Deploying and Monitoring Elastic Stack on GCP
A presentation at Google Cloud Next ’19 UK in in London, UK by David Pilato
DZ105-Deploying and Monitoring Elastic Stack on GCP Speaker: David Pilato
David Pilato Developer | Evangelist Elastic @dadoonet https://speaker.pilato.fr/
Deploying on Google Cloud Platform 01
Deploying beats
Deploying beats Elasticsearch Master Nodes (3) Ingest Nodes (X) Data Nodes Hot (X) Data Notes Warm (X)
Deploying beats Elasticsearch Master Nodes (3) Auditbeat Ingest Nodes (X) Packetbeat Data Nodes Hot (X) Metricbeat Data Notes Warm (X) Filebeat
Deploying beats Elasticsearch Master Nodes (3) Auditbeat Auditbeat Ingest Nodes (X) Packetbeat Packetbeat Data Nodes Hot (X) Metricbeat Metricbeat Data Notes Warm (X) Filebeat Filebeat
Deploying beats Elasticsearch Master Nodes (3) Heartbeat Auditbeat Auditbeat Ingest Nodes (X) Packetbeat Packetbeat Data Nodes Hot (X) Metricbeat Metricbeat Data Notes Warm (X) Filebeat Filebeat
Deploying beats Elasticsearch Master Nodes (3) Heartbeat Heartbeat Auditbeat Auditbeat Ingest Nodes (X) Packetbeat Packetbeat Data Nodes Hot (X) Metricbeat Metricbeat Data Notes Warm (X) Filebeat Filebeat
Elastic Platform Beats Elasticsearch Logstash Log Files Wire Data Master Nodes (3) Ingest Nodes (X) Metrics your{beat} Kafka Nodes (X) Redis Data Store Web APIs Social Sensors Messaging Queue Kibana Instances (X) Data Nodes Hot (X) Data Notes Warm (X) Queues Storage Metrics Notification
Elastic Platform Beats Elasticsearch Logstash Log Files Wire Data Master Nodes (3) Ingest Nodes (X) Metrics your{beat} Kafka Nodes (X) Redis Data Store Web APIs Social Sensors Messaging Queue Kibana Instances (X) Data Nodes Hot (X) Data Notes Warm (X) Queues Storage Metrics Notification
Elasticsearch manual deployment 02
Manual deployment Create the first machine $ gcloud beta compute instances create \ gnext-dadoonet-1 \ —machine-type=n1-standard-1 \ —image=debian-9-stretch-v20191115 \ —image-project=debian-cloud \ —boot-disk-size=10GB \ —boot-disk-type=pd-standard \ —boot-disk-device-name=gnext-dadoonet-1
Manual deployment Create the second machine. For High Avaibility. $ gcloud beta compute instances create \ gnext-dadoonet-2 \ —machine-type=n1-standard-1 \ —image=debian-9-stretch-v20191115 \ —image-project=debian-cloud \ —boot-disk-size=10GB \ —boot-disk-type=pd-standard \ —boot-disk-device-name=gnext-dadoonet-2
Manual deployment Create the third machine. To avoid split brain issues. $ gcloud beta compute instances create \ gnext-dadoonet-3 \ —machine-type=n1-standard-1 \ —image=debian-9-stretch-v20191115 \ —image-project=debian-cloud \ —boot-disk-size=10GB \ —boot-disk-type=pd-standard \ —boot-disk-device-name=gnext-dadoonet-3
$ gcloud beta compute instances list NAME INTERNAL_IP STATUS gnext-dadoonet-1 10.154.0.2 RUNNING gnext-dadoonet-2 10.154.0.3 RUNNING gnext-dadoonet-3 10.154.0.4 RUNNING Manual deployment Check that all machines are running
SSH to the instance $ gcloud beta compute ssh gnext-dadoonet-1
Manual deployment Install Elasticsearch
Install Elasticsearch $ wget -qO - \ https://artifacts.elastic.co/GPG-KEY-elasticsearch | \ sudo apt-key add – $ sudo apt-get install apt-transport-https $ echo \ “deb https://artifacts.elastic.co/packages/7.x/apt stable main”|\ sudo tee -a /etc/apt/sources.list.d/elastic-7.x.list $ sudo apt-get update && sudo apt-get install elasticsearch # Automatic restart $ sudo /bin/systemctl daemon-reload $ sudo /bin/systemctl enable elasticsearch.service
Generate a certificate and a private key $ sudo /usr/share/elasticsearch/bin/elasticsearch-certutil ca $ sudo /usr/share/elasticsearch/bin/elasticsearch-certutil cert \ —ca elastic-stack-ca.p12 # Copy certificates /usr/share/elasticsearch/*.p12 on each node # In /etc/default/elasticsearch/certs
Manual deployment On the first node $ sudo mkdir /etc/elasticsearch/certs $ sudo cp /usr/share/elasticsearch/*.p12 /etc/elasticsearch/certs $ sudo chmod g+rw /etc/elasticsearch/certs/elasticcertificates.p12
Manual deployment Setup elasticsearch
Configure Elasticsearch $ sudo vi /etc/elasticsearch/elasticsearch.yml network.host: site discovery.seed_hosts: - 10.154.0.2 - 10.154.0.3 - 10.154.0.4 xpack.security.enabled: true xpack.security.transport.ssl.enabled: true xpack.security.transport.ssl.verification_mode: certificate xpack.security.transport.ssl.keystore.path: certs/elasticcertificates.p12 xpack.security.transport.ssl.truststore.path: certs/elasticcertificates.p12 # Optionally # xpack.security.http.ssl.enabled: true # xpack.security.http.ssl.keystore.path: certs/elasticcertificates.p12 # xpack.security.http.ssl.truststore.path: certs/elasticcertificates.p12
Start Elasticsearch $ sudo systemctl start elasticsearch.service # Check logs $ sudo tail -f /var/log/elasticsearch/elasticsearch.log [INFO ][o.e.n.Node
Manual deployment Start elasticsearch ] [gnext-dadoonet-1] started
Repeat on the 2 other nodes
$ sudo /usr/share/elasticsearch/bin/elasticsearch-setup-passwords interactive Manual deployment Generate stack users
Manual deployment Check the cluster
Check the cluster $ curl “10.154.0.2:9200/_cat/nodes?v&h=ip,node.role,master,name”\ -u elastic Enter host password for user ‘elastic’: ip node.role master name 10.154.0.3 dilm gnext-dadoonet-2 10.154.0.4 dilm gnext-dadoonet-3 10.154.0.2 dilm * gnext-dadoonet-1
03 Use discovery-gce plugin
Install GCE Discovery plugin $ sudo /usr/share/elasticsearch/bin/elasticsearch-plugin install discovery-gce
With discovery-gce plugin Install the plugin
Change elasticsearch settings $ sudo vi /etc/elasticsearch/elasticsearch.yml # Remove the following lines # discovery.seed_hosts: # - 10.154.0.2 # - 10.154.0.3 # - 10.154.0.4 # And add cloud.gce.project_id: elastic-community cloud.gce.zone: europe-west2-c discovery.seed_providers: gce
Add compute-rw scope for each $ gcloud beta compute instances $ gcloud beta compute instances —scopes=default,compute-rw $ gcloud beta compute instances
machine stop gnext-dadoonet-1 set-scopes gnext-dadoonet-1 \ start gnext-dadoonet-1 With discovery-gce plugin $ gcloud beta compute instances stop gnext-dadoonet-2 $ gcloud beta compute instances set-scopes gnext-dadoonet-2 \ —scopes=default,compute-rw $ gcloud beta compute instances start gnext-dadoonet-2 Stop the VMs and add compute-rw rights $ gcloud beta compute instances stop gnext-dadoonet-3 $ gcloud beta compute instances set-scopes gnext-dadoonet-3 \ —scopes=default,compute-rw $ gcloud beta compute instances start gnext-dadoonet-3
With discovery-gce plugin Check the cluster
Check the cluster $ curl “10.154.0.2:9200/_cat/nodes?v&h=ip,node.role,master,name”\ -u elastic Enter host password for user ‘elastic’: ip node.role master name 10.154.0.3 dilm gnext-dadoonet-2 10.154.0.4 dilm gnext-dadoonet-3 10.154.0.2 dilm * gnext-dadoonet-1
Elastic Platform Beats Elasticsearch Logstash Log Files Wire Data Master Nodes (3) Ingest Nodes (X) Metrics your{beat} Kafka Nodes (X) Redis Data Store Web APIs Social Sensors Messaging Queue Kibana Instances (X) Data Nodes Hot (X) Data Notes Warm (X) Queues Storage Metrics Notification
04 Kibana manual deployment
Create firewall rules $ gcloud beta compute firewall-rules create kibana \ —direction=INGRESS —priority=1000 \ —network=default —action=ALLOW —rules=tcp:5601 —source-ranges=0.0.0.0/0 —target-tags=kibana
Kibana deployment Create Kibana VM
Create Kibana GCE Instance $ gcloud beta compute instances create \ gnext-dadoonet-k \ —machine-type=n1-standard-1 \ —tags kibana \ —image=debian-9-stretch-v20191115 \ —image-project=debian-cloud \ —boot-disk-size=10GB \ —boot-disk-type=pd-standard \ —boot-disk-device-name=gnext-dadoonet-k
SSH to the instance $ gcloud beta compute ssh gnext-dadoonet-k
Manual deployment Install Kibana
Install Kibana $ wget -qO - \ https://artifacts.elastic.co/GPG-KEY-elasticsearch | \ sudo apt-key add $ sudo apt-get install apt-transport-https $ echo \ “deb https://artifacts.elastic.co/packages/7.x/apt stable main”|\ sudo tee -a /etc/apt/sources.list.d/elastic-7.x.list $ sudo apt-get update && sudo apt-get install kibana # Automatic restart $ sudo /bin/systemctl daemon-reload $ sudo /bin/systemctl enable kibana.service
Manual deployment Setup and Start Kibana
Configure Kibana $ sudo vi /etc/kibana/kibana.yml server.host: “10.154.0.5” elasticsearch.hosts: [“http://10.154.0.2:9200”, “http://10.154.0.3:9200”, “http://10.154.0.4:9200”] xpack.security.encryptionKey: “something_at_least_32_characters_gnext-dadoonet-k_gnextdadoonet-k” elasticsearch.username: “kibana” elasticsearch.password: “my_incredible_password” # Start Kibana sudo systemctl start kibana.service # Check logs sudo journalctl —unit kibana -f
Manual deployment Open http://35.197.195.120:5601/
Manual deployment Check with Dev Tools GET /
Elastic Platform Beats Elasticsearch Logstash Log Files Wire Data Master Nodes (3) Ingest Nodes (X) Metrics your{beat} Kafka Nodes (X) Redis Data Store Web APIs Social Sensors Messaging Queue Kibana Instances (X) Data Nodes Hot (X) Data Notes Warm (X) Queues Storage Metrics Notification
Using ECK 05
$ gcloud beta container clusters create “gnext-dadoonet” \ —zone “europe-west2-c” —machine-type “n1-standard-4” Kubernetes deployment Create Kubernetes cluster $ gcloud beta container clusters get-credentials gnext-dadoonet \ —zone europe-west2-c
Install the operator $ kubectl apply -f https://download.elastic.co/downloads/eck/1.0.0-beta1/all-inone.yaml
Kubernetes deployment Deploy ECK in Kubernetes
Monitor the operator logs $ kubectl -n elastic-system logs -f statefulset.apps/elasticoperator
Kubernetes deployment Deploy an Elasticsearch cluster
Deploy 3 elasticsearch nodes $ cat <<EOF | kubectl apply -f apiVersion: elasticsearch.k8s.elastic.co/v1beta1 kind: Elasticsearch metadata: name: elasticsearch-gnext spec: version: 7.4.2 nodeSets: - name: default count: 3 config: node.store.allow_mmap: false EOF # Check if it’s running and ready $ kubectl get elasticsearch NAME HEALTH NODES VERSION PHASE elasticsearch-gnext green 3 7.4.2 Ready # Check Elasticsearch logs from 1st node $ kubectl logs -f elasticsearch-gnext-es-default-0
AGE 1m
Get the credentials $ PASSWORD=$(kubectl get secret elasticsearch-gnext-es-elasticuser -o=jsonpath=’{.data.elastic}’ | base64 —decode)
Kubernetes deployment
Connect to elasticsearch from your workstation $ kubectl port-forward service/elasticsearch-gnext-es-http 9200 $ curl -u “elastic:$PASSWORD” -k “https://elasticsearch-gnext-eshttp:9200” { “name” : “elasticsearch-gnext-es-default-0”, “cluster_name” : “elasticsearch-gnext”, “cluster_uuid” : “e4nry98BTjCsLSZASIcrRg”, “version” : { … }, “tagline” : “You Know, for Search”
Connect to an Elasticsearch cluster }
Kubernetes deployment Deploy a Kibana instance
Deploy Kibana $ cat <<EOF | kubectl apply -f apiVersion: kibana.k8s.elastic.co/v1beta1 kind: Kibana metadata: name: kibana-gnext spec: version: 7.4.2 count: 1 elasticsearchRef: name: elasticsearch-gnext EOF # Check until it’s running $ kubectl get kibana
Get the elastic password $ kubectl get secret elasticsearch-gnext-es-elastic-user o=jsonpath=’{.data.elastic}’ | base64 —decode; echo
Kubernetes deployment Connect to Kibana
Connect to Kibana from your workstation $ kubectl port-forward service/kibana-gnext-kb-http 5601
Manual deployment Open https://localhost:5601
Manual deployment Check with Dev Tools GET / Check nodes
06 One click install
From Google Cloud Marketplace
Or https://cloud.elastic.co/
One click deploy
One click deploy
DZ105-Deploying and Monitoring Elastic Stack on GCP Speaker: David Pilato
To deploy elastic stack on Google Compute Platform (GCP), you have several options:
- Start GCE instances, install and configure elasticsearch nodes discovery, SSL and security
- Do the same but install
discovery-gceplugin to simplify nodes discovery - Use Elastic Cloud Enterprise and deploy it on GCE instances
- Use Elastic Cloud Kubernetes operator on GKE
Or, you can let Elastic, the company behind elastic stack, deploy and manage it for you with cloud.elastic.co
Resources
The following resources were mentioned during the presentation or are useful additional information.
Buzz and feedback
Here’s what was said about this presentation on social media.
