DZ105-Deploying and Monitoring Elastic Stack on GCP Speaker: David Pilato

David Pilato Developer | Evangelist Elastic @dadoonet https://speaker.pilato.fr/

Deploying on Google Cloud Platform 01

Deploying beats

Deploying beats Elasticsearch Master Nodes (3) Ingest Nodes (X) Data Nodes Hot (X) Data Notes Warm (X)

Deploying beats Elasticsearch Master Nodes (3) Auditbeat Ingest Nodes (X) Packetbeat Data Nodes Hot (X) Metricbeat Data Notes Warm (X) Filebeat

Deploying beats Elasticsearch Master Nodes (3) Auditbeat Auditbeat Ingest Nodes (X) Packetbeat Packetbeat Data Nodes Hot (X) Metricbeat Metricbeat Data Notes Warm (X) Filebeat Filebeat

Deploying beats Elasticsearch Master Nodes (3) Heartbeat Auditbeat Auditbeat Ingest Nodes (X) Packetbeat Packetbeat Data Nodes Hot (X) Metricbeat Metricbeat Data Notes Warm (X) Filebeat Filebeat

Deploying beats Elasticsearch Master Nodes (3) Heartbeat Heartbeat Auditbeat Auditbeat Ingest Nodes (X) Packetbeat Packetbeat Data Nodes Hot (X) Metricbeat Metricbeat Data Notes Warm (X) Filebeat Filebeat

Elastic Platform Beats Elasticsearch Logstash Log Files Wire Data Master Nodes (3) Ingest Nodes (X) Metrics your{beat} Kafka Nodes (X) Redis Data Store Web APIs Social Sensors Messaging Queue Kibana Instances (X) Data Nodes Hot (X) Data Notes Warm (X) Queues Storage Metrics Notification

Elasticsearch manual deployment 02

Manual deployment Create the first machine $ gcloud beta compute instances create \ gnext-dadoonet-1 \ —machine-type=n1-standard-1 \ —image=debian-9-stretch-v20191115 \ —image-project=debian-cloud \ —boot-disk-size=10GB \ —boot-disk-type=pd-standard \ —boot-disk-device-name=gnext-dadoonet-1

Manual deployment Create the second machine. For High Avaibility. $ gcloud beta compute instances create \ gnext-dadoonet-2 \ —machine-type=n1-standard-1 \ —image=debian-9-stretch-v20191115 \ —image-project=debian-cloud \ —boot-disk-size=10GB \ —boot-disk-type=pd-standard \ —boot-disk-device-name=gnext-dadoonet-2

Manual deployment Create the third machine. To avoid split brain issues. $ gcloud beta compute instances create \ gnext-dadoonet-3 \ —machine-type=n1-standard-1 \ —image=debian-9-stretch-v20191115 \ —image-project=debian-cloud \ —boot-disk-size=10GB \ —boot-disk-type=pd-standard \ —boot-disk-device-name=gnext-dadoonet-3

$ gcloud beta compute instances list NAME INTERNAL_IP STATUS gnext-dadoonet-1 10.154.0.2 RUNNING gnext-dadoonet-2 10.154.0.3 RUNNING gnext-dadoonet-3 10.154.0.4 RUNNING Manual deployment Check that all machines are running

SSH to the instance $ gcloud beta compute ssh gnext-dadoonet-1

Manual deployment Install Elasticsearch

Install Elasticsearch $ wget -qO - \ https://artifacts.elastic.co/GPG-KEY-elasticsearch | \ sudo apt-key add – $ sudo apt-get install apt-transport-https $ echo \ “deb https://artifacts.elastic.co/packages/7.x/apt stable main”|\ sudo tee -a /etc/apt/sources.list.d/elastic-7.x.list $ sudo apt-get update && sudo apt-get install elasticsearch # Automatic restart $ sudo /bin/systemctl daemon-reload $ sudo /bin/systemctl enable elasticsearch.service

Generate a certificate and a private key $ sudo /usr/share/elasticsearch/bin/elasticsearch-certutil ca $ sudo /usr/share/elasticsearch/bin/elasticsearch-certutil cert \ —ca elastic-stack-ca.p12 # Copy certificates /usr/share/elasticsearch/*.p12 on each node # In /etc/default/elasticsearch/certs

Manual deployment On the first node $ sudo mkdir /etc/elasticsearch/certs $ sudo cp /usr/share/elasticsearch/*.p12 /etc/elasticsearch/certs $ sudo chmod g+rw /etc/elasticsearch/certs/elasticcertificates.p12

Manual deployment Setup elasticsearch

Configure Elasticsearch $ sudo vi /etc/elasticsearch/elasticsearch.yml network.host: site discovery.seed_hosts: - 10.154.0.2 - 10.154.0.3 - 10.154.0.4 xpack.security.enabled: true xpack.security.transport.ssl.enabled: true xpack.security.transport.ssl.verification_mode: certificate xpack.security.transport.ssl.keystore.path: certs/elasticcertificates.p12 xpack.security.transport.ssl.truststore.path: certs/elasticcertificates.p12 # Optionally # xpack.security.http.ssl.enabled: true # xpack.security.http.ssl.keystore.path: certs/elasticcertificates.p12 # xpack.security.http.ssl.truststore.path: certs/elasticcertificates.p12

Start Elasticsearch $ sudo systemctl start elasticsearch.service # Check logs $ sudo tail -f /var/log/elasticsearch/elasticsearch.log [INFO ][o.e.n.Node

Manual deployment Start elasticsearch ] [gnext-dadoonet-1] started

Repeat on the 2 other nodes

$ sudo /usr/share/elasticsearch/bin/elasticsearch-setup-passwords interactive Manual deployment Generate stack users

Manual deployment Check the cluster

Check the cluster $ curl “10.154.0.2:9200/_cat/nodes?v&h=ip,node.role,master,name”\ -u elastic Enter host password for user ‘elastic’: ip node.role master name 10.154.0.3 dilm gnext-dadoonet-2 10.154.0.4 dilm gnext-dadoonet-3 10.154.0.2 dilm * gnext-dadoonet-1

03 Use discovery-gce plugin

Install GCE Discovery plugin $ sudo /usr/share/elasticsearch/bin/elasticsearch-plugin install discovery-gce

With discovery-gce plugin Install the plugin

Change elasticsearch settings $ sudo vi /etc/elasticsearch/elasticsearch.yml # Remove the following lines # discovery.seed_hosts: # - 10.154.0.2 # - 10.154.0.3 # - 10.154.0.4 # And add cloud.gce.project_id: elastic-community cloud.gce.zone: europe-west2-c discovery.seed_providers: gce

Add compute-rw scope for each $ gcloud beta compute instances $ gcloud beta compute instances —scopes=default,compute-rw $ gcloud beta compute instances

machine stop gnext-dadoonet-1 set-scopes gnext-dadoonet-1 \ start gnext-dadoonet-1 With discovery-gce plugin $ gcloud beta compute instances stop gnext-dadoonet-2 $ gcloud beta compute instances set-scopes gnext-dadoonet-2 \ —scopes=default,compute-rw $ gcloud beta compute instances start gnext-dadoonet-2 Stop the VMs and add compute-rw rights $ gcloud beta compute instances stop gnext-dadoonet-3 $ gcloud beta compute instances set-scopes gnext-dadoonet-3 \ —scopes=default,compute-rw $ gcloud beta compute instances start gnext-dadoonet-3

With discovery-gce plugin Check the cluster

Check the cluster $ curl “10.154.0.2:9200/_cat/nodes?v&h=ip,node.role,master,name”\ -u elastic Enter host password for user ‘elastic’: ip node.role master name 10.154.0.3 dilm gnext-dadoonet-2 10.154.0.4 dilm gnext-dadoonet-3 10.154.0.2 dilm * gnext-dadoonet-1

Elastic Platform Beats Elasticsearch Logstash Log Files Wire Data Master Nodes (3) Ingest Nodes (X) Metrics your{beat} Kafka Nodes (X) Redis Data Store Web APIs Social Sensors Messaging Queue Kibana Instances (X) Data Nodes Hot (X) Data Notes Warm (X) Queues Storage Metrics Notification

04 Kibana manual deployment

Create firewall rules $ gcloud beta compute firewall-rules create kibana \ —direction=INGRESS —priority=1000 \ —network=default —action=ALLOW —rules=tcp:5601 —source-ranges=0.0.0.0/0 —target-tags=kibana

Kibana deployment Create Kibana VM

Create Kibana GCE Instance $ gcloud beta compute instances create \ gnext-dadoonet-k \ —machine-type=n1-standard-1 \ —tags kibana \ —image=debian-9-stretch-v20191115 \ —image-project=debian-cloud \ —boot-disk-size=10GB \ —boot-disk-type=pd-standard \ —boot-disk-device-name=gnext-dadoonet-k

SSH to the instance $ gcloud beta compute ssh gnext-dadoonet-k

Manual deployment Install Kibana

Install Kibana $ wget -qO - \ https://artifacts.elastic.co/GPG-KEY-elasticsearch | \ sudo apt-key add $ sudo apt-get install apt-transport-https $ echo \ “deb https://artifacts.elastic.co/packages/7.x/apt stable main”|\ sudo tee -a /etc/apt/sources.list.d/elastic-7.x.list $ sudo apt-get update && sudo apt-get install kibana # Automatic restart $ sudo /bin/systemctl daemon-reload $ sudo /bin/systemctl enable kibana.service

Manual deployment Setup and Start Kibana

Configure Kibana $ sudo vi /etc/kibana/kibana.yml server.host: “10.154.0.5” elasticsearch.hosts: [“http://10.154.0.2:9200”, “http://10.154.0.3:9200”, “http://10.154.0.4:9200”] xpack.security.encryptionKey: “something_at_least_32_characters_gnext-dadoonet-k_gnextdadoonet-k” elasticsearch.username: “kibana” elasticsearch.password: “my_incredible_password” # Start Kibana sudo systemctl start kibana.service # Check logs sudo journalctl —unit kibana -f

Manual deployment Open http://35.197.195.120:5601/

Manual deployment Check with Dev Tools GET /

Elastic Platform Beats Elasticsearch Logstash Log Files Wire Data Master Nodes (3) Ingest Nodes (X) Metrics your{beat} Kafka Nodes (X) Redis Data Store Web APIs Social Sensors Messaging Queue Kibana Instances (X) Data Nodes Hot (X) Data Notes Warm (X) Queues Storage Metrics Notification

Using ECK 05

$ gcloud beta container clusters create “gnext-dadoonet” \ —zone “europe-west2-c” —machine-type “n1-standard-4” Kubernetes deployment Create Kubernetes cluster $ gcloud beta container clusters get-credentials gnext-dadoonet \ —zone europe-west2-c

Install the operator $ kubectl apply -f https://download.elastic.co/downloads/eck/1.0.0-beta1/all-inone.yaml

Kubernetes deployment Deploy ECK in Kubernetes

Monitor the operator logs $ kubectl -n elastic-system logs -f statefulset.apps/elasticoperator

Kubernetes deployment Deploy an Elasticsearch cluster

Deploy 3 elasticsearch nodes $ cat <<EOF | kubectl apply -f apiVersion: elasticsearch.k8s.elastic.co/v1beta1 kind: Elasticsearch metadata: name: elasticsearch-gnext spec: version: 7.4.2 nodeSets: - name: default count: 3 config: node.store.allow_mmap: false EOF # Check if it’s running and ready $ kubectl get elasticsearch NAME HEALTH NODES VERSION PHASE elasticsearch-gnext green 3 7.4.2 Ready # Check Elasticsearch logs from 1st node $ kubectl logs -f elasticsearch-gnext-es-default-0

AGE 1m

Get the credentials $ PASSWORD=$(kubectl get secret elasticsearch-gnext-es-elasticuser -o=jsonpath=’{.data.elastic}’ | base64 —decode)

Kubernetes deployment

Connect to elasticsearch from your workstation $ kubectl port-forward service/elasticsearch-gnext-es-http 9200 $ curl -u “elastic:$PASSWORD” -k “https://elasticsearch-gnext-eshttp:9200” { “name” : “elasticsearch-gnext-es-default-0”, “cluster_name” : “elasticsearch-gnext”, “cluster_uuid” : “e4nry98BTjCsLSZASIcrRg”, “version” : { … }, “tagline” : “You Know, for Search”

Connect to an Elasticsearch cluster }

Kubernetes deployment Deploy a Kibana instance

Deploy Kibana $ cat <<EOF | kubectl apply -f apiVersion: kibana.k8s.elastic.co/v1beta1 kind: Kibana metadata: name: kibana-gnext spec: version: 7.4.2 count: 1 elasticsearchRef: name: elasticsearch-gnext EOF # Check until it’s running $ kubectl get kibana

Get the elastic password $ kubectl get secret elasticsearch-gnext-es-elastic-user o=jsonpath=’{.data.elastic}’ | base64 —decode; echo

Kubernetes deployment Connect to Kibana

Connect to Kibana from your workstation $ kubectl port-forward service/kibana-gnext-kb-http 5601

Manual deployment Open https://localhost:5601

Manual deployment Check with Dev Tools GET / Check nodes

06 One click install

From Google Cloud Marketplace

Or https://cloud.elastic.co/

One click deploy

DZ105-Deploying and Monitoring Elastic Stack on GCP Speaker: David Pilato