Les Vendredis noirs : même pas peur !
A presentation at Devopsdays Geneva 2019 in in Geneva, Switzerland by David Pilato
Les vendredi noirs ? Même pas peur ! David Pilato Developer | Evangelist, @dadoonet
Data Platform Architectures
The Elastic Journey of Data Beats Log Files Wire Data Metrics your{beat} !3 @dadoonet sli.do/elastic
The Elastic Journey of Data Beats Log Files Elasticsearch Master Nodes (3) Wire Data Ingest Nodes (X) Metrics your{beat} Data Nodes Hot (X) Data Notes Warm (X) !4 @dadoonet sli.do/elastic
The Elastic Journey of Data Beats Log Files Elasticsearch Master Nodes (3) Wire Data Ingest Nodes (X) Metrics your{beat} Data Nodes Hot (X) Data Notes Warm (X) !5 @dadoonet sli.do/elastic Kibana Instances (X)
The Elastic Journey of Data Beats Elasticsearch Logstash Log Files Wire Data Master Nodes (3) Ingest Nodes (X) Metrics your{beat} Nodes (X) Data Nodes Hot (X) Data Notes Warm (X) !6 @dadoonet sli.do/elastic Kibana Instances (X)
The Elastic Journey of Data Beats Elasticsearch Logstash Log Files Wire Data Master Nodes (3) Ingest Nodes (X) Metrics your{beat} !7 Data Store Web APIs Social Sensors Nodes (X) Data Nodes Hot (X) Data Notes Warm (X) @dadoonet sli.do/elastic Kibana Instances (X)
The Elastic Journey of Data Beats Elasticsearch Logstash Log Files Wire Data Master Nodes (3) Kibana Ingest Nodes (X) Metrics your{beat} Data Store Web APIs Social Sensors Nodes (X) Data Nodes Hot (X) Data Notes Warm (X) Queues !8 Instances (X) @dadoonet Storage Metrics Notification sli.do/elastic
The Elastic Journey of Data Beats Elasticsearch Logstash Log Files Wire Data Master Nodes (3) Kibana Ingest Nodes (X) Metrics your{beat} Kafka Nodes (X) Redis Messaging Queue Data Store Web APIs Social Sensors Data Nodes Hot (X) Data Notes Warm (X) Queues !9 Instances (X) @dadoonet Storage Metrics Notification sli.do/elastic
App Search Site Search Enterprise Search Logging APM Business Analytics Security Analytics Metrics Elastic Stack Future Solutions Kibana Visualize & Manage Elasticsearch Store, Search, & Analyze Beats SaaS Elastic Cloud Logstash Ingest Self Managed Standalone Elastic Cloud Enterprise Deployment
Elasticsearch Cluster Sizing
Terminology Cluster my_cluster Server 1 Node A d1 d3 d6 d2 d4 d9 d12 d11 d7 d8 d5 d3 d1 d10 d4 Index twitter d6 d2 d5 Index logs !12 @dadoonet sli.do/elastic
Partition Cluster my_cluster Server 1 d1 4 d3 3 Node A d2 d4 d9 d12 2 Shards d6 d11 d7 d8 d5 d10 0 1 1 d3 d1 d4 Index twitter d6 d2 d5 0 Index logs !13 @dadoonet sli.do/elastic
Distribution Cluster my_cluster Server 1 Server 2 twitter shard P4 d1 d6 Node B Node A twitter shard P1 d2 d5 d3 d10 d11 twitter shard P0 d7 d8 d4 d9 twitter shard P3 d12 twitter shard P2 !14 d3 d1 @dadoonet d6 logs shard P0 d2 d4 d5 logs shard P1 sli.do/elastic
Replication • Primaries • Replicas Cluster my_cluster Server 1 Server 2 twitter shard P4 Node B d1 d6 d2 d11 d7 d3 twitter shard R3 d9 d12 twitter shard P2 !15 twitter shard P1 d5 d2 d2 d11 d5 d1 @dadoonet d6 d8 d9 logs shard R0 twitter shard P3 logs shard P0 twitter shard R2 twitter shard P0 d7 d6 d2 d4 d5 d10 d3 d1 twitter shard R1 d12 d3 Node A d4 twitter shard R0 logs shard R1 d4 d6 d3 d10 d8 d4 twitter shard R4 d1 d5 logs shard P1 sli.do/elastic
Replication Server 3 twitter shard R4 d1 d6 Server 2 Node C twitter shard P4 twitter shard P1 d2 d10 twitter shard R3 d9 twitter shard P2 !16 d6 Node A d2 d8 d4 d2 d3 d1 logs shard P1 @dadoonet logs shard R0 twitter shard P3 twitter shard R1 d12 d5 d5 d8 d9 d2 d6 logs shard P0 twitter shard R2 sli.do/elastic twitter shard P0 d7 d4 twitter shard R0 logs shard R1 d4 d11 d3 d7 d3 d4 Node B d1 d11 d5 d12 • Primaries • Replicas Cluster my_cluster Server 1 d5 d10 d1 d3 d6
Scaling Data !17 @dadoonet sli.do/elastic
Scaling Data !18 @dadoonet sli.do/elastic
Scaling Data !19 @dadoonet sli.do/elastic
Scaling Big Data … !20 @dadoonet … sli.do/elastic
Scaling Big Data … … • In Elasticsearch, shards are the working unit • More data -> More shards But how many shards? !21 @dadoonet sli.do/elastic
How much data? • ~1000 events per second • 60s * 60m * 24h * 1000 events => ~87M events per day • 1kb per event => ~82GB per day • 3 months => ~7TB !22 @dadoonet sli.do/elastic
Shard Size • It depends on many different factors ‒ document size, mapping, use case, kinds of queries being executed, desired response time, peak indexing rate, budget, … • After the shard sizing*, each shard should handle 45GB • Up to 10 shards per machine
- https://www.elastic.co/elasticon/conf/2016/sf/quantitative-cluster-sizing !23 @dadoonet sli.do/elastic
How many shards? • Data size: ~7TB • Shards per machine: 10* • Shard Size: ~45GB* • Total Servers: 16 • Total Shards: ~160 3 months of logs Cluster my_cluster … * https://www.elastic.co/elasticon/conf/2016/sf/quantitative-cluster-sizing !24 @dadoonet sli.do/elastic
But… • How many indices? • What do you do if the daily data grows? • What do you do if you want to delete old data? !25 @dadoonet sli.do/elastic
Time-Based Data • Logs, social media streams, time-based events • Timestamp + Data • Do not change • Typically search for recent events • Older documents become less important • Hard to predict the data size !26 @dadoonet sli.do/elastic
Time-Based Data • Time-based Indices is the best option ‒ create a new index each day, week, month, year, … ‒ search the indices you need in the same request !27 @dadoonet sli.do/elastic
Daily Indices Cluster my_cluster d3 d1 d4 d6 d2 d5 logs-2018-04-10 !28 @dadoonet sli.do/elastic
Daily Indices Cluster my_cluster d3 d1 d4 d6 d2 d5 logs-2018-04-10 d3 d1 d4 d6 d2 d5 logs-2018-04-11 !29 @dadoonet sli.do/elastic
Daily Indices Cluster my_cluster d3 d1 d4 d6 d2 d5 logs-2018-04-10 d3 d1 d4 d6 d2 d5 logs-2018-04-11 d3 d1 d4 d6 d2 d5 logs-2018-04-12 !30 @dadoonet sli.do/elastic
Templates • Every new created index starting with ‘logs-’ will have ‒ 2 shards ‒ 1 replica (for each primary shard) ‒ 60 seconds refresh interval PUT _template/logs { “template”: “logs-*”, “settings”: { “number_of_shards”: 2, “number_of_replicas”: 1, “refresh_interval”: “60s” } } More on that later !31 @dadoonet
Alias Cluster my_cluster d3 d1 d4 logs-write logs-2018-04-10 users @dadoonet d2 d5 logs-read Application !32 d6 sli.do/elastic
Alias Cluster my_cluster d3 d1 d4 logs-write d2 d5 logs-2018-04-10 d3 logs-read Application d6 d1 d4 d6 d2 d5 logs-2018-04-11 users !33 @dadoonet sli.do/elastic
Alias Cluster my_cluster d3 d1 d4 logs-write d2 d5 logs-2018-04-10 d3 logs-read Application d6 d1 d4 d6 d2 d5 logs-2018-04-11 d3 users d1 d4 d6 d2 d5 logs-2018-04-12 !34 @dadoonet sli.do/elastic
Detour: Rollover API https://www.elastic.co/guide/en/elasticsearch/reference/6.3/indices-rollover-index.html
Do not Overshard don’t keep default values! Cluster my_cluster d3 d6 d1 • 3 different logs d2 d4 d5 • 1 index per day each access-… • 1GB each d5 • 5 shards (default): so 200mb / shard vs 45gb d1 d7 d6 d9 d5 • 6 months retention application-… • ~900 shards for ~180GB d59 • we needed ~4 shards! d0 d4 d10 d3 d5 mysql-… !36 @dadoonet sli.do/elastic
Scaling Big Data 1M users … … But what happens if we have 2M users? !37 @dadoonet sli.do/elastic
Scaling Big Data !38 1M users … … 1M users … … @dadoonet sli.do/elastic
Scaling Big Data !39 1M users … … 1M users … … 1M users … … @dadoonet sli.do/elastic
Scaling Big Data U s e r s !40 … … … … … … @dadoonet sli.do/elastic
Shards are the working unit • Primaries ‒ More data -> More shards ‒ write throughput (More writes -> More primary shards) • Replicas ‒ high availability (1 replica is the default) ‒ read throughput (More reads -> More replicas) !41 @dadoonet sli.do/elastic
Detour: Shrink API https://www.elastic.co/guide/en/elasticsearch/reference/6.3/indices-shrink-index.html
Detour: Split API https://www.elastic.co/guide/en/elasticsearch/reference/6.3/indices-split-index.html
Optimal Bulk Size
What is Bulk? 1000 index requests with 1 document __________ __________ __________ __________ __________ __________ __________ __________ __________ __________ __________ __________ __________ __________ __________ __________ __________ __________ __________ __________ __________ __________ __________ __________ __________ __________ __________ __________ __________ __________ __________ __________ __________ __________ __________ __________ __________ __________ __________ __________ _____ Master Nodes (3) Ingest Nodes (X) Beats Logstash Application Data Nodes Hot (X) 1000 log events 1 bulk request with 1000 documents !45 Elasticsearch @dadoonet sli.do/elastic Data Notes Warm (X)
What is the optimal bulk size? Elasticsearch
Beats Logstash Application 1000 log events !46 @dadoonet 4 * 250? 2 * 500? 1 * 1000? Master Nodes (3) Ingest Nodes (X) Data Nodes Hot (X) Data Notes Warm (X) sli.do/elastic
It depends… • on your application (language, libraries, …) • document size (100b, 1kb, 100kb, 1mb, …) • number of nodes • node size • number of shards • shards distribution !47 @dadoonet sli.do/elastic
Test it ;) Elasticsearch
1000000 log events 4000 * Beats Logstash Application 2000 * 250-> 160s Master Nodes (3) 500-> 164s Ingest Nodes (X) 1000 * 1000-> 155s Data Nodes Hot (X) Data Notes Warm (X) !48 @dadoonet sli.do/elastic
Test it ;) input { stdin{} } filter {} output { elasticsearch { hosts => [“10.12.145.189”] flush_size => “${SIZE}” } }
In Beats set “bulk_max_size” in the output.elasticsearch
DATE=date +%Y.%m.%d LOG=logs/logs.txt exec_test () { curl -s -XDELETE “http://USER:PASS@HOST:9200/logstash-$DATE” sleep 10 export SIZE=$1 time cat $LOG | ./bin/logstash -f logstash.conf } for SIZE in 100 500 1000 3000 5000 10000; do for i in {1..20}; do exec_test $SIZE done; done; !49
@dadoonet
sli.do/elastic
Test it ;) • 2 node cluster (m3.large) ‒ 2 vCPU, 7.5GB Memory, 1x32GB SSD • 1 index server (m3.large) ‒ logstash ‒ kibana !50
docs
100 500 1000 3000 5000 10000 time(s) 191.7 161.9 163.5 160.7 160.7 161.5 @dadoonet
Distribute the Load
Avoid Bottlenecks Elasticsearch single node _________ _________ _________ _________ _________ _________ _________ _________ _________ _________ _________ _________ _________ _________ _________ _________ _________ _________ _________ _________ _________ _________ _________ _________ _________ _________ _________ _________ _________ _________ _________ _________ _________ _________ _________ Node 1 Beats Logstash Application 1000000 log events round robin Node 2 output { elasticsearch { hosts => [“node1”,”node2”] } } !52 @dadoonet sli.do/elastic
Load Balancer Elasticsearch
Node 1 Beats Logstash LB Application Node 2 1000000 log events !53 @dadoonet sli.do/elastic
Coordinating-only Node Elasticsearch
Node 1 Beats Logstash Node 3 co-node Application Node 2 1000000 log events !54 @dadoonet sli.do/elastic
Test it ;) • 2 node cluster (m3.large) ‒ 2 vCPU, 7.5GB Memory, 1x32GB SSD • 1 index server (m3.large) ‒ logstash (round robin configured) ‒ hosts => [“10.12.145.189”, “10.121.140.167”] ‒ kibana !55 #docs time(s) 1000 5000 10000 NO Round Robin 163.5 160.7 161.5 Round Robin 161.3 158.2 159.4 @dadoonet sli.do/elastic
Optimizing Disk IO
Durability time buffer index a doc buffer index a doc buffer index a doc buffer !57 @dadoonet lucene flush segment
refresh_interval • Dynamic per-index setting PUT logstash-2017.05.16/_settings { “refresh_interval”: “60s” } • Increase to get better write throughput to an index • New documents will take more time to be available for Search. !58 #docs time(s) 1000 5000 10000 1s refresh 60s refresh 161.3 156.7 158.2 152.1 159.4 152.6 @dadoonet
Durability time buffer index a doc doc op trans_log buffer lucene flush segment trans_log buffer elasticsearch flush lucene commit trans_log !59 @dadoonet segment segment
Translog fsync every 5s (1.7) buffer Primary index a doc doc op trans_log buffer Replica index a doc doc op trans_log redundancy doesn’t help if all nodes lose power !60 @dadoonet
Translog fsync on every request • For low volume indexing, fsync matters less • For high volume indexing, we can amortize the costs and fsync on every bulk • Concurrent requests can share an fsync bulk 1 single fsync bulk 2 !61 @dadoonet sli.do/elastic
Async Transaction Log • index.translog.durability ‒ request (default) ‒ async • index.translog.sync_interval (only if async is set) • Dynamic per-index settings • Be careful, you are relaxing the safety guarantees !62 #docs time(s) 1000 5000 10000 Request fsync 161.3 158.2 159.4 5s sync 152.4 149.1 150.3 @dadoonet
Final Remarks
App Search Site Search Enterprise Search Logging APM Business Analytics Security Analytics Metrics Elastic Stack Future Solutions Kibana Visualize & Manage Elasticsearch Store, Search, & Analyze Beats SaaS Elastic Cloud Logstash Ingest Self Managed Standalone Elastic Cloud Enterprise Deployment
Final Remarks • Primaries ‒ More data -> More shards ‒ Do not overshard! • Replicas ‒ high availability (1 replica is the default) ‒ read throughput (More reads -> More replicas) Big Data U s e r s !65 @dadoonet … … … … … … sli.do/elastic
Final Remarks • Bulk and Test • Distribute the Load • Refresh Interval • Async Trans Log (careful) !66 #docs per bulk 1000 5000 10000 Default 163.5 160.7 161.5 RR+60s+Async5s 152.4 149.1 150.3 @dadoonet
Les vendredi noirs ? Même pas peur ! David Pilato Developer | Evangelist, @dadoonet
Surveiller une application complexe n’est pas une tâche aisée, mais avec les bons outils, ce n’est pas si sorcier. Néanmoins, des périodes fortes telles que les opérations de type “Black Friday” (Vendredi noir) ou période de Noël peuvent pousser votre application aux limites de ce qu’elle peut supporter, ou pire, la faire crasher. Parce que le système est fortement sollicité, il génère encore davantage de logs qui peuvent également mettre à mal votre système de supervision.
Dans cette session, j’aborderai les bonnes pratiques d’utilisation de la suite Elastic pour centraliser et monitorer vos logs. Je partagerai également avec vous quelques trucs et astuces pour vous aider à passer sans souci vos Vendredis noirs !
Resources
The following resources were mentioned during the presentation or are useful additional information.
-
Quantitative Cluster Sizing
How many shards should I have? How many nodes should I have? What about replicas? Do these questions sound familiar? The answer is often ‘it depends’. This talk will outline the factors that affect sizing and walk you through a quantitative approach to estimating the configuration and size of your cluster.
-
Rollover API
The rollover index API rolls an alias over to a new index when the existing index is considered to be too large or too old.
-
Shrink Index API
The shrink index API allows you to shrink an existing index into a new index with fewer primary shards.
-
Split Index API
The split index API allows you to split an existing index into a new index, where each original primary shard is split into two or more primary shards in the new index.
-
Talk video from a past session
Talk give at DevopsDDay Marseille, November 2018.
-
Official photos from DevopsDays Geneva
Buzz and feedback
Here’s what was said about this presentation on social media.
