do MORE with stateLESS Elasticsearch
A presentation at Meetup ElasticFR #99 - BBQ & Serverless in in Paris, France by David Pilato
David Pilato Developer | Evangelist do MORE with stateLESS Elasticsearch Sponsors
github.com/elastic/elasticsearch ) ( ● Distributed, scalable, highly available, resilient search & analytics engine ● HTTP based JSON interface ● Based on Apache Lucene ● Not only grep or SQL’s LIKE = ‘%quick% ●Ranked results (BM25, recency, popularity), fuzzy matching ●Complex search expressions ●Spelling, Synonyms, Phrases, Stemming ● Timeseries, geospatial ● Vector search, G AI/ML, RAG search apps db-engines.com/en/ranking/search+engine
) ●Cluster ●Nodes ●Index ●Shards ●Segments ●Cluster state ●Data streams ●Index Lifecycle Management (ILM
Shared-nothing stateful architecture Node 1 Node 2 Node 7 Disk Disk Disk Cluster state Cluster state Cluster state logs.2025.9.5 shard 1, primary logs.2025.9.5 shard 1, replica Cache logs.2025.9.5 shard 2, primary logs.2025.9.5 shard 2, replica logs.2025.9.4 shard 1, replica logs.2025.9.4 shard 1, primary CSP object storage (e.g., Amazon S3) Snapshots: logs.2025.8.31 … … … … logs.2025.8.30 Partially mounted: logs.2025.8.31 shard 1, primary logs.2025.8.30 shard 1, primary logs.2025.8.29 shard 1, primary logs.2025.8.29 Deployment options: Public Cloud Hybrid Hot (more performant, expensive) Warm Data tiers Cold Frozen (less performant, cheaper) On-Premises
Disadvantages of stateful ● ● ● ● ● User defines cluster (RAM, CPU, disk) CPU & RAM coupled with storage CPU shared by ingestion & search Needs to think of data tiers Adding/removing nodes moves data
Traditional On-Premise On-Prem) Infrastructure as a Service IaaS Platform as a Service PaaS Software as a Service SaaS Dining Table Dining Table Dining Table Dining Table Drinks Drinks Drinks Drinks Electricity Electricity Electricity Electricity Oven Oven Oven Oven Fire Fire Fire Fire Pizza Dough Pizza Dough Pizza Dough Pizza Dough Base Sauce Base Sauce Base Sauce Base Sauce Toppings Toppings Toppings Toppings Cheese Cheese Cheese Cheese Home Made Take & Bake Delivery Dined Out ) ) Self Managed ) ( ( ( ( Pizza as a Service Managed by Vendor
Introducing Elastic Cloud Serverless Elastic Cloud Elastic Self Managed Elastic Cloud on K8s Elastic Cloud Hosted Elastic Cloud Serverless Download and run it yourself We give the orchestration to you We orchestrate the infrastructure We manage everything Control Simplicity
Serverless Elasticsearch ● ● ● ● ● ● ● SaaS Exploits Cloud Service Provider Stateless: decouples compute from storage Same API as stateful Elasticsearch Pay-as-you-go model (retention, indexing, searching) No more cluster specifics, sizing, upgrading, HA 2 simplified data tiers: indexing, search
New stateless architecture Node 1 Node 2 Disk Disk Cluster state Cluster state Cache Cache Partially mounted: logs.2025.9.5 shard 1, index Partially mounted: logs.2025.9.5 shard 2, index logs.2025.9.4 shard 1, index logs.2025.9.2 shard 1, index logs.2025.9.3 shard 1, index logs.2025.9.1 shard 1, index Indexing tier CSP object storage (e.g., Amazon S3) Cluster state Index data: logs.2025.9.5 logs.2025.9.4 logs.2025.9.3 logs.2025.9.2 logs.2025.9.1 Write Read Node 3 Node 4 Disk Disk Cluster state Cluster state Cache Cache Partially mounted: logs.2025.9.5 shard 1, search Partially mounted: logs.2025.9.5 shard 2, search logs.2025.9.4 shard 1, search logs.2025.9.2 shard 1, search logs.2025.9.3 shard 1, search logs.2025.9.1 shard 1, search Search tier
● ● ● ● ● ● ● : ( Super) Thin shards Loaded with metadata until data is indexed/searched Indexing generates files, uploads them and deletes them when unused Searching loads from object store only the required data into the cache Indexing & searching virtually limitless data Not dependent on disk, not restricted by disk capacity Shard relocation (and recovery) is very fast WIP Super Thin Indexing Shards
Stateful example data inside a shard
Data inside a shard (stateful) Node 1 New docs [1, 5] Index “logs.2025.9.5”, shard 1, primary Lucene in-memory indexing buffers Translog generation: Docs: 1 [1, 5]
Data inside a shard (stateful) Node 1 New docs [6, 12] Index “logs.2025.9.5”, shard 1, primary Lucene in-memory indexing buffers Translog generation: Docs: 1 [1, 5] 2 [6, 12]
Data inside a shard (stateful) Node 1 Index “logs.2025.9.5”, shard 1, primary refresh Lucene segments generation: Docs: Translog generation: Docs: 1 [1, 12] 1 [1, 5] 2 [6, 12] Lucene in-memory indexing buffers
Data inside a shard (stateful) Node 1 Search documents Index “logs.2025.9.5”, shard 1, primary Lucene segments generation: Docs: Translog generation: Docs: Lucene in-memory indexing buffers 1 [1, 12] 1 [1, 5] 2 [6, 12]
Data inside a shard (stateful) Node 1 New docs [13, 15] Index “logs.2025.9.5”, shard 1, primary Lucene segments generation: Docs: Translog generation: Docs: Lucene in-memory indexing buffers 1 [1, 12] 1 [1, 5] 2 [6, 12] 3 [13, 15]
Data inside a shard (stateful) Node 1 Index “logs.2025.9.5”, shard 1, primary refresh Lucene segments generation: Docs: Translog generation: Docs: 1 [1, 12] 2 [13, 15] 1 [1, 5] 2 [6, 12] 3 [13, 15] Lucene in-memory indexing buffers
Data inside a shard (stateful) Node 1 Index “logs.2025.9.5”, shard 1, primary Lucene segments generation: Docs: Flushed commits: Translog generation: Docs: Lucene in-memory indexing buffers 1 [1, 12] 2 [13, 15] Index flush Commit 2 1 [1, 5] 2 [6, 12] 3 [13, 15]
Data inside a shard (stateful) Node 1 Index “logs.2025.9.5”, shard 1, primary Lucene segments generation: Docs: Flushed commits: Translog generation: Docs: Lucene in-memory indexing buffers 1 [1, 12] Commit 2 2 [13, 15]
Data inside a shard (stateful) Node 1 New docs [16, 20] Index “logs.2025.9.5”, shard 1, primary Lucene segments generation: Docs: Flushed commits: Translog generation: Docs: Lucene in-memory indexing buffers 1 [1, 12] 2 [13, 15] Commit 2 4 [16, 20]
Data inside a shard (stateful) Node 1 Index “logs.2025.9.5”, shard 1, primary refresh Lucene segments generation: Docs: Flushed commits: Translog generation: Docs: 1 [1, 12] 2 [13, 15] 3 [16, 20] Commit 2 4 [16, 20] Lucene in-memory indexing buffers
Data inside a shard (stateful) Node 1 Index “logs.2025.9.5”, shard 1, primary Lucene segments generation: Docs: Flushed commits: Translog generation: Docs: Lucene in-memory indexing buffers 1 [1, 12] 2 [13, 15] Commits 2, 3 3 [16, 20] Index flush
Data inside a shard (stateful) Node 1 Index “logs.2025.9.5”, shard 1, primary Lucene segments generation: Docs: Flushed commits: Translog generation: Docs: 1 [1, 12] 2 [13, 15] Commits 2, 3 3 [16, 20] Searchable refreshed segments Flushed commits (on disk) Non-flushed operations (on disk) for recovery
Stateless example offloading shard data to an object store
Offloading index data to the object store (stateless) Node 1 (indexing) New docs [1, 5] Index “logs.2025.9.5”, shard 1 Lucene in-memory indexing buffers Lucene segments generation: Docs: Flushed commits: Translog generation: Docs: CSP object storage 1 [1, 5] (e.g., Amazon S3)
Offloading index data to the object store (stateless) Node 1 (indexing) New docs [6, 12] Index “logs.2025.9.5”, shard 1 CSP object storage Lucene in-memory indexing buffers Lucene segments generation: Docs: Flushed commits: Translog generation: Docs: 1 [1, 5] 2 [6, 12] (e.g., Amazon S3)
Offloading index data to the object store (stateless) Node 1 (indexing) New docs [6, 12] Index “logs.2025.9.5”, shard 1 CSP object storage Lucene in-memory indexing buffers Lucene segments generation: Translog data: Docs: Upload (< 200ms) Flushed commits: Translog generation: Docs: (e.g., Amazon S3) 1 [1, 5] 2 [6, 12] Node 1, file 1
Offloading index data to the object store (stateless) Node 1 (indexing) Acknowledged writes Index “logs.2025.9.5”, shard 1 CSP object storage Lucene in-memory indexing buffers Lucene segments generation: Translog data: Docs: Node 1, file 1 Flushed commits: Translog generation: Docs: (e.g., Amazon S3) 1 [1, 5] 2 [6, 12]
Offloading index data to the object store (stateless) Node 1 (indexing) Index “logs.2025.9.5”, shard 1 CSP object storage refresh Lucene segments generation: Docs: Flushed commits: Translog generation: Docs: 1 [1, 12] Lucene in-memory indexing buffers (e.g., Amazon S3) Translog data: Node 1, file 1 Commit 1
Offloading index data to the object store (stateless) Node 1 (indexing) New docs [13, 15] Index “logs.2025.9.5”, shard 1 Lucene segments generation: Docs: Flushed commits: Translog generation: Docs: CSP object storage Lucene in-memory indexing buffers 1 [1, 12] (e.g., Amazon S3) Translog data: Node 1, file 1 Commit 1 3 [13, 15]
Offloading index data to the object store (stateless) Node 1 (indexing) New docs [13, 15] Index “logs.2025.9.5”, shard 1 Lucene segments generation: Docs: Flushed commits: Translog generation: Docs: CSP object storage Lucene in-memory indexing buffers 1 [1, 12] (e.g., Amazon S3) Translog data: Node 1, file 1 Commit 1 Upload (< 200ms) 3 [13, 15] Node 1, file 2
Offloading index data to the object store (stateless) Node 1 (indexing) Acknowledged writes Index “logs.2025.9.5”, shard 1 Lucene segments generation: Docs: Flushed commits: Translog generation: Docs: CSP object storage Lucene in-memory indexing buffers 1 [1, 12] (e.g., Amazon S3) Translog data: Node 1, file 1 Commit 1 Node 1, file 2 3 [13, 15]
Offloading index data to the object store (stateless) Node 1 (indexing) Index “logs.2025.9.5”, shard 1 Lucene segments generation: Docs: Flushed commits: Translog generation: Docs: CSP object storage refresh Lucene in-memory indexing buffers 1 [1, 12] 2 [13, 15] (e.g., Amazon S3) Translog data: Node 1, file 1 Commits 1, 2 Node 1, file 2
Offloading index data to the object store (stateless) Node 1 (indexing) Index “logs.2025.9.5”, shard 1 Lucene segments generation: Docs: Flushed commits: Translog generation: Docs: CSP object storage Lucene in-memory indexing buffers 1 [1, 12] 2 [13, 15] Commits 1, 2 (e.g., Amazon S3) Translog data: Index flush Upload (deleted) Index data: logs.2025.9.5 shard 1, BCC1 (Batched compound commit)
Offloading index data to the object store (stateless) Node 2 (search) Index “logs.2025.9.5”, shard 1 Lucene segments generation: Docs: Flushed commits: Translog generation: Docs: CSP object storage Lucene in-memory indexing buffers 1 [1, 12] 2 [13, 15] Commits 1, 2 Node Cache (e.g., Amazon S3) Translog data: Partially mounted: Index data: logs.2025.9.5 shard 1, BCC1 (Batched compound commit)
Offloading index data to the object store (stateless) Node 2 (search) Search documents 2 & 14 Index “logs.2025.9.5”, shard 1 Lucene segments generation: Docs: Flushed commits: CSP object storage Lucene in-memory indexing buffers 1 [1, 12] 2 [13, 15] Commits 1, 2 Node Cache (e.g., Amazon S3) Translog data: Partially mounted: Files of CC1 Translog generation: Docs: Index data: Files of CC2 logs.2025.9.5 shard 1, BCC1 (Batched compound commit)
Batched Compound Commit Commit 1 Commit 2 File File File … BCC gen=3 blob file CC3 Header File File File File File … CC2 Header File File File File … CC1 Header BCC gen=1 blob file Commit 3
Autoscaling Autoscaler Autoscaling metrics API, Search power Master node Ingestion load, min heap (e.g., mappings) Indexing node 1 Search load Search node 1
Limitations ● ● ● ● ● Links to external resources in Kibana is not supported Refresh rate non adjustable to less than x per minute No dashboard export as PNG/PDF (no headless browser) Subset of APIs only Can not define the number of shards
Do MORE with StateLESS Elasticsearch David Pilato
Comment feriez-vous pour créer Elasticsearch si vous commenciez ce projet en 2025 ?
- Découpler le calcul (compute) du stockage (storage)
- Externaliser la gestion de la persistence et la réplication à un blob store comme S3, Google Cloud Storage ou encore Azure Blob Storage
- Dynamiquement ajouter ou supprimer des instances
- Avoir les bonnes valeurs par défaut
- Et un chemin hyper clair et fluide pour les développeurs
C’est exactement ce que nous avons fait avec Elastic Serverless.
Lors de cette session, vous allez découvrir comment nous avons re-conçu Elasticsearch pour lui permettre d’en faire davantage avec une architecture Stateless qui peut exécuter des requêtes sur un espace de stockage froid (cold storage).
Video
Resources
The following resources were mentioned during the presentation or are useful additional information.
Buzz and feedback
Here’s what was said about this presentation on social media.
