do MORE with stateLESS Elasticsearch

David Pilato Developer | Evangelist do MORE with stateLESS Elasticsearch Sponsors

github.com/elastic/elasticsearch ) ( ● Distributed, scalable, highly available, resilient search & analytics engine ● HTTP based JSON interface ● Based on Apache Lucene ● Not only grep or SQL’s LIKE = ‘%quick% ●Ranked results (BM25, recency, popularity), fuzzy matching ●Complex search expressions ●Spelling, Synonyms, Phrases, Stemming ● Timeseries, geospatial ● Vector search, G AI/ML, RAG search apps db-engines.com/en/ranking/search+engine

) ●Cluster ●Nodes ●Index ●Shards ●Segments ●Cluster state ●Data streams ●Index Lifecycle Management (ILM

Shared-nothing stateful architecture Node 1 Node 2 Node 7 Disk Disk Disk Cluster state Cluster state Cluster state logs.2025.9.5 shard 1, primary logs.2025.9.5 shard 1, replica Cache logs.2025.9.5 shard 2, primary logs.2025.9.5 shard 2, replica logs.2025.9.4 shard 1, replica logs.2025.9.4 shard 1, primary CSP object storage (e.g., Amazon S3) Snapshots: logs.2025.8.31 … … … … logs.2025.8.30 Partially mounted: logs.2025.8.31 shard 1, primary logs.2025.8.30 shard 1, primary logs.2025.8.29 shard 1, primary logs.2025.8.29 Deployment options: Public Cloud Hybrid Hot (more performant, expensive) Warm Data tiers Cold Frozen (less performant, cheaper) On-Premises

Disadvantages of stateful ● ● ● ● ● User defines cluster (RAM, CPU, disk) CPU & RAM coupled with storage CPU shared by ingestion & search Needs to think of data tiers Adding/removing nodes moves data

Traditional On-Premise On-Prem) Infrastructure as a Service IaaS Platform as a Service PaaS Software as a Service SaaS Dining Table Dining Table Dining Table Dining Table Drinks Drinks Drinks Drinks Electricity Electricity Electricity Electricity Oven Oven Oven Oven Fire Fire Fire Fire Pizza Dough Pizza Dough Pizza Dough Pizza Dough Base Sauce Base Sauce Base Sauce Base Sauce Toppings Toppings Toppings Toppings Cheese Cheese Cheese Cheese Home Made Take & Bake Delivery Dined Out ) ) Self Managed ) ( ( ( ( Pizza as a Service Managed by Vendor

Introducing Elastic Cloud Serverless Elastic Cloud Elastic Self Managed Elastic Cloud on K8s Elastic Cloud Hosted Elastic Cloud Serverless Download and run it yourself We give the orchestration to you We orchestrate the infrastructure We manage everything Control Simplicity

Serverless Elasticsearch ● ● ● ● ● ● ● SaaS Exploits Cloud Service Provider Stateless: decouples compute from storage Same API as stateful Elasticsearch Pay-as-you-go model (retention, indexing, searching) No more cluster specifics, sizing, upgrading, HA 2 simplified data tiers: indexing, search

New stateless architecture Node 1 Node 2 Disk Disk Cluster state Cluster state Cache Cache Partially mounted: logs.2025.9.5 shard 1, index Partially mounted: logs.2025.9.5 shard 2, index logs.2025.9.4 shard 1, index logs.2025.9.2 shard 1, index logs.2025.9.3 shard 1, index logs.2025.9.1 shard 1, index Indexing tier CSP object storage (e.g., Amazon S3) Cluster state Index data: logs.2025.9.5 logs.2025.9.4 logs.2025.9.3 logs.2025.9.2 logs.2025.9.1 Write Read Node 3 Node 4 Disk Disk Cluster state Cluster state Cache Cache Partially mounted: logs.2025.9.5 shard 1, search Partially mounted: logs.2025.9.5 shard 2, search logs.2025.9.4 shard 1, search logs.2025.9.2 shard 1, search logs.2025.9.3 shard 1, search logs.2025.9.1 shard 1, search Search tier

● ● ● ● ● ● ● : ( Super) Thin shards Loaded with metadata until data is indexed/searched Indexing generates files, uploads them and deletes them when unused Searching loads from object store only the required data into the cache Indexing & searching virtually limitless data Not dependent on disk, not restricted by disk capacity Shard relocation (and recovery) is very fast WIP Super Thin Indexing Shards

Stateful example data inside a shard

Data inside a shard (stateful) Node 1 New docs [1, 5] Index “logs.2025.9.5”, shard 1, primary Lucene in-memory indexing buffers Translog generation: Docs: 1 [1, 5]

Data inside a shard (stateful) Node 1 New docs [6, 12] Index “logs.2025.9.5”, shard 1, primary Lucene in-memory indexing buffers Translog generation: Docs: 1 [1, 5] 2 [6, 12]

Data inside a shard (stateful) Node 1 Index “logs.2025.9.5”, shard 1, primary refresh Lucene segments generation: Docs: Translog generation: Docs: 1 [1, 12] 1 [1, 5] 2 [6, 12] Lucene in-memory indexing buffers

Data inside a shard (stateful) Node 1 Search documents Index “logs.2025.9.5”, shard 1, primary Lucene segments generation: Docs: Translog generation: Docs: Lucene in-memory indexing buffers 1 [1, 12] 1 [1, 5] 2 [6, 12]

Data inside a shard (stateful) Node 1 New docs [13, 15] Index “logs.2025.9.5”, shard 1, primary Lucene segments generation: Docs: Translog generation: Docs: Lucene in-memory indexing buffers 1 [1, 12] 1 [1, 5] 2 [6, 12] 3 [13, 15]

Data inside a shard (stateful) Node 1 Index “logs.2025.9.5”, shard 1, primary refresh Lucene segments generation: Docs: Translog generation: Docs: 1 [1, 12] 2 [13, 15] 1 [1, 5] 2 [6, 12] 3 [13, 15] Lucene in-memory indexing buffers

Data inside a shard (stateful) Node 1 Index “logs.2025.9.5”, shard 1, primary Lucene segments generation: Docs: Flushed commits: Translog generation: Docs: Lucene in-memory indexing buffers 1 [1, 12] 2 [13, 15] Index flush Commit 2 1 [1, 5] 2 [6, 12] 3 [13, 15]

Data inside a shard (stateful) Node 1 Index “logs.2025.9.5”, shard 1, primary Lucene segments generation: Docs: Flushed commits: Translog generation: Docs: Lucene in-memory indexing buffers 1 [1, 12] Commit 2 2 [13, 15]

Data inside a shard (stateful) Node 1 New docs [16, 20] Index “logs.2025.9.5”, shard 1, primary Lucene segments generation: Docs: Flushed commits: Translog generation: Docs: Lucene in-memory indexing buffers 1 [1, 12] 2 [13, 15] Commit 2 4 [16, 20]

Data inside a shard (stateful) Node 1 Index “logs.2025.9.5”, shard 1, primary refresh Lucene segments generation: Docs: Flushed commits: Translog generation: Docs: 1 [1, 12] 2 [13, 15] 3 [16, 20] Commit 2 4 [16, 20] Lucene in-memory indexing buffers

Data inside a shard (stateful) Node 1 Index “logs.2025.9.5”, shard 1, primary Lucene segments generation: Docs: Flushed commits: Translog generation: Docs: Lucene in-memory indexing buffers 1 [1, 12] 2 [13, 15] Commits 2, 3 3 [16, 20] Index flush

Data inside a shard (stateful) Node 1 Index “logs.2025.9.5”, shard 1, primary Lucene segments generation: Docs: Flushed commits: Translog generation: Docs: 1 [1, 12] 2 [13, 15] Commits 2, 3 3 [16, 20] Searchable refreshed segments Flushed commits (on disk) Non-flushed operations (on disk) for recovery

Stateless example offloading shard data to an object store

Offloading index data to the object store (stateless) Node 1 (indexing) New docs [1, 5] Index “logs.2025.9.5”, shard 1 Lucene in-memory indexing buffers Lucene segments generation: Docs: Flushed commits: Translog generation: Docs: CSP object storage 1 [1, 5] (e.g., Amazon S3)

Offloading index data to the object store (stateless) Node 1 (indexing) New docs [6, 12] Index “logs.2025.9.5”, shard 1 CSP object storage Lucene in-memory indexing buffers Lucene segments generation: Docs: Flushed commits: Translog generation: Docs: 1 [1, 5] 2 [6, 12] (e.g., Amazon S3)

Offloading index data to the object store (stateless) Node 1 (indexing) New docs [6, 12] Index “logs.2025.9.5”, shard 1 CSP object storage Lucene in-memory indexing buffers Lucene segments generation: Translog data: Docs: Upload (< 200ms) Flushed commits: Translog generation: Docs: (e.g., Amazon S3) 1 [1, 5] 2 [6, 12] Node 1, file 1

Offloading index data to the object store (stateless) Node 1 (indexing) Acknowledged writes Index “logs.2025.9.5”, shard 1 CSP object storage Lucene in-memory indexing buffers Lucene segments generation: Translog data: Docs: Node 1, file 1 Flushed commits: Translog generation: Docs: (e.g., Amazon S3) 1 [1, 5] 2 [6, 12]

Offloading index data to the object store (stateless) Node 1 (indexing) Index “logs.2025.9.5”, shard 1 CSP object storage refresh Lucene segments generation: Docs: Flushed commits: Translog generation: Docs: 1 [1, 12] Lucene in-memory indexing buffers (e.g., Amazon S3) Translog data: Node 1, file 1 Commit 1

Offloading index data to the object store (stateless) Node 1 (indexing) New docs [13, 15] Index “logs.2025.9.5”, shard 1 Lucene segments generation: Docs: Flushed commits: Translog generation: Docs: CSP object storage Lucene in-memory indexing buffers 1 [1, 12] (e.g., Amazon S3) Translog data: Node 1, file 1 Commit 1 3 [13, 15]

Offloading index data to the object store (stateless) Node 1 (indexing) New docs [13, 15] Index “logs.2025.9.5”, shard 1 Lucene segments generation: Docs: Flushed commits: Translog generation: Docs: CSP object storage Lucene in-memory indexing buffers 1 [1, 12] (e.g., Amazon S3) Translog data: Node 1, file 1 Commit 1 Upload (< 200ms) 3 [13, 15] Node 1, file 2

Offloading index data to the object store (stateless) Node 1 (indexing) Acknowledged writes Index “logs.2025.9.5”, shard 1 Lucene segments generation: Docs: Flushed commits: Translog generation: Docs: CSP object storage Lucene in-memory indexing buffers 1 [1, 12] (e.g., Amazon S3) Translog data: Node 1, file 1 Commit 1 Node 1, file 2 3 [13, 15]

Offloading index data to the object store (stateless) Node 1 (indexing) Index “logs.2025.9.5”, shard 1 Lucene segments generation: Docs: Flushed commits: Translog generation: Docs: CSP object storage refresh Lucene in-memory indexing buffers 1 [1, 12] 2 [13, 15] (e.g., Amazon S3) Translog data: Node 1, file 1 Commits 1, 2 Node 1, file 2

Offloading index data to the object store (stateless) Node 1 (indexing) Index “logs.2025.9.5”, shard 1 Lucene segments generation: Docs: Flushed commits: Translog generation: Docs: CSP object storage Lucene in-memory indexing buffers 1 [1, 12] 2 [13, 15] Commits 1, 2 (e.g., Amazon S3) Translog data: Index flush Upload (deleted) Index data: logs.2025.9.5 shard 1, BCC1 (Batched compound commit)

Offloading index data to the object store (stateless) Node 2 (search) Index “logs.2025.9.5”, shard 1 Lucene segments generation: Docs: Flushed commits: Translog generation: Docs: CSP object storage Lucene in-memory indexing buffers 1 [1, 12] 2 [13, 15] Commits 1, 2 Node Cache (e.g., Amazon S3) Translog data: Partially mounted: Index data: logs.2025.9.5 shard 1, BCC1 (Batched compound commit)

Offloading index data to the object store (stateless) Node 2 (search) Search documents 2 & 14 Index “logs.2025.9.5”, shard 1 Lucene segments generation: Docs: Flushed commits: CSP object storage Lucene in-memory indexing buffers 1 [1, 12] 2 [13, 15] Commits 1, 2 Node Cache (e.g., Amazon S3) Translog data: Partially mounted: Files of CC1 Translog generation: Docs: Index data: Files of CC2 logs.2025.9.5 shard 1, BCC1 (Batched compound commit)

Batched Compound Commit Commit 1 Commit 2 File File File … BCC gen=3 blob file CC3 Header File File File File File … CC2 Header File File File File … CC1 Header BCC gen=1 blob file Commit 3

Autoscaling Autoscaler Autoscaling metrics API, Search power Master node Ingestion load, min heap (e.g., mappings) Indexing node 1 Search load Search node 1

Limitations ● ● ● ● ● Links to external resources in Kibana is not supported Refresh rate non adjustable to less than x per minute No dashboard export as PNG/PDF (no headless browser) Subset of APIs only Can not define the number of shards

Do MORE with StateLESS Elasticsearch David Pilato