Elasticsearch Query Language: ES|QL

A presentation at FinistDevs in December 2024 in Brest, France by David Pilato

Slide 1

Slide 1

Elasticsearch Query Language ES|QL o s e d li David Pilato - @dadoonet Developer | Evangelist & s m e d

Slide 2

Slide 2

A brief history of Elasticsearch’s analytical capabilities 2010 2013 2014 2015 2023 Elasticsearch 0.9 Elasticsearch < 0.90 Elasticsearch 1.0 Elasticsearch 2.0 Elasticsearch 8.11 Facets Facet terms-stats Aggregations Pipeline aggregations ES|QL

Slide 3

Slide 3

ES|QL • Language • Engine • Visualization

Slide 4

Slide 4

o e d li ES|QL in action https://github.com/dadoonet/esql-demo s & s m e d

Slide 5

Slide 5

PROJECTIONS Each language client will offer a selection of projections relevant to that language ecosystem. RESULT DATA Ways to consume ES|QL results Users can consume raw data directly from the server output in one of several formats. DataFrame Object / Dict Cursor For mapping domain objects within a client application For incremental consumption of results, with implicit pagination For data science and analytics; integration with frameworks like Pandas Text CSV JSON Human-readable format ideal for interactive work, CLIs, etc Raw CSV data to load directly into spreadsheets and ETL processes Structured response containing metadata and data in a 2D value array Bring your own Custom projections built atop raw server output Apache Arrow Dataframe IPC format

Slide 6

Slide 6

Object API https://github.com/dadoonet/elasticsearch-java-client-demo String query = “”” FROM persons | WHERE name == “David” | KEEP name | LIMIT 1 “”“; Iterable<Person> persons = client.esql() .query(ObjectsEsqlAdapter.of(Person.class), query); for (Person person : persons) { assertNull(person.getId()); assertNotNull(person.getName()); }

Slide 7

Slide 7

ResultSet JDBC API https://github.com/dadoonet/elasticsearch-java-client-demo String query = “”” FROM persons | WHERE name == “David” | KEEP name | LIMIT 1 “”“; try (ResultSet resultSet = client.esql() .query(ResultSetEsqlAdapter.INSTANCE, query)) { assertTrue(resultSet.next()); assertEquals(“David”, resultSet.getString(1)); }

Slide 8

Slide 8

POST /_query 8. 16 { “query”: “”” from logs-* | stats x = ?function(?field) by ?breakdownField A better dashboard experience with named parameters | where x >= ?value “”“, “params”: [ {“function” : {“identifier” : “avg”}}, {“field” : {“identifier” : “network.bytes”}}, {“breakdownField” : {“identifier” : “agent.name”}}, {“value”: 1000} ] }

Slide 9

Slide 9

TD B

Slide 10

Slide 10

TD B

Slide 11

Slide 11

TD B

Slide 12

Slide 12

  1. 17 Coming next WHERE MATCH(actors, “Marlon*”) WHERE QSTR(“bytes:[1024 TO 2048]”)

Slide 13

Slide 13

  1. 18 Coming next WHERE KQL(“bytes>=1024”)

Slide 14

Slide 14

TB joinType JOIN indexName (AS qualifier)? condition? joinType: LOOKUP | LEFT | RIGHT | INNER condition: ON identifier == identifier | USING identifier JOINS! INLINESTATS total_visits = COUNT() FROM employees | SORT emp_no | LOOKUP JOIN languages_lookup ON language_code | KEEP emp_no, language_name ● No need to create an enrich policy ● A drag and drop experience in the UI D

Slide 15

Slide 15

Elasticsearch Query Language ES|QL o s e d li David Pilato - @dadoonet Developer | Evangelist & s m e d