Elasticsearch

BBL at

$ curl -XPOST https://localhost:9200/speaker/_doc -d ‘{ “name” : “David Pilato”, “jobs” : [ { “name” : “SRA Europe (SSII)”, “date” : “1995” }, { “name” : “SFR”, “date” : “1997” }, { “name” : “e-Brands / Vivendi”, “date”: “2000” }, { “name” : “DGDDI (douane)”, “date” : “2005” }, { “name” : “elastic”, “date” : “2013” } ], “motivations” : [ “family”, “job”, “deejay” ], “blog” : “https://david.pilato.fr/”, “twitter” : [ “@dadoonet”, “@elasticfr” ], “bluesky” : [ “@pilato.fr” ], “email” : “david@elastic.co” }’ -H ‘Content-Type: application/json’

One Search AI Platform Two Out-of-the-Box Solutions The Freedom to Build Anything Out-of-the-Box Solutions Elastic Observability Build Your Own Elastic Security Elastic Search The Elastic Search AI Platform Ingest Secure & Scalable Storage AI / ML Search Visualization Workflow Automation

st a rt -lo ca l

el a st ic cl ou d

se rv er le ss

A typical search implementation… CREATE TABLE user ( name VARCHAR(100), comments VARCHAR(1000) ); INSERT INTO user VALUES (‘David Pilato’, ‘Developer at elastic’); INSERT INTO user VALUES (‘Malloum Laya’, ‘Worked with David at french customs service’); INSERT INTO user VALUES (‘David Gageot’, ‘Engineer at Doctolib’); INSERT INTO user VALUES (‘David David’, ‘Who is that guy?’); David

Search on term INSERT INTO user VALUES INSERT INTO user VALUES customs service’); INSERT INTO user VALUES INSERT INTO user VALUES (‘David Pilato’, ‘Developer at elastic’); (‘Malloum Laya’, ‘Worked with David at french (‘David Gageot’, ‘Engineer at Doctolib’); (‘David David’, ‘Who is that guy?’); SELECT * FROM user WHERE name=”David”; Empty set (0,00 sec) David

Search like INSERT INTO user VALUES INSERT INTO user VALUES customs service’); INSERT INTO user VALUES INSERT INTO user VALUES (‘David Pilato’, ‘Developer at elastic’); (‘Malloum Laya’, ‘Worked with David at french (‘David Gageot’, ‘Engineer at Doctolib’); (‘David David’, ‘Who is that guy?’); SELECT * FROM user WHERE name LIKE “%David%”; +———————+———————————+ | name | comments | +———————+———————————+ | David Pilato | Developer at elastic | | David Gageot | Engineer at Doctolib | | David David | Who is that guy? | +———————+———————————+ David

Search for terms INSERT INTO user VALUES INSERT INTO user VALUES customs service’); INSERT INTO user VALUES INSERT INTO user VALUES (‘David Pilato’, ‘Developer at elastic’); (‘Malloum Laya’, ‘Worked with David at french (‘David Gageot’, ‘Engineer at Doctolib’); (‘David David’, ‘Who is that guy?’); SELECT * FROM user WHERE name LIKE “%David Pilato%”; +———————+———————————+ | name | comments | +———————+———————————+ | David Pilato | Developer at elastic | +———————+———————————+ David Pilato

Search with inverted terms INSERT INTO user VALUES INSERT INTO user VALUES customs service’); INSERT INTO user VALUES INSERT INTO user VALUES (‘David Pilato’, ‘Developer at elastic’); (‘Malloum Laya’, ‘Worked with David at french (‘David Gageot’, ‘Engineer at Doctolib’); (‘David David’, ‘Who is that guy?’); SELECT * FROM user WHERE name LIKE “%Pilato David%”; Empty set (0,00 sec) SELECT * FROM user WHERE name LIKE “%Pilato%David%”; Empty set (0,00 sec) Pilato David

Search for terms INSERT INTO user VALUES INSERT INTO user VALUES customs service’); INSERT INTO user VALUES INSERT INTO user VALUES (‘David Pilato’, ‘Developer at elastic’); (‘Malloum Laya’, ‘Worked with David at french (‘David Gageot’, ‘Engineer at Doctolib’); (‘David David’, ‘Who is that guy?’); SELECT * FROM user WHERE name LIKE “%David%” AND name LIKE “%Pilato%”; +———————+———————————+ | name | comments | +———————+———————————+ | David Pilato | Developer at elastic | +———————+———————————+ Pilato David

Search in two fields INSERT INTO user VALUES INSERT INTO user VALUES customs service’); INSERT INTO user VALUES INSERT INTO user VALUES (‘David Pilato’, ‘Developer at elastic’); (‘Malloum Laya’, ‘Worked with David at french (‘David Gageot’, ‘Engineer at Doctolib’); (‘David David’, ‘Who is that guy?’); SELECT * FROM user WHERE name LIKE “%David%” OR comments LIKE “%David%”; +———————+——————————————————————-+ | name | comments | +———————+——————————————————————-+ | David Pilato | Developer at elastic | | Malloum Laya | Worked with David at french customs service | | David Gageot | Engineer at Doctolib | | David David | Who is that guy? | +———————+——————————————————————-+ David

Search with typos INSERT INTO user VALUES INSERT INTO user VALUES customs service’); INSERT INTO user VALUES INSERT INTO user VALUES (‘David Pilato’, ‘Developer at elastic’); (‘Malloum Laya’, ‘Worked with David at french (‘David Gageot’, ‘Engineer at Doctolib’); (‘David David’, ‘Who is that guy?’); SELECT * FROM user WHERE name LIKE “%Dadid%”; Empty set (0,00 sec) Dadid

Search with typos INSERT INTO user VALUES INSERT INTO user VALUES customs service’); INSERT INTO user VALUES INSERT INTO user VALUES (‘David Pilato’, ‘Developer at elastic’); (‘Malloum Laya’, ‘Worked with David at french (‘David Gageot’, ‘Engineer at Doctolib’); (‘David David’, ‘Who is that guy?’); SELECT * FROM user WHERE name LIKE “%adid%” OR name LIKE “%D_did%” OR name LIKE “%Da_id%” OR name LIKE “%Dad_d%” OR name LIKE “%Dadi%”; +———————+———————————+ | name | comments | +———————+———————————+ | David Pilato | Developer at elastic | | David Gageot | Engineer at Doctolib | | David David | Who is that guy? | +———————+———————————+ Dadid

User Interface

What is a search engine? ● Index engine (indexing documents) ● Search engine (within the created indices)

Demo time!

Elasticsearch You Know, for Search

GET /_analyze { “char_filter”: [ “html_strip” ], “tokenizer”: “standard”, “filter”: [ “lowercase”, “stop”, “snowball” ], “text”: “These are <em>not</em> the droids you are looking for.” }

“char_filter”: “html_strip” These are <em>not</em> the droids you are looking for. These are not the droids you are looking for.

“tokenizer”: “standard” These are not the droids you are looking for. These are not the droids you are looking for

“filter”: “lowercase” These are not the droids you are looking for these are not the droids you are looking for

“filter”: “stop” These are not the droids you are looking for these are not the droids you are looking for droids you looking

“filter”: “snowball” These are not the droids you are looking for these are not the droids you are looking for droids you droid you looking look

These are <em>not</em> the droids you are looking for. { “tokens”: [{ “token”: “droid”, “start_offset”: 27, “end_offset”: 33, “type”: “<ALPHANUM>”, “position”: 4 },{ “token”: “you”, “start_offset”: 34, “end_offset”: 37, “type”: “<ALPHANUM>”, “position”: 5 }, { “token”: “look”, “start_offset”: 42, “end_offset”: 49, “type”: “<ALPHANUM>”, “position”: 7 }]}

Elasticsearch You Know, for Vector Search

Example: 1-dimensional vector Character Vector [ 1 ] ] Realistic

[ Embeddings represent your data Cartoon 1

represent different data aspects Human Character Vector [ 1, 1 Realistic Cartoon ] ] Machine

[ Multiple dimensions 1, 0

is grouped together Human Character Vector [ 1.0, 1.0 Realistic Cartoon 1.0, 0.0 [ 1.0, 0.8 ] ] ]

Machine

[ Similar data

Vector search ranks objects by similarity (~relevance) to the query Human Rank Query 1 Realistic Cartoon 2 3 4 5 Machine Result

Data Ingestion and Embedding Generation POST /_doc { “_id”:”product-1234”, “product_name”:”Summer Dress”, “description”:”Our best-selling…”, “Price”: 118, “color”:”blue”, “fabric”:”cotton”, “fabric”:”cotton” } “desc_embedding”:[0.452,0.3242,…], “desc_embedding”:[0.452,0.3242,…] } “img_embedding”:[0.012,0.0,…] } Source data POST /_doc

Vector Query GET product-catalog/_search { “query” : { “bool”: { “must”: [{ “knn”: { “field”: “desc_embbeding”, “num_candidates”: 50, “query_vector”: [0.123, 0.244,…] } }], “filter”: { “term”: { “department”: “women” } } } } }, “size”: 10

Similarity Human q cos(θ) = d1 d2 Realistic θ q⃗ × d ⃗ | q⃗ | × | d |⃗ _score = 1 + cos(θ) 2

Similarity: cosine (cosine) θ Similar vectors θ close to 0 cos(θ) close to 1 1+1 _score = =1 2 θ Orthogonal vectors θ close to 90° cos(θ) close to 0 1+0 _score = = 0.5 2 θ Opposite vectors θ close to 180° cos(θ) close to -1 1−1 _score = =0 2

https://djdadoo.pilato.fr/

https://github.com/dadoonet/music-search/

One Search AI Platform Two Out-of-the-Box Solutions The Freedom to Build Anything Out-of-the-Box Solutions Elastic Observability Build Your Own Elastic Security Elastic Search The Elastic Search AI Platform Ingest Secure & Scalable Storage AI / ML Search Visualization Workflow Automation

www.meetup.com/ElasticFR @elasticfr discuss.elastic.co

Thank You