Elasticsearch
A presentation at BBL Ekino (private event) by David Pilato
BBL at
$ curl -XPOST https://localhost:9200/speaker/_doc -d ‘{ “name” : “David Pilato”, “jobs” : [ { “name” : “SRA Europe (SSII)”, “date” : “1995” }, { “name” : “SFR”, “date” : “1997” }, { “name” : “e-Brands / Vivendi”, “date”: “2000” }, { “name” : “DGDDI (douane)”, “date” : “2005” }, { “name” : “elastic”, “date” : “2013” } ], “motivations” : [ “family”, “job”, “deejay” ], “blog” : “https://david.pilato.fr/”, “twitter” : [ “@dadoonet”, “@elasticfr” ], “bluesky” : [ “@pilato.fr” ], “email” : “david@elastic.co” }’ -H ‘Content-Type: application/json’
One Search AI Platform Two Out-of-the-Box Solutions The Freedom to Build Anything Out-of-the-Box Solutions Elastic Observability Build Your Own Elastic Security Elastic Search The Elastic Search AI Platform Ingest Secure & Scalable Storage AI / ML Search Visualization Workflow Automation
st a rt -lo ca l
el a st ic cl ou d
se rv er le ss
A typical search implementation… CREATE TABLE user ( name VARCHAR(100), comments VARCHAR(1000) ); INSERT INTO user VALUES (‘David Pilato’, ‘Developer at elastic’); INSERT INTO user VALUES (‘Malloum Laya’, ‘Worked with David at french customs service’); INSERT INTO user VALUES (‘David Gageot’, ‘Engineer at Doctolib’); INSERT INTO user VALUES (‘David David’, ‘Who is that guy?’); David
Search on term INSERT INTO user VALUES INSERT INTO user VALUES customs service’); INSERT INTO user VALUES INSERT INTO user VALUES (‘David Pilato’, ‘Developer at elastic’); (‘Malloum Laya’, ‘Worked with David at french (‘David Gageot’, ‘Engineer at Doctolib’); (‘David David’, ‘Who is that guy?’); SELECT * FROM user WHERE name=”David”; Empty set (0,00 sec) David
Search like INSERT INTO user VALUES INSERT INTO user VALUES customs service’); INSERT INTO user VALUES INSERT INTO user VALUES (‘David Pilato’, ‘Developer at elastic’); (‘Malloum Laya’, ‘Worked with David at french (‘David Gageot’, ‘Engineer at Doctolib’); (‘David David’, ‘Who is that guy?’); SELECT * FROM user WHERE name LIKE “%David%”; +———————+———————————+ | name | comments | +———————+———————————+ | David Pilato | Developer at elastic | | David Gageot | Engineer at Doctolib | | David David | Who is that guy? | +———————+———————————+ David
Search for terms INSERT INTO user VALUES INSERT INTO user VALUES customs service’); INSERT INTO user VALUES INSERT INTO user VALUES (‘David Pilato’, ‘Developer at elastic’); (‘Malloum Laya’, ‘Worked with David at french (‘David Gageot’, ‘Engineer at Doctolib’); (‘David David’, ‘Who is that guy?’); SELECT * FROM user WHERE name LIKE “%David Pilato%”; +———————+———————————+ | name | comments | +———————+———————————+ | David Pilato | Developer at elastic | +———————+———————————+ David Pilato
Search with inverted terms INSERT INTO user VALUES INSERT INTO user VALUES customs service’); INSERT INTO user VALUES INSERT INTO user VALUES (‘David Pilato’, ‘Developer at elastic’); (‘Malloum Laya’, ‘Worked with David at french (‘David Gageot’, ‘Engineer at Doctolib’); (‘David David’, ‘Who is that guy?’); SELECT * FROM user WHERE name LIKE “%Pilato David%”; Empty set (0,00 sec) SELECT * FROM user WHERE name LIKE “%Pilato%David%”; Empty set (0,00 sec) Pilato David
Search for terms INSERT INTO user VALUES INSERT INTO user VALUES customs service’); INSERT INTO user VALUES INSERT INTO user VALUES (‘David Pilato’, ‘Developer at elastic’); (‘Malloum Laya’, ‘Worked with David at french (‘David Gageot’, ‘Engineer at Doctolib’); (‘David David’, ‘Who is that guy?’); SELECT * FROM user WHERE name LIKE “%David%” AND name LIKE “%Pilato%”; +———————+———————————+ | name | comments | +———————+———————————+ | David Pilato | Developer at elastic | +———————+———————————+ Pilato David
Search in two fields INSERT INTO user VALUES INSERT INTO user VALUES customs service’); INSERT INTO user VALUES INSERT INTO user VALUES (‘David Pilato’, ‘Developer at elastic’); (‘Malloum Laya’, ‘Worked with David at french (‘David Gageot’, ‘Engineer at Doctolib’); (‘David David’, ‘Who is that guy?’); SELECT * FROM user WHERE name LIKE “%David%” OR comments LIKE “%David%”; +———————+——————————————————————-+ | name | comments | +———————+——————————————————————-+ | David Pilato | Developer at elastic | | Malloum Laya | Worked with David at french customs service | | David Gageot | Engineer at Doctolib | | David David | Who is that guy? | +———————+——————————————————————-+ David
Search with typos INSERT INTO user VALUES INSERT INTO user VALUES customs service’); INSERT INTO user VALUES INSERT INTO user VALUES (‘David Pilato’, ‘Developer at elastic’); (‘Malloum Laya’, ‘Worked with David at french (‘David Gageot’, ‘Engineer at Doctolib’); (‘David David’, ‘Who is that guy?’); SELECT * FROM user WHERE name LIKE “%Dadid%”; Empty set (0,00 sec) Dadid
Search with typos INSERT INTO user VALUES INSERT INTO user VALUES customs service’); INSERT INTO user VALUES INSERT INTO user VALUES (‘David Pilato’, ‘Developer at elastic’); (‘Malloum Laya’, ‘Worked with David at french (‘David Gageot’, ‘Engineer at Doctolib’); (‘David David’, ‘Who is that guy?’); SELECT * FROM user WHERE name LIKE “%adid%” OR name LIKE “%D_did%” OR name LIKE “%Da_id%” OR name LIKE “%Dad_d%” OR name LIKE “%Dadi%”; +———————+———————————+ | name | comments | +———————+———————————+ | David Pilato | Developer at elastic | | David Gageot | Engineer at Doctolib | | David David | Who is that guy? | +———————+———————————+ Dadid
User Interface
What is a search engine? ● Index engine (indexing documents) ● Search engine (within the created indices)
Demo time!
Elasticsearch You Know, for Search
GET /_analyze { “char_filter”: [ “html_strip” ], “tokenizer”: “standard”, “filter”: [ “lowercase”, “stop”, “snowball” ], “text”: “These are <em>not</em> the droids you are looking for.” }
“char_filter”: “html_strip” These are <em>not</em> the droids you are looking for. These are not the droids you are looking for.
“tokenizer”: “standard” These are not the droids you are looking for. These are not the droids you are looking for
“filter”: “lowercase” These are not the droids you are looking for these are not the droids you are looking for
“filter”: “stop” These are not the droids you are looking for these are not the droids you are looking for droids you looking
“filter”: “snowball” These are not the droids you are looking for these are not the droids you are looking for droids you droid you looking look
These are <em>not</em> the droids you are looking for. { “tokens”: [{ “token”: “droid”, “start_offset”: 27, “end_offset”: 33, “type”: “<ALPHANUM>”, “position”: 4 },{ “token”: “you”, “start_offset”: 34, “end_offset”: 37, “type”: “<ALPHANUM>”, “position”: 5 }, { “token”: “look”, “start_offset”: 42, “end_offset”: 49, “type”: “<ALPHANUM>”, “position”: 7 }]}
Elasticsearch You Know, for Vector Search
Example: 1-dimensional vector Character Vector [ 1 ] ] Realistic
[ Embeddings represent your data Cartoon 1
represent different data aspects Human Character Vector [ 1, 1 Realistic Cartoon ] ] Machine
[ Multiple dimensions 1, 0
is grouped together Human Character Vector [ 1.0, 1.0 Realistic Cartoon 1.0, 0.0 [ 1.0, 0.8 ] ] ]
Machine
[ Similar data
Vector search ranks objects by similarity (~relevance) to the query Human Rank Query 1 Realistic Cartoon 2 3 4 5 Machine Result
Data Ingestion and Embedding Generation POST /_doc { “_id”:”product-1234”, “product_name”:”Summer Dress”, “description”:”Our best-selling…”, “Price”: 118, “color”:”blue”, “fabric”:”cotton”, “fabric”:”cotton” } “desc_embedding”:[0.452,0.3242,…], “desc_embedding”:[0.452,0.3242,…] } “img_embedding”:[0.012,0.0,…] } Source data POST /_doc
Vector Query GET product-catalog/_search { “query” : { “bool”: { “must”: [{ “knn”: { “field”: “desc_embbeding”, “num_candidates”: 50, “query_vector”: [0.123, 0.244,…] } }], “filter”: { “term”: { “department”: “women” } } } } }, “size”: 10
Similarity Human q cos(θ) = d1 d2 Realistic θ q⃗ × d ⃗ | q⃗ | × | d |⃗ _score = 1 + cos(θ) 2
Similarity: cosine (cosine) θ Similar vectors θ close to 0 cos(θ) close to 1 1+1 _score = =1 2 θ Orthogonal vectors θ close to 90° cos(θ) close to 0 1+0 _score = = 0.5 2 θ Opposite vectors θ close to 180° cos(θ) close to -1 1−1 _score = =0 2
https://djdadoo.pilato.fr/
https://github.com/dadoonet/music-search/
One Search AI Platform Two Out-of-the-Box Solutions The Freedom to Build Anything Out-of-the-Box Solutions Elastic Observability Build Your Own Elastic Security Elastic Search The Elastic Search AI Platform Ingest Secure & Scalable Storage AI / ML Search Visualization Workflow Automation
www.meetup.com/ElasticFR @elasticfr discuss.elastic.co
Thank You
Vous utilisez encore des requêtes SQL pour faire des recherches ? Vos utilisateurs vous reprochent de ne pas pouvoir chercher sur toutes les rubriques ? Votre temps de réponse moyen est supérieur à la demi-seconde avec seulement quelques millions de documents ? Il vous faut 3 jours pour produire des statistiques sur vos données ? Vous rêvez d’offrir une recherche “à la google” sur les données de votre SI ?
Ne cherchez plus ! Cette conférence est faite pour vous.
Venez découvrir comment Elasticsearch peut résoudre ces problèmes et bien d’autres lors d’une session basée principalement sur des démonstrations live.
Resources
The following resources were mentioned during the presentation or are useful additional information.
-
Ekino
-
Person dataset injector
-
Dev Console demo
You can copy/paste this in the Kibana Dev Console.
-
Music Search demo
Buzz and feedback
Here’s what was said about this presentation on social media.
