Search, Observability, Security: a new era David Pilato | @dadoonet
Slide 1
Slide 2
Elasticsearch You Know, for Vector Search
Slide 3
Embeddings represent your data Example: 1-dimensional vector Character Vector [ -1 ] Realistic Cartoon [1]
Slide 4
Multiple dimensions represent different data aspects Human Character Vector [ -1, 1 ] Realistic Cartoon Machine [ 1, 0 ]
Slide 5
Similar data is grouped together Human Character Vector [ -1.0, 1.0 ] Realistic Cartoon [ 1.0, 0.0 ] [ -1.0, 0.8 ] Machine
Slide 6
Vector search ranks objects by similarity (~relevance) to the query Human Rank Query 1 Realistic Cartoon 2 3 4 5 Machine Result
Slide 7
Similarity: cosine (cosine) Human q d1 d2 Realistic θ 1 + 𝑐𝑜𝑠(𝜃) _𝑠𝑐𝑜𝑟𝑒 = 2
Slide 8
Data Ingestion and Embedding Generation { “_id”:”product-1234”, “product_name”:”Summer Dress”, “description”:”Our best-selling…”, “Price”: 118, “color”:”blue”, “fabric”:”cotton”, “desc_embedding”:[0.452,0.3242,…], “img_embedding”:[0.012,0.0,…] } Source data POST /_doc
Slide 9
Vector Query GET product-catalog/_search { “knn”: { “field”: “desc_embbeding”, “k”: 5, “num_candidates”: 50, “query_vector”: [0.123, 0.244,…], “filter”: { “term”: { “department”: “women” } } }, “size”: 10 }
Slide 10
ChatGPT Elastic and LLM
Slide 11
Gen AI Search engines
Slide 12
LLM: opportunities and limits your question one answer your question GAI / LLM public internet data
Slide 13
Slide 14
Retrieval Augmented Generation your question the right answer your question + context window GAI / LLM public internet data your business data documents images audio
Slide 15
Demo: Elastic + Azure OpenAI
Slide 16
AI Assistant for security and observability
Slide 17
Slide 18
Slide 19
Search, Observability, Security: a new era David Pilato | @dadoonet
