Elasticsearch
A presentation at BBL Accenture (Private Event) in in Amsterdam, Netherlands by David Pilato
Elastic Stack Overview Search. Observe. Protect.
Who? $ curl http://localhost:9200/speaker/_doc/dpilato { “name” : “David Pilato”, “jobs” : [ { “name” : “SRA Europe (SSII)”, “date” : “1995” }, { “name” : “SFR”, “date” : “1997” }, { “name” : “e-Brands / Vivendi”, “date”: “2000” }, { “name” : “DGDDI (douane)”, “date” : “2005” }, { “name” : “elastic”, “date” : “2013” } ], “motivations” : [ “family”, “job”, “deejay” ], “blog” : “http://david.pilato.fr/”, “twitter” : [ “@dadoonet”, “@elasticfr” ], “email” : “david@pilato.fr” } 2
The Elastic Platform Enterprise Search Observability Security Kibana Explore, Visualize, Engage Elasticsearch Store, Search, Analyze Integrations Connect, Collect, Alert Public cloud Hybrid On-premises
FILEBEAT MASTER (3) METRICBEAT PACKETBEAT WINGLOBEAT AUDITBEAT HEARTBEAT CUSTOM UI INGEST (X) Coordinating (X) DATA - HOT (X) FUNCTION BEAT KAFKA ELASTICSEARCH CLIENTS DATA - WARM (X) REDIS DATA STORE SOCIAL 4 WEB APIS SENSORS MESSAGING QUEUE WORKERS (2+) ALERTING (X) MACHINE LEARNING (2+) INSTANCES (X)
A typical search implementation… CREATE TABLE user ( name VARCHAR(100), comments VARCHAR(1000) ); INSERT INTO user VALUES (‘David Pilato’, ‘Developer at elastic’); INSERT INTO user VALUES (‘Malloum Laya’, ‘Worked with David at french customs service’); INSERT INTO user VALUES (‘David Gageot’, ‘Engineer at Doctolib’); INSERT INTO user VALUES (‘David David’, ‘Who is that guy?’); David 7
Search on term INSERT INSERT french INSERT INSERT INTO user VALUES (‘David Pilato’, ‘Developer at elastic’); INTO user VALUES (‘Malloum Laya’, ‘Worked with David at customs service’); INTO user VALUES (‘David Gageot’, ‘Engineer at Doctolib’); INTO user VALUES (‘David David’, ‘Who is that guy?’); SELECT * FROM user WHERE name=”David”; Empty set (0,00 sec) David 8
Search like INSERT INSERT french INSERT INSERT INTO user VALUES (‘David Pilato’, ‘Developer at elastic’); INTO user VALUES (‘Malloum Laya’, ‘Worked with David at customs service’); INTO user VALUES (‘David Gageot’, ‘Engineer at Doctolib’); INTO user VALUES (‘David David’, ‘Who is that guy?’); SELECT * FROM user WHERE name LIKE “%David%”; +———————+———————————+ | name | comments | +———————+———————————+ | David Pilato | Developer at elastic | | David Gageot | Engineer at Doctolib | | David David | Who is that guy? | +———————+———————————+ David 9
Search for terms INSERT INSERT french INSERT INSERT INTO user VALUES (‘David Pilato’, ‘Developer at elastic’); INTO user VALUES (‘Malloum Laya’, ‘Worked with David at customs service’); INTO user VALUES (‘David Gageot’, ‘Engineer at Doctolib’); INTO user VALUES (‘David David’, ‘Who is that guy?’); SELECT * FROM user WHERE name LIKE “%David Pilato%”; +———————+———————————+ | name | comments | +———————+———————————+ | David Pilato | Developer at elastic | +———————+———————————+ David Pilato 10
Search with inverted terms INSERT INSERT french INSERT INSERT INTO user VALUES (‘David Pilato’, ‘Developer at elastic’); INTO user VALUES (‘Malloum Laya’, ‘Worked with David at customs service’); INTO user VALUES (‘David Gageot’, ‘Engineer at Doctolib’); INTO user VALUES (‘David David’, ‘Who is that guy?’); SELECT * FROM user WHERE name LIKE “%Pilato David%”; Empty set (0,00 sec) SELECT * FROM user WHERE name LIKE “%Pilato%David%”; Empty set (0,00 sec) Pilato David 11
Search for terms INSERT INSERT french INSERT INSERT INTO user VALUES (‘David Pilato’, ‘Developer at elastic’); INTO user VALUES (‘Malloum Laya’, ‘Worked with David at customs service’); INTO user VALUES (‘David Gageot’, ‘Engineer at Doctolib’); INTO user VALUES (‘David David’, ‘Who is that guy?’); SELECT * FROM user WHERE name LIKE “%David%” AND name LIKE “%Pilato%”; +———————+———————————+ | name | comments | +———————+———————————+ | David Pilato | Developer at elastic | +———————+———————————+ Pilato David 12
Search in two fields INSERT INSERT french INSERT INSERT INTO user VALUES (‘David Pilato’, ‘Developer at elastic’); INTO user VALUES (‘Malloum Laya’, ‘Worked with David at customs service’); INTO user VALUES (‘David Gageot’, ‘Engineer at Doctolib’); INTO user VALUES (‘David David’, ‘Who is that guy?’); SELECT * FROM user WHERE name LIKE “%David%” OR comments LIKE “%David%”; +———————+——————————————————————-+ | name | comments | +———————+——————————————————————-+ | David Pilato | Developer at elastic | | Malloum Laya | Worked with David at french customs service | | David Gageot | Engineer at Doctolib | | David David | Who is that guy? | +———————+——————————————————————-+ David 13
Search with typos INSERT INSERT french INSERT INSERT INTO user VALUES (‘David Pilato’, ‘Developer at elastic’); INTO user VALUES (‘Malloum Laya’, ‘Worked with David at customs service’); INTO user VALUES (‘David Gageot’, ‘Engineer at Doctolib’); INTO user VALUES (‘David David’, ‘Who is that guy?’); SELECT * FROM user WHERE name LIKE “%Dadid%”; Empty set (0,00 sec) Dadid 15
Search with typos INSERT INSERT french INSERT INSERT INTO user VALUES (‘David Pilato’, ‘Developer at elastic’); INTO user VALUES (‘Malloum Laya’, ‘Worked with David at customs service’); INTO user VALUES (‘David Gageot’, ‘Engineer at Doctolib’); INTO user VALUES (‘David David’, ‘Who is that guy?’); SELECT * FROM user WHERE name LIKE “%adid%” OR name LIKE “%D_did%” OR name LIKE “%Da_id%” OR name LIKE “%Dad_d%” OR name LIKE “%Dadi%”; +———————+———————————+ | name | comments | +———————+———————————+ | David Pilato | Developer at elastic | | David Gageot | Engineer at Doctolib | | David David | Who is that guy? | +———————+———————————+ 16
User Interface 18
What is a search engine? Index engine (indexing documents) Search engine (within the created indices) 19
Demo time!
The Elastic Platform Enterprise Search Observability Security Kibana Explore, Visualize, Engage Elasticsearch Store, Search, Analyze Integrations Connect, Collect, Alert Public cloud Hybrid On-premises
Elastic Enterprise Search Enterprise Search Observability Security Kibana Explore, Visualize, Engage Elasticsearch Store, Search, Analyze Integrations Connect, Collect, Alert Public cloud Hybrid On-premises
ELASTIC ENTERPRISE SEARCH Search everything, anywhere Easily implement powerful, modern search experiences across your website, app, or digital workplace. Search it all, simply.
Elastic Observability Enterprise Search Observability Security Kibana Explore, Visualize, Engage Elasticsearch Store, Search, Analyze Integrations Connect, Collect, Alert Public cloud Hybrid On-premises
ELASTIC OBSERVABILITY Unified visibility across your entire ecosystem Bring your logs, metrics, and traces together into a single stack so you can monitor, detect, and react to events with speed.
Elastic Security Enterprise Search Observability Security Kibana Explore, Visualize, Engage Elasticsearch Store, Search, Analyze Integrations Connect, Collect, Alert Public cloud Hybrid On-premises
ELASTIC SECURITY Security how it should be: open Elastic Security integrates endpoint security and SIEM to give you prevention, collection, detection, and response capabilities for unified protection across your infrastructure.
Searching for Rides
Searching for Restaurants
Searching for Love
Elastic Virtual https://community.elastic.co/emea-virtual/ @elastic discuss.elastic.co
Do you still use SQL queries to search? Your users blame you for not being able to search any and all data? Your average response time is greater than half a second with only a few million documents? Do you need 3 days to produce statistics on your data? Do you dream of offering a “google search” on your IS data?
Enter Elasticsearch: A simple-to-setup and simple-to-use search service. You simply feed it the information you want to be searchable using its index APIs and use its simple-yet-powerful search APIs to let users search for this information in milliseconds.
Elasticsearch is an open source horizontally-scalable, distributed search and analytics engine based on Apache Lucene. Elasticsearch is one of the most popular projects on Github with 1100+ contributors, 12,000+ forks and 43,000+ commits. Organizations around the world, from startups to governments to large corporations, are using Elasticsearch for numerous use cases to gain real-time insights from large volumes of data.
This talk will introduce you to Elasticsearch in a conceptual and hands-on, practical way. We will talk about documents, indices, search, aggregations and language analysis. But we will also see each of these concepts in action via live demos. We will feed actual information into the Elasticsearch service and later retrieve it by searching for it! At the end of this talk you will be comfortable spinning up your very own Elasticsearch service for your very own application!
Resources
The following resources were mentioned during the presentation or are useful additional information.
-
Kibana Demo Script
This is the Kibana script I’m starting from when doing the talk
-
Accenture
Buzz and feedback
Here’s what was said about this presentation on social media.
