Search: a new era

A presentation at jPrime 2024 in in Sofia, Bulgaria by David Pilato

Search is not just traditional TF/IDF any more but the current trend of machine learning and models has opened another dimension for search.

This talk gives an overview of:

  • “Classic” search and its limitations
  • What is a model and how can you use it
  • How to use vector search or hybrid search in Elasticsearch
  • Where OpenAI’s ChatGPT or similar LLMs come into play to with Elastic

This talk covers the state of the art in terms of search nowadays: BM25, Vector search, Embeddings, Hybrid Search, Reciprocal Rank Fusion, and OpenAI integration. The main demo covers how to generate embeddings from a piece of music and then use the techniques we learned to propose the most probable version of it when we hum a song.

Resources

The following resources were mentioned during the presentation or are useful additional information.

Buzz and feedback

Here’s what was said about this presentation on social media.