Un moteur de recherche de documents d’entreprise

A presentation at JDLL in April 2023 in Lyon, France by David Pilato

Slide 1

Slide 1

0 p e t t! s r n a u t r s e e s w a e l P fore be Indexer ses documents bureautique avec la suite Elastic et FSCrawler David Pilato Developer | Evangelist, Community @dadoonet https://github.com/dadoonet/JDLL

Slide 2

Slide 2

Lab 0 setup https://github.com/dadoonet/JDLL 2

Slide 3

Slide 3

The Elastic Search Platform Enterprise Search Observability Security Kibana Explore, Visualize, Engage Elasticsearch Store, Search, Analyze Integrations Connect, Collect, Alert Public cloud Hybrid On-premises

Slide 4

Slide 4

Lab 1 indexing json documents 4

Slide 5

Slide 5

ingest-attachment processor extracting from BASE64 or CBOR 5

Slide 6

Slide 6

Slide 7

Slide 7

Slide 8

Slide 8

Parsing a stream and getting content and metadata static void extractTextAndMetadata(InputStream stream) throws Exception { BodyContentHandler handler = new BodyContentHandler(); Metadata metadata = new Metadata(); try (stream) { new DefaultParser().parse(stream, handler, metadata, new ParseContext()); String extractedText = handler.toString(); String title = metadata.get(TikaCoreProperties.TITLE); String keywords = metadata.get(TikaCoreProperties.KEYWORDS); String author = metadata.get(TikaCoreProperties.CREATOR); } }

Slide 9

Slide 9

An ingest pipeline

Slide 10

Slide 10

ingest-attachment processor using Tika behind the scene

Slide 11

Slide 11

Lab 2 ingest attachment 11

Slide 12

Slide 12

FSCrawler You know, for files… 12

Slide 13

Slide 13

Slide 14

Slide 14

Disclaimer This project is a community project. It is not officially supported by Elastic. Support is only provided by FSCrawler community on discuss and stackoverflow. http://discuss.elastic.co/ https://stackoverflow.com/questions/tagged/fscrawler

Slide 15

Slide 15

FSCrawler Architecture FSCrawler Local Dir JSON (noop) Mount Point XML SSH / SCP / FTP Apache Tika ES 6/7/8 HTTP Rest Inputs Filters Outputs

Slide 16

Slide 16

Lab 3 fscrawler 16

Slide 17

Slide 17

FSCrawler even better with a UI 17

Slide 18

Slide 18

FSCrawler Architecture FSCrawler Local Dir JSON (noop) Mount Point XML SSH / SCP / FTP Apache Tika WP 7/8 Filters Outputs ES 6/7/8 HTTP Rest Inputs

Slide 19

Slide 19

Lab 4 workplace search 19

Slide 20

Slide 20

Si nc 8. e 2 Network drives connector package for Enterprise Search https://github.com/elastic/enterprise-search-network-drives-connector/

Slide 21

Slide 21

Thanks! PR are warmly welcomed! https://github.com/dadoonet/fscrawler