Indexing your office documents with Elastic stack and FSCrawler

Slide 1

Indexing your office documents with Elastic and FSCrawler David Pilato Developer | Evangelist, Community @dadoonet 1

Slide 2

Slide 3

Slide 4

Parsing a stream and getting content and metadata static void extractTextAndMetadata(InputStream stream) throws Exception { BodyContentHandler handler = new BodyContentHandler(); Metadata metadata = new Metadata(); try (stream) { new DefaultParser().parse(stream, handler, metadata, new ParseContext()); String extractedText = handler.toString(); String title = metadata.get(TikaCoreProperties.TITLE); String keywords = metadata.get(TikaCoreProperties.KEYWORDS); String author = metadata.get(TikaCoreProperties.CREATOR); } }

Slide 5

ingest-attachment plugin extracting from BASE64 or CBOR 5

Slide 6

An ingest pipeline

Slide 7

ingest-attachment processor plugin using Tika behind the scene

Slide 8

Demo https://cloud.elastic.co 8

Slide 9

FSCrawler You know, for files… 9

Slide 10

Slide 11

Disclaimer This project is a community project. It is not officially supported by Elastic. Support is only provided by FSCrawler community on discuss and stackoverflow. http://discuss.elastic.co/ https://stackoverflow.com/questions/tagged/fscrawler

Slide 12

FSCrawler Architecture FSCrawler Local Dir JSON (noop) Mount Point XML SSH / SCP / FTP Apache Tika ES 6/7/8 HTTP Rest Inputs Filters Outputs

Slide 13

FSCrawler Key Features • • • Much more formats than ingest attachment plugin OCR (Tesseract) Much more metadata than ingest attachment plugin (See https://fscrawler.readthedocs.io/en/latest/admin/fs/elasticsearch.html#generated-fields) • Language detection (But see also https://github.com/spinscale/elasticsearch-ingest-langdetect)

Slide 14

Documentation • • • • https://fscrawler.readthedocs.io/ https://fscrawler.readthedocs.io/en/latest/user/tutorial.html https://fscrawler.readthedocs.io/en/latest/user/formats.html https://fscrawler.readthedocs.io/en/latest/admin/fs/index.html https://fscrawler.readthedocs.io/en/latest/

Slide 15

Demo https://cloud.elastic.co 15

Slide 16

FSCrawler even better with a UI 16

Slide 17

FSCrawler Architecture FSCrawler Local Dir JSON (noop) Mount Point XML SSH / SCP / FTP Apache Tika WP 7/8 Filters Outputs ES 6/7/8 HTTP Rest Inputs

Slide 18

Demo https://cloud.elastic.co 18

Slide 19

Be t 8. a 2 Network drives connector package for Enterprise Search https://github.com/elastic/enterprise-search-network-drives-connector/

Slide 20

Thanks! PR are warmly welcomed! https://github.com/dadoonet/fscrawler