Indexing your office documents with Elastic and FSCrawler y o David Pilat Developer | Evangelist, Communit @dadoonet
Slide 1
Slide 2
Slide 3
Slide 4
and getting content and metadata ; { ; ; ; ; ; ; { static void extractTextAndMetadata(InputStream stream) throws Exception BodyContentHandler handler = new BodyContentHandler() Metadata metadata = new Metadata() try (stream) new DefaultParser().parse(stream, handler, metadata, new ParseContext()) String extractedText = handler.toString() String title = metadata.get(TikaCoreProperties.TITLE) String keywords = metadata.get(TikaCoreProperties.KEYWORDS) String author = metadata.get(TikaCoreProperties.CREATOR) } } Parsing a stream
Slide 5
n
ingest-attachment plugi extracting from BASE64 or CBOR
Slide 6
An ingest pipeline
Slide 7
ingest-attachment processor plugin using Tika behind the scene
Slide 8
Demo 8 https://cloud.elastic.co
Slide 9
r FSCrawle You know, for files…
Slide 10
Slide 11
Disclaime This project is a community project. It is not officially supported by Elastic. Support is only provided by FSCrawler community on discuss and stackoverflow.
r http://discuss.elastic.co/ https://stackoverflow.com/questions/tagged/fscrawler
Slide 12
FSCrawle Architecture FSCrawler Local Dir JSON (noop) ES6 Mount Point XML ES7 SSH / SCP Apache Tika HTTP Rest r Inputs Filters Outputs
Slide 13
FSCrawle Key Features • • • Much more formats than ingest attachment plugi OCR (Tesseract Much more metadata than ingest attachment plugin (See https://fscrawler.readthedocs.io/en/latest/admin/fs/elasticsearch.html#generated-fields) n r Language detection ) •
Slide 14
Documentation • • • • https://fscrawler.readthedocs.io/ https://fscrawler.readthedocs.io/en/latest/user/tutorial.html https://fscrawler.readthedocs.io/en/latest/user/formats.html https://fscrawler.readthedocs.io/en/latest/admin/fs/index.html https://fscrawler.readthedocs.io/en/latest/
Slide 15
Demo 15 https://cloud.elastic.co
Slide 16
r FSCrawle even better with a UI
Slide 17
FSCrawle r Workplace Search integration
Slide 18
FSCrawle Architecture FSCrawler Local Dir JSON (noop) ES6 Mount Point XML ES7 SSH / SCP Apache Tika WP7 Filters Outputs HTTP Rest r Inputs
Slide 19
Slide 20
Demo 20 https://cloud.elastic.co
Slide 21
Thanks PR are warmly welcomed! ! https://github.com/dadoonet/fscrawler
