FSCrawler! You know, for files!
A presentation at Meetup ElasticFR #59 - FSCrawler! You know, for files! by David Pilato
FSCrawler: you know, for files!
Laetitia Richard
David Pilato
Disclaimer
This project is a community project. It is not officially supported by Elastic. Support is only provided by FSCrawler community on discuss and stackoverflow.
FSCrawler Architecture
- Inputs:
- Local Dir
- Mount Point
- SSH / SCP
- HTTP Rest
- Filters
- JSON (noop)
- XML
- Apache Tika
- Outputs
- ES6
- ES7
FSCrawler Key Features
- Much more formats than ingest attachment plugin
- OCR (Tesseract)
- Much more metadata than ingest attachment plugin (See Generated Fields)
- Language detection
Documentation
Documentation
FSCrawler Workplace Search integration
FSCrawler Architecture
It now adds Workplace Search 7
Need to enrich your data?
Need to analyze your data?
Observe and analyze your data
Beware of the settings
- FSCrawler with Workplace Search output is not in watch mode (you can use systemd)
- To transform your Workplace Search index you will have to set dynamic mapping to true first (default is strict)
- If you have other standard Workplace Search connectors, you will have to transform your data in another index because the full sync refresh the content source from scratch
Needs to be done (Help Wanted!)
Avez-vous déjà rêvé de pouvoir indexer vos fichiers locaux, genre PDF, OpenOffice, images et autres dans la suite Elastic pour pouvoir en rechercher facilement le contenu ou les meta-données ? C’est toute la promesse du projet open-source FSCrawler.
Au cours de cette session, vous découvrirez les possibilités de ce projet et ce qu’il apporte par rapport au plugin ingest-attachment processor d’Elasticsearch ainsi qu’une implémentation concrète.
Video
Resources
The following resources were mentioned during the presentation or are useful additional information.
Buzz and feedback
Here’s what was said about this presentation on social media.
