Implementing the procedures for creating and maintaining an index of scientific publications

Employer: Institute of Neuroinformatics University of Zurich & ETH Zurich Project Objective: Creating an ElasticSearch index of scientific publications (metadata and full-text) by aggregating content from various data sources (i.e. scientific publication databases). Keeping the ElasticSearch index updated as new content is added to the data sources. Implementation of a flexible workflow to integrate additional future data sources. Being able to handle changes to the APIs of data sources. Technologies to use: Node.js, Python, Elastic Stack, Docker. Open for other suggestions Data sources (example list): 1) Crossref ([login to view URL]): contains the metadata of all publications having a digital object identifier (DOI). Content can be downloaded by querying the database through a REST API in a rate-limited fashion. 2) MEDLINE/Pubmed ([login to view URL]): contains metadata and abstracts of most publications related to life sciences. Also contains publications not having DOIs. Content can be bulk downloaded. 3) CORE ([login to view URL]): An aggregate database of most open access publications, including full text of some. If full-text is not available (e.g. papers from arXiv), a link is provided to the original source, which should be crawled to fetch the full text content. Contains data from large databases such as arXiv, CiteSeerX. Database can be bulk downloaded. The workflow should be flexible to include additional data sources as they become available. Project Parts/Tasks: Different tasks should be handled by individual Docker microservices 1) Downloading and parsing the entire content of listed data sources and indexing this in individual ElasticSearch indices 2) Parsing PDF files to extract the full-text (unstructured) if data source only provides PDFs (e.g. CORE) 3) Aggregating downloaded content from data sources in an “meta” ElasticSearch index 4) Keeping meta index updated as new publications appear 5) Maintaining the meta index: handling duplicates, handling different versions of a publication (e.g. arXiv preprints vs their final publication in a journal), adding new fields to the index, etc. 6) Index backup, revisioning and rollback tasks ElasticSearch index fields: Title, journal, page, publication date, authors, affiliations, abstract, full-text, references, figures, data source, data source ID, DOI….. [to be extended]