Repository files navigation

This repo contains the code to train, evaluate and reproduce the representation learning models and results on the benchmark introduced inSciRepEval.

Clone the repo and setup the environment as follows:

git clone git@github.com:allenai/scirepeval.gitcd scirepevalconda create -n scirepeval python=3.8conda activate scirepevalpip install -r requirements.txt

Please refer to the following for further usage:

Training - Train multi-task/multi-format transformer models or adapter modules

Inference - Using the trained SciRepEval models to generate embeddings.

Evaluation - Evaluate trained models on custom tasks OR customize existing evaluation config for SciRepEval benchmark tasks

Benchmarking - Simply evaluate models(pretrained from HuggingFace/local checkpoints) on SciRepEval and generate a report

SciRepEval consists of 24 scientific document tasks to train and evaluate scientific document representation models. The tasks are divided across 4 task formats- classificationCLF, regressionRGN, proximity (nearest neighbors) retrievalPRX and ad-hoc searchSRCH. The table below gives a brief overview of the tasks with their HuggingFace datasets config names, if applicable.The benchmark dataset can be downloaded from AWS S3 or HuggingFace as follows:

mkdir scirepeval_data&& mkdir scirepeval_data/train&& mkdir scirepeval_data/test&&cd scirepeval_dataaws s3 --no-sign-request sync s3://ai2-s2-research-public/scirepeval/train trainaws s3 --no-sign-request sync s3://ai2-s2-research-public/scirepeval/testtest

The AWS CLI commands can be run with the--dryrun flag to list the files being copied. The entire dataset is ~24 GB in size.

The training, validation and raw evaluation data is available atallenai/scirepeval, while the labelled test examples are available atallenai/scirepeval_test.

importdatasets#training/validation/eval metadatadataset=datasets.load_dataset(allenai/scirepeval,<hfconfigname>)#labelled test examplesdataset=datasets.load_dataset(allenai/scirepeval_test,<hfconfigname>)

Since we want to evaluate document representations, every dataset consists of two parts: test metadata (text for representation generation available under allenai/scirepeval) and labelled examples (available under allenai/scirepeval_test)

*S2AND requires the evaluation dataset in a specific format so to evaluate your model on the task please followthese instructions.

**Combinations of multiple datasets -1,2,3, also dataset of papers authored by potential reviewers is required for evaluation; hence the multiple dataset configs.

The aggregate benchmark is released underODC-BY license. By downloading this data you acknowledge that you have read and agreed to all the terms in this license.For constituent datasets, also go through the individual licensing requirements, as applicable.

Please cite the SciRepEval work as:

@article{Singh2022SciRepEvalAM,title={SciRepEval: A Multi-Format Benchmark for Scientific Document Representations},author={Amanpreet Singh and Mike D'Arcy and Arman Cohan and Doug Downey and Sergey Feldman},journal={ArXiv},year={2022},volume={abs/2211.13308}}