Repository files navigation

Code for Findings of EMNLP 2022 short paper"CDGP: Automatic Cloze Distractor Generation based on Pre-trained Language Model".

• Paper
• Demo
• ACL

• paper/: "CDGP: Automatic Cloze Distractor Generation based on Pre-trained Language Model"
• models/: models in CDGP
  • CSG/: the models as Candidate Set Generator
  • DS/: the models as Distractor Selector
• datasets/: datasets for fine-tuneing and testing
  • CLOTH.zip: CLOTH dataset
  • DGen.zip: DGen dataset
• fine-tune/: code for fine-tuning
• test/: code for testing
  • dis_generator(BERT).py: distractors generator based on BERT
  • dis_generator(SciBERT).py: distractors generator based on SciBERT
  • dis_generator(RoBERTa).py: distractors generator based on RoBERTa
  • dis_generator(BART).py: distractors generator based on BART
  • dis_evaluator.py: distractors evaluator
  • results/: results of distractors generator
  • evaluations/: evaluations of distractors evaluator
• demo.ipynb: code for CDGP demo

Models are available at Hugging Face.

Its input are stem and answer, and output is candidate set of distractors.

Its input are stem, answer and candidate set of distractors, and output are top 3 distractors.

fastText:cdgp-ds-fasttext

Datasets are available at Hugging Face and GitHub.

CLOTH is a dataset which is a collection of nearly 100,000 cloze questions from middle school and high school English exams. The detail of CLOTH dataset is shown below.

You can download CLOTH dataset fromHugging Face orGitHub.

DGen is a cloze questions dataset which covers multiple domains including science, vocabulary, common sense and trivia. It is compiled from a wide variety of datasets including SciQ, MCQL, AI2 Science Questions, etc. The detail of DGen dataset is shown below.

You can download CLOTH dataset fromHugging Face orGitHub.

The evaluations of these model as a Candidate Set Generator in CDGP is shown as follows:

1. Clone or download this repo.

git clone https://github.com/AndyChiangSH/CDGP.git

2. Move into this repo.

cd ./CDGP/

3. Setup a virtual environment.

python -m venv CDGP-env

Python version: 3.8.8

4. Pip install the required packages.

pip install -r requirements.txt

Our model is fine-tuned onColab, so you can uploadthese Jupyter Notebook to Colab and run it by yourself!

We are testing in local, so we need to download the datasets and models.

1. Unzip the CLOTH or DGen datasets in/datasets/.
2. CSG models will download from Hugging Face when you run the code, so you don't have to do anything!
3. If you want to use your own CSG model, you can put it in the new directory/models/CSG/.
4. However, you have to download the DS models by yourself.
5. Then, move the DS models into the new directory/models/DS/.
6. Run/test/dis_generator(BERT).py to generate the distractors based on BERT.
7. Run/test/dis_generator(SciBERT).py to generate the distractors based on SciBERT.
8. Run/test/dis_generator(RoBERTa).py to generate the distractors based on RoBERTa.
9. Run/test/dis_generator(BART).py to generate the distractors based on BART.
10. Check the generating results as.json files in/test/results/.
11. Run/test/dis_evaluator.py to evaluate the generating results.
12. Check the evaluations as.csv file in/test/evaluations/.

@inproceedings{chiang-etal-2022-cdgp,    title = "{CDGP}: Automatic Cloze Distractor Generation based on Pre-trained Language Model",    author = "Chiang, Shang-Hsuan  and      Wang, Ssu-Cheng  and      Fan, Yao-Chung",    booktitle = "Findings of the Association for Computational Linguistics: EMNLP 2022",    month = dec,    year = "2022",    address = "Abu Dhabi, United Arab Emirates",    publisher = "Association for Computational Linguistics",    url = "https://aclanthology.org/2022.findings-emnlp.429",    pages = "5835--5840",    abstract = "Manually designing cloze test consumes enormous time and efforts. The major challenge lies in wrong option (distractor) selection. Having carefully-design distractors improves the effectiveness of learner ability assessment. As a result, the idea of automatically generating cloze distractor is motivated. In this paper, we investigate cloze distractor generation by exploring the employment of pre-trained language models (PLMs) as an alternative for candidate distractor generation. Experiments show that the PLM-enhanced model brings a substantial performance improvement. Our best performing model advances the state-of-the-art result from 14.94 to 34.17 (NDCG@10 score). Our code and dataset is available at https://github.com/AndyChiangSH/CDGP.",}

• Shang-Hsuan Chiang (@AndyChiangSH)
• Ssu-Cheng Wang (@shiro-wang)
• Yao-Chung Fan (@yfan)