Repository files navigation

Models to perform neural summarization (extractive and abstractive) using machine learning transformers and a tool to convert abstractive summarization datasets to the extractive task.

TransformerSum is a library that aims to make it easy totrain,evaluate, anduse machine learningtransformer models that performautomatic summarization. It features tight integration withhuggingface/transformers which enables the easy usage of awide variety of architectures andpre-trained models. There is a heavy emphasis on codereadability andinterpretability so that both beginners and experts can build new components. Both the extractive and abstractive model classes are written usingpytorch_lightning, which handles the PyTorch training loop logic, enablingeasy usage of advanced features such as 16-bit precision, multi-GPU training, andmuch more.TransformerSum supports both the extractive and abstractive summarization oflong sequences (4,096 to 16,384 tokens) using thelongformer (extractive) andLongformerEncoderDecoder (abstractive), which is a combination ofBART (paper) and the longformer.TransformerSum also contains models that can run on resource-limited devices while still maintaining high levels of accuracy. Models are automatically evaluated with theROUGE metric but human tests can be conducted by the user.

Check outthe documentation for usage details.

• For extractive summarization, compatible with everyhuggingface/transformers transformer encoder model.

• For abstractive summarization, compatible with everyhuggingface/transformers EncoderDecoder and Seq2Seq model.

• Currently, 10+ pre-trained extractive models available to summarize text trained on 3 datasets (CNN-DM, WikiHow, and ArXiv-PebMed).

• Contains pre-trained models that excel at summarization on resource-limited devices: On CNN-DM,mobilebert-uncased-ext-sum achieves about 97% of the performance ofBertSum while containing 4.45 times fewer parameters. It achieves about 94% of the performance ofMatchSum (Zhong et al., 2020), the current extractive state-of-the-art.

• Contains code to train models that excel at summarizing long sequences: Thelongformer (extractive) andLongformerEncoderDecoder (abstractive) can summarize sequences of lengths up to 4,096 tokens by default, but can be trained to summarize sequences of more than 16k tokens.

• Integration withhuggingface/nlp means any summarization dataset in thenlp library can be used for both abstractive and extractive training.

• "Smart batching" (extractive) and trimming (abstractive) support to not perform unnecessary calculations (speeds up training).

• Use ofpytorch_lightning for code readability.

• Extensive documentation.

• Three pooling modes (convert word vectors to sentence embeddings): mean or max of word embeddings in addition to the CLS token.

All pre-trained models (including larger models and other architectures) are located inthe documentation. The below is a fraction of the available models.

Installation is made easy due to conda environments. Simply run this command from the root project directory:conda env create --file environment.yml and conda will create and environment calledtransformersum with all the required packages fromenvironment.yml. The spacyen_core_web_sm model is required for theconvert_to_extractive.py script to detect sentence boundaries.

1. Clone this repository:git clone https://github.com/HHousen/transformersum.git.
2. Change to project directory:cd transformersum.
3. Run installation command:conda env create --file environment.yml.
4. (Optional) If using theconvert_to_extractive.py script then download theen_core_web_sm spacy model:python -m spacy download en_core_web_sm.

Hayden Housen –haydenhousen.com

Distributed under the GNU General Public License v3.0. See theLICENSE for more information.

https://github.com/HHousen

• Code heavily inspired by the following projects:
  • Adapting BERT for Extractive Summariation:BertSum
  • Text Summarization with Pretrained Encoders:PreSumm
  • Word/Sentence Embeddings:sentence-transformers
  • CNN/CM Dataset:cnn-dailymail
  • PyTorch Lightning Classifier:lightning-text-classification
• Important projects utilized:
  • PyTorch:pytorch
  • Training code:pytorch_lightning
  • Transformer Models:huggingface/transformers

All Pull Requests are greatly welcomed.

Questions? Commends? Issues? Don't hesitate to open anissue and briefly describe what you are experiencing (with any error logs if necessary). Thanks.

1. Fork it (https://github.com/HHousen/TransformerSum/fork)
2. Create your feature branch (git checkout -b feature/fooBar)
3. Commit your changes (git commit -am 'Add some fooBar')
4. Push to the branch (git push origin feature/fooBar)
5. Create a new Pull Request