Repository files navigation

This repository is now deprecated. Please useSimple Transformers instead.

The underlyingPytorch-Transformers library by HuggingFace has been updated substantially since this repo was created. As such, this repo might not be compatible with the current version of the Hugging Face Transformers library. This repo will not be updated further.

I recommend usingSimple Transformers (based on the updated Hugging Face library) as it is regularly maintained, feature rich, as well as (much) easier to use.

This repository is based on thePytorch-Transformers library by HuggingFace. It is intended as a starting point for anyone who wishes to use Transformer models in text classification tasks.

Please refer to thisMedium article for further information on how this project works.

Check out the new librarysimpletransformers for one line training and evaluating!

• Setup
  • Simple Transformers
  • Quickstart using Colab
  • With Conda
• Usage
  • Yelp Demo
  • Custom Datasets
  • Current Pretrained Models
  • Evaluation Metrics
• Acknowledgements

If you want to go directly to training, evaluating, and predicting with Transformer models, take a look at theSimple Transformers library. It's the easiest way to use Transformers for text classification with only 3 lines of code required. It's based on this repo but is designed to enable the use of Transformers without having to worry about the low level details. However, ease of usage comes at the cost of less control (and visibility) over how everything works.

Try thisGoogle Colab Notebook for a quick preview. You can run all cells without any modifications to see how everything works. However, due to the 12 hour time limit on Colab instances, the dataset has been undersampled from 500 000 samples to about 5000 samples. For such a tiny sample size, everything should complete in about 10 minutes.

1. Install Anaconda or Miniconda Package Manager fromhere
2. Create a new virtual environment and install packages.
   conda create -n transformers python pandas tqdm jupyter
conda activate transformers
   If using cuda:
   conda install pytorch cudatoolkit=10.0 -c pytorch
   else:
   conda install pytorch cpuonly -c pytorch
conda install -c anaconda scipy
conda install -c anaconda scikit-learn
pip install pytorch-transformers
pip install tensorboardX
3. Clone repo.git clone https://github.com/ThilinaRajapakse/pytorch-transformers-classification.git

This demonstration uses the Yelp Reviews dataset.

Linux users can executedata_download.sh to download and set up the data files.

If you are doing it manually;

1. DownloadYelp Reviews Dataset.
2. Extracttrain.csv andtest.csv and place them in the directorydata/.

Once the download is complete, you can run thedata_prep.ipynb notebook to get the data ready for training.

Finally, you can run therun_model.ipynb notebook to fine-tune a Transformer model on the Yelp Dataset and evaluate the results.

The table below shows the currently available model types and their models. You can use any of these by setting themodel_type andmodel_name in theargs dictionary. For more information about pretrained models, seeHuggingFace docs.

When working with your own datasets, you can create a script/notebook similar todata_prep.ipynb that will convert the dataset to a Pytorch-Transformer ready format.

The data needs to be intsv format, with four columns, and no header.

This is the required structure.

• guid: An ID for the row.
• label: The label for the row (should be an int).
• alpha: A column of the same letter for all rows. Not used in classification but still expected by theDataProcessor.
• text: The sentence or sequence of text.

The evaluation process in therun_model.ipynb notebook outputs the confusion matrix, and the Matthews correlation coefficient. If you wish to add any more evaluation metrics, simply edit theget_eval_reports() function in the notebook. This function takes the predictions and the ground truth labels as parameters, therefore you can add any custom metrics calculations to the function as required.

None of this would have been possible without the hard work by the HuggingFace team in developing thePytorch-Transformers library.