Repository files navigation

This repository consists of:

• torchtext.datasets: The raw text iterators for common NLP datasets
• torchtext.data: Some basic NLP building blocks (tokenizers, metrics, functionals etc.)
• torchtext.nn: NLP related modules
• torchtext.vocab: Vocab and Vectors related classes and factory functions
• examples: Example NLP workflows with PyTorch and torchtext library.

Note: The legacy code discussed intorchtext v0.7.0 release note has been retired totorchtext.legacy folder. Those legacy code will not be maintained by the development team, and we plan to fully remove them in the future release. Seetorchtext.legacy folder for more details.

We recommend Anaconda as a Python package management system. Please refer topytorch.org for the details of PyTorch installation. The following are the correspondingtorchtext versions and supported Python versions.

Using conda:

conda install -c pytorch torchtext

Using pip:

pip install torchtext

If you want to use English tokenizer fromSpaCy, you need to install SpaCy and download its English model:

pip install spacypython -m spacy download en_core_web_sm

Alternatively, you might want to use theMoses tokenizer port inSacreMoses (split fromNLTK). You have to install SacreMoses:

pip install sacremoses

For torchtext 0.5 and below,sentencepiece:

conda install -c powerai sentencepiece

To build torchtext from source, you needgit,CMake and C++11 compiler such asg++.:

git clone https://github.com/pytorch/text torchtextcd torchtextgit submodule update --init --recursive# Linuxpython setup.py clean install# OSXMACOSX_DEPLOYMENT_TARGET=10.9 CC=clang CXX=clang++ python setup.py clean install# or ``python setup.py develop`` if you are making modifications.

Note

When building from source, make sure that you have the same C++ compiler as the one used to build PyTorch. A simple way is to build PyTorch from source and use the same environment to build torchtext.If you are using the nightly build of PyTorch, checkout the environment it was built withconda (here) andpip (here).

Find the documentationhere.

The datasets module currently contains:

• Language modeling: WikiText2, WikiText103, PennTreebank, EnWik9
• Machine translation: IWSLT2016, IWSLT2017, Multi30k
• Sequence tagging (e.g. POS/NER): UDPOS, CoNLL2000Chunking
• Question answering: SQuAD1, SQuAD2
• Text classification: AG_NEWS, SogouNews, DBpedia, YelpReviewPolarity, YelpReviewFull, YahooAnswers, AmazonReviewPolarity, AmazonReviewFull, IMDB

For example, to access the raw text from the AG_NEWS dataset:

>>>fromtorchtext.datasetsimportAG_NEWS>>>train_iter=AG_NEWS(split='train')>>># Iterate with for loop>>>for (label,line)intrain_iter:>>>print(label,line)>>># Or send to DataLoader>>>fromtorch.utils.dataimportDataLoader>>>train_iter=AG_NEWS(split='train')>>>dataloader=DataLoader(train_iter,batch_size=8,shuffle=False)

To get started with torchtext, users may refer to the following tutorials available on PyTorch website.

• Text classification with AG_NEWS dataset
• Translation trained with Multi30k dataset using transformers and torchtext
• Language modeling using transforms and torchtext

In the v0.9.0 release, we moved the following legacy code totorchtext.legacy. This is part of the work to revamp the torchtext library and the motivation has been discussed inIssue #664:

• torchtext.legacy.data.field
• torchtext.legacy.data.batch
• torchtext.legacy.data.example
• torchtext.legacy.data.iterator
• torchtext.legacy.data.pipeline
• torchtext.legacy.datasets

We have amigration tutorial to help users switch to the torchtext datasets inv0.9.0 release. For the users who still want the legacy components, they can addlegacy to the import path.

In the v0.10.0 release, we retire the Vocab class totorchtext.legacy. Users can still access the legacy Vocab viatorchtext.legacy.vocab. This class has been replaced by a Vocab module that is backed by efficient C++ implementation and provides common functional APIs for NLP workflows.

This is a utility library that downloads and prepares public datasets. We do not host or distribute these datasets, vouch for their quality or fairness, or claim that you have license to use the dataset. It is your responsibility to determine whether you have permission to use the dataset under the dataset's license.

If you're a dataset owner and wish to update any part of it (description, citation, etc.), or do not want your dataset to be included in this library, please get in touch through a GitHub issue. Thanks for your contribution to the ML community!