Repository files navigation

Roadmap

After OpenAI's ChatGPT was released the whole world got caught up in the AI frenzy (and for a good reason). While systems such as Meta's Llama and OpenAI's GPT-4 are incredibly powerful - they only focus on English language.

Similarly, for machine translation systems, Meta released a powerful NLLB ("no language left behind") MT system that supports 202 languages! - but they didn't release open-source checkpoints (they were released under non-commercial CC-NC-BY 4.0 license).

Themain goal of this effort is to release trulyopen-source NLLB checkpoints that can be freely used even for commercial purposes.

Theextended goal of this project is to scale up beyond the original 3.3B parameters dense transformers (7B+) and also support non-English LLMs.

We strongly believe that focusing on multilingual AI models can help democratize this technology, and help businesses & researchers around the world.

Check out ourgetting started guide if you wish to contribute! Or just play around!

👨‍👩‍👧‍👦 JoinAleksa Gordić - The AI Epiphany Discord server and the#open-nllb channel if you want to engage with the rest of the community!

📺Daily YouTube video streams here! 👀

Here is a list of data contributors who are owners of their respective languages (they are all native speakers). 🙏

Below is the original fairseq README:

No Language Left Behind (NLLB) is a first-of-its-kind, AI breakthrough project that open-sources models capable of delivering high-quality translations directly between any pair of 200+ languages — including low-resource languages like Asturian, Luganda, Urdu and more. It aims to help people communicate with anyone, anywhere, regardless of their language preferences.

To enable the community to leverage and build on top of NLLB, we open source all our evaluation benchmarks(FLORES-200, NLLB-MD, Toxicity-200), LID models and training code, LASER3 encoders, data mining code, MMT training and inference code and our final NLLB-200 models and their smaller distilled versions, for easier use and adoption by the research community.

This code repository contains instructions to get the datasets, optimized training and inference code for MMT models, training code for LASER3 encoders as well as instructions for downloading and using the final large NLLB-200 model and the smaller distilled models.In addition to supporting more than 200x200 translation directions, we also provide reliable evaluations of our model on all possible translation directions on the FLORES-200 benchmark. By open-sourcing our code, models and evaluations, we hope to foster even more research in low-resource languages leading to further improvements in the quality of low-resource translation through contributions from the research community.

• Paper
• Website
• Blog
• NLLB Demo
• Request for Proposals : Translation Support for African Languages

All models are licensed under CC-BY-NC 4.0 available inModel LICENSE file. We provide FLORES-200 evaluation results for all the models. For more details see theModeling section README.

⭐ NEW ⭐ : We are releasing all the translations of NLLB-200 MoE model. CheckEvaluation section README for more details.

Please usewget --trust-server-names <url> to download the provided links in proper file format.

LID (LanguageIDentification) model to predict the language of the input text is availablehere underCC-BY-NC 4.0 license.

LASER3 models are available atLASER.

Support for the dense models is available through the Hugging Face Hub under theNLLB tag. It is supported in thetransformers library and the documentation for this model is availablehere.

Input and output languages are entirely customizable with BCP-47 codes used by the FLORES-200 dataset, here's an example usage with a translation from Romanian to German:

>>>fromtransformersimportAutoModelForSeq2SeqLM,AutoTokenizer>>>tokenizer=AutoTokenizer.from_pretrained("facebook/nllb-200-distilled-600M",use_auth_token=True,src_lang="ron_Latn")>>>model=AutoModelForSeq2SeqLM.from_pretrained("facebook/nllb-200-distilled-600M",use_auth_token=True)>>>article="Şeful ONU spune că nu există o soluţie militară în Siria">>>inputs=tokenizer(article,return_tensors="pt")>>>translated_tokens=model.generate(...**inputs,forced_bos_token_id=tokenizer.lang_code_to_id["deu_Latn"],max_length=30... )>>>tokenizer.batch_decode(translated_tokens,skip_special_tokens=True)[0]

Result:

UN-Chef sagt, es gibt keine militärische Lösung in Syrien

Follow installation instructions inINSTALL guide for running training/generation. For general instructions aboutfairseq and working with the codebase refer tofairseq README. Forstopes andLASER follow their README files for installation.

NLLB project uses data from three sources : public bitext, mined bitext and data generated using backtranslation. Details of different datasets used and open source links are provided in detailshere.

We provide a download script forpublic bitext data, and links to downloadNLLB-Seed data. For more details checkhere.

LASER3 teacher-student training code is open sourcedhere. LASER3 encoders and mined bitext metadata are open sourced inLASER repository.Global mining pipeline and monolingual data filtering pipelines are released and available in ourstopes repository.

Follow the instructionshere to generate backtranslated data from a pretrained model.

We open source our dataset preparation pipeline for filtering/encoding/binarizing large scale datasets instopes. Encoding the datasets are done using the newSPM-200 model which was trained on 200+ languages used in the NLLB project. For more details seelink.

We open source all our model training and generation code in this repo. We also share code for finetuning our models on different domains like NLLB-MD. Additionally, we also share the code for online distillation that produced our 1.3B and 600M distilled models. For more details check theModeling section Readme.

NLLB project includes release of evaluation datasets like Flores-200, NLLB-MD and Toxicity-200. For instructions to run evaluation see instructionshere and for instructions to produce generations from the models follow instructionshere.

Flores200 |NLLB-MD |Toxicity-200

(Added Jan - 2023) We open-source additional guidelines and training materials for conducting the human evaluation protocol we utilized (XSTS), as well the calibration utilized and the entire human translation evaluation set for NLLB-200 and it's published baselinehere.

If you use NLLB in your work or any models/datasets/artifacts published in NLLB, please cite :

@article{nllb2022,title={No Language Left Behind: Scaling Human-Centered Machine Translation},author={{NLLB Team} and Costa-jussà, Marta R. and Cross, James and Çelebi, Onur and Elbayad, Maha and Heafield, Kenneth and Heffernan, Kevin and Kalbassi, Elahe and Lam, Janice and Licht, Daniel and Maillard, Jean and Sun, Anna and Wang, Skyler and Wenzek, Guillaume and Youngblood, Al and Akula, Bapi and Barrault, Loic and Mejia-Gonzalez, Gabriel and Hansanti, Prangthip and Hoffman, John and Jarrett, Semarley and Sadagopan, Kaushik Ram and Rowe, Dirk and Spruit, Shannon and Tran, Chau and Andrews, Pierre and Ayan, Necip Fazil and Bhosale, Shruti and Edunov, Sergey and Fan, Angela and Gao, Cynthia and Goswami, Vedanuj and Guzmán, Francisco and Koehn, Philipp and Mourachko, Alexandre and Ropers, Christophe and Saleem, Safiyyah and Schwenk, Holger and Wang, Jeff},year={2022}}

NLLB code and fairseq(-py) is MIT-licensed available inLICENSE file.