Repository files navigation

This repository documents the XLM-V Integration into 🤗 Transformers.

Basic steps were also documented in thisissue.

Please openan issue or PR for bugs/comments - it is highly appreciated!!

• 08.05.2023: XLM-V model is available underMeta AI organization and it was also added to🤗 TransformersDocumentation.
• 06.05.2023: Mentionfairseq PR for XLM-V and add results on XQuAD.
• 05.02.2023: Initial version of this repo.

XLM-V is multilingual language model with a one million token vocabulary trained on 2.5TB of data from Common Crawl (same as XLM-R).It was introduced in theXLM-V: Overcoming the Vocabulary Bottleneck in Multilingual Masked Language Modelspaper by Davis Liang, Hila Gonen, Yuning Mao, Rui Hou, Naman Goyal, Marjan Ghazvininejad, Luke Zettlemoyer and Madian Khabsa.

From the abstract of the XLM-V paper:

Large multilingual language models typically rely on a single vocabulary shared across 100+ languages.As these models have increased in parameter count and depth, vocabulary size has remained largely unchanged.This vocabulary bottleneck limits the representational capabilities of multilingual models like XLM-R.In this paper, we introduce a new approach for scaling to very large multilingual vocabularies byde-emphasizing token sharing between languages with little lexical overlap and assigning vocabulary capacityto achieve sufficient coverage for each individual language. Tokenizations using our vocabulary are typicallymore semantically meaningful and shorter compared to XLM-R. Leveraging this improved vocabulary, we train XLM-V,a multilingual language model with a one million token vocabulary. XLM-V outperforms XLM-R on every task wetested on ranging from natural language inference (XNLI), question answering (MLQA, XQuAD, TyDiQA), andnamed entity recognition (WikiAnn) to low-resource tasks (Americas NLI, MasakhaNER).

At the moment, XLM-V is not officially integrated intofairseq library, but the model itself can be loaded with it. But here's an openmerge requests that adds model and usage readme intofairseq.

The first author of the XLM-V paper, Davis Liang,tweetedabout the model weights, so they can be downloaded via:

$ wget https://dl.fbaipublicfiles.com/fairseq/xlmv/xlmv.base.tar.gz

The scriptconvert_xlm_v_original_pytorch_checkpoint_to_pytorch.py is needed to load these weights and converts them intoa 🤗 Transformers PyTorch model. It also checks, if everything went right during weight conversion:

torch.Size([1, 11, 901629]) torch.Size([1, 11, 901629])max_absolute_diff = 7.62939453125e-06Do both models output the same tensors? 🔥Saving model to /media/stefan/89914e9b-0644-4f79-8e65-a8c5245df168/xlmv/exported-workingConfiguration savedin /media/stefan/89914e9b-0644-4f79-8e65-a8c5245df168/xlmv/exported-working/config.jsonModel weights savedin /media/stefan/89914e9b-0644-4f79-8e65-a8c5245df168/xlmv/exported-working/pytorch_model.bin

Notice: On my laptop, 16GB of CPU RAM were not enough to convert the model weights. So I had to convert it on my server...

Another crucial part of integrating a model into 🤗 Transformers is on the Tokenizer side. The tokenizer in 🤗 Transformersshould output the same ids/subtokens as thefairseq tokenizer.

For this reason, thexlm_v_tokenizer_comparison.py script loads all 176 languages from theWikiANN dataset,tokenizes each sentence and compares it.

Unfortunately, some sentences have a slightly different output compared to thefairseq tokenizer, but this happens not quite often.The output of thexlm_v_tokenizer_comparison.py script with all tokenizer differences can be viewedhere.

After the model conversion and tokenizer checks, it is time to check the MLM performance:

fromtransformersimportpipelineunmasker=pipeline('fill-mask',model='stefan-it/xlm-v-base')unmasker("Paris is the <mask> of France.")

It outputs:

[{'score':0.9286897778511047,'token':133852,'token_str':'capital','sequence':'Paris is the capital of France.'}, {'score':0.018073994666337967,'token':46562,'token_str':'Capital','sequence':'Paris is the Capital of France.'}, {'score':0.013238662853837013,'token':8696,'token_str':'centre','sequence':'Paris is the centre of France.'}, {'score':0.010450296103954315,'token':550136,'token_str':'heart','sequence':'Paris is the heart of France.'}, {'score':0.005028395913541317,'token':60041,'token_str':'center','sequence':'Paris is the center of France.'}]

Results for masked LM are pretty good!

The last part of integrating a model into 🤗 Transformers is to test the performance on downstream tasks and compare theirperformance with the paper results. Both QA and NER downstream tasks are covered here.

A recentmaster version of Transformers (commit:59d5ede) is used to reproduce the XQuAD results using the PyTorchquestion answering exampleon a single A100 (40GB) GPU.

First, 5 models (with different seed!) are fine-tuned on English SQuAD dataset.

Fine-tuning for first model (XLM-R):

python3 run_qa.py \--model_name_or_path xlm-roberta-base \--dataset_name squad \--do_train \--do_eval \--max_seq_length 512 \--doc_stride 128 \--per_device_train_batch_size 6 \--learning_rate 3e-5 \--weight_decay 0.0 \--warmup_steps 0 \--num_train_epochs 2 \--seed 1 \--output_dir xlm-r-1 \--fp16 \--save_steps 14646

For XLM-V is looks similar:

 python3 run_qa.py \ --model_name_or_path stefan-it/xlm-v-base \ --dataset_name squad \ --do_train \ --do_eval \ --max_seq_length 512 \ --doc_stride 128 \ --per_device_train_batch_size 6 \ --learning_rate 3e-5 \ --weight_decay 0.0 \ --warmup_steps 0 \ --num_train_epochs 2 \ --seed 1 \ --output_dir xlm-v-1 \ --fp16 \ --save_steps 14618

Then this fine-tuned model can be zero-shot evaluated on the 11 languages in XQuAD. Here's an example for Hindi (shortened):

python3 run_qa.py --model_name_or_path xlm-r-1 \--dataset_name xquad \--dataset_config_name xquad.hi \--do_eval \--max_seq_length 512 \--doc_stride 128 \--output_dir xlm-r-1-hi \--fp16

This is done for each fine-tuned model on each language. Detailed results for all 5 different models can be seen here:

• XLM-R (Base) Results (Development and Test result)
• XLM-V (Base) Results (Development and Test result)

Here's the overall performance table (inspired by Table 9 in the XLM-V paper with their results):

Summary: The F1-Score results for XLM-V could be reproduced (56.3 vs. 56.4). For exact match there are slightly differentresults (71.9 vs. 71.3). For the XLM-R model there's a larger difference: our XLM-R models perform better on XQuAD comparedto their XLM-R reimplementation. Our XLM-R model also achieves better results than XLM-V on XQuAD.

For NER, theflair-fine-tuner.py fine-tunes a model on the English WikiANN (Rahimi et al.) split with the hyper-parameters,mentioned in the paper (only difference is that we use 512 as sequence length compared to 128!). We fine-tune 5 models withdifferent seeds and average performance over these 5 different models. The scripts expects a model configuration as first input argument.All configuration files are located under the./configs folder. Fine-tuning XLM-V can be started with:

$ python3 flair-fine-tuner.py ./configs/xlm_v_base.json

Fine-tuning is done on a A100 (40GB) instances fromLambda Cloud using Flair.A 40GB is definitely necessary to fine-tune this model with that given batch size! Latest Flair master (commit23618cd) is also needed.

The scriptmasakhaner-zero-shot.py performs zero-shot evaluation on the MasakhaNER v1 datset, that is used in the XLM-V paper.One crucial part is to deal withDATE entities: they do not exist in the English WikiANN (Rahimi et al.) split, but they areannotated in MasakhaNER v1. For this reason, we convert allDATE entities intoO to disable them for evaluation. The scriptmasakhaner-zero-shot.py is used for performing zero-shot evaluation and will output a nice results table.

Detailed results for all 5 different models can be seen here:

• XLM-R (Base) Results (Development and Test result)
• XLM-V (Base) Results (Development and Test result)

Here's the overall performance table (inspired by Table 11 in the XLM-V paper with their results):

Diff. between XLM-V and XLM-R in the paper: (32.1 - 20.9) = 11.2%.

Diff. between reproduced XLM-V and XLM-R: (39.7 - 22.9) = 16.8%.

The scriptwikiann-zero-shot.py performs zero-shot evaluation on the WikiANN (Rahimi et al.) dataset. Ths scriptwikiann-zero-shot.pyis used for zero-shot evaluation and will also output a nice results table. Notice: it uses a high batch size for evaluating the model,so a A100 (40GB) GPU is definitely useful.

Detailed results for all 5 different models can be seen here:

• XLM-R (Base) Results (Development and Test result)
• XLM-V (Base) Results (Development and Test result)

Here's the overall performance table (inspired by Table 10 in the XLM-V paper with their results):

Diff. between XLM-V and XLM-R in the paper: (64.7 - 61.3) = 3.4%.

Diff. between reproduced XLM-V and XLM-R: (65.0 - 62.6) = 2.4%.

After all checks (weights, tokenizer and downstream tasks) the model was uploaded to the 🤗 Transformers Model Hub:

• facebook/xlm-v-base

XLM-V was also added to the 🤗 Transformers Documentation withthis PRand now lives athere.