Repository files navigation

LUKE (LanguageUnderstanding withKnowledge-basedEmbeddings) is a new pretrained contextualized representation of words andentities based on transformer. It was proposed in our paperLUKE: Deep Contextualized Entity Representations with Entity-aware Self-attention.It achieves state-of-the-art results on important NLP benchmarks includingSQuAD v1.1 (extractivequestion answering),CoNLL-2003 (named entityrecognition),ReCoRD(cloze-style question answering),TACRED (relationclassification), andOpen Entity(entity typing).

This repository contains the source code to pretrain the model and fine-tune itto solve downstream tasks.

November 9, 2022: The large version of LUKE-Japanese is available

The large version of LUKE-Japanese is available on the Hugging Face Model Hub:

• luke-japanese-large
• luke-japanese-large-lite

This model achieves state-of-the-art results on three datasets inJGLUE.

October 27, 2022: The Japanese version of LUKE is available

The Japanese version of LUKE is now available on the Hugging Face Model Hub:

• luke-japanese-base
• luke-japanese-base-lite

This model outperforms other base-sized models on four datasets inJGLUE.

April 13, 2022: The mLUKE fine-tuning code is available

The example code is updated. Now it is based onallennlp andtransformers. You can reproducethe experiments in theLUKE andmLUKE papers with this implementation. Forthe details, please seeREADME.md under each example directory. The older codeused inthe LUKE paper has been moved toexamples/legacy.

April 13, 2022: The detailed instructions for pretraining LUKE models areavailable

For those interested in pretraining LUKE models, we explain how to preparedatasets and run the pretraining code onpretraining.md.

November 24, 2021: Entity disambiguation example is available

The example code of entity disambiguation based on LUKE has been added to thisrepository. This model was originally proposed inour paper, and achieved state-of-the-artresults on five standard entity disambiguation datasets: AIDA-CoNLL, MSNBC,AQUAINT, ACE2004, and WNED-WIKI.

For further details, please refer toexamples/entity_disambiguation.

August 3, 2021: New example code based on Hugging Face Transformers andAllenNLP is available

New fine-tuning examples of three downstream tasks, i.e.,NER,relationclassification, andentity typing, have been added to LUKE. These examplesare developed based on Hugging Face Transformers and AllenNLP. The fine-tuningmodels are defined using simple AllenNLP's Jsonnet config files!

The example code is available inexamples.

May 5, 2021: LUKE is added to Hugging Face Transformers

LUKE has been added to themaster branch of the Hugging Face Transformers library.You can now solve entity-related tasks (e.g., named entity recognition, relationclassification, entity typing) easily using this library.

For example, the LUKE-large model fine-tuned on the TACRED dataset can be usedas follows:

fromtransformersimportLukeTokenizer,LukeForEntityPairClassificationmodel=LukeForEntityPairClassification.from_pretrained("studio-ousia/luke-large-finetuned-tacred")tokenizer=LukeTokenizer.from_pretrained("studio-ousia/luke-large-finetuned-tacred")text="Beyoncé lives in Los Angeles."entity_spans= [(0,7), (17,28)]# character-based entity spans corresponding to "Beyoncé" and "Los Angeles"inputs=tokenizer(text,entity_spans=entity_spans,return_tensors="pt")outputs=model(**inputs)logits=outputs.logitspredicted_class_idx=int(logits[0].argmax())print("Predicted class:",model.config.id2label[predicted_class_idx])# Predicted class: per:cities_of_residence

We also provide the following three Colab notebooks that show how to reproduceour experimental results on CoNLL-2003, TACRED, and Open Entity datasets usingthe library:

• Reproducing experimental results of LUKE on CoNLL-2003 Using Hugging Face Transformers
• Reproducing experimental results of LUKE on TACRED Using Hugging Face Transformers
• Reproducing experimental results of LUKE on Open Entity Using Hugging Face Transformers

Please refer to theofficial documentationfor further details.

November 5, 2021: LUKE-500K (base) model

We released LUKE-500K (base), a new pretrained LUKE model which is smaller thanexisting LUKE-500K (large). The experimental results of the LUKE-500K (base) andLUKE-500K (large) on SQuAD v1 and CoNLL-2003 are shown as follows:

We tuned only the batch size and learning rate in the experiments based onLUKE-500K (base).

LUKE outperforms the previous state-of-the-art methods on five important NLPtasks:

These numbers are reported inour EMNLP 2020 paper.

LUKE can be installed usingPoetry:

poetry install# If you want to run pretraining for LUKEpoetry install --extras"pretraining opennlp"# If you want to run pretraining for mLUKEpoetry install --extras"pretraining icu"

The virtual environment automatically created by Poetry can be activated bypoetry shell.

A note on installingtorch

The pytorch installed viapoetry install does not necessarily match yourhardware. In such case, seethe official site andreinstall the correct version with thepip command.

poetry run pip3 uninstall torch torchvision torchaudio# Example for Linux with CUDA 11.3poetry run pip3 install torch torchvision torchaudio --extra-index-url https://download.pytorch.org/whl/cu113

Our pretrained models can be used with thetransformers library. The modeldocumentations can be found in the following links:LUKE andmLUKE.

Currently, the following models are available onthe Hugging Face Model Hub.

The entity embeddings cause a large memory footprint as they contain all theWikipedia entities that we used in pretraining. However, in some downstreamtasks (e.g., entity typing, named entity recognition, and relationclassification), we only need special entity embeddings such as[MASK]. Also,you may want to only use the word representations.

With such use-cases in mind, to make our models easier to use, we have uploadedlite models only with special entity embeddings. These models perform exactlythe same as the full models but have much fewer parameters, which enablefine-tuning the model with small GPUs.

We release the fine-tuning code based onallennlp andtransformers underexamples. You can run fine-tuning experiments very easily withpre-defined config files and theallennlp train command. For the details andexample commands for each task, please see the task directory underexamples.

The detailed instructions for pretraining luke models can be found onpretraining.md.

If you use LUKE in your work, please cite theoriginal paper.

@inproceedings{yamada-etal-2020-luke,    title = "{LUKE}: Deep Contextualized Entity Representations with Entity-aware Self-attention",    author = "Yamada, Ikuya  and      Asai, Akari  and      Shindo, Hiroyuki  and      Takeda, Hideaki  and      Matsumoto, Yuji",    booktitle = "Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)",    year = "2020",    publisher = "Association for Computational Linguistics",    url = "https://aclanthology.org/2020.emnlp-main.523",    doi = "10.18653/v1/2020.emnlp-main.523",}

For mLUKE, please citethis paper.

@inproceedings{ri-etal-2022-mluke,    title = "m{LUKE}: {T}he Power of Entity Representations in Multilingual Pretrained Language Models",    author = "Ri, Ryokan  and      Yamada, Ikuya  and      Tsuruoka, Yoshimasa",    booktitle = "Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)",    year = "2022",    publisher = "Association for Computational Linguistics",    url = "https://aclanthology.org/2022.acl-long.505",}