実行環境の準備

pip install -U pip wheel setuptools poetry

poetry installpoetry run poe force-cuda11# CUDA 11 を使いたい場合

データセットの準備

git clone https://github.com/ku-nlp/KWDLC.git datasets/kwdlc/repoallennlp make-dataset kwdlc --output-dir datasets/kwdlc --source datasets/kwdlc/repo/

git clone https://github.com/stockmarkteam/ner-wikipedia-dataset.git datasets/stockmark-ner-wiki/repoallennlp make-dataset stockmark_ner_wiki --output-dir datasets/stockmark-ner-wiki --source datasets/stockmark-ner-wiki/repo/ner.json

モデルの学習

CUDA_VISIBLE_DEVICES=0 GPU=0 allennlp train configs/kwdlc/bert.jsonnet -s outputs/kwdlc/bert

CUDA_VISIBLE_DEVICES=0 GPU=0 allennlp train configs/stockmark-ner-wiki/bert.jsonnet -s outputs/stockmark-ner-wiki/bert

モデルの予測

CUDA_VISIBLE_DEVICES=0 allennlp predict \  outputs/kwdlc/bert/model.tar.gz \  datasets/kwdlc/tst_ner.txt \  --output-file outputs/kwdlc/bert/tst_ner.jsonl \  --cuda-device 0 --predictor sentence_tagger --use-dataset-reader

CUDA_VISIBLE_DEVICES=0 allennlp predict \  outputs/stockmark-ner-wiki/bert/model.tar.gz \  datasets/stockmark-ner-wiki/tst_ner.txt \  --output-file outputs/kwdlc/stockmark-ner-wiki/tst_ner.jsonl \  --cuda-device 0 --predictor sentence_tagger --use-dataset-reader

Acknowledgements