Repository files navigation

A Japanese DistilBERT pretrained model, which was trained onWikipedia.

Findhere for a quickstart guidance in Japanese.

1. Introduction
2. Requirements
3. Usage
4. License

DistilBERT is a small, fast, cheap and light Transformer model based on Bert architecture. It has 40% less parameters than BERT-base, runs 60% faster while preserving 97% of BERT's performance as measured on the GLUE language understanding benchmark.

This model was trained with the official Hugging Face implementation fromhere for 2 weeks on AWS p3dn.24xlarge instance.

More details about distillation can be found in following paper."DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter" by Sanh et al. (2019).

The teacher model isthe pretrained Japanese BERT models from TOHOKU NLP LAB.

Currently only PyTorch compatible weights are available. Tensorflow checkpoints can be generated by following theofficial guide.

torch>=1.3.1torchvision>=0.4.2transformers>=2.5.0tensorboard>=1.14.0tensorboardX==1.8scikit-learn>=0.21.0mecab-python3

Please download and unzipDistilBERT-base-jp.zip.

# Read from local pathfromtransformersimportAutoModel,AutoTokenizertokenizer=AutoTokenizer.from_pretrained("bert-base-japanese-whole-word-masking")model=AutoModel.from_pretrained("LOCAL_PATH")

LOCAL_PATH means the path which above file is unzipped. 3 files should be included:

• pytorch_model.bin
• config.json
• vocal.txt

or

# Download from model library from huggingface.cofromtransformersimportAutoModel,AutoTokenizertokenizer=AutoTokenizer.from_pretrained("bert-base-japanese-whole-word-masking")model=AutoModel.from_pretrained("bandainamco-mirai/distilbert-base-japanese")

Copyright (c) 2020 BANDAI NAMCO Research Inc.

Released under the MIT license

https://opensource.org/licenses/mit-license.php