Repository files navigation

English Version |中文版说明

This repository contains a PyTorch implementation of the albert model from the paper

A Lite Bert For Self-Supervised Learning Language Representations

by Zhenzhong Lan. Mingda Chen....

• pytorch=1.10
• cuda=9.0
• cudnn=7.5
• scikit-learn
• sentencepiece

Official download links:google albert

Adapt to this version，download pytorch model (google drive):

v1

• albert_base_v1.zip
• albert_large_v1.zip
• albert_xlarge_v1.zip
• albert_xxlarge_v1.zip

v2

• albert_base_v2.zip
• albert_large_v2.zip
• albert_xlarge_v2.zip
• albert_xxlarge_v2.zip

１. Placeconfig.json and30k-clean.model into theprev_trained_model/albert_base_v2 directory.example:

├── prev_trained_model|  └── albert_base_v2|  |  └── pytorch_model.bin|  |  └── config.json|  |  └── 30k-clean.model

2．convert albert tf checkpoint to pytorch

pythonconvert_albert_tf_checkpoint_to_pytorch.py \--tf_checkpoint_path=./prev_trained_model/albert_base_tf_v2 \--bert_config_file=./prev_trained_model/albert_base_v2/config.json \--pytorch_dump_path=./prev_trained_model/albert_base_v2/pytorch_model.bin

TheGeneral Language Understanding Evaluation (GLUE) benchmark is a collection of nine sentence- or sentence-pair language understanding tasks for evaluating and analyzing natural language understanding systems.

Before running anyone of these GLUE tasks you should download theGLUE data by runningthis script and unpack it to some directory $DATA_DIR.

3．runsh scripts/run_classifier_sst2.shto fine tuning albert model

Performance of ALBERT on GLUE benchmark results using a single-model setup ondev: