Repository files navigation

GLM is a General Language Model pretrained with an autoregressive blank-filling objective and can be finetuned onvarious natural language understanding and generation tasks.

Please refer to our paper for a detailed description of GLM:

GLM: General Language Model Pretraining with Autoregressive Blank Infilling (ACL 2022)

Zhengxiao Du*, Yujie Qian*, Xiao Liu, Ming Ding, Jiezhong Qiu, Zhilin Yang, Jie Tang (*: equal contribution)

News: We releaseChatGLM-6B, an open pre-trained language model with 6 billion parameters optimized for Chinese QA and dialogue based on the GLM framework.

You can download the pretrained models used in the paperfromOneDriveorTsinghua-Cloud.

Unzip the downloaded file into a local folder and setCHECKPOINT_PATH in the corresponding scripts to the folder path.

dev set, single model, single-task finetuning

CNN/Daily Mail (test set, no additional data used)

XSum (test set, no additional data used)

test set, zero-shot

You can access GLM models via HuggingFace Hub. Pleaseinstalltransformers>=4.23.1 and find all the available modelshere.

fromtransformersimportAutoTokenizer,AutoModelForSeq2SeqLMtokenizer=AutoTokenizer.from_pretrained("THUDM/glm-10b",trust_remote_code=True)model=AutoModelForSeq2SeqLM.from_pretrained("THUDM/glm-10b",trust_remote_code=True)model=model.half().cuda()model.eval()# Inferenceinputs=tokenizer("Ng is an adjunct professor at [MASK] (formerly associate professor and Director of its Stanford AI Lab or SAIL ). Also a pioneer in online education, Ng co-founded Coursera and deeplearning.ai.",return_tensors="pt")inputs=tokenizer.build_inputs_for_generation(inputs,max_gen_length=512)inputs=inputs.to('cuda')outputs=model.generate(**inputs,max_length=512,eos_token_id=tokenizer.eop_token_id)print(tokenizer.decode(outputs[0].tolist()))# Traininginputs=tokenizer(    ["Tsinghua University is located in [MASK].","One minus one equals zero, is it correct? Answer: [MASK]"],return_tensors="pt",padding=True)inputs=tokenizer.build_inputs_for_generation(inputs,targets=["Beijing","No"],max_gen_length=8,padding=False)inputs=inputs.to('cuda')outputs=model(**inputs)loss=outputs.losslogits=outputs.logits

fromtransformersimportAutoTokenizer,AutoModelForMultipleChoicetokenizer=AutoTokenizer.from_pretrained("THUDM/glm-10b",trust_remote_code=True)model=AutoModelForMultipleChoice.from_pretrained("THUDM/glm-10b",trust_remote_code=True)model=model.half().cuda()model.eval()inputs=tokenizer(["Tsinghua University is located in [MASK].","One minus one equals zero, is it correct? Answer: [MASK]"],return_tensors="pt",padding=True)choices= [["Beijing","Shanghai"], ["Yes","No"]]inputs=tokenizer.build_inputs_for_multiple_choice(inputs,choices)inputs=inputs.to('cuda')outputs=model(**inputs)logits=outputs.logits

You can also convert the finetuned checkpoints withscripts/convert_glm_checkpoint_to_transformers.py.

We prepare two docker images based on CUDA 10.2 and CUDA 11.2. You can pull the pre-built images from Docker Hub and runwith docker v19.03+

docker run --gpus all --rm -it --ipc=host zxdu20/glm-cuda102

or replaceglm-cuda102 withglm-cuda112.

You can also modify the image according to your requirements indocker/cuda102.dockerfileand build the image yourself

  docker build -f cuda102.dockerfile. -t glm-cuda102

Please first install PyTorch (we use 1.7.0) andapex, and then install otherdependencies bypip install -r requirements.txt

git clone https://github.com/THUDM/GLMcd GLM

If your encounter theCUDA out of memory error, which means you GPU memory is limited, you can try the modelparallelism to divide the parameters into multiple GPUs. Take the two-way model parallelism as an example. Firstrunchange_mp.py to divide the checkpoint:

python change_mp.py path_to_the_checkpoint 2

Then update the checkpoint path in the model config file (suchasconfig_tasks/model_blocklm_10B.sh) and changeMP_SIZE in the script (suchasscripts/ds_finetune_superglue.sh) to2.

We provide scripts for finetuning GLM on some downstream tasks.

• ChangeCHECKPOINT_PATH to your local path. Run the following script

bash scripts/generate_block.sh \     config_tasks/model_blocklm_10B_chinese.sh

Some models (GLM-2B, GLM-10B, and GLM-10B-Chinese) use three different mask tokens:[MASK] for short blankfilling,[sMASK] for sentence filling, and[gMASK] for left-to-right generation.

You can also add multiple[MASK] and[sMASK] in a single example. The model will fill the blanks one by one from left to right. The answer to each blank always begins with a special<|startofpiece|>.

• Download theSuperGlue data and check the experiment setup inscripts/ds_finetune_superglue.sh. NotethatDATA_ROOT, CHECKPOINT_PATH, SAVE_PATHneed to be changed to your local path. You may also change thebatch-size andnproc_per_node according to youravailable hardware.

• Run the following script (use the COPA dataset as an example)

bash scripts/ds_finetune_superglue.sh \     config_tasks/model_blocklm_10B.sh \     config_tasks/task_copa.sh

• We also implementP-Tuning in our code. Run the following script to integratep-tuning:

bash scripts/ds_finetune_superglue_prompt.sh \     config_tasks/model_blocklm_10B.sh \     config_tasks/task_copa.sh

• To apply GLM to a new NLU dataset with cloze-filling finetuning, implement aDataProcessor intasks/superglue/dataset.py for data loading and add aPVP intasks/superglue/pvp.py for the cloze question. More details can be foundhere.

• Download theGigaword,CNN/Daily MailorXSum dataset and check the experiment setup inscripts/ds_finetune_seq2seq.sh. ChangeDATA_ROOT, CHECKPOINT_PATH, SAVE_PATH toyourlocal path.

• Run the following script (use the CNN/Daily Mail dataset as an example)

  bash scripts/ds_finetune_seq2seq.sh \    config_tasks/model_blocklm_10B.sh \    config_tasks/seq_cnndm_org.sh

• The summaries are written into./runs/experiment_name/test.jsonl.hyps. The references are writtenintotest.jsonl.refs in the same directory. For calculating rouge,installfile2rouge and download Stanford CoreNLPfromhere. Run the following script

  bash scripts/evaluate_seq2seq.sh \ ./runs/experiment_name/test.jsonl.hyps ./runs/experiment_name/test.jsonl.refs

Process your seq2seq data into{split}.source and{split}.target, with each line being the context or the target ofa sample, andsplit beingtrain,val, andtest.

Run the following script

bash scripts/ds_finetune_seq2seq.sh\   config_tasks/model_blocklm_10B.sh\   config_tasks/seq_customization.sh

You can specify the hyperparameters inconfig_tasks/seq_customization.shandconfig_tasks/config_blocklm_10B_cnndm.json

bash scripts/evaluate_multichoice.sh config_tasks/model_blocklm_10B.sh

Note thatCHECKPOINT_PATH andDATA_PATH need to be changed to your local path.

The format of each line of the data file should be

{"inputs_pretokenized": "Context and question here", "choices_pretokenized": ["Choice 1", "Choice 2", "Choice 3"], "label": int}

• Download theLAMBADA data and changeDATA_ROOT, CHECKPOINT_PATH inscripts/evaluate_lm.sh
• Run the following script

bash scripts/evaluate_lm.sh\     config_tasks/model_blocklm_large_generation.sh \     config_tasks/zero_lambada.sh

• Downloadourtest set of wikibookorWikitext103 dataset andchangeDATA_ROOT, CHECKPOINT_PATHinscripts/evaluate_lm.sh
• Run the following script

  bash scripts/evaluate_lm.sh\   config_tasks/model_blocklm_large_generation.sh \   config_tasks/zero_wikitext.sh

• Download theYahoo dataset and check the experiment setup inscripts/finetune_blank.sh. ChangeDATA_ROOT, CHECKPOINT_PATH, SAVE_PATH to yourlocal path.

• Run the following script

bash scripts/finetune_blank.sh \      config_tasks/model_blocklm_large.sh \      config_tasks/seq_blank.sh

Run the following script to pre-train the GLM-Large model

bash scripts/ds_pretrain_nvidia.sh config/ds_block_large.sh

The scriptscripts/ds_pretrain_nvidia.sh launches the training program with DeepSpeed.You should changeNUM_WORKERS andNUM_GPUS_PER_WORKER to the number of workers and the number of gpus per worker.Also changeHOST_FILE_PATH to the path to an OpenMPI-style hostfile. More details about DeepSpeed launcher can befoundhere.

The fileconfig/ds_block_large.sh defines the hyperparameters for pretraining. Most of thearguments are fairly self-explanatory. Specifically,--train-data can be multiple keywords defined inNAMED_CORPORAindata_utils/corpora.py. The hyperparameters of the optimizer are defined in the correspondingjson file underconfig. The semantics of the json file can be foundhere.

Part of the code is based onMegatron-LMandPET.

Please cite our paper if you find this code useful for your research:

@article{DBLP:conf/acl/DuQLDQY022,  author    = {Zhengxiao Du and               Yujie Qian and               Xiao Liu and               Ming Ding and               Jiezhong Qiu and               Zhilin Yang and               Jie Tang},  title     = {{GLM:} General Language Model Pretraining with Autoregressive Blank Infilling},  booktitle = {Proceedings of the 60th Annual Meeting of the Association for Computational               Linguistics (Volume 1: Long Papers), {ACL} 2022, Dublin, Ireland,               May 22-27, 2022},  pages     = {320--335},  publisher = {Association for Computational Linguistics},  year      = {2022},}