Index

環境構築手順

For Ubuntu

前提条件 / Prerequisites

gcc-12 installation steps

sudo apt updatesudo apt upgradesudo apt install make build-essential libssl-dev zlib1g-dev libbz2-dev libreadline-dev libsqlite3-dev wget curl llvm libncursesw5-dev xz-utils tk-dev libxml2-dev libxmlsec1-dev libffi-dev liblzma-dev gitsudo apt install gcc-12 g++-12sudo ln -s -f /usr/bin/gcc-12 /usr/bin/gccsudo ln -s -f /usr/bin/g++-12 /usr/bin/g++

nvidia-driver-535 installation steps

nvidia-smi

sudo apt-get --purge remove nvidia-*sudo apt-get --purge remove cuda-*

sudo add-apt-repository ppa:graphics-drivers/ppasudo apt updatesudo apt install nvidia-driver-535sudo reboot

nvidia-smi

sudo nano /etc/apt/apt.conf.d/20auto-upgrades

APT::Periodic::Update-Package-Lists "1";APT::Periodic::Unattended-Upgrade "0";

CUDA 12.1 installation steps

wget https://developer.download.nvidia.com/compute/cuda/12.1.1/local_installers/cuda_12.1.1_530.30.02_linux.runsudo sh cuda_12.1.1_530.30.02_linux.run

│ Existing package manager installation of the driver found. It is strongly    ││ recommended that you remove this before continuing.                          ││ Abort                                                                        ││ Continue                                                                     │

Do you accept the above EULA? (accept/decline/quit):accept

│ CUDA Installer                                                               ││ - [ ] Driver                                                                 ││      [ ] 530.30.02                                                           ││ + [X] CUDA Toolkit 12.1                                                      ││   [ ] CUDA Demo Suite 12.1                                                   ││   [ ] CUDA Documentation 12.1                                                ││ - [ ] Kernel Objects                                                         ││      [ ] nvidia-fs                                                           ││   Options                                                                    ││   Install                                                                    │

/usr/local/cuda/bin/nvcc -V

For WSL2

前提条件 / Prerequisites

Windows側でNVIDIA Driverをインストール / Install NVIDIA Driver on Windows side

WSL2でUbuntu 22.04をインストール/ Install Ubuntu 22.04 with WSL2

管理者権限でPowerShellを起動する / Start PowerShell with administrator privileges

WSL2の更新 / Update WSL2

wsl --set-default-version 2wsl --update

WSL2上でのUbuntu 22.04のインストール / Installing Ubuntu 22.04 on WSL2

wsl --install -d Ubuntu-22.04

nvidia-smi

Ubuntu側でのCUDAのインストール / Installing CUDA on Ubuntu side

Windowsターミナルのインストール / Installing Windows Terminal

For macOS

前提条件/ Prerequisites

Installing Command Line Tools

xcode-select --install

Installing Python3.10.11 and PATH setting

実験ソースコード

Software Installation

CUDAの動作確認 / Checking the operation of CUDA

/usr/local/cuda/bin/nvcc -V

環境変数LD_LIBRARY_PATHにCUDAのパスを追加 / Add CUDA path to environment variable LD_LIBRARY_PATH

echo'export LD_LIBRARY_PATH="$LD_LIBRARY_PATH:/usr/local/cuda-12.1/lib64"'>>~/.bashrcsource~/.bashrc

python3でvenvが使える状態かの確認 / Check if venv is usable in python3

python3 -Vpython3 -m venv venvsource venv/bin/activatedeactivaterm -r venv

pyenv環境の構築 / Building a pyenv environment

pyenv未導入の場合 / If pyenv is not installed

curl https://pyenv.run| bash

pyenv導入済みの場合 / If pyenv has been installed

cd~/.pyenv/plugins/python-build/../..&& git pull&&cd -

pyenvのパス追加 / Add pyenv to the PATH

echo'export PYENV_ROOT="$HOME/.pyenv"'>>~/.bashrcecho'export PATH="$PYENV_ROOT/bin:$PATH"'>>~/.bashrcecho'eval "$(pyenv init --path)"'>>~/.bashrcsource~/.bashrc

echo'export PYENV_ROOT="$HOME/.pyenv"'>>~/.bash_profileecho'export PATH="$PYENV_ROOT/bin:$PATH"'>>~/.bash_profileecho'eval "$(pyenv init --path)"'>>~/.bash_profilesource~/.bash_profile

pyenvでPython 3.10.13をインストール / Install Python 3.10.13 with pyenv

pyenv install 3.10.13

実験ディレクトリとvenv環境の作成・有効化・バージョン確認 / Creation, activation, and version confirmation of experiment directory and venv environment

mkdir my-llmcd my-llmpyenvlocal 3.10.13python -m venv venvsource venv/bin/activatewhich pythonpython -Vpip -V

PyTorchのインストールと動作確認 / Installing PyTorch and checking its operation

pip install torch

importtorchtorch.cuda.is_available()torch.cuda.device_count()torch.cuda.get_device_name()

importtorchtorch.backends.mps.is_available()

BERTでTransformersの動作確認 / Check the operation of Transformers with BERT

pip install transformers fugashi unidic-lite

fromtransformersimportAutoModelForMaskedLM,AutoTokenizer,pipelinemodel_name="cl-tohoku/bert-large-japanese-v2"tokenizer=AutoTokenizer.from_pretrained(model_name)model=AutoModelForMaskedLM.from_pretrained(model_name)model=model.to("cuda:0")mlm=pipeline("fill-mask",model=model,tokenizer=tokenizer,device="cuda:0")mlm("語りえぬものについては、[MASK]しなければならない。")[:2]

fromtransformersimportAutoModelForMaskedLM,AutoTokenizer,pipelinemodel_name="cl-tohoku/bert-large-japanese-v2"tokenizer=AutoTokenizer.from_pretrained(model_name)model=AutoModelForMaskedLM.from_pretrained(model_name)model=model.to("mps")mlm=pipeline("fill-mask",model=model,tokenizer=tokenizer,device="mps")mlm("語りえぬものについては、[MASK]しなければならない。")[:2]

Inference and Evaluation

text-generation

pip install accelerate safetensors bitsandbytes

1.3B

importtorchfromtransformersimportAutoModelForCausalLM,AutoTokenizer,pipelinemodel_name="llm-jp/llm-jp-1.3b-v1.0"tokenizer=AutoTokenizer.from_pretrained(model_name)model=AutoModelForCausalLM.from_pretrained(model_name,torch_dtype=torch.float32,device_map="auto")pipe=pipeline("text-generation",model=model,tokenizer=tokenizer,device_map="auto",pad_token_id=tokenizer.pad_token_id)print(pipe("語りえぬものについては、",max_length=128))

importtorchfromtransformersimportAutoModelForCausalLM,AutoTokenizer,pipelinemodel_name="llm-jp/llm-jp-1.3b-v1.0"tokenizer=AutoTokenizer.from_pretrained(model_name)model=AutoModelForCausalLM.from_pretrained(model_name,torch_dtype=torch.float16,device_map="auto")pipe=pipeline("text-generation",model=model,tokenizer=tokenizer,device_map="auto",pad_token_id=tokenizer.pad_token_id)print(pipe("語りえぬものについては、",max_length=128))

importtorchfromtransformersimportAutoModelForCausalLM,AutoTokenizer,pipelinemodel_name="llm-jp/llm-jp-1.3b-v1.0"tokenizer=AutoTokenizer.from_pretrained(model_name)model=AutoModelForCausalLM.from_pretrained(model_name,torch_dtype=torch.bfloat16,device_map="auto")pipe=pipeline("text-generation",model=model,tokenizer=tokenizer,device_map="auto",pad_token_id=tokenizer.pad_token_id)print(pipe("語りえぬものについては、",max_length=128))

13B

importtorchfromtransformersimportAutoModelForCausalLM,AutoTokenizer,pipelinemodel_name="llm-jp/llm-jp-13b-v1.0"tokenizer=AutoTokenizer.from_pretrained(model_name)model=AutoModelForCausalLM.from_pretrained(model_name,torch_dtype=torch.float16,device_map="auto")pipe=pipeline("text-generation",model=model,tokenizer=tokenizer,device_map="auto",pad_token_id=tokenizer.pad_token_id)print(pipe("語りえぬものについては、",max_length=128))

importtorchfromtransformersimportAutoModelForCausalLM,AutoTokenizer,pipeline,BitsAndBytesConfigmodel_name="llm-jp/llm-jp-13b-v1.0"tokenizer=AutoTokenizer.from_pretrained(model_name)quantization_config=BitsAndBytesConfig(load_in_4bit=True)model=AutoModelForCausalLM.from_pretrained(model_name,torch_dtype=torch.float16,device_map="auto",quantization_config=quantization_config)

llm-jp-eval

Installation

deactivate

git clone https://github.com/llm-jp/llm-jp-eval.gitcd llm-jp-evalcp configs/config_template.yaml configs/config.yamlpython -m venv venvsource venv/bin/activatepip install -e.wandb disabled

jasterのビルドとディレクトリ構成の確認 / Building jaster and checking the directory structure

python scripts/preprocess_dataset.py --dataset-name all --output-dir jaster/ls jaster/ls jaster/1.2.0/ls jaster/1.2.0/evaluation

dataset_dirの設定 / Setting dataset_dir

dataset_dir:"jaster/1.2.0/evaluation/dev"

精度評価 / Accuracy evaluation

JNLI devセット全件の評価 / Evaluation of all JNLI dev sets

python scripts/evaluate_llm.py torch_dtype=fp32 \  target_dataset="[jnli]" \  metainfo.max_num_samples=-1 \  wandb.run_name=llm-jp-1.3b-v1.0_fp32_dev-jnli \  model.pretrained_model_name_or_path=llm-jp/llm-jp-1.3b-v1.0 \  tokenizer.pretrained_model_name_or_path=llm-jp/llm-jp-1.3b-v1.0

python scripts/evaluate_llm.py torch_dtype=fp16 \  target_dataset="[jnli]" \  metainfo.max_num_samples=-1 \  wandb.run_name=llm-jp-1.3b-v1.0_fp16_dev-jnli \  model.pretrained_model_name_or_path=llm-jp/llm-jp-1.3b-v1.0 \  tokenizer.pretrained_model_name_or_path=llm-jp/llm-jp-1.3b-v1.0

python scripts/evaluate_llm.py torch_dtype=bf16 \  target_dataset="[jnli]" \  metainfo.max_num_samples=-1 \  wandb.run_name=llm-jp-1.3b-v1.0_bf16_dev-jnli \  model.pretrained_model_name_or_path=llm-jp/llm-jp-1.3b-v1.0 \  tokenizer.pretrained_model_name_or_path=llm-jp/llm-jp-1.3b-v1.0

jaster全データセット先頭100件の評価 / Evaluate the first 100 results for each of all datasets in jaster

python scripts/evaluate_llm.py torch_dtype=fp32 \  wandb.run_name=llm-jp-1.3b-v1.0_fp32_dev-all \  model.pretrained_model_name_or_path=llm-jp/llm-jp-1.3b-v1.0 \  tokenizer.pretrained_model_name_or_path=llm-jp/llm-jp-1.3b-v1.0

python scripts/evaluate_llm.py torch_dtype=fp16 \  wandb.run_name=llm-jp-1.3b-v1.0_fp16_dev-all \  model.pretrained_model_name_or_path=llm-jp/llm-jp-1.3b-v1.0 \  tokenizer.pretrained_model_name_or_path=llm-jp/llm-jp-1.3b-v1.0

python scripts/evaluate_llm.py torch_dtype=bf16 \  wandb.run_name=llm-jp-1.3b-v1.0_bf16_dev-all \  model.pretrained_model_name_or_path=llm-jp/llm-jp-1.3b-v1.0 \  tokenizer.pretrained_model_name_or_path=llm-jp/llm-jp-1.3b-v1.0

Supervised Fine-tuning

Installation

deactivatecd ..

git clone https://github.com/llm-jp/llm-jp-sft.gitcd llm-jp-sftpython -m venv venvsource venv/bin/activatepip install -r requirements.txtwandb disabled

jasterの参照

ln -s ../llm-jp-eval/jaster.

Ichikara-instruction公開データのプロンプト化 / Converting Ichikara-instruction public data to prompt format

importjsonimportrandomimportsysif__name__=="__main__":inst="以下は、タスクを説明する指示です。要求を適切に満たす応答を書きなさい。"records= []forfinsys.argv[1:]:withopen(f,"r",encoding="utf8")asfin:forrinjson.load(fin):records.append({"ID":r["ID"],"text":f'{inst}\n\n### 指示:\n{r["text"]}\n\n### 応答:\n{r["output"]}',                })random.shuffle(records)dev_len=len(records)//10dev,train=records[:dev_len],records[dev_len:]json.dump(train,sys.stdout,indent=1,ensure_ascii=False)json.dump(dev,sys.stderr,indent=1,ensure_ascii=False)

python convert_ichikara.py Distribution20231115/*.json \> jaster/1.2.0/tuning/train/ichikara.json \2> jaster/1.2.0/tuning/dev/ichikara.jsonhead -n 5 jaster/1.2.0/tuning/dev/ichikara.json

LoRA SFT BF16 - Ubuntu / WSL2

python train.py \  --num_train_epochs 1 \  --per_device_train_batch_size 4 \  --gradient_accumulation_steps 16 \  --learning_rate 2e-5 \  --warmup_ratio 0.1 \  --lr_scheduler_type cosine \  --bf16 \  --max_seq_length 2048 \  --gradient_checkpointing \  --data_files`ls jaster/1.2.0/tuning/train/*.json` \  --use_peft \  --model_name_or_path llm-jp/llm-jp-1.3b-v1.0 \  --output_dir results/llm-jp-1.3b-lora-jaster_ichikara-v1.0

LoRA SFT 4bit - Ubuntu / WSL2

python train.py \  --num_train_epochs 1 \  --per_device_train_batch_size 8 \  --gradient_accumulation_steps 8 \  --learning_rate 2e-5 \  --warmup_ratio 0.1 \  --lr_scheduler_type cosine \  --bf16 \  --load_in_4bit True \  --max_seq_length 2048 \  --gradient_checkpointing \  --data_files`ls jaster/1.2.0/tuning/train/*.json` \  --use_peft \  --model_name_or_path llm-jp/llm-jp-1.3b-v1.0 \  --output_dir results/llm-jp-1.3b-lora-jaster_ichikara-v1.0

Full Parameter SFT

compute_environment:LOCAL_MACHINEdebug:falsedistributed_type:DEEPSPEEDdowncast_bf16:'no'machine_rank:0main_training_function:mainmixed_precision:bf16num_machines:1num_processes:8rdzv_backend:staticsame_network:truetpu_env:[]tpu_use_cluster:falsetpu_use_sudo:falseuse_cpu:falsedeepspeed_config:zero_stage:3offload_optimizer_device:noneoffload_param_device:cpuzero3_init_flag:truezero3_save_16bit_model:true

accelerate launch --config_file configs/accelerate_config_zero3.yaml \  train.py \  --model_name_or_path llm-jp/llm-jp-13b-v1.0 \  --tokenizer_name_or_path llm-jp/llm-jp-13b-v1.0 \  --num_train_epochs 1 \  --per_device_train_batch_size 1 --gradient_accumulation_steps 32 \  --learning_rate 1e-4 --warmup_ratio 0.1 --lr_scheduler_type cosine \  --bf16 \  --max_seq_length 2048 \  --gradient_checkpointing \  --data_files`ls jaster/1.2.0/tuning/train/*.json` \  --output_dir results/llm-jp-13b-full-jaster_ichikara-v1.0

Direct Preference Optimization

Clone the repository

deactivatecd ..

git clone https://github.com/llm-jp/llm-jp-dpo.gitcd llm-jp-dpo

Ubuntu / WSL2

poetry installpoetry shellwandb disabled

compute_environment:LOCAL_MACHINEdebug:falsedistributed_type:'NO'downcast_bf16:'no'machine_rank:0main_training_function:mainmixed_precision:bf16num_machines:1num_processes:1rdzv_backend:staticsame_network:truetpu_env:[]tpu_use_cluster:falsetpu_use_sudo:falseuse_cpu:false

accelerate launch --config_file accelerate_configs/single_gpu.yaml train.py --model llm-jp/llm-jp-1.3b-v1.0 --per-device-train-batch-size 4 --per-device-eval-batch-size 8

macOS (パフォーマンスに難があるため改良中です / Performance needs improvement)

python -m venv venvsource venv/bin/activatepip install torch==2.2.0 transformers==4.37.2 trl==0.7.10 peft==0.8.2 datasets==2.16.1 accelerate==0.26.1 wandbwandb disabled

defmain():importtorch._dynamotorch._dynamo.config.suppress_errors=True

torch_dtype=torch.float16,# bfloat16,

# bf16=True,

python train.py --model llm-jp/llm-jp-1.3b-v1.0 --per-device-train-batch-size 4 --per-device-eval-batch-size 8

Pretraining

環境構築 / Environment construction

deactivatecd ..

echo'export LD_LIBRARY_PATH="$LD_LIBRARY_PATH:/usr/local/cuda-11.8/lib64"'>>~/.bashrcsource~/.bashrcgit clone https://github.com/microsoft/Megatron-DeepSpeedcd Megatron-DeepSpeedmkdir -p tmppython3 -m venv venvsource venv/bin/activatepip install torch==2.0.1 torchvision==0.15.2 --index-url https://download.pytorch.org/whl/cu118pip install"pip>=23.1""setuptools>=65.5.0" wheel pybind11 six regex nltk numpy deepspeed==0.12.2 einops tensorboard transformers sentencepiece"protobuf<3.21.0"

Installing apex

git clone https://github.com/NVIDIA/apex -b 23.08;cd apexpip install -v --disable-pip-version-check --no-cache-dir --no-build-isolation --config-settings"--build-option=--cpp_ext" --config-settings"--build-option=--cuda_ext" ./cd ..

Installing FlashAttention2

ninja --version

pip install"flash-attn<2.4.0" --no-build-isolation

トークナイザの準備 / Preparing the tokenizer

curl -O -L https://github.com/llm-jp/llm-jp-tokenizer/raw/main/models/ver2.1/code10k_en20k_ja30k.ver2.1.model

事前学習データの準備 / Preparation of pre-training data

importjsonimportsysfromdatasetsimportload_datasetif__name__=="__main__":dataset=load_dataset('mc4','ja',split='train',streaming=True)limit=int(sys.argv[1])count=0fordocindataset:json.dump(doc,sys.stdout,ensure_ascii=False)print()count+=1ifcount==limit:break

python download_mc4_ja.py 10000> mc4-ja-10k.jsonl

python tools/preprocess_data.py \  --input ./mc4-ja-10k.jsonl \  --output-prefix dataset/mc4-ja-10k \  --tokenizer-model ./code10k_en20k_ja30k.ver2.1.model \  --append-eod \  --tokenizer-type SentencePieceTokenizer \  --dataset-impl mmap --workers 8ls -l dataset/mc4-ja-10k*

事前学習スクリプトの準備 / Preparing the pretraining script

cp examples_deepspeed/pretrain_llama2_distributed.sh.chmod +x pretrain_llama2_distributed.sh

< DATASET_1="./tmp/data/bookcorpus_train_1m_text_sentence"> DATASET_1="./dataset/mc4-ja-10k_text_document"< TOKENIZER_PATH=./tmp/tokenizer.model # official llama tokenizer.model> TOKENIZER_PATH=./code10k_en20k_ja30k.ver2.1.model> export NCCL_IB_GID_INDEX=3> export NCCL_IB_TC=106<        --tokenizer-type GPTSentencePieceTokenizer \>        --tokenizer-type SentencePieceTokenizer \

# args_defaults={'tokenizer_type': 'GPT2BPETokenizer'},

事前学習の実行 / Execute pretraining

./pretrain_llama2_distributed.sh