Repository files navigation

When I opened this repository in 2017, there was no official code yet.I tried to implement the paper as I understood, but to no surpriseit had several bugs. I realized them mostly thanks to people who issued here, soI'm very grateful to all of them. Though there is theofficial implementation as well asseveral other unofficial github repos, I decided to update my own one.This update focuses on:

• readable / understandable code writing
• modularization (but not too much)
• revising known bugs. (masking, positional encoding, ...)
• updating to TF1.12. (tf.data, ...)
• adding some missing components (bpe, shared weight matrix, ...)
• including useful comments in the code.

I still stick to IWSLT 2016 de-en. I guess if you'd like to test on a big data suchas WMT, you would rely on the official implementation.After all, it's pleasant to check quickly if your model works.The initial code for TF1.2 is moved to thetf1.2_lecacy folder for the record.

• python==3.x (Let's move on to python 3 if you still use python 2)
• tensorflow==1.12.0
• numpy>=1.15.4
• sentencepiece==0.1.8
• tqdm>=4.28.1

• STEP 1. Run the command below to downloadIWSLT 2016 German–English parallel corpus.

bash download.sh

It should be extracted toiwslt2016/de-en folder automatically.

• STEP 2. Run the command below to create preprocessed train/eval/test data.

python prepro.py

If you want to change the vocabulary size (default:32000), do this.

python prepro.py --vocab_size 8000

It should create two foldersiwslt2016/prepro andiwslt2016/segmented.

• STEP 3. Run the following command.

python train.py

Checkhparams.py to see which parameters are possible. For example,

python train.py --logdir myLog --batch_size 256 --dropout_rate 0.5

• STEP 3. Or download the pretrained models.

wget https://dl.dropbox.com/s/4lom1czy5xfzr4q/log.zip; unzip log.zip; rm log.zip

• Run

python test.py --ckpt log/1/iwslt2016_E19L2.64-29146 (OR yourCkptFile OR yourCkptFileDirectory)

• Typically, machine translation is evaluated with Bleu score.
• All evaluation results are available ineval/1 andtest/1.

• Beam decoding will be added soon.
• I'm going to update the code when TF2.0 comes out if possible.