Repository files navigation

A fast LSTM Language Model for large vocabulary language like Japanese and Chinese.

It focuses onaccelerating inference time andreducing model size to fit requirement of real-time applications especially in client side. It is85% smaller, and are50x faster than standard LSTM solution with softmax. See the paperJLM - Fast RNN Language Model with Large Vocabulary for performance detail.

The training part is done with TensorFlow. Instead of depending on a big dynamic library of TF to run in client app, we dumped the trained weights out. The inference and decoding can be done by python with numpy or C++ with Eigen. There is no black box.

We implemented the standard LSTM ,tie-embedding,D-softmax, andD-softmax* in both training and numpy inference stages as a comparison. In practice, please consider just use D-softmax* for your best interest.

A standard Viterbi decoder with beam search is implemented. It batches prediction to save decoding time. We also implementedEnabling Real-time Neural IME with Incremental Vocabulary Selection at NAACL 2019. It further more reduced the softmax cost during decoding by ~95%, reaching real-time in commodity CPU.

We useBCCWJ Japanese corpus for example.

Make sure you have a txt file with a white space segmented sentence per line.

If not, use tools likeMeCab to get the job done first.

In JLM, the unitword is the in the format ofdisplay/reading/POS. For example "品川/シナガワ/名詞-固有名詞-地名-一般".

python data.py -v 50000

Make sure you put your corpus.txt in the data folder first. The script will generate a encoded corpus and a bunch of pickles for later usage. The lexicon is also generated in this stage. Words in lexicon are ordered by their frequency for the convenience of vocabulary segmentation.

• Config the experiment folder path (put the corpus with vocab you would like to use in the config)

data_path = os.path.abspath(os.path.join(root_path, "data/corpus_50000"))

• Enter the train folder, and edit the run filetrain.py file

parameters= {"is_debug":False,"batch_size":128*3,"embed_size":256,"hidden_size":512,"num_steps":20,"max_epochs":10,"early_stopping":1,"dropout":0.9,"lr":0.001,"share_embedding":True,"gpu_id":0,"tf_random_seed":101,"D_softmax":False,"V_table":True,"embedding_seg": [(200,0,12000), (100,12000,30000), (50,30000,None)]}

V_table is the D-softmax* here. Theembedding_seg means how you want to separate your vocabulary. Make your own best result by tuning the hyper parameters. The results will be saved to folder "experiments" with a ID,sacred framework will take care of all the experiment indexing.

A small hint here, use up your GPU memory by setting a biggernum_steps andbatch_size to best reduce training time.

Run thetest.py in train folder, it will auto generate a random sentence with the trained model. If the sentence doesn't make sense to your language knowledge at all. One of the stage is not correctly setup. Here is an example of random generated sentence.

自宅 に テレビ 局 で 電話 し たら 、 で も 、 すべて の 通話 が 実現 。

It is a common issue to bring trained model into clients like mobile device or Windows environment. We recommend you use ONNX for the purpose. But for this example, we wantfull control and want to cutoff dependency to TF. Run theweights.py in the train folder to get a pickle of all the trainable parameters in the numpy format. You can use the following command to even dump the txt format for other usage.

python weiths.py -e 1 -v True

Runeval.py to see the conversion accuracy of your trained model with offline model.

python eval.py

We also implemented the k-means quantization mentioned in the ICLR 2016 paperDeep Compression. The code is atcomp.py. Run the script with the experiment id. It will generate the code and codebook. By our experiments with 8 bits, there is almost no accuracy loss for conversion task.

go tohttp://nlpfun.com/ime