Repository files navigation

This repo is a tensorflow implementation of the synthetic gradient, or DNI, forrecurrent neural network (RNN). The architecture contains a multilayer LSTM RNN thatis used for language modeling to do word-level prediction. For a detailed description of how synthetic gradient is applied to train this architecture, check out the blog posthere.

The data required to run the model is in the data/ dir of thePTB dataset from Tomas Mikolov's webpage:

$ wget http://www.fit.vutbr.cz/~imikolov/rnnlm/simple-examples.tgz$ tar xvf simple-examples.tgz

To run the RNN model,

$ python sgrnn/main.py --model=small --data_path=simple-examples/data/ \    --num_gpus=0 --rnn_mode=BASIC --save_path=/tmp/sgrnn

Reference:

• Decoupled Neural Interfaces using Synthetic Gradients
• Understanding Synthetic Gradients and Decoupled Neural Interfaces
• nitarshan's implementation of synthetic gradient for MLP
• Tensorflow PTB tutorial