Repository files navigation

This repository contains an implementation of the two models (the Deep LSTM and the Attentive Reader) described inTeaching Machines to Read and Comprehend by Karl Moritz Hermann and al., NIPS, 2015. This repository also contains an implementation of a Deep Bidirectional LSTM.

The three models implemented in this repository are:

• deepmind_deep_lstm reproduces the experimental settings of the DeepMind paper for the LSTM reader
• deepmind_attentive_reader reproduces the experimental settings of the DeepMind paper for the Attentive reader
• deep_bidir_lstm_2x128 implements a two-layer bidirectional LSTM reader

We trained the three models during 2 to 4 days on a Titan Black GPU. The following results were obtained:

Here is an example of attention weights used by the attentive reader model on an example:

Software dependencies:

• Theano GPU computing library library
• Blocks deep learning framework
• Fuel data pipeline for Blocks

Optional dependencies:

• Blocks Extras and a Bokeh server for the plot

We recommend usingAnaconda 2 and installing them with the following commands (wherepip refers to thepip command from Anaconda):

pip install git+git://github.com/Theano/Theano.gitpip install git+git://github.com/mila-udem/fuel.gitpip install git+git://github.com/mila-udem/blocks.git -r https://raw.githubusercontent.com/mila-udem/blocks/master/requirements.txt

Anaconda also includes a Bokeh server, but you still need to installblocks-extras if you want to have the plot:

pip install git+git://github.com/mila-udem/blocks-extras.git

The corresponding dataset is provided byDeepMind but if the script does not work (or you are tired of waiting) you can checkthis preprocessed version of the dataset byKyunghyun Cho.

Set the environment variableDATAPATH to the folder containing the DeepMind QA dataset. The training questions are expected to be in$DATAPATH/deepmind-qa/cnn/questions/training.

Run:

cp deepmind-qa/* $DATAPATH/deepmind-qa/

This will copy our vocabulary listvocab.txt, which contains a subset of all the words appearing in the dataset.

To train a model (see list of models at the beginning of this file), run:

./train.py model_name

Be careful to set yourTHEANO_FLAGS correctly! For instance you might want to useTHEANO_FLAGS=device=gpu0 if you have a GPU (highly recommended!)

Teaching Machines to Read and Comprehend, by Karl Moritz Hermann, Tomáš Kočiský, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman and Phil Blunsom, Neural Information Processing Systems, 2015.

Thomas Mesnard

Alex Auvolat

Étienne Simon

We would like to thank the developers of Theano, Blocks and Fuel at MILA for their excellent work.

We thank Simon Lacoste-Julien from SIERRA team at INRIA, for providing us access to two Titan Black GPUs.