Repository files navigation

This repository contains a implementation of a Advanced Differentiable Neural Computer (ADNC) for a more robust andscalable usage in Question Answering. This work is published on theMRQA workshop at theACL 2018. The ADNC is applied to the20 bAbI QA tasks withSOTA mean results and to theCNN Reading Comprehension Task withpassable results without any adaptation or hyper-parameter tuning.

The repository contains the following features:

• Modular implementation of controller and memory unit
• Fully configurable model/experiment with a yaml-config-file
• Unit tests for all key parts (memory unit, controller, etc. )
• Pre-trained models on bAbI task and CNN RC task
• Plots of the memory unit functionality during sequence inference
• The following advancements to the DNC:

Bypass Dropout	DNC Normalization	Content Based Memory Unit	Bidirectional Controller
Dropout to reduce the bypass connectivity Forces an earlier memory usage during training	Normalizes the memory unit's input Increases the model stability during training	Memory Unit without temporal linkage mechanism Reduces memory consumption by up to 70	Bidirectional DNC Architecture Allows to handle variable requests and rich information extraction

Please find detailed information about the advancements and the experiments in

• MRQA 2018 paper submissionRobust and Scalable Differentiable Neural Computer for Question Answering
• My master thesis about theAdvanced DNC for Question Answering with a detailed DNC/ADNC description.

The plot below shows the impact of the different advancements in the word error rate with the bAbI task 1.

Furthermore, it contains a set of rich analysis tools to get a deeper insight in the functionality of the ADNC. For examplethat the advancements lead to a more meaningful gate usage of the memory cell as you can see in the following plots:

To setup an virtual environment and install ADNC:

git clone https://github.com/joergfranke/ADNC.gitcd ADNC/python3 -m venv venvsource venv/bin/activatepip install -e .

The repository contains different pre-trained models in the experiments folder.ForbAbI inference, choose pre-trained model e.g.adnc and run:

python scripts/inference_babi_task.py adnc

Possible models arednc,adnc,biadnc on bAbi Task 1 andbiadnc-all,biadnc-aug16-all for all bAbI tasks with or without augmentation of task 16. The augmentation provides equal word distribution during training.

ForCNN inference of pre-trained ADNC run:

python scripts/inference_babi_task.py

The configuration filescripts/config.yml contains the full config of the ADNC training. The training script can be run with:

python scripts/start_training.py

It starts a bAbI training and plots every epoch a function plot to control the training progress.

To plot a function plot of the bAbI task choose pre-trained model e.g.adnc and run:

python scripts/plot_function_babi_task.py

Possible models arednc,adnc,biadnc on bAbi Task 1 andbiadnc-all,biadnc-aug16-all for all bAbI tasks with or without augmentation of task 16.

• Joint trained on all 20 tasks.
• Mean results of 5 training runs with different initializations.
• Similar hyper-parameter as theoriginal DNC
• The unidirectional controller has one LSTM layer and 256 hidden units and the bidirectional has 172 hidden units in each direction.
• The memory unit has 192 locations, a width of 64 and 4 read heads.
• Bypass Dropout is applied with a dropout rate of 10%.
• The model is optimized with RMSprop with fixed learning rate of 3e-05 and momentum of 0.9.
• Task 16 Augmentation: The task contains a strong local minimum. Given the most common color as answer leads to a correct answer in 50% of the cases.

• All hyper-parameters are chosen inspired by related work.
• The controller is a LSTM with one hidden layer and a layer size of 512 and a memory matrix with 256 locations, a width of 128 and four read heads.
• Bypass Dropout is applied with a dropout rate of 10%.
• The maximum sequence length during training is limited to 1400 words.
• The model is optimized with RMSprop with fixed learning rate of 3e-05 and momentum of 0.9.