Repository files navigation

This repository contains the code for the following paper:

• D. Fried*, R. Hu*, V. Cirik*, A. Rohrbach, J. Andreas, L.-P. Morency, T. Berg-Kirkpatrick, K. Saenko, D. Klein**, T. Darrell**,Speaker-Follower Models for Vision-and-Language Navigation. in NeurIPS, 2018. (PDF)

@inproceedings{fried2018speaker,  title={Speaker-Follower Models for Vision-and-Language Navigation},  author={Fried, Daniel and Hu, Ronghang and Cirik, Volkan and Rohrbach, Anna and Andreas, Jacob and Morency, Louis-Philippe and Berg-Kirkpatrick, Taylor and Saenko, Kate and Klein, Dan and Darrell, Trevor},  booktitle={Neural Information Processing Systems (NeurIPS)},  year={2018}}

(*, **: indicates equal contribution)

Project Page:http://ronghanghu.com/speaker_follower

If you only want to use our data augmentation on the R2R dataset but don't need our models, you can directly download our augmented data on R2R (JSON file containing synthetic data generated by our speaker model)here. This JSON file is in the same format as the original R2R dataset, with one synthetic instruction per sampled new trajectory.

Note that we first trained on the combination of the original and the augmented data, and then fine-tuned on the original training data.

1. Install Python 3 (Anaconda recommended:https://www.continuum.io/downloads).
2. Install PyTorch following the instructions onhttps://pytorch.org/ (we used PyTorch 0.3.1 in our experiments).
3. Download this repository or clonerecursively with Git, and then enter the root directory of the repository:

# Make sure to clone with --recursivegit clone --recursive https://github.com/ronghanghu/speaker_follower.gitcd speaker_follower

If you didn't clone with the--recursive flag, then you'll need to manually clone the pybind submodule from the top-level directory:

git submodule update --init --recursive

4. Install the dependencies for the Matterport3D Simulator:

sudo apt-get install libopencv-dev python-opencv freeglut3 freeglut3-dev libglm-dev libjsoncpp-dev doxygen libosmesa6-dev libosmesa6 libglew-dev

5. Compile the Matterport3D Simulator:

mkdir build && cd buildcmake ..makecd ../

Note: This repository is built upon theMatterport3DSimulator codebase. Additional details on the Matterport3D Simulator can be found inREADME_Matterport3DSimulator.md.

1. Download the Precomputing ResNet Image Features, and extract them intoimg_features/:

mkdir -p img_features/cd img_features/wget https://www.dropbox.com/s/o57kxh2mn5rkx4o/ResNet-152-imagenet.zip?dl=1 -O ResNet-152-imagenet.zipunzip ResNet-152-imagenet.zipcd ..

(In case the URL above doesn't work, it is likely because the Room-to-Room dataset changed its feature URLs. You can find the latest download linkshere.)

After this step,img_features/ should containResNet-152-imagenet.tsv. (Note that you only need to download the features extracted from ImageNet-pretrained ResNet to run the following experiments. Places-pretrained ResNet features or actual images are not required.)

2. Download the R2R dataset and our sampled trajectories for data augmentation:

./tasks/R2R/data/download.sh

1. Train the speaker model:

python tasks/R2R/train_speaker.py

2. Generate synthetic instructions from the trained speaker model as data augmentation:

# the path prefix to the speaker model (trained in Step 1 above)export SPEAKER_PATH_PREFIX=tasks/R2R/speaker/snapshots/speaker_teacher_imagenet_mean_pooled_train_iter_20000python tasks/R2R/data_augmentation_from_speaker.py \    $SPEAKER_PATH_PREFIX \    tasks/R2R/data/R2R

After this step,R2R_literal_speaker_data_augmentation_paths.json be generated undertasks/R2R/data/. This JSON file contains synthetic instructions generated by the speaker model on sampled new trajectories in the train environment (i.e. the speaker-driven data augmentation in our paper).

Alternatively, you can directly download our precomputed speaker-driven data augmentation with./tasks/R2R/data/download_precomputed_augmentation.sh.

3. Train the follower model on the combination of the original and the augmented training data.

python tasks/R2R/train.py \  --use_pretraining --pretrain_splits train literal_speaker_data_augmentation_paths

The follower will be first trained on the combination of the originaltrain environment and the newliteral_speaker_data_augmentation_paths (generated in Step 2 above) for 50000 iterations, and then fine-tuned on theoriginaltrain environment for 20000 iterations. This step may take a long time. (It look approximately 50 hours using a single GPU on our local machine.)

• All the commands above run on a single GPU. You may choose a specific GPU by settingCUDA_VISIBLE_DEVICES environment variable (e.g.export CUDA_VISIBLE_DEVICES=1 to use GPU 1).
• You may directly download our trained speaker model and follower model with

./tasks/R2R/snapshots/release/download_speaker_release.sh  # Download speaker./tasks/R2R/snapshots/release/download_follower_release.sh  # Download follower

The scripts above will save the downloaded models under./tasks/R2R/snapshots/release/. To use these downloaded models, set the speaker and follower path prefixes as follows:

export SPEAKER_PATH_PREFIX=tasks/R2R/snapshots/release/speaker_final_releaseexport FOLLOWER_PATH_PREFIX=tasks/R2R/snapshots/release/follower_final_release

• One can also train the follower only on the original training datawithout using the augmented data from the speaker as follows:

python tasks/R2R/train.py

1. Set the path prefixes for the trained speaker and follower model:

# the path prefixes to the trained speaker and follower model# change these path prefixes if you are using downloaded models.export SPEAKER_PATH_PREFIX=tasks/R2R/speaker/snapshots/speaker_teacher_imagenet_mean_pooled_train_iter_20000export FOLLOWER_PATH_PREFIX=tasks/R2R/snapshots/follower_with_pretraining_sample_imagenet_mean_pooled_train_iter_11100

2. Generate top-ranking trajectory predictions with pragmatic inference:

# Specify the path prefix to the output evaluation fileexport EVAL_FILE_PREFIX=tasks/R2R/eval_outputs/pragmaticspython tasks/R2R/rational_follower.py \    $FOLLOWER_PATH_PREFIX $SPEAKER_PATH_PREFIX \    --batch_size 15 --beam_size 40 --state_factored_search \    --use_test_set \    --eval_file $EVAL_FILE_PREFIX

This will generate the prediction files in the directory ofEVAL_FILE_PREFIX, and also print the performance onval_seen andval_unseen splits. (The displayed performance will be zero on thetest split, since the test JSON file does not contain ground-truth target locations.) The predicted trajectories with the above script contain only thetop-scoring trajectories among all candidate trajectories, ranked with pragmatic inference. The expected success rates are 70.1% and 54.6% onval_seen andval_unseen, respectively.

3. For participating in theVision-and-Language Navigation Challenge, add--physical_traversal option to generate physically-plausible trajectory predictions with pragmatic inference:

# Specify the path prefix to the output evaluation fileexport EVAL_FILE_PREFIX=tasks/R2R/eval_outputs/pragmatics_physicalpython tasks/R2R/rational_follower.py \    $FOLLOWER_PATH_PREFIX $SPEAKER_PATH_PREFIX \    --batch_size 15 --beam_size 40 --state_factored_search \    --use_test_set --physical_traversal \    --eval_file $EVAL_FILE_PREFIX

This will generate the prediction files in the directory ofEVAL_FILE_PREFIX. These prediction files can be submitted tohttps://evalai.cloudcv.org/web/challenges/challenge-page/97/overview for evaluation. The expected success rate on the challenge test set is 53.5%.

The major difference with--physical_traversal is that now the generated trajectories containall states visited by the search algorithm in the order they are traversed. The agent expands each route one step forward at a time, and then switches to expand the next route. The details are explained in Appendix E inour paper.

4. In addition, it is also possible to evaluate the performance of the follower alone, using greedy decoding (without pragmatic inference from the speaker):

export EVAL_FILE_PREFIX=tasks/R2R/eval_outputs/greedypython tasks/R2R/validate.py \    $FOLLOWER_PATH_PREFIX \    --batch_size 100 \    --use_test_set \    --eval_file $EVAL_FILE_PREFIX

This will generate the prediction files in the directory ofEVAL_FILE_PREFIX, and also print the performance onval_seen andval_unseen splits. (The displayed performance will be zero on thetest split, since the test JSON file does not contain ground-truth target locations.) The expected success rates are 66.4% and 35.5% onval_seen andval_unseen, respectively.

This repository is built upon theMatterport3DSimulator codebase.