Repository files navigation

This repository contains a PyTorch implementation of the PVSE network and the MRW dataset proposed in our paperPolysemous Visual-Semantic Embedding for Cross-Modal Retrieval (CVPR 2019). The code and data are free to use foracademic purposes only.

Please also visit ourproject page

1. MRW Dataset
2. Setting up an environment
3. Download and prepare data
4. Evaluate pretrained models
5. Train your own model

OurMy Reaction When (MRW) dataset contains50,107 video-sentence pairs crawled from social media, where videos display physical or emotional reactions to the situations described in sentences. This subreddit/r/reactiongifs contains several examples; below shows some representative examples pairs;

(a) Physical Reaction	(b) Emotional Reaction	(c) Animal Reaction	(d) Lexical Reaction
MRW a witty comment I wanted to make was already said	MFW I see a cute girl on Facebook change her status to single	MFW I cant remember if I've locked my front door	MRW a family member askes me why his computer isn't working

Below shows the descriptive statistics of the datset. The word vocabulary size is34,835. The dataset can be used for evaluting cross-modal retrieval systems underambiguous/weak-association between vision and language.

We provide detailed analysis of the dataset in thesupplementary material of the main paper.

Follow the instructionbelow to download the dataset.

We recommend creating a virtual environment and install packages there. Note, you must install the Cython package first.

python3 -m venv <your virtual environment name>source <your virtual environment name>/bin/activatepip3 install Cythonpip3 install -r requirements.txt

cd databash prepare_mrw_dataset.sh

This will download the dataset (without videos) in a JSON format, a vocabulary file, and train/val/test splits. It will then prompt an option:

Do you wish to download video data and gulp them? [y/n]

We provide two ways to obtain the data. A recommended option is to download pre-compiled data in aGulpIO binary storage format, which contains video frames sampled at 8 FPS. For this, simpliy hitn (this will terminate the script) and download our pre-compiled GulpIO data inthis link (54 GB). After finish downloading, extract the tarball underdata/mrw/gulp to train and/or test our models.

If you wish to download raw video clips and gulp them on your own, hity once prompted with the message above. This will start downloading videos and, once finished, start gulping the video files at 8 FPS (you can change this indownload_gulp_mrw.py). If you encounter any problem downloading the video files, you may also download them directly fromthis link (19 GB), and then continue gulping them using the scriptdownload_gulp_mrw.py.

cd databash prepare_tgif_dataset.sh

This will download the dataset (without videos) in a TSV format, a vocabulary file, and train/val/test splits. Please note, we use a slightly modified version of theTGIF dataset because of invalid video files; the script will automatically download the modified version.

It will then prompt an option:

Do you wish to gulp the data? [y/n]

Similar to the MRW data, we provide two options to obtain the data: (1) download pre-compiled GulpIO data, or (2) download raw video clips and gulp them on your own, and we recommend the first option for an easy start. For this, simply hitn and download our pre-compiled GulpIO data inthis link (89 GB). After finish downloadingtgif-gulp.tar.gz, extract the tarball underdata/tgif/gulp.

If you wish to gulp your own dataset, hity and follow the prompt. Note that you must first download a tarball containing the videos before gulping. You can download the filetgif.tar.gz (124 GB) fromthis link and place it under./data/tgif. Once you have the video data, the script will start gulping the video files.

cd databash prepare_coco_dataset.sh

Download all six pretrained models in a tarball atthis link. You can also download each individual files using the links below.

Using the pretrained models you should be able to reproduce the results in the table below

You can train your own model usingtrain.py; checkoption.py for all available options.

For example, you can train our PVSE model (k=2) on COCO using the command below. It uses ResNet152 as a backbone CNN, GloVe word embedding, MMD loss weight 0.01 and DIV loss weight 0.1, and bacth size of 256:

python3 train.py --data_name coco --cnn_type resnet152 --wemb_type glove --margin 0.1 --max_violation --num_embeds 2 --img_attention --txt_attention --mmd_weight 0.01 --div_weight 0.1 --batch_size 256

For video models, you should set the parameter--max_video_length; otherwise it defaults to 1 (single frame). Here's an example command:

python3 train.py --data_name mrw --max_video_length 4 --cnn_type resnet18 --wemb_type glove --margin 0.1 --num_embeds 4 --img_attention --txt_attention --mmd_weight 0.01 --div_weight 0.1 --batch_size 128

If you use any of the material in this repository we ask you to cite:

@inproceedings{song-pvse-cvpr19,  author    = {Yale Song and Mohammad Soleymani},  title     = {Polysemous Visual-Semantic Embedding for Cross-Modal Retrieval},  booktitle = {CVPR},  year      = 2019

Our code is based onthe implementation by Faghri et al.

Last edit: Tuesday July 16, 2019