Repository files navigation

This project is presented as spotlight in CVPR2018.

Humans have strong ability to make inferences about the appearance of the invisible and occluded parts of scenes. For example, when we look at the scene on the left we can make predictions about what is behind the coffee table, and can even complete the sofa based on the visible parts of the sofa, the coffee table, and what we knowin general about sofas and coffee tables and how they occlude each other.

SeGAN can learn to

1. Generate theappearance of the occluded parts of objects,
2. Segment the invisible parts of objects,
3. Although trained on synthetic photo realistic images reliably segmentnatural images,
4. By reasoning about occluder-occludee relations inferdepth layering.

If you find this project useful in your research, please consider citing:

@inproceedings{ehsani2018segan,  title={Segan: Segmenting and generating the invisible},  author={Ehsani, Kiana and Mottaghi, Roozbeh and Farhadi, Ali},  booktitle={CVPR},  year={2018}}

• Using Torch 7 and dependencies fromthis repository.
• Linux OS
• NVIDIA GPU + CUDA + CuDNN

1. Clone the repository using the command:

    git clone https://github.com/ehsanik/SeGAN cd SeGAN

2. Download the dataset fromhere and extract it.

3. Make a link to the dataset.

    ln -s /PATH/TO/DATASET dyce_data

4. Download pretrained weights fromhere and extract it.

5. Make a link to the weights' folder.

    ln -s /PATH/TO/WEIGHTS weights

We introduce DYCE, a dataset of syntheticoccluded objects. This is a synthetic dataset withphoto-realistic images and natural configuration of objectsin scenes. All of the images of this dataset are taken in indoorscenes. The annotations for each image contain thesegmentation mask for the visible and invisible regions ofobjects. The images are obtained by taking snapshots fromour 3D synthetic scenes.

The number of the synthetic scenes that we use is 11,where we use 7 scenes for training and validation, and 4scenes for testing. Overall there are 5 living rooms and 6 kitchens, where 2 living rooms and 2 kitchen are used fortesting. On average, each scene contains 60 objects and thenumber of visible objects per image is 17.5 (by visible wemean having at least 10 visible pixels). There is no commonobject instance in train and test scenes.

The dataset can be downloaded fromhere.

To train your own model:

th main.lua -baseLR 1e-3 -end2end -istrain "train"

Seedata_settings.lua for additional commandline options.

To test using the pretrained model and reproduce the results in the paper:

th main.lua -weights_segmentation "weights/segment" -end2end -weights_texture "weights/texture" -istrain "test" -predictedSV

For testing using the groundtruth visible mask as input instead of the predicted mask:

th main.lua -weights_segmentation "weights/segment_gt_sv" -end2end -weights_texture "weights/texture_gt_sv" -istrain "test"

Code for GAN network borrows heavily frompix2pix.