Repository files navigation

This repository provides a Torch implementation of the framework proposed in CVPR 2017 paperSynthesizing 3D Shapes via Modeling Multi-View Depth Maps and Silhouettes with Deep Generative Networks byAmir A. Soltani,Haibin Huang,Jiajun Wu,Tejas Kulkarni andJoshua Tenenbaum

Slides used for two invited talks at CVPR 2017Vision Meets Cognition Workshop andMIT Vision Seminar (contains new results):Here

CVPR 2017 Poster:Here

• Torch
• nn
• nngraph
• cutorch
• cunn
• paths
• image
• optim
• gnuplot
• xlua -- Needed only when processing/preparing your own data set throught1_dataLoader.lua
• OpenCV -- for final 3D-model reconstruction and visualization

• tSNE -- For running the tSNE experiment
• cudnn v6 or higher --git clone https://github.com/soumith/cudnn.torch.git -b R7 && cd cudnn.torch && luarocks make cudnn-scm-1.rockspec

The following installs luaJIT and luarocks locally in$HOME/usr. If youwant a system-wide installation, remove the-DCMAKE_INSTALL_PREFIX=$HOME/usr option.

git clone https://github.com/torch/luajit-rocks.gitcd luajit-rocksmkdir build;cd buildcmake .. -DCMAKE_INSTALL_PREFIX=$HOME/usr -DWITH_LUAJIT21=OFFmake -j 4make installcd ../..

We assumeluarocks andluajit are in$PATH. Ifthey are not - and assuming you installed them locally in$HOME/usr - youcan instead run~/usr/bin/luarocks and~/usr/bin/luajit.

If you run into compilation issues or you are willing to use CUDA version 9.1 and higher, you may try usingthis Git repo to compile Torch7 as follow:

git clone https://github.com/nagadomi/distro.git~/torch --recursivecd~/torch./install-deps./install.sh

luarocks install torchluarocks install cudnn# for GPU supportluarocks install cunn# for GPU support

We recommend using a machine with ~200GBs of free storage (~60GBs if you're usingModelNet40), ~10GBs of memory and a GPU with ~5GBs of memory with the default arguments. You will need less than 2GBs of free GPU memory when using the model for running experiments (4_0_runExps.lua). GPU memory and RAM requirement could be reduced by settingnCh andmaxMemory arguments to a smaller values respectively.

Usemain.lua to train new models or run experiments for a pre-trained model. Before running the code make sure you specify a directory name for your model-to-be-trained by setting the input arguments:

• benchmark: Set to1 if you want to use a benchmark data set (E.g. ModelNet40 here)
• fromScratch: If set to1 the code will load the 2D images to Torch tensors and save them onto disk. Make sure you have unzipped the contents of the.zip files either manually or via running the code with thezip argument set to1
• modelDirName: Name of the directory to save model and results of each run of the code
• dropoutNet: Set to1 to train or use a pre-trainedDropOutNet model
• singleVPNet: Set to1 to trainor or use a pre-trainedSingleVPNet model
• conditional: Set to1 for conditional models
• silhouetteInput: Set to1 to use silhouettes for training/testing
• KLD: The KLD term's gradient coefficient (smaller values like 10 delivers better reconstructions after training)
• experiment: Set to0 when you want to train a model and to1 when you want to use a pre-trained model
• fromEpoch: Set to the epoch number from which you want to load a model and use it for testing (use whenexperiment is set to1). Set to0 to use one of the pre-trained models (download fromhere)

Setting bothdropoutNet andsingleVPNet to0 means that you are working with aAllVPNet model.

You can download the pre-rendered data sets forShapNet Core andModelNet 40 fromhere. You will need about ~30GBs and ~8GBs for the .zip files forShapeNet Core andModelNet40 data sets respectively. The data sets only contain raw depth map rendering results in float or int (stored in png files). To process the data sets and store the viewpoint depth maps intoTorch tensors you would need an additional ~160GBs and ~55GBs for eachShapeNet Core orModelNet40 respectively. Set thebenchmark argument to 0 or 1 to use the ShapeNet core or ModelNet40 data sets respectively.

Run the following command to store the rendered 2D images of depth maps into Torch tensors:

th main.lua -zip 1 -fromScratch 1 -maxMemory 3000 -train 0

If you want to create your own data set you can use the rendering code provided in/depth_render_reconstruction/code/renderDepth/. The obj/ply files names should be inmodel_x format wherex could be any integer. You need to have install Microsoft Visual C++ 2013 Redistributable. After rendering view points to create your data set make sure you follow the following pattern for creating your .zip files:

CategoryName.zip└── CategoryName_depth_[rgb/float]    ├── model_x_Cam_0.jpg     ├── model_x_Cam_0.jpg    ├── ...    └── model_x_Cam_19.jpg

Note that the filecamPosList.txt contains thex,y andz coordinates for20 camera positions that we used. In case you want to compile the rendering pipeline and not use the provided.exe file for rendering views, you may compilethis code. You may also compile a newer version of the same code provided.

Here are a few examples on how to train different models. The models will be trained on the ShapeNet core data set unless you setbenchmark to1:

Train an unconditional AllVPNet model using depth maps:th main.lua  -modelDirName "AllVPNet" -dropoutNet 0 -singleVPNet 0 -silhouetteInput 0 -conditional 0 -benchmark 0Train a conditional `DropoutNet` model using depth maps:th main.lua  -modelDirName "DropoutNet-Conditional" -dropoutNet 1 -singleVPNet 0 -silhouetteInput 0 -conditional 1 -benchmark 0Train a conditional SingleVPNet using silhouettes:th main.lua  -modelDirName "SingleVPNet-Cond" -dropoutNet 0 -singleVPNet 1 -silhouetteInput 1 -conditional 1 -benchmark 0Train an unconditional AllVPNet model for `ModelNet40` data set with silhouettes:th main.lua  -modelDirName "AllVPNet" -dropoutNet 0 -singleVPNet 0 -silhouetteInput 0 -conditional 0 -benchmark 1

Make sure you set thetestPhase argument to0 when training or running experiments

To load a model and use it for testing (reconstruction, sampling etc) download a pre-trained model fromhere and do the followings:

Make sure you have stored the depth maps into tensors (th main.lua -zip 1 -fromScratch 1 -maxMemory 3000 -train 0)Set the optional arguments in accordance to the model downloadedCopy model.t7 and mean_logvar.t7 to modelDirName/model/th main.lua -modelDirName 'someModelDirName' -experiment 1 -expType 'randomSampling' -conditional 1 -sampleCategory 'chair, car, sofa, laptop, guitar, airplane'expType could be either randomSampling, interpolation, forwardPass, NNs or tSNE

Optional:

Set the testPhase argument to 1 to only use the test set for quick runs/tests

To generate the final 3D shape do the followings:

Unzip `depth_render_reconstruction.zip`Compile the code in /depth_render_reconstruction/code/depthReconstruction_Ubuntu/depthReconstruction (Make sure OpenCV is installed)Copy the built executable file `depthReconstruction` and into the results directory; camPosList.txt must be present in the same directory.Run the 4_1_visualization.lua -inputDir 'anywhereOnDisk' -outputDir 'anywhereOnDisk' -experiment 'sampling'The valid inputs for experiment are reconstruction, sampling, interpolation and `NN`. Use reconstruction for nyud and userData experiments.

th main.lua -modelDirName 'someModelDirName' -experiment 1 -expType 'randomSampling' -conditional 1 -sampleCategory 'chair, car, sofa, laptop, guitar, airplane'

For conditional models you can specify which categories to get samples for by having comma-separated category names in thesampleCategory argument. Simply removesampleCategory argument if you want to get samples for all categories.

Here are some random samples obtained from unconditionally-trained models:

And here are some conditional samples. The models are able to quickly learn the generative distribution for a new category through hierarchical priors and using the learned representations for other categories. For instance, we only had about 50 samples of headphones in the training set but the model generates good-looking samples of headphones after training:

We show that our models do not memorize the training samples by showing the nearest neighbor examples (on the Z layer embeddings) from the test set. You can run the nearest neighbor experiment for both unconditional and conditional samples. You can download the samples selected manually (manual selection is done via viz.txt) fromhere. To get the nearest neighbor samples:

Make sure you have your selected samples here `/modelDirName/experiments/[conditionalSamples/randomSamples]-empirical/``th main.lua  -modelDirName "AllVPNet-Conditional-Sil"  -silhouetteInput 1 -conditional 1 -benchmark 0 -experiment 1 -fromEpoch 0 -expType 'NNs'`

If you want to use the model trained on yoru machine, make sure you set thefromEpoch argument and create a viz.txt file in each sample set directory(e.g.../randomSamples-empirical/sample1/ or.../conditionalSamples-empirical/airplane/sample1/)

th main.lua -modelDirName 'someModelDirName' -experiment 1 -conditional 0 -expType 'forwardPass' -forwardPassType 'randomReconstruction' -nReconstructions 80

Run the following to reconstruct all samples in the test data set:

th main.lua -modelDirName 'someModelDirName' -experiment 1 -conditional 1 -expType 'forwardPass' -forwardPassType 'reconstructAllSamples'

Make sure you extract thezip-ExtraData.zip into the repository directory before running the following commands. ForuserData the code will convert rgb images into silhouettes ifsilhouetteInput is set to1. If you set it to0 the code will assume you only have depth maps in/ExtraData/userData/ and are working with a model trained on depth maps.

th main.lua -modelDirName 'someModelDirName' -experiment 1 -conditional 0 -expType 'forwardPass' -forwardPassType 'nyud'

th main.lua -modelDirName 'someModelDirName' -experiment 1 -conditional 0 -silhouetteInput 0 -expType 'forwardPass' -forwardPassType 'userData'

th main.lua -modelDirName 'someModelDirName' -conditional 0 -experiment 1 -expType 'interpolation' -nSamples 3 -sampleCategory 'chair, car, sofa, laptop, guitar, airplane'

Remove-sampleCategory 'chair, car, sofa, laptop, guitar, airplane' if you want to get interpolation results for all categories

Run the following command for applying the tSNE algorithm on the Z layer embeddings.

th main.lua -modelDirName 'someModelDirName' -experiment 1 -conditional 0 -silhouetteInput 0 -expType 'tSNE''

Here is the tSNE visualization for all of the ShapeNet Core training samples:

Clickhereto view it with the original resolution.

SingleVPNet models give similar reconstructions/IoU/classification accuracy for most viewpoints, meaning that they obtain some consistent representations. You can run the representation consistency experiment on ourShapeNet Core test set by running the following command:

th main.lua  -modelDirName "SingleVPNet-Conditional-Depth"  -silhouetteInput 0 -singleVPNet 1 -conditional 1 -benchmark 0 -experiment 1 -fromEpoch 0 -expType 'forwardPass' -forwardPassType 'reconstructAllSamples' -allViewsExp 1th main.lua  -modelDirName "SingleVPNet-Conditional-Silhouette"  -silhouetteInput 1 -singleVPNet 1 -conditional 1 -benchmark 0 -experiment 1 -fromEpoch 0 -expType 'forwardPass' -forwardPassType 'reconstructAllSamples' -allViewsExp 1

Execute the following to get the reconstructions for all test samples:

th main.lua -modelDirName 'someModelDirName' -experiment 1 -expType 'forwardPass' -forwardPassType 'reconstructAllSamples'`

For aSingleVPNet model execute the following to get reconstructions for each views separately:

th main.lua -experiment 1 -expType 'forwardPass' -forwardPassType 'reconstructAllSamples' -allViewsExp 1`

To get the IoU numbers, in a Windows machine open theREADME.txt inzip-IoU.zip and install the requirements. Then Runrun.bat

Please cite our work if you found our paper helpful and/or use our code for your research:

@inproceedings{{3DVAE,  title={Synthesizing 3D Shapes via Modeling Multi-View Depth Maps and Silhouettes with Deep Generative Networks},  author={Soltani, Amir Arsalan and Huang, Haibin and Wu, Jiajun and Kulkarni, Tejas D and Tenenbaum, Joshua B}  booktitle={Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition},  pages={1511-1519},  year={2017}}