Repository files navigation

By Chen Liu, Jimei Yang, Duygu Ceylan, Ersin Yumer, and Yasutaka Furukawa

This paper presents the first end-to-end neural architecture for piece-wise planar reconstruction from a single RGB image. The proposed network, PlaneNet, learns to directly infer a set of plane parameters and corresponding plane segmentation masks. For more details, please refer to our CVPR 2018paper or visit ourproject website.

We developed a better technique, PlaneRCNN, for piece-wise planar detection as described in our recent arXivpaper. Unfortunately, we cannot release the code and data yet.

We add script for extracting plane information from the original ScanNet dataset and rendering 3D planar segmentation results to 2D views. Please see the README in folderdata_preparation/ for details. Note that we made some modifications to the heuristic-heavy plane fitting algorithms when cleaning up the messy codes developed over time. So the plane fitting results will be slightly different with the training data we used (provided in the.tfrecords files).

PyTorch training and testing codes are available now (still experimental and without the CRF module).

Python 2.7, TensorFlow (>= 1.3), numpy, opencv 3.

Please run the following commands to compile the library for thecrfasrnn module.

cd cppsh compile.shcd ..

To train the network, you also need to run the following commands to compile the library for computing the set matching loss. You need Eigen (I am using Eigen 3.2.92) for the compilation. (Please seehere for details.)

cd nndistancemakecd ..

We convertScanNet data to.tfrecords files for training and testing. The training data can be downloaded fromhere (orhere if you cannont access the previous one), and the validation data can be downloaded fromhere (orhere).

If you download the training data from the BOX link, please run the following command to merge downloaded files into one.tfrecords file.

cat training_data_segments/*> planes_scannet_train.tfrecords

To train the network from the pretrained DeepLab network, please first download the DeepLab modelhere (under the Caffe to TensorFlow conversion), and then run the following command.

python train_planenet.py --restore=0 --modelPathDeepLab="path to the deep lab model" --dataFolder="folder which contains tfrecords files"

Please first download our trained network fromhere (orhere) and put the uncompressed folder under ./checkpoint folder.

To evaluate the performance against existing methods, please run:

python evaluate.py --dataFolder="folder which contains tfrecords files"

A plane is represented by three parameters and a segmentation mask. If the plane equation isnx=d wheren is the surface normal and d is the plane offset, then plane parameters arend. The plane equation is in the camera frame, wherex points to the right,y points to the front, andz points to the up.

Please first download our trained network (see [Evaluation](### Evaluation) section for details). Scriptpredict.py predicts and visualizes custom images (if "customImageFolder" is specified) or ScanNet testing images (if "dataFolder" is specified).

python predict.py --customImageFolder="folder which contains custom images"python predict.py --dataFolder="folder which contains tfrecords files" [--startIndex=0] [--numImages=30]

This will generate visualization images, a webpage containing all the visualization, as well as cache files under folder "predict/".

Same commands can be used for various applications by providing optional arguments,applicationType,imageIndex,textureImageFilename, and some application-specific arguments. The following commands are used to generate visualizations in the submission. (The TV application needs more manual specification for better visualization.)

python predict.py --dataFolder=/mnt/vision/Data/PlaneNet/ --textureImageFilename=texture_images/CVPR.jpg --imageIndex=118 --applicationType=logo_texture --startIndex=118 --numImages=1python predict.py --dataFolder=/mnt/vision/Data/PlaneNet/ --textureImageFilename=texture_images/CVPR.jpg --imageIndex=118 --applicationType=logo_video --startIndex=118 --numImages=1python predict.py --dataFolder=/mnt/vision/Data/PlaneNet/ --textureImageFilename=texture_images/checkerboard.jpg --imageIndex=72 --applicationType=wall_texture --wallIndices=7,9 --startIndex=72 --numImages=1python predict.py --dataFolder=/mnt/vision/Data/PlaneNet/ --textureImageFilename=texture_images/checkerboard.jpg --imageIndex=72 --applicationType=wall_video --wallIndices=7,9 --startIndex=72 --numImages=1python predict.py --customImageFolder=my_images/TV/ --textureImageFilename=texture_images/TV.mp4 --imageIndex=0 --applicationType=TV --wallIndices=2,9python predict.py --customImageFolder=my_images/ruler --textureImageFilename=texture_images/ruler_36.png --imageIndex=0 --applicationType=ruler --startPixel=950,444 --endPixel=1120,2220

Note that, the above script generate image sequences for video applications. Please run the following command under the image sequence folder to generate a video:

ffmpeg -r 60 -f image2 -s 640x480 -i %04d.png -vcodec libx264 -crf 25 -pix_fmt yuv420p video.mp4

To check out the pool ball application, please run the following commands.

python predict.py --customImageFolder=my_images/pool --imageIndex=0 --applicationType=pool --estimateFocalLength=Falsecd poolpython pool.py

Use mouse to play:)

If you have any questions, please contact me atchenliu@wustl.edu.