Repository files navigation

This repository contains MATLAB codes and data for sparse depth sensing, the problem of dense depth image reconstruction from very limited amount of measurements. Please refer to our paperSparse Depth Sensing for Resource-Constrained Robots and click on theYouTube video below for more details.

• Matlab R2015a or later versions, including
  • the Computer Vision System Toolbox
  • the Robotics System Toolbox

• The code is self-contained.No installation required.
• However, if you are interested in trying out a differnet solverCVX, please download it from thislink and follow the installation instructions there. CVX is a Matlab-based modeling system for convex optimization and it has slightly higher accuracy than our fast solver NESTA. We recommend obtaining anacademic license for the most optimized performance.

• rundemo_single_frame.m for a demo of the reconstruction algorithm on samples from each individual frame of depth images.
• rundemo_multi_frame.m for a demo of the reconstruction algorithm on samples collected across multiple frames, given odometry information.
• rundemo_middlebury.m for a demo of the reconstruction algorithm on the Middlebury dataset.

The code is structured as follows.

• lib/algorithm contains the core code, i.e., the formulation of our optimization problem (inl1ReconstructionOnImage.m).
• lib/nesta_solver is the implementation of the NESTA fast solver tailored to our problem.
• lib/geometry handles all geometry related implementations (e.g., rigid body transformation, image projection).
• lib/sampling provides functions for create a small set of measurements from the ground truth depth image.
• lib/utility contains other helper functions.

Thedata folder contains two datasets, including

• ZED: rgb images and depth images collected from theZED Stereo Camera.
• lids_floor6: rgb and depth images collected from the Kinect sensor, along with odometry information obtained from a odometer.

Tested on following hosts

• Matlab R2017a on Windows 7 Enterprise
• Matlab R2016b on Ubuntu 16.04 LTS
• Matlab R2017a on macOS Sierra (Version 10.12.5)

Future work:

• add CUDA code for parallelization on GPU

If you use our code in your research, please consider citing:

@article{Ma2017SparseDepthSensing,  title={Sparse Depth Sensing for Resource-Constrained Robots},  author={Ma, Fangchang and Carlone, Luca and Ayaz, Ulas and Karaman, Sertac},  journal={arXiv preprint arXiv:1703.01398},  year={2017}}@inproceedings{ma2016sparse,  title={Sparse sensing for resource-constrained depth reconstruction},  author={Ma, Fangchang and Carlone, Luca and Ayaz, Ulas and Karaman, Sertac},  booktitle={Intelligent Robots and Systems (IROS), 2016 IEEE/RSJ International Conference on},  pages={96--103},  year={2016},  organization={IEEE}}

Please email Fangchang Ma (fcma@mit.edu) for problems and bugs. Thanks!