Repository files navigation

[CoRL 2022] SurroundDepth: Entangling Surrounding Views for Self-Supervised Multi-Camera Depth Estimation
Yi Wei*,Linqing Zhao*,Wenzhao Zheng,Zheng Zhu,Yongming Rao, Guan Huang,Jiwen Lu,Jie Zhou

Depth estimation from images serves as the fundamental step of 3D perception for autonomous driving and is an economical alternative to expensive depth sensors like LiDAR. The temporal photometric consistency enables self-supervised depth estimation without labels, further facilitating its application. However, most existing methods predict the depth solely based on each monocular image and ignore the correlations among multiple surrounding cameras, which are typically available for modern self-driving vehicles. In this paper, we propose a SurroundDepth method to incorporate the information from multiple surrounding views to predict depth maps across cameras.

• python 3.8, pytorch 1.8.1, CUDA 11.4, RTX 3090

git clone https://github.com/weiyithu/SurroundDepth.gitconda create -n surrounddepth python=3.8conda activate surrounddepthpip install -r requirements.txt

Since we usedgp codebase to generate groundtruth depth, you should also install it.

Datasets are assumed to be downloaded underdata/<dataset-name>.

• Please download the officialDDAD dataset and place them underdata/ddad/raw_data. You may refer to officialDDAD repository for more info and instructions.
• Please downloadmetadata of DDAD and place these pkl files indatasets/ddad.
• We provide annotated self-occlusion masks for each sequences. Please downloadmasks and place them indata/ddad/mask.
• Export depth maps for evaluation

cd toolspython export_gt_depth_ddad.py val

• Generate scale-aware SfM pseudo labels for pretraining (it may take several hours). Note that since we use SIFT descriptor in cv2, you need to change cv2 version and we suggest start a new conda environment.

conda create -n sift python=3.6conda activate siftpip install opencv-python==3.4.2.16 opencv-contrib-python==3.4.2.16python sift_ddad.pypython match_ddad.py

• The final data structure should be:

SurroundDepth├── data│   ├── ddad│   │   │── raw_data│   │   │   │── 000000|   |   |   |── ...|   |   |── depth│   │   │   │── 000000|   |   |   |── ...|   |   |── match│   │   │   │── 000000|   |   |   |── ...|   |   |── mask│   │   │   │── 000000|   |   |   |── ...

• Please download the officialnuScenes dataset todata/nuscenes/raw_data
• Export depth maps for evaluation

cd toolspython export_gt_depth_nusc.py val

• Generate scale-aware SfM pseudo labels for pretraining (it may take several hours).

conda activate siftpython sift_nusc.pypython match_nusc.py

• The final data structure should be:

SurroundDepth├── data│   ├── nuscenes│   │   │── raw_data│   │   │   │── samples|   |   |   |── sweeps|   |   |   |── maps|   |   |   |── v1.0-trainval|   |   |── depth│   │   │   │── samples|   |   |── match│   │   │   │── samples

Take DDAD dataset as an example.Train scale-ambiguous model.

python -m torch.distributed.launch --nproc_per_node 8 --num_workers=8 run.py  --model_name ddad  --config configs/ddad.txt

Train scale-aware model. First we should conduct SfM pretraining.

python -m torch.distributed.launch --nproc_per_node 8  run.py  --model_name ddad_scale_pretrain  --config configs/ddad_scale_pretrain.txt

Then we select the best pretrained model.

python -m torch.distributed.launch --nproc_per_node 8  run.py  --model_name ddad_scale  --config configs/ddad_scale.txt  --load_weights_folder=${best pretrained}

We observe that the training on nuScenes dataset is unstable and easy to overfit. Also, the results with 4 GPUs are much better than 8 GPUs. Thus, we set fewer epochs and use 4 GPUs for nuScenes experiments. We also provide SfM pretrained model onDDAD andnuScenes.

python -m torch.distributed.launch --nproc_per_node${NUM_GPU}  run.py  --model_nametest  --config configs/${TYPE}.txt --models_to_load depth encoder   --load_weights_folder=${PATH}  --eval_only

Our code is based onMonodepth2.

If you find this project useful in your research, please consider cite:

@article{wei2022surround,    title={SurroundDepth: Entangling Surrounding Views for Self-Supervised Multi-Camera Depth Estimation},    author={Wei, Yi and Zhao, Linqing and Zheng, Wenzhao and Zhu, Zheng and Rao, Yongming and Huang ,Guan and Lu, Jiwen and Zhou, Jie},    journal={arXiv preprint arXiv:2204.03636},    year={2022}}