Repository files navigation

S³Gaussian: Self-Supervised Street Gaussians for Autonomous Driving

Nan Huang*,Xiaobao Wei,Wenzhao Zheng$^\dagger$, Pengju An,Ming Lu,Wei Zhan,Masayoshi Tomizuka,Kurt Keutzer,Shanghang Zhang$^\ddagger$

* Work done while interning at UC Berkeley$\dagger$ Project leader$\ddagger$ Corresponding author

S³Gaussian employs 3D Gaussians to model dynamic scenes for autonomous drivingwithout other supervisions (e.g., 3D bounding boxes).

• [2023/5/31] Training & evaluation code release!
• [2024/5/31] Paper released onarXiv.

To tackle the challenges in self-supervised street scene decomposition, we propose a multi-resolution hexplane-based encoder to encode 4D grid into feature planes and a multi-head Gaussian decoder to decode them into deformed 4D Gaussians. We optimize the overall model without extra annotations in a self-supervised manner and achieve superior scene decomposition ability and rendering quality.

Our code is developed on Ubuntu 22.04 using Python 3.9 and pytorch=1.13.1+cu116. We also tested on pytorch=2.2.1+cu118. We recommend using conda for the installation of dependencies.

git clone https://github.com/nnanhuang/S3Gaussian.git --recursivecd S3Gaussianconda create -n S3Gaussian python=3.9 conda activate S3Gaussianpip install -r requirements.txtpip install -e submodules/depth-diff-gaussian-rasterizationpip install -e submodules/simple-knn

Follow detailed instructions inPrepare Dataset.

We only use dynamic32 and static32 split.

For training first clip (eg. 0-50 frames), run

python train.py -s $data_dir --port 6017 --expname "waymo" --model_path $model_path

If you want to try novel view synthesis, use

--configs "arguments/nvs.py"

For instance, you can try:

python train.py -s "./data/processed/dynamic32/training/022" --expname "waymo" --model_path "./work_dirs/phase1/dynamic/recon/022"

For training next clip (eg. 51-100 frames), run

python train.py -s $data_dir --port 6017 --expname "waymo" --model_path $model_path --prior_checkpoint "$prior_dir/chkpnt_fine_50000.pth" --configs "arguments/stage2.py"

For instance, you can try:

python train.py -s "./data/processed/dynamic32/training/022" --expname "waymo" --model_path "./work_dirs/phase1/dynamic/recon/p2/022" --prior_checkpoint "./work_dirs/phase1/dynamic/recon/022/chkpnt_fine_50000.pth" --configs "arguments/stage2.py"

Also, you can load an existing checkpoint with:

pythontrain.py-s $data_dir--port6017--expname"waymo"--start_checkpoint"$ckpt_dir/chkpnt_fine_30000.pth"--model_path $model_path

For more scripts examples, please checkhere.

You can visualize and eval a checkpoints follow:

pythontrain.py-s $data_dir--port6017--expname"waymo"--start_checkpoint"$ckpt_dir/chkpnt_fine_50000.pth"--model_path $model_path--eval_only

If you use different configs, you will need to add them as well:

--configs "arguments/nvs.py"

Then you can get rendering RGB videos, ground truth RGB videos, depth videos, dynamic rgb videos and static rgb videos.

Credits to @Korace0v0 for building 3D Gaussians for street scenes. Many thanks!

Special thanks toStreetGaussians for sharing visualization results!

Our code is based on4D Gaussians andEmerNeRF.

Thanks to these excellent open-sourced repos!

If you find this project helpful, please consider citing the following paper:

@article{huang2024s3gaussian,        title={S3Gaussian: Self-Supervised Street Gaussians for Autonomous Driving},        author={Huang, Nan and Wei, Xiaobao and Zheng, Wenzhao and An, Pengju and Lu, Ming and Zhan, Wei and Tomizuka,    Masayoshi and Keutzer, Kurt and Zhang, Shanghang},        journal={arXiv preprint arXiv:2405.20323},        year={2024}      }