Official Implementation of Self-Supervised Street Gaussians for Autonomous Driving

S³Gaussian: Self-Supervised Street Gaussians for Autonomous Driving

Nan Huang*, Xiaobao Wei, Wenzhao Zheng$^\dagger$, Pengju An, Ming Lu, Wei Zhan, Masayoshi Tomizuka, Kurt Keutzer, Shanghang Zhang$^\ddagger$

* Work done while interning at UC Berkeley $\dagger$ Project leader $\ddagger$ Corresponding author

S³Gaussian employs 3D Gaussians to model dynamic scenes for autonomous driving without other supervisions (e.g., 3D bounding boxes).

• [2023/5/31] Training & evaluation code release!
• [2024/5/31] Paper released on arXiv.

To tackle the challenges in self-supervised street scene decomposition, we propose a multi-resolution hexplane-based encoder to encode 4D grid into feature planes and a multi-head Gaussian decoder to decode them into deformed 4D Gaussians. We optimize the overall model without extra annotations in a self-supervised manner and achieve superior scene decomposition ability and rendering quality.

Our code is developed on Ubuntu 22.04 using Python 3.9 and pytorch=1.13.1+cu116. We also tested on pytorch=2.2.1+cu118. We recommend using conda for the installation of dependencies.

git clone https://github.com/nnanhuang/S3Gaussian.git --recursive
cd S3Gaussian
conda create -n S3Gaussian python=3.9 
conda activate S3Gaussian

pip install -r requirements.txt
pip install -e submodules/depth-diff-gaussian-rasterization
pip install -e submodules/simple-knn

Follow detailed instructions in Prepare Dataset.

For training first clip (eg. 0-50 frames), run

python train.py -s $data_dir --port 6017 --expname "waymo" --model_path $model_path 

If you want to try novel view synthesis, use

--configs "arguments/nvs.py"

For training next clip (eg. 51-100 frames), run

python train.py -s $data_dir --port 6017 --expname "waymo" --model_path $model_path 
--prior_checkpoint "$prior_dir/chkpnt_fine_50000.pth" --configs "arguments/stage2.py"

Also, you can load an existing checkpoint with:

python train.py -s $data_dir --port 6017 --expname "waymo" --start_checkpoint "$ckpt_dir/chkpnt_fine_30000.pth"

For more scripts examples, please check here.

You can visualize and eval a checkpoints follow:

python train.py -s $data_dir --port 6017 --expname "waymo" --start_checkpoint "$ckpt_dir/chkpnt_fine_50000.pth"
--eval_only

Then you can get rendering RGB videos, ground truth RGB videos, depth videos, dynamic rgb videos and static rgb videos.

Credits to @Korace0v0 for building 3D Gaussians for street scenes. Many thanks!

Special thanks to StreetGaussians for sharing visualization results!

Our code is based on 4D Gaussians and EmerNeRF.

Thanks to these excellent open-sourced repos!

If you find this project helpful, please consider citing the following paper:

@article{huang2024s3gaussian,
        title={S3Gaussian: Self-Supervised Street Gaussians for Autonomous Driving},
        author={Huang, Nan and Wei, Xiaobao and Zheng, Wenzhao and An, Pengju and Lu, Ming and Zhan, Wei and Tomizuka,    Masayoshi and Keutzer, Kurt and Zhang, Shanghang},
        journal={arXiv preprint arXiv:2405.20323},
        year={2024}
      }