Repository files navigation

Level 2 Autonomous Driving on a Single Device: Diving into the Devils of Openpilot

This repository is the PyTorch implementation for our Openpilot-Deepdive.In contrast to most traditional autonomous driving solutions where the perception, prediction, and planning module are apart,Openpilot uses an end-to-end neural network to predict the trajectory directly from the camera images, which is called Supercombo. We try to reimplement the training details and test the pipeline on public benchmarks. Experimental results of OP-Deepdive on nuScenes, Comma2k19, CARLA, and in-house realistic scenarios (collected in Shanghai) verify that a low-cost device can indeed achieve most L2 functionalities and be on par with the original Supercombo model. We also test on CommaTwo device with a dual-model deploeyment framework, which is in this repo:Openpilot-Deployment.

• Directory Structure
• Changelog
• Quick Start
  • Installation
  • Dataset
  • Training and Testing
  • Demo
• Baselines
• Citation
• License

Openpilot-Deepdive ├── tools           - Tools to generate split on Comma2k19 and nuScenes datasets.  ├── utils_comma2k19 - The utils provided by comma, copied from `commaai/comma2k19.git/utils`├── data      ├── nuscenes  -> soft link to the nusSenes-all dataset      ├── comma2k19 -> soft link to the Comma2k19 dataset

2022-6-17: We released the v1.0 code for Openpilot-Deepdive.

2022-6-26: We fix some problems and update the readme for using the code on bare-metal machines. Thanks @EliomEssaim and @MicroHest!

2022-7-13: We released the v1.0 code forOpenpilot-Deployment for dual-model deployment in the Openpilot framework.

Before starting, we refer you to read thearXiv to understand the details of our work.

Clone repo and install requirements.txt in aPython>=3.7.0 environment, includingPyTorch>=1.7.

git clone https://github.com/OpenPerceptionX/Openpilot-Deepdive.git  # clonecd Openpilot-Deepdivepip install -r requirements.txt  # install

We train and evaluate our model on two datasets,nuScenes andComma2k19.The table shows some key features of them.

Please create adata folder and create soft links to the datasets.

For dataset splits, you may create your own by running the scripts in thetools folder, or download it inhttps://github.com/OpenPerceptionX/Openpilot-Deepdive/issues/4.

By default, the batch size is set to be 6 per GPU, which consumes 27 GB GPU memory. When using 8 V100 GPUs, it takes approximate 120 hours to train 100 epochs on Comma2k19 dataset.

Note: Our lab useslurm to run and manage the tasks. Then, the PyTorch distributed training processes are initialized manually byslurm, since the automaticmp.spawn may cause unknown problems on slurm clusters. For most people who do not use a cluster, it's okay to launch the training process on bare-metal machines, but you will have to open multiple terminals and set some environmental variables manually if you want to use multiple GPUs. We will explain it below.

Warning: Since we have to extract all the frames from the video before sending them into the network, the program is hungry for memory. The actual memory usage is related tobatch_size andn_workers. By default, each process withn_workers=4 andbatch_size=6 consumes around 40 to 50 GB memory. You'd better open anhtop to monitor the memory usage, before the machine hangs.

# Training on a slurm clusterexport DIST_PORT = 23333  # You may use whatever you wantexport NUM_GPUS = 8PORT=$DIST_PORT$ srun -p $PARTITION$ --job-name=openpilot -n $NUM_GPUS$ --gres=gpu:$NUM_GPUS$ --ntasks-per-node=$NUM_GPUS$ python main.py

# Training on a bare-metal machine with a single GPUPORT=23333 SLURM_PROCID=0 SLURM_NTASKS=1 python main.py

# Training on a bare-metal machine with multiple GPUs# You need to open multiple terminals# Let's use 4 GPUs for example# Terminal 1PORT=23333 SLURM_PROCID=0 SLURM_NTASKS=4 python main.py# Terminal 2PORT=23333 SLURM_PROCID=1 SLURM_NTASKS=4 python main.py# Terminal 3PORT=23333 SLURM_PROCID=2 SLURM_NTASKS=4 python main.py# Terminal 4PORT=23333 SLURM_PROCID=3 SLURM_NTASKS=4 python main.py# Then, the training process will start after all 4 processes are launched.

By default, the program will not output anything once the training process starts, for the widely-usedtqdm might be buggy on slurm clusters. So, you may see some debugging info like the one below and the program seems to be stuck.

[1656218909.68] starting job... 0 of 1[1656218911.53] DDP Initialized at localhost:23333 0 of 1Comma2k19SequenceDataset: DEMO mode is on.Loaded pretrained weights for efficientnet-b2

Don't worry, you can open a tensorboard to see the loss and validation curves.

tensorboard --logdir runs --bind_all

Otherwise, you may want to parse--tqdm=True to show the progress bar inTerminal 1.

By default, the test process will be executed once every epoch. So we did not implement the independent test script.

See more demo and test cases on ourwebpage！

You can generate your own demo video usingdemo.py. It will generate some frames in the./vis folder. (You may have to create it first.) Then, you can generate a video usingffmpeg.

Here we list several baselines to perform trajectory prediction task on different datasets. You are welcome to pull request and add your work here!

nuScenes

Comma2k19

*: The lower, the better. To comparison, the average jerk of human driver's trajectories is 0.3232 m/s^2.

Please use the following citation when referencing our repo orarXiv.

@article{chen2022op,   title={Level 2 Autonomous Driving on a Single Device: Diving into the Devils of Openpilot},   author={Li Chen and Tutian Tang and Zhitian Cai and Yang Li and Penghao Wu and Hongyang Li and Jianping Shi and Junchi Yan and Yu Qiao},   journal={arXiv preprint arXiv:2206.08176},   year={2022}}

All code within this repository is underApache License 2.0.