work_dirs/

tools/*.sh

-\[2024-03\] We first release the data and baselines for the challenge. Please fill in the[form](https://docs.google.com/forms/d/e/1FAIpQLScUXEDTksGiqHZp31j7Zp7zlCNV7p_08uViwP_Nbzfn3g6hhw/viewform?usp=sf_link) to apply for downloading the data and try our baselines. Welcome any feedback!

-\[2024-02\] We will co-organize[Autonomous Grand Challenge](https://opendrivelab.com/challenge2024/) in CVPR 2024. Welcome to try the Multi-View 3D Visual Grounding track! We will release more details about the challenge with the baseline after the Chinese New Year.

-\[2023-12\] We release the[paper](./assets/EmbodiedScan.pdf) of EmbodiedScan. Please check the[webpage](https://tai-wang.github.io/embodiedscan) and view our demos!

We will release the code for model training and benchmark with pretrained checkpoints in the 2024 Q1.

<palign="center">

We provide configs for different tasks[here](configs/) and you can run the train and test script in the[tools folder](tools/) for training and inference.

For example, to train a multi-view 3D detection model with pytorch, just run:

python tools/train.py configs/mv-det3d_8xb4_embodiedscan-3d-284class-9dof.py --work-dir=work_dirs/mv-3ddet --launcher="pytorch"

Or on the cluster with multiple machines, run the script with the slurm launcher following the sample script provided[here](tools/mv-grounding.sh).

NOTE: To run the multi-view 3D grounding experiments, please first download the 3D detection pretrained model to accelerate its training procedure. After downloading the detection checkpoint, please check the path used in the config, for example, the`load_from`[here](https://github.com/OpenRobotLab/EmbodiedScan/blob/main/configs/grounding/mv-grounding_8xb12_embodiedscan-vg-9dof.py#L210), is correct.

To inference and evaluate the model (e.g., the checkpoint`work_dirs/mv-3ddet/epoch_12.pth`), just run the test script:

python tools/test.py configs/mv-det3d_8xb4_embodiedscan-3d-284class-9dof.py work_dirs/mv-3ddet/epoch_12.pth --launcher="pytorch"

Please see the[paper](./assets/EmbodiedScan.pdf) for details of our two benchmarks, fundamental 3D perception and language-grounded benchmarks. This dataset is still scaling up and the benchmark is being polished and extended. Please stay tuned for our recent updates.

We preliminarily provide several baseline results here with their logs and pretrained models.

Note that the performance is a little different from the results provided in the paper because we re-split the training set as the released training and validation set while keeping the original validation set as the test set for the public benchmark.

| Method| Input| AP@0.25| AR@0.25| AP@0.5| AR@0.5| Download|

|:------:|:-----:|:-------:|:-------:|:------:|:------:|:------:|

|[Baseline](configs/detection/mv-det3d_8xb4_embodiedscan-3d-284class-9dof.py)| RGB-D| 15.22| 52.23| 8.13| 26.66|[Model](https://pjlab-my.sharepoint.cn/:u:/g/personal/wangtai_pjlab_org_cn/Efl363DOsXdAiikGcHIC3aQB_rjqHKsgxACyUgrHzqRmMA?e=XQDeY7)|[Log](https://pjlab-my.sharepoint.cn/:u:/g/personal/wangtai_pjlab_org_cn/ET5FOsjHqOBBsGs_WHeUIpUBY-iLXeWdYPNeWZ3nh9wbYg?e=1fsirH)|

| Method|AP@0.25| AP@0.5| Download|

|:------:|:-----:|:-------:|:------:|

|[Baseline-Mini](configs/grounding/mv-grounding_8xb12_embodiedscan-vg-9dof.py) | 33.59 | 14.40 |[Model](https://pjlab-my.sharepoint.cn/:u:/g/personal/wangtai_pjlab_org_cn/EbrSZQM6bLROuVdG5MgRhqABJH0Cs91vHE9B-PfjZXvE0w?e=D5wbIK) |[Log](https://pjlab-my.sharepoint.cn/:u:/g/personal/wangtai_pjlab_org_cn/EbhDHknA5nNMiBTiYZhnyCIBSZ881MJXUSfPgQGMm-spEw?e=Fp9fwZ)

|[Baseline-Mini (w/ FCAF box coder)](configs/grounding/mv-grounding_8xb12_embodiedscan-vg-9dof_fcaf-coder.py)| -| -| -|

|[Baseline-Full](configs/grounding/mv-grounding_8xb12_embodiedscan-vg-9dof-full.py)| -| -| -|

Please see the[paper](./assets/EmbodiedScan.pdf) for more details of our two benchmarks, fundamental 3D perception and language-grounded benchmarks. This dataset is still scaling up and the benchmark is being polished and extended. Please stay tuned for our recent updates.

-\[x\] Release the paper and partial codes for datasets.

-\[x\] Release EmbodiedScan annotation files.

-\[x\] Release partial codes for models and evaluation.

-\[\] Polish dataset APIs and related codes.

-\[\] Release Embodied Perceptron pretrained models.

-\[\] Release multi-modal datasets and codes.

-\[\] Release codes for baselines and benchmarks.

-\[x\] Release Embodied Perceptron pretrained models.

-\[x\] Release multi-modal datasets and codes.

-\[x\] Release codes for baselines and benchmarks.

-\[\] Full release and further updates.

If you find our work helpful, please cite:

set -x

CKPT_PATH=/mnt/petrelfs/wangtai/EmbodiedScan/work_dirs

PARTITION=test

JOB_NAME=mv-grounding-challenge-benchmark

TASK=mv-grounding-challenge-benchmark

CONFIG=configs/grounding/mv-grounding_8xb12_embodiedscan-vg-9dof.py

WORK_DIR=${CKPT_PATH}/${TASK}

CKPT=${CKPT_PATH}/${TASK}/latest.pth

CPUS_PER_TASK=16

GPUS=8

GPUS_PER_NODE=8

PORT=29320

PYTHONPATH="$(dirname$0)/..":$PYTHONPATH \

export NCCL_IB_DISABLE=1;export NCCL_P2P_DISABLE=1; \

srun -p${PARTITION} \

--job-name=${JOB_NAME} \

--gres=gpu:${GPUS_PER_NODE} \

--ntasks=${GPUS} \

--ntasks-per-node=${GPUS_PER_NODE} \

--cpus-per-task=${CPUS_PER_TASK} \

--kill-on-bad-exit=1 \

--quotatype=reserved \

python -u tools/train.py${CONFIG} --work-dir=${WORK_DIR} --launcher="slurm" --cfg-options env_cfg.dist_cfg.port=${PORT} --resume