- Notifications
You must be signed in to change notification settings - Fork34
NeurIPS-2021: Direct Multi-view Multi-person 3D Human Pose Estimation
License
sail-sg/mvp
Folders and files
| Name | Name | Last commit message | Last commit date | |
|---|---|---|---|---|
Repository files navigation
This is the official implementation of our NeurIPS-2021 work: Multi-view Pose Transformer (MvP).MvP is a simple algorithm that directly regresses multi-person 3D human pose from multi-view images.
⭐⭐⭐[News] A Re-implementation is integrated into xrmocap: athttps://github.com/openxrlab/xrmocap
@article{wang2021mvp, title={Direct Multi-view Multi-person 3D Human Pose Estimation}, author={Tao Wang and Jianfeng Zhang and Yujun Cai and Shuicheng Yan and Jiashi Feng}, journal={Advances in Neural Information Processing Systems}, year={2021}}- Set the project root directory as ${POSE_ROOT}.
- Install all the required python packages (with requirements.txt).
- compile deformable operation for projective attention.
cd ./models/opssh ./make.shPlease followVoxelPose to downloadthe CMU Panoptic Dataset and PoseResNet-50 pre-trained model.
The directory tree should look like this:
${POSE_ROOT}|-- models| |-- pose_resnet50_panoptic.pth.tar|-- data| |-- panoptic| | |-- 16060224_haggling1| | | |-- hdImgs| | | |-- hdvideos| | | |-- hdPose3d_stage1_coco19| | | |-- calibration_160224_haggling1.json| | |-- 160226_haggling1| | |-- ...Please followVoxelPose to downloadthe Shelf/Campus Dataset.
Due to the limited and incomplete annotations of the two datasets, we use psudoground truth 3D pose generated from VoxelPose to train the model, we expect mvp wouldperform much better with absolute ground truth pose data.
Please use voxelpose or other methods to generate psudo ground truth for the training set,you can also use our generated psudo GT:psudo_gt_shelf.psudo_gt_campus.psudo_gt_campus_fix_gtmorethanpred.
Due to the small dataset size, we fine-tune Panoptic pre-trained model to Shelf and Campus.Download the pretrained MvP on Panoptic frommodel_best_5view andmodel_best_3view_horizontal_view ormodel_best_3view_2horizon_1lookdown
The directory tree should look like this:
${POSE_ROOT}|-- models| |-- model_best_5view.pth.tar| |-- model_best_3view_horizontal_view.pth.tar| |-- model_best_3view_2horizon_1lookdown.pth.tar|-- data| |-- Shelf| | |-- Camera0| | |-- ...| | |-- Camera4| | |-- actorsGT.mat| | |-- calibration_shelf.json| | |-- pesudo_gt| | | |-- voxelpose_pesudo_gt_shelf.pickle| |-- CampusSeq1| | |-- Camera0| | |-- Camera1| | |-- Camera2| | |-- actorsGT.mat| | |-- calibration_campus.json| | |-- pesudo_gt| | | |-- voxelpose_pesudo_gt_campus.pickle| | | |-- voxelpose_pesudo_gt_campus_fix_gtmorethanpred_case.picklePlease followCHUNYUWANG/H36M-Toolbox to prepare the data.
The data and pre-trained model directory tree should look like this, you can only downloadthe Panoptic dataset and PoseResNet-50 for reproducing the main MvP result and ablation studies:
${POSE_ROOT}|-- models| |-- pose_resnet50_panoptic.pth.tar| |-- model_best_5view.pth.tar| |-- model_best_3view_horizontal_view.pth.tar| |-- model_best_3view_2horizon_1lookdown.pth.tar|-- data| |-- pesudo_gt| | |-- voxelpose_pesudo_gt_shelf.pickle| | |-- voxelpose_pesudo_gt_campus.pickle| | |-- voxelpose_pesudo_gt_campus_fix_gtmorethanpred_case.pickle| |-- panoptic| | |-- 16060224_haggling1| | | |-- hdImgs| | | |-- hdvideos| | | |-- hdPose3d_stage1_coco19| | | |-- calibration_160224_haggling1.json| | |-- 160226_haggling1| | |-- ...| |-- Shelf| | |-- Camera0| | |-- ...| | |-- Camera4| | |-- actorsGT.mat| | |-- calibration_shelf.json| | |-- pesudo_gt| | | |-- voxelpose_pesudo_gt_shelf.pickle| |-- CampusSeq1| | |-- Camera0| | |-- Camera1| | |-- Camera2| | |-- actorsGT.mat| | |-- calibration_campus.json| | |-- pesudo_gt| | | |-- voxelpose_pesudo_gt_campus.pickle| | | |-- voxelpose_pesudo_gt_campus_fix_gtmorethanpred_case.pickle| |-- HM36The evaluation result will be printed after every epoch, the best result can be found in the log.
We train and validate on the five selected camera views. We trained our models on 8 GPUs and batch_size=1 for each GPU, note the total iteration per epoch should be3205, if not, please check your data.
python -m torch.distributed.launch --nproc_per_node=8 --use_env run/train_3d.py --cfg configs/panoptic/best_model_config.yaml| Datasets | AP25 | AP25 | AP25 | AP25 | MPJPE | pth |
|---|---|---|---|---|---|---|
| Panoptic | 92.3 | 96.6 | 97.5 | 97.7 | 15.8 | here |
You can find several ablation experiment configs under./configs/panoptic/, for example, removing RayConv:
python -m torch.distributed.launch --nproc_per_node=8 --use_env run/train_3d.py --cfg configs/panoptic/ablation_remove_rayconv.yamlAs shelf/campus are very small dataset with incomplete annotation, we finetune pretrained MvP with pseudo ground truth 3D pose extracted with VoxelPose, we expect more accurate GT would help MvP achieve much higher performance.
python -m torch.distributed.launch --nproc_per_node=8 --use_env run/train_3d.py --cfg configs/shelf/mvp_shelf.yaml| Datasets | Actor 1 | Actor 2 | Actor 2 | Average | pth |
|---|---|---|---|---|---|
| Shelf | 99.3 | 95.1 | 97.8 | 97.4 | here |
| Campus | 98.2 | 94.1 | 97.4 | 96.6 | here |
MvP also applies to the naive single-person setting, with dataset like Human3.6, to come
python -m torch.distributed.launch --nproc_per_node=8 --use_env run/train_3d.py --cfg configs/h36m/mvp_h36m.yamlTo evaluate a trained model, pass the config and model pth:
python -m torch.distributed.launch --nproc_per_node=8 --use_env run/validate_3d.py --cfg xxx --model_path xxxThis repo is under the Apache-2.0 license. For commercial use, please contact the authors.