Repository files navigation

This repository contains code for the following paper:

MeTRAbs: Metric-Scale Truncation-Robust Heatmaps for Absolute 3D Human Pose Estimation
by István Sárándi, Timm Linder, Kai O. Arras, Bastian Leibe
IEEE Transactions on Biometrics, Behavior, and Identity Science (T-BIOM), Selected Best Works FromAutomated Face and Gesture Recognition 2020.

The repo has been updated to an improved version employed in the following paper:

Learning 3D Human Pose Estimation from Dozens of Datasets using a Geometry-Aware Autoencoder to Bridge Between Skeleton Formats
by István Sárándi, Alexander Hermans, Bastian Leibe
IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), 2023.

• [2023-08-02] Major codebase refactoring, models as described in ourWACV'23 paper, several components factored out into separate repos, PyTorch support for inference, and more.
• [2021-12-03] Added new backbones, including the ResNet family from ResNet-18 to ResNet-152
• [2021-10-19] Released new best-performingmodels based on EfficientNetV2 and super fastones using MobileNetV3, simplifiedAPI, multiple skeleton conventions, support forradial/tangential distortion, improved antialiasing, plausibility filtering and other newfeatures.
• [2021-10-19] Full codebase migrated to TensorFlow 2 and Keras
• [2020-11-19] Oral presentation at the IEEE Conference on Automatic Face and Gesture Recognition(FG'20) (Talk VideoandSlides)
• [2020-11-16] Training and evaluation code now released along with dataset pre-processing scripts!Code and models upgraded to Tensorflow 2.
• [2020-10-06]Journal paper accepted for publication in theIEEE Transactions on Biometrics, Behavior, and Identity Science (T-BIOM), Best of FG Special Issue
• [2020-08-23] Short presentation at ECCV2020's 3DPWworkshop (slides)
• [2020-08-06] Our method has wonthe3DPW Challenge

We releasestandalone TensorFlow models (SavedModel) to allow easy application in downstreamresearch. After loading the model, you can run inference in a single line of Pythonwithout havingthis codebase as a dependency. Try it in action inGoogle Colab.

importtensorflowastfimporttensorflow_hubastfhubmodel=tfhub.load('https://bit.ly/metrabs_l')image=tf.image.decode_jpeg(tf.io.read_file('img/test_image_3dpw.jpg'))pred=model.detect_poses(image)pred['boxes'],pred['poses2d'],pred['poses3d']

See also thedemos folder for more examples.

NOTE: The models can only be used fornon-commercial purposes due to the licensing of the usedtraining datasets.

Alternatively, you can try the experimental PyTorch version:

wget -O - https://bit.ly/metrabs_l_pt| tar -xzvf -python -m metrabs_pytorch.scripts.demo_image --model-dir metrabs_eff2l_384px_800k_28ds_pytorch --image img/test_image_3dpw.jpg

• ./demo.py to auto-download the model, predict on a sample image and display theresult with Matplotlib orPoseViz (if installed).
• ./demo_video.py filepath-or-url-to-video.mp4 to run inference on a video.

• How-to Guide with Examples
• Full API Reference

• Several skeleton conventions supported through the keyword argumentskeleton (e.g. COCO,SMPL, H36M)
• Multi-image (batched) and single-image predictions both supported
• Advanced, parallelized cropping logic behind the scenes
  • Anti-aliasing through image pyramid andsupersampling,gamma-correct rescaling.
  • GPU-accelerated undistortion of pinhole perspective (homography) and radial/tangential lensdistortions
• Estimates returned in3D world space (when calibration is provided) and2D pixel space
• Built-in, configurabletest-time augmentation (TTA) with rotation, flip and brightness (keywordargumentnum_aug sets the number of TTA crops per detection)
• Automaticsuppression of implausible poses and non-max suppression on the 3D pose level (can be turned off)
• Multiple backbones with different speed-accuracy trade-off (EfficientNetV2, MobileNetV3)

See the docs directory.

If you find this work useful in your research, please cite it as:

@article{sarandi2021metrabs,title={{MeTRAbs:} Metric-Scale Truncation-Robust Heatmaps for Absolute 3{D} Human Pose Estimation},author={S\'ar\'andi, Istv\'an and Linder, Timm and Arras, Kai O. and Leibe, Bastian},journal={IEEE Transactions on Biometrics, Behavior, and Identity Science},year={2021},volume={3},number={1},pages={16-30},doi={10.1109/TBIOM.2020.3037257}}

The above paper is an extended journal version of the FG'2020 conference paper:

@inproceedings{Sarandi20FG,title={Metric-Scale Truncation-Robust Heatmaps for 3{D} Human Pose Estimation},author={S\'ar\'andi, Istv\'an and Linder, Timm and Arras, Kai O. and Leibe, Bastian},booktitle={IEEE International Conference on Automatic Face and Gesture Recognition},pages={677-684},year={2020}}

The newer large-scale models correspond to the WACV'23 paper:

@inproceedings{Sarandi2023dozens,author ={S\'ar\'andi, Istv\'an and Hermans, Alexander and Leibe, Bastian},title ={Learning {3D} Human Pose Estimation from Dozens of Datasets using a Geometry-Aware Autoencoder to Bridge Between Skeleton Formats},booktitle ={IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)},year ={2023}}

Code in this repository was written byIstván Sárándi (RWTH AachenUniversity) unless indicated otherwise.

Got any questions or feedback? Drop a mail tosarandi@vision.rwth-aachen.de!