Repository files navigation

This project implements a 3D human skinning model - SMPL: A SkinnedMulti-Person Linear Model with C++. The official SMPL model is available athttp://smpl.is.tue.mpg.de.

The author-provided implementation based onChumpy andOpenDR contains spaghetti code, and it cannot run on GPUs yet. I convert and update another Tensorflowversion of SMPL contributed byCalciferZh toC++ style.You can find the Tensorflow implementationhere.However, Tensorflow C++ APIs are not user-friendly, so I choose the PytorchC++ APIs - libTorch - instead.

For more details, see thepaperpublished by Max Planck Institute for Intelligent Systems on SIGGRAPH ASIA2015.

I have tested the codes on my machine, but I'm not sure the performance on other environments.

• GPU

  NVIDIA GeForce GTX 960M

• OS

  Ubuntu 18.04 LTS

• Packages

1. xtensor: A C++ library meant for numerical analysis with multi-dimensional array expressions.

   A C++ interpretation ofNumpy, you can even find functions with similar names in it. A [cheat sheet (https://xtensor.readthedocs.io/en/latest/) fromNumpy toXtensor is helpful.

   Currently, I only useXtensor as an IO interface for module testing with random inputs and restoring hyperparameters in JSON format. Share the buffer of anXtensor array with a correspondingPyTorch tensor is straightforward.

2. nlohmann_json: JSON for Modern C++.

   Xtensor loads data from and dumps data into JSONs through nlohmann's toolkit.

3. libTorch: Pytorch C++ API.

   PyTorch C++ API simplifies tensor computing and introduces GPUacceleration to this work, using CUDA and cuDNN.

   Note: I installed the nightly version oflibTorch with CUDA 10.0 support.

4. CUDA: NVIDIA parallelcomputing platform.

   CUDA 10.0 works well on my machine. I think the other versions fit thelibTorch download list should work as well.

5. CMake: A tool to build, test and pack upC++ program.

   The CMake installed byapt-get is CMake 3.5.1 which causes afailure whenlibTorch tries to find CUDA. A description of the issue:https://discuss.pytorch.org/t/install-libtorch-error-pytorch-c-api/26756/2

   You should update it to a newer version, such as 3.13.4 (>=3.12.2 should work). Delete the old CMake completely, download the latest official source codes, and build it from scratch.

• Package Installation

  Here are procedures to install packages into the root directory correctly. If you want to link libraries manually, skip this part and install the following packages in the usual ways — official documentation and Google helps a lot.

  • Source Code

    Compile libraries and headers into the root directory, e.g.,Xtensor:

  1. create a directory in the Xtensor repo you have just cloned from Github to build the package.

      cd <xtensor-dir> mkdir build cd build

  2. configure CMake and generate Makefiles. Remember to redirect theinstallation directory to the root.

      cmake -D CMAKE_INSTALL_PREFIX=/usr/local ..

     The "/usr/local" can be replaced with any other locations as long as CMake can find the package automatically.

  3. compile and install.

      make -j#X# sudo make install

     #X# is the number of threads specified by yourself.

  • Binary Library

    Move pre-built packages into the root directory, e.g.,libTorch:

  1. redirect into thelibTorch directory.

      cd <libTorch-dir>

  2. copyCMake configurations to the "lib" directory.

      cp -rv share/cmake lib

  3. copy headers and libraries to the root directory.

      cp -rv include lib /usr/local

     Afterward, you won't need to specify the path to thelibTorch library manually whenever buildinglibTorch-dependent projects withCMake. The operations are a little bit different from the official guidance.

• Data preprocessing

  Download and extract the official data fromhttp://smpl.is.tue.mpg.de/, youwill get two files:

  basicModel_f_lbs_10_207_0_v1.0.0.pklbasicmodel_m_lbs_10_207_0_v1.0.0.pkl

  Runpreprocess.py inSMPL/scripts with Python 2 to convert the data into.json and. npz format. If you change the location of these data, remember to edit themain.cpp as well since I have hardcoded the paths.

  Note: If you have a strange mesh visualization, remember to double-check the face indices. (MeshLab favors the face indices starting from 1. However, this rule may not be compatible with other software.)

• Build and Run

  After installing all packages, you can compile SMPL++ from source:

  mkdir buildcd buildcmake ..make

  These commands produce an executable program named as "smplpp". To run it, just type

  ./smplpp

  in the terminal.

  To track the usage of GPUs, use the following command:

  nvidia-smi -lms

Follow these steps:#5 (comment).

The project is simply a raw framework for SMPL++. I have written many comments in the source codes, but there may be some typoes. Sorry about that.

• Forward Propagation

  SMPL++ implements a part of the algorithm described in the paper. The inputs of the system are shape coefficients \beta, pose axis-angle parameterization \theta, and body translation \vec{t}. Each of them controls a trait of the body mesh.

  

  Note: Backward propagation has not been available yet.

• Render Meshes

  I don't have a GUI to render the output now! If you would like to see themeshes, try to render them inMeshLab.

• Pipeline

  Following the paper, we can generate a mesh with four steps.

  1. Generate pose blend shape and shape blend shape.

  2. Regress joints from vertices.

  3. Compute the transformation matrices for each joint.

  4. Linear Blend Skinning

• Kinematic Tree

  An kinematic tree for SMPL looks like this:

  

• Hyperparameters restore from.npz files instead of.json files.(.json neither saves storage nor performances efficiently when beingimported.)

  Update: I have uploaded a script to convert data from official.pkl, check the fold SMPL++/scripts for more details.

• A OpenGL GUI to render and manipulate the 3D mesh.

• Fit the 3D mesh to a 2D image - SMPLify.

• Export SMPL++ into static or dynamic library.

• A trainable SMPL.

Note: The importance of each demand decreases in this list.

If you find any problem, error, or even typo, feel free to contact me.

SMPL++ is for research purposes only. Any commercial usage should be allowed by original authors.

[1] Official Website of SMPL:http://smpl.is.tue.mpg.de.

[2] Official Website of SMPLify:http://smplify.is.tue.mpg.de.