Repository files navigation

This repository contains code to train and render Mixture of VolumetricPrimitives (MVP) models.

If you use Mixture of Volumetric Primitives in your research, please cite:
Mixture of Volumetric Primitives for Efficient Neural Rendering
Stephen Lombardi, Tomas Simon, Gabriel Schwartz, Michael Zollhoefer, Yaser Sheikh, Jason Saragih
ACM Transactions on Graphics (SIGGRAPH 2021) 40, 4. Article 59

@article{Lombardi21,author = {Lombardi, Stephen and Simon, Tomas and Schwartz, Gabriel and Zollhoefer, Michael and Sheikh, Yaser and Saragih, Jason},title = {Mixture of Volumetric Primitives for Efficient Neural Rendering},year = {2021},issue_date = {August 2021},publisher = {Association for Computing Machinery},address = {New York, NY, USA},volume = {40},number = {4},issn = {0730-0301},url = {https://doi.org/10.1145/3450626.3459863},doi = {10.1145/3450626.3459863},journal = {ACM Trans. Graph.},month = {jul},articleno = {59},numpages = {13},keywords = {neural rendering}}

• Python (3.8+)
  • PyTorch
  • NumPy
  • SciPy
  • Pillow
  • OpenCV
• ffmpeg (in $PATH to render videos)
• CUDA 10 or higher

The repository contains two CUDA PyTorch extensions. To build, cd to eachdirectory and usemake:

cd extensions/mvpraymarchermakecd -cd extensions/utilsmake

There are two main scripts in the root directory: train.py and render.py. Thescripts take a configuration file for the experiment that defines the datasetused and the options for the model (e.g., the type of decoder that is used).

Download the latest release on Github to get the experiments directory.

To train the model:

python train.py experiments/dryice1/experiment1/config.py

To render a video of a trained model:

python render.py experiments/dryice1/experiment1/config.py

See ARCHITECTURE.md for more details.

See the latest Github release for data.

Implement your own Dataset class to return images and camera parameters. Anexample is given in data.multiviewvideo. A dataset class will need to returncamera pose parameters, image data, and tracked mesh data.

See ARCHITECTURE.md

See the LICENSE file for details.