Repository files navigation

The last few years have seen a rise in novel differentiable graphics layerswhich can be inserted in neural network architectures. From spatial transformersto differentiable graphics renderers, these new layers leverage the knowledgeacquired over years of computer vision and graphics research to build new andmore efficient network architectures. Explicitly modeling geometric priors andconstraints into neural networks opens up the door to architectures that can betrained robustly, efficiently, and more importantly, in a self-supervisedfashion.

At a high level, a computer graphics pipeline requires a representation of 3Dobjects and their absolute positioning in the scene, a description of thematerial they are made of, lights and a camera. This scene description is theninterpreted by a renderer to generate a synthetic rendering.

In comparison, a computer vision system would start from an image and try toinfer the parameters of the scene. This allows the prediction of which objectsare in the scene, what materials they are made of, and their three-dimensionalposition and orientation.

Training machine learning systems capable of solving these complex 3D visiontasks most often requires large quantities of data. As labelling data is acostly and complex process, it is important to have mechanisms to design machinelearning models that can comprehend the three dimensional world while beingtrained without much supervision. Combining computer vision and computergraphics techniques provides a unique opportunity to leverage the vast amountsof readily available unlabelled data. As illustrated in the image below, thiscan, for instance, be achieved using analysis by synthesis where the visionsystem extracts the scene parameters and the graphics system renders back animage based on them. If the rendering matches the original image, the visionsystem has accurately extracted the scene parameters. In this setup, computervision and computer graphics go hand in hand, forming a single machine learningsystem similar to an autoencoder, which can be trained in a self-supervisedmanner.

Tensorflow Graphics is being developed to help tackle these types of challengesand to do so, it provides a set of differentiable graphics and geometry layers(e.g. cameras, reflectance models, spatial transformations, mesh convolutions)and 3D viewer functionalities (e.g. 3D TensorBoard) that can be used to trainand debug your machine learning models of choice.

See theinstalldocumentation for instructions on how to install TensorFlow Graphics.

You can find the API documentationhere.

TensorFlow Graphics is fully compatible with the latest stable release ofTensorFlow, tf-nightly, and tf-nightly-2.0-preview. All the functions arecompatible with graph and eager execution.

Tensorflow Graphics heavily relies on L2 normalized tensors, as well as havingthe inputs to specific function be in a pre-defined range. Checking for all ofthis takes cycles, and hence is not activated by default. It is recommended toturn these checks on during a couple epochs of training to make sure thateverything behaves as expected. Thispageprovides the instructions to enable these checks.

To help you get started with some of the functionalities provided by TFGraphics, some Colab notebooks are available below and roughly ordered bydifficulty. These Colabs touch upon a large range of topics including, objectpose estimation, interpolation, object materials, lighting, non-rigid surfacedeformation, spherical harmonics, and mesh convolutions.

NOTE: the tutorials are maintained carefully. However, they are not consideredpart of the API and they can change at any time without warning. It is notadvised to write code that takes dependency on them.

Object pose estimation	Camera intrinsics optimization

B-spline and slerp interpolation	Reflectance	Non-rigid surface deformation

Spherical harmonics rendering	Environment map optimization	Semantic mesh segmentation

Visual debugging is a great way to assess whether an experiment is going in theright direction. To this end, TensorFlow Graphics comes with a TensorBoardplugin to interactively visualize 3D meshes and point clouds.This demoshows how to use the plugin. Followthese instructionsto install and configure TensorBoard 3D. Note that TensorBoard 3D is currentlynot compatible with eager execution nor TensorFlow 2.

Among many things, we are hoping to release resamplers, additional 3Dconvolution and pooling operators, and a differentiable rasterizer!

Follow us onTwitter to hear about thelatest updates!

You may use this software under theApache 2.0 License.

As part of TensorFlow, we're committed to fostering an open and welcomingenvironment.

• Stack Overflow: Askor answer technical questions.
• GitHub: Report bugs or makefeature requests.
• TensorFlow Blog: Stay up to date on contentfrom the TensorFlow team and best articles from the community.
• Youtube Channel: Follow TensorFlow shows.

If you use TensorFlow Graphics in your research, please reference it as:

@inproceedings{TensorflowGraphicsIO2019,   author = {Oztireli, Cengiz and Valentin, Julien and Keskin, Cem and Pidlypenskyi, Pavel and Makadia, Ameesh and Sud, Avneesh and Bouaziz, Sofien},   title = {TensorFlow Graphics: Computer Graphics Meets Deep Learning},   year = {2019}}

Want to reach out? E-mail us attf-graphics-contact@google.com!

• Sofien Bouaziz (sofien@google.com)
• Jay Busch
• Forrester Cole
• Ambrus Csaszar
• Boyang Deng
• Ariel Gordon
• Christian Häne
• Cem Keskin
• Ameesh Makadia
• Cengiz Öztireli
• Rohit Pandey
• Romain Prévost
• Pavel Pidlypenskyi
• Stefan Popov
• Konstantinos Rematas
• Omar Sanseviero
• Aviv Segal
• Avneesh Sud
• Andrea Tagliasacchi
• Anastasia Tkach
• Julien Valentin
• He Wang
• Yinda Zhang