Repository files navigation

Seamlessly generate high-quality 3D meshes from 2D images and retexture them using text-based texture generation directly in Unity.

This project integratesTripoSR, a "a state-of-the-art open-source model for fast feedforward 3D reconstruction from a single image" by StabilityAI and TripoAI, andtriposr-texture-gen directly into Unity Editor.

• Transformation of 2D images into textured 3D meshes, available both in Editor and in Playmode.
• Text-based texture generation, applied directly to the generated 3D meshes.

The generated 3D meshes are imported using a modified vertex color importer (based onAndrew Raphael Lukasik's importer) and auto-assigned to a base Material with custom shader to utilize and display the vertex colors correctly (without surface normals, though).

Tested with Unity 2022.3.22f1 running on Windows 11 and Ubuntu 22.04.

The first demo clip shows the creation of a 3D mesh based on a 2D Texture within the Editor.

The generated meshes are colored (vertex colors), react to light, and (optionally) automatically useMeshCollider andRigidbody for physics interaction.

This clip shows the text-based retexturing of a TripoSR-generated 3D mesh:

This demo clip shows 3D mesh generation in Playmode:

0. Download the latest release.unitypackage and import it into your project (Assets > Import Package).
1. cd into theAssets folder of your Unity project (using Command Prompt, Terminal, ...) and clone the latest repo of TripoSR:git clone https://github.com/VAST-AI-Research/TripoSR.git.
2. After cloning TripoSR,cd into the TripoSR folder that you just created by cloning the repo.
3. Runpip install --upgrade setuptools,pip install torch (in case you don't have PyTorch installed) andpip install -r requirements.txt.
4. Add theTripoSR Prefab (found inAssets > Prefabs) to your scene.
5. Configure the path to your python executable in theTripoSR GameObject within your scene: For Windows, runwhere python within Command Prompt. For Unix, runwhich python within Terminal.
6. Configure the other public variables in the Inspector as needed.

0. Clone this repo.
1. Ensure you have Python installed on your system.
2. Runpip install --upgrade setuptools,pip install torch (in case you don't have PyTorch installed) andpip install -r requirements.txt from within this project'sAssets/TripoSR folder.
3. In Unity, add theTripoSR Prefab to the scene (or simply open up mySampleScene).
4. Configure the path to your python executable in theTripoSR GameObject within your scene: For Windows, runwhere python within Command Prompt. For Unix, runwhich python within Terminal.
5. Configure the other public variables in the Inspector as needed.

I've adapted thetext2texture.py script fromtriposr-texture-gen, so it doesn't open up the 3D viewer and works in-situ. When you clone the repo on your own, this won't be included.

0. Follow the general setup for TripoSR from above
1. cd into theAssets folder and clone the latesttriposr-texture-gen.
2. Runpip install -r -requirements.txt from within theAssets/triposr-texture-gen directory.
3. Runpip install huggingface-hub --upgrade to fix some compatibility problems.
4. Add theTextureGenerator Prefab to your scene.
5. Configure the path to your python executable.
6. Configure the public vars to your liking.

0. Follow the general setup from above
1. Runpip install -r -requirements.txt from within theAssets/triposr-texture-gen directory.
2. Runpip install huggingface-hub --upgrade to fix some compatibility problems.
3. Add theTextureGenerator Prefab to your scene (or use mySampleScene).
4. Configure the path to your python executable.
5. Configure the public vars to your liking.

Once you have set up your scene with theTripoSRForUnity component and configured the parameters, you can run the process by clicking theRun TripoSR button in the inspector.

When you run TripoSR for the first time, the model weights will be downloaded and cached - this only occurs once; subsequent runs use the cached model.

• autoAddMesh: When enabled, the generated mesh is automatically added to the Unity scene as GameObject.
• autoAddPhysicsComponents: Whether or not to automatically add physics components (i.e., convexMeshCollider andRigidbody) to the generated mesh.
• autoFixRotation: If enabled, will automatically correct the wrong object rotation after adding the mesh to the scene.
• moveAndRename: Moves and renames the output.obj file based on the input image's filename if enabled.
• moveAndRenamePath: The directory to which the.obj file will be moved. Must start withAssets/.
• showDebugLogs: Enables the display of debug outputs from therun.py script in the Unity console.

All TripoSR parameters are exposed by my script. Feel free to change them as you see fit.

I've made some of themReadOnly within the Inspector, since you shouldn't really change those vars (e.g. model name, device to use). You can still change them within the script of course.

Likewise, you can run the texture generator by clicking onGenerate Textured Object within theTextureGenerator-component in the inspector.

When you run the texture generator for the first time, model weights will also be downloaded and cached, which also only occurs the first time.

Set the model file in the inspector to your.obj fromAssets > Models, describe the design you would like as model description, and start the generator.

TripoSR.objs only consist ofvs andfs, surface normals are not calculated. When Unity calculates the normals upon import, they are not smoothed correctly (even when using high smoothing angles).

For now, I've disabled normal calculation - feel free to create a PR if you know how to correctly handle this issue.

This project is licensed under the MIT License. SeeLICENSE for more information.

Special thanks to StabilityAI, TripoAI, and contributors of the TripoSR project.

Please cite their work adequately in case you use it in your own publications:

@article{TripoSR2024,title={TripoSR: Fast 3D Object Reconstruction from a Single Image},author={Tochilkin, Dmitry and Pankratz, David and Liu, Zexiang and Huang, Zixuan and and Letts, Adam and Li, Yangguang and Liang, Ding and Laforte, Christian and Jampani, Varun and Cao, Yan-Pei},journal={arXiv preprint arXiv:2403.02151},year={2024}}

Also special thanks to @ejones fortriposr-texture-gen.