Repository files navigation

Use Unity 3D character and Python deep learning algorithms to stream as a VTuber!

This is part of theOpenVTuberProject, which provides many toolkits for becoming a VTuber.

Youtube Playlist (Chinese) (Covers videos 1-4):

First of all, I'd like to give credits to the following projects that I borrow code from:

And the virtual characterunity-chan © UTJ/UCL.

• OS: Ubuntu 16.04 (18.04 may also work) or Windows 10 64bits or MacOS
• (Optional but recommended) An NVIDIA GPU (tested with CUDA 9.0, 10.0 and 10.1, but may also work with other versions)

• Python3.x (installation viaAnaconda is recommended;mandatory for Windows users)

  • (Optional) It is recommended to use conda environments. Runconda create -n vtuber python=3.6. Activate it byconda activate vtuber.

• Python libraries

  • Ubuntu:
    • Install the requirements bypip install -r requirements_(cpu or gpu).txt
    • If you have CUDA 10.1,pip install onnxruntime-gpu to get faster inference speed using onnx model.
  • Windows:
    • CPU:
      • pip install -r requirements_cpu.txt
      • ifdlib cannot be properly installed, followhere.
    • GPU:
      • Installpytorch usingconda. Example:conda install pytorch==1.2.0 torchvision==0.4.0 cudatoolkit=10.0 -c pytorch
      • Install other dependencies bypip install -r requirements_gpu.txt.
      • If you have CUDA 10,pip install onnxruntime-gpu to get faster inference speed using onnx model.

• Optional

  • OBS Studio if you want to embed the virtual character into your videos.
  • Unity Editor if you want to customize the virtual character.
    • Linux installation
    • Windows installation

Here we assume that you have installed the requirements and activated the virtual environment you are using.

You need to download the modelshere, extract and put intoface_alignment/ckpts.

If you don't useonnxruntime, you can omit this step as the script will automatically download them for you.

Runpython demo.py --debug. (add--cpu if you have CPU only)

You should see the following:

^{Left: CPU model. Right: GPU model run on a GTX1080Ti.}

1. Download and launch the binarieshere depending on your OS to launch the unity window featuring the virtual character (unity-chan here).Important: Ensure that only one window is opened at a time!
2. After the vitual character shows up, runpython demo.py --connect to synchronize your face features with the virtual character. (add--debug to see your face and--cpu if you have CPU only as step 1.)

You should see the following:

^{Left: CPU model. Right: GPU model run on a GTX1080Ti.}

Enjoy your VTuber life!

In this section, I will describe the functionalities implemented and a little about the technology behind.

Usinghead-pose-estimation andface-alignment, deep learning methods are applied to do the following: face detection and facial landmark detection. A face bounding box and the 68-point facial landmark is detected, then a PnP algorithm is used to obtain the head pose (the rotation of the face). Finally, kalman filters are applied to the pose to make it smoother.

The character's head pose is synchronized.

As for the visualization, the white bounding box is the detected face, on top of which 68 green face landmarks are plotted. The head pose is represented by the green frustum and the axes in front of the nose.

UsingGazeTracking, The eyes are first extracted using the landmarks enclosing the eyes. Then the eye images are converted to grayscale, and a pixel intensity threshold is applied to detect the iris (the black part of the eye). Finally, the center of the iris is computed as the center of the black area.

The character's gaze is not synchronized. (Since I didn't find a way to move unity-chan's eyes)

As for the visualization, the red crosses indicate the iris.

1. Estimateeye aspect ratio: The eye aspect ratio can be used to detect blinking, but currently I just use auto blinking since this estimation is not so accurate.

2. Estimatemouth aspect ratio: I use this number to synchronize with the character's mouth.

3. The mouth distance is used to detect smile and synchronize with the character.

If you want to customize the virtual character, you can find the unity project inrelease.

MIT License