nihui/rife-ncnn-vulkanPublic

NotificationsYou must be signed in to change notification settings
Fork86
Star941

RIFE, Real-Time Intermediate Flow Estimation for Video Frame Interpolation implemented with ncnn library

License

MIT license

941 stars 86 forks Branches Tags Activity

Star

Notifications

You must be signed in to change notification settings

Branches Tags

Folders and files

Name		Name	Last commit message	Last commit date
Latest commit History 49 Commits
.github		.github
images		images
models		models
src		src
.gitmodules		.gitmodules
LICENSE		LICENSE
README.md		README.md

Repository files navigation

RIFE ncnn Vulkan

ncnn implementation of RIFE, Real-Time Intermediate Flow Estimation for Video Frame Interpolation.

rife-ncnn-vulkan usesncnn project as the universal neural network inference framework.

Download

Download Windows/Linux/MacOS Executable for Intel/AMD/Nvidia GPU

https://github.com/nihui/rife-ncnn-vulkan/releases

This package includes all the binaries and models required. It is portable, so no CUDA or PyTorch runtime environment is needed :)

About RIFE

RIFE (Real-Time Intermediate Flow Estimation for Video Frame Interpolation)

https://github.com/hzwer/arXiv2020-RIFE

Huang, Zhewei and Zhang, Tianyuan and Heng, Wen and Shi, Boxin and Zhou, Shuchang

https://rife-vfi.github.io

https://arxiv.org/abs/2011.06294

Usages

Input two frame images, output one interpolated frame image.

Example Commands

./rife-ncnn-vulkan -0 0.jpg -1 1.jpg -o 01.jpg./rife-ncnn-vulkan -i input_frames/ -o output_frames/

Example below runs on CPU, Discrete GPU, and Integrated GPU all at the same time. Uses 2 threads for image decoding, 4 threads for one CPU worker, 4 threads for another CPU worker, 2 threads for discrete GPU, 1 thread for integrated GPU, and 4 threads for image encoding.

./rife-ncnn-vulkan -i input_frames/ -o output_frames/ -g -1,-1,0,1 -j 2:4,4,2,1:4

Video Interpolation with FFmpeg

mkdir input_framesmkdir output_frames# find the source fps and format with ffprobe, for example 24fps, AACffprobe input.mp4# extract audioffmpeg -i input.mp4 -vn -acodec copy audio.m4a# decode all framesffmpeg -i input.mp4 input_frames/frame_%08d.png# interpolate 2x frame count./rife-ncnn-vulkan -i input_frames -o output_frames# encode interpolated frames in 48fps with audioffmpeg -framerate 48 -i output_frames/%08d.png -i audio.m4a -c:a copy -crf 20 -c:v libx264 -pix_fmt yuv420p output.mp4

Full Usages

Usage: rife-ncnn-vulkan -0 infile -1 infile1 -o outfile [options]...       rife-ncnn-vulkan -i indir -o outdir [options]...  -h                   show this help  -v                   verbose output  -0 input0-path       input image0 path (jpg/png/webp)  -1 input1-path       input image1 path (jpg/png/webp)  -i input-path        input image directory (jpg/png/webp)  -o output-path       output image path (jpg/png/webp) or directory  -n num-frame         target frame count (default=N*2)  -s time-step         time step (0~1, default=0.5)  -m model-path        rife model path (default=rife-v2.3)  -g gpu-id            gpu device to use (-1=cpu, default=auto) can be 0,1,2 for multi-gpu  -j load:proc:save    thread count for load/proc/save (default=1:2:2) can be 1:2,2,2:2 for multi-gpu  -x                   enable spatial tta mode  -z                   enable temporal tta mode  -u                   enable UHD mode  -f pattern-format    output image filename pattern format (%08d.jpg/png/webp, default=ext/%08d.png)

input0-path,input1-path andoutput-path accept file path
input-path andoutput-path accept file directory
num-frame = target frame count
time-step = interpolation time
load:proc:save = thread count for the three stages (image decoding + rife interpolation + image encoding), using larger values may increase GPU usage and consume more GPU memory. You can tune this configuration with "4:4:4" for many small-size images, and "2:2:2" for large-size images. The default setting usually works fine for most situations. If you find that your GPU is hungry, try increasing thread count to achieve faster processing.
pattern-format = the filename pattern and format of the image to be output, png is better supported, however webp generally yields smaller file sizes, both are losslessly encoded

If you encounter a crash or error, try upgrading your GPU driver:

Build from Source

Download and setup the Vulkan SDK fromhttps://vulkan.lunarg.com/

For Linux distributions, you can either get the essential build requirements from package manager

dnf install vulkan-headers vulkan-loader-devel

apt-get install libvulkan-dev

pacman -S vulkan-headers vulkan-icd-loader

Clone this project with all submodules

git clone https://github.com/nihui/rife-ncnn-vulkan.gitcd rife-ncnn-vulkangit submodule update --init --recursive

Build with CMake

You can pass -DUSE_STATIC_MOLTENVK=ON option to avoid linking the vulkan loader library on MacOS

mkdir buildcd buildcmake ../srccmake --build. -j 4

Model

model	upstream version
rife	1.2
rife-HD	1.5
rife-UHD	1.6
rife-anime	1.8
rife-v2	2.0
rife-v2.3	2.3
rife-v2.4	2.4
rife-v3.0	3.0
rife-v3.1	3.1
rife-v4	4.0
rife-v4.6	4.6

Sample Images

Original Image

Interpolate with rife rife-anime model

rife-ncnn-vulkan.exe -m models/rife-anime -0 0.png -1 1.png -o out.png

Interpolate with rife rife-anime model + TTA-s

rife-ncnn-vulkan.exe -m models/rife-anime -x -0 0.png -1 1.png -o out.png

Original RIFE Project

https://github.com/hzwer/arXiv2020-RIFE

Other Open-Source Code Used

https://github.com/Tencent/ncnn for fast neural network inference on ALL PLATFORMS
https://github.com/webmproject/libwebp for encoding and decoding Webp images on ALL PLATFORMS
https://github.com/nothings/stb for decoding and encoding image on Linux / MacOS
https://github.com/tronkko/dirent for listing files in directory on Windows