Repository files navigation

The network structure of LiteFlowNet. For the ease of representation, only a 3-level design is shown.

A cascaded flow inference module M:S in NetE.

This repository (https://github.com/twhui/LiteFlowNet) is the offical release ofLiteFlowNet for my paperLiteFlowNet: A Lightweight Convolutional Neural Network for Optical Flow Estimation in CVPR 2018 (Spotlight paper, 6.6%).The up-to-date version of the paper is available onarXiv.

LiteFlowNet is a lightweight, fast, and accurate opitcal flow CNN. We develop several specialized modules including (1) pyramidal features, (2) cascaded flow inference (cost volume + sub-pixel refinement), (3) feature warping (f-warp) layer, and (4) flow regularization by feature-driven local convolution (f-lconv) layer. LiteFlowNet outperforms PWC-Net (CVPR 2018) on KITTI and has a smaller model size (less than PWC-Net by ~40%). For more details about LiteFlowNet, you may visitmy project page.

Oral presentation at CVPR 2018 is also available onYouTube.

NEW! Our extended work (LiteFlowNet2, TPAMI 2020) is now available athttps://github.com/twhui/LiteFlowNet2.

LiteFlowNet2 in TPAMI 2020, another lightweight convolutional network, is evolved from LiteFlowNet (CVPR 2018) to better address the problem of optical flow estimation by improving flow accuracy and computation time. Comparing to our earlier work, LiteFlowNet2 improves the optical flow accuracy on Sintel clean pass by 23.3%, Sintel final pass by 12.8%, KITTI 2012 by 19.6%, and KITTI 2015 by 18.8%. Its runtime is 2.2 times faster!

Note: *Runtime is averaged over 100 runs for a Sintel's image pair of size 1024 × 436.

NEW! Our extended work (LiteFlowNet3, ECCV 2020) is now available athttps://github.com/twhui/LiteFlowNet3.

We ameliorate the issue of outliers in the cost volume by amending each cost vector through an adaptive modulation prior to the flow decoding. We further improve the flow accuracy by exploring local flow consistency. To this end, each inaccurate optical flow is replaced with an accurate one from a nearby position through a novel warpingof the flow field. LiteFlowNet3 not only achieves promising results on public benchmarks but also has a small model size and a fast runtime.

Note: *Runtime is averaged over 100 runs for a Sintel's image pair of size 1024 × 436.

This software and associated documentation files (the "Software"), and the research paper (LiteFlowNet: A Lightweight Convolutional Neural Network for Optical Flow Estimation) including but not limited to the figures, and tables (the "Paper") are provided for academic research purposes only and without any warranty. Any commercial use requires my consent. When using any parts of the Software or the Paper in your work, please cite the following paper:

@InProceedings{hui18liteflownet,     author = {Tak-Wai Hui and Xiaoou Tang and Chen Change Loy},     title = {{LiteFlowNet: A Lightweight Convolutional Neural Network for Optical Flow Estimation}},     booktitle = {{Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR)}},     year = {2018},   pages = {8981--8989}, url = {http://mmlab.ie.cuhk.edu.hk/projects/LiteFlowNet/} }

1. FlyingChairs dataset (31GB) andtrain-validation split.
2. RGB image pairs (clean pass) (37GB) and flow fields (311GB) for Things3D dataset.
3. Sintel dataset (clean + final passes) (5.3GB).
4. KITTI12 dataset (2GB) and KITTI15 dataset (2GB) (Simple registration is required).

The code package comes as the modified Caffe fromDispFlowNet andFlowNet2 with our new layers, scripts, and trained models.

Reimplementations inPytorch and TensorFlow are also available.

Installation was tested under Ubuntu 14.04.5/16.04.2 with CUDA 8.0, cuDNN 5.1 and openCV 2.4.8/3.1.0.

Edit Makefile.config (and Makefile) if necessary in order to fit your machine's settings.

For openCV 3+, you may need to changeopencv2/gpu/gpu.hpp toopencv2/cudaarithm.hpp in/src/caffe/layers/resample_layer.cu.

If your machine installed a newer version of cuDNN, you do not need to downgrade it. You can do the following trick:

1. Downloadcudnn-8.0-linux-x64-v5.1.tgz and untar it to a temp folder, saycuda-8-cudnn-5.1

2. Renamecudnn.h tocudnn-5.1.h in the folder/cuda-8-cudnn-5.1/include

3. $ sudo cp cuda-8-cudnn-5.1/include/cudnn-5.1.h /usr/local/cuda/include/$ sudo cp cuda-8-cudnn-5.1/lib64/lib* /usr/local/cuda/lib64/

4. Replace#include <cudnn.h> to#include <cudnn-5.1.h> in/include/caffe/util/cudnn.hpp.

$ cd LiteFlowNet$ make -j 8 tools pycaffe

The source files include/src/caffe/layers/warp_layer.cpp,/src/caffe/layers/warp_layer.cu, and/include/caffe/layers/warp_layer.hpp.

The grid pattern that is used by f-warp layer is generated by a grid layer. The source files include/src/caffe/layers/grid_layer.cpp and/include/caffe/layers/grid_layer.hpp.

It is implemented using off-the-shelf components. More details can be found in/models/testing/depoly.prototxt or/models/training_template/train.prototxt.template by locating the code segmentNetE-R.

Two custom layers (ExpMax andNegSquare) are optimized in speed for forward-pass. Generalized Charbonnier loss is implemented in l1loss_layer. The power factor (alpha) can be adjusted inl1_loss_param { power: alpha l2_per_location: true }.

1. Prepare the training set. In/data/make-lmdbs-train.sh, changeYOUR_TRAINING_SET andYOUR_TESTING_SET to your favourite dataset.

$ cd LiteFlowNet/data$ ./make-lmdbs-train.sh

2. Copy files from/models/training_template to a new model folder (e.g.NEW). Edit all the files and make sure the settings are correct for your application. Model for the complete network is provided. LiteFlowNet uses stage-wise training to boost the performance. Please refer tomy paper for more details.

$ mkdir LiteFlowNet/models/NEW$ cd LiteFlowNet/models/NEW$ cp ../training_template/solver.prototxt.template solver.prototxt$ cp ../training_template/train.prototxt.template train.prototxt$ cp ../training_template/train.py.template train.py

3. Create a soft link in your new model folder

$ ln -s ../../build/tools bin

4. Run the training script

$ ./train.py -gpu 0 2>&1 | tee ./log.txt

The trained models (liteflownet,liteflownet-ft-sintel,liteflownet-ft-kitti) are available in the folder/models/trained. Untar the files to the same folder before you use it.

liteflownet: Trained on Chairs and then fine-tuned on Things3D.

liteflownet-ft-sintel: Model used for Sintel benchmark.

liteflownet-ft-kitti: Model used for KITTI benchmark.

1. Open the testing folder

$ cd LiteFlowNet/models/testing

2. Create a soft link in the folder/testing

$ ln -s ../../build/tools bin

3. ReplaceMODE in./test_MODE.py tobatch if all the images has the same resolution (e.g. Sintel dataset), otherwise replace it toiter (e.g. KITTI dataset).

4. ReplaceMODEL in lines 9 and 10 oftest_MODE.py to one of the trained models (e.g.liteflownet-ft-sintel).

5. Run the testing script. Flow fields (MODEL-0000000.flo,MODEL-0000001.flo, ... etc) are stored in the folder/testing/results having the same order as the image pair sequence.

$ test_MODE.py img1_pathList.txt img2_pathList.txt results

Average end-point error can be computed using the provided script/models/testing/util/endPointErr.m

1. A PyTorch-based reimplementation of LiteFlowNet is available athttps://github.com/sniklaus/pytorch-liteflownet.
2. A TensorFlow-based reimplementation of LiteFlowNet is also available athttps://github.com/keeper121/liteflownet-tf2.

The early version of LiteFlowNet was submitted to ICCV 2017 for review in March 2017. The improved work was published inCVPR 2018.