Repository files navigation

This repo contains several scripts that allow to transfer the weights from the tensorflow implementation of I3Dfrom the paperQuo Vadis, Action Recognition? A New Model and the Kinetics Dataset by Joao Carreira and Andrew Zisserman to PyTorch.

The original (and official!) tensorflow code can be foundhere.

The heart of the transfer is thei3d_tf_to_pt.py script

Launch it withpython i3d_tf_to_pt.py --rgb to generate the rgb checkpoint weight pretrained from ImageNet inflated initialization.

To generate the flow weights, usepython i3d_tf_to_pt.py --flow.

You can also generate both in one run by using both flags simultaneouslypython i3d_tf_to_pt.py --rgb --flow.

Note that the master version requires PyTorch 0.3 as it relies on the recent addition of ConstantPad3d that has been included in this latest release.

If you want to use pytorch 0.2 checkout the branch pytorch-02 which contains a simplified model with even padding on all sides (and the corresponding pytorch weight checkpoints).The difference is that the 'SAME' option for padding in tensorflow allows it to pad unevenly both sides of a dimension, an effect reproduced on the master branch.

This simpler model produces scores a bit closer to the original tensorflow model on the demo sample and is also a bit faster.

There is a slight drift in the weights that impacts the predictions, however, it seems to only marginally affect the final predictions, and therefore, the converted weights should serve as a valid initialization for further finetuning.

This can be observed by evaluating the same sample as theoriginal implementation.

For a demo, launchpython i3d_pt_demo.py --rgb --flow.This script will print the scores produced by the pytorch model.

Pytorch Flow + RGB predictions:

1.0          44.53513 playing cricket1.432034e-09 24.17096 hurling (sport)4.385328e-10 22.98754 catching or throwing baseball1.675852e-10 22.02560 catching or throwing softball1.113020e-10 21.61636 hitting baseball9.361596e-12 19.14072 playing tennis

Tensorflow Flow + RGB predictions:

1.0         41.8137 playing cricket1.49717e-09 21.4943 hurling sport3.84311e-10 20.1341 catching or throwing baseball1.54923e-10 19.2256 catching or throwing softball1.13601e-10 18.9153 hitting baseball8.80112e-11 18.6601 playing tennis

PyTorch RGB predictions:

[playing cricket]: 9.999987E-01[playing kickball]: 4.187616E-07[catching or throwing baseball]: 3.255321E-07[catching or throwing softball]: 1.335190E-07[shooting goal (soccer)]: 8.081449E-08

Tensorflow RGB predictions:

[playing cricket]: 0.999997[playing kickball]: 1.33535e-06[catching or throwing baseball]: 4.55313e-07[shooting goal (soccer)]: 3.14343e-07[catching or throwing softball]: 1.92433e-07

PyTorch Flow predictions:

[playing cricket]: 9.365287E-01[hurling (sport)]: 5.201872E-02[playing squash or racquetball]: 3.165054E-03[playing tennis]: 2.550464E-03[hitting baseball]: 1.729896E-03

Tensorflow Flow predictions:

[playing cricket]: 0.928604[hurling (sport)]: 0.0406825[playing tennis]: 0.00415417[playing squash or racquetbal]: 0.00247407[hitting baseball]: 0.00138002

To time the forward and backward passes, you can installkernprof, an efficient line profiler, and then launch

kernprof -lv i3d_pt_profiling.py --frame_nb 16

This launches a basic pytorch training script on a dummy dataset that consists of replicated images as spatio-temporal inputs.

On my GeForce GTX TITAN Black (6Giga) a forward+backward pass takes roughly 0.25-0.3 seconds.

Visualization of the weights and matching activations for the first convolutions

Weights

Activations

Weights

Activations