Repository files navigation

Our Tensorflow implementation of Recurrent Pose Attention in Du et al.: "RPAN: An End-to-End Recurrent Pose-attention Network for Action Recognition in Videos".

Note that we are not associated with the original authors.

Tested with python 2.7. The following additional packages are required:tensorflow, numpy, csv, cv2

Our simple RPAN model inmodel_simple.py drops the parameter sharing method in Equation (2) of the paper. This is the version used in our submission for CVPR 2018 Moments in Time challenge.

We also attempt provide a model with the original parameter sharing scheme described. It can be found inmodel_shared.py.

We provide an example on how to generate the joint maps indata_handler.py, seegkern() andload_pose_map(). Note that we useOpenpose format (published as Cao et al.: "Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields") throughout the project. If you are using a different pose detector, you will need to modify the code.

The filevideo1.npy is an example of whatload_pose_map() expects. The format is Tx18x3, where each triplet is (x,y,confidence), and all values are in range [0,1]. In case there are multiple poses, we currently picked the one with highest confidence; if no pose is detected, we set the frame to all-zeros.

We assume that the video is stored as a collection of jpeg files, sampled at 25 fps. The files are organized as:PREFIX/video_id/frame_000001.jpg etc.

Unlike the published paper, we use ResNet v2-50 to extract the convolutional cube. You can download our ResNet weights athttp://cmlab.csie.ntu.edu.tw/~agethen/resnet_v2.npy . Please do not forget to edit util.py and adjust the path to the ResNet weights.

For any feedback or questions, feel free to send a message to

s [dot] agethen [at] gmail [dot] com.