Repository files navigation

A Tennis dataset and models for event detection & commentary generation. Discussed in:

"TenniSet: A Dataset for Dense Fine-Grained Event Recognition, Localisation and Description"

NOTE:The results in the paper were with oldKeras models, new results arewithMXNet and Gluon models.

The tennis dataset consists of 5 matches and has manually annotated temporal events and commentary captions.

Individual shots (serve and hit) are used to generate 11 temporal event categories:

More about the sample numbers for these individual classes can be seen below in the split information.

.......

Theannotator was used to annotate the videos withdense temporal events.

.......

Seedata for download and organisation information.

Once you have JSON annotation files from the annotator, you can run:utils/annotations/preprocess.py to perform pre-processing on the annotations,OR you can just download from myGoogle Drive

.......

Due to the limited size of the dataset, there are two varieties of train, validation and testing splits. The first (01) uses the the entire V010 as the validation and test while the second (02) splits across all videos evenly.

The resulting statistics per event class are as follows:

.......

There is one commentary style caption for each of the 746 points, as well as another 10817 captions not aligned to any imagery. Some examples are:

Both groups of captions are utilised to generate a word embedding for the 250 unique words in the vocabulary. The embedding is generated withtrain_embeddings.py utilising aSkipGram model. Below the 100 dimensional word embedding is visualised post t-SNE. The full embeddings can be found indata/embeddings-ex.txt.

There are a number of different models architectures to chose from, more information and download links to pretrained models can be found in the README in themodels directory.

.......

These models are trained withtrain.py and evaluated withevaluate.py

Features can be extracted using--save_feats argument, and will save them as .npy in\data\features\$model_id$\ with the same structure as\data\frames\.

The table below shows theF1 scores per class on the test set for some of the different models

Below is a video of the CNN-RNN model on the 02 test set. This can be generated using--vis when runningevaluate.py

YouTube video of results

.......

NOTE:The captioning scripts require thenlg-eval package. Please install prior as recommended by thier README

These models are trained withtrain_gnmt.py and evaluated withevaluate_gnmt.py

The table below shows some example generated captions on the test split, the underline marks errors.03 represents the point in the GIF at the top of this page.

If you find any data or models useful please reference and cite

@inproceedings{faulkner2017tenniset,  title={TenniSet: A Dataset for Dense Fine-Grained Event Recognition, Localisation and Description},  author={Faulkner, Hayden and Dick, Anthony},  booktitle={2017 International Conference on Digital Image Computing: Techniques and Applications (DICTA)},  pages={1--8},  organization={IEEE}}