SumMe Video Summarization Dataset

Our experiments were conducted using raw videos from the SumMe videosummarization benchmark dataset⁸. The SumMe dataset contains 25 raw videos together withvideo summaries that are generated by human observers. The videos depict variousevents such as cooking, plane landing, and base jumping. The length of thevideos varies from one to six minutes. In the manuscript, we refer to the peoplewho created the personal video summaries for the SumMe dataset as“users”, and those whose eye movements we recorded as“subjects.”

A total of 41 users (19 males and 22 females) participated in the videosummarization for the SumMe dataset⁸. The users were instructed to generate video summaries thatcontain the most important content of the original raw video, but with only 5%to 15% of the original length. The audio was muted to ensure only visual stimuliwere used for summarization. Frame-level importance scores were calculated basedon the probability that a frame was included in the user summaries. Finally, agroup-level summary was generated based on the frames with the highest 15% ofthe importance scores⁸.

Human Eye Movement Data Collection

We collected human eye movement data from a separate group of sixsubjects (two males and four females) while they viewed the raw videos of theSumMe dataset. All subjects had normal or corrected to normal vision. The studywas approved by the Institutional Review Board at the Smith-Kettlewell EyeResearch Institute, and also adhered to the Declaration of Helsinki. Informedconsent was obtained for experimentation with human subjects.

Videos were presented on a Samsung screen (resolution: 2560×1440,refresh rate: 60 Hz) and generated by Psychtoolbox-3 for MATLAB on a MacBook Procomputer. Observers’ heads were stabilized by a chin and forehead rest.Viewing distance was 57 cm, and the display was 58.2°×33.3°. Eye movements were recorded with an SR Research EyeLink 1000video-based eye tracker at 1000 Hz.

All of the original videos from the SumMe dataset were resized to havethe same width (1920 pixels; 43.6 degree of visual angle). The audio was muted.Subjects were asked to watch the entire video without any additionalinstructions. The experiment was divided into six blocks, each of which lastedapproximately 10 minutes. Before each video, observers fixated a blank screenwith a red central dot for 2s.

Since the temporal resolution of the eye movement data (1000 Hz) ishigher than the frame rate of the videos (15 to 30 Hz), we first down-sampledthe eye movement data by averaging the gaze positions over the samplescorresponding to each frame.

Eye movement consistency across subjects

We first tested whether the eye movements were consistent acrosssubjects within the same video. For each subject, we obtained gaze location oneach video frame. We then calculated gaze velocity from the gaze locationdifference between every two contiguous video frames.Figure 1a andbshow two example frames with gaze locations and velocities from six subjectssuperimposed on them.

Eye movement consistency reflects the importance of a frame

Although subjects’ eye movements were generally consistent whilewatching videos, there were frames in which eye movements were inconsistent (seeFigure 1b). Could gaze consistencyserve as an indicator of the importance of individual frames? To test this, wecompared eye movement consistency between subjects in the frames that wereselected for the user summarized videos with those that were excluded. Forconsistency of gaze location, an independent t-test showed that the consistencylevel of the selected frames was significantly higher than that of theunselected frames (t(1998)=6.85,p<0.001,Figure2d). For consistency of gaze velocity, a pair-wise t-test showed thatacross the 15 pairs of subjects, the difference in eye velocity direction in theselected frames was significantly lower than that in the unselected frames(t(14)=9.96,p<0.001,Fig 2e). Similarly, the difference in eyespeed in the selected frames was also significantly lower than that in theunselected frames (t(14)=10.07,p<0.001,Fig 2f).Therefore, subjects’ eye movement patterns were more similar to eachother while viewing the frames that were included in the user summaries thanthose that were excluded. The results suggest that human gaze patterns canindicate whether a video frame is important for summarizing a video, and thatpeople tend to make similar eye movements while viewing the important videoframes.

Gaze-Aware Deep Learning Model for Video Summarization

We employed a deep learning model similar to those used in manyvideo-related tasks¹². Thesemodels rely on multiple layers of nonlinear processing to extract usefulfeatures in a task (e.g. object detection, classification, etc.). Since deeplearning networks are effective in processing images and videos, they might bewell suited to learn the importance of video frames from low level visualfeatures as well as gaze patterns.

We constructed a novel multi-stream deep learning model for videosummarization that we call DLSum (Figure3a). There are three streams in the model: 1) the spatial stream, 2) thetemporal stream, and 3) the gaze stream.

The gaze stream takes the raw eye position and velocity as inputs. The“Gaze” representation of an arbitrary framet isdenoted asg_t.$g_t^x$ and$g_t^y$ are the horizontal and vertical components ofg_t. Both$g_t^x$ and$g_t^y$ are matrices that have the same size as theoriginal video frame, and the values are initialized to zero. We then centered a1° diameter circle on each subject’s gaze location at framet. The values within each circle reflect gaze velocity fromframe t to framet+1 of that subject. Positive values indicaterightward and downward movements for the horizontal component$g_t^x$ and vertical component$g_t^y$, respectively. The absolute values reflecthorizontal and vertical gaze speeds. Finally, all values were normalized to the(0, 255) range to generate gray-scaleg_t. representations for eachframe. A schematic illustration of the construction of one subject’s Gazerepresentation is shown inFigure 3b. Westacked multiple Gaze representations to represent the movement of gaze acrossframes. A 2L input based on gaze positions and velocities for gaze stream wasformed by stacking the Gaze representations from framet to thenextL−1 frames. The final gaze input${\text{Y}}_t\in {\text{R}}^{w\times h\times 2L}$ for framet is defined as:

The spatial stream takes the RGB values of each pixel as inputsto represent visual characteristics of the video frames. The temporal streamtakes multi-frame motion vectors as inputs to convey the motion information ofthe video to the model. The motion vectors of an arbitrary framet are denoted asm_t, which contains thedisplacement from framet to framet +1.$m_t^x$ and$m_t^y$ represent the horizontal and verticalcomponents ofm_t. A2L input was stacked to convey the motion informationbetween framet and the nextL −1frames withL represents the stacking length. The multi-framemotion vectors$F_t\in {\text{R}}^{w\times h\times 2L}$ for an arbitrary framet weredefined as:

We followed Simonyan and Zisserman (2014)¹² to set the stacking lengthL of Gaze representation and multi-frame motion vectorsequal to 10. In the training stage, each stream is trained separately. In thespatial stream, RGB images with their corresponding labels are input to theconvolutional neural network. Temporal stream and gaze stream are trainedsimilarly with different inputs (multi-frame motion vectors and gazerepresentations). Then, the outputs in the second fully-connected layer with4096 neurons of each stream are combined in order to define the features foreach video frame. The combined features with their corresponding labels areinput to the support vector regression (SVR) algorithm¹³ to train a regression model. In the teststage, we first use the trained ConvNets to extract features of the test frames.Then, the learned SVR model is used to predict an importance score for eachframe. The final summary is composed of the video frames within the top 15% ofthe model-generated importance scores^8,14.

Implementation and Evaluation Details

We compared the gaze aware model (DLSum-RealGaze) with a variation thatonly used the video-based spatial and temporal streams as inputs(DLSum–NoGaze). Since collecting eye movement data during video watchingrequires additional human effort, the DLSumRealGaze model might be inefficientfor practical applications. Therefore, we also tested another variation in whichthe gaze stream used gaze locations predicted by the class activation mapping(CAM) technique¹⁵ as input(DLSum-PredictedGaze). This model but uses predicted instead of collected realgaze information.

For each variation of our model, we performed a 3-fold cross validationfor training and testing. The VGG-16 deep convolutional neural networks werepre-trained on the ImageNet dataset to avoid over-fitting. We implemented ourmodel on the Caffe deep learning framework with Tesla K80 GPUs. In addition, agrid search was run to optimize the parameters for SVR.

We compared the automatic summary (A) with the human-annotatedsummarizations (H) and report the F-score to evaluate the performance of thedifferent models^8,14. The F-score is defined as:

wherer is the recall andp isthe precision. Following conventions in the video summarizationliterature^8,14, we report the average mean F-scoreacross all users (AMF) and the average highest F-score (AHF) by comparing themodel generated summaries with the summaries generated by all users. The highestF-score was defined as the score of the model when it best matched one of theusers’ summaries. AMF reflects how well the model prediction matchesaverage user preferences, and AHF reflects how well the model prediction matchesits most similar user’s summary.

TABLE 1.

• 1.Henderson JM. Human gaze control during real-world sceneperception.Trends in cognitive sciences2003;7(11):498–504. [DOI] [PubMed] [Google Scholar]
• 2.Najemnik J, Geisler WS. Optimal eye movement strategies in visualsearch.Nature2005;434(7031):387. [DOI] [PubMed] [Google Scholar]
• 3.Goldstein RB, Woods RL, Peli E. Where people look when watching movies: Do all viewerslook at the same place?Computers in biology and medicine2007;37(7):957–64. [DOI] [PMC free article] [PubMed] [Google Scholar]
• 4.Dorr M, Martinetz T, Gegenfurtner KR, Barth E. Variability of eye movements when viewing dynamicnatural scenes.Journal of vision, 2010;10(10):28–28. [DOI] [PubMed] [Google Scholar]
• 5.Shepherd SV, Steckenfinger SA, Hasson U, Ghazanfar AA. Human-monkey gaze correlations reveal convergent anddivergent patterns of movie viewing.Current Biology2010;20(7):649–56. [DOI] [PMC free article] [PubMed] [Google Scholar]
• 6.Kienzle W, Schölkopf B, Wichmann FA, Franz MO, editors. How to find interesting locations in video: aspatiotemporal interest point detector learned from human eyemovements.Joint Pattern Recognition Symposium; 2007:Springer. [Google Scholar]
• 7.Truong BT, Venkatesh S. Video abstraction: A systematic review andclassification.ACM transactions on multimedia computing, communications, andapplications (TOMM)2007;3(1):3. [Google Scholar]
• 8.Gygli M, Grabner H, Riemenschneider H, Van Gool L, editors. Creating summaries from uservideos.European conference on computer vision; 2014:Springer. [Google Scholar]
• 9.Mathe S, Sminchisescu C. Dynamic eye movement datasets and learnt saliency modelsfor visual action recognition.Computer Vision–ECCV 2012:Springer; 2012. p.842–56. [Google Scholar]
• 10.Torralba A, Oliva A, Castelhano MS, Henderson JM. Contextual guidance of eye movements and attention inreal-world scenes: the role of global features in objectsearch.Psychological review2006;113(4):766. [DOI] [PubMed] [Google Scholar]
• 11.Cousineau DConfidence intervals in within-subject designs: A simplersolution to Loftus and Masson’s method.Tutorials in quantitative methods for psychology2005;1(1):42–5. [Google Scholar]
• 12.Simonyan K, Zisserman A, editors. Two-stream convolutional networks for actionrecognition in videos.Advances in neural information processing systems;2014.
• 13.Drucker H, Burges CJ, Kaufman L, Smola AJ, Vapnik V, editors. Support vector regressionmachines.Advances in neural information processing systems;1997.
• 14.Song Y, Vallmitjana J, Stent A, Jaimes A, editors. Tvsum: Summarizing web videos usingtitles.Proceedings of the IEEE Conference on Computer Vision and PatternRecognition; 2015. [Google Scholar]
• 15.Zhou B, Khosla A, Lapedriza A, Oliva A, Torralba A. Learning deep features for discriminativelocalization. In: Proceedings of the 2016 IEEEconference on computer vision and patternrecognition. [Google Scholar]
• 16.Mahasseni B, Lam M, Todorovic S. Unsupervised video summarization with adversarial LSTMnetworks. In Proceedings of the 2017 IEEEconference on computer vision and patternrecognition. [Google Scholar]
• 17.Loftus GR, Mackworth NH. Cognitive determinants of fixation location duringpicture viewing.Journal of Experimental Psychology: Human perception andperformance1978;4(4):565. [DOI] [PubMed] [Google Scholar]
• 18.Wu J, Zhong S-h, Ma Z, Heinen SJ, Jiang J. Foveated convolutional neural networks for videosummarization.Multimedia Tools and Applications2018:1–23.
• 19.Zhang K, Chao WL, Sha F, Grauman K. Video summarization with long short-termmemory.European conference on computer vision;2016.
• 20.Xu J, Mukherjee L, Li Y, Warner J, Rehg JM, Singh V.: Gaze-enabled ego-centric video summarization viaconstrained submodular maximization. InProceedings of the 2015 IEEE conference on computer vision andpattern recognition, pp.2235–2244. [DOI] [PMC free article] [PubMed] [Google Scholar]
• 21.Salehin MM, Paul M: A novel framework for video summarization based onsmooth pursuit information from eye tracker data. InProceedings of the 2017 international conference on multimediaretrieval, pp.692–697 [Google Scholar]