US11727201B2 - Annotation framework for video - Google Patents

Abstract

A system and method for transferring annotations associated with a media file. An annotation associated with a media file is indexed to a first instance of that media file. By comparing features of the two instances, a mapping is created between the first instance of the media file and a second instance of the media file. The annotation can be indexed to the second instance using the mapping between the first and second instances. The annotation can be processed (displayed, stored, or modified) based on the index to the second instance.

Description

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No. 17/107,018, filed Nov. 30, 2020, which is a continuation of U.S. patent application Ser. No. 16/384,289, filed Apr. 15, 2019, which is a continuation of U.S. patent application Ser. No. 15/795,635, filed Oct. 27, 2017, which is a continuation of U.S. patent application Ser. No. 14/145,641 filed Dec. 31, 2013, which is a continuation of U.S. patent application Ser. No. 13/414,675, filed Mar. 7, 2012, which is a continuation of U.S. patent application Ser. No. 12/477,762, filed Jun. 3, 2009, which is a continuation of U.S. patent application Ser. No. 11/615,771, filed Dec. 22, 2006, each of which is hereby incorporated by reference herein in its entirety.

TECHNICAL FIELD

The disclosed embodiments relate generally to the authoring and display of annotations for video, and to the collaborative sharing and editing of annotations over a network.

BACKGROUND

Annotations provide a mechanism for supplementing video with useful information. Annotations can contain, for example, metadata describing the content of the video, subtitles, or additional audio tracks. Annotations can be of various data types, including text, audio, graphics, or other forms. To make their content meaningful, annotations are typically associated with a particular video, or with a particular portion of a video.

One method by which the useful information contained in annotations can be exchanged is by transferring annotated video over a network. However, transferring video content over a network introduces several obstacles. First, video files are generally quite large, and transferring video requires substantial amounts of bandwidth, as well as host and recipient computers that can support the required bandwidth and storage needs. Second, many video files are likely to be copyrighted, or to be otherwise prohibited from distribution without payment of a fee. Compliance with copyright restrictions requires additional software and hardware investments to prevent unauthorized copying. Third, as the recipient of an annotated video may already have an unannotated copy of the video, from a data efficiency perspective the transfer of an annotated copy of the video to such a recipient unnecessarily consumes both bandwidth and storage.

Thus, exchanging annotated video by transferring a complete copy of the video is an inadequate solution.

SUMMARY

Annotations associated with a media file are transferred between devices independently of the associated media file, while maintaining the appropriate temporal or spatial relationship of the annotation with any segment of the media file. An annotation associated with a media file is indexed to a first instance of that media file. A mapping is created between the first instance of the media file and a second instance of the media file by comparing features of the two instances. The annotation can be indexed to the second instance using the mapping between the first and second instances. The annotation can be displayed, stored, or modified based on the index to the second instance.

Comparing features of instances allows the annotations to be consistently indexed to a plurality of independently acquired instances of a media file. Consistent indexing of annotations supports sharing of annotations and allows for a collaborative community of annotation authors, editors, and consumers. Annotations can include advertisements or premium for-pay content. Privileges for submitting, editing or viewing annotations can be offered for sale on a subscription basis, free of charge, or can be bundled with purchase of media files.

According to one embodiment, a first user submits to an annotation server annotations that are indexed to his instance of a media file. The annotation server maps the first user's instance of the media file to a canonical instance of the media file and stores the submitted annotation indexed to the canonical instance of the media file. A second user requests annotations, and the annotation server maps the second user's instance of the media file to the canonical instance of the media file. The annotation server sends the annotation to the second user indexed to the second user's instance of the media file.

The features and advantages described in this summary and the following detailed description are not all-inclusive. Many additional features and advantages will be apparent to one of ordinary skill in the art in view of the drawings, specification, and claims hereof.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG.1 shows a network connecting a community of video providers and consumers.

FIG.2 illustrates frames of a video, and the indexing of annotations to one or more frames.

FIG.3 illustrates frames of two instances of a video.

FIG.4(a) illustrates annotations indexed to a canonical instance of video.

FIG.4(b) illustrates mapping a client instance of video to a canonical instance of video.

FIG.5 illustrates one embodiment for storing video and annotations.

FIG.6 is an event trace of the display and modification of annotations associated with a video.

FIG.7(a) illustrates a user interface for viewing, creating, and editing annotations.

FIG.7(b) illustrates a user interface for creating a new annotation.

FIG.8 illustrates a method for determining which annotations to display.

The figures depict various embodiments of the present invention for purposes of illustration only. One skilled in the art will readily recognize from the following discussion that alternative embodiments of the structures and methods illustrated herein may be employed without departing from the principles of the invention described herein.

DESCRIPTION OF EMBODIMENTS

FIG.1 shows a network connecting a community of video providers and consumers.FIG.1 illustrates one embodiment by which a plurality of users can exchange videos and annotations. Video is used herein as an example of a media file with which annotation can be associated. This example is chosen for the purposes of illustration and is not limiting. Other types of media files with which annotations can be associated include, but are not limited to, audio programs, Flash, movies (in any encoding format), slide presentations, photo collections, animated programs, and other documents. Other examples will be apparent to one of skill in the art without departing from the scope of the present invention.

A user views, authors, and edits annotations using aclient104. An annotation is any data which can usefully supplement a media file. For example, an annotation can be an audio or textual commentary, translation, advertisement or summary, rating on a predetermined scale (1-5 stars), metadata, or a command for how the media file should be displayed. An annotation can also include video content. Theclients104 include software and hardware for displaying video. For example, aclient104 can be implemented as a television, a personal computer, a digital video recorder (DVR), a personal digital assistant (PDA), a cellular telephone, or another device having or connected to a display device; software includes any video player adapted to decode video files, such as MPEG-2, MPEG-4, QuickTime, VCD, or any other current or future video format. Other examples of clients will be apparent to one of skill in the art without departing from the scope of the present invention. A graphical user interface used by theclient104 according to one embodiment is described herein with references toFIGS.7(a) and7(b).

Theclients104 are connected to anetwork105. Thenetwork105 can be implemented as any electronic medium by which annotation content can be transferred. Through thenetwork105, theclients104 can send and receive data fromother clients104. Thenetwork105 can be a global (e.g., the Internet), regional, wide-area, or local area network.

Avideo server106 stores a collection of videos on an electronic medium. Responsive to a request by aclient104 for a particular video (or a set of videos matching certain criteria), thevideo server106 transfers a video over thenetwork105 to theclient104. Thevideo server106 may be configured to charge a fee for the service of providing the video to the client, or it may provide the video free of charge. Thevideo server106 can be implemented, for example, as an on-demand content service, an online store, or a streaming video server. Other examples of video servers will be apparent to one of skill in the art without departing from the scope of the present invention.

Some of theclients104 are also connected to video sources102. A video source102 is a device providing video to the client. For example, a video source102 could be a cable box, a television antenna, a digital video recorder, a video cassette player, a camera, a game console, a digital video disk (DVD) unit, or any other device capable of producing a video output in a format readable by theclient104. Other examples of video sources102 will be apparent to one of skill in the art without departing from the scope of the present invention.

According to one embodiment of the present invention,clients104 can send video over thenetwork105. For example, theclient104B can receive video from thevideo source102B and transfer it through the network to another client, such as theclient104D.Clients104 can also send video through thenetwork105 to thevideo server106. Video sent from aclient104 to thevideo server106 is stored on an electronic medium and is available toother clients104.

Annotation server110 is connected to thenetwork105. Theannotation server110 stores annotations on an electronic medium. Responsive to a request from aclient104 for an annotation associated with a particular media file, theannotation server110 sends one or more annotations associated with the media file to theclient104 through thenetwork105. Responsive to a submission by theclient104 of one or more annotations associated with a media file, theannotation server110 stores the one or more annotations in association with the media file. Theannotation server110 stores annotations indexed to instances of one or more media files or portions thereof. A method used by theannotation server110, according to various embodiments of the present invention, is described herein with reference toFIGS.4-6.

Optionally, avideo server108 is communicatively connected to theannotation server110, either locally or over thenetwork105. Thevideo server108 can have many of the same capabilities as described herein with reference to thevideo server106. Thevideo server108 can transfer video to theclients104 over thenetwork105. In one embodiment, theannotation server110 andvideo server108 in combination transfer annotated video to aclient104. In another embodiment, thevideo server108 stores a canonical instance of a video, as described herein with reference toFIG.5.

As shown in the figure, any given client may have access to video from a variety of sources. For example, theclient104A can receive video directly from thevideo source102A or from thevideo server106 via thenetwork105. Different clients sometimes have access to different video sources. For example, like theclient104A, theclient104B can receive video from thevideo server106 via thenetwork105, but, in contrast to theclient104A, has direct access to thevideo source102B instead of thevideo source102A.

Although a client can obtain video from a potentially wide range of video sources, the present invention allows annotations sent from theannotation server110 to the client to be consistently associated with a particular media file and portion thereof, regardless of the source from which the client's copy of the video was obtained. The consistent association of annotations with media files facilitates the exchange of annotations between users having different instances (or copies) of a given media file. The present invention enables the sharing and exchange of annotations among a plurality of clients by reindexing annotations for various instances of client media files. For example, theannotation server110 sends annotations indexed to theclient104A's instance of a video and sends annotations indexed to theclient104B's instance of the video, despite the fact that the two clients may have acquired their copies of the video from different sources. Theannotation server110 beneficially provides annotations that are not only appropriate for the video displayed by theclient104, but for the particular instance of the video which theclient104 is displaying, as described herein with reference toFIG.4.

Referring now toFIG.2, there is shown a conceptual diagram illustrating how annotations are associated temporally and/or spatially with a video file and one or more frames of thereof.FIG.2 shows a series of video frames, running fromframe200 to frame251. Theclient104 displays these frames, and can also pause, rewind, fast-forward, skip, or otherwise adjust the order or speed with which the frames are displayed.

For the purposes of illustration, the following discussion refers to a video as being composed of frames. Video is sometimes stored or transmitted as blocks of frames, fields, macroblocks, or in sections of incomplete frames. When reference is made herein to video being composed of frames, it should be understood that during intermediate steps video may in fact be stored as any one of various other forms. The term “frame” is used herein for the sake of clarity, and is not limiting to any particular format or convention for the storage or display of video.

Some of the frames have annotations associated with them as provided by a particular user. In the example illustrated,frame201 is drawn in greater to detail to illustrate some of its associated annotations. As shown in the figure, annotations can be associated with a particular spatial location of a frame, or they can be associated with an entire frame. For example,annotation1 is associated with a rectangular box in the upper-left corner offrame201. In contrast,annotation4 is associated with the entire frame.

Annotations can also be associated with overlapping spatial locations. For example,annotation1 is associated with a rectangular box overlapping a different rectangular box associated withannotation2. In one embodiment, annotations can be associated with a spatial location defined by any closed form shape. For example, as shown inFIG.2,annotation3 is associated with spatial locations defined by an elliptical shape.

Annotation list280 maintains associations between the spatial definition of annotations and the content of annotations.Annotation1, associated with a rectangular box inframe201, includes the text “Vice President.”Annotation1 is an example of an annotation useful for highlighting or adding supplemental information to particular portions of a frame.Annotation4 is associated with theentire frame201 and contains the text “State of the Union.”Annotation4 is an example of an annotation used to summarize the content of a frame.Annotation5 is associated with theentire frame201 and contains some audio, which, in this case, is a French audio translation.Annotation5 is an example of an annotation used to provide supplemental audio content.

Annotations can also have temporal associations with a media file or any portion thereof. For example, an annotation can be associated with a specific frame, or a specific range of frames. InFIG.2, for example,annotation2 could be associated withframe200 to frame251, whileannotation5 is associated only withframe201. The spatial definition associated with an annotation can also change over time. For example,annotation1 can be associated with a first region inframe201, and with a second region inframe202. Time and spatially-dependent annotation associations are particularly useful for providing supplemental information regarding objects in motion, and can accommodate, as in the example shown in the figure, the movement of the Vice-President of the United States. The temporal associations can be defined in terms of frame numbers, timecodes, or any other indexing basis. The illustration of theannotation list280 as a table is not meant to limit the underlying storage format used; any format or organization of the annotation information may be employed including optimized formats that reduce storage requirements and/or increase retrieval speed.

During playback of a media file, theclient104 is adapted to display the annotations associated with the frames of the file. Annotations can be displayed, for example, as text superimposed on the video frame, as graphics shown alongside the frame, or as audio reproduced simultaneously with video; annotations may also appear in a separate window or frame proximate to the video. Annotations can also include commands for how the media file with which they are associated is to be displayed. Displaying command annotations can include displaying video as instructed by the annotation. For example, responsive to an annotation, theclient104 might skip to a different place in a video, display a portion of the video in slow motion, or jump to a different video altogether.

Theclient104 is capable of displaying a subset of the available annotations. For example, a user watching the video ofFIG.2 can select which annotations should be displayed by theclient104 by designation of various criteria. The user can choose to receive only certain types of annotations (e.g. commentary, text, graphic, audio), or only annotations that are defined by a particular region of the display. The user can choose to receive only annotations in a particular language, matching a certain search criteria (such as keywords), or authored by a particular user. As another example, when annotations are written and edited in a collaborative community of users, a user can choose to receive only annotations authored by users with reputations above a certain threshold, or to receive only annotations with ratings above a certain threshold. Users can also search for annotations, and retrieve associated video based on the results of the annotation search.

Certain annotations can be given a priority that does not allow a user to prevent them from being displayed. For example, annotations can include advertisements, which may be configured so that no other annotations are displayed unless the advertisement annotations are also displayed. Such a configuration would prevent users from viewing certain annotations while avoiding paid advertisement annotations. A method for determining which annotations to display is described herein with reference toFIG.8.

Users can also edit annotations using theclient104. For example, a user viewing the annotations shown inFIG.2 may be dissatisfied withannotation1. The user changes the annotation text “Vice President” to “Vice President of the United States” using an input device connected to theclient104. Future display of the annotation (to this user or possibly other users) would include the modified text “Vice President of the United States.” As another option, a user can change the temporal or spatial definition with which annotations or associated. For example, the astute user may recognize that the documents shown on the right side of the frame are actually excerpts from 15 USC §§ 78dd-1, and that the Constitution (despite being almost completely obscured by the position of the President) is just barely visible on the left side of the frame. The user can change the temporal definition with whichAnnotation3 is associated accordingly, for example, by dragging (for example, in a direct manipulation user interface illustrating frames of the video) the spatial definition to a different location using an input device connected to theclient104.

Theannotation list280 is shown inFIG.2 for the purposes of illustration as one example of how a client can organize annotations and their associated frames. Theannotation list280 is useful for managing and displaying annotations associated with a frame or range of frames, but various clients can organize annotations differently without departing from the scope of the present invention.

As shown inFIG.1, a client sometimes has access to multiple instances of the same video, and different clients frequently have access to various different instances.FIG.3 illustrates sequences of the frames making up two instances of the same video. For example,video instance302 could be a copy of a video received from a cable channel, whilevideo instance304 is a copy of the same video received from an online video store. As another example,video instance302 could be a copy of a video recorded by a first user's digital video recorder receiving a signal from a first broadcast station, whilevideo instance304 is a copy of the same video recorded by a second user's digital video recorder receiving a signal from a second broadcast station.

Asvideo instance302 is acquired independently ofvideo instance304, it is likely that the two copies are not time-synchronized, and/or are of different lengths. For example,video instance302 might have been recorded from The Zurich Channel, a television affiliate known for its punctuality and good taste.Video instance304, on the other hand, might have been recorded from TV Tulsa, a television affiliate known for its slipshod programming and haphazard timing. Thus, as shown inFIG.3, the frames of the first instance might not necessarily correspond to the frames of the second instance. In addition, there are numerous other types of differences that can arise between different instances of a given program or broadcast. These include, and are not limited to, differences in encoding parameters (e.g., resolution, frame rate) and differences in file formats.

In the example illustrated, theframes306 ofvideo instance302 are time-shifted with respect to theframes308 of thevideo instance304. The first frame of theframes308 contains the same content as the third frame of theframes306. When annotations are associated with specific frames of a video by one user, it is desirable that they be displayed with those frames when shown to another user in spite of the possibility of time shifting between various instances of the video. Notice as well thatvideo instance302 has 6 frames, whereasvideo instance304 has 4 frames.

Theannotation server110 accounts for this time shifting of frames so that annotations can be properly displayed with various instances of the video. For example, suppose an annotation describes the driver who enters the third frame of theframes306. If this annotation is indexed with respect to theframes306, theannotation server110 translates this index to an index with respect to theframes308 so that the annotation can be properly displayed with thevideo instance304. Theannotation server110 translates the annotation indexes by mapping one video instance to another.

Referring now toFIG.4(a), annotations404 are indexed to a canonical instance ofvideo406. For the purposes of illustration, the instance of video having annotations indexed to it is referred to as the canonical instance, and the instance of video that will be displayed at the client is referred to as the client instance. According to one embodiment, annotations can be shared in multiple directions between two or more client peers. As such, it is possible that there is no definitively canonical instance of video. It should be understood that the term “canonical instance” refers to a role that an instance of video plays in one case of annotation exchange, and not necessarily to the status of that copy of the video in the video distribution system or in the annotation framework as a whole.

Thevideo server108 may store video content in chunks. One system and method for storing video in chunks is disclosed in U.S. patent application Ser. No. 11/428,319, titled “Dynamic Media Serving infrastructure” to Manish Gupta, et al., filed Jun. 30, 2006, and U.S. Provisional Patent Application Ser. No. 60/756,787, titled “Discontinuous Download of Media Articles” to Michael Yu, et al., filed Jan. 6, 2006, both of which are incorporated herein by reference in their entirety.FIG.4(a) shows a canonical instance ofvideo406 stored aschunk402A andchunk402B. A chunk is a data element for storing video. Storing video in chunks is beneficial for the efficient indexing and transfer of video, and allows for the manipulation as video data of more manageable size.

As described herein with reference toFIG.2, an annotation can be associated with a specific frame in a video. The association between the annotation and the specific frame is stored by indexing the annotation to a frame in a particular instance of the video.Annotation404A, for example, is indexed to a frame of the canonical instance ofvideo406, in this case to a frame in thechunk402A.

As also described herein with reference toFIG.2, an annotation can be associated with a range of frames in a video. A set of one or more frames of video is sometimes referred to as a segment of video.Annotation404D, for example, is indexed to a segment of video of the canonical instance ofvideo406, in this case the segment including one or more frames of thechunk402B.

The client receives a video from a video source or server (such as one of those described herein with reference toFIG.1) and stores a copy as the client instance ofvideo408. As the client displays the video, the client periodically requests, from the annotation server, annotations associated with frames of video about to be displayed. To ensure that annotations are requested, retrieved, transmitted and received in sufficient time for display with their associated frames, the client requests annotations associated with a frame some time before that frame is to be displayed.

For increased efficiency, the client can combine requests for annotations associated with particular frames into a request for annotations associated with a segment of video. A request could, for example, seek to retrieve all of the annotations associated with a given video. In the example shown, the client requests annotations associated with the segment ofvideo409. The request for annotations will return annotations associated with individual frames of the segment, or annotations associated with a superset or subset of the frames of the segment. For example, the client can request annotations associated with exactly the segment ofvideo409, associated with individual frames of the segment ofvideo409, or associated with the entire video.

Referring now toFIG.4(b), theannotation server110 maps the client instance ofvideo408 to a canonical instance ofvideo406. Themapping412 describes the correspondence between frames of the client instance ofvideo408 and frames in the canonical instance ofvideo406. Theannotation server110 can map the client instance of thevideo408 to the canonical instance ofvideo406 using a variety of techniques. According to one embodiment of the present invention, the client's request for annotations includes a feature of the client instance ofvideo408. A feature is a succinct representation of the content of one or more frames of video that are similar. For example, theannotation server110 may group the frames into logical units, such as scenes or shots. Theannotation server110 may use scene detection algorithms to group the frames automatically. One scene detection algorithm is described in Naphade, M. R., et al., “A High-Performance Shot Boundary Detection Algorithm Using Multiple Cues”, 1998 International Conference on Image Processing (Oct. 4-7 1998), vol. 1, pp. 884-887, which is incorporated by reference herein.

Thus, theannotation server110 can compute one feature set for all frames that belong to the same scene. The feature can be, for example, a description of a characteristic in the time, spatial, or frequency domains. For example, a client can request annotations associated with a specific frame, and can describe that frame by its time, position, and frequency domain characteristics. The client can use any technique for determining features of video, such as those described in Zabih, R., Miller, J., and Mai, K., “Feature-Based Algorithms for Detecting and Classifying Scene Breaks”, Proc. ACM Multimedia 95, San Francisco, Calif. (November 1993), pp. 189-200; Arman, F., Hsu, A., and Chiu, M-Y., “Image Processing on Encoded Video Sequences”, Multimedia Systems (1994), vol. 1, no. 5, pp. 211-219; Ford, R. M., et al., “Metrics for Shot Boundary Detection in Digital Video Sequences”, Multimedia Systems (2000), vol. 8, pp. 37-46, all of the foregoing being incorporated by reference herein. One of ordinary skill in the art would recognize various techniques for determining features of video.

Generally, a distance function is defined over the universe of features that captures the closeness of the underlying sets of frames. When theannotation server110 receives a request for annotation for a frame, along with its feature set, the server first attempts to map the frame in the request to the closest frame in the canonical instance ofvideo406. Theannotation server110 uses the temporal position of the frame in the client instance of video408 (one of the features in the feature set) to narrow down the set of frames in thecanonical video406 that may potentially map to this frame, e.g., by limiting the candidate set to frames within a fixed amount of time or frames before and after the selected frame. For all of the frames in the candidate set, theannotation server110 computes the distance between the feature set of the frame from theclient408 and feature set of the frame fromcanonical video406. The frame from thecanonical video406 with the shortest distance is termed as the matching frame. The client frame is then mapped to the matching frame. If the distance to the closest frame is greater than a certain threshold, indicating absence of a good match, no annotations are returned. The components described by a feature used to create the mapping can reside in the segment of video for which annotations are being requested, but need not be. Similarly, the components described by a feature may or may not reside in the segment of video to which an annotation is indexed.

Features may be represented as strings, allowing theannotation server110 to search for features using an inverted index from feature strings to frames, for example. Theannotation server110 may also search for features by defining a distance metric over the feature set and selecting the candidate frame with the smallest distance. Such mapping could take place at the time theserver110 receives the client request, or theannotation server110 can pre-compute and maintain the distances in an offline process.

Using themapping412, theannotation server110 determines a corresponding segment ofvideo414 in the canonical instance of video. The corresponding segment ofvideo414 has content that closely matches the content of the segment ofvideo409, as described above. Under ideal conditions, the corresponding segment ofvideo414 contains instances of the same frames as the segment ofvideo409. Theannotation server110 associates each frame in theclient video408 that maps to a frame in the canonical instance of video with a frame number and maintains a list of frame numbers for each frame mapping. In one example, the length of the list of frame numbers is equal to the number of frames in the client instance ofvideo408, where each entry maps the corresponding frame to the frame in the canonical instance ofvideo406.

The annotation server determines the annotations that are indexed to the corresponding segment of video414 (or to a superset or subset of the corresponding segment of video414). As the example ofFIG.4(b) illustrates, theannotation404D is indexed to a segment of video that falls in the corresponding segment ofvideo414. In response to the request for annotations for thesegment409, theannotation server110 transmits theannotation404D to the client.

Optionally, the annotation server can also transmit information describing the segment of the video that the annotation is associated with. For example, using a feature as a reference point, the annotation server can describe a frame (or range of frames) with respect to that reference point.

FIG.5 illustrates the organization of video and annotations.FIG.5 shows how annotations can be indexed to a canonical instance of video in an annotation server.

According to one embodiment, annotations are stored in an annotation repository. Canonical instances of video are stored in a video repository. The annotation and repositories can be included in the same server, or they can be included in different servers. For example, the annotations can be stored in theannotation server110 and video can be stored in thevideo server108.

An annotation includes a reference to a segment of video. For example, theannotation404D includes a temporal definition501D. A temporal definition specifies one or more frames of a canonical instance of video. In the example illustrated, the temporal definition501D refers to one of theframes504 of the canonical instance ofvideo406. As another example, theannotation404F includestemporal definition510F.Temporal definition510F refers to a range of the frames of the canonical instance ofvideo406. A temporal definition can be described using a variety of metrics including, but not limited to, document identifiers, frame identifiers, timecodes, length in frames, length in milliseconds, and various other combinations.

The temporal definition is one example of how annotations can be associated with segments of video. Other methods for associating annotations with segments of video will be apparent to one of skill in the art without departing from the scope of the present invention.

An annotation also includes annotation content511. Annotation content can include, for example, audio, text, metadata, commands, or any other data useful to be associated with a media file. An annotation can optionally include a spatial definition509, which specifies the area of the frame (or frames) with which that annotation is associated. Use of a spatial definition509 is an example of one method for associating an annotation with a specific spatial location on a frame.

As an example, suppose the corresponding segment ofvideo414 includes theframes504. The corresponding segment ofvideo414 can be defined as a range of timecodes. The annotation server retrieves annotations by searching for annotations with references to timecodes that are within or overlapping with the range of timecodes defining the corresponding segment ofvideo414. The annotation server retrievesannotation404D, including theannotation content511D. The annotation server transmits theannotation content511D (or theannotation404D, which includes theannotation content511D) to the client, which displays theannotation content511D.

FIG.6 is an event trace of the display and modification of annotations associated with a video, according to one embodiment of the present invention. Theclient104 receives a segment of video from avideo server106 or a video source102, and stores a copy as the client instance of video. The client processes the segment using a feature detection algorithm and determines602 a feature based on a first segment of video. The client sends a request for annotations associated with a second segment of video, the request including the feature, to theannotation server110.

The first segment of video may contain some frames in common with the second segment of video, but need not. The feature included in the request for annotations associated with the second segment of video may additionally include features from adjacent segments to the second segment of video.

The request can also include metadata describing the content or title of the video so that the annotation server can retrieve the appropriate annotations. For example, video purchased from an online store may have a video title that can be used to filter the set of available annotations. As another example, the metadata sent to the annotation server for video acquired from broadcast television or cable can include a description of the time and channel at which the video was acquired. The annotation server can use this time and channel information to determine the appropriate video and retrieve annotations associated with that video.

Theannotation server110 receives the request for annotations. Theannotation server110searches604 for the feature included in the request in a canonical instance of the video and creates a mapping between the client instance of the video and the canonical instance of the video. In one embodiment, the request for annotations includes metadata indicating a particular video for which to retrieve annotations, and theannotation server110searches604 in a canonical instance in the video indicated by this metadata for the feature.

Theannotation server110searches608 an annotation repository for annotations associated with the video and returns an annotation. For example, theannotation server110 can search for annotations indexed to the canonical instance of the video. Using the mapping between the two instances, theannotation server110 can translate the index to the canonical instance of the video to an index to the client instance of the video

Theannotation server110 transmits an annotation associated with the video to the client. According to one embodiment, the annotation also includes index information defining the set of one or more frames associated with the annotation. Theannotation server110 can define frames associated with the annotation, for example, by indexing the association with respect to the feature.

Theclient104 receives and displays610 the annotation. Theclient104 can also process index information for the annotation so that the annotation is displayed appropriately along with the client instance of the video.

Optionally, the client receives612 changes to the annotation from the user. For example, a user can edit text, re-record audio, modify metadata included in the annotation content, or change an annotation command. Theclient104 transmits the modified annotation to theannotation server110, or, alternatively, transmits a description of the modifications theannotation server110.

Theannotation server110 receives the modified annotation. Theannotation server110stores614 the modified annotation and indexes the modified annotation to the canonical instance of the video. Theannotation server110 can index the modified annotation with the canonical instance of the video using a variety of methods. For example, theannotation server110 can translate an index to the client instance of the video using a previously established mapping. As another example, theclient104 can include a feature with the modified annotation, and theannotation server110 can establish a new mapping between the client instance of the video and the canonical instance of the video.

For the purposes of illustration, features have been shown as flowing from theclient104 to theannotation server110. However, for the purpose of establishing a mapping between the client instance of the video and the canonical instance of the video, features can flow in either direction. The example of theannotation server110 maintaining this mapping on the basis of features sent by theclient104 is given for the purposes of illustration and is not limiting. In another embodiment, the client maintains the mapping between the client instance of the video and the canonical instance of the video, for example, on the basis of features of the canonical instance of the video sent by theannotation server110 to theclient104. In yet another embodiment, a third party maintains the mapping between the client instance of the video and the canonical instance of the video by receiving features from both theannotation server110 and theclient104.

Theclient104 can also be used to submit a new annotation. For example, a user can create annotation content and associate it with a video. The user can also specify a spatial definition for the new annotation and choose a range of frames of the client instance of the video to which the annotation will be indexed. Theclient104 transmits the new annotation to theannotation server110 for storage.

Referring now toFIG.7(a), a user can search, create, or edit annotations using a graphical user interface. In the example illustrated, the graphical user interface for annotations is integrated into a video playergraphical user interface702. The video playergraphical user interface702 is an example of an interface that might be shown on the display device of aclient104. The video playergraphical user interface702 includes a display area for presenting the media file (in the example illustrated, a video), as well as control buttons for selecting, playing, pausing, fast forwarding, and rewinding the media file. The video playergraphical user interface702 can also include advertisements, such as the advertisement for the National Archives and Records Administration shown inFIG.7(a).

The video playergraphical user interface702 presents a frame of video. Shown along with the frame of video is anannotation definition704. Theannotation definition704 graphically illustrates the spatial definition and/or the temporal definition of an annotation. For example, theannotation definition704 shown inFIG.7(a) delineates a subset of the frame with which an annotation is associated. As another example, anannotation definition704 can delineate a range of frames with which an annotation is associated. While asingle annotation definition704 is shown inFIG.7(a), the video playergraphical user interface702 can include a plurality ofannotation definitions704 without departing from the scope of the invention.

Theannotation definition704 can be displayed in response to a user selection, or as part of the display of an existing annotation. For example, the user can use an input device to select a region of the frame with which a new annotation will be associated, and in response to that selection the video playergraphical user interface702 displays theannotation definition704 created by the user. As another example, the video playergraphical user interface702 can display video and associated annotations, and can display theannotation definition704 in conjunction with displaying an associated annotation.

The video playergraphical user interface702 also includesannotation control buttons706, which allow the user to control the content and display of annotations. For example, the video playergraphical user interface702 can include a button for searching annotations. In response to the selection of the search annotations button, the client searches for annotations associated with the annotation definition704 (or a similar definition), or for annotations associated with a keyword. The results of the search can then be displayed on the video playergraphical user interface702. As another example, the video playergraphical user interface702 can include a button for editing annotations. In response to the selection of the edit annotations button, the video playergraphical user interface702 displays one or more annotations associated with theannotation definition704 and allows the user to modify the one or more annotations. As yet another example, the video playergraphical user interface702 can include a button for creating a new annotation. In response to the selection of the create new annotation button, the video playergraphical user interface702 displays options such as those shown inFIG.7(b).

Referring now toFIG.7(b), theannotation control buttons706 indicate that the create new annotation button has been selected. The video playergraphical user interface702 includes a display area for receiving user input of the new annotation content. In the example illustrated, the new annotation content includes somenew annotation text708. As shown inFIG.7(b), as the user enters the description “General MacArthur”, thenew annotation text708 is displayed. In response to a further user selection indicating the authoring of annotation content is complete, the new annotation is submitted, for example, to theannotation server110, and displayed in the video playergraphical user interface702.

The entering ofnew annotation text708 has been shown as an example of the authoring of annotation content. The video playergraphical user interface702 can be adapted to receive other types of annotation content as well. For example, annotation content can include audio, and the video playergraphical user interface702 can include a button for starting recording of audio through a microphone, or for selecting an audio file from a location on a storage medium. Other types of annotations and similar methods for receiving their submission by a user will be apparent to one of skill in the art without departing from the scope of the invention.

FIG.8 illustrates a method for determining which annotations to display. In one embodiment, theclient104 displays only some of the received annotations. Theclient104 performs a method such as the one illustrated inFIG.8 to determine which annotations should be displayed and which should not.

Theclient104 receives802 an annotation. The client determines804 if the annotation is high-priority. A high-priority annotation is displayed regardless of user settings for the display of annotations. High-priority annotations can include, for example, advertisements, emergency broadcast messages, or other communications whose importance that should supersede local user settings.

If theclient104 determines804 that the annotation is high-priority, the client displays812 the annotation. If theclient104 determines804 that the annotation is not high-priority, the client determines806 if annotations are enabled. Annotations can be enabled or disabled, for example, by a user selection of an annotation display mode. If the user has selected to disable annotations, theclient104 does not display810 the annotation. If the user has selected to enable annotations, theclient104 determines808 if the annotation matches user-defined criteria.

As described herein, theclient104 allows the user to select annotations for display based on various criteria. In one embodiment, the user-defined criteria can be described in the request for annotation, limiting the annotations sent by theannotation server110. In another embodiment, the user-defined criteria can be used to limit which annotations to display once annotations have been received at theclient104. User defined-criteria can specify which annotations to display, for example, on the basis of language, annotation content, particular authors or groups of authors, or other annotation properties.

If theclient104 determines808 that the annotation satisfies the user-defined criteria, theclient104 displays812 the annotation. If theclient104 determines808 that the annotation does not satisfy the user-defined criteria, theclient104 does not display810 the annotation.

FIG.8 illustrates one example of how theclient104 may determine which annotations to display. Other methods for arbitrating annotation priorities established by the annotation provider and the annotation consumer will be apparent to one of skill in the art without departing from the scope of the present invention.

Turning now to the canonical instance of video disclosed herein, the canonical instance of video can be implemented in a variety of ways according to various embodiments. In some cases, theannotation server110 has selected a canonical instance of the video prior to the submission of the new annotation. Theclient104 can send a feature to facilitate the indexing of the new annotation to the canonical instance of the video. In other cases, for example, when the annotation is the first to be associated with a particular video, theannotation server110 may not have yet identified a canonical instance of the video. Theannotation server110 stores the annotation indexed to the client instance of the video, and establishes the client instance of the video as the canonical instance of the video for future annotation transactions.

According to one embodiment of the present invention, annotations are stored indexed to features of the instance of video used by the client that submitted that annotation. Annotations can be stored and retrieved without any underlying canonical instance of video. For example, each annotation can be indexed to its own “canonical instance of video”, which refers to the instance of video of the submitter. Such an approach is particularly beneficial for situations in which theannotation server110 does not maintain or have access to copies of the video itself. Essentially, theannotation server110 can serve as a blind broker of annotations, passing annotations from authors to consumers without its own copy of the video with which those annotations are associated.

A content-blind annotation server can be beneficial, for example, when the video content is copyrighted, private, or otherwise confidential. For example, a proud mother may want to annotate a film of her son's first bath, but might be reticent to submit even a reference instance of the video to a central annotation server. The content-blind annotation server stores annotations indexed to the mother's instance of the video, without access to an instance of its own. When an aunt, uncle, or other trusted user with an instance of the video requests annotations, his instance is mapped to the mother's instance by comparison of features of his instance to features of the mother's instance received with the submission of the annotation. Features can be determined in such a way that cannot be easily reversed to find the content of a frame, thus preserving the privacy of the video.

The case of an annotation server and a client is but one example in which the present invention may be usefully employed for the sharing and distribution of annotations for video. It will be apparent to one of skill in the art that the methods described herein for transmitting annotations without the need to transmit associated video will have a variety of other uses without departing from the scope of the present invention. For example, the features described herein could be used in an online community in which users can author, edit, review, publish, and view annotations collaboratively, without the burdens of transferring or hosting video directly. Such a community would allow for open-source style production of annotations without infringing the copyright protections of the video with which those annotations are associated.

As an added feature, a user in such a community could also accumulate a reputation, for example based on other users' review of the quality of that user's previous authoring or editing. A user who wants to view annotations could have the option of ignoring annotations from users with reputations below a certain threshold, or to search for annotations by users with reputations of an exceedingly high caliber. As another example, a user could select to view annotations only from a specific user, or from a specific group of users.

As described herein, annotations can also include commands describing how video should be displayed, for example, commands that instruct a display device to skip forward in that video, or to jump to another video entirely. A user could author a string of jump-to command annotations, effectively providing a suggestion for the combination of video segments into a larger piece. As an example, command annotations can be used to create a new movie from component parts of one or more other movies. The annotation server provides the annotations to the client, which acquires the various segments specified by the annotations and assembles the pieces for display to the user.

The present invention has applicability to any of a variety of hosting models, including but not limited to peer-to-peer, distributed hosting, wiki-style hosting, centralized serving, or other known methods for sharing data over a network.

The annotation framework described herein presents the opportunity for a plurality of revenue models. As an example, the owner of the annotation server can charge of fee for including advertisements in annotations. The annotation server can target advertisement annotations to the user based on a variety of factors. For example, the annotation server could select advertisements for transmission to the client based on the title or category of the video that the client is displaying, known facts about the user, recent annotation search requests (such as keyword searches), other annotations previously submitted for the video, the geographic location of the client, or other criteria useful for effectively targeting advertising.

Access to annotations could be provided on a subscription basis, or annotations could be sold in a package with the video content itself. For example, a user who purchases a video from an online video store might be given permission for viewing, editing, or authoring annotations, either associated with that video or with other videos. An online video store might have a promotion, for example, in which the purchase of a certain number of videos in a month gives the user privileges on an annotation server for that month.

Alternatively, the purchase of a video from an online video store might be coupled to privileges to author, edit, or view annotations associated with that video. If a particular annotation server becomes particularly popular with users, controlled access to the annotation server could assist with the protection of the copyrights of the video. For example, a user might have to prove that he has a certified legitimately acquired copy of a video before being allowed to view, edit, or author annotations. Such a requirement could reduce the usefulness or desirability of illegally acquired copies of video.

These examples of revenue models have been given for the purposes of illustration and are not limiting. Other applications and potentially profitable uses will be apparent to one of skill in the art without departing from the scope of the present invention.

Reference in the specification to “one embodiment” or to “an embodiment” means that a particular feature, structure, or characteristic described in connection with the embodiments is included in at least one embodiment of the invention. The appearances of the phrase “in one embodiment” in various places in the specification are not necessarily all referring to the same embodiment.

It should be noted that the process steps and instructions of the present invention can be embodied in software, firmware or hardware, and when embodied in software, can be downloaded to reside on and be operated from different platforms used by a variety of operating systems.

The present invention also relates to an apparatus for performing the operations herein. This apparatus may be specially constructed for the required purposes, or it may comprise a general-purpose computer selectively activated or reconfigured by a computer program stored in the computer. Such a computer program may be stored in a computer readable storage medium, such as, but is not limited to, any type of disk including floppy disks, optical disks, CD-ROMs, magnetic-optical disks, read-only memories (ROMs), random access memories (RAMs), EPROMs, EEPROMs, magnetic or optical cards, application specific integrated circuits (ASICs), or any type of media suitable for storing electronic instructions, and each coupled to a computer system bus. Furthermore, the computers referred to in the specification may include a single processor or may be architectures employing multiple processor designs for increased computing capability.

The algorithms and displays presented herein are not inherently related to any particular computer or other apparatus. Various general-purpose systems may also be used with programs in accordance with the teachings herein, or it may prove convenient to construct more specialized apparatus to perform the required method steps. The required structure for a variety of these systems will appear from the description below. In addition, the present invention is not described with reference to any particular programming language. It will be appreciated that a variety of programming languages may be used to implement the teachings of the present invention as described herein, and any references below to specific languages are provided for disclosure of enablement and best mode of the present invention.

While the invention has been particularly shown and described with reference to a preferred embodiment and several alternate embodiments, it will be understood by persons skilled in the relevant art that various changes in form and details can be made therein without departing from the spirit and scope of the invention.

Finally, it should be noted that the language used in the specification has been principally selected for readability and instructional purposes, and may not have been selected to delineate or circumscribe the inventive subject matter. Accordingly, the disclosure of the present invention is intended to be illustrative, but not limiting, of the scope of the invention, which is set forth in the following claim

Claims (18)

What is claimed is:

1. A method for providing annotations to client devices, the method comprising:

causing a video to be presented on a display associated with a client device;

causing a first annotation to be presented, wherein the first annotation corresponds to a first frame of the video being presented on the display;

in response to receiving a request to modify annotations presented in connection with the video, identifying a second annotation; and

causing the second annotation to be presented in connection with presentation of a second frame of the video on the display.

2. The method ofclaim 1, wherein the first annotation is text in a first language, and wherein the second annotation is text in a second language.

3. The method ofclaim 1, wherein the second annotation is identified in response to receiving a selection of an annotation from a list of annotations that is presented on the display.

4. The method ofclaim 1, further comprising:

receiving an indication of a change in resolution of presentation of the video on the display from a first resolution to a second resolution; and

identifying a portion of the second frame in which the second annotation is to be presented based on the second resolution.

5. The method ofclaim 1, wherein the second frame of the video is identified based on a frame rate of the video.

6. The method ofclaim 1, wherein the first annotation and the second annotation are identified based on an annotation definition associated with the video.

7. A system for providing annotations to client devices, the system comprising:

a hardware processor that is configured to:

cause a video to be presented on a display associated with a client device;

cause a first annotation to be presented, wherein the first annotation corresponds to a first frame of the video being presented on the display;

in response to receiving a request to modify annotations presented in connection with the video, identify a second annotation; and

cause the second annotation to be presented in connection with presentation of a second frame of the video on the display.

8. The system ofclaim 7, wherein the first annotation is text in a first language, and wherein the second annotation is text in a second language.

9. The system ofclaim 7, wherein the second annotation is identified in response to receiving a selection of an annotation from a list of annotations that is presented on the display.

10. The system ofclaim 7, wherein the hardware processor is configured to:

receive an indication of a change in resolution of presentation of the video on the display from a first resolution to a second resolution; and

identify a portion of the second frame in which the second annotation is to be presented based on the second resolution.

11. The system ofclaim 7, wherein the second frame of the video is identified based on a frame rate of the video.

12. The system ofclaim 7, wherein the first annotation and the second annotation are identified based on an annotation definition associated with the video.

13. A non-transitory computer-readable medium containing computer executable instructions that, when executed by a processor, cause the processor to perform a method for providing annotations to client devices, the method comprising:

causing a video to be presented on a display associated with a client device;

causing a first annotation to be presented, wherein the first annotation corresponds to a first frame of the video being presented on the display;

in response to receiving a request to modify annotations presented in connection with the video, identifying a second annotation; and

causing the second annotation to be presented in connection with presentation of a second frame of the video on the display.

14. The non-transitory computer-readable medium ofclaim 13, wherein the first annotation is text in a first language, and wherein the second annotation is text in a second language.

15. The non-transitory computer-readable medium ofclaim 13, wherein the second annotation is identified in response to receiving a selection of an annotation from a list of annotations that is presented on the display.

16. The non-transitory computer-readable medium ofclaim 13, the method comprising:

receiving an indication of a change in resolution of presentation of the video on the display from a first resolution to a second resolution; and

identifying a portion of the second frame in which the second annotation is to be presented based on the second resolution.

17. The non-transitory computer-readable medium ofclaim 13, wherein the second frame of the video is identified based on a frame rate of the video.

18. The non-transitory computer-readable medium ofclaim 13, wherein the first annotation and the second annotation are identified based on an annotation definition associated with the video.

Images