USRE48546E1 - System and method for presenting content with time based metadata - Google Patents

Abstract

Disclosed herein is a system and method for rendering media related content and web services. According to one or more embodiments, media content is rendered from a source. In one or more segments of the media source, at least one of an object, person or location is identified and additional information, relevant content or services about the person, object or location are displayed or made accessible while or after the media content is being rendered, on one or more devices and/or on one or more paired devices.

Description

RELATED APPLICATIONS

This application is a reissue of U.S. patent application Ser. No. 13/523,829, filed Jun. 14, 2012, and issued as U.S. Pat. No. 9,762,967 on Sep. 12, 2017, which claims benefit of priority to Provisional U.S. Patent Application No. 61/497,023, filed Jun. 14, 2011; the aforementioned priority application applications being hereby incorporated by reference for all purposes.

TECHNICAL FIELD

Embodiments described herein pertain generally to a system and method for presenting content, and more specifically, a system and method for presenting content with time based metadata.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1A-1D illustrate systems for including time based metadata information as a form of supplemental content in connection with the presentation of media content, according to one or more embodiments.

FIG. 2 illustrates a method for generating time based metadata for a media content, according to embodiments.

FIG. 3 illustrates a method for providing media data that is packaged or integrated with time based metadata, according to an embodiment.

FIG. 4 illustrates a method for providing time based metadata to a media source, for transmission to a media playback device with media content, under an embodiment.

FIG. 5 illustrates a method for providing time based metadata to a playback device that receives the media content from a media source, according to another embodiment.

FIG. 6A throughFIG. 6F illustrate examples of media content presentation that includes metadata content, according to some embodiments described herein.

FIG. 7A throughFIG. 7F illustrate examples of media content presentations that include metadata content for playback and/or paired devices, according to some embodiments described herein.

FIG. 8 throughFIG. 9C illustrate various interface tools for facilitating different metadata placement or selection operators in generating metadata for a media content, according to an embodiment.

FIG. 10A-10B illustrate examples of media content interfaces generated using metadata content for a paired device, according to embodiments described herein.

FIG. 11 illustrates a method for providing media data to be shared or published, according to an embodiment.

FIG. 12A-12E illustrate example interfaces for providing media data to be shared or published, according to an embodiment.

FIG. 13 is a block diagram that illustrates a computer system upon which embodiments described herein may be implemented.

DETAILED DESCRIPTION

Embodiments described herein include a service that enables devices to utilize various web services independent of the communication protocol and format utilized by the devices.

In particular, devices that are capable of media playback and/or web browsing typically use communication protocols such as XML, EBIF (set-top boxes), REST or JSON. For such devices, content publishers make available various forms of content, often to supplement or augment the user's playback and/or web browsing experience. For example, cinematic studios often publish movie previews and bonus clips on websites, which can be downloaded and viewed by a device that uses an XML communication format (e.g. ANDROID device), EBIF (set-top box) or JSON. Under conventional approaches, the content publishers are required to address the various communication protocols and formats used by the various devices individually. In contrast, embodiments described herein include a service that is capable of interfacing a device using any of multiple possible communication protocols with a web service that may or may not handle the particular communication protocol in question.

Moreover, under embodiments described, the web service providers may update functionality and content provided without having to address various possible communication protocols that are in use. Accordingly, otherwise incompatible devices that normally would not communicate with each other may be able to connect, communicate and/or share information.

As used herein, BD refers to Blu-ray Disc®, and BD specification refers to the various documents that define the behavior and requirements of BD players, software and related systems, and, in particular, “System Description Blu-ray Disc Read-Only Format: Part 3. Audio Visual Basic Specifications”. The BD specification includes a section pertaining to virtual file system (“VFS”) updates.

In some embodiments, reference is made to playback devices that operate as a Blu-ray player. A Blu-ray player generally supports a virtual file system (VFS) that utilizes files located on an optical disc itself, as well as files that reside in a local storage area of the playback device. The Blu-ray environment generally permits updates to the environment by way of new files and content. With reference to embodiments described herein, a Blu-ray player is able to retrieve information and data from a network, under programmatic instructions that originate from logic or software operating on the player, as well as those executing with the disc. In such implementations, the Blu-ray player may be configured to access and retrieve metadata for presentation in connection with media content that is provided at least in part from the optical disc. In this way, the original contents and functionality of the optical disc can be augmented through the addition of content, namely, audio and video files, application files, or data files, such as those downloaded from a network server. Information that updates the playback environment of the player is received through the VFS of the playback device. With further reference to the Blu-ray environment, the service may perform functions, such as dynamically generating a manifest for the data communicated to the player, as well as a digital signature that is sufficiently in compliance with Blu-ray Specifications (e.g. See System Description Blu-ray Disc Read-Only Format: Part Audio Visual Basic Specifications”). In a Blu-ray platform the manifest defines modifications that are to be made to a file system in the playback device. More specifically, the manifest file maps files located on the Blu-ray disc and in local storage to their relative positions and names in the virtual file system.

One or more embodiments described herein provide that methods, techniques and actions performed by a computing device are performed programmatically, or as a computer-implemented method. Programmatically means through the use of code, or computer-executable instructions. A programmatically performed step may or may not be automatic.

One or more embodiments described herein may be implemented using programmatic modules or components. A programmatic module or component may include a program, a subroutine, a portion of a program, or a software component or a hardware component capable of performing one or more stated tasks or functions. As used herein, a module or component can exist on a hardware component independently of other modules or components. Alternatively, a module or component can be a shared element or process of other modules, programs or machines.

Furthermore, one or more embodiments described herein may be implemented through the use of instructions that are executable by one or more processors. These instructions may be carried on a computer-readable medium. Machines shown or described with figures below provide examples of processing resources and computer-readable mediums on which instructions for implementing embodiments of the invention can be carried and/or executed. In particular, the numerous machines shown with embodiments of the invention include processor(s) and various forms of memory for holding data and instructions. Examples of computer-readable mediums include permanent memory storage devices, such as hard drives on personal computers or servers. Other examples of computer storage mediums include portable storage units, such as CD or DVD units, flash memory (such as carried on many cell phones and personal digital assistants (PDAs)), and magnetic memory. Computers, terminals, network enabled devices (e.g. mobile devices such as cell phones) are all examples of machines and devices that utilize processors, memory, and instructions stored on computer-readable mediums. Additionally, embodiments may be implemented in the form of computer-programs, or a computer usable carrier medium capable of carrying such a program.

System Description

FIGS. 1A-1D illustrate a system for including time based metadata information as a form of supplemental content in connection with the presentation of media content, according to one or more embodiments. Each of the embodiments described with respect toFIG. 1A-FIG. 1D, including components and programmatic processes described with each embodiment, may be used individually or in combination with one another. In particular, embodiments described herein enable the rendering of content, such as movies and/or television programs, to be enhanced with the display of relevant metadata information that pertains to events that are part of the content being watched. For example, the appearance of an actor in the movie or program may be supplemented with relevant metadata information that includes the name of the actor who is playing the character, as well as additional information about the actor or character. Likewise, (i) the presence of objects that appear in the content may be highlighted by metadata, particularly as to commercial products; (ii) the use of locations in the content may also be supplemented with information about such location; or (iii) music and/or a soundtrack playing in the audio background or as the primary audio track may be supplemented with metadata. Numerous other examples are provided herein for enhancing or supplementing the presentation of content, such as provided by movies and programs, with metadata information regarding the events that occur as part of the content being watched.

As used herein, metadata, in the context of movies and programs, refers to information and content that is displayed to the user, but which is not part of the content itself. Rather, such metadata is delivered to the user watching the movie or program as an additional or independent layer.

InFIG. 1A,system100 is described in the context of multiple operational environments. According to embodiments, asystem100 may be operational in one or more of the operational environments described.

System100 includesmetadata generation subsystem110 and time metadata service (or services)120. Themetadata generation subsystem110processes media files102 corresponding to movies and/or programs. The output of themetadata generation110 includes ametadata file111, which includes a combination of time stamps113 and associatedinformation115. Themetadata file111 includes the time stamps113 by which the sync time triggers131 can be programmatically or automatically activated by a playback device in order to cause the rendering of associated information or content items for a particular trigger.

Within the structure of the timeline for the media file, the time stamps113 are timed to coincide with the occurrence of a particular event in the media content. Each time stamp113 may signify the beginning of an event, and last for a duration that coincides with the ending of the event. For example, the introduction of an actor into a scene may signify the beginning of a particular time stamp113, and the actor's exit from the scene may signify the end of the event associated with the time stamp113. Similarly, a time stamp113 may be associated when music or a particular song is played in the background of the media content, and a time stamp113 may have the duration equivalent to the length of the song (e.g., the duration of time the music is playing). One frame or series of frames in the media content can be associated with multiple time stamps113. Furthermore, each time stamp113 is associated with a corresponding item of information orcontent115. Such information orcontent115 can be displayed over the movie/program content on a playback device, or on a separate, paired device, as a mechanism for providing the viewer with additional information.

In some variations, the information and content displayed to the user can be interactive. For example, the user may select a link presented with the information orcontent item115 as a mechanism for viewing additional information, or partaking in a supplemental activity (e.g. make online purchase).

In order to generate themetadata file111, one or more embodiments may include a combination of manual and programmatic resources comprising themetadata generation system110. In one embodiment, a programmatic/manual enhancement112 component may be used to facilitate operators to identify time stamps113, and associated information orcontent115. With reference to an embodiment ofFIG. 8, the programmaticmanual enhancement112 may include auser interface tool800 that displays ageneral timeline812 for a particular movie or program (e.g. the “Incredibles”). In an embodiment depicted, theuser interface tool800 enables an operator to create events for scenes of the media content. The events may correspond to an actor, playing a role in the content, appearing in a scene of the media content.

Theuser interface tool800 may also include one ormore event columns814. For example, theevent columns814 may include a list of actors, as well as other events relating to the media content.

As also shown inFIG. 8, thetimeline812 extends horizontally relative to theevent columns814. Within thetimeline812,various time stamps813 are identified by the operator each of which corresponds to one or more events in the event column. Eachtime stamp813 is represented by brackets that can vary in dimension to represent a duration of thetime stamp813 over the course of thetimeline812. In one implementation, an operator is enabled to (i) view a given scene or frame; (ii) view the exact time within the timeline in which the scene takes place; and (iii) have the ability to add new events, as well as to associate a particularsenior time stamp813 within an existing event in theevent column814.

According to some embodiments, themetadata generation subsystem110 can also include programmatic resources to automate the generation of some or all of the metadata. In particular,programmatic automation114 may be used to identify some or all of the time stamps113, as well as to generate some or all of the information or content items associated with individual time stamps113. In one implementation, theprogrammatic automation114 uses a script for the media content in order to determine when events occur in the media content (e.g. the introduction of a character into a scene), what the event pertains to (e.g. the character involved), and what the associated information orcontent item115 should include (e.g. the name of actor who plays the character of the event). As an alternative or addition to the script, closed-captioned text associated with the media content can be similarly used.

Still further, some embodiments may use image analysis to identify persons or objects appearing in the content. For example, individual frames of the media content may be analyzed for faces, which are then compared to a facial recognition database of actors, in order to determine who the actor is that appears in the content.

As additional resources, theinterface tool800 can include information such as the cast and the soundtrack list. When an operator creates atime stamp813, the operator can create a record that contains or references information about the particular event. The information or content items can be listed for the operator to edit (see list of records818).

The time basedmetadata service120 may utilize themetadata file111 in one or more operational environments. In particular, theservice120 may provide (i) media content that is packaged with time based metadata (as generated by the metadata generation subsystem110); or (ii) time based metadata in connection with media that is to be played back or rendered from a media source independent of theservice120.

In one operational environment, theservice120 combines the time based data113 andmedia content121 for broadcast and/or distribution. An intermediate service or step122 may retransmit or deliver the combined time based data113 andmedia content123 to aplayback device118. For example, a combined transmission time based metadata andmedia content121 may be supplied to a cable head end, which in turn broadcasts (or retransmits) the combined time based data113 andmedia content123 to its user population. As another example, the time based data andmedia content121 may be stored onto media (e.g. Blu-ray disc) and distributed through Blu-ray disc distribution channels. Other forms of media may also be used to store the combined time based data andmedia content121. For example, the combination of time based data andmedia content121 may be supplied as a download from a website. Still further, theplayback device118 may correspond to a set-top box, and the intermediate service or step122 may correspond to a cable head end that delivers the combined time based metadata andmedia123 to the set-top box in the appropriate format (e.g. EBIF format).

In another operational environment, theservice120 operates to supply the time basedmetadata125 independent of the media content that pertains to the metadata. Accordingly, for a particular media content, theservice120 supplies a delivery node or trigger126 with the time basedmetadata125 and media content identifier. Thedelivery node126 may include a broadcast ordistribution mechanism127, and an application orresource129 to integrate or otherwise combine themetadata125 with the underlying media content. In particular, themetadata125 may be structured to provide time stamps113 and information orcontent items115 all along a timeline that is based on the events that occur during the playback or rendering of the associated media content. Theintegration resource129 may synchronize themetadata125 with the media content in retransmitting, or otherwise making available (e.g. download), in enhanced form, themedia content130 that includes metadata.

In this way, the delivery node or trigger126 is able to deliver to aplayback device128, a combination of integrated content orenhanced content130 that includes the time based metadata and the associated content.

Various mediums of playback may be encompassed by an operational mode such as described withdelivery node126. For example, thedelivery node126 may correspond to a cable head end, and theplayback device128 may correspond to (or be provided through) a set-top box. Theintegrated content130 may be delivered as an EBIF formatted content.

In some variations, thedistribution node126 acquires the time based metadata for particular media content on a need to or real time basis. For example, in an implementation in which theplayback device128 is provided as or through a set-top box, one more embodiments enable the set-top box or cable head end to request the time based metadata based on user input. In such an implementation, thedistribution node126 may supply the media content identifier to theservice120, in order to receive the time basedmetadata125 for the corresponding media content.

In another operational environment, theservice120 supplies the time basedmetadata143 to aplayback device138 that receives the media content from another source. In particular, amedia source132 may correspond to, for example, a cable head end, or a website for downloading content, or a media that contains stored media. In the latter case, themedia source132 andmedia playback device138 may be provided from the same device of the user.

The user may operate theplayback device138 to rendermedia content135 from themedia source132. According to some embodiments, the end user can elect to enhance the media playback with the time basedmetadata143 provided throughservice120. In one implementation, the user may make arequest141 for time based metadata information from theservice120. Therequest141 may specify the media content the user is about to watch. As an alternative or variation, therequest141 may be generated from themedia source132, which responds to programmatic conditions and/or user input.

Upon receiving the request, theservice120 obtainssynchronization timing information131 from the media source. In an embodiment in which themedia source132 corresponds to, for example, a cable head end or website, theservice120 may query themedia source132 by specifying themedia ID133 for the media content that is to be rendered on theplayback device138. The media source in turn may provide theservice120 with thesynchronization timing information131. Thesynchronization timing information131 may include information that precisely indicates when a content item is to initiate, or where (e.g. which scene or frame) in the timeline of a particular content is being rendered at the time of the query. As an alternative or variation, theservice120 may query, for example, the user'splayback device138, which can include media corresponding to the media source. For example, in some Blu-ray environments, the Blu-ray player may communicate withservice120 over network in order to receive enhanced content in the form of time basedmetadata143. In some embodiments, the playback device may be a television.

In some implementations, themedia source132 may supply thetiming information131 as a one-time event for the particular media content, thus initiating a clock or counter on theservice120. Alternatively, thetiming information131 may be provided repeatedly, or on an ongoing basis. With this information, theservice120 supplies time basedmetadata143 to theplayback device138 in a manner that synchronizes the timeline of themetadata143 with that of the media content.

In another operational environment, theservice120 supplies enhanced content in the form of time based metadata and the associated media content to a user end device for playback. The user and device may correspond to, for example, a tablet or personal computer that connects to the service120 (or its affiliates or partners) via, for example, a website or web service. In some variations, theplayback device148 may correspond to, for example, a Blu-ray player or television which receives content and metadata from over a network.

In the operational environment depicted, the user of theplayback device148 makes amedia request147 from theservice120. Themedia request147 may identify an underlying media content, such as a movie or program. Theservice120 supplies enhancedmedia149 in that it includes both media and its associated time based metadata. Theservice120 may pre-synchronize the time based metadata with the underlying media content so as to transmit the content and metadata concurrently or at one time. Theplayback device148 may include programming resources that enable it to render both the media content and the information or content item included with the metadata.

As a variation, theservice120 may be configured to supply the time basedmetadata157 to a paireddevice160, in connection with corresponding media being rendered or played back on auser playback device158. Theuser playback device158 may correspond to a media source device, such as a Blu-ray player that plays back content from a disk, or tablet that plays back the content from the stored media file. Alternatively, theplayback device158 may receive themedia155 from theservice120. Still further, theplayback device158 may receive themedia155 from any media source, such as a cable head end or website.

In the example shown, themedia playback device158 receives the media from theservice120. Rather than communicating the time basedmetadata157 to themedia playback device158, themetadata157 may be communicated to the paireddevice160. In one embodiment, the paireddevice160 makes therequest161 for time based media data of an identified media content. The paireddevice160 may be configured to identify itself as being a paired device, rather than a playback (or primary playback) device. Alternatively, theservice120 may be configured to identify from therequest161 that the paireddevice160 is to receive the time basedmetadata157 information for an identified media content, in which the media content is played back on a different device or medium. In response to therequest161 theservice120 supplies a time basedmetadata157 to the paireddevice160.

In an embodiment in which theservice120 provides themedia155, the service controls transmission of both the media content and the metadata. As such, theservice120 is able to synchronize the delivery of metadata information to the paireddevice160, while transmittingmedia155 to theplayback device158.

In an embodiment in which themedia155 is transmitted from an alternative media source, such as a cable head end or website, theservice120 may receive synchronization timing information for media content being played back, so that the delivery of the time basedmetadata157 to the paireddevice160 is synchronized so that the timelines of the metadata and the media content are aligned. The synchronization timing information may be received by querying the media source (e.g. cable head end), as similarly described with other embodiments. Alternatively, themedia playback device158 may provide the necessary synchronization timing information, by initiating playback of the media content upon the paireddevice160 receiving the time basedmetadata157.

In an embodiment in which themedia155 is stored or otherwise local to theplayback device158, the paireddevice160 may be operated to request the time basedmetadata161 that is associated with the media content that is to be played back on thedevice158. For example, the user may operate the paireddevice160 to identify the media content that is to be played back on themedia playback device158. Alternatively, the twodevices158,160 may communicate with one another across, for example, a local communication port (e.g. Bluetooth or WiFi). Still further, theplayback device158 may be triggered to contact theservice120 in response to a playback request from a user (e.g. insertion or play command of a Blu-ray disc, play command for stored media file on a tablet etc.).

In response to receiving therequest161,service120 may query themedia playback device158 for synchronization timing information, in response to receiving therequest161. For example, a user may be associated with an account on theservice120. Once the user accesses theservice120, theservice120 locates and communicates with both of the user devices, thus enabling identification of the paired device for receiving the time basedmetadata157.

Still further, in some variations, the time basedmetadata157 may be communicated to the paireddevice160 from theplayback device158. For example theplayback device158 may synchronize both the time based metadata and the media content by receiving respective transmissions of one or both from an external source (e.g. content source or service120), and then retransmitting the metadata to the paireddevice160. The transmission of themetadata157 to the paireddevice160 can be done through a communication medium such as, for example, Bluetooth or other short radiofrequency wave communication channels.

FIG. 1B is a block diagram illustrating the architecture of the time based metadata service(s)120 according to some embodiments described withFIG. 1A. More specifically, According to some embodiments, the services described in connection with the use of the time based metadata service (s)120 may include one or more components, modules or services shown inFIG. 1B depending on the operating environment of the implementation.

With regard to embodiments described, time based metadata service(s)120 may also be configured to map multiple media playback devices to each other thereby enabling the devices to share metadata and/or associated content, even if one or more of the playback devices are not capable of communicating directly with each other.

As described withFIG. 1A, in certain embodiments, themetadata generation subsystem110processes media files102 corresponding to movies and/or programs. The output of themetadata generation subsystem110 includes ametadata file111 having a combination of time stamps113 and associatedinformation115 or content (FIG. 1A).

In another embodiment such as illustrated inFIG. 1B, themetadata file111 may be sent frommetadata generation subsystem110 while the associatedinformation115 or content may be provided from a second source, such as, for example, content service(s)195. The timebase metadata service120 can include a contentservice communication interface173 to receive content (e.g. movies or programs) from different sources, either in real-time or asynchronously with re-transmission of such content in enhanced form. As examples, the contentservice communication interface173 includes a program interface to receive content as file uploads from content providers. In other variations,content interface173 receives content in form of streaming or via broadcast medium (e.g. cable, satellite) or data communications only (e.g. questions for a quiz, consumer user's commentary and web posts, lists of friends related to the content).

In some embodiments, time metadata service(s)120 is configured to associate or map the content (e.g. as received from content service(s)195) to themetadata file111 and provide the newly associated content to one or more playback devices. For example, themetadata file111 may be received via a metadata services(s)communication170 of the time metadata service(s)120 and provided to a metadatacontent mapping component171. Themapping component171 then associates or maps themetadata111 with content (e.g., movie or programming content) received from the content service(s)195 via content service(s)communication173. In some embodiments, themetadata111 includes time stamps that correlate to scenes, frames and/or events in the content. The metadata tocontent mapping171 programmatically links the time stamps and the associated metadata to the run time of the corresponding content. For example, one file, or two linked files may be created that can be played simultaneously by a playback device in order to render the content (e.g. movie) and time based metadata at appropriate times (e.g. frame indicated by the time stamp).

Theservice120 communicates the mapped metadata/content to a playback device using, for example, adelivery node126 such as described above with respect toFIG. 1A. Other delivery nodes such as described withFIG. 1A may alternatively be used.

Theservice120 may also include a user accounts management service175. Theservice120 may associate individual users with accounts, which can incorporate pay for service functionality or other capabilities requiring the activation of an account. For example, functionality provided via theservice120 can be provided via a pay per view or monthly subscription basis. As an addition or alternative, account information can carry device and user profile information of a particular user. Such information may include (i) contact information about the user; (ii) one or more identifiers as to a type or types of playback device(s) the user is utilizing; (iii) historical data indicating the interactions of the user with a particular media file; (iv) linked accounts for other web services which the user may have registered to; and/or (v) preferences of the user regarding the output and display of the supplemental content.

According to some embodiments,service120 provides programmatic enhancements to enable third-parties (e.g. advertisers) to identify appropriate topics and information to supply with content. Theservice120 includes a programmatic service communication (PSC)176 that provides an interface for third-parties (e.g. advertisers, content owners, movie studios or sponsors) to research content and time based metadata from a library or collection of media files (e.g. movie library of studio). ThePSC176 can be used by the third-party (e.g. via service190) in order to specify additional metadata content, such as, for example, promotional metadata content. For example, the advertiser may use thePSC176 to specify overlay ads or content that relate to a particular commercial product, or, conversely, a particular commercial product approved for display with a particular media asset. The advertiser may also specify the media content, or supply information to have the media content selected programmatically. Theprogrammatic mapping component177 can use information provided from the advertiser to perform one or more of the following: (i) select the media content on which the advertiser content is to be displayed, (ii) select the scene(s) or events of the media content, and/or (iii) identify the particular time stamp within the media content that is to serve as the stamp for the supplied content.

As an addition or alternative, theprogrammatic mapping177 may include logic (algorithms) to scan media content, its associated metadata and/or time base metadata in order to identify one or more semantic categories corresponding to the media being viewed by the user. This information may be communicated to the programmatic service(s)190. The service can enable manual or programmatic selection of supplemental metadata content, in connection with time based metadata.

For example, if the user is watching a program or movie about dogs and/or one or more of the time stamps is identified as corresponding to dogs, mapping177 can communicate the information to the programmatic service(s)190. Theservice190 may then determine which supplemental content (e.g., advertisements regarding dog food, dog breeders, etc.) can be added to the media and provided to the user.

In certain embodiments,service120 may also include adevice identification service178 configured to identify a type of playback device utilized by a user. For example, thedevice identification service178 may determine that a user has access to, or is using, a particular device such as, for example, tablet computer, Blu-ray player, set-top box, etc., as well as the communication format for each device (such as determined by device communications179). In one or more embodiments, once the device information has been determined, the information may be associated with a user and stored using the user accounts management175. Device identification may also trigger theservice120 to operate in one of multiple possible computing environments, as described with prior embodiments.

Service120 may also includemapping tool180 configured for associating content received from programmatic mapping service(s)190 tometadata111. Anexemplary mapping tool180 is shown inFIG. 9A.

Referring toFIG. 9A, mapping tool900 includes aselector interface910 to enable a manual operator to filter the time stamps and/or metadata based on scenes. The manual operator may utilize the interface through, for example, theservice190. For example, theinterface910 may be generated or provided with themapping tool180. In using theinterface910, the operator can view the context of numerous scenes in a media content, view the time based metadata and its information, generate new time based metadata, verify the additional information and/or select other information, if any, that could be added. For example, if a particular song by an artist is playing in the given scene, additional content or advertisements corresponding to additional songs performed by that artist may be selected by the operator to later be provided to the user at the time of content playback. In another embodiment, songs and/or artists from a particular genre of the identified song and/or artist may be associated by the operator with the metadata which can be later presented to the user. The mapping tool900 may be used on any number of categories such as, for example, clothing, food, scene or shooting locations, scene mood, atmosphere or ambiance, objects present in screen, etc. As an additional example, the mapping tool may identify presence of commercial objects, or objects that are identifiable to a theme or topic (e.g. advertisement theme), or other factors, including user specified or user generated (“crowd-sourced”) factors.

Referring back toFIG. 1B,service120 may be configured to map one playback device to another playback device without each of the playback devices having capabilities of communicating directly with each other. Thus, a first playback device may deliverrelevant information115 throughservice120 to a second playback device or paired device. In certain embodiments,device communications179 determines a communication protocol of one device and may convert therelevant information115 from a first communication protocol utilized by the first playback device, to a second communication protocol, utilized by the second playback device.

In one embodiment, the imported metadata and related time stamps may be imported in a first format (e.g., the format in which they were created) and transferred to a playback as is. In another embodiment, the time stamps and metadata are programmatically processed to match the structure of themetadata file111 and/or the playback device to which the information will be sent.

FIG. 1C-FIG. 1D illustrate variations or additions to embodiments described inFIG. 1A and/orFIG. 1B.FIG. 1C illustrates an architecture in which theservice120 provides themedia149 to a playback device (e.g., playback device148). Theplayback device158 communicates anaudio fingerprint151 to theservice120. The audio fingerprint can be used to determine information about the media content, such as which media content is being played back, or alternatively, which scene is being played back. Theaudio fingerprint151 can be communicated to the service and used to identify the media content, or the particular portion or scene of the media content. For example, themedia ID133 can be determined from theaudio fingerprint151. This information is provided to theservice120 which, under some embodiments, can map or associate the relevant time stamps113 to the identifiedmedia ID133 and provides theInformation115 back to theplayback device148.

In another embodiment illustrated byFIG. 1D, an audio signal153 associated with themedia149 may be played on a first device (e.g., playback device158) and listened to on a paired device (e.g., paired device160) after which the paireddevice160 provides theaudio fingerprint151 to theservice120. As discussed above, theaudio fingerprint151 is identified and mapped to amedia ID133 and the information is provided to theservice120.Service120 then maps or associates the relevant time stamps113 to the identifiedmedia ID133 and provides theinformation115 back to the paireddevice160. In another embodiment, the relationship between the paireddevice160 andplayback device158 may be reversed such that the paireddevice160 transmits the audio signal to theplayback device158, which in turn, transmits the audio fingerprint to theservice120. In the various implementations shown, the paired device may be used to receive metadata, either from the service or from theplayback device158,160. The paired device can alternatively display portions of the content.

Methodology

FIG. 2 throughFIG. 5 illustrate various embodiments for providing time based metadata in connection with the rendering of a media content. Methods, such as described byFIG. 2 throughFIG. 5, may be implemented using components such as described with a system ofFIG. 1. Accordingly, reference to elements ofsystem100 is intended to illustrate suitable components or subsystems for performing a step or sub step being described.

FIG. 2 illustrates a method for generating time based metadata for a media content, according to embodiments. The media content may correspond to, for example, a movie, a television program, a short film, an animation, a homemade video, or other audio/video work or creation. The media content may be processed or analyzed (210) using programmatic resources that facilitate manual operation and/or automation. Accordingly, embodiments may include ametadata generation sub-system110 that includes a manual editor interface (e.g. SeeFIG. 8), and one or more programmatic resources that enable automated processing. The automated resources can include, for example, image analysis (214) which uses image recognition to identify persons and objects appearing in the content. As an addition or alternative, the automated process can include steps for analyzing script and/or closed-captioned analysis (216), which uses text content associated with the content in order to facilitate programmatic detection of events (e.g. the presence of a person in a scene, the use of a location, etc.).

The media content is processed in order to identify the occurrence of events for which supplemental information or content can be provided (220). Various kinds of events may be detected in the media content. These include, for example, events pertaining to (i) an actor or character (222), (ii) the use of objects (224), such as commercial objects, (iii) the playing of songs or supplemental media (e.g. television show in background) as part of the media content, and/or (iv) depiction of locations (226) (e.g. landmarks, geographic regions etc.).

The events of the media content may be individually associated with time stamps that are included as part of an associated metadata file (230). The metadata file is structured to reflect a timeline of the media content. The occurrence of an event in the media content may be signified by a trigger, which is marked in the metadata file to coincide with the occurrence of the event at the playtime of the media content. In this way, individual triggers are associated with corresponding events of the media content.

Once events are identified, the events may be linked to pertinent information or content (240). For example, time stamps that signify the presence of an actor or character in a scene of the media content may be linked to a data item that includes additional information about the actor, the character or the scene. Each time stamp can be linked to a data file that contains additional information pertaining to the event signified by the time stamp. The data file may be created programmatically and/or manually. For example, if an actor is signified by a time stamp, programmatic resources may automatically identify a profile of the actor from a movie database and include information (or a link) to the movie profile as part of the data item that is associated with the time stamp. As an alternative or addition, an operator may manually edit or create content for the data item that is associated with the trigger.

In some implementations,metadata generation subsystem110 processes a media content in order to detect events that occur as part of the media content. Themetadata generation sub-system110 associates the detected events with time stamps, which are in turn associated with timing information that signifies when the events occur during the playback of the media content. Additionally, the detected events can be associated with information or content items that supplement the content during portions that coincide with the placement of a corresponding time stamps (signifying when the event is taking place as part of the content). In one implementation, themetadata generation sub-system110 includes a manual editor interface (e.g. seeFIG. 8) that includes programmatic enhancements for facilitating an operator in (i) inserting time stamps along an appropriate timeline of the corresponding media content, (ii) associating time stamps with data items (e.g. information/content), orprogrammatic services190 orcontent services195 that pertain to (e.g. describe or supplement) the event identified by each of the time stamps.

As an alternative or addition, the process of event detection and subsequent data item association may be performed programmatically and with automation. In one implementation, individual frames that comprise the video portion of the media content are analyzed using image analysis. For example, individual frames are analyzed to detect faces, and to associate the faces with identities of persons (e.g. actors) from a database. Likewise, frames of the media content may be analyzed to detect objects (e.g. commercial objects) and to recognize the objects by type of brand using image analysis. The objects can then be mapped toprogrammatic services190 orcontent services195.

Once the metadata file is created with time stamps and linked information, the metadata file is made available to manual operators. In certain embodiments, the manual operators may utilize tools shown and described with respect toFIG. 9A throughFIG. 9C to check the accuracy of the time stamps, and the associations of those time stamps withrelevant information115 andprogrammatic services190 orcontent services195. As an addition or alternative, such tools may be used by third-party metadata providers.

The time stamps and metadata are then communicated to playback devices and/or media sources for use with playback of the associated media file (250). Depending on the implementation, the contents of the metadata file can be transmitted in real-time, to coincide with playback of the associated media content on a playback device. Alternatively, the contents of the metadata file can be transmitted and stored in a playback environment, for use when the associated media content is played back from a particular source or on a paireddevice160 at a later time. For example, in a cable/set-top box environment, the metadata file and/or its contents can be transmitted to a media source (e.g. cable head-end, website) where it is re-transmitted to a user playback device (e.g. set-top box).

Alternatively, the metadata may be transmitted to the media source (e.g. cable broadcast, website). The receiving media source may combine a media content transmission with the metadata, so as to transmit the media content and the metadata to the user for playback. According to some embodiments, the media source may include programmatic resources for (i) requesting the metadata that is to accompany the media content transmission, and (ii) combining transmission of the metadata with the media content, including synchronizing the delivery of metadata with the transmission of the media content along a timeline that is reflected by the media content.

Still further, the contents of the metadata file can be communicated to the playback device while the playback device receives the media content from another media source (e.g. cable/set top box, website). For example, the playback device may receive the metadata fromtime metadata service120, while receiving the coinciding media content from a media source (e.g. cable head end). Theservice120 may synchronize the transmission of the metadata using reference timing information for the playback of the media content (e.g. start time). Alternatively, the playback device may synchronize the receipt of metadata from one source with the receipt of media content from another source in creating an enhanced playback containing both the media content and the time based metadata.

In some variations, the contents of the metadata file may be transmitted and stored by, for example, a user of a playback device. The stored file can be accessed and used when the playback device renders the associated media content or file.

As another variation or alternative, the metadata file, or its contents, may be transmitted to a paired device of a user, for use in conjunction with rendering of the associated media content on a playback device. For example, the metadata file may be transmitted over a network to a tablet device that the user can hold and view, while watching the associated media content on a set-top box or Blu-ray environment.

Numerous variations to how the generated metadata is communicated and used are described with, for example, embodiments ofFIG. 3 throughFIG. 5.

FIG. 3 illustrates a method for providing media content that is packaged or integrated with time based metadata, according to an embodiment. As described with an embodiment ofFIG. 2, a metadata file is generated for a particular media content (310). The metadata file may be structured to include timing information that identifies the occurrence of events in the runtime of the corresponding media content. The metadata file may be provided as a separate file from the media content, but the packaging of the metadata with the media content can inherently include programmatic resources that can trigger the metadata file and the media content to synchronously initiate their respective playback, so that the timeline for each of the metadata and the media content are aligned during the playback of the media content.

In an embodiment ofFIG. 3, the metadata is packed with the media file and then delivered (320). In one implementation, the combined media and metadata are transmitted to a media source, such as a broadcast source (e.g. cable head end in distribution) or website, where the data is retransmitted to a playback device of the user (322). In another implementation, the combined media and metadata are transmitted to the media playback device of a user without an intervening distribution of broadcast source (324). For example,service120 may be provided through a website that can deliver the combined media and metadata package to an end user operating a computing device, such as a laptop, personal computer, game station, or Internet television. As another variation, the combined metadata and media may be provided on a storage medium, such as on a Blu-ray disc, or flash memory device (e.g. accessory device, tablet) (326).

FIG. 4 illustrates a method for providing time based metadata to a media source, for transmission to a media playback device with media content, under an embodiment. A method such as described withFIG. 4 may be implemented to enable a media source to deliver or otherwise provide media content in an enhanced form (as provided by time based metadata). In response to a media playback request to a user, a media source (e.g. broadcaster) may request metadata for the requested media content (410). In an alternative implementation, the request may be initiated by the media playback device of the user, which may use a network connection to communicate with theservice120 separately from the media source. Theservice120 may respond to the request by delivering metadata associated with the media content identified in the request to the media source (420). The media source may be provided with programmatic resources to synchronize the timing of the metadata with the playback of the media content (430). For example, the media source may include an application that synchronizes the metadata with the playback of the media content, so as to transmit both metadata and media content in synchronous fashion to the media playback device.

In some variations, the media playback device may request and receive the metadata independently from the media source. For example, the media playback device may request the metadata from theservice120. Theservice120 may acquire the synchronization timing information from the media source, and deliver metadata in real-time to the playback device. Alternatively, the playback device may receive the metadata in real-time or synchronously, and include programmatic resources for synchronizing the playback of the metadata with the associated media content. Numerous such variations are possible.

FIG. 5 illustrates a method for providing time based metadata to a playback device that receives the media content from a media source, according to another embodiment. In one implementation, a user requests metadata for a media content that the user is to playback (510). The media content may, for example, reside on the user's device, or be provided from a separate media source (e.g. cable or website).

Theservice120 may respond to the request in real-time, so as to transmit metadata concurrently and in-sync with playback of the media content on the playback device. In responding to the request,service120 may acquire synchronization timing information in order to transmit the media content in sync (520). The synchronization timing information may correspond to, for example, the exact time that the media content was initiated, or the precise frame or portion of the media content that is rendered at a reference time. In one implementation, theservice120 acquires the synchronization timing information from the media source (522). In another implementation, theservice120 acquires the synchronization timing information from the media playback device (524).

In some variations, the media source may reside on a user device. For example, the media source may correspond to a Blu-ray player. The synchronization timing information may be communicated from the Blu-ray player of the user.

Theservice120 may then transmit the metadata information in real-time to the device of the user (530). In one embodiment, the metadata is transmitted to the media playback device of the user (532). In a variation, the metadata is transmitted to a paired device of the user (534). The paired device may correspond to, for example, a tablet, smart phone or laptop, while the media playback device may correspond to a set-top box and television, personal computer, Blu-ray player or Internet television. As an example, theservice120 may maintain an account for the user from which it maintains information to identify and locate the media playback device and the paired device on a network. In another embodiment, theservice120 can broker a request. For example,service120 may provide response actions to a user interacting with the media, either through the playback device or paired device and connect the user toprogrammatic services190 orcontent services195.

FIG. 6A throughFIG. 6F illustrate examples of media content presentation that includes metadata content, according to some embodiments described herein. The enhanced presentation of media content may be implemented using, for example, a system such as described with an embodiment ofFIG. 1. Accordingly, reference is made to elements ofFIG. 1 for purpose of illustrating suitable components or elements for creating metadata presentations as shown.

FIG. 6A illustrates a media content presentation that is enhanced with presentation of time based metadata, in accordance with one or more embodiments. In an embodiment described, the media content is a movie, and the metadata that is presented with the movie includes actor names for characters that appear in individual scenes. In the example provided, a given scene is marked by the presence of three actors. Each are identified bymetadata items602. Themetadata items602 may also include additional information, such as the name of the character that the actor is playing in the given content item. Additionally, a song or track that starts playing in a scene may include aninformational tag604. With reference to prior embodiments, the metadata file may be provided in form of a file that includes time based triggers that coincide with the timeline of the movie. At scenes (e.g. frames) that coincide with introduction or presence of an event (e.g. an actor, a song playing), the informational tag is presented as an overlay.

FIG. 6B illustrates the information tag being interactive or able to include supplemental information, such as thefilmography606 of an individual actor. In some implementations, the user is able to interact with the information tags that appear over the media content. The user interaction may, for example, be the user operating the playback or paired device requesting additional or supplemental metadata from the metadata source (which may be the media source or the service120). In another embodiment, the user interaction may be a request to receive (e.g., video on demand, purchase a digital copy, or Blu-ray copy etc.) and/or view additional media content the selected actor(s) is in.

FIG. 6C illustrates additional examples of metadata content that can be overlaid with the media content. InFIG. 6C, themetadata content612 can occupy a majority of the screen space, so as to present additional information about the actor (or event that is being described).

FIG. 6D illustrates an example in which metadata includes supplemental information or services, such as an interactive feature that enables the user to purchase a product associated with an event of media content (or the media content itself). In the example shown, the user is able to purchase a soundtrack by interacting withmetadata content622. Themetadata content622 can be displayed to the user at, for example, the occurrence of a relevant event, such as the playback of a song that appears on the soundtrack.

FIG. 6E andFIG. 6F illustrate the user of metadata content that coincide with product placement within a media content. In the example shown, the product placement event coincides with an actor of a television show riding in a sports car. Themetadata content624 identifies, for example, the make and model of the car. Additional information, such as information about thecharacter626 or program, may also be displayed as an overlay of the content.

FIG. 7A throughFIG. 7F illustrate examples of media content presentations that include metadata content for playback and/or paired devices, according to some embodiments described herein. InFIG. 7A andFIG. 7B, a user is able to login to a service that is provided in connection with the playback of the media content. For example, the user may include an account withservice120, and have the ability to login in order to receive metadata in connection with media content that is either received from the service, from a different media source (e.g. cable or website), or played back from a local source (e.g. Blu-ray player).FIG. 7C illustrates an implementation in which the metadata presentation over the media content is personalized. In variations, the metadata content presentation may also be configured or tailored for the particular user based on, for example, preferences of the user. For example, as shown inFIG. 7C, the user preference may be to have all spoken dialogue displayed in real time.

FIG. 7D throughFIG. 7F illustrate presentations that supplement the media content independently of the media content presentations. Such supplemental presentations may be displayed on a playback device in lieu of, for example, the media content (e.g. the media content may be paused), or displayed on a paired device (e.g. tablet) while the user watches the media content on a media playback device (television). Accordingly, some implementations provide for the supplemental metadata content to be provided independent of the occurrence of events or triggers. For example, the supplemental content showing the website content may be accessible to the user throughout the playback of the media content.

InFIG. 7D andFIG. 7E, the presentations include, for example, website content which can display various forms of content about individual characters of a program. InFIG. 7F, the supplemental content includes online auction or e-commerce content fromprogrammatic services190 orcontent services195. Various alternatives to the form of supplemental content that can be displayed are possible.

Metadata Placement and Development

Various parties may utilize an interface or tool to develop metadata themes of content for inclusion with a media content. For example, an operator of themetadata generation sub-system110 may place metadata according to a default criteria (e.g. identify all actors, songs, bibliography etc.). Other operators, such as users of theservice190 may place metadata by themes. For example, the operators may be provided an interface (e.g. SeeFIG. 9A andFIG. 9C) where metadata is identified, and commercial or user-selected themes can be specified for scene(s). The operator can view the content, the product placement, see tags associated with placed content etc. to develop product placement, themes and other content.

For example, referring toFIG. 9C, a user may be able to check the relevant information in a particular scene such as items of clothing worn by the actors, information regarding the character played by the actor, and/or a story line associated with the scene. Additionally, the user interface tool ofFIG. 9C may presentmetadata information115 as an overlay to themedia file102. As such, a manual operator may verify the accuracy of the metadata mapping to the time stamps113. Additionally, the manual operator may also determine the relevance of the information to the content of the media. In such embodiments, the user interface tool is dynamic, and the visual elements of the interface appear and disappear as the timeline of themedia115 is traversed.

Referring toFIG. 9C, a web based tool may be provided to enable the manual operator to selectively verify information in a particular scene. For example, the manual operator may be able to select one or more categories of the supplemental content to verify (e.g., products, portrayals etc.). As a result, only information corresponding to the selected categories will be displayed. Additionally, the manual operator may be able to add events and/or content using this tool.

Second Screen Examples

FIG. 10A-10B illustrate examples of media content interfaces generated using metadata content for a paired device, according to embodiments described herein. As discussed above regardingFIG. 1, a paireddevice160 may receive time based metadata, such as frommetadata service120, in connection with corresponding media being rendered or played back on auser playback device158. Embodiments provide for the programmatic services on the paired device1000 (e.g. a phone, tablet or personal computer) receiving metadata corresponding to a media being rendered or played back on a main playback device, and providing selectable features to enable a user to access supplemental content related to the received metadata in aninterface1000. The interfaces ofFIGS. 10A and 10B may be in lieu of or in addition to any interfaces displayed on the main playback device. Anindicator1002 may be provided to specify to a user that the features being displayed include content relating to the media playback, relative to the beginning and end of the media.

The interfaces ofFIG. 10A-10B include a plurality ofselectable features1004 that relate to persons or objects appearing in the media. The features include graphical and/or text content that describe persons or objects appearing in the media being rendered or played back on the user playback device, and further describe supplemental content which may be generated and displayed in response to a user selection of the feature. For example, the graphical content of the feature may include an image from the media being rendered or played back on the user playback device, and when selected may cause the interface to be removed and a supplemental content (e.g. graphic and/or text content) relating to the media and/or feature to be displayed on the paired device. For example, generating the supplemental content may include, in response to a user selection of a selectable feature, causing a larger version of the image and a text caption to be displayed on the paired device.

In the embodiment ofFIG. 10A, a series offeatures1004,1006 and1008 are displayed on theinterface1000. Embodiments provide for the features to be selectable by a user. For example, where the paired device is a mobile computing device that includes a touch screen display, and theinterface1000 is provided on the touch screen display, in an embodiment the user may select a particular feature by touching the portion of the screen corresponding to the selectable feature. In such embodiments,service120 can detect selection of the feature and provide supplemental content on the main playback device and/or on the paireddevice1000 corresponding to the selected feature. The supplemental content may include, for example, information about actors or objects that relate to a description of the selected feature, and the information may be displayed on the paired device. For example, in response to selection offeature1006, theservice120 may cause content to be displayed on the paired device relating to characters who appear in the scene and/or the graphic of the feature.

FIG. 10B illustrates a media content interface in an embodiment of the invention. A series ofselectable features1010 are displayed on the interface. When selected, theselectable features1010 may cause supplemental content to be displayed on the user playback device and/or the paired device. The example of solid-square features1012, which display solid squares without any graphical or text content, may be replaced withselectable features1010 by programmatic services on the paired device including graphical and/or text content based on time based metadata such as supplied bymetadata services120. Similarly, theselectable features1010 may be replaced by solid-squares features1012 (or other selectable features) based on the time based metadata.

Link Sharing

FIG. 11 illustrates a method for providing media data to be shared or published, according to an embodiment. A method such as described by an embodiment ofFIG. 11 can be implemented using components such as described with other embodiments (e.g., a system ofFIG. 1). Accordingly, reference may be made to elements ofFIG. 1 for purpose of illustrating suitable components for performing a step or sub-step being described.

With reference toFIG. 11, a user identifies a movie segment to share (1110). Referring toFIG. 1, for example, a user can specify input that corresponds to the movie segment usinguser playback device158 and/or on the paired device160 (1112). The input can identify the start/finish times, in reference to a movie's runtime. In this way, the timing specification can designate a beginning time and an end time of the movie segment.

In some variations, the user input can identify a predefined segment (1114). The predefined segment may have a set of predefined features, for example a predefined start/finish time.

Theuser playback device158 and/or the paireddevice160 may respond to the user input, by, for example, communicating the identified segment to a service or set top box, etc. (1116). The paired device1160 can, for example, determine, from user input, the start/end times for a movie, and then communicate the start/end times (along with an identifier of the movie) to the service or set top box.

In some variations, for example, a user can operate the paireddevice160 in order to provide input that identifies the desired segment (1118). For example, a user's interaction with a second screen (e.g., touch screen) can be used to identify a movie segment or its start and/or end times. In such an embodiment asecond screen interaction1118 may be provided to a user in response to determine, for example, a predefined segment. Various other input mechanisms and/or interfaces can be used to enable a user to specify an input corresponding to a segment of the movie.

After responding to a user input and/or second screen interaction, a link is generated (1120). The link may be generated to include, for example, data elements that provide or correspond to one or more of the following: (i) data that identifies a location of source file for the movie or movie segment (1122), (ii) start and end times for the segment (1124), (iii) time-based metadata that is associated with the segment of the movie (persons and/or objects that appear or are referenced in the identified segment) (1126). In some variations, an identifier for the movie segment can be identified by the link. As an example, the link can be a selectable hypertext link that, when triggered, causes the identifiedmovie segment1110 to be played or otherwise displayed. The generated link can be published (1130). This can include publishing the link on a social networking account (1132) (e.g. Facebook, or any social network known in the art) or sharing the link through messaging (1134). For example, the link can be shared by sending the link by e-mail, text message, or another direct form of communication.

FIG. 12A-12E illustrate example interfaces for providing media data to be shared or published, according to an embodiment. In the example provided, the interfaces can be provided for use on, for example, a mobile computing device or tablet. For example, inFIG. 12A, theinterface1210 can enable a user to provide inputs through interaction with, for example, amobile computing device1200. In some examples, the mobile computing device (or tablet or other computing device) can operate as a second screen.

In the example provided, the user triggers aselectable feature1210. Thefeature1210 can programmed to, for example, identify a segment, or alternatively provide user input that identifies a start/finish for the movie segment. In the example provided, once the segment is identified, time-based metadata (e.g., related shopping panel1214) can be displayed to the user. The user input can also be used to enable the user to publish the clip1215 of the segment. For example, the user can communicate the clip1215 to the service, either with or without, for example, the time-based or rich metadata.

FIG. 12B illustrates an interface where a user initiates sharing or publishing of the clip1215 by preparing an email message. The user may designate a recipient of the email message. A subject line can optionally be automatically pre-populated based on, for example, metadata associated with the selected segment or movie.

FIG. 12C illustrates an interface where the user may view an image for inclusion in the email message. The image may, for example, correspond to a frame from the clip.

InFIG. 12D, the interface can enable the user to provide commentary, such as a text comment, with the link that is shared. For example, the text commentary can be provided with an email message. For example, a user may provide a comment (“This is my favorite moment in this show”) which will be included in the email message. In variations, the interface can enable the user to provide the commentary in connection with a link that is published on, for example, a social networking account.

FIG. 12E illustrates an interface where the user has shared the clip by sending the email message. The message included an image corresponding to the clip, a text comment prepared by the user, and the published link.

Computer System

FIG. 13 is a block diagram that illustrates a computer system upon which embodiments described herein may be implemented. For example, in the context ofFIG. 1A,system100 may be implemented using a computer system such as described byFIG. 13. In particular, service120 (or portions thereof) may be implemented on a computer system such as shown and described withFIG. 13. Thecomputer service120 may communicate with playback devices, paired devices and/or distribution sources of media content and/or associated metadata using a network such as the Internet.

Additionally, ametadata generation sub-system110 may be implemented using a computing system such as shown and described byFIG. 13.

In an embodiment,computer system1300 includesprocessor1304,main memory1306,ROM1308,storage device1310, andcommunication interface1316.Computer system1300 includes at least oneprocessor1304 for processing information.Computer system1300 also includes amain memory1306, such as a random access memory (RAM) or other dynamic storage device, for storing information and instructions to be executed byprocessor1304.Main memory1306 also may be used for storing temporary variables or other intermediate information during execution of instructions to be executed byprocessor1304.Computer system1300 may also include a read only memory (ROM)1308 or other static storage device for storing static information and instructions forprocessor1304. Astorage device1310, such as a magnetic disk or optical disk, is provided for storing information and instructions. Thecommunication interface1316 may enable thecomputer system1300 to communicate with one or more networks through use of thenetwork link1320.

Computer system1300 can includedisplay1312, such as a cathode ray tube (CRT), a LCD monitor, and a television set, for displaying information to a user. Aninput device1314, including alphanumeric and other keys, is coupled tocomputer system1300 for communicating information and command selections toprocessor1304. Other non-limiting, illustrative examples ofinput device1314 include a mouse, a trackball, or cursor direction keys for communicating direction information and command selections toprocessor1304 and for controlling cursor movement ondisplay1312. While only oneinput device1314 is depicted inFIG. 13, embodiments may include any number ofinput devices1314 coupled tocomputer system1300.

Embodiments described herein are related to the use ofcomputer system1300 for implementing the techniques described herein. According to one embodiment, those techniques are performed bycomputer system1300 in response toprocessor1304 executing one or more sequences of one or more instructions contained inmain memory1306. Such instructions may be read intomain memory1306 from another machine-readable medium, such asstorage device1310. Execution of the sequences of instructions contained inmain memory1306 causesprocessor1304 to perform the process steps described herein. In alternative embodiments, hard-wired circuitry may be used in place of or in combination with software instructions to implement embodiments described herein. Thus, embodiments described are not limited to any specific combination of hardware circuitry and software.

Although illustrative embodiments have been described in detail herein with reference to the accompanying drawings, variations to specific embodiments and details are encompassed by this disclosure. It is intended that the scope of embodiments described herein be defined by claims and their equivalents. Furthermore, it is contemplated that a particular feature described, either individually or as part of an embodiment, can be combined with other individually described features, or parts of other embodiments. Thus, absence of describing combinations should not preclude the inventor(s) from claiming rights to such combinations.

Claims (20)

What is claimed is:

1. A method comprising:

receiving media content;

causing playback of at least a portion of the media content;

determining, based on the playback of at least the portion of the media content, an audio fingerprint;

transmitting a request for metadata, wherein the request comprises the determined audio fingerprint;

receiving, in response to the request, time-based metadata associated with the media content, wherein the time-based metadata comprises a plurality of time stamps that are triggerable at different instances of a timeline of the media content, and wherein one or more time stamps of the plurality of time stamps are associated with corresponding metadata content;

receiving, based on the playback of the at least the portion of the media content, an indication of occurrence of one or more events in the timeline of the media content; and

triggering, based on the indication of occurrence of the one or more events, a sequence of time stamps of the plurality of time stamps, wherein the triggered sequence of time stamps is synchronized with the occurrence of the one or more events in the timeline of the media content;

wherein triggering the sequence of time stamps causes determining associated information for one or more time stamps of the sequence of time stamps, and rendering at least a portion of the metadata content based on the associated information; and

wherein the associated information for one or more of the time stamps of the sequence of time stamps comprises a link to the portion of the metadata content.

2. The method ofclaim 1, wherein the rendered portion of the metadata content comprises an interactive user feature configured to be displayed for user input at a particular instance in the timeline of the media content.

3. The method ofclaim 2, further comprising linking the interactive user feature to a service to enable the user to purchase a preselected item from an online store.

4. The method ofclaim 1, wherein the rendering at least a portion of the metadata content comprises causing the portion of the metadata content to overlay the playback of the media content, wherein the metadata content is associated with a soundtrack of the media content.

5. The method ofclaim 1, wherein the metadata content comprises an advertisement.

6. The method ofclaim 1, wherein at least a portion of the associated information is associated with a geographic location of a storyline of the media content.

7. The method ofclaim 1, wherein the media content is packaged with the time-based metadata for delivery to media playback devices.

8. The method ofclaim 7, wherein the packaged media content originated from a head end, and wherein the media content is a broadcast.

9. A method comprising:

receiving media content;

causing playback of at least a portion of the media content;

determining, based on the playback of at least the portion of the media content, an audio fingerprint;

transmitting a request comprising an indication of the determined audio fingerprint;

receiving, in response to the request, time-based metadata corresponding to a timeline of the media content, wherein the time-based metadata comprises a plurality of time stamps that are triggerable at different instances of the timeline, wherein one or more time stamps of the plurality of time stamps are associated with corresponding metadata content;

receiving, based on the playback of the at least the portion of the media content, an indication of occurrence one or more events in the timeline of the media content; and

triggering, based on the indication of occurrence of the one or more events, a sequence of time stamps of the plurality of time stamps, wherein the triggered sequence of time stamps is synchronized with occurrences of the one or more events in the timeline of the media content, wherein the triggering the sequence of time stamps causes at least a portion of the metadata content to be rendered.

10. The method ofclaim 9, wherein the media content is received from a first source and the time-based metadata is received from a second source distinct from the first source.

11. The method ofclaim 9, wherein the determining an audio fingerprint comprises recording an audio output associated with the playback of the media content.

12. The method ofclaim 9, wherein the one or more events comprise a scene change in the playback of the media content.

13. The method ofclaim 9, wherein the one or more events comprise an audio change associated with the playback of the media content.

14. The method ofclaim 9, wherein the metadata content comprises an advertisement.

15. The method ofclaim 9, wherein the metadata content is contextually related to the media content.

16. A method comprising:

receiving media content;

causing playback of at least a portion of the media content;

determining, based on the playback of at least the portion of the media content, an audio fingerprint;

transmitting a request comprising an indication of the determined audio fingerprint;

receiving, in response to the request, time-based metadata comprising a plurality of time stamps associated with a timeline of the media content, wherein one or more of the plurality of time stamps are associated with corresponding metadata content;

receiving, based on the playback of the at least the portion of the media content, an indication of occurrence of one or more events in the timeline of the media content;

determining, based on the indication of occurrence of one or more events in the timeline of the media content, one or more time stamps of the plurality of time stamps; and

causing, based on the determined one or more time stamps, at least a portion of the metadata content to be rendered.

17. The method ofclaim 16, wherein the media content is received from a first source and the time-based metadata is received from a second source distinct from the first source.

18. The method ofclaim 16, wherein the determining an audio fingerprint comprises recording an audio output associated with the playback of the media content.

19. The method ofclaim 16, wherein the one or more events comprise a scene change in the playback of the media content.

20. The method ofclaim 16, wherein the one or more events comprise an audio change associated with the playback of the media content.

Images