TECHNICAL FIELDThe present application relates generally to the technical field of media packages; and more specifically to delivery and management of dynamic mixed media packages.
BACKGROUNDThe entertainment industry does not exercise complete control over their raw assets delivered through digital distribution channels. Conventional delivery of these raw assets suffers from several limitations, which is perhaps most obvious with conventional distribution of music content. First, conventional digital music delivery over the Internet or other digital distribution channels (e.g., digital radio broadcast, Compact Disc, Audio or Video on Demand services through cable, terrestrial broadcast or via satellite, cellular phone networks, etc.) is limited to delivery of an individual or delivery of multiple individual tracks. This limitation constrains a consumer to single track playback. Second, an individual audio track is separate from other related assets. Third, the format of audio tracks is often proprietary (i.e. non-standard) and only in one format or resolution. These limitations hamper control over their raw assets and the ability of music industry members to innovate with respect to their raw assets. Although described in the context of music content, these same issues plague other digital content spaces (e.g., eBooks, videos, games, image data, etc.).
The conventional delivery model restrains the entertainment industry owners' and creators' ability to innovate. The conventional delivery model relies heavily on an intermediate entity (i.e. content aggregators and distributors, download store front and network operators, content delivery and playback software/device manufacturers etc.). The intermediate entity that delivers content separates the owners and creators of the content from their customers. This separation interferes with the owners' and creators' ability to collect helpful statistical data and interact closer with their end customers. Instead, distributors and software/hardware providers (e.g., Apple iTunes® music service, Real Network Rhapsody® music service, etc.) substantially control the consumer experience of consuming content via the Internet, digital media files, and media streams. Allowing these intermediate entities to possess control over distribution and the consumer interaction hinders progress in product differentiation by members of the entertainment industry. Lastly, the intermediate entities have the best abilities to influence the consumption behavior and experience of consumers since they are the closest to the consumers.
BRIEF DESCRIPTION OF THE DRAWINGSSome embodiments are illustrated by way of example and not limitation in the figures of the accompanying drawings in which:
FIG. 1 depicts an example system that bundles media together into a dynamic mixed media package.
FIG. 2 depicts an example system that propagates package modifications to media consumers.
FIG. 3 depicts an example of a general structure of a dynamic mixed media package.
FIG. 4 depicts a flowchart of example operations for creating a dynamic mixed media package.
FIG. 5 depicts a flowchart of example operations for querying an entity for a package modification responsive to a query event.
FIG. 6 depicts a flowchart of example operations for handling submissions.
FIG. 7 depicts a flowchart for example operations to search for derivative media.
FIG. 8 depicts a flowchart for example operations for handling a package query for a dynamic mixed media package.
FIG. 9 depicts an example system that maintains package modifications.
FIG. 10 depicts an example implementation of a media query module.
FIG. 11 depicts an example interface for a dynamic mixed media package player.
FIG. 12 depicts a diagrammatic representation of a machine in the example form of aprocessing system1200 within which a set of instructions, for causing the machine to perform any of the functionality discussed herein, may be executed.
FIG. 13 depicts a flowchart for example application of a composite set of rules to media that may be performed inblock607 ofFIG. 6 orblock709 ofFIG. 7.
FIG. 14 depicts an example revenue stream from derivative media.
FIG. 15 depicts an example format of a dynamic mixed media package in accordance with the Multimedia Content Description Interface (“the MPEG 7 standard”).
FIG. 16 depicts an example presentation of media from an example dynamic mixed media package.
DETAILED DESCRIPTIONOverviewAn innovative experience can be provided with a dynamic mixed media package, as well as instituting a media delivery and management model that leverages networks. An encompassing and comprehensive media experience can be presented to the consumer with a package that renders mixed media (e.g. video, additional audio, interviews, lyrics, image artwork, etc.) related to a seed media (e.g. a recording of a particular song), which may be an individual work or a collection of works. The seed media can be associated with supplemental media as a mixed media package (e.g., a file may have pointers to the seed media and the supplemental media, a file may actually contain the seed media and the supplemental media, etc.). Identifying information is generated and associated with the seed media that allows management and tracking of the seed media. Identifying information may be embedded into the seed media (e.g., watermark), derived from the seed media (e.g., a hash value generated, fingerprint data generated, etc.). Identifying information may also be generated and associated with the supplemental media. In addition, a reference to a package query checkpoint is embedded into the package. Accessing the package query checkpoint (or multiple package query checkpoints), such as an IP address of an online network server, allows dynamic modification/updates to be indicated with the dynamic mixed media package.
With a dynamic mixed media package, a user purchases an experience or level of service instead of an individual media file. In addition, the dynamic mixed media package is a product and service with value beyond the individual media files that can expand during the life span of the dynamic mixed media package through the addition of newly released, updated or altered related rich media content. Examples of such expansion include, but are not limited to, additional video clips, music tracks, streaming audio or video, live concert video, music news, editorial reviews, song lyrics, alternate versions of a track or lyric, karaoke versions and lyrics synchronization data, photographic or image art data, ring tones, data usable to categorize and navigate content (e.g., genre, tempo, mood, release year, country of origin, etc.), and user generated content (e.g., user created music videos, user comments, user re-edited videos or altered soundtracks for videos, user remixes of audio tracks, etc.). The delivery of additional dynamic media can be done on a promotional basis, tied to commerce or advertising, by contest with consumer participation, etc. The media assets (i.e., seed media and/or supplemental media) may also be upgraded or downgraded in quality of size, supported software codecs and bit rates, rendering limitations (e.g., audio only, audio and image, audio and video), etc., for various reasons, such as to fit a particular playback device.
The subsequent description includes illustrative systems, methods, techniques, instruction sequences and computing machine program products that embody the present invention. For purposes of explanation, numerous specific details are set forth in the following in order to provide an understanding of various embodiments of the inventive subject matter. It will be evident, however, to one skilled in the art that embodiments of the inventive subject matter may be practiced without these specific details. For instance, examples are described below in the context of being performed on a single machine. Multiple machines, however, may be involved in tracking seed media and collecting package modifications. Furthermore, the term media is used frequently throughout within the context of audio and video. The term media, however, should not be limited to these particular examples of media, and include many other types of media, such as photographic images, art images, literature, streaming media, etc. In general, well-known instruction instances, protocols, structures and techniques have not been shown in detail.
For the purposes of this specification, “processing system” includes a system using one or more processors, microcontrollers and/or digital signal processors having the capability of running a “program”, which is a set of executable machine code. Processing systems include communication and electronic devices such as (but not limited to) cell phones, music players, personal data assistants (PDAs), automotive entertainment systems and consumer electronics products designed for use at home. Processing systems also include computers, or “computing devices” of all forms (desktops, laptops, palmtops, etc.). A “program” as used herein, includes user-level applications as well as system-directed applications or daemons.
FIG. 1 depicts an example system that bundles media together into a dynamic mixed media package. A media owner or creator (e.g., author, recording company, production company, etc.) generates seed media (e.g., video, audio, images, etc.) and transmits the seed media to amedia packager104. Themedia packager104 also receivessupplemental media109 and113 from amedia service provider107 and amedia publisher111, respectively. Supplemental media may be previews, lyrics, trailers, reviews, artwork, etc. For example,media service provider107 may provide advertisements and the media publisher may provide artwork.
Themedia packager104 generates data for management of the seed media. For example, themedia packager104 generates a watermark, fingerprint (e.g., audio fingerprint, video fingerprint, image fingerprint, tandem fingerprint, etc.), and/or hash (e.g., using the MD5 hash function, a SHA hash function, etc.) with the seed media. The media packager, or another entity, can later use the generated management data to perform various management operations with the data, such as track use of the seed media, identify the seed media in a derivative work, etc. Of course, it is not necessary for the media packager to generate the management data. For instance, the media owner/creator may generate the data and communicate the data and/or location of the data to themedia packager104. Themedia packager104 may also generate management data with thesupplemental media109 and113.
Themedia packager104 assembles the seed media103 and thesupplemental media109 and113 into amixed media package115. Themedia packager104 writes into themixed media package115, perhaps in a header, management data that has not been incorporated into the media and a reference to themedia packager104 as a package query checkpoint. Themedia packager104 also writes structural information and directives for presenting the media into the package header and/or section headers. Instances of themixed media package115 are then delivered via a network102 (e.g., LAN, WAN, Internet, cellular networks, etc.) or through comparable distribution processes for non-network connected devices to media consumers117a-117c,directly or indirectly. For example, the media consumers117a-117cmay have purchased a membership from themedia packager104, purchased an instance of thepackage115 from themedia packager104, purchased an instance of the package from another entity that receives the instance of thepackage115 from themedia packager104 and forwards the instance of thepackage115 to a media consumer, etc.
The media consumers117a-117ccan then transmit or transfer the instance (or a copy of the instance) to another device. InFIG. 1, themedia consumer117ctransmits the received instance to a portableaudio media player123, a mobile phone125, and an automobile entertainment device127. The transfer of the instance to these devices may involve additional operations. For example, themedia consumer117cmay be required to acquire a lower quality version (e.g., more compact, smaller display size, etc.) of certain media in the mixed media package instance for playback on devices with limited resources.
FIG. 2 depicts an example system that propagates package modifications to media consumers. The media creator/owner101 delivers rules that govern themixed media package115 to themedia packager104. The media packager receives amedia submission201 from the media service provider, such as karaoke style lyrics for seed media that is audio. Themedia packager104 also receives consumer generated submissions from the media consumers117a-117c.Themedia consumer117asubmitsartwork207. Themedia consumer117bsubmits rating information. Themedia consumer117csubmits consumer generatedderivative media205. Theconsumer117cmay choose to only share a portion of her consumer generated derivative media. For example, theconsumer117cmay have re-mixed a song and taken digital photos with friends and an artist at a concert. Theconsumer117cmay choose to keep the photos private, and submit the re-mixed song as the consumer generatedmedia205. Themedia packager104 processes the submissions and approves or rejects the submissions in accordance with the rules from the media creator/owner101. Themedia packager104 updates apackage tracking structure203 accordingly to reflect the approval or rejection. Content can also be supplied from the author of the seed media, the owner/creator101 (assuming the author and owner/creator are not the same), and other service providers. Furthermore, the types of package modifications can include enhancements to the content of a mixed media package, modifications to structural information, modifications to presentation directives, etc.
A dynamic mixed media package allows product flexibility, new sources of revenue, the opportunity for product differentiation, and greater consumer involvement. A dynamic mixed media package can be dynamically modified throughout the life of the package, thus providing the capability to modify the package as well as expand the products/services offered with the package. In addition to adding content from an entity such as a record label, movie studio, or production company, consumers can create derivative media from one or more seed media. The derivative media can be incorporated into the dynamic mixed media package. Consumers can also contribute feedback (e.g., commentary, ratings, etc.) and supplemental media that is not derivative media (e.g., artwork for a seed video or seed audio).
The dynamic mixed media package also allows media owners/creators to collect consumer feedback and nimbly react to the consumer feedback to increase attractiveness of a product or service. A media owner/creator can adjust the contents of a package based on feedback, modify services, etc. A media owner/creator can also identify those consumers that generate the most popular media.
The flexibility and capability for expansion and/or change in the dynamic mixed media package also provides new business models and sources of revenue. Business models may spawn to offer various management services for the dynamic mixed media package, such as statistic collection, tracking and storing of package modifications, etc. The dynamic mixed media package will attract consumers and change consumer behavior with respect to purchasing of media online to create new sources of revenue or increase revenue. Consumers will be motivated to purchase this dynamic mixed media package for the enhanced experience it offers that cannot be achieved with the seed media alone. Consumers can also benefit, reputably or monetarily, when they contribute media that becomes popular. For instance, a consumer generated media associated with a seed media may be associated with an advertisement that generates advertisement revenues for the consumer and/or owner of the seed media. In fact, a consumer may create an advertisement that becomes associated with a seed media in a dynamic mixed media package.
FIG. 14 depicts an example revenue stream from derivative media. A media owner/creator1401 performs operations to generate amixed media package1409. The media owner/creator1401 generates fingerprint data for theseed media1405. The media owner/creator1401 may also generate one or more fingerprints forsupplemental media1407. The media owner/creator1401 stores the fingerprint(s) in a fingerprint andhash database1403, which may or may not be controlled by the media owner/creator1401. The media owner/creator1401 creates the dynamicmixed media package1409 with theseed media1405 and thesupplemental media1407. The media owner/creator1401 then marks instances of theseed media1405 with watermarks prior to delivery of the instances tomedia consumers1411 and1413. The instance delivered to themedia consumer1411 includes a watermark in the instance of theseed media1405 that brands it with an indication of the media consumer1411 (e.g., a customer account number, a username, etc.). Likewise, the instance delivered to themedia consumer1413 includes a watermark that brands the seed media instance with an indication of themedia consumer1413. The media owner/creator1401 may also apply a hash function to each of the instances prior to delivery. The media owner/creator1401 then stores the generated hash values into the fingerprint andhash database1403. Of course, these operations are not necessarily all performed by the media owner/creator1401. For instance, the media owner/creator1401 may only generate the fingerprint(s) and leave it to a media packager to embed watermarks and generate hash values.
Multiple watermarks may be applied to a seed media. A media owner/creator and a media packager (and any other entity in the distribution path of a seed media) may embed one or more different watermarks. There are watermarking techniques that allow tandem watermarking (e.g., embedding multiple watermarks on top of each other). Such tandem watermarking techniques allow for watermarking at multiple stages within the content distribution chain. For example, three different watermarks could be applied to a seed media. A first watermark that contains a generic content identifier (e.g., the ISRC code for a particular recording) can be embedded in a seed media. Then a second watermark that includes a distributor's ID is embedded into the seed media. Finally, a third watermark that includes a customer's ID is embedded into the seed media.
Deployment of a tandem watermarking technique may employ bit stream watermarking algorithms. With these bit stream watermarking algorithms, the watermark is inserted in the encoded/compressed audio or video signal stream, thus avoiding decoding and re-encoding. Although avoiding decoding and re-encoding may be less interesting when the signal is available uncompressed (e.g., at the production stage), it becomes more interesting when the signal is not readily available uncompressed (e.g., at the distribution stage).
Themedia consumers1411 and1413 generate a consumer generatedmedia1415 based on theseed media1405 in themixed media package1409. Themedia consumer1411 first creates a derivative media with the seed media of themixed media package1409. For example, themedia consumer1411 creates an audio re-mix with theseed media1405 and potentially, but not necessarily, with other audio (e.g., consumer created audio, audio from the same artist as the seed media, audio from another artist, etc.). The derivative media is provided to themedia consumer1413. Themedia consumer1413 creates a video to accompany the re-mix derivative media to generate themedia1415. For example, themedia consumer1413 creates a video from various animated videos. Themedia consumers1411 and1413 may operate entirely independently, as collaborative partners, as part of a creative community (e.g., an online video sharing community, an online social network community, an online digital image sharing community, etc.), etc. The consumer generatedmedia1415 is transmitted to acontent identifier system1417, which can entirely or in part reside locally on the consumer's computer or remotely on one or multiple servers.
Thecontent identifier system1417 processes themedia1415 to determine contribution percentage. Thecontent identifier system1417 accesses the fingerprint andhash database1403 to identify content of the consumer generatedmedia1415. Thecontent identifier system1417 then computes relative percent contribution from different authors or media owners/creators. The consumer generatedmedia1415 is then automatically categorized for destination selection and tagged based on the computed percent contribution.
In another embodiment, author attribution is determined based on identifiers, such as watermarks, previously embedded in the media. Using the example illustrated inFIG. 14, thecontent identifier system1417 examines themedia1415 to compute relative contribution by authors, whether consumer authors or seed media authors, using techniques such as watermarking. Although the author attribution computation is performed by thecontent identifier system1417, a separate system is not necessary for such functionality. Content identification functionality may be implemented with a program proximate to the consumer (e.g., a module or process that works in the background or foreground of the application used by the consumer to mix media, and perhaps generates a watermark to identify media originally created by the consumer), or another third party (e.g., in the media sharing server1419).
Themedia1415 is then provided to amedia sharing server1419, which results in a revenue stream. Amedia consumer1421 accesses the consumer generatedmedia1415. Access of themedia1415 by theconsumer1421 can be considered a revenue generating event. For example, advertisers pay advertising fees for advertising on the web page that presents themedia1415. As themedia1415 increases in popularity, greater advertising fees are generated, assuming the greater exposure leads to more clicks on the advertising links. In another example, consumers pay fees for accessing media hosted by a network including themedia sharing server1419. A portion of these fees are paid to owners/creators of media presented from the network as royalties. Advertising and/oruse fees1423 are paid to the media owner/creator1401. The media owner/creator1401 may then pay royalties to themedia consumers1411 and1413 based on percentage of contribution from themedia consumers1411 and1413. Such payments to consumers may spur creativity and increase consumer involvement. If themedia1415 includes seed media from another media owner/creator, then the owner of the media sharing server may apportion the payout of fees in accordance with the determined percent contribution. Furthermore, funds may be held in escrow for media contributed by unknown authors. These funds held in escrow may be held indefinitely until the authors are discovered, may be held for a limited period of time and then donated to an artist community, etc.
An entity may also assume accounting responsibilities and act as a clearinghouse for all fees received from media sharing sites and dispense royalties according to the percent contribution to the media owners/creators. For instance, themedia packager104 ofFIG. 1 may charge fees to multiple media owners/creators that send their seed media to themedia packager104 for assembly into a dynamic mixed media package. Themedia packager104 may charge fees to media consumers for membership in the dynamic mixed media package service. A service may maintain package modifications and propagate the modifications to members, maintain a community of independent artists that generate media (e.g., supplemental media, derivative media, etc.) and provide exposure to the media owner/creators (e.g., producers, publishing companies, other artists, etc.). Themedia packager104 may take a percentage of each package purchased by a media consumer. The revenue to the media packager may be flat fee based, variable based, or a hybrid of flat fee and variable fee. Variation in fees may be tied into the number of submissions from consumers, media consumer community activity related to a seed media, etc.
Whatever entity maintains the dynamic mixed media package, the package is created to be flexible to accommodate the management and modification operations discussed above.FIG. 3 depicts an example of a general structure of a dynamic mixed media package. A dynamicmixed media package301 includes several sections. A first section, which may be referred to as a package header, is apackage information section303. Thepackage information section303 includes package content and structure information and a reference (e.g., uniform resource locator, internet protocol address, etc.) to a package query checkpoint. The package query checkpoint is a checkpoint location to start querying for package modifications. Thepackage information section303 may also include one or more references to approved service providers that provide package modifications. Thepackage information section303 may also include access and authentication information and/or code, directives that govern presentation of content from different sections of themixed media package301, service level information, package level information, membership information, etc.
The mixed media package can be implemented as one or multiple instances (containing different media related to a particular seed media). A mixed media package may also be implemented as a virtual package. For instance, a link between various media might just consist in one identifier. This identifier can be absolute (e.g., a unique number or a set of numbers, a fingerprint, or a text string, or a combination thereof that is shared across multiple entities and acts as binding element). This identifier can also be recursive. For instance, one media package contains an index that points to a second package, which in turn contains a different index that references a third media package, etc.
Themixed media package301 also includes a clear mediasegment information section305 and aclear media segment307. A mixed media package does not necessarily include sections for clear media, but clear media sections can accommodate promotional content (e.g., samples, trailers, previews, reviews, etc.), revenue generating content (e.g., advertisements), etc. The clearmedia segment section305 includes information about content and structure of theclear media segment307, and, perhaps, presentation directives. For example, theclear media segment307 may include various type of content. Presentation directives in the clear mediasegment information section305 may restrict presentation of advertisements to every fifth access of the mixed media package, rotate promotional material, present content each time a new host device is encountered, etc. Theclear media segment307 includes unprotected and/or unrestricted media, such as promotional content as already mentioned. Although not protected and/or restricted, the content in theclear media segment307 may be marked (e.g., with a watermark) or fingerprinted for management purposes, such as collecting statistics.
The dynamicmixed media package301 includes a seed mediasegment information section309 and aseed media segment311. The content of theseed media segment311 is protected and/or restricted. The protection mechanism (e.g., digital rights management mechanism) may be implemented completely or partially in the seed mediasegment information section309. The seed mediasegment information section309 also includes content and structural information about theseed media segment311. Theseed media segment311 includes seed media, supplemental media, references related to the seed media (e.g., links to review of the seed media, links to an author website, code that loads a page from the author website, pointers to content at a remote or local location different than the dynamic mixed media package, etc.). For example, theseed media segment311 may include a reference to access streaming media in a different folder, at a remote server, on a network attached storage device, etc. The streaming media may be played immediately, played when accessed, cached for offline playing, etc. The content of theseed media segment311 may be videos, audio tracks, an audio collection, images, animations, text, games, podcasts, etc. The seed mediasegment information section309 may also include code for collecting statistics about the seed media and/or statistics collected about the supplemental media.
The third portion of the dynamicmixed media package301 includes a consumer generated mediasegment information section313 and a consumer generated media segment315. The consumer generated media segment315 may include derivative media created by consumers, independent media created by consumers that relate to the seed media, consumer comments about the seed media, references to consumer websites related to the seed media, code that accesses content from other consumer websites related to the seed media, pointers to content at a remote or local location different than the dynamic mixed media package, etc. The consumer generated mediasegment information section313 includes content and structural information about the content of the consumer generated media segment315. The consumer generated mediasegment information section313 may identify individual media in the segment315, indicate percent contribution for a particular media in the segment315, indicate popularity of media in the segment315, etc. The media that may be generated by consumers and added to a package covers a wide gamut of media, such as games, videos, audio, animation, lyrics, poems, commentary, re-mixes, alternative lyrics, photos, etc. A consumer will have the option to share their personal media with other media package owners (linked to the same seed media) or to keep their personal media for private consumption only.
Those of ordinary skill in the art should appreciate that the example dynamic mixed media package depicted inFIG. 3 is illustrative and not intended to be limiting upon embodiments. For instance, the package is described as including seed media and supplemental media. An implementation of a dynamic mixed media package does not necessarily literally “include” media. The package may include pointers to content, and the content may be in different locations. There are multiple types of file containers already defined that provide guidelines for implementing pointers to contents in different locations (e.g., MPEG-4 Systems (ISO/IEC 14496-1), MPEG-7 (ISO/IEC 15938), MPEG-21 (ISO/IEC 21000), mxf (Material eXchange Format), and aaf (Advanced Authoring Format)).
The dynamic mixed media package can be implemented in accordance with any of a number of techniques, both standard and proprietary. Although a standard implementation, such as in accordance with an MPEG standard, seems more desirable for wide-spread adoption in the market, a proprietary format may be optimal and/or preferable for other purposes. As a matter of fact, multiple physical formats can conceivably coexist, where conversion prescriptions will allow transitions from one format to another. For instance, a particular format might be suitable for the media exchange across PC platforms, where a significant amount of computational power is available for processing. This format might not be suitable in a more restrained platform environment where the necessity for compact and energy preserving devices might demand a more limited format.FIG. 15 depicts an example format of a dynamic mixed media package in accordance with the Multimedia Content Description Interface standard (“the MPEG-7 standard”). InFIG. 15, a dynamicmixed media package1500 includes a package header, clear content header, clear content section, seed media header, seed media section, and consumer generated content section. The header may indicate general information about the package, such as creation time, size, access, privileges, etc. The clear content header includes descriptive metadata for themixed media package1500. The clear content header also includes a package query checkpoint reference. The clear content section includes preview pages, preview photos, and preview audio. The preview photos includeimage media1503 with images compressed in accordance with JPEG and corresponding metadata and identifying data. The preview videos includevideo media1505 with flash video and corresponding metadata and identifying data. The seed media header includes an XML experience description. The seed media section includes menus, photos, video, audio, and lyrics that are watermarked and with a digital rights management (DRM) technology applied. The video in the seed media section includesvideo media1507 encoded according to FairPlay® DRM technology with corresponding metadata and identifying data. Similarly, the audio in the seed media section includes FairPlay encodedaudio media1509 with metadata and identifying data. The consumer generated section includes consumer generated content and syndicated content via an RSS feed. The consumer generated section includes consumer generatedmedia1511, examples of which include news, reviews, media from blogs, and feeds from external internet feeds.
Preview and seed media of a mixed media package may be implemented as a single media and not necessarily as separate media. The previews associated with a seed media can be implemented using scalable coding techniques, such as those defined in the MPEG-2 and MPEG-4 standards. The media content is coded in multiple layers, where each layer adds perceptual quality to the decoded/reconstructed signal. It is thus possible to decode only the basic layer of an audio signal and obtain AM quality monaural audio. Decoding the second layer will yield high quality audio with some (inaudible) artifacts, while decoding a third layer will allow the perfect (lossless) reconstruction of the original studio recording itself. This allows the encryption/protection of only a part of the content bit stream, while the first layer will be made available unencrypted as a pre-listening sample.
A similar approach can be taken with spatial information for audio. While the stereo signal might be made available unprotected for public consumption, multi-channel rendering information might be available in a protected format that can be unlocked upon acquisition of the necessary rights.
Though it is technically simpler to keep these different layers of content data in one bit stream format for synchronization upon reconstruction, for distribution purposes, it might be desirable to keep them in separate packages (i.e., the multi-channel information might only be available at a later point in time, after the release of the original media item).
The particular technique used to create a dynamic mixed media package will vary with the type of digital rights management utilized, the desired degree of flexibility for the package, etc. Regardless the specific details of encoding, protection, metadata, etc., the dynamic mixed media package begins with seed media.
FIG. 4 depicts a flowchart of example operations for creating a dynamic mixed media package. Atblock401, seed media is received, which may be one or more files. Of course, if the media owner/creator is creating the dynamic mixed media package, then block401 can be skipped. Atblock403, management data is generated for the seed media. Management of the seed media may utilize layering of multiple types of data. For example, fingerprinting data and a hash value are generated for the seed media. In addition, the seed media is marked with digital watermarking data. The hash value can be used to quickly identify the media. The fingerprinting data can be utilized to identify a portion or all of the media when combined with other media. The watermarking can be used to track the media and filter user generated media. Although the hash value provides expediency, this operation may be skipped. Furthermore, the fingerprinting may be done at a later time. Also, multiple fingerprinting algorithms can be deployed that each fulfill different robustness and fingerprint data size requirements. For example, one fingerprint format might be highly robust against even drastic changes in the signal (e.g., equalization, pitch shifting, time stretching, dynamic compression, perceptual coding), while another format will be significantly more compact (e.g., the amount of fingerprint data extracted for a certain duration of audio or video content is smaller). A more compact format may be more suitable for transmission through channels with bandwidth limitations. Atblock405, it is determined whether there is supplemental media. If there is no supplemental media, then control flows to block407. If there is supplemental media, then control flows to block409.
Atblock407, a dynamic mixed media package is generated with the seed media. Control flows to block411 fromblock407.
Atblock409, a dynamic mixed media package is generated with the seed media and the supplemental media. Atblock411, management data and/or management code (e.g., statistic collection code) is embedded into the generated package. Atblock413, it is determined whether data from a service provider is available. If a service provider has provided data (e.g., supplemental media, reviews, advertisements, etc.), then control flows to block415. If not, then control flows to block417.
Atblock415, the data from the service provider is written into the package. Atblock417, information about the package is written into the package. For example, structural and content information is written into the package header, clear content header, and/or seed media header. Atblock419, a reference to a package query checkpoint is written into the package, as well as any references to service providers that provide package modifications, if any.
After creation of a dynamic mixed media package and delivery of an instance of the dynamic mixed media package, the package can be modified. Modifications to the package can include various media generated by any one of owners, authors, controllers, consumers, and service providers. A modification to a package may be an upgrade, or even a downgrade, in quality of certain package content. For example, video may be enhanced (or higher quality video added to the package) for presentation over a home theatre system, or downgraded for presentation over a compact mobile device. This may not only affect the size of the rendered image or audio resolution, but also the compactness of the encoded media to a point where actual recoding into a different compression scheme might be necessary, because the original codec is not supported in the mobile device. A package modification may replace content, modify content, transcode content, or be added to the package. Since package modifications can be generated by any of a variety of sources at various times during the life span of a dynamic media package, a service may aggregate, review, and distribute the modifications for efficient maintenance of the package modifications.
FIG. 9 depicts an example system that maintains package modifications. Arules database911 hosts rules that govern packages that are associated with particular seed media. For example, a rule may require automatic acceptance of any submission from the corresponding media owner/creator. Another rule may reject any encoding submission that modifies the protective measures of a package unless created by a particular author. Therule database911 is accessed by asubmission handler module901 and a derivativemedia search module903. Themodules901 and903 are implemented in one or more machines, and may or may not be implemented at a same physical entity or location. When thesubmission handler module901 receives a submission, thesubmission handler module901 evaluates the submission against corresponding rules in therules database911. The corresponding rules may be determined by examining a submission for identifying data, such as a hash value(s), watermark, and/or fingerprint data. Thesubmission handler module901 indicates a result from evaluation of the submission against the appropriate rule(s) in anevaluation structure907. Maintaining indications of evaluation results allows for expedient dispensation of previously evaluated submissions. If thesubmission handler module901 approves a submission, then the submission is indicated in apackage tracking structure905 that tracks approved submissions. If approved, the submission and/or a reference to the submission are stored in an approvedmedia database909. It is not necessary to discard rejected submissions, however. A rejected submission may be stored in the same or a separate database for various reasons, such as archiving, comparison purposes, gathering of statistical data etc.
In addition to being submitted, package modifications may be discovered on the Internet. Thederivative search module903 searches a network (e.g., the Internet, a LAN, a particular online community, etc.) for consumer generated media derived from seed media. For example, thesearch module903 may search using fingerprint data, hash values, etc., of seed media. Thesearch module903 evaluates discovered derivative media against appropriate rules in therules database911. Similar to thesubmission handler module901, thesearch module903 updates thestructures907 and905 and thedatabase909 in accordance with evaluations. Whether rejected or approved, an indication of an evaluation result for a particular discovered derivative media is recorded in theevaluation structure907. If approved, the approval is indicated in the package tracking structure and the discovered derivative media and/or a reference thereto is stored in the approvedmedia database909. Indication of approval of a submission may also be accompanied by tracking information, such as a package version or date of approval, when distributing the approved submission.
FIG. 6 depicts a flowchart of example operations for handling submissions. Atblock601, a submission is received. At block603, it is determined whether the received submission has previously been evaluated. If so, then the submission is discarded atblock604. Otherwise, the corresponding seed media is determined at block603. Atblock605, the author of the submission is determined. If the author is a service provider, then control flows to block609. If the author is a consumer, then control flows to block607. If the author is an owner/creator of the seed media, then control flows to block615.
Atblock609, it is determined whether the service provider is pre-approved. For instance, the service provider has an agreement in place with the seed media creator/owner to provide submissions. If the service provider is pre-approved, then control flows to block615. Otherwise, control flows to block607.
Atblock607, the submission is evaluated against the rules for the corresponding seed media. Atblock608, the result of the evaluation is indicated. Atblock611, it is determined whether the submission is rejected or approved. If rejected, then the author is notified of the rejection atblock613. If approved, then information is recorded for the approved submission atblock615. For example, information about authorship, rights ownership, creation date, approval date, size, media type, attribution, etc., is recorded. Atblock617, the submission and/or a reference to the submission is stored. An author or representative of the author/rights owner may also be notified of approved submissions. A notification of an approved submission may also invite the author/rights owner to participate in a royalty scheme that compensates the author/rights owner based on popularity and percentage contribution.
FIG. 7 depicts a flowchart for example operations to search for derivative media. Atblock701, search for new consumer generated derivative media commences. Atblock703 it is determined if derivative media has been found. If not, then control returns to block701. If new derivative media has been found, then the corresponding seed media is determined atblock705. For example, the derivative media is examined for any watermarking, or a fingerprint of the derivative media is generated and compared against a fingerprint database. In another example, the derivative media indicates attribution information in a header segment. Atblock707, rules for the corresponding seed media are selected. Atblock709, the discovered media is evaluated against the selected rules. Atblock710, a result of the evaluation is indicated. Atblock711, it is determined whether the discovered derivative media is approved or rejected. If approved, control flows to block715. If rejected, control flows to block713.
At block715, an indication of the discovered media is recorded in a search structure with an approve flag set and indication of the corresponding seed media. For example, a structure is employed to track results of the search to avoid redundant evaluations. Additional information may also be recorded in the search structure to avoid certain network addresses, allow for evaluation of media against new or modified rules, etc. Atblock717, the discovered derivative media is indicated in a package tracking structure. Control flows fromblock717 to block719.
At block713, indication of the discovered derivative media is recorded in the search structure and a rejected flag is set along with indication of the corresponding seed media. The seed media is also indicated in case a submission is allowed for a first seed media, while rejected for a second seed media. Control flows to block719 from block713.
Atblock719, it is determined whether other seed media correspond to the discovered derivative media. If so, then control flows to block707. If there are no other corresponding seed media, then control returns to block701.
As stated above, multiple seed media may correspond to a submission. To conform to various rules for different seed media, a composite of different rules may be applied to media.FIG. 13 depicts a flowchart for example application of a composite set of rules to media that may be performed inblock607 ofFIG. 6 or block709 ofFIG. 7. Atblock1301, it is determined whether the submitted or discovered media (or accompanying information, such as in a header) includes data identifying different seed media. If the media (or accompanying information) does not include data identifying different seed media, then control flows to block1321. If the media includes data that identifies different seed media, then control flows to block1303.
Atblock1303, the media is examined to determine attribution to different seeds. Atblock1305, the rules for the individual seeds are looked up. Atblock1307, it is verified whether the rules are the same. If the rules are the same, then control flows to block1323. If the rules are not the same, then control flows to block1309.
Atblock1309, it is determined whether the individual rules allow for composite rules. If composite rules are not allowed, then control flows to block1311. If composite rules are allowed then control flows to block1313.
At block1311, the media is rejected. Control flows from block1311 to block1317.
At block1313, a composite of the different rules are generated based on seed attribution. Other factors may also be considered in the generation of composite rules, such as priority, pre-configured conflict resolution policy, etc. At block1315, the media is evaluated against the composite rules. At block1317, a result of the evaluation is generated. Control flows to either block608 or710 from block1317.
Atblock1321, rules for the seed media are looked up. Atblock1323, the media is evaluated against the rules. If the rules were determined to be the same atblock1307, then a rule or one of the sets of rules is selected. Control flows fromblock1323 to block1317.
Using various business models and delivery protocols, approved modifications are made available to media consumers with dynamic mixed media packages. Delivery of dynamic mixed media packages can be implemented in various manners. The entity that maintains package modifications may push all approved package modifications. The entity may prompt media consumers to accept or reject installation of approved package modifications. A query event may be detected at a consumer machine or device that triggers querying of an entity for any package modifications.
FIG. 5 depicts a flowchart of example operations for querying an entity for a package modification responsive to a query event. Atblock501, a dynamic mixed media package event is detected for a dynamic mixed media package. Atblock503, the dynamic mixed media package is accessed to determine a reference for a package query checkpoint, data that identifies the seed media of the package, and optionally a time of last query. For example, a hash value for the seed media, URL of a media packager, timestamp, and a unique index such as a customer or session identifier are written into a request message. Additional information may also be written into the query that affects the query result, such as consumer service level, community membership, etc. Atblock505, the package query checkpoint is queried with the information determined atblock503. A dashed line fromblock505 to block507 represents a lapse of time until a response is received to the query.
FIG. 8 depicts a flowchart for example operations for handling a package query for a dynamic mixed media package. Atblock801, a package query is received. At block803, data identifying a seed media and a previous query timestamp are determined from the query. At block805, a package tracking structure is accessed with the data determined at block803 to determine package modifications available since the previous query timestamp. The available package modifications may also be filtered based on consumer service level, privilege, geography, etc. Atblock807, it is determined whether any package modifications are available. If package modifications are available, then an indication of the available package modifications is returned to the media consumer with a new query timestamp or other identifier atblock809. For example, a message is returned that includes some new media to be added to the package, and references to other media. If there are no available package modifications, then a null value is returned with a new query timestamp.
Referring again toFIG. 5, an indication of package modifications is received atblock507. Atblock509, it is determined if additional media is to be added to the package. If so, control flows to block511. If there is no additional media, then control flows to block513.
Atblock511, the additional media is added to the package and the package information is updated accordingly. Atblock513, it is determined whether the indication of package modifications included a reference(s) and/or data. For example, it is determined whether the response to the query indicated network addresses, a new encoding scheme, ratings data, etc. If so, then control flows to block515. Otherwise, control flows to block517.
Atblock515, the package is modified in accordance with the reference(s) and/or data. For example, the ratings data is written into the clear content section of the package, a reference is written into a header for the consumer generated media section of the package, etc. Atblock517, the new query timestamp is written into the package.
A variety of implementations are possible for querying a checkpoint for modifications. For instance, functionality that detects an event and generates a query may be implemented as a component of a media player, a background process, daemon, plug-in, etc.FIG. 10 depicts an example implementation of a media query module. InFIG. 10, amedia player1003 is separate from apackage query module1005 in amedia consumer machine1007, which may be a mobile device, consumer electronic device, computer, one or more components in an automobile, etc. Themedia query module1005 may be a plug-in to themedia player1003, process that runs in the background, etc. Themedia consumer machine1007 hosts dynamic mixed media packages1001a-1001h.Themedia player1003 loads and activates the dynamicmixed media package1001bat a time a. At a time b, themedia player1003 invokes thepackage query module1005 or notifies thepackage query module1005 of the activation of thedynamic media package1001b.In various embodiments a query event is detected differently (e.g., thepackage query module1005 monitors the address space occupied by the dynamic mixed media packages1001a-1001h,an inter-process communication mechanism notifies thepackage query module1005 when a dynamic mixed media package is accessed, etc.). At a time c, thepackage query module1005 gathers query information from the dynamicmixed media package1001b.In another embodiment, a registration structure maintains query information for the packages hosted on themedia consumer machine1007, and the package query module accesses the structure based on a package identifier communicated by themedia player1003. At a time d, the package query module generates a query with the gathered information. At a time e, the package query module causes the generated query to be transmitted to the checkpoint indicated for seed media of the dynamicmixed media package1001b.At a time f, an indication of a package modification(s) is received and handled by thepackage query module1005. Thepackage query module1005 queues the modification(s) for application to thepackage1001bat time g. At a time h, themedia player1003 applies the queued modification(s) to thepackage1001b.It should be appreciated by those of ordinary skill in the art that the illustration ofFIG. 10 is intended to aid in understanding as one example implementation and not meant to be limiting. For instance, the modification to the package may be applied by thepackage query module1005 when themedia player1003 completes a current presentation. In another example, thepackage query module1005 modifies thepackage1001bwhile the media player presents media from a copy of thepackage1001bto be discarded after presentation.
A media player presents media of a dynamic mixed media package as directed by corresponding presentation directives. Directives may direct a player to overlap media, stream media concurrently, enforce a sequence upon media, etc. Some presentation directives may be pre-defined in the package, while others are commands from a user.FIG. 16 depicts an example presentation of media from an example dynamic mixed media package. A mixedmedia package player1613 loads and activates a dynamicmixed media package1621. Themixed media package1621 includes anadvertisement1603 for an upcoming musical album,synchronized lyrics1605,seed audio1607, a consumer generated video, and a consumer generated re-mix1611 that includes theseed audio1607. Assume a consumer commands theplayer1613 to play the consumer generatedvideo1609 withlyrics1605. A presentation directive(s) in the header for the consumer media section directs theplayer1613 to play theseed audio1607 with the consumer generatedvideo1609. The current play directive causes theplayer1613 to overlay thelyrics1605 onto thevideo1609 to create thevideo1605 with overlaid lyrics. A directive for thepackage1613 directs the player to sequence theadvertisement1603 for presentation after completion of thevideo1605. Theplayer1613 concurrently sends theseed audio1607 to speaker(s)1615 and thevideo1605 to thedisplay1613. The directive also directs the player to send theadvertisement1603 to thedisplay1613 after thevideo1605 has completed.
A player may utilize an interface that accommodates video play and a few controls, or a more complicated interface that divides a display area among various content of a dynamic mixed media package.FIG. 11 depicts an example interface for a dynamic mixed media package player. InFIG. 11, a display area of an interface has been divided into 10 regions. Avideo region1101 presents video from a mixed media package. Apackage directory1103 presents accessible content of a package to user (e.g., in tree hierarchy format, icon format, etc.). A directory of consumer generatedmedia region1105 presents a directory of consumer generated media that has been added to the package. A seed media owner/creator feed region1107 streams information from a recording company, publisher, and/or artist. A lyrics and coverart region1121 presents cover art and lyrics for the seed audio in a double truck layout that can be expanded to allow a consumer to navigate similar to flipping through pages of an album jacket. Aconcert schedule region1119 presents a concert schedule for the author of the seed audio of the package, perhaps, as filtered by current geographic information of the machine or device hosting the player. A region117 presents advertisements for concerts in the area by musicians of the same genre of music. Aregion1109 presents music reviews by consumers and critics, depending on the level of service purchased. Aregion1111 presents upgrades available for the seed audio, such as richer sound, improved player, etc. Aregion1115 presents live media feedback. For example, live comments from consumers in a music community are displayed in theregion1115.
FIG. 11 illustrates just one example of many possible examples. Numerous features and permutations of interfaces are possible with a dynamic mixed media package. For example, when audio from an album is played, a digital representation of an album booklet can be displayed in a double truck layout. Photos can be on one side, with lyrics and credits on the other side. The pages of the booklet can be flipped on a device with a large display area or scrolled through on a device with a constrained display area. Lyrics can be synchronized with audio on the level of individual words, playback of audio may be triggered by clicking on the lyrics, etc.
For image media, various functionality is also possible. A slideshow can be generated with all images of a particular artist or label. Consumer photos can be mixed in with musical artist photos and set to the audio of the artist.
In addition to the functionality allowed by dynamic mixed media packages, additional products and services can be spawned. A dynamic mixed media package can define themes for devices. For example, the sounds and display may be configured to comport with a theme as defined for a dynamic mixed media package for a particular album. For instance, the first few notes of the 4 most popular songs of the album may be utilized for 4 different ring/alarm sounds of a phone and the wallpaper for the phone set to cover art for the album. Design tools can be developed to mix media in a dynamic mixed media package. Moreover, new services can be offered that maintain package modifications, review submissions, track statistics, compensate consumers that generate popular media, RSS feeds, blogs, news services, user ratings, etc.
The described embodiments may be provided as a computing machine program product, or software, that may include a machine-readable medium having stored thereon instructions, which may be used to program a processing system (or other electronic devices) to perform a process according to embodiments of the invention, whether presently described or not, since every conceivable variation is not enumerated herein. A machine readable medium includes any mechanism for storing or transmitting information in a form (e.g., software, processing application) readable by a machine (e.g., a computer, a personal data assistant, a cellular phone, a media center, game console, etc.). The machine-readable medium may include, but is not limited to, magnetic storage medium (e.g., floppy diskette); optical storage medium (e.g., CD-ROM); magneto-optical storage medium; read only memory (ROM); random access memory (RAM); erasable programmable memory (e.g., EPROM and EEPROM); flash memory; or other types of medium suitable for storing electronic instructions. In addition, embodiments may be embodied in an electrical, optical, acoustical or other form of propagated signal (e.g., carrier waves, infrared signals, digital signals, etc.), or wireline, wireless, or other communications medium.
FIG. 12 depicts a diagrammatic representation of a machine in the example form of aprocessing system1200 within which a set of instructions, for causing the machine to perform any of the functionality discussed herein, may be executed. The machine may operate as a standalone device or may be connected (e.g., networked) to other machines. In a networked deployment, the machine may operate in the capacity of a server or a client machine in server-client network environment, or as a peer machine in a peer-to-peer (or distributed) network environment. The machine may be a server computer, a client computer, a personal computer (PC), a tablet PC, a set-top box (STB), a Personal Digital Assistant (PDA), a cellular telephone, a web appliance, a network router, switch or bridge, or any machine capable of executing a set of instructions (sequential or otherwise) that specify actions to be taken by that machine. Further, while only a single machine is illustrated, the term “machine” shall also be taken to include any collection of machines that individually or jointly execute a set (or multiple sets) of instructions to perform any one or more of the methodologies discussed herein.
Theexample processing system1200 includes a processor1202 (e.g., a central processing unit (CPU) a graphics processing unit (GPU) or both), amain memory1204 and astatic memory1206, which communicate with each other via abus1208. Theprocessing system1200 may further include a video display unit1210 (e.g., a liquid crystal display (LCD) or a cathode ray tube (CRT)). Theprocessing system1200 also includes an alphanumeric input device1212 (e.g., a keyboard), a cursor control device1214 (e.g., a mouse), adisk drive unit1216, a signal generation device1218 (e.g., a speaker) and anetwork interface device1220.
Thedisk drive unit1216 includes a machine-readable medium1222 on which is stored one or more sets of instructions (e.g., software1224) embodying any one or more of the methodologies or functions described herein. Thesoftware1224 may also reside, completely or at least partially, within themain memory1204 and/or within theprocessor1202 during execution thereof by theprocessing system1200, themain memory1204 and theprocessor1202 also constituting machine-readable media.
Thesoftware1224 may further be transmitted or received over a network1026 via thenetwork interface device1220.
While the invention(s) is (are) described with reference to various implementations and exploitations, it will be understood that these embodiments are illustrative and that the scope of the invention(s) is not limited to them. In general, techniques for access-based security evaluation of files introduced from a source external to a machine may be implemented with facilities consistent with any hardware system or hardware systems defined herein. Many variations, modifications, additions, and improvements are possible.
Plural instances may be provided for components, operations or structures described herein as a single instance. Finally, boundaries between various components, operations and data stores are somewhat arbitrary, and particular operations are illustrated in the context of specific illustrative configurations. Other allocations of functionality are envisioned and may fall within the scope of the invention(s). In general, structures and functionality presented as separate components in the exemplary configurations may be implemented as a combined structure or component. Similarly, structures and functionality presented as a single component may be implemented as separate components. These and other variations, modifications, additions, and improvements fall within the scope of the invention(s).