US20140068695A1 - Advanced Digital TV System - Google Patents

Abstract

A system for displaying electronic program guide information about a plurality of video items on a screen, the system including a display module to display a plurality of pages layering into the screen, each of the pages including a plurality of graphical panels associated with the video items, and a user input module to receive a user input to turn at least one of the pages such that the at least one page is peeled away from other ones of the pages in order to bring the graphical panels of the at least one page in or out of view, wherein the display module is operative to show the turning of the at least one page bringing the at least one page in or out of view. Related apparatus and methods are also described.

Description

RELATED APPLICATION INFORMATION
• The present application claims priority from U.S. Provisional Patent Application Ser. No. 60/705,430 of Shelton, et al., filed 4 Aug. 2005, the disclosure of which is hereby incorporated herein by reference.
FIELD OF THE INVENTION
• The present invention relates to an advanced digital television (TV) system.
BACKGROUND OF THE INVENTION
• By way of introduction, viewers are now expecting more advanced graphics, video and audio in relation to television as a result of their experience with computers, and in particular the Internet. However, capabilities of set top boxes are generally limited due to inherent limitations of the set top boxes, mainly in relation to the hardware thereof.
• One solution is to use a similar hardware set up used in computers and other advanced graphic, video and/or audio systems. However, the computer and similar systems are generally too expensive to be implemented in a set top box.
• The following references are also believed to represent the state of the art:
• US PublishedPatent Application 2005/0155061 of Kim;
• US Published Patent Application 2004/004717 of Gordon, et al.;
• US Published Patent Application 2003/0156220 of Narita;
• US Published Patent Application 2004/0017378 of Lin, et al.;
• US Published Patent Application 2004/0190614 of Schlockermann, et al.;
• PCT Published Patent Application WO 2004/072935 of NDS Limited;
• US Published Patent Application 2006/0050790 of Eerenberg, et al.;
• European Published Patent Application EP1365385 of Broadcom Corporation;
• Japan Published Patent Application 11-168729 of Toshiba Corp.;
• Japan Published Patent Application 2002-016876 of Toshiba Corp.;
• U.S. Pat. No. 6,466,200 to Cesana, et al.; and
• A publication entitled “Rapid Serial Visual Presentation Techniques for Consumer Digital Video Devices” by the Mitsubishi Electric Research Laboratory, August 2003.
• The disclosures of all references mentioned above and throughout the present specification, as well as the disclosures of all references mentioned in those references, are hereby incorporated herein by reference.
SUMMARY OF THE INVENTION
• The present invention, in preferred embodiments thereof, includes an advanced set top box system including hardware and software for providing enhanced graphics (including overlay graphics and animations), video, audio, user input and timeline and synchronization.
• In preferred embodiments thereof, the advanced set top box system includes some or all of the following functionality: display a mixture of synchronized graphics and video in such a way that the graphics and video merge and it is generally difficult to see which one is which; present video transitions and effects which are preferably at least of the quality seen in studio presentation equipment; jump around video clips from any suitable frame on one stream to any suitable frame on another stream at any suitable speed and with the stream playing out in any direction; change the speed and direction of playout smoothly and without any visible defects; present audio and audio effects which are at least of the quality available on modern consumer entertainment equipment; separately control the playout of audio regardless of what is happening to the video or graphics; perform audio mixing from multiple sources; improve the synchronization and timeline control; control playout and transitions of graphics, video and audio with field-accuracy; control playout to achieve the presentation quality to at least that seen in studio presentation equipment; support multiple controllers of different types simultaneously; and allow for advanced input types, for example, but not limited to, iMotion and voice recognition.
• The present invention, in preferred embodiments thereof also includes a system and method for compressing electronic program guide (EPG) screenshots and video clips, for example, but not limited to, as a sequence of P or B coded MPEG video frames.
• The present invention, in preferred embodiments thereof also includes a high definition EPG having a series of tabbed graphical paneled pages which layer into the screen in much the same way as a series of files in a filing cabinet or a series of Music compact disks (CDs) in a rack of CDs. A remote control, for example, is then be used to peel away or bring back layers, effectively giving the illusion of moving through a stack of pages of information.
• There is thus provided in accordance with a preferred embodiment of the present invention, a system for displaying electronic program guide information about a plurality of video items on a screen, the system including a display module to display a plurality of pages layering into the screen, each of the pages including a plurality of graphical panels associated with the video items, and a user input module to receive a user input to turn at least one of the pages such that the at least one page is peeled away from other ones of the pages in order to bring the graphical panels of the at least one page in or out of view, wherein the display module is operative to show the turning of the at least one page bringing the at least one page in or out of view.
• Further in accordance with a preferred embodiment of the present invention each of the pages includes a tab to facilitate selection of any one of the pages.
• Still further in accordance with a preferred embodiment of the present invention each of the graphical panels has a shape, the display module being operative to display on the screen a non-linear changing of the shape of the graphical panels of the at least one page as the at least one page is being turned.
• Additionally in accordance with a preferred embodiment of the present invention the display module is operative to display the video items in the graphical panels.
• Moreover in accordance with a preferred embodiment of the present invention each of the video items displayed in the graphical panels has a shape, the display module being operative to display on the screen a non-linear changing of the shape of the videos of the at least one page as the at least one page is being turned.
• Further in accordance with a preferred embodiment of the present invention the video items include a plurality of live video items.
• Still further in accordance with a preferred embodiment of the present invention the video items include a plurality of non-live sourced video items.
• Additionally in accordance with a preferred embodiment of the present invention the display module is operative to display the at least one page turning about an axis.
• Moreover in accordance with a preferred embodiment of the present invention the display module is operative to display the at least one page turning vertically about a horizontal axis.
• Further in accordance with a preferred embodiment of the present invention the user input module is operative to receive a user input to extend one of the pages to an extended page, the display module being operative to show the extended page on the screen, the extended page including additional information relating to one or more of the graphical panels of the extended page.
• Still further in accordance with a preferred embodiment of the present invention each of the pages is uniquely associated with one of a plurality of television channels.
• Additionally in accordance with a preferred embodiment of the present invention the pages are arranged alphabetically.
• Moreover in accordance with a preferred embodiment of the present invention the graphical panels of each of the pages are arranged alphabetically.
• Further in accordance with a preferred embodiment of the present invention the pages are arranged chronologically.
• Still further in accordance with a preferred embodiment of the present invention the graphical panels of each of the pages are arranged chronologically.
• Additionally in accordance with a preferred embodiment of the present invention the user input module is operative to receive a user input to move a focus between graphical panels in one of the pages, the display module being operative to display the focus on one of the graphical panels.
• Moreover in accordance with a preferred embodiment of the present invention the user input module is operative to receive a user input to turn at least one of the pages and move a position of a focus of the graphical panels in the pages at the same time.
• There is also provided in accordance with still another preferred embodiment of the present invention, a set-top box system including a decoder to decode a compressed video stream into a decoded video stream, and an alpha generator operationally connected to the decoder, the alpha generator being operative to generate an alpha channel for the decoded video stream.
• Further in accordance with a preferred embodiment of the present invention, the system includes a compositor operationally connected to the alpha generator, the compositor being operative to alpha blend the decoded video stream with another decoded video stream based on the alpha channel.
• Still further in accordance with a preferred embodiment of the present invention the decoded video stream includes a plurality of video frames, the alpha generator being operative to generate the alpha channel based on at least one of the video frames.
• Additionally in accordance with a preferred embodiment of the present invention the one video frame includes a set of RGB values, the alpha generator being operative to generate the alpha channel based on the set of RGB values.
• Moreover in accordance with a preferred embodiment of the present invention the decoded video stream includes a plurality of video frames, the alpha channel includes a value applied to at least one of the video frames.
• Further in accordance with a preferred embodiment of the present invention the decoded video stream includes a plurality of video frames, each of the video frames including a plurality of pixels, the alpha channel includes a value applied to each of the pixels of at least one of the video frames.
• Still further in accordance with a preferred embodiment of the present invention the alpha generator is operative to apply the value to each of the pixels of at least one of the video frames based on a vector based graphics system.
• Additionally in accordance with a preferred embodiment of the present invention the alpha generator is operative to apply the value to each of the pixels of at least one of the video frames based on at least one bitmap sequence.
• Moreover in accordance with a preferred embodiment of the present invention, the system includes a memory arrangement having a plurality of areas, the memory arrangement being operationally connected to the alpha generator, the decoded video stream including a plurality of video frames, each of the video frames including a plurality of blocks of pixels, and an image transform engine operationally connected to the memory arrangement, the image transform engine being operative to move the blocks between the areas of the memory arrangement.
• Further in accordance with a preferred embodiment of the present invention the image transform engine is a blitter.
• Still further in accordance with a preferred embodiment of the present invention the image transform engine is operative to perform image transform operations on the pixels.
• Additionally in accordance with a preferred embodiment of the present invention the image transform engine is operative to perform at least one of scaling and shaping of at least part of least one of the video frames.
• Moreover in accordance with a preferred embodiment of the present invention the image transform engine and the alpha generator are operative to perform a video transition effect between the decoded video stream and another decoded video stream.
• There is also provided in accordance with still another preferred embodiment of the present invention, a set-top box system for cueing up of material, the system including a storage arrangement to store a plurality of videos, and a first decoder operationally connected to the storage arrangement, the first decoder being operative to decode one of the videos off of the storage arrangement, stop decoding the one video at a first specific frame, save at least one reference frame and a plurality of decoding state information, the at least one reference frame and the decoding state information being necessary to allow restarting the decoding from the first specific frame, loading the at least one reference frame and the decoding state information, and then restarting the decoding from the first specific frame at a future point in time.
• Further in accordance with a preferred embodiment of the present invention, the system includes a second decoder to decode a main streamed video, and a compositor operationally connected to the first decoder and the second decoder, the compositor being operative to composite the outputs of the first decoder and the second decoder such that the one video from the first specific frame is inserted into the main streamed video.
• Still further in accordance with a preferred embodiment of the present invention the first decoder is operative to decode another one of the videos off of the storage arrangement to a second specific frame, while the one video is still cued up in the first decoder, such that the other video is also cued up ready to be decoded from the second specific frame at another future point in time.
• Additionally in accordance with a preferred embodiment of the present invention the first decoder is operative to decode the other video from the second specific frame at the other future point in time.
• Moreover in accordance with a preferred embodiment of the present invention the first decoder is operative to de-cue the previously cued up other video.
• Further in accordance with a preferred embodiment of the present invention the first decoder is operative to select the one video for decoding instead of the other video.
• Still further in accordance with a preferred embodiment of the present invention the first decoder is operative to decode the one video off of the storage arrangement to the first specific frame faster than real time.
• There is also provided in accordance with still another preferred embodiment of the present invention, a screenshot compression system for compressing uncompressed screenshots, the system including an encoder including an analyzing module to analyze the screenshots to identify commonality among the screenshots, and create a reference frame based on the commonality of the screenshots, and a compression module to compress the screenshots based on the reference frame.
• Additionally in accordance with a preferred embodiment of the present invention the screenshots include a plurality of macroblock boundaries, the analyzing module being operative to analyze the screenshots based on the macroblock boundaries.
• Moreover in accordance with a preferred embodiment of the present invention the compression module is operative to compress the screenshots into a stream including a plurality of compressed frames, the compressed frames including at least one of an MPEG B-frame and an MPEG P-frame.
• Further in accordance with a preferred embodiment of the present invention the compression module is operative to compress the reference frame as an MPEG I-frame in the stream of the compressed frames.
• Still further in accordance with a preferred embodiment of the present invention the reference frame is compressed and sent in a different stream to the stream of compressed frames.
• Additionally in accordance with a preferred embodiment of the present invention the analyzing module is operative to reorder a sequence of the screenshots so as to improve efficiency of the compression of the screenshots.
• There is also provided in accordance with still another preferred embodiment of the present invention, a screenshot compression system for compressing uncompressed screenshots, the system including an encoder including an analyzing module to analyze the screenshots to identify commonality among the screenshots, and determine a suitable reference frame based on the commonality of the screenshots, the suitable reference frame being selected from a pre-existing frame, and a compression module to compress the screenshots based on the reference frame.
• Moreover in accordance with a preferred embodiment of the present invention the compression module is operative to compress the screenshots into a stream including a plurality of compressed frames, the compressed frames including at least one of an MPEG B-frame and an MPEG P-frame.
• Further in accordance with a preferred embodiment of the present invention the reference frame is part of a dictionary of images that resides in a set-top box.
• Still further in accordance with a preferred embodiment of the present invention the compression module is operative to compress the screenshots into a stream including a plurality of compressed frames, the compressed frames including an MPEG I-frame and at least one of an MPEG B-frame and an MPEG P-frame, the MPEG I-frame corresponding to the reference frame.
• Additionally in accordance with a preferred embodiment of the present invention the screenshots include a plurality of macroblock boundaries, the analyzing module being operative to analyze the screenshots based on the macroblock boundaries.
• Moreover in accordance with a preferred embodiment of the present invention the analyzing module is operative to reorder a sequence of the screenshots so as to improve efficiency of the compression of the screenshots.
• There is also provided in accordance with still another preferred embodiment of the present invention, a set-top box system to display a video of a first video stream including a window displaying a video of a second video stream using a long-term reference frame, the system including a first decoder to decode the first video stream, the first video stream including a reference to the long-term reference frame, and a second decoder to decode the second video stream, the second decoder having an output frame, the long-term reference frame pointing to the output frame, the second decoder being operative to be start decoding the second video stream without driving an output.
• Further in accordance with a preferred embodiment of the present invention the at least one long-term reference frame is not received by the set-top box system in a main video stream.
• There is also provided in accordance with still another preferred embodiment of the present invention, a set-top box system for managing reference frames, the system including a decoder independent buffer to store at least one reference frame, and a video decoder to decode a compressed video based on the at least one reference frame, the decoder having a dedicated buffer to store the at least one reference frame, the decoder being operative to transfer the at least one reference frame between the dedicated buffer and the decoder independent buffer.
• There is also provided in accordance with still another preferred embodiment of the present invention, a set-top box system, including a storage arrangement to store a compressed video clip formed from a plurality of sub-regions of at least one video frame, the video clip having a start and an end, and a decoder arrangement to loop-decode the sub-regions of the at least one video frame with a seamless cut between the start and the end of the video clip.
• There is also provided in accordance with still another preferred embodiment of the present invention, a set-top box system, including a storage arrangement to store a plurality of compressed video images, each of the video images having a size, and a flexible video decoder, the decoder having a maximum processing speed, the decoder being operative to decode a variable number of the compressed video images at the same time, the variable number being based on the maximum processing speed of the decoder and the size of the video images for decoding.
• There is also provided in accordance with still another preferred embodiment of the present invention, a set-top box system, including a decoder operative to receive a compressed frame which is too large for decoding by the decoder, and decode only a sub-region of the compressed frame.
• There is also provided in accordance with still another preferred embodiment of the present invention, a method for displaying electronic program guide information about a plurality of video items on a screen, the method including displaying a plurality of pages layering into the screen, each of the pages including a plurality of graphical panels associated with the video items, receiving a user input to turn at least one of the pages such that the at least one page is peeled away from other ones of the pages in order to bring the graphical panels of the at least one page in or out of view, and showing the turning of the at least one page bringing the at least one page in or out of view.
• There is also provided in accordance with still another preferred embodiment of the present invention, a method including decoding a compressed video stream into a decoded video stream, and generating an alpha channel for the decoded video stream.
• There is also provided in accordance with still another preferred embodiment of the present invention, a method for cueing up of material, the method including storing a plurality of videos, decoding one of the videos, stopping the decoding of the one video at a first specific frame, saving at least one reference frame and a plurality of decoding state information, the at least one reference frame and the decoding state information being necessary to allow restarting the decoding from the first specific frame, loading the at least one reference frame and the decoding state information, and then restarting the decoding from the first specific frame at a future point in time.
• There is also provided in accordance with still another preferred embodiment of the present invention, a screenshot compression method for compressing uncompressed screenshots, the method including analyzing the screenshots to identify commonality among the screenshots, creating a reference frame based on the commonality of the screenshots, and compressing the screenshots based on the reference frame.
• There is also provided in accordance with still another preferred embodiment of the present invention, a screenshot compression method for compressing uncompressed screenshots, the method including analyzing the screenshots to identify commonality among the screenshots, determining a suitable reference frame based on the commonality of the screenshots, the suitable reference frame being selected from a pre-existing frame, and compressing the screenshots based on the reference frame.
• There is also provided in accordance with still another preferred embodiment of the present invention, a method for playing a video clip, including storing a compressed video clip formed from a plurality of sub-regions of at least one video frame, the video clip having a start and an end, and loop-decoding the sub-regions of the at least one video frame with a seamless cut between the start and the end of the video clip.
BRIEF DESCRIPTION OF THE DRAWINGS
• The present invention will be understood and appreciated more fully from the following detailed description, taken in conjunction with the drawings in which:
• FIG. 1 is a sample screen of an interactive TV application running on a set-top box system constructed and operative in accordance with a preferred embodiment of the present invention;
• FIGS. 2 and 3 are sample screens of a game application running on the set-top box system ofFIG. 1;
• FIG. 4 is a view of a high definition EPG constructed and operative in accordance with a preferred embodiment of the present invention;
• FIG. 5 is a view of a plurality of tabbed graphical paneled pages of the EPG ofFIG. 4;
• FIG. 6 is a view of the tabbed graphical paneled pages ofFIG. 5 showing one of the paneled pages being peeled away;
• FIG. 7 is a view of an extended paneled page of the tabbed graphical paneled pages ofFIG. 5;
• FIG. 8 is a view of a plurality of tabbed graphical paneled pages for a channel viewing mode of the EPG ofFIG. 5;
• FIG. 9 is a view of the tabbed graphical paneled pages ofFIG. 8 showing one of the paneled pages being peeled away;
• FIG. 10 is a view illustrating right navigation in a paneled page of the tabbed graphical paneled pages ofFIG. 8;
• FIG. 11 is a view illustrating tabbed graphical paneled pages labeled with channel numbers of the EPG ofFIG. 4;
• FIG. 12 is a view illustrating tabbed graphical paneled pages labeled alphabetically of the EPG ofFIG. 4;
• FIG. 13 is a block diagram view of the set-top box system ofFIG. 1;
• FIG. 14 is a view of an image transform engine for use with the set-top box system ofFIG. 13;
• FIG. 15 is a view of a symbol indicating the addition of an alpha channel in the set-top box system ofFIG. 13;
• FIG. 16 is a detailed block diagram view of an alpha generator of the set-top box system ofFIG. 13;
• FIG. 17 is a block diagram view of a compositor for use with the set-top box system ofFIG. 13;
• FIG. 18 is a block diagram view of an example process performed by the compositor ofFIG. 17;
• FIG. 19 is a block diagram showing combined use of compositors, image transform engines and memory buffers for use with the set-top box system ofFIG. 13;
• FIG. 20 is a block diagram view of a broadcaster decoder subsystem for use with the set-top box system ofFIG. 13;
• FIG. 21 is a block diagram view of a preferred method of converting all video feeds to the same format for use with the set-top box system ofFIG. 13;
• FIG. 22 is a block diagram view illustrating a preferred method of insertion of alternative material for use with the set-top box system ofFIG. 13;
• FIG. 23 is a detailed block diagram view of the broadcast decoder subsystem of the set-top box system ofFIG. 13;
• FIG. 24 is a block diagram view of a basic level video effects processor of the set-top box system ofFIG. 13;
• FIG. 25 is a block diagram view illustrating a preferred method of creating a video overlay using the video effects processor ofFIG. 24;
• FIG. 26 is a block diagram view illustrating a preferred method of creating a background or levels of a game using the video effects processor ofFIG. 24;
• FIG. 27 is a block diagram view of the basic level video effects processor ofFIG. 24 with a PNG decoder arrangement;
• FIG. 28 is a block diagram view of an advanced level video effects processor for use with the set-top box system ofFIG. 13;
• FIG. 29 is a block diagram view of a managed decoder for use with the set-top box system ofFIG. 13;
• FIG. 30 is a block diagram view of a multiple input managed decoder for use with the set-top box system ofFIG. 13;
• FIG. 31 is a block diagram view of another advanced level video effects processor for use with the set-top box system ofFIG. 13;
• FIG. 32 is a block diagram view of an illustration of a preferred method of playing out multiple clips simultaneously using the set-top box system ofFIG. 13;
• FIG. 33 is a block diagram view illustrating a preferred compression of screenshots for use with the set-top box system ofFIG. 13;
• FIG. 34 is a view of image screenshots including a reference screenshot for use in a compression system constructed and operative in accordance with a preferred embodiment of the present invention;
• FIG. 35 is a view a plurality of image screenshots being ordered for use in the compression system ofFIG. 34;
• FIG. 36 is a view a common image of a plurality of image screenshots for use in the compression system ofFIG. 34;
• FIG. 37 is a view a plurality of smaller than full-screen size screenshots being combined into a plurality of full-screen size screenshots for use in the compression system ofFIG. 34;
• FIG. 38 is a view of a plurality of images including equal sized spare area for use in the compression system ofFIG. 34;
• FIG. 39 is a view of a plurality of images including unequal sized spare area for use in the compression system ofFIG. 34;
• FIG. 40 is a block diagram view of the compression system ofFIG. 34 constructed and operative in accordance with a preferred embodiment of the present invention;
• FIG. 41 is a block diagram view of a picture-in-picture system constructed and operative in accordance with a preferred embodiment of the present invention;
• FIG. 42 is a block diagram view of a most preferred basic level video effects processor for use with the set-top box system ofFIG. 13;
• FIG. 43 is a block diagram view of a most preferred advanced level video effects processor for use with the set-top box system ofFIG. 13;
• FIG. 44 is a block diagram view of a graphical effects processor for use with the set-top box system ofFIG. 13;
• FIG. 45 is a block diagram view of a 2D accelerator of the graphical effects processor ofFIG. 44;
• FIG. 46 is a block diagram view of an audio transfer engine for use in the set-top box system ofFIG. 13; and
• FIG. 47 is a block diagram view of an audio effects processor of the set-top box system ofFIG. 13.
DETAILED DESCRIPTION OF A PREFERRED EMBODIMENT
• By way of introduction, the preferred embodiments provided herein are preferably implemented in hardware and/or software. It will be appreciated that the exact specification for the silicon chip(s) used in implementation is very much driven by the platform operator and the chipset vendor. Therefore, the description below specifies the general requirements necessary in order to enable the preferred embodiments to be implemented. For example, a particular set-top box (STB) platform may require more TV outputs, or more input tuners. It is appreciated that each STB platform needs to be treated on a case by case basis and may require multiple copies of the system building blocks described in the preferred embodiments, below, in order to achieve the desired functionality. Similarly, different chipset vendors implement hardware in different ways and therefore the vendors derive acceleration in different ways.
• FIGS. 1-3 illustrate the kind of graphical, video and audio effects that are preferably produced in preferred embodiments of the present invention.
• Reference is now made toFIG. 1, which is asample screen10 of an interactive TV application running on a set-top box system12 constructed and operative in accordance with a preferred embodiment of the present invention. The set-top box system12 is preferably operative to support one or more of the following (typically derived from TV studio presentation mixers): scaleable text with different weightings in the same line; wide color range, anti-aliased, drop shadowed text; animated and rotating/revealing text into place (arrow14); high quality scaleable bitmap image processing that preferably does not introduce visual artifacts into the bitmap graphics for example using bilinear and bicubic algorithms; pseudo three-dimensional (3D) rotations or text or graphical constructions (arrow16); multiple layers of arbitrarily size video animated into view with alpha reveal (arrow18); bitmaps scale up to appear in the screen (arrow20); all operations synchronized to a timeline; multiple areas of constantly moving overlays; overlays revealed through changes in transparency; pseudo 3D rotating logos; animated overlays exploding from the screen to form the backdrop to areas of text or other graphics; multiple moving video overlays of any suitable size or shape and in any suitable position on the screen; large moving backgrounds; bitmap images animated across the screen at any suitable speed; any suitable number of bitmap images effortlessly moved at different places and with different speeds; flares or flashes tracked across the screen to highlight areas or draw the eye to a particular area of the screen; text revealed through changes in alpha; text revealed in synchronization with a background area reveal; text animated in from somewhere and revealed through changes in transparency; text textured with bitmap or gradient fills; text scaled up/down with precision and substantially no loss of quality; TV Channel identification text animated across the screen and positioned against the edge of a moving video transition; graphical elements drawn or filled at a controlled rate; text rotated into view; flashes or flare effects flash through the text as the text rotates; multiple layers of graphical overlays and high quality text (anti-aliased, textured, filtered, blended and scalable text) flies into position from different areas of the screen in a pre-determined sequence with accurate animation and changing transparency; tables of text lines scrolled or rotated into view according to a predetermined order; timing of scrolling or rotations is controlled; transparency of table elements is also controlled; lines of tables rotate at different rates to give a cascading rotation effect; flashes or flares are added to a table to highlight key parts of the table; and complex alpha based reveals or transitions performed between different layers of graphics.
• Other general functionality of the set-top box system12 is now described below.
• The set-top box system12 is also operative to preferably provide functionality for 3D games, for example, but not limited to: enabling multiple tracks of video (typically four or more tracks) to be seamlessly switched from one part of one track to another part of another track; high quality text rendering (anti-aliased, textured, filtered, blended and scalable text); 3D-like Play objects and audio mixing.
• Additionally or alternatively, the set-top box system12 is also preferably operative to provide functionality such that if an advertisement is stored with content to disk, the advertisement can be frame accurately replaced with an alternative suitable piece of content and any suitable transitions are generally possible.
• Reference is now made toFIGS. 2 and 3, which show asample screen22 and asample screen24 of a game application running on the set-top box system12 ofFIG. 1, respectively. The set-top box system12 is preferably operative to provide functionality for the enhanced presentation of 2D games as shown in thesample screen22 ofFIG. 2 and thesample screen24 ofFIG. 3. The following features are typically provided: full screen moving backgrounds (arrow26); text effects including shimmering and drop shadows (arrow28); precision animation of large bitmaps (arrow30); gradient light simulating fills (arrow32); multiple layer of graphics moving independently (arrow34); large number of (for example, but not limited to 30 or more) animated play objects including lighting effects, highlights and rotations (arrow36).
• Reference is now made toFIG. 4, which is a view of ahigh definition EPG54 having asample screen38 constructed and operative in accordance with a preferred embodiment of the present invention. By way of introduction, high definition (HD) TV is starting to grow in popularity and the screens used tend to be larger in size than screens traditionally used for standard definition (SD) TV. On average people sit relatively closer to high definition TVs, giving the viewer a greater field of view. However, electronic program guides tend to just be up-scaled versions of the SD counterparts and are not taking full advantage of the new space available. In addition, set top boxes have to increasingly handle more connectivity and user interactivity options.
• A visually simple way to make more and more user interface options available to the viewer is required. The visual features available to the viewer must also dynamically scale as one set top box in a system may have more features available than another or new features may be added during the life time of the TV system. Also, TV systems increasingly have to support more channels and features. Simple ways to sort and navigate around lots of channels is therefore required.
• Thehigh definition EPG54 is preferably implemented on the set-top box system12 (FIG. 13). The high definition EPG54 typically includes one or more of the following features: animated text which is scaleable, animated, drop shadowed and/or textured (arrow40); full sized moving background at the same time as any other suitable effect (arrow42); scaleable textured graphics including lighting, shading and/or reflection effects (arrow44); pseudo 3D effects including drop down tabs, rotations of video or graphics and/or page turns bulges (arrow46); down-scaled, non linear shaping of live video where the shape is changed smoothly on a frame by frame basis and movement is less than a pixel (arrow48); multiple arbitrarily shaped and scaled video clips from disk are placed on screen (arrow50); large areas of graphics are smoothly scaled up/down to or from full screen; mosaics of video clips are formed from video which originates from any suitable source; full moving backgrounds at the same time as fill mosaics; reflections of the mosaic and or the graphical constructions; multiple movies are placed on screen from any suitable source; movies are smoothly scaled up/down on screen with graphical constructions around the video that scale up/down at the same time/rate such that the video appears to fit in the graphics; transitions between video sequences from any suitable source; the transition effects are performed with a combination of transparency changes and deformations of video sequences; transitions are performed where an intermediate video or still image are placed mid transition; and complex overlaid graphics are added and sequenced to the video transition.
• Reference is now made toFIG. 5, which is a view of a plurality of tabbed graphical paneledpages52 of thehigh definition EPG54 ofFIG. 4. Reference is also made toFIG. 6, which is a view of the tabbed graphical paneledpages52 ofFIG. 5 showing a paneledpage56 being peeled away from the other tabbed graphical paneled pages52. Reference is also made toFIG. 4. Instead of designing a flat layout EPG, thehigh definition EPG54 is designed as a series of tabbed graphical paneled areas orpages52 that layer into the screen in much the same way as a series of files in a filing cabinet or a series of Music CDs in a rack of CDs. A remote control (not shown) is then typically used to peel away or bring back the layers of thepages52, effectively giving the illusion of moving through a stack of pages of information.
• Thehigh definition EPG54 preferably includes adisplay module55 and auser input module57. Thedisplay module55 is typically operative to display thepages52 layering into thescreen38. Eachpage52 typically includes a plurality ofgraphical panels59 associated with video items. Thegraphical panels59 are typically live or non-live (for example, but not limited to, disk sourced) video items or graphical representations of the associated videos. Therefore, thedisplay module55 is preferably operative to display the video items in thegraphical panels59. Each of thepages52 preferably includes atab61 to facilitate selection of thepages52.
• Theuser input module57 is generally operative to receive a user input, typically from a remote control, to turn the page56 (or more of the pages52), such that thepage56 is peeled away from theother pages52 in order to bring thegraphical panels59 of thepage56 in or out of view (out of view in the example ofFIG. 6).
• Thedisplay module55 is preferably operative to show the turning of thepage56 bringing thepage56 in or out of view, the turning typically being vertically about ahorizontal axis63. Additionally, thedisplay module55 is preferably operative to display on the screen38 a non-linear changing of the shape of the video items, or the graphical representations, in thegraphical panels59 of thepage56 as thepage56 is being turned.
• The design of thehigh definition EPG54 has the advantage that the number of paneled pages stretching back into the distance may be different for each set top box (STB) and the number of paneled pages may change dynamically or over time. For example, if the STB needs to support locally connected devices because the hardware configuration of the STB supports the feature, an extra paneled page(s) is preferably added to the paneled pages allowing the viewer to see and interact with locally connected devices. Similarly, if an operator wants to add a new feature to all STBs on a network, an extra paneled page(s) is preferably added to thehigh definition EPG54.
• Reference is now made toFIG. 7, which is a view of an extendedpaneled page58 of the tabbed graphical paneledpages52 ofFIG. 5. The viewer typically navigates to a paneled page and then the viewer extends the paneled page to obtain more information on a subject area yielding the extendedpaneled page58. The extended paneledpage58 typically includes additional information relating to one or more of thegraphical panels59. Additionally or alternatively, the extendedpaneled page58 includes one or more of the following: text; graphical arrangements; images; or videos, depending on the purpose and the features available on the network. InFIG. 7, a number ofsmall videos60 show the programs available for viewing. Theuser input module57 is preferably operative to receive user input to extend one of thepages52 to the extendedpaneled page58. Thedisplay module55 is preferably operative to show the extendedpaneled page58 on thescreen38.
• Reference is now made toFIG. 8, which is a view of a plurality of tabbed graphical paneledpages62 for a channel viewing mode of thehigh definition EPG54 ofFIG. 5. Reference is also made toFIG. 9, which is a view of the tabbed graphical paneledpages62 ofFIG. 8 showing a paneledpage64 being peeled away from the other tabbed graphical paneled pages62. Reference is also made toFIG. 4.
• The tabbed graphical paneledpages62 preferably include a plurality ofvideos66. Thevideos66 may be arranged in the tabbed graphical paneledpages62 in many ways to help with navigation and, searching. Thehigh definition EPG54 is preferably operative such that the viewer can switch the view at any suitable time from one where the facilities of the set-top box system12 are shown, to, for example, one where all of the TV channels are seen.
• In TV channel mode, each channel is shown on a different paneled page of the tabbed graphical paneled pages62. In other words, each of thepages62 is uniquely associated with one TV channel. Therefore, the channels layer back into the distance with one tab of the tabbed graphical paneledpages62 showing one channel. Thevideos66 on each paneled page, in the example ofFIG. 8, span from left to right.
• Thehigh definition EPG54 is preferably operative such that the viewer can move left and right on each paneledpage62 or press a button to extend any of the paneledpages62 upwards, if required. Navigating left and right moves a focus to a different piece of content which can be played by viewer selection. The different pieces of content may be the next piece of content to be broadcast on a channel or other items such as the last piece of content recorded or the last game played on the channel, by way of example only.
• Traditionally, EPGs use a timeline associated with each piece of content such that the viewer determines when a piece of content is to be aired. Thehigh definition EPG54 generally assumes that content is increasingly recorded and available locally so the paneled pages show only what is available. Therefore, thehigh definition EPG54 typically uses the screen space to promote the content rather than the scheduling time.
• Reference is now made toFIG. 10, which is a view illustrating right navigation in a paneledpage68 of the tabbed graphical paneledpages62 ofFIG. 8. Eachpage62 typically includes a plurality ofgraphical panels79. In the channel viewing mode, the viewer moves in and out of the screen to look at different channels or moves left/right to see what content is available on the channel. The viewer can move in and out of the screen at the same time as moving left/right.FIG. 10 shows a viewer moving the paneledpage68 to the right.
• Theuser input module57 is preferably operative to receive a user input to move afocus77 between thegraphical panels79 in thepage68. Thedisplay module55 is typically operative to display thefocus77 on one of thegraphical panels79.
• Theuser input module57 is also preferably operative to receive a user input to turn one or more of thepages62 and move a position of thefocus77 of thegraphical panels79 in thepages62 at the same time, for example, so that the viewer can move to agraphical panel79 which is in a diagonal position from where the viewer is currently positioned.
• Reference is now made toFIG. 11, which is a view illustrating a plurality of tabbed graphical paneledpages70 labeled with a plurality ofchannel numbers72 of thehigh definition EPG54 ofFIG. 4. The paneled pages70 show channels layering back into the screen and content available on the channel is arranged on each paneledpage70. However, it will be appreciated by those ordinarily skilled in the art that the tabs can be arranged in different ways, for example, by viewer configuration, thus helping with navigation and searching.
• Reference is now made toFIG. 12, which is a view illustrating a plurality of tabbed graphical paneledpages74 labeled alphabetically of thehigh definition EPG54 ofFIG. 4. Each of the tabbed graphical paneledpages74 includes a plurality ofgraphical panels75. The content shown inFIG. 12 is arranged alphabetically from front to back. In other words, the tabbed graphical paneledpages74 are arranged alphabetically from front to back. Thegraphical panels75 on aparticular page74 are also arranged alphabetically from left to right. By way of example only, the content may be arranged chronologically where time stretches back into the distance so that thepages74 are arranged chronologically and thegraphical panels75 in each paneled page74 (available during a particular time period) are arranged chronologically from left to right.
• Reference is now made toFIG. 13, which is a block diagram view of the set-top box system12 ofFIG. 1.FIG. 13 describes the highest level functional blocks of the set-top box system12.
• The set-top box system12 is preferably operative to provide a mix of high definition (HD) and standard definition (SD) inputs and a mix of TV output standards in the same set-top box. The set-top box system12 typically incorporates a personal video recorder (PVR). In order to keep costs and risks of the set-top box system12 low, the set-top box system12 is typically implemented by designing the silicon chips of the set-top box system12 as a natural evolution of existing and proven set-top box building blocks.
• Inevitably more screen movement, color and blending means bigger and more graphical assets. Therefore, the set-top box system12 is preferably operative to keep the costs of storage and transmission down through the extensive use of compression or vector techniques described hereinbelow, in particular with reference toFIGS. 25-40.
• The set-top box system12 is preferably operative to include one or more of the following general features: support the decoding and processing of multiple video feeds; support multiple locally generated graphical layers; support multiple locally generated video layers; support the compositing of multiple layers of video and graphics to create a final viewer experience; support the decoding and processing of multiple incoming audio feeds; support multiple locally generated audio samples to create a final viewer experience; all operations are inherently synchronized and controlled against one common set of timelines; video and graphics are allowed to merge such that it is not generally possible to tell the difference between the video and graphics; and support complex multi-layer graphical constructions.
• The set-top box system12 is preferably scalable to allow for more outputs and inputs of different types.
• Additionally or alternatively, the set-top box system12 is preferably operative so that video and audio of different line standards and resolutions are handled.
• The set-top box system12 is preferably operative such that any loss in video or audio quality through thesystem12 is zero in situations where no processing is done. When processing is performed, the degradation in the video or audio quality is not typically perceived by a viewer/listener.
• Optionally, the set-top box system12 supports: video and audio input from local audio-visual devices attached to the set-top box system12; and plug-ins which are used to process video and audio flowing through the set-top box system12.
• The set-top box system12 preferably supports: 2-way remote controls; allowing multiple controllers to be used simultaneously; and multiple key presses simultaneously from any suitable controller and optionally key-repeat and button-up functionality. The controller(s) are typically implemented using technology such as Bluetooth, IR blasters, and wireless LAN technology standard IEEE 802.11 as known to those ordinarily skilled in the art of remote controllers.
• The set-top box system12 optionally supports one or more of the following: video and audio streams which originate from connected devices; video and audio which originate from dedicated base band inputs; and plug-able advanced controllers.
• The set-top box system12 preferably includes abroadcast decoder subsystem76, avideo effects processor78, agraphical effects processor80, avideo compositor82 and anaudio effects processor84.
• Thebroadcast decoder subsystem76 is now described briefly. Thebroadcast decoder subsystem76 is described in more detail with reference toFIGS. 20-23.
• Thebroadcast decoder subsystem76 includes the primary decoders for the set-top box system12. Thebroadcast decoder subsystem76 preferably includes at least two video broadcast decoders176 (FIG. 20) in order to allow for channel changing transition effects and to allow for content to be jumped around seamlessly.
• The compression technology of the received video is assumed to depend on other broadcaster requirements. At the time of writing, MPEG-2 is typically the default compression technology for Standard Definition systems and MPEG-2 or H.264 for High definition systems. Therefore, whilst the set-top box system12 does not assume a particular technology, the set-top box system12 does make some assumptions about the way the compression systems work and the features available. For example, at the time of writing, an alpha channel is not carried with the MPEG-2 and H264 compression systems. It will be appreciated by those ordinarily skilled in the art that the components of the set-top box system12 may need to be adapted to the compression technology used by the broadcaster.
• Thebroadcast decoder subsystem76 is preferably operative to support: all suitable output TV resolutions for example, but not limited to, up to and including 1080×1920; all appropriate frame rates, for example, but not limited to, up to 60 Hz; interlaced and non-interlaced standards; and all appropriate video aspect ratios for example, but not limited to, 16:9 to/from 4:3, and Letterbox/Pillbox to full screen.
• Thebroadcast decoder subsystem76 is also preferably operative to support: the set up and timing of video transitions from one decoded video input to another; the scaling and transformation of video under software control on a frame by frame basis; an alpha channel to be added to the incoming video under software control on a frame by frame basis; and application of different levels of alpha from a simple fixed value to a complex per pixel alpha which changes on a frame by frame bases.
• Thebroadcast decoder subsystem76 is preferably timeline aware such that at all times it is preferably possible for software to read the timing related information for the current frames being decoded where the timing information preferably includes one or more of the following: timeline control protocol information from a separate packet identification (PID); time codes derived from the compressed video group of pictures (GOP) and/or Picture headers; and from presentation time stamp (PTS) and/or decoder time stamp (DTS) and/or program clock reference (PCR). Thebroadcast decoder subsystem76 is typically operative to enable a field or frame based interrupt which is synchronized to the relevant vertical blanking interval (VBI) period, preferably by software control.
• Thebroadcast decoder subsystem76 is preferably operative to cue up and play content from storage, typically including: under software control, one of thevideo broadcast decoders176 cues up, ahead of time, such that a specific frame or field is decoded to the output of the video broadcast decoder; under software control, for content which has already been cued up to be played (the maximum time required between cueing up content and playing the content is typically less than a second; inserting content off disk into either an incoming live broadcast feed or another disk based feed using the above cueing technique; cue-up up to four (or more) alternative disk based streams using the cueing method and select only one of the disk based streams for play-out; and de-cue material which has previously been cued up. Cueing of material is described in more detail with reference toFIGS. 22 and 29.
• The set-top box system12 is preferably operative such that frames from multiplevideo broadcast decoders176 are synchronized (genlocked) to a common time base.
• Thevideo effects processor78 is now described briefly. Thevideo effects processor78 is described in more detail with reference toFIGS. 24-43.
• Thevideo effects processor78 handles all video-based effects for the set-top box system12. TV graphics are becoming more televisual and there is generally more motion, meaning that constantly moving overlays and effect sequences synchronized to the main video feeds are both becoming more important. Thevideo effects processor78 therefore includes all the decoders and handlers for pre-rendered bitmaps and decoders.
• Thevideo effects processor78 is preferably operative to playout moving video backgrounds or overlays preferably including: a plurality (for example, but not limited to, two) separately controllable full screen overlays or underlays; clips with a duration in the region of between 100 milliseconds and several hours, for example; play back in a loop with a seamless cut between start and end from/to any suitable frame; and seamlessly concatenate new clips onto the end of the clip that is currently playing.
• Thevideo effects processor78 is optionally operative to playout moving video thumbnails preferably including: movie thumbnails of arbitrary size which collectively cover up to the full size of the output screen; medium duration clips with a duration in the region of between 10 seconds and 20 minutes for example; play back in a loop with a seamless cut between start and end from/to any suitable frame in the sequence; and seamlessly concatenate new clips onto the end of the clip that is currently playing.
• Thevideo effects processor78 is preferably operative to support full 24 bit color, by way of example only, for all video related effects.
• Thevideo effects processor78 is typically operative to apply a simple alpha channel (which is fixed), or a complex alpha channel (which changes every frame), to the video layers.
• Thevideo effects processor78 is preferably operative such that: the speed and direction of the playback of any suitable video clip can be varied; there is substantially no restrictions on the size and shape of any video overlays (for example, but not limited to, the video being the bottom 30 lines of the screen); and the cue up and frame accurate triggering of play-out start and stop is supported as well as the storage and loading of reference frames and a decoding state.
• Thevideo effects processor78 is preferable operative such that any reference image generated in the decoding process may be transferred to an external storage independent of the dedicated decoder buffer. The video effects processor is preferable operative such that any reference images used in the decoding processor may be transferred from (loaded from) an external source. The video effects processor is operative such that loaded reference images need not necessarily have been generated from the video effects processor. Similarly, any state information used in the decoding process of a coded stream, preferably at any point in the processing of the stream, may be saved, and later loaded back into the video effects processor.
• Thevideo effects processor78 is preferably operative such that, when informed of a region of interest in frames of an input stream, only the region of interest is selectively decoded from the frames of the input stream. Thevideo effects processor78 preferably ignores or skips components of the stream that do not include the region of interest. The region of interest is typically at least one of the decoding data or the output data (for example, thevideo effects processor78 may decode a slightly larger region than the region outputted by the video effects processor78). The above method generally saves both memory and processing bandwidth.
• Thevideo effects processor78 is preferably operative to: loop-decode sub-regions of a frame(s) with a seamless cut between the start and end of a video clip; and seamlessly concatenate new clips formed from sub-regions of a frame(s) onto the end of a clip that is currently playing.
• Thevideo effects processor78 preferably notifies applications running on amain CPU212 of the set-top box system12 (through an interrupt) of the following: each frame VBI start point; end of clip and start of clip; and embedded frame time code.
• Thevideo effects processor78 is preferably operative such that MPEG I-Frame tiling or PNG tiling is supported. Thevideo effects processor78 preferably supports the direct reading/writing between themain CPU212 and any suitable video effects buffer (not shown).
• Thegraphical effects processor80 is now described briefly. Thegraphical effects processor80 is described in more detail with reference toFIGS. 44-45.
• Thegraphical effects processor80 typically includes contains all of the graphics handling for the system, for example, but not limited to, all the graphics which are generated in real time in the set-top box system12. Examples include Vector based graphics and text rendering.
• Thegraphical effects processor80 is preferably operative to include one or more of the following text rendering features: antialiasing; transforms (3×3 affine; supporting scaling, rotation, and shearing); animations; rendered stroked and filled (solid color, gradient filled and texture); shape effects such as bevels; highlighting effects such as drop shadows; and bitmap or vector formats.
• Thegraphical effects processor80 is preferably operative to include one or more of the following vector graphic features: arbitrary shapes supported using paths comprising straight line segments; quadratic and cubic Bezier curve segments; ellipse arc segments; antialiasing; transforms (3×3 affine; supporting scaling rotation, shearing); animations; rendered stroked and filled (solid color, gradient filled and texture); mapping SVG Basic and/or Macromedia Flash to hardware.
• Thegraphical effects processor80 is preferably operative to support hardware decoding of I-frame or PNG bitmap formats.
• Thegraphical effects processor80 is optionally operative to support hardware decoding of one or more of the following formats: GM; JPG; DV or BMP images, depending on system requirements.
• Thegraphical effects processor80 is preferably operative to support the following color formats and conversions, by way of example only: up to 32 bits per pixel RGBA or YUVA (4:4:4 or 4:2:0); all CLUTx formats; 1 or 8 bpp alpha channel; conversion from YUV 4:2:0 to RGB 4:4:4; conversion from CLUTx to RGB888; separately addressable alpha channel. It should be noted that the set-top box system12 may have to support other color formats if, for example, the set-top box system12 needs to support DV standard format.
• Thegraphical effects processor80 is preferably operative to support a direct drawing interface (pixel level access to a memory surface) and the reading from the main CPU212 (FIG. 13) of any suitable memory buffer used by thegraphical effects processor80.
• Thegraphical effects processor80 is optionally operative to support the rendering of 3D models.
• The set-top box system12 is preferably operative to ensure that the geometry of other elements in thesystem12 is consistent with thegraphical effects processor80 by issuing geometry co-ordinate commands originating from thegraphical effects processor80.
• Thegraphical effects processor80 is preferably operative to support the delivery of vector generated alpha planes to other parts of thesystem12.
• Thegraphical effects processor80 is typically operative to accept assets incoming from thevideo effects processor78 for textures, by way of example only.
• Thevideo compositor82 is the main compositor for the set-top box system12. Thevideo compositor82 typically takes a plurality of full resolution, full frame rate, full color feeds86 from the other subsystems of the set-top box system12 and composites thefeeds86 together. Thevideo compositor82 outputs acomposite feed88 to an output stage (not shown) of the set-top box system12.
• Theaudio effects processor84 is now described briefly. Theaudio effects processor84 is described in more detail with reference toFIGS. 46-47.
• Theaudio effects processor84 handles all of the audio in the set-top box system12. Banks of a plurality ofsamples90 are played out, looped and mixed with a plurality of other audio feeds92. Theaudio effects processor84 includes an audio mixer394 (FIG. 47) which mixes thesamples90 and other audio feeds92 with a plurality of audio feed(s)96 from thebroadcast decoder subsystem76, creating afinal audio output98.
• Theaudio effects processor84 is preferably operative to support: the selection, mixing and transitions between incoming broadcast audio feeds96; and multi-language system capabilities.
• Theaudio effects processor84 is preferably operative to support the locally injectedaudio samples90, for example, but not limited to: up to 16 or more compressed streams where the compressed audio formats supported include formats supported by theaudio effects processor84 such asMPEG1 Layers 1,2,3; up to 16 or more stereo PCM uncompressed streams; all sample rates/precisions supported by the compression standards used in the platform for example up to 48 KHz, 16 bit max; mixing of different input rates and precisions; seamless looping and concatenation of clips; ramp-like transitions between clips where the ramp up/down rate and the shape of the ramp is controllable (s-shaped ramps for example).
• Theaudio effects processor84 is preferably linked into the overall synchronization mechanism of the set-top box system12, described below.
• Additionally, theaudio effects processor84 is preferably operative to: cue up and play an audio clip with sample accurate in/out points; start and stop play-out of audio clips with a timing that is at least as accurate as that used for the video playout; audio ramp up/down to lead/lag video or graphics; support 4.1, 5.1, 6.1, 7.1 multi-channel audio systems by way of example only; support the mixing of audio in any suitable combination of locally insertedclips90 and otheraudio inputs92; split all inputs into the mono components of the input before entering the audio mixer394 (FIG. 47) in order to allow precise control of the position of any suitable audio track in a multi-channel surround-sound system; and support the control of attenuation on any of the inputs and outputs to the audio mixer394 (FIG. 47).
• Theaudio effects processor84 optionally supports: the addition of audio effects to any suitable channel into, or out of, the audio mixer394 (FIG. 47) for example, but not limited to, echo, reverb, panning, time stretching, pitch shifting, dynamic range compression; and input from local audio devices attached to the set-top box system12.
• Theaudio effects processor84 is preferably operative such that any suitable audio source can be transferred between the main CPU212 (FIG. 13) and theaudio effects processor84.
• There is preferably inherent synchronization throughout the set-top box system12. All video coming in from external sources is aligned on acommon time base100. If necessary, frames are typically dropped or inserted to keep the synchronization. All video frames and audio samples are preferably time stamped and the time order is typically constantly maintained as the information is clocked through the set-top box system12.
• In addition, the set-top box system12 is continuously aware of broadcast or local timelines. The broadcast timelines may be present in incoming audio/video (A/V) feeds, in which case the broadcast timelines are always kept with video frames as the video frames are clocked through the set-top box system12. The broadcast timelines may be present in incoming audio/video/data feeds. The broadcast timelines are typically used to calibrate one of a plurality of high-resolution clocks (for example, but not limited to, 90 KHz or above) (not shown) included in the set-top box system12 to enable low-level timing control of the output audio and video signals. Local timelines are typically generated from the clock without external calibration.
• Events may then be scheduled against the timelines, for example, but not limited to, a compositor may reconfigure itself synchronously to an entry in a given timeline.
• Several components appear in a number of places in the set-top box system12. The same components are all assumed to have substantially the same functionality, and the same components may physically be the same device. In the description below, the functionality is partitioned to aid understanding and suggest where partitioning may be designed into the silicon, if necessary.
• Reference is now made toFIG. 14, which is a view of an image transform engine102 (ITE) for use with the set-top box system12 ofFIG. 13. Theimage transform engine102 is operative to quickly move blocks of pixels from one area of memory to another, optionally perform an operation on the pixels and alpha-blend the pixels into a target area. The operations are direct memory access (DMA) based and operate asynchronously to the main CPU212 (FIG. 13) processing. The format of the input and output buffers typically conforms to any format listed in the features of theimage transform engine102 listed below. Theimage transform engine102 is preferably controlled through a list of instructions which, once enabled executes automatically in sequence, triggered either from the completion of execution of the previous instruction or linked to a timing trigger. It should be noted that it is typically possible to modify the list of instructions dynamically (“on the fly”). Theimage transform engine102 is generally referred to as a BitBlt device or Blitter in STB hardware systems.
• TheImage Transform Engine102 generally has one input and one output, the input and output memory area ranges typically having an arbitrary size.
• On transfer of memory from input to output, the image transform engine102 optionally allows one or more of the following operations to be performed: moving and/or copying including sub-pixel copy/move to allow for slow/accurate animation and source area removal to allow for animation; color including color space conversion (YCbCr (4:2:0)->RGB888(4:4:4)), color expansion (CLUTx->RGB888, channel splitting, channel mixing, channel combining, separately addressable color/alpha channels), color correction (gamma, contrast, gain, brightness, hue/saturation/lightness); scaling including unlimited up-scaling, unlimited down-scaling, high quality scaling algorithm (for example, but not limited to, N-tap filters where N is larger than 2, and the tap-filters may be applied in several dimensions such as Bilinear and Bicubic algorithms), any suitable number of source/target lines or columns; shaping including transforms (3×3 affine; supporting scaling, rotation and shearing)), other non-linear deformations from/to any suitable arbitrarily shaped area (e.g.: page turns); filtering including blurs (Gaussian), sharpening, edge detection/embossing, filtering applied to any 4 points for example or any suitable bounded area as specified through vector based co-ordinate system; blending including alpha blend a new source item onto a target item (in other words, every pixel transferred is a read, blend, write operation (alpha based porter-duff)).
• It should be noted that color or alpha fill typically resides in a vector engine.
• Theimage transform engine102 also typically allows alpha planes to be separately addressed. By way of a non-limiting example, an 8 bit PNG can be decoded to form the alpha plane of an existing RGB image.
• Theimage transform engine102 is typically operative to perform at least 20 full output screen sized operations per frame period or the equivalent pixel transfer rate.
• The set-top box system12 includes several image transformengines102, described herein below. Eachimage transform engine102 performs substantially the same function.
• The set-top box system12 also includes a memory pool (not shown). The memory pool preferably includes all memory buffers used for storing transient images, audio and video frames in the set-top box system12. As far as possible, all blocks of data in the set-top box system12 are preferably accessible from the main CPU212 (FIG. 13) to allow for any suitable intermediate processing or plug-in like architectures. It is generally always possible to read flows of data going through the set-top box system12, modify the data flows, and re-inject the data flows into the correct flow.
• To ensure that the main CPU212 (FIG. 13) can interpret the blocks of data, the blocks of data preferably have a standard header which is used by application software to track and process frames.
• Reference is now made toFIG. 15, which is a view of asymbol103 used to represent the addition of an alpha channel in the set-top box system12 ofFIG. 13. Alpha blending is typically used extensively in the set-top box system12 by the image transformengines102 and compositors (described in more detail with reference toFIGS. 17-19) to make graphics and video appear to blend into one overall visual presentation or to create effects like transitions. To enable alpha blending, it is necessary to specify an alpha plane for any suitable video or graphics based source.
• For MPEG or other broadcast compression formats, an alpha channel is not always supported as part of the standard, and rarely used even when supported. Therefore, if an alpha channel is required to create an effect like a video transition for example, an alpha channel must be defined dynamically (“on the fly”) as the video is being decoded.
• Reference is now made toFIG. 16, which is a detailed block diagram view of analpha generator104 of the set-top box system12 ofFIG. 13. The set-top box system12 includes thealpha generator104 which is typically operative to apply an alpha channel. The alpha channel may be applied in a number of different ways, each of the ways supporting a different kind of effect.
• Thealpha generator104 is used to generate analpha channel106 for aframe116 where one is not already available. Thealpha generator104 is typically used in all the video and graphics sub systems of the set-top box system12.
• Thealpha generator104 preferably supports one or more of the following alpha capabilities: a simple 1, 8, or 16 bit value is applied to each frame116 (block108); a per-pixel value which is filled from the vector based graphics system allowing for selective areas, gradient effects and shaped area fills (block110); a per pixel value which is filled from a sequence of 1, 8 or 16 bit PNG images (block112); thealpha channel106 is derived from the original image (block114), in particular the RGB (block128), thealpha channel106 being used to generate drop shadows, luma or chroma keying whereby the color and tolerance or a luma threshold are defined.
• Thealpha generator104 has many ways of adding thealpha channel106 to eachframe116 being clocked through the set-top box system12. The ways of adding an alpha channel are set up from the main CPU212 (FIG. 13) as a sequence of instructions (block118) which are triggered without CPU intervention of VBI triggers (block120). The overall system geometry control (block122) is an input to thealpha generator104. The overall system geometry control is used in thealpha generator104 to define the position and shape of the area to be filled. For vector based fills, the overall system geometry control also defines the type of fill. The timelines (block124) are also an input to thealpha generator104. Thetimelines124 and the VBI triggers120 are synchronized (block126) allowing instructions (block118) loaded into thealpha generator104 to be frame accurately actioned.
• Reference is now made toFIG. 17, which is a block diagram view of acompositor130 for use with the set-top box system12 ofFIG. 13. There areseveral compositors130 in the set-top box system12. Each of thecompositors130 shown in the set-top box system12 may embodied as a single component. However, thecompositors130 have been described as separate components herein in order to allow for partitioning of hardware, should partitioning prove necessary. Each of thecompositors130 typically performs substantially the same function(s). Eachcompositor130 typically includes one or more of the following features described below.
• Eachcompositor130 preferably composites a plurality (for example, but not limited to, 6) separate full resolution, full frame rate, full color input video sequences together to form a single full resolution, fullframe rate output134. The input video sequences are input to thecompositor130 via a plurality of V/G feeds132.
• Eachcompositor130 preferably allows a background color to be defined as an RGB888 value (arrow136). The background color is the color against which alpha blending is preferably performed. No background is generally assumed for upstream compositors.
• The Z-order of any of the input video sequences is preferably definable through software control (arrow138). Only the background color plane is generally effectively fixed in position as the last or furthest layer away.
• The input frames of the input video sequences may be alpha blended together according to the values of the “per frame” or “per pixel” alpha attributes via a geometry control input (arrow140).
• Thecompositor130 also preferably has a timeline control input (arrow142) for synchronizing the input video sequences.
• The compositor typically includes ascaler156, the functionality of which is described in more detail with reference toFIG. 18.
• Reference is now made toFIG. 18, which is a block diagram view of an example process performed by thecompositor130 ofFIG. 17. Thecompositor130 typically takes input from the various input video sequences and composites, and/or alpha blends, the sequences together into thefinal output134.FIG. 18 shows aninput window146 of one of the input video sequences in aninput buffer150.FIG. 18 also shows anoutput window148 in aninternal buffer152 of thecompositor130 and adownstream buffer154.
• Theinput buffer150 is connected to one of the V/G feeds132 of thecompositor130. Theinput window146 is defined and stored in theinput buffer150. Theinput window146 gives thecompositor130 an input region to be transferred to theoutput window148. Theoutput window148 is preferably defined through software setup.
• Thewindows146,148 define the area and position of the input pixels and define where the input pixels are positioned in theoutput134. Therefore, by implication thewindows146,148 define whether any scaling is performed by thescaler156 on transfer of the pixels from theinput window146 to theoutput window148. There is an output window defined for each input video sequence.
• The video sequences may be alpha blended together as defined by the Z order (arrow138 ofFIG. 17).
• Theinternal buffer152 is clocked out of thecompositor130 to either the TV output circuitry or thedownstream buffer154, depending on where thecompositor130 is located in the set-top box system12.
• Reference is now made toFIG. 19, which is a block diagram showing combined use of operationally connected elements preferably including thecompositors130, the image transformengines102, one of thealpha generators104 and a plurality ofmemory buffers158 for use with the set-top box system12 ofFIG. 13. Starting at thedownstream compositor130, namely acompositor160, thecompositor160 preferably composites a plurality ofinputs162 together. The memory buffers158 and a plurality ofwindows164 of the memory buffers158 are typically defined through software control. The memory buffers158 acting as input buffers for thecompositor160 are typically filled from anyother memory buffer158 in the same memory domain by using one of the image transformengines102. The image transformengines102 typically transform blocks of pixels as already described above with reference toFIG. 14.
• Upstream compositors typically operate in substantially the same way as thecompositor160. Anoutput166 of anupstream compositor168 is preferably output to a software definedwindow172 of adownstream buffer170.
• It should be noted that thebuffers158 connected to the input of theupstream compositor168 may be partitioned in a different memory domain than the memory buffers158 attached to thedownstream compositor160, for example, but not limited to, use in bus partitioning, if appropriate.
• It should also be noted that the main CPU212 (FIG. 13) preferably needs to have read/write access to all the memory buffers158 except the buffers that are internal to thecompositors130. The access to the memory buffers158 generally allows the main CPU212 (FIG. 13) to perform intermediate processing on video, audio or graphics passing through the set-top box system12. The only point that is generally not accessible by the main CPU212 (FIG. 13) is a final compositedoutput174 of thedownstream compositor160. In higher security implementations, access by theCPU212 to buffers containing compressed video data, and possibly to uncompressed video data, may be prohibited, or severely limited,
• The number of outputs and the way that the hardware of the set-top box system12 is constructed to accommodate the outputs is typically very much platform dependant, but it is assumed that there are generally two different types of multiple output configurations as follows.
• The first configuration is a “same output presentation” which is typically used where an exact copy of the TV output is required, and possibly at a different scale, for example, but not limited to, to feed a VCR or second screen in another room. In the cases of the previous examples, it might be necessary to scale the TV output up or down. The scaling is assumed to be performed through the addition of a scaler (not shown) on the output side of the finaldownstream compositor160.
• The second configuration is a “different output presentation” which is typically adopted where a completely different output presentation is required, for example, but not limited to, for an independent TV output or for showing the same TV presentation at the same scale, but with a different composition. The “different output presentation” configuration might be appropriate if the set-top box system12 is operative to support multiple TVs in different rooms, for example, where each room can independently use the facilities of the set-top box system12.
• The “different output presentation” configuration typically necessitates multiplevideo effects processors78,graphical effects processors80 and audio effects processors84 (FIG. 13). However, multiplevideo effects processors78,graphical effects processors80 andaudio effects processors84 may not be necessary if reduced functionality is acceptable for some TV outputs. The exact layout is therefore platform dependent.
• Reference is now made toFIG. 20, which is a block diagram view of thebroadcast decoder subsystem76 for use with the set-top box system12 ofFIG. 13. Thevideo broadcast decoders176 of thebroadcast decoder subsystem76 are typically the main input decoders to the set-top box system12. Thevideo broadcast decoders176 are preferably operative to decode a plurality ofbroadcast streams178 in real time off-air or from disk. Thevideo broadcast decoders176 preferably handle any suitable compression format required by the set-top box system12. Features like “trick mode” are supported by the set-top box system12. As described above with reference toFIG. 13, a minimum of twovideo broadcast decoders176 are required for the set-top box system12. If appropriate, morevideo broadcast decoders176 are added to thebroadcast decoder subsystem76 by increasing the number of component chains.
• In thebroadcast decoder subsystem76 subsystem, thevideo broadcast decoders176 are augmented with additional components to form a larger sub-system. The following section describes the typical features of thebroadcast decoder subsystem76.
• Thebroadcast decoder subsystem76 is preferably operative to: enable the decoding of at least two broadcast video streams178 yielding a plurality of decodedvideo streams180; and to condition the decodedvideo streams180 to allow downstream mixing and effects to be added to the decodedvideo streams180; to extract timing and time-line information from thestreams180 in order to allow downstream effects to be performed on, or added to, the decodedvideo streams180 in way that can be synchronized to the underlying content of thestreams180; and to allow video transition effects to be performed on the decoded video streams180.
• The compressed broadcast streams178 may be from any suitable “broadcast originating” source, but are generally either from a live streamed source or a stored file based source.
• The typical flow through thebroadcast decoder subsystem76 is now described.
• First, thevideo broadcast decoders176 typically decode the compressed broadcast streams178 to base band YCbCr 4:2:0 decoded video streams180. A plurality of timing data184 or metadata embedded in thevideo stream180 is extracted and passed up to controlling software typically managed by aneffect controller182.
• The video streams180 are then processed to pre-condition the video streams180 ready for the rest of the set-top box system12 including: pan and scan vector processing; up/down scaling that is required as part of the compression syntax; features relating to conditioning the content for the rest of the set-top box system12; de-interlacing; optionally frame rate conversion (depending on the feature set of the set-top box system12); any additional up/down sampling required for video resolution matching prior to any transitions.
• The actual video transitions are typically performed through a combination of the image transformengines102, thealpha generator104 and thedownstream video compositor160. Thevideo broadcast decoders176, thealpha generator104, the image transformengines102, thecompositor160, thearea selector186 and theeffect controller182 are preferably operationally connected as shown inFIG. 20.
• Thealpha controller104 is used to create alpha channels for each of the decoded video streams180. In the example ofFIG. 20, the created alpha channels are assumed be applied to a whole picture before any suitable transformations are performed by the image transformengines102.
• The image transformengines102 are typically used to perform any suitable transforms necessary to create the desired transitions. If the transition(s) requires a video to shrink into the distance for example, one of the image transformengines102 may be programmed by anarea selector186 to shrink thevideo180 over the next few frames. Thearea selector186 is typically a software control module running on the main CPU212 (FIG. 13).
• The created alpha channels are typically used to create effects like cross fades, whereby the alpha is increased on one the video feeds180 and reduced on anothervideo feed180, for example. When the video feeds180 are composited downstream by thecompositor160, the cross fade effect is seen. In combination, thealpha generator104 and the image transformengines102 are typically used to create most transitions under software control.
• It will be appreciated by those ordinarily skilled in the art that the order of the image transformengines102 and thealpha generator104 may be reversed, for example the image transformengines102 can process thevideos180 before thealpha generator104. Additionally, it will be appreciated by those ordinarily skilled in the art that the created alpha channels can be applied to a region or regions of the picture before or after transformations are performed by the image transformengines102, as appropriate.
• Reference is now made toFIG. 21, which is a block diagram view of a preferred method of converting a plurality of video feeds188 to the same format for use with the set-top box system12 ofFIG. 13. Theincoming feeds188 may be different combinations of resolution, frame rate and aspect ratio. To allow for a transition between thefeeds188, the resolution, frame rate and aspect ratio factors need to be rationalized into a common format. However, the rationalization must preferably be done in a way that minimizes any reduction in video quality.
• Therefore, to ensure that interlaced frames can co-exist with non interlaced frames, interlaced frames are expanded to full frames including creating intermediate lines.
• The set-top box system12 preferably operates at a system frame/field rate of 50 or 60 Hz by way of example only. Therefore, any of thefeeds188 coming into thesystem12, whether interlaced or not, are expanded to the system frame/field rate. Hence interlacedfeeds188 with 60 Hz field rate are converted to a 60 Hz frame rate by interpolating the missing lines (block190). Alsonon-interlaced feeds188 with 30 Hz field rate are converted to 60 Hz by interpolating intermediate frames (block194).
• The above method ensures that all thefeeds188 are in the same format, meaning that all thefeeds188 can be composited together. However, thefeeds188 can also be converted back to the original format losslessly by removal of the additional lines (block196). Additionally, thefeeds188 may be left at 60 Hz frame rate or reduced to 30 Hz frame rate for progressive displays (block198).
• It should be noted that 1080i (interlaced) HD feeds are preferably converted to 1080P, thus doubling the bus bandwidth required for the set-top box system12. It should be noted that any suitable common format can be used in place of p60 format either to reduce the bandwidth requirements or to match the input and output formats. The format can even change dynamically rather than being a static system fixed format.
• Separate feeds in the set-top box system12 preferably need to be locked or synchronized together to ensure that a seamless switch from one video source to another is achieved. Given that the video sources can come from anywhere, a clock synchronization mechanism (not shown) is preferably provided in the set-top box system12.
• The set-top box system12 is preferably operative such that all switching is performed on VBI at the frame rate of the set-top box system12.
• All content time synchronization information is typically extracted as described with reference toFIG. 13. The synchronization information is typically from several sources which can be enabled under software control. The timing information is made available to controlling software running on the main CPU212 (FIG. 13).
• Reference is now made toFIG. 22, which is a block diagram view illustrating a preferred method of insertion of a plurality ofalternative material200 for use with the set-top box system12 ofFIG. 13.
• The mainvideo broadcast decoders176 have some special features used to handle the frame accurate switching or insertion of the alternative A/V material200. Thealternative material200 is typically stored in adisk arrangement205 as shown inFIG. 22. It will be appreciated by those ordinarily skilled in the art that thedisk arrangement205 may be a single storage unit or separate storage units which are operationally connected to each other.
• One of thevideo broadcast decoders176, namely avideo broadcast decoder204, is preferably operative to handle cueing up of the alternative material200 (off disk material) for insertion into either incoming live feeds from a tuner (not shown) or another disk basedcontent202 with frame accuracy. When thecontent202 is disk based content, thecontent202 is typically stored in thedisk arrangement205. As shown inFIG. 22: thecontent202 is typically decoded by the othervideo broadcast decoder176; the output of thedecoders176 is preferably composited by acompositor203 such that thealternative material200 is inserted into thecontent202; and thevideo broadcast decoders176, thedisk arrangement205 and thecompositor203 are preferably operationally connected.FIG. 22 shows up to four disk basedsources200, by way of example only, being cued up in thevideo broadcast decoder204 ready for insertion. The cueing method is preferably used for applications where alternative material needs be inserted, like advertising material, or applications where a key press is required to make the final decision as to which piece of material is to be inserted. Telescopic Advertising, Xtreamplay of NDS Limited or DVD-like menuing and games are examples where the cueing feature is typically used. Cueing of material is described in more detail with reference toFIG. 29.
• The requirements for trick mode are generally defined or limited by the platform operator and the compression algorithms being used. For the set-top box system12, however, any changes in speed and direction are generally substantially smooth and deterministic. Frames are preferably not be dropped or repeated below a known threshold (for example, but not limited to 12 frames per second or less) and any latencies and speed increments are preferably minimized thereby giving smooth ramp up/down/acceleration control to the video playback.
• Reference is now made toFIG. 23, which is a detailed block diagram view of thebroadcast decoder subsystem76 of the set-top box system12 ofFIG. 13.FIG. 23 shows thebroadcast decoder subsystem76 including thevideo decoders176 and a plurality ofaudio decoders206.
• Under normal viewing, it is expected that a plurality ofaudio streams208 associated with the video streams178 are either decoded by theaudio decoders206 and passed through the set-top box system12 to theaudio effects processor84 or fed through the set-top box system12 without any change, depending on audio type and the final audio output. Theaudio effects processor84 is described in more detail with reference toFIGS. 46-47.
• Thebroadcast decoder subsystem76 includes anunpacker210 for extracting data of a plurality of supplementarysynchronized streams214 for subtitles, closed captions and possibly other streams such as DVD format including sub-picture auxiliary streams. The supplementarysynchronized streams214 are extracted and fed to themain CPU212. The data is preferably processed within themain CPU212 and dropped back into the graphics system through thegraphical effects processor80 described in more detail with reference toFIGS. 44-45. The extracting is typically hardware accelerated.
• Additionally, a plurality oftimeline data216 is preferably extracted by atimeline extraction module218 and forwarded to themain CPU212 and a plurality of video/audio data buffers220.
• By way of introduction, several video effects processors are now described with reference toFIGS. 24-43. It will be appreciated that any of the video effects processors described with reference toFIGS. 24-43 can be used alone or in any suitable combination for use as thevideo effects processor78.
• Thevideo effects processor78 is preferably used to create composites of moving video, used to provide constantly moving backgrounds, effects or overlays. Thevideo effects processor78 is also preferably used to create collections of assets to be used by other elements in the set-top box system12. Thevideo effects processor78 is typically linked to, and complimentary to, the graphical effects processor80 (FIG. 44).
• The objectives of thebroadcast decoder subsystem76 are generally as follows: to create a scalable, cost sensitive, sub-system for generating moving video backgrounds, overlays and effects; to handle any timing and synchronization necessary to link the video effects to the rest of the set-top box system12; and to handle a media package loading mechanism described with reference toFIGS. 33-40.
• The video effects processors described with reference toFIGS. 24-43 are divided into two broad categories, namely a basic level video effects processor and an advanced level video effects processor.
• The basic level is generally for short duration looping backgrounds or overlays (max 2 mins) based on MPEG I-frames or PNGs. Any EPG movie thumbnails are typically bitmap stills.
• The advanced level is generally for long duration looping backgrounds or movie thumbnails based on Long GOP compression. The actual compression format used is typically arbitrary as long as the storage and transmission bandwidth requirements are met.
• Reference is now made toFIG. 24, which is a block diagram view of a basic levelvideo effects processor222 of the set-top box system12 ofFIG. 13.
• One of a plurality of I-frames224 are typically presented to an I-frame decoder226 during a frame period. The I-frame decoder226 preferably decodes any suitable shape of I-frame224 as long as the I-frame224 includes an integral number of macro blocks. The decoded output or regions of the I-frame decoder226 are then generally processed and copied with pixel accuracy onto a surface by theimage transform engine102. Following composition, thealpha generator104 typically adds an alpha layer to the I-frames224. The basic levelvideo effects processor222 preferably includes a frame selector225 for selecting individual I-frames as required.
• Overall, the basic levelvideo effects processor222 generally allows multiple regions of moving, full color video to be used as a background or overlay for example, but not limited to, providing background images for EPGs and other graphics utilizing I-frames, requiring interruption of the decoding process in order to display the I-frames, thereby providing a significant advance over existing systems. The existing systems typically use static screen shots generally requiring interrupting the CPU rather than having a specialized separate parallel decoder for I-frames.
• Reference is now made toFIG. 25, which is a block diagram view illustrating a preferred method of creating a plurality ofvideo overlays228 using thevideo effects processor222 ofFIG. 24. The basic level video effects processor222 (FIG. 24) preferably allows a set oftiles230 to be defined as a set of I-frames. Then, a decode-blit process (block232) is typically used to create an output (the video overlays228) from the tiles. The overall effect is to enable multiple regions of full color moving video to be used.
• In the example ofFIG. 25, regions (tiles230) of whole I-frames (I-frame 1 and I-frame n) are generally blitted in a timed sequence to createvideo overlays228. Obviously, thetiles230 of the I-frames (I-frame 1 and I-frame n) can be arranged in any suitable way, from each I-frame including one frame (tile230) of each region, to each I-frame including areas of many video overlays. Ultimately, the arrangement of the I-frames depends on which option gives the best compression. In a similar manner, it may be appropriate to alter the compression across the picture to give the correct amount of quality to a particular graphic.
• Reference is now made toFIG. 26, which is a block diagram view illustrating a preferred method of creating abackground234 or levels of a game using the basic levelvideo effects processor222 ofFIG. 24. In the example ofFIG. 26, a set of I-frames can be arranged as a plurality oftiles236 and used to createmany backgrounds234 or levels of a game, for example. It should be noted that combining the example ofFIG. 26 with the example ofFIG. 25, video tiles and animated and moving sprites can be supported, typically requiring the addition of an alpha channel. Alpha channels are described with reference toFIG. 27.
• Reference is now made toFIG. 27, which is a block diagram view of the basic levelvideo effects processor222 ofFIG. 24 with a PNG decoder arrangement238. To allow for more efficient palette based storage and/or higher precision color images, a chain of PNG decoding parallel to the I-frame decoding described with reference toFIG. 24 is preferably applied as shown inFIG. 27. The PNG decoder arrangement238 includes: aPNG decoder242 for decoding a plurality ofPNGs240. The output of thePNG decoder242 is preferably processed by another of the image transformengines102. The addition of an alpha channel to the PNG track is optional as the PNG image may already have an alpha channel.
• Optionally, the basic levelvideo effects processor222 includes separate tracks for JPEG, GIF or DV decoders, if necessary.
• With alpha capability, the basic levelvideo effects processor222 is very flexible and typically achieves one or more of the following effects: create multiple moving video regions of any suitable size, anywhere on the screen with alpha reveal and animation; create tile based backgrounds, where tiles are substituted on a frame by frame basis; create arbitrarily shaped, animated sprite sets using the pixel perfect PNG alpha capability; and generate sets of full color depth and alpha channel bitmap assets to be used in local compositions or passed to the graphical effects processor80 (FIG. 44) for further processing.
• Using I-frames and PNGs with long or large movie sequences typically requires large storage capabilities and transmission bandwidth, so a long GOP based version of theVideo Effects Processor78 is preferably required. Long GOP versions of theVideo Effects Processor78 are described below with reference toFIGS. 28-31 and43.
• Reference is now made toFIG. 28, which is a block diagram view of an advanced levelvideo effects processor244 for use with the set-top box system12 ofFIG. 13. The architecture of the advanced levelvideo effects processor244 is substantially the same as the architecture of the basic levelvideo effects processor222 except that the advanced levelvideo effects processor244 is preferably operative to handle long GOP compressed sequences (for example, a plurality of video sequences246) as well.
• The advanced levelvideo effects processor244 typically includes a plurality oflong GOP decoders248.
• The implementation of thelong GOP decoders248 is typically silicon dependent. However, it should be noted that the total output video area of each of thelong GOP decoders248 is never greater than one screen area. Therefore, through careful management of thelong GOP decoders248, it is generally possible to use the advanced levelvideo effects processor244 to decode one screen area from many sources.
• Reference is now made toFIG. 29, which is a block diagram view of a manageddecoder250 for use with the set-top box system12 ofFIG. 13. The manageddecoder250 typically includes asequence selector252, a longGOP decoder control254 and twolong GOP decoders256.
• The manageddecoder250 is a long GOP based decoder which preferably has the following attributes: reference frames used by thelong GOP decoders254 are managed such that multiple streams or stills can be decoded simultaneously; the manageddecoder250 is timeline aware (block258) and is controlled by the longGOP decoder control254 to decode to a specific frame in the video sequence246 (in other words, to decode to frame X, and put frame X in an output buffer of the manageddecoder250 until the manageddecoder250 is instructed to play the video).
• In all the video effects processors described herein, it is assumed that: the clips played out come from a storage arrangement (for example, but not limited to, thedisk arrangement205 ofFIG. 22); and the storage mechanism is managed in such a way that the storage mechanism is operative to cope with the required throughput.
• Using the two decoders256 (or alternatively thelong GOP decoders248 ofFIG. 28), the twovideo sequences246 may be decoded, each of thesequences246 at full screen, to base band video simultaneously, and if, the outputs of thelong GOP decoders256 are synchronized together, a switch can be used to seamlessly switch from one of the outputs to another of the outputs of thelong GOP decoders256. The above method is preferably used to seamlessly concatenate content together or to give seamless loops. The difficult part is to get the timing right such that the correct frame on onesequence246 is concatenated to the correct frame of theother sequence246. However, if content is coming from storage, and the manageddecoder250 is aware of which frame the manageddecoder250 is decoding, the manageddecoder250 is instructed by the main CPU212 (FIG. 13) to cue up to a specific frame (typically faster than real-time) in thevideo sequence246, ready to play (typically at real-time) from where the manageddecoder250 cued up to. Therefore, the manageddecoder250 decodes video up to a specific frame and saves the reference frame(s) and other decoding state information necessary to allow restarting the decoding from the exact same specific frame at which the manageddecoder250 stopped decoding. At a future point in time, the manageddecoder250 loads the reference frame(s) and the decoding state information and restarts decoding from the specific frame.
• Using the above cueing technique typically has two benefits. First, any timing issues are simplified, because the manageddecoder250 has the content at the correct frame, ready to playout on demand. Second, the concept of cueing up ahead of time is preferably used to reduce any real time constraints on the set-top box system12 and to manage the flow of content out of storage.
• Note that the control mechanisms required are substantially the same as the control mechanisms described in the video aspects of thebroadcast decoder subsystem76.
• InFIG. 29, acompositor260 is used as a switch with the relevant input being routed through to the output and clocked through on a VBI period.
• InFIG. 29, there are twodecoders256. However it is well known that decoders typically run faster than real time, so for real time playout, one decoder can be used to generate multiple outputs, by careful management of the reference frames that are used to generate the output, as will be described below with reference toFIG. 30.
• Reference is now made toFIG. 30, which is a block diagram view of a multiple input manageddecoder262 for use with the set-top box system12 ofFIG. 13. The multiple input manageddecoder262 preferably includes asingle decoder264 which processes theinput sequences246 and a plurality ofcorresponding reference frames266 on a time sharing basis. For each frame period, thedecoder264 typically decodes one frame fromvideo sequence1 usingreference frames1. Then, thedecoder264 typically switches tovideo sequence2 usingreference frames2, pushing the decoded frames for each sequence out in one frame period.
• The number of streams that thedecoder264 decodes is only typically limited by CPU capability, memory and the compression scheme used.
• For small video that only fill part of the screen such as promos, advertisements or banners, by way of example only, two full screen loads of video may processed by thedecoder264 at once.
• The reference frames266 may be stored in, and/or loaded from a decoderindependent buffer434, which is not a dedicated buffer of the multiple input manageddecoder262, even though the multiple input manageddecoder262 may make some use of thebuffer434. In other words, the reference frames266 may be transferred between the decoderindependent buffer434 and a dedicated buffer of the multiple input manageddecoder262. The term “transfer between” as used in the specification and claims is defined as loading a reference frame from thebuffer434 and/or storing a reference frame in thebuffer434. The term “dedicated” buffer as used in the specification and claims is defined as a buffer for exclusive use by a decoder. Thebuffer434 is typically a buffer of a host memory or thehost CPU212. It should be noted that the reference frames266 may or may not be physically moved when the reference frames266 are moved into or out of the dedicated buffer of the multiple input manageddecoder262 For example, the dedicated buffer of the multiple input manageddecoder262 may be reassigned to another memory location, which moves the reference frames266 into, or out of, of the dedicated buffer or thebuffer434. Therefore, the term “transfer between” as used in the specification and claims is defined to include moving a reference frame by reassigning the physical memory location of the dedicated buffer.
• Reference is now made toFIG. 31, which is a block diagram view of another advanced levelvideo effects processor268 for use with the set-top box system12 ofFIG. 13. The advanced levelvideo effects processor268 preferably includes a long GOP decoder272 managed by a longGOP decoder control274.
• In a similar way to that described with reference to the basic levelvideo effects processor222 ofFIG. 27, the advanced levelvideo effects processor268 typically includes theimage transform engine102 and thealpha generator104 to allow for the shaping, animation and alpha effects aspects.
• The advanced levelvideo effects processor268 also preferably includes aPNG decoder270 so that a plurality of I-frame references276 for the long GOP decoder272 may also come from aPNG source278.
• The basic level video effects processor222 (FIG. 24), the managed decoder250 (FIG. 29) and the advanced level video effects processor268 (FIG. 31) are typically operative to handle arbitrarily sized (in space and time) video clips as well as pushing out any suitable region of any suitable frame of any suitable clip on cue (given a disk access and cue up time). Hence, the basic level video effects processor222 (FIG. 24), the managed decoder250 (FIG. 29) and the advanced level video effects processor268 (FIG. 31) typically have much more advanced uses as described below with reference toFIGS. 32-40.
• Reference is now made toFIG. 32, which is a block diagram view of an illustration of a preferred method of playing out multiple clips simultaneously using the set-top box system12 ofFIG. 13. A plurality oftiles284 are located in an I-frame280 and an I-frame282. Thetiles284 are associated with different video clips or stills. For example, all thetiles284 labeled “tile1” are associated with a single video clip. Anoutput286 is formed using Bitblt operations (block288). The Bitblt operations are typically performed by theimage transform engine102 selecting theappropriate tiles284 from the I-frames280,282 in order to form theoutput286.
• As discussed above, the basic level video effects processor222 (FIG. 24), the managed decoder250 (FIG. 29) and the advanced level video effects processor268 (FIG. 31) are preferably operative to playout multiple clips simultaneously. Playing out multiple clips simultaneously is typically used to generate mosaics and moving video selection panes (arrow50) shown in thehigh definition EPG54 ofFIG. 4. The individual clips are typically looped or concatenated with other clips. At the same time, the clips are optionally animated on/off screen, rotated, or a completely new set of mosaics are revealed through folding away the old set of mosaics. Individual entries in the mosaic table could be highlighted, and spun round to reveal a new movie clip.
• FIG. 32 shows the way that the input video sequences are typically constructed of thetiles284 in a completely different way than the video sequences appear on theoutput286 thereby allowing for improved compression of the video clips.
• Reference is now made toFIG. 33, which is a block diagram view illustrating a preferred compression of a plurality ofscreenshots292 for use with the set-top box system12 ofFIG. 13. Long GOP compression is used, typically at the broadcaster Headend, to compress packages of movie thumbnails, stills, animations, moving backgrounds and overlays. The compressed packages are then received and stored by the set-top box system12. The compressed packages can be decoded in the set-top box system12 by the managed decoder250 (FIG. 29) or the advanced level video effects processor268 (FIG. 31). Then, the content is preferably distributed either in memory to the next stage of the set-top box system12 or back to mass storage, ready for any future required effects.
• In many systems the basis of EPG Screens are compressed as an MPEG-2 I-frame which is uncompressed when the screen is needed. With the increase in number of screens, the space available in flash memory to store the compressed screenshots is at a premium. Additionally, HD displays have increased the quality expectations of the viewers. Therefore, acompression system298 for compressing image screenshots constructed and operative in accordance with a preferred embodiment of the present invention is described with reference toFIGS. 34-40.
• Reference is now made toFIG. 34, which is a view of a plurality of image screenshots294 (only two shown for simplicity) including a reference screenshot296 (only one shown for simplicity) for use in thecompression system298.
• Instead of compressing the sequence ofscreenshots294 as a sequence of I-frames, the sequence ofscreenshots294 are preferably compressed as a sequence of P, or B frames, all of which are predicted from one, two or more, common images (for example, the reference screenshot296) not otherwise used for display. In other words, thereference screenshot296 is not displayed on a display screen (e.g.: Television) after thereference screenshot296 is decoded. In order to perform the compression, the sequence of screenshot frames294 are typically analyzed to identify parts which are common among thescreenshots294, typically based on macroblock boundaries. The parts of thescreenshots294 that are common to a sufficient number of thescreenshots294 are then preferably composed into one or more new screenshots. The new screen shots orreference screenshots296 preferably become the basis of prediction, for all other “real” screenshot frames294. Thereference screenshots296 are then generally compressed as I-frames. The actual,original screenshots294 are then typically compressed (encoded) as P or B frames, using the newreference image screenshots296 in the normal predictive manner used by MPEG. Given sufficient commonality between thescreenshots294 generally leads to an improvement in the compression quality and/or efficiency.
• When decoding the compressed screenshots, the decoder (the managed decoder250 (FIG. 29) or the advanced level video effects processor268 (FIG. 31), by way of example only) typically needs to first decode therelevant reference screenshots296 and then the actual P or B frames representing the desired screenshot. Optionally, thenew reference screenshots296 are cached in dynamic random access memory (DRAM) to save repeated decoding of each screenshot used. However, if the decoder is fast enough, storing thereference screenshot296 in DRAM may not be necessary. Given correct formatting of the compressed (encoded) screenshots, it is generally trivial to either concatenate the bitstreams that represent the images of the screenshots or to send the images to the decoder one after the other. It should be noted that the newreference image screenshots296 ofFIG. 33 are never displayed. Thereference screenshots296 are typically created from the images to be compressed (encoded).
• InFIG. 34, thereference screenshot296 is created from parts of theimage screen shots294. Thereference screenshot296 will not be displayed on screen. Theimage screen shots294 are typically encoded as P frames derived from thedummy reference screenshot296. It should be noted that thedummy reference screenshot296 is optionally: (a) sent as part of the compression scheme used to compress the video sequence; or (b) sent out of band using a completely different compression scheme like PNG for example; or (c) part of a standard image or dictionary of images that resides in the decoder in the set-top box system12. In case (b), other compression schemes may give the benefit of a very high quality reference image (for example, but not limited to, a losslessly compressed reference frame) from which the other images are derived. In case (c), compression benefits may be yielded when something is already known about the images to be broadcast, for example, if the images to be broadcast are always cartoons or are all in the style of a resident EPG, then a set of reference frames in the decoder may give great benefits in terms of compression efficiency.
• Reference is now made toFIG. 35, which is a view a plurality ofimage screenshots300 being ordered for use in thecompression system298 ofFIG. 34. Optionally, thecompression system298 is operative to order (re-order or rearrange) the sequence ofscreenshots300 to ensure efficient compression. In the illustration ofFIG. 35, it is generally more efficient to sendpicture 1 beforepicture 3, aspicture 3 can be efficiently derived frompicture 1, but very inefficiently derived frompicture 2.
• Reference is now made toFIG. 36, which is a view of acommon image302 of a plurality ofimage screenshots304 for use in thecompression system298 ofFIG. 34. Optionally, thecompression system298 is operative to identify which of theimage screenshots304 has the greatest commonality with all of theother screenshots304. If there is sufficient commonality, thecompression system298 preferably uses thecommon image302 in place of an artificially created reference image. InFIG. 36, it generally makes more sense to sendframe3 first, as the common image302 (reference frame), and then frames 1 and 2 as P-Frames derived fromframe 3.
• If the timing requirements allow more than two images to be decoded, it may be desirable to allow one EPG image to depend on a sequence of previous images. For example, a screenshot D could depend on screenshot C which is predicted and reconstructed from a screenshot B and a screenshot A. Hence decoding the screenshot D requires the screenshots A, B and C to be decoded in order to generate the required reference images for the screenshot D. Altering the “order” of the screenshots (relative to the ID of the screenshots) may improve compression efficiency.
• The reference image screenshot(s) that are generated do not need to be encoded as I-frames; where applicable, a more appropriate mechanism for encoding may be used instead, for example, but not limited to, JPEG2000. The screenshots are still decoded as normal, but before decoding, the MPEG decoder sets up pointers to the reference image screenshots which have been decoded previously using a mechanism other than an MPEG decoding engine (such as an MPEG-2 decoding engine).
• Reference is now made toFIG. 37, which is a view a plurality of smaller than full-screen size screenshots306 being combined into a plurality of full-screen size screenshots308 for use in thecompression system298 ofFIG. 34. Where thescreenshots306 are smaller than full screen, for example, but not limited to, on-screen displays (OSDs), thescreenshots306 are optionally combined into the full-screen size screenshots308. When the full-screen size screenshots308 are decoded, only the required section of the image is used. Combining thescreenshots306 into the full-screen size screenshots308 reduces the total number of image frames requiring storage. Additional information preferably conveys the actual location and size of the sub-image to be extracted from the full-screen size screenshots308, such as Video Service Information (VSI) from Xtreamplay of NDS Limited.
• Thecompression system298 preferably also determines the areas of full-screen size screenshots308 that need to be decoded for each smaller than full-screen size screenshots306 to be decoded. The decoding area information of the sub-pictures is typically conveyed to the decoder used in decoding thescreenshots306 to improve the efficient decoding of thescreenshots306. Alternatively, thecompression system298 may preferably work from a fixed decoding area and ensure that all the items needed to decode a givensub-picture screenshot306 occur within the fixed decoding area.
• InFIG. 37, thescreenshots306 which are similar have been ordered and placed in the full-screen size screenshots308 to take advantage of the space available and the way that most long GOP compression schemes derive one picture from the previous picture very efficiently.
• Reference is now made toFIG. 38, which is a view of a plurality ofimages310 including a plurality of equally sizedspare areas312 for use in thecompression system298 ofFIG. 34. If theimages310 do not fill the full area of the screen, then thespare area312 may be filled with still images or other smaller video sequences (not shown).
• Reference is now made toFIG. 39, which is a view of a plurality ofimages314 including a plurality of unequal sizedspare areas316 for use incompression system298 ofFIG. 34. Ifvideo images314 expand in size over time, then thespare area316, available for the insertion of other still or video images, around theimages314, changes with time.
• In accordance with a most preferred embodiment of the present invention, where multiple compression mechanisms exist (for example, but not limited to, JPEG-2000, MPEG I-frames, GIF, Vector Graphics), including local software generation of images on the set-top box system12, a static analysis typically chooses the most appropriate mechanism for any individual part of the screenshot. The parts of the screen shots could then be combined using the motion prediction techniques of MPEG-2 which are able to extract macroblocks of data from one or two reference images. If more than two motion prediction techniques are available, then short sections of MPEG-2 are typically used to combine the relevant parts of several images into the maximum of two reference images for generation of the final image.
• The managed decoder250 (FIG. 29) or the advanced level video effects processor268 (FIG. 31), by way of example only, are typically suitable for decoding the compressed sequences of thecompression system298.
• Reference is now made toFIG. 40, which is a block diagram view ofcompression system298 ofFIG. 34. Thecompression system298 preferably includes anencoder318 which scans a plurality ofinput images320, a plurality ofvideos322, a plurality ofcoordinates324 of theinput images320 and of thevideos322 and a plurality ofparameters326 for thecompression system298 such as, maximum resolution and decoder characteristics, by way of example only. Theencoder318 is either a real time encoder or a non-real time encoder.
• If theencoder318 is non real time, theencoder318 is typically a computer operative to iterate many different layouts and orders to find the optimum compression layout. Once the optimum layout has been found, a package ofcompressed frames325 and associated metadata are typically written out for sending to a decoder.
• If theencoder318 is a real time encoder, theencoder318 is typically operative to work in substantially same way as the non-real time encoder, but combinations of layout are generally more limited due to the reduced compression time available.
• Theencoder318 preferably includes ananalyzing module319 and acompression module321 for performing the functions of thecompression system298 described with reference toFIGS. 33-39. The analyzingmodule319 andcompression module321 are now described below in order to describe how the functionality of thecompression system298 described with reference toFIGS. 33-39 is allocated between the analyzingmodule319 and thecompression module321.
• The analyzingmodule319 is preferably operative to: analyze thescreenshots320,322 to identify commonality among thescreenshots320,322 typically based on macroblock boundaries of thescreenshots320;322; create a reference frame based on the commonality of thescreenshots320,322 or determine a suitable pre-existing frame as the reference frame based on the commonality of thescreenshots320,322 (by way of example only, the pre-existing frame is typically a part of a dictionary of images that resides in a set-top box or the pre-existing frame is one of thescreenshots320,322); and reorder a sequence of thescreenshots320,322 so as to improve efficiency of the compression of thescreenshots320,322.
• Thecompression module321 is typically operative to compress thescreenshots320,322, based on the reference frame, into a stream ofcompressed frames325 including one or more MPEG B-frames and/or one or more MPEG P-frames.
• If the reference frame(s) is not already resident on the receiving devices (the set-top boxes), the reference frame is typically compressed by the compression module as an MPEG I-frame into the stream of the compressed frames325. Alternatively, the reference frame is compressed in a different format (such as PNG by way of example only) and sent in a different stream to the stream ofcompressed frames325.
• In a similar way to the tile processing described above, sets of sprites are typically stored along with a precise PNG alpha mask and a combination of the sprites and the PNG alpha mask are used to define sprite sets which can be loaded into memory.
• The decoders of thevideo effects processor78 described with reference toFIGS. 24-30 typically pull multiple assets off a disk (not shown) of the set-top box system12 continuously. The decoders preferably cue up content ahead of time, so therefore, using cueing, and considering disk issues in the design of the video effects, the load on the disk can be managed.
• Local video is optionally inserted into the video effects processor78 (FIG. 13) from special base band inputs and connected input devices, by way of example only. If the local video input is from a local DV based camera device, then a DV based chain including a DV hardware decoder, one of the image transform engines102 (FIG. 14) and one of the alpha generators104 (FIG. 16) is preferably included in thevideo effects processor78 parallel with the other MPEG I-frame and PNG decoders. The main CPU212 (FIG. 13) is typically operative to extract frames from the input device and pass the frames to thevideo effects processor78 through the DV input route, by way of example only.
• Similarly, if the local video input is from a base band video input, then frames of video preferably enter the video effects processor78 (FIG. 13) through a chain including one of the image transform engines102 (FIG. 14) and one of the alpha generators104 (FIG. 16).
• Reference is now made toFIG. 41, which is a block diagram view of a picture-in-picture system412 constructed and operative in accordance with a preferred embodiment of the present invention. By way of introduction, newer compression standards such as 14496-10 (AVC) allow the decoder to utilize long term reference frames (LTRFs) which are typically used as a source for the final decoded image. The LTRFs are not subject to the usual update and replace cycles whereby reference frames are updated and/or replaced by new reference frames arriving in the video sequence being decoded.
• The set-top box system12 is preferably implemented withtwin decoders418,420 such that a picture-in-picture style display is typically performed utilizing an LTRF414 without any specific additional hardware. Consider two video streams, avideo stream416 and avideo stream422. Thevideo stream416 is specially encoded such that part of thevideo stream416 includes one ormore references432 to the LTRF414, but the LTRF414 is never transmitted in the main video transport stream. Thedecoder418 for thestream416 is set up such that the LTRF414 points to anoutput frame424 of thedecoder420. Thedecoder420 is started in order to decode thevideo stream422, but without driving an output. When thestream416 is decoded and displayed, the result is anoutput426 including avideo window428 that displays avideo430 of thestream422. By control over which stream is played, including which is displayed off disk, a personalized insert is available. By control over the encoder, the location of the display is also controllable.
• Reference is now made toFIG. 42, which is a block diagram view of a most preferred basic levelvideo effects processor328 for use with the set-top box system12 ofFIG. 13. The basic levelvideo effects processor328 includes anMPEG chain330, aPNG chain332 and one ormore chains334 for other compressed video formats.
• Reference is now made toFIG. 43, which is a block diagram view of a most preferred advanced levelvideo effects processor336 for use with the set-top box system12 ofFIG. 13. The advanced levelvideo effects processor336 includes: aLong GOP processor338 for a plurality ofvideo sequences342 and a plurality of I-frames344; and aPNG decoder340 for decoding a plurality of PNGs346. The advanced levelvideo effects processor336 includes one of thealpha generators104. The advanced levelvideo effects processor336 also includes a plurality ofother decoders348 for decoding a plurality of other compressed video formats350. The advanced levelvideo effects processor336 also preferably includes anasset loading interface349 to capture frames before the frames are sent for display, the frames typically being routed back to disk storage (not shown).
• The advanced levelvideo effects processor336 is also preferably operative to seemlessly loop-decode and concatenate. A storage arrangement, such as thedisk arrangement205 ofFIG. 22, is preferably operative to store acompressed video clip438 formed from a plurality ofsub-regions440 of one or more video frames442. Thevideo clip438 has astart444 and anend446. TheLong GOP processor338 is preferably operative to: decode only thesub-regions440 of the video frames442 with a seamless cut between thestart444 and theend446 of thevideo clip438; and/or seamlessly concatenate thevideo clip438 onto the end of a clip (not shown) that is currently playing.
• The decoding limit of thedecoder338 is generally not restricted to a fixed number ofstreams342 at a fixed decoding rate. The advanced levelvideo effects processor336 is preferably operative such that the number of movie video thumbnails or image decodes (video sequences342) decoded at any one time by thedecoder338 depends on the total size of the images for decoding and the maximum processing speed of thedecoder338. By way of example only, it may be possible to decode16 streams, each with a resolution of 352 by 288 pixels, or two streams with a resolution of 1408 by 576 pixels. Additionally, the flexibility of thedecoder338 may be used to decode a stream faster than would normally be allowed with a fixed limit decoder. By way of example only, an image having a resolution of 704 by 576 pixels may be decoded in one-quarter of the normal time taken by a fixed limit decoder.
• Additionally, the advanced levelvideo effects processor336 is preferably operative to: receive a compressed frame (for example, but not limited to, the compressed frame442) which is too large for decoding by theLong GOP processor338; and decode only a sub-region (for example one of the sub-regions440) of the compressed frame. The sub-region is up to the maximum image size supported by the speed of theLong GOP processor338. The compressed frame is larger than the maximum image size supported by theLong GOP processor338. By way of example only, thedecoder338 can decode an image of resolution 2816 by 576 pixels contained within an image having a resolution of 4096 by 4096 pixels.
• Reference is now made toFIG. 44, which is a block diagram view of thegraphical effects processor80 for use with the set-top box system12 ofFIG. 13. Thegraphical effects processor80 is preferably used to create any suitable graphical elements or effects which are created dynamically in the set top box12 (rather than being pre-rendered by the broadcaster/content provider). The graphical elements or effects generally include, by way of example only, vector and image based graphics, text, 3D geometry based graphics and any suitable effects dynamically applied to vector and image based graphics, text and 3D geometry based graphics. Thegraphical effects processor80 is linked to and complimentary to the video effects processor78 (FIG. 13). Thegraphical effects processor80 is operative to accept any suitable bitmap based assets from the video effects processor78 (FIG. 13).
• Thegraphical effects processor80 is preferably operative to support the creation of suitable graphical features in the set-top box system12, in particular, where doing so reduces the overall cost of bandwidth and storage such that the savings result in a net benefit to the manufacturer of the set-top box system12. Thegraphical effects processor80 is preferably operative to support atimeline synchronization356 required to allow for time sequenced presentation effects for example, but not limited to, knowing when to ask thevideo effects processor78 to extract and decode requested assets.
• FIG. 44 shows ahardware stack352 and asoftware stack354 of thegraphical effects processor80. Thesoftware stack354 is shown in order to describe the full context for thegraphical effects processor80. It should be noted that it may also be necessary to partition some of thesoftware stack354 functionality into a separate dedicated graphics CPU (not shown). The descriptions below are therefore separated in to hardware and software in order to describe the full functionality of thegraphical effects processor80.
• Reference is now made toFIG. 45, which is a block diagram view of a2D accelerator358 of thegraphical effects processor80 ofFIG. 44. The2D accelerator358 is arranged as apipeline362 which is used to generate raster based graphics from paths and paints. The2D accelerator358 includes aninterface360 which is typically an OpenVG v1.0 Interface. The flow of information through thepipeline362 is preferably as described below.
• The2D accelerator358 typically includes apaint generator364, which generates paints. The paints are typically bitmaps loaded through theinterface360. Alternatively or additionally, the paints are generated mathematically from within thepaint generator364, in which case, the paints are typically solid color or gradient fills, by way of example only.
• The2D accelerator358 includes a rasterizer366 which subsequently uses the paints generated by thepaint generator364.
• The rasterizer366 typically generates bitmap rasters. The inputs to the rasterizer366 are paths, transforms, strokes and fills. Paths are sets of points, joined together in a particular way. Paths are typically joined by straight lines and/or curved lines, by way of example only, to form a line, curve and/or bounded area/region. The lines or regions are then either filled by a paint and/or stroked by a paint. Fills are performed based on rules. Strokes are typically painted along a path with a paint. The paints are typically applied to a brush which is used to stroke along paths, or fill an area bounded by a closed path. The brush has characteristics like beveled edges and/or rounded ends (by way of example only). Transforms typically include rotations, scaling and/or skewing. Transforms are typically the same as the transforms performed in the raster domain by theimage transform engine102, but the transforms performed by the rasterizer366 are typically performed before any rasterization, in order not to compromise quality. By way of example only, scaling in the vector domain generally incurs no quality logs, but scaling in the raster domain typically results in quality loss. Vector scaling is particularly useful for scaling text, by way of example only.
• Any output from the rasterizer366 is full 32 bit color with an 8 bit alpha channel and is fully anti-aliased.
• The rasterizer366 typically generates a plurality ofdifferent raster images368 per frame. Theraster images368 generally continue through thepipeline362 to become part of complex graphical constructions. The final part of thepipeline362 uses theimage transform engine102 to perform functions in the raster domain typically including a deflicker operation, for example, before blending sets of vector originated rasters into the final composition. A plurality ofmemory buffers370 including the final compositions are then passed downstream for compositing with the main broadcast feeds in the video compositor82 (FIG. 13).
• The2D accelerator358 preferably supports at least two contexts (two separate fully composited outputs), at the maximum resolution and frame rate supported by the outputs. It should be noted that more outputs may be required if the set-top box system12 needs to support multiple simultaneous TV outputs.
• OpenVG only specifies a nearest neighbor interpolation kernel. Therefore, theimage transform engine102 is typically operative to support a bilinear kernel, and preferably, support a bicubic kernel.
• The2D accelerator358 preferably supports YUV images (4:4:4, 4:2:0, 4:2:2, 4:1:1) by way of example only.
• Reference is again made toFIG. 44.
• The2D accelerator358 is a general purpose engine which preferably accelerates the basic functions required for a vector graphics operation. Thesoftware stack354 disposed above the2D accelerator358 preferably performs functions generally required for the set-top box system12 described below.
• Thegraphical effects processor80 optionally includes a3D hardware accelerator372 which is typically OpenGL-ES based.
• Thesoftware stack354 preferably includes avector library374 preferably operative to support drawing vector primitives not defined in OpenVG. Thesoftware stack354 preferably includes the VGU Utility Library as defined in Open VG 1.0. The VGU Utility Library typically supports creation of, and appending to paths, the following shapes, by way of example only: lines, polylines, polygons, rectangles, rounded rectangles, ellipses (including circles) and arcs. Thevector library374 preferably supports transforms (for example, but not limited to, 3×3 affine; supporting scaling, rotation and shearing).
• Thesoftware stack354 preferably includes atext library376 generally operative to provide advanced text rendering support for example: at the character level including face, color, style, weight, size, kerning, vector transforms (3×3 affine, supporting scaling, rotation and shearing); and text layout functions, such as flowing text to fit arbitrary shaped bounding boxes, line spacing and breaking and hyphenation, handling of international character sets and presentation, rendering text to arbitrary paths including curves; of vector and/or bitmap font library.
• Thesoftware stack354 preferably also includes animage processing library378 typically operative to support image manipulations including for example: image blending (alpha based porter-duff); transform manipulation (3×3 projective); other deformations such as page turns; color manipulation (brightness, contrast, hue/saturation/lightness); channel splitting, mixing and combining; convolution kernel based filtering (maximum kernel size as per OpenVG) including blurs (Gaussian), sharpening, edge detection and/or embossing; additive noise and dithering; sub-pixel shift re-sampling; pixel format conversion; programmatic texture generation (tile based, radial and linear gradient fills); and image file format decoding.
• It should be noted that the2D accelerator358 is generally assumed to use the image transform engine102 (FIG. 14) or decoders of the video effects processor78 (FIG. 13) for some or all of the functionality of theimage processing library378.
• Thesoftware stack354 also preferably includes an animation/effects library380 preferably operative to support: TV-centric effects including moving highlights and lens flare; motion path based animation of vector primitives; physics based motion path generator that models ballistics, gravity, bounces and/or collisions; auto generation of alpha planes (drop shadow and luma/chroma key); and text specific effects such as reveals, fades, flips, smooth scrolling banners.
• Thesoftware stack354 also typically includes avector file parser382 preferably operative to parse the following file formats: Macromedia Flash (SWF) file format version; and SVG Basic 1.2 (documents and document fragments) including gzip and BinXML compressed variants.
• Thesoftware stack354 preferably includes a software graphics stack384 and a control andtiming module436 to manage the control and timing of a graphics effect. Thesoftware stack354 optionally includes a3D library386 to support the3D hardware accelerator372.
• In the absence of the3D accelerator372, the3D library386 preferably includes a software vertex transformation engine (not shown) that is typically used to generate screen co-ordinates for vertices in a simple 3D model. The screen co-ordinates are then typically used to derive projected and affine transformations used by the image transform engine(s)102 (FIG. 14) for video and graphics scaling used in ‘pseudo 3D’ effects and transitions.
• Reference is now made toFIG. 46, which is a block diagram view of anaudio transfer engine388 for use in theaudio effects processor84 of the set-top box system ofFIG. 13.
• As TV systems migrate towards HD, it is expected that the quality of audio will have to increase at the same time, necessitating higher quality audio outputs, more audio outputs, analogue and digital formats simultaneously.
• The audio effects processor84 (FIG. 13) is used to generate all audio outputs and effects in the set-top box system12, and co-ordinate the audio outputs and effects together with the rest of the set-top box system12. It should be noted that it is assumed that any multi-language audio issues have been resolved in the main CPU212 (FIG. 13) prior to the audio streams entering theaudio effects processor84.
• The sources of audio in the set-top box system12 are generally: live off-air broadcasts with formats ranging from mono to multi-channel; locally stored disk based audio, originating from live-off air broadcasts with formats ranging from mono to multi-channel; locally stored disk based assets such as MP3s and home made videos with formats ranging from mono to multi-channel; local audio feeds for example, but not limited to, local microphones or attached devices with formats ranging from mono to multi-channel; local audio samples; and locally created audio such as synthesis. The term “local” as used in the specification and claims is defined as being in the location of the set-top box system12, as opposed to being in the location of a Headend or a broadcaster or a content provider.
• Theaudio effects processor84 is preferably operative to include the following processing options: process nothing, for example, but not limited to, making sure that for normal viewing of TV programs, as little processing as possible is made to the audio channel; transitions so that in the same way that video transitions are described hereinabove, the corresponding audio transitions are preferably supported, including control of volume and the speed/timing/shape of the associated ramp is required, for example, but not limited to, when changing channels or when new content is inserted or when transitioning to alternative screens; mixing, including control over which of the many input audio feeds makes it through the set-top box system12, the volume of the input and which channel the input appears on; audio sound effects, for example, but not limited to, reverb may be added to any of the audio streams; compression so that a rich audio experience is provided while minimizing transmission bandwidths or storage requirements, for example, extensive use of compression is preferably made for stored or transmitted audio clips; synthesis for the generation of musical sound tracks or effects for games and EPGs, by way of example only. The synthesis is preferably performed in order to minimize the transmission and storage requirements for theaudio effects processor84. The synthesis is preferably Wavetable based, polyphonic and provides the quality of audio experience that is at least as good as mobile phone polyphonic synthesizers.
• Theaudio transfer engine388 is preferably a building block of theaudio effects processor84. Theaudio transfer engine388 preferably performs effects on audio samples as the audio travels through theaudio transfer engine388. Theaudio transfer engine388 preferably performs the following functions: changing the attenuation of audio channels under software control including fixed attenuation or ramp-like gain profiles where the shape of the attenuation profile is definable; and effects including reverb, echo, time stretching, panning, pitch shifting and dynamic range compression.
• Theaudio transfer engine388 is preferably operative to apply multiple operations to audio streams as the audio streams are passing through theaudio effects processor84. Eachaudio transfer engine388 only operates on one mono audio stream, although there are many audio streams in the set-top box system12.
• Reference is now made toFIG. 47, which is a block diagram view of theaudio effects processor84 of the set-top box system12 ofFIG. 13. Theaudio effects processor84 typically includes a managedaudio decoder390 preferably operative to playout a plurality of stereo audio streams392, for example, but not limited to,16 stereo audio streams392, each of the stereo audio streams392 being controlled with sample accuracy. Transitions between the stereo audio streams392 and other streams in theaudio effects processor84 are made through a combination of theaudio transfer engine388 and theaudio mixer394. To do a cross fade audio transition, for example, theaudio transfer engines388 typically change the attenuation of audio channels with a software defined ramp profile, such that once mixed at theaudio mixer394, the effect is to fade one channel out and the other channel in.
• A plurality ofaudio streams396 entering theaudio effects processor84 are typically mono, stereo or multi-channel in pulse code modulated (PCM) format. The audio streams396 may have different sample precisions and rates. In order to rationalize the different formats, theaudio transfer engines388 and theaudio mixer394 preferably operate on mono-only streams and at a multiple of the highest sample rate.
• Theaudio effects processor84 preferably includes an audio crossbar switch andsplitter398. All theaudio streams396 are split into channels on entry to the audio crossbar switch andsplitter398. The channels of theaudio streams396 are routed out of the audio crossbar switch andsplitter398 on a plurality of software defined outputs402.
• Theaudio transfer engines388 are preferably operative to perform operations on thevarious outputs402 of the audio crossbar switch andsplitter398, for example, but not limited to, attenuation or reverb effects. Eachmono stream output402 processed by theaudio transfer engines388 is then typically input as aninput feed404 into theaudio mixer394 for mixing into a final output presentation.
• As the input feeds404 are mono, theaudio mixer394 preferably has complete control over where any audio is positioned in a multi-channel audio presentation.
• When anaudio stream406 leaves theaudio mixer394, theaudio stream406 is optionally processed by one of theaudio transfer engines388, for example, but not limited to, applying an overall attenuation or an effect such as dynamic range compression.
• As described above, the managedaudio decoder390 is preferably operative to handle multiple stored compressed stereo audio streams392. The managedaudio decoder390 is typically an audio equivalent of a managed video decoder such as the manageddecoder250 ofFIG. 29. The managedaudio decoder390 preferably has the following features: cueing-up separate audio streams, for example, but not limited to, up to 16 separate audio streams, to a known audio sample start point; playing an audio stream which was previously been cued up; looping an audio sample with the loop point at a sample accurate location in the stream; concatenating and inserting audio streams with sample accuracy; allowing control software (not shown) to define the stream, sample start/end; providing interrupts to software to indicate start/end; coordinating withaudio transfer engines388 to allow for downstream audio effects and transitions; synchronizing playout with the rest of the system (block408); and any suitable audio compression format supported by theaudio effects processor84 is typically supported, so by way of example only, if theaudio effects processor84 supports a multi-channel compressed format, then so does the managedaudio decoder390.
• Theaudio effects processor84 also preferably includes anaudio midi synthesizer410. Theaudio midi synthesizer410 is preferably used to generate music for games, iTV applications and EPGs where bandwidth and storage requirements cannot be satisfied using other methods. Theaudio midi synthesizer410 is preferably a low-cost Wavetable synthesizer similar to Wavetable synthesizers in polyphonic mobile phones, by way of example only.
• User input is assumed to come from any number of suitable sources, and is generally platform dependent. User input for the set-top box system12 preferably includes the following features: any suitable remote control functionality is preferably two-way; multiple controllers are preferably supported; and pressing keys simultaneously is preferably supported, for example, but not limited to, up to four keys simultaneously.
• Any suitable user inputs having an audio capability are preferably either: hard-wired to one of the inputs of theaudio effects processor84 and therefore the inputs are controllable through theaudio effects processor84; or the audio is accessible to the main CPU212 (FIG. 13) and any suitable audio streams may be fed into one of theaudio transfer engines388.
• Any suitable user inputs having video capability are preferably either hard wired into the video effects processor78 (FIG. 13), or frames of the video feeds are read and inserted into the video effects processor78 (FIG. 13) with minimal overhead of the main CPU212 (FIG. 13).
• It should be noted that if a plug-in architecture is required by the platform operator to handle features like additional downloadable video or audio processing, for example, a plug-in architecture is typically provided by supplying additional spare CPU power in the hardware design of the set-top box system12.
• The set-top box system12 optionally includes a middleware based plug-in architecture (not shown) by giving full access to all video and audio paths through the set-top box system12 from any suitable general purpose CPU. The plug-in feature is typically arranged as described above in order to minimize hardware costs.
• It is appreciated that software components of the present invention may, if desired, be implemented in ROM (read only memory) form. The software components may, generally, be implemented in hardware, if desired, using conventional techniques.
• It will be appreciated that various features of the invention which are, for clarity, described in the contexts of separate embodiments may also be provided in combination in a single embodiment. Conversely, various features of the invention which are, for brevity, described in the context of a single embodiment may also be provided separately or in any suitable sub-combination. It will also be appreciated by persons skilled in the art that the present invention is not limited by what has been particularly shown and described hereinabove. Rather the scope of the invention is defined only by the claims which follow.

Claims (17)

1-37. (canceled)

38. A screenshot compression system for compressing uncompressed screenshots, the system comprising an encoder including:

an analyzing module to: analyze the screenshots to identify commonality among the screenshots; and create a reference frame based on the commonality of the screenshots; and

a compression module to compress the screenshots based on the reference frame.

39. The system according toclaim 38, wherein the screenshots include a plurality of macroblock boundaries, the analyzing module being operative to analyze the screenshots based on the macroblock boundaries.

40. The system according toclaim 38, wherein the compression module is operative to compress the screenshots into a stream including a plurality of compressed frames, the compressed frames including at least one of: an MPEG B-frame and an MPEG P-frame.

41. The system according toclaim 38, wherein the compression module is operative to compress the reference frame as an MPEG I-frame in the stream of the compressed frames.

42. The system according toclaim 40, wherein the reference frame is compressed and sent in a different stream to the stream of compressed frames.

43. The system according toclaim 38, wherein the analyzing module is operative to reorder a sequence of the screenshots so as to improve efficiency of the compression of the screenshots.

44. A screenshot compression system for compressing uncompressed screenshots, the system comprising an encoder including:

an analyzing module to: analyze the screenshots to identify commonality among the screenshots; and determine a suitable reference frame based on the commonality of the screenshots, the suitable reference frame being selected from a pre-existing frame; and

a compression module to compress the screenshots based on the reference frame.

45. The system according toclaim 44, wherein the compression module is operative to compress the screenshots into a stream including a plurality of compressed frames, the compressed frames including at least one of: an MPEG B-frame and an MPEG P-frame.

46. The system according toclaim 44, wherein the reference frame is part of a dictionary of images that resides in a set-top box.

47. The system according toclaim 44, wherein the compression module is operative to compress the screenshots into a stream including a plurality of compressed frames, the compressed frames including an MPEG I-frame and at least one of: an MPEG B-frame and an MPEG P-frame, the MPEG I-frame corresponding to the reference frame.

48. The system according toclaim 44, wherein the screenshots include a plurality of macroblock boundaries, the analyzing module being operative to analyze the screenshots based on the macroblock boundaries.

49. The system according toclaim 44, wherein the analyzing module is operative to reorder a sequence of the screenshots so as to improve efficiency of the compression of the screenshots.

50-58. (canceled)

59. A screenshot compression method for compressing uncompressed screenshots, the method comprising:

analyzing the screenshots to identify commonality among the screenshots;

creating a reference frame based on the commonality of the screenshots; and

compressing the screenshots based on the reference frame.

60. A screenshot compression method for compressing uncompressed screenshots, the method comprising:

analyzing the screenshots to identify commonality among the screenshots;

determining a suitable reference frame based on the commonality of the screenshots, the suitable reference frame being selected from a pre-existing frame; and

compressing the screenshots based on the reference frame.

61. (canceled)

Images