USRE48845E1 - Video decoding system supporting multiple standards - Google Patents

Abstract

System and method for decoding digital video data. The decoding system employs hardware accelerators that assist a core processor in performing selected decoding tasks. The hardware accelerators are configurable to support a plurality of existing and future encoding/decoding formats. The accelerators are configurable to support substantially any existing or future encoding/decoding formats that fall into the general class of DCT-based, entropy decoded, block-motion-compensated compression algorithms. The hardware accelerators illustratively comprise a programmable entropy decoder, an inverse quantization module, a inverse discrete cosine transform module, a pixel filter, a motion compensation module and a de-blocking filter. The hardware accelerators function in a decoding pipeline wherein at any given stage in the pipeline, while a given function is being performed on a given macroblock, the next macroblock in the data stream is being worked on by the previous function in the pipeline.

Description

CROSS-REFERENCECROSS REFERENCE TO RELATED APPLICATIONS

This application is a reissue of U.S. patent application Ser. No. 13/608,221 filed Sep. 10, 2012 (now U.S. Pat. No. 9,417,883) entitled, “Video Decoding System Supporting Multiple Standards,” which is a divisional application of and claims priority to U.S. patent application Ser. No. 10/114,798, filed on Apr. 1, 2002, having the title “VIDEO DECODING SYSTEM SUPPORTING MULTIPLE STANDARDS,” and issued as U.S. Pat. No. 8,284,844 on Oct. 9, 2012, which is incorporated by reference herein as if expressly set forth in its entirety.

FIELD OF THE INVENTION

The present invention relates generally to video decoding systems, and more particularly to a video decoding system supporting multiple standards.

BACKGROUND

Digital video decoders decode compressed digital data that represent video images in order to reconstruct the video images. A relatively wide variety of encoding /decoding algorithms and encoding/decoding standards presently exist, and many additional algorithms and standards are sure to be developed in the future. The various algorithms and standards produce compressed video bit streams of a variety of formats. Some existing public format standards include MPEG -1, MPEG-2 (SD/HD), MPEG-4, H.263, H.263+ and H.26LIJVT. Also, private standards have been developed by Microsoft Corporation (Windows Media), RealNetworks, Inc., Apple Computer, Inc. (QuickTime), and others. It would be desirable to have a multi-format decoding system that can accommodate a variety of encoded bit stream formats, including existing and future standards, and to do so in a cost-effective manner.

A highly optimized hardware architecture can be created to address a specific video decoding standard, but this kind of solution is typically limited to a single format. On the other hand, a fully software based solution is often flexible enough to handle any encoding format, but such solutions tend not to have adequate performance for real time operation with complex algorithms, and also the cost tends to be too high for high volume consumer products. Currently a common software based solution is to use a general-purpose processor running in a personal computer, or to use a similar processor in a slightly different system. Sometimes the general-purpose processor includes special instructions to accelerate digital signal processor (DSP) operations such as multiply-accumulate (MAC); these extensions are intimately tied to the particular internal processor architecture. For example, in one existing implementation, an Intel Pentium processor includes an MMX instruction set extension. Such a solution is limited in performance, despite very high clock rates, and does not lend itself to creating mass market, commercially attractive systems.

Others in the industry have addressed the problem of accommodating different encoding/decocting algorithms by designing special purpose DSPs in a variety of architectures. Some companies have implemented Very Long Instruction Word (VLIW) architectures more suitable to video processing and able to process several instructions in parallel. In. these cases, the processors are difficult to program when compared to a general-purpose processor. Despite the fact that the DSP and VLIW architectures are intended for high performance, they still tend not to have enough performance for the present purpose of real time decoding of complex video algorithms. In special cases, where the processors are dedicated for decoding compressed video, special processing accelerators are tightly coupled to the instruction pipeline and are part of the core of the main processor.

Yet others in the industry have addressed the problem of accommodating different encoding/decoding algorithms by simply providing, multiple instances of hardware, each dedicated to a single algorithm. This solution is inefficient and is not cost-effective.

Further limitations and disadvantages of conventional and traditional approaches will become apparent to one of skill in the art through comparison of such systems with the present invention as set forth in the remainder of the present application with reference to the drawings.

SUMMARY

One aspect of the present invention is directed to a digital media decoding system having a processor and a hardware accelerator. The processor is adapted to control a decoding process. The hardware accelerator is coupled to the processor and performs a decoding function on a digital media data stream. The accelerator is configurable to perform the decoding function according to a plurality of decoding methods.

Another aspect of the present invention is directed to a method of decoding a digital media data stream. Pursuant to the method, in a first stage, a first decoding function is performed on an i^thdata element of the data stream with a first decoding accelerator. In a second stage, after the first stage, a second decoding function is performed on the i^thdata element with a second decoding accelerator, while the first decoding function is performed on an i+1^stdata element in the data stream with the first decoding accelerator.

Another aspect of the present invention is directed to a method of decoding a digital video data stream. Pursuant to the method, in a first stage, entropy decoding is performed on an i^thdata element of the data stream. In a second stage, after the first stage, inverse quantization is performed on a product of the entropy decoding of the i^thdata element, while entropy decoding is performed on an i+1^stdata element in the data stream.

Still another aspect of the present invention is directed to a method of decoding a digital media data stream. Pursuant to this method, media data of a first encoding/decoding format is received. At least one external decoding function is configured based on the first encoding/decoding format. Media data of the first encoding/decoding format is decoded using the at least one external decoding function. Media data of a second encoding/decoding format is received. The at least one external decoding function is configured based on the second encoding/decoding format. Then media data of the second encoding/decoding format is decoded using the at least one external decoding function.

It is understood that other embodiments of the present invention will become readily apparent to those skilled in the art from the following detailed description, wherein embodiments of the invention are shown and described only by way of illustration of the best modes contemplated for carrying out the invention. As will be realized, the invention is capable of other and different embodiments and its several details are capable of modification in various other respects, all without departing from the spirit and scope of the present invention. Accordingly, the drawings and detailed description are to be regarded as illustrative in nature and not as restrictive.

BRIEF DESCRIPTION OF THE DRAWINGS

These and other features, aspects, and advantages of the present invention will become better understood with regard to the following description, appended claims, and accompanying drawings where:

FIG. 1 is a functional block diagram of a digital media system in which the present invention may be illustratively employed.

FIG. 2 is a functional block diagram demonstrating a video decode data flow according to an illustrative embodiment of the present invention.

FIG. 3 is a high-level functional block diagram of a digital video decoding system according to an illustrative embodiment of the present invention.

FIG. 4a is a functional block diagram of a digital video decoding system according to an illustrative embodiment of the present invention.

FIG. 4b is a functional block diagram of a motion compensation filter engine according to an illustrative embodiment of the present invention.

FIG. 5 is a block diagram depicting a clocking scheme for a decoding system according to an illustrative embodiment of the present invention.

FIG. 6 is a chart representing a decoding pipeline according to an illustrative embodiment of the present invention.

FIG. 7 is a flowchart representing a macroblock decoding loop according to an illustrative embodiment of the present invention.

FIG. 8 is a flowchart representing a method of decoding a digital video data stream containing more than one video data format, according to an illustrative embodiment of the present invention.

DETAILED DESCRIPTION OF THE EMBODIMENTS

The present invention forms an integral part of a complete digital media system and provides flexible and programmable decoding resources.FIG. 1 is a functional block diagram of a digital media system in which the present invention may be illustratively employed. It will be noted, however, that the present invention can be employed in systems of widely varying architectures and widely varying designs.

The digital media system ofFIG. 1 includestransport processor102,audio decoder104, direct memory access (DMA)controller106, system memory controller108,system memory110,host CPU interface112,host CPU114,digital video decoder116,display feeder118,display engine120,graphics engine122,display encoders124 andanalog video decoder126. Thetransport processor102 receives and processes a digital media data stream. Thetransport processor102 provides the audio portion of the data stream to theaudio decoder104 and provides the video portion of the data stream to thedigital video decoder116. In one embodiment, the audio and video data is stored inmain memory110 prior to being provided to theaudio decoder104 and thedigital video decoder116. Theaudio decoder104 receives the audio data stream and produces a decoded audio signal.DMA controller106 controls data transfer amongstmain memory110 and memory units contained in elements such as theaudio decoder104 and thedigital video decoder116. The system memory controller108 controls data transfer to and fromsystem memory110. In an illustrative embodiment,system memory110 is a dynamic random access memory (DRAM) unit. Thedigital video decoder116 receives the video data stream, decodes the video data and provides the decoded data to thedisplay engine120 via thedisplay feeder118. Theanalog video decoder126 digitizes and decodes an analog video signal (NTSC or PAL) and provides the decoded data to thedisplay engine120. Thegraphics engine122 processes graphics data in the data stream and provides the processed graphics data to thedisplay engine120. Thedisplay engine120 prepares decoded video and graphics data for display and provides the data to displayencoders124, which provide an encoded video signal to a display device.

FIG. 2 is a functional block diagram demonstrating a video decode data flow according to an illustrative embodiment of the present invention. Transport streams are parsed by thetransport processor102 and written tomain memory110 along with access index tables. Thevideo decoder116 retrieves the compressed video data for decoding, and the resulting decoded frames are written back tomain memory110. Decoded frames are accessed by thedisplay feeder interface118 of the video decoder for proper display by a display unit. InFIG. 2, two video streams are shown flowing to thedisplay engine120, suggesting that, in an illustrative embodiment, the architecture allows multiple display streams by means of multiple display feeders.

Aspects of the present invention relate to the architecture ofdigital video decoder116. In accordance with the present invention, a moderately capable general purpose CPU with widely available development tools is used to decode a variety of coded streams using hardware accelerators designed as integral parts of the decoding process.

Specifically, the most widely-used compressed video formats fall into a general class of DCT-based, variable-length coded, block-motion-compensated compression algorithms. As mentioned above, these types of algorithms encompass a wide class of international, public and private standards, including MPEG-1, MPEG-2 (SD/HD), MPEG-4, H.263, H.263-F, H.26LINT, Microsoft Corp, Real Networks, QuickTime, and others. Fundamental functions exist that are common to most or all of these formats. Such functions include, for example, programmable variable-length decoding (VLD), arithmetic decoding (AC), inverse quantization (IQ), inverse discrete cosine transform (IDCT), pixel filtering (PF), motion compensation (MC), and deblocking/de-ringing (loop filtering or post-processing). The term “entropy decoding” may be used generically to refer to variable length decoding, arithmetic decoding, or variations on either of these. According to the present invention, these functions are accelerated by hardware accelerators.

However, each of the algorithms mentioned above implement some or all of these functions in different ways that prevent fixed hardware implementations from addressing all requirements without duplication of resources. In accordance with one aspect of the present invention, these hardware modules are provided with sufficient flexibility or programmability enabling a decoding system that decodes a variety of standards efficiently and flexibly.

The decoding system of the present invention employs high-level granularity acceleration with internal programmability or configurability to achieve the requirements above by implementation of very fundamental processing structures that can be configured dynamically by the core decoder processor. This contrasts with a system employing fine-granularity acceleration, such as multiply-accumulate (MAC), adders, multipliers, FFT functions, DCT functions, etc. In a fine-granularity acceleration system, the decompression algorithm has to be implemented with firmware that uses individual low-level instructions (such as MAC) to implement a high-level function, and each instruction runs on the core processor. In the high-level granularity system of the present invention, the firmware configures each hardware accelerator, which in turn represent high-level functions (such as motion compensation) that run (using a well-defined specification of input data) without intervention from the main core processor. Therefore, each hardware accelerator runs in parallel according to a processing pipeline dictated by the firmware in the core processor. Upon completion of the high-level functions, each accelerator notifies the main core processor, which in turn decides what the next processing pipeline step should be.

The software control typically consists of a simple pipeline that orchestrates decoding by issuing commands to each hardware accelerator module for each pipeline stage, and a status reporting mechanism that makes sure that all modules have completed their pipeline tasks before issuing the start of the next pipeline stage.

FIG. 3 is a high-level functional block diagram of a digitalvideo decoding system300 according to an illustrative embodiment of the present invention. The digitalvideo decoding system300 ofFIG. 3 can illustratively be employed to implement thedigital video decoder116 ofFIGS. 1 and 2. Thecore processor302 is the central control unit of thedecoding system300. Thecore processor302 prepares the data for decoding. Thecore processor302 also orchestrates the macroblock (MB) processing pipeline for all modules and fetches the required data from main memory via thebridge304. Thecore processor302 also handles some data processing tasks. Picture level processing, including sequence headers, GOP headers, picture headers, time stamps, macroblock-level information except the block coefficients, and buffer management, are performed directly and sequentially by thecore processor302, without using theaccelerators304,306,308,309,310,312 and314 other than the PVLD306 (which accelerates general bitstream parsing). Picture level processing does not overlap with slice level/macroblock decoding in this embodiment.

Programmable variable length decoder (PVLD)306,inverse quantizer308,inverse transform module309,pixel filter310,motion compensation module312 and loop/post filter314 are hardware accelerators that accelerate special decoding tasks that would otherwise be bottlenecks for real-time video decoding if these tasks were handled by thecore processor302 alone. Eachhardware module306,308,309,310,312 and314 is internally configurable or programmable to allow changes according to various processing algorithms. In an alternative embodiment,modules308 and309 are implemented in the form of atransform engine307 that handles all functionality, but which is conceptually equivalent to the union of308 and309. In a further alternative embodiment,modules310 and312 are implemented in the form of afilter engine311 which consists of an internal SIMD (single instruction multiple data) processor and a general purpose controller to interface to the rest of the system, but which is conceptually equivalent to the union of310 and312. In a further alternative embodiment,module314 is implemented in the form of another filter engine similar to311 which consists of an internal SIMD (single instruction multiple data) processor and a general purpose controller to interface to the rest of the system, but which is conceptually equivalent to314. In a further alternative embodiment,module314 is implemented in the form of thesame filter engine311 that can also implement the equivalent function of the combination of310 and311. Eachhardware module306,308,309,310,312 and314 performs its task after being so instructed by thecore processor302. In-an illustrative embodiment of the present invention, each hardware module includes a status register that indicates whether the module has completed its assigned tasks. Theore processor302 polls the status register to determine whether the hardware module has completed its task. In an alternative embodiment, the hardware accelerators share a status register.

In an illustrative embodiment, thePVLD engine306 performs variable-length code (VLD) decoding of the block DCT coefficients. It also helps thecore processor302 to decode the header information in the compressed bitstream. In an illustrative embodiment of the present invention, thePVLD module306 is designed as a coprocessor to thecore processor302, while the rest of themodules308,309,310,312 and314 are designed as hardware accelerators. Also, in an illustrative embodiment, thePVLD module306 includes two variable length decoders. Each of the two programmable variable-length decoders can be hardwired to efficiently perform decoding according to a particular video compression standard, such as MPEG2 HD. One of them can be optionally set as a programmable VLD engine, with a code RAM to hold VLC tables for media coding formats other than MPEG2. The two VLD engines are controlled independently by thecore processor302, and either one or both of them will be employed at any given time, depending on the application.

TheIQ engine308 performs run-level pair decoding, inverse scan and quantization. Theinverse transform engine309 performs IDCT operations or other inverse transform operations like the Integer Transform of the H.26x standards. In an illustrative embodiment of the present invention, theIQ module308 and theinverse transform module309 are part of a common hardware module and use a similar interface to thecore processor302.

Thepixel filter310 performs pixel filtering and interpolation. Themotion compensation module312 performs motion compensation. Thepixel filter310 andmotion compensation module312 are shown as one module in the diagram to emphasize a certain degree of direct cooperation between them. In an illustrative embodiment of the present invention, thePF module310 and theMC module312 are part of a commonprogrammable module311 designated as a filter engine capable of performing internal SIMD instructions to process data in parallel with an internal control processor.

Thefilter module314 performs the de-blocking operation common in many low bit-rate coding standards. In one embodiment of the present invention, the filter module comprises a loop filter that performs de-blocking within the decoding loop. In another embodiment, the filter module comprises a post filter that performs de-blocking outside the decoding loop. In yet another embodiment, the filter module comprises a de-ringing filter, which may function as either a loop filter or a post filter, depending on the standard of the video being processed. In yet another embodiment, thefilter module314 includes both a loop filter and a post filter. Furthermore, in yet another embodiment, thefilter module314 is implemented using thesame filter engine311 implementation as for310 and312, except thatmodule311 is programmed to produce deblocked or deringed data as the case may be.

Thebridge module304 arbitrates and moves picture data betweendecoder memory316 and main memory. Thebridge interface304 includes an internal bus network that includes arbiters and a direct memory access (DMA) engine. Thebridge304 serves as an interface to the system buses.

In an illustrative embodiment of the present invention, thedisplay feeder module318 reads decoded frames from main memory and manages the horizontal scaling and displaying of picture data. Thedisplay feeder318 interfaces directly to a display module. In an illustrative embodiment, thedisplay feeder318 converts from420 to422 color space. Also, in an illustrative embodiment, thedisplay feeder318 includes multiple feeder interfaces, each including its own independent color space converter and horizontal scaler. Thedisplay feeder318 handles its own memory requests via thebridge module304.

Decoder memory316 is used to store macroblock data and other time-critical data used during the decode process. Eachhardware block306,308,309,310,312,314 accessesdecoder memory316 to either read the data to be processed or write processed data back. In an illustrative embodiment of the present invention, all currently used data is stored indecoder memory316 to minimize accesses to main memory. Eachhardware module306,308,309,310,312,314 is assigned one or more buffers indecoder memory316 for data processing. Each module accesses the data indecoder memory316 as the macro blocks are processed through the system. In an exemplary embodiment,decoder memory316 also includes parameter buffers that are adapted to hold parameters that are needed by the hardware modules to do their job at a later macroblock pipeline stage. The buffer addresses are passed to the hardware modules by thecore processor302. In an illustrative embodiment,decoder memory316 is a static random access memory (SRAM) unit.

FIG. 4a is a functional block diagram of digitalvideo decoding system300 according to an illustrative embodiment of the present invention. InFIG. 4a, elements that are common toFIG. 3 are given like reference numbers. InFIG. 4a, various elements are grouped together to illustrate a particular embodiment where308 and309 form part of atransform engine307,310 and312 form part of afilter engine311 that is a programmable module that implements the functionality of PF and MC,313 and315 form part of anotherfilter engine314 which is another instance of the same programmable module except that it is programmed to implement the functionality of aloop filter313 and apost filter315. In addition to the elements shown inFIG. 3,FIG. 4a shows, phase-locked loop (PLL)element320, internal data bus322,register bus324 and separate loop and postfilters313 and315 embodied in afilter engine module314 which implements the functionality of313 and315.

Thecore processor302 is the master of thedecoding system300. It controls the data flow of decoding processing. All video decode processing, except where otherwise noted, is performed in the core processor. ThePVLD306,IQ308,inverse transform309,PF310 andMC312, and filter314 are hardware accelerators to help the core processor achieve the required performance. In an illustrative embodiment of the present invention, thecore processor302 is a MIPS processor, such as a MIPS32 implementation, for example. Thecore processor302 incorporates a D cache and an I cache. The cache sizes are chosen to ensure that time critical operations are not impacted by cache misses. For example, instructions for macroblock-level processing of MPEG-2 video runs from cache. For other algorithms, time-critical code and data also reside in cache. The determination of exactly which functions are stored in cache involves a trade-off between cache size, main memory access time, and the degree of certainty of the firmware implementation for the various algorithms. The cache behavior with proprietary algorithms depends in part in the specific software design. In an illustrative embodiment, the cache sizes are 16 kB for instructions and 4 kB for data. These can be readily expanded if necessary.

At the macroblock level, thecore processor302 interprets the decoded bits for the appropriate headers and decides and coordinates the actions of the hardware blocks306,308,309,310,312 and314. Specifically, all macroblock header information, from the macroblock address increment (MBAinc) to motion vectors (MV s) and to the cbp pattern in the case of MPEG2 decoding, for example, is derived by thecore processor302. Thecore processor302 stores related information in a particular format or data structure (determined by the hardware module specifications) in the appropriate buffers in thedecoder memory316. For example, the quantization scale is passed to the buffer theIQ engine308; macroblock type, motion type and pixel precision are stored in the parameter buffer for thepixel filter engine310. The core processor keeps track of certain information in order to maintain the correct pipeline, and it may store some such information in its D cache, some in main system memory and some in thedecoder memory316, as required by the specific algorithm being performed. For example, for some standards, motion vectors of the macroblock are kept as the predictors for future motion vector derivation.

In an illustrative embodiment the programmablevariable length decoder306 performs decoding of variable length codes (VLC) in the compressed bit stream to extract values, such as DCT coefficients, from the compressed data stream. Different coding formats generally have their own unique VLC tables. ThePVLD306 is completely configurable in terms of the VLC tables it can process. ThePVLD306 can accommodate a dynamically changing set of VLC tables, for example they may change on a macroblock-to-macroblock basis. In an illustrative embodiment of the present invention, thePVLD306 includes a register that the core processor can program to guide thePVLD306 to search for the VLC table of the appropriate encoding/decoding algorithm. ThePVLD306 decodes variable length codes in as little as one clock, depending on the specific code table in use and the specific code being decoded.

ThePVLD306 is designed to support the worst-case requirement for VLD operation with MPEG-2 HDTV (MP@HL), while retaining its full programmability. ThePVLD306 includes a code table random access memory (RAM) for fastest performance. Code tables such a MPEG-2 video can fit entirely within the code RAM. Some formats, such as proprietary formats, may require larger code tables that do not fit entirely within the code RAM in thePVLD306. For such cases, thePVLD306 can make use of both thedecoder memory316 and the main memory as needed. Performance of VLC decoding is reduced somewhat when codes are searched invideo memory316 and main memory. Therefore, for formats that require large tables of VLC codes, the most common codes are typically stored in the PVLD code RAM, the next most common codes are stored in decoder memory, and the least common codes are stored in main memory. Also, such codes are stored indecoder memory316 and main memory such that even when extended look-ups indecoder memory316 and main memory are required, the most commonly occurring codes are found more quickly. This allows the overall performance to remain exceptionally high.

In an illustrative embodiment of the present invention, thePVLD306 is architected as a coprocessor of thecore processor302. That is, it can operate on a single-command basis where the core processor issues a command (via a coprocessor instruction) and waits (via a Move From Coprocessor instruction) until it is executed by thePVLD306, without polling to determine completion of the command. This increases performance when a large number of VLC codes are parsed under software control. Additionally, thePVLD306 can operate on a block-command basis where thecore processor302 commands thePVLD306 to decode a complete block of VLC codes, such as DCT coefficients, and thecore processor302 continues to perform other tasks in parallel. In this case, thecore processor302 verifies the completion of the block operation by checking a status bit in thePVLD306. The PVLD produces results (tokens) that are stored indecoder memory316.

ThePVLD306 checks for invalid codes and recovers gracefully from them. Invalid codes may occur in the coded bit stream for a variety of reasons, including errors in the video encoding, errors in transmission, and improper discontinuities in the stream.

Theinverse quantizer module308 performs run-level code (RLC) decoding, inverse scanning (also called zig-zag scanning), inverse quantization and mismatch control. The coefficients, such as DCT coefficients, extracted by thePVLD306 are processed by theinverse quantizer308 to bring the coefficients from the quantized domain to the DCT domain. In an exemplary embodiment of the present invention, theIQ module308 obtains its input data (run-level values) from thedecoder memory316, as the result of thePVLD module306 decoding operation. In an alternative embodiment, theIQ module308 obtains its input data directly from thePVLD306. This alternative embodiment is illustratively employed in conjunction with encoding/decoding algorithms that are relatively more involved, such as MPEG-2 HD decoding, for best performance. The run-length, value and end-of-block codes read by theIQ module308 are compatible with the format created by the PVLD module when it decodes blocks of coefficient VLCs, and this format is not dependent on the specific video coding format being decoded. In an exemplary embodiment, theIQ308 andinverse transform309 modules form part of a tightly coupled module labeledtransform engine307. This embodiment has the advantage of providing fast communication betweenmodules308 and309 by virtue of being implemented in the same hardware block.

The scan pattern of theIQ module308 is programmable in order to be compatible with any required pattern. The quantization format is also programmable, and mismatch control supports a variety of methods, including those specified in MPEG-2 and MPEG-4. In an exemplary embodiment, theIQ module308 can accommodate block sizes of 16×16, 8×8, 8×4, 4×8 and 4×4. In an illustrative embodiment of the present invention, theIQ module308 includes one or more registers that are used to program the scan pattern, quantization matrix and mismatch control method. These registers are programmed by thecore processor302 to dictate the mode of operation of the IQ module. TheIQ module306 is designed in such a way that thecore processor302 can intervene at any point in the process, in case a particular decoding algorithm requires software processing of some aspect of the algorithmic steps performed by theIQ module308. For example, there may be cases where an unknown algorithm could require a different form of rounding; this can be performed in thecore processor302. TheIQ module308 has specific support for AC prediction as specified in MPEG-4 Advanced Simple Profile. In an exemplary embodiment, theIQ module308 also has specific support for the inverse quantization functions of the ISO-ITU NT (Joint Video Team) standard under development.

Theinverse transform module309 performs the inverse transform to convert the coefficients produced by theIQ module308 from the frequency domain to the spatial domain. The primary transform supported is the IDCT, as specified in MPEG-2, MPEG-4, IEEE, and several other standards. The coefficients are programmable, and it can support alternative related transforms, such as the “linear” transform in H.26L (also known as JVT), which is not quite the same as IDCT. Theinverse transform module309 supports a plurality of matrix sizes, including 8×8, 4×8, 8×4 and 4×4 blocks. In an illustrative embodiment of the present invention, theinverse transform module309 includes a register that is used to program the matrix size. This register is programmed by thecore processor302 according to the appropriate matrix size for the encoding/decoding format of the data stream being decoded.

In an illustrative embodiment of the present invention, the coefficient input to theinverse transform module309 is read fromdecoder memory316, where it was placed after inverse quantization by theIQ module308. The transform result is written back todecoder memory316. In an exemplary embodiment, theinverse transform module309 uses the same memory location indecoder memory316 for both its input and output, allowing a savings in on-chip memory usage. In an alternative embodiment, the coefficients produced by the IQ module are provided directly to theinverse transform module309, without first depositing them indecoder memory316. To accommodate this direct transfer of coefficients, in one embodiment of the present invention, theIQ module308 andinverse transform module309 use a common interface directly between them for this purpose. In an exemplary embodiment, the transfer of coefficients from theIQ module308 to theinverse transform module309 can be either direct or viadecoder memory316. For encoding/decoding algorithms that require very high rates of throughput, such as MPEG-2 HD decoding, the transfer is direct in order to save time and improve performance.

In an illustrative embodiment, the functionality of thePF310 andMC312 are implemented by means of a filter engine (FE)311. The FE is the combination of an 8-way SIMD processor2002 and a 32-bit RISC processor2004, illustrated inFIG. 4b. Both processors operate at the same clock frequency. TheSIMD engine2002 is architected to be very efficient as a coprocessor to the RISC processor (internal MIPS)2004, performing specialized filtering and decision-making tasks. TheSIMD2002 includes: a split X-memory2006 (allowing simultaneous operations), a Y-memory, a Z-register input with byte shift capability, 16 bit per element inputs, and no branch or jump functions. TheSIMD processor2002 has hardware for three-level looping, and it has a hardware function call and return mechanism for use as a coprocessor. All of these help to improve performance and minimize the area. TheRISC processor2004 controls the operations of theFE311. Its functions include the control of the data flow and scheduling tasks. It also takes care of part of the decision-making functions. TheFE311 operates like the other modules on a macro block basis under the control of themum core processor302.

Referring again toFIG. 4a, thepixel filter310 performs pixel filtering and interpolation as part of the motion compensation process. Motion compensation uses a small piece of an image from a previous frame to predict a piece of the current image; typically the reference image segment is in a different location within the reference frame. Rather than recreate the image anew from scratch, the previous image is used and the appropriate region of the image moved to the proper location within the frame; this may represent the image accurately, or more generally there may still be a need for coding the residual difference between this prediction and the actual current image. The new location is indicated by motion vectors that denote the spatial displacement in the frame with respect to the reference frame.

Thepixel filter310 performs the interpolation necessary when a reference block is translated (motion-compensated) by a vector that cannot be represented by an integer number of whole-pixel locations. For example, a hypothetical motion vector may indicate to move a particular block 10.5 pixels to the right and 0.25 pixels down for the motion-compensated prediction. In an illustrative embodiment of the present invention, the motion vectors are decoded by the PVLD 3D6 in a previous processing pipeline stage and are further processed in thecore processor302 before being passed to the pixel filter, typically via thedecoder memory316. Thus, thepixel filter310 gets the motion information as vectors and not just bits from the bitstream. In an illustrative embodiment, the reference block data that is used by the motion compensation process is read by thepixel filter310 from thedecoder memory316, the required data having been moved todecoder memory316 fromsystem memory110; alternatively the pixel filter obtains the reference block data fromsystem memory110. Typically the pixel filter obtains the processed motion vectors fromdecode memory316. The pixel data that results from motion compensation of a given macroblock is stored in memory after decoding of said macroblock is complete. In an illustrative embodiment, the decoded macroblock data is written todecoder memory316 and then transferred tosystem memory110; alternatively, the decoded macro block data may be written directly tosystem memory110. If and when that decoded macroblock data is needed for additional motion compensation of another macroblock, thepixel filter310 retrieves the reference macroblock pixel information from memory, as above, and again the reconstructed macroblock pixel information is written to memory, as above.

Thepixel filter310 supports a variety of filter algorithms, including ½ pixel and ¼ pixel interpolations in either or both of the horizontal and vertical axes; each of these can have many various definitions, and the pixel filter can be configured or programmed to support a wide variety of filters, thereby supporting a wide range of video formats, including proprietary formats. The PF module can process block sizes of 4, 8 or 16 pixels per dimension (horizontal and vertical), or even other sizes if needed. Thepixel filter310 is also programmable to support different interpolation algorithms with different numbers of filter taps, such as 2, 4, or 6 taps per filter, per dimension. In an illustrative embodiment of the present invention, thepixel filter309 includes one or more registers that are used to program the filter algorithm and the block size. These registers are programmed by thecore processor302 according to the motion compensation technique employed with the encoding/decoding format of the data stream being decoded. In another illustrative embodiment, the pixel filter is implemented using the filter engine (FE) architecture, which is programmable to support any of a wide variety of filter algorithms. As such, in either type of embodiment, it supports a very wide variety of motion compensation schemes.

Themotion compensation module312 reconstructs the macroblock being decoded by performing the addition of the decoded difference (or residual or “error”) pixel information from theinverse transform module309 to the pixel prediction data from the output of thepixel filter310. Themotion compensation module312 is programmable to support a wide variety of block sizes, including 16×16, 16×8, 8×16, 8×8, 8×4, 4×8 and 4×4. Themotion compensation module312 is also programmable to support different transform block types, such as field-type and frame-type transform blocks. Themotion compensation module312 is further programmable to support different matrix formats. Furthermore,MC module312 supports all the intra and inter prediction modes in the H.26L/JVT proposed standard. In an illustrative embodiment of the present invention, themotion compensation module312 includes one or more registers that are configurable to select the block size and format. These registers are programmed by thecore processor302 according to the motion compensation technique employed with the encoding/decoding format of the data stream being decoded. In another illustrative embodiment, the motion compensation module is a function of a filter engine (FE) that is serving as the pixel filter and motion compensation modules, and it is programmable to perform any of the motion compensation functions and variations that are required by the format being decoded.

Theloop filter313 and postfilter315 perform de-blocking filter operations. In an illustrative embodiment of the present invention, theloop filter313 and postfilter315 are combined in onefilter module314, as shown inFIG. 3. Thefilter module314 in an illustrative embodiment is the same processing structure as described for311, except that it is programmed to perform the functionality of313 and315. Some decoding algorithms employ a loop filter and others employ a post filter. Therefore, the filter module314 (orloop filter313 and postfilter315 independently) is programmable to turn on either theloop filter313 or thepost filter315 or both. In an illustrative embodiment, the filter module314 (orloop filter313 and post filter315) has a register that controls whether a loop filter or post filter scheme is employed. Thecore processor302 programs the filter module register(s) according to the bit-stream semantics. Theloop filter313 and postfilter315 each have programmable coefficients and thresholds for performing a variety of de-blocking algorithms in either the horizontal or vertical directions. Deblocking is required in some low bit-rate algorithms. De-blocking is not required in MPEG-2. However, in one embodiment of the present invention, de-blocking is used to advantage with MPEG-2 at low bit rates.

In one embodiment of the present invention, the input data to theloop filter313 and postfilter315 comes fromdecoder memory316, the input pixel data having been transferred fromsystem memory110 as appropriate, typically at the direction of thecore processor302. This data includes pixel and block/macroblock parameter data generated by other modules in thedecoding system300. The output data from theloop filter313 and postfilter315 is written intodecoder memory316. Thecore processor302 then causes the processed data to be put in its correct location insystem memory110. Thecore processor302 can program operational parameters intoloop filter313 and postfilter315 registers at any time. In an illustrative embodiment, all parameter registers are double buffered. In another illustrative embodiment theloop filter313 and postfilter315 obtain input pixel data fromsystem memory110, and the results may be written tosystem memory110.

Theloop filter313 and postfilter315 are both programmable to operate according to any of a plurality of different encoding/decoding algorithms. In the embodiment whereinloop filter313 and postfilter315 are separate hardware units, theloop filter313 and postfilter315 can be programmed similarly to one another. The difference is where in the processing pipeline eachfilter313,315 does its work. Theloop filter313 processes data within the reconstruction loop and the results of the filter are used in the actual reconstruction of the data. Thepost filter315 processes data that has already been reconstructed and is fully decoded in the two-dimensional picture domain. In an illustrative embodiment of the present invention, the coefficients, thresholds and other parameters employed by theloop filter313 and the post filter315 (or, in the alternative embodiment, filter module314) are programmed by thecore processor302 according to the de-blocking technique employed with the encoding/decoding format of the data stream being decoded.

Thecore processor302,bridge304,PVLD306,IQ308,inverse transform module309,pixel filter310,motion compensation module312,loop filter313 and postfilter315 have access todecoder memory316 via the internal bus322 or via equivalent functionality in thebridge304. In an exemplary embodiment of the present invention, thePVLD306,IQ308,inverse transform module309,pixel filter310,motion compensation module312,loop filter313 and postfilter315 use thedecoder memory316 as the source and destination memory for their normal operation. In another embodiment, thePL VD306 uses thesystem memory110 as the source of its data in normal operation. In another embodiment, thepixel filter310 andmotion compensation module312, or the equivalent function in thefilter module314, use thedecoder memory316 as the source for residual pixel information and they usesystem memory110 as the source for reference pixel data and as the destination for reconstructed pixel data. In another embodiment, theloop filter313 andpost processor315, or the equivalent function in thefilter module314,use system memory110 as the source and destination for pixel data in normal operation. The CPU has access todecoder memory316, and theDMA engine304 can transfer data betweendecoder memory316 and themain system memory110. The arbiter fordecoder memory316 is in thebridge module304. In an illustrative embodiment,decoder memory316 is a static random access memory (SRAM) unit.

Thebridge module304 performs several functions. In an illustrative embodiment, thebridge module304 includes an interconnection network to connect all the other modules of the MVP as shown schematically as internal bus322 and registerbus324. It is the bridge between the various modules ofdecoding system300 and the system memory. It is the bridge between theregister bus324, thecore processor302, and the main chip-level register bus. It also includes a DMA engine to service the memories within thedecoder system300, includingdecoder memory316 and local memory units within individual modules such asPVLD306. The bridge module illustratively includes an asynchronous interface capability and it supports different clock rates in thedecoding system300 and the main memory bus, with either clock frequency being greater than the other.

Thebridge module304 implements a consistent interface to all of the modules of thedecoding system300 where practical.Logical register bus324 connects all the modules and serves the purpose of accessing control and status registers by themain core processor302. Coordination of processing by themain core processor302 is accomplished by a combination of accessing memory, control and status registers for all modules.

In an illustrative embodiment of the present invention, thedisplay feeder318 module reads decoded pictures (frames or fields, as appropriate) from main memory in their native decoded format (4:2:0, for example), converts the video into 4:2:2 format, and performs horizontal scaling using a polyphase filter. According to an illustrative embodiment of the present invention, the coefficients, scale factor, and the number of active phases of the polyphase filter are programmable. In an illustrative embodiment of the present invention, thedisplay feeder318 includes one or more registers that are used to program these parameters. These registers are programmed by thecore processor302 according to the desired display format. In an exemplary embodiment the polyphase filter is an 8 tap, 11 phase filter. The output is illustratively standard 4:2:2 format YCrCb video, in the native color space of the coded video (for example, ITU-T 709-2 or ITU-T 601-B color space), and with a horizontal size that ranges, for example, from 160 to 1920 pixels. The horizontal scaler corrects for coded picture sizes that differ from the display size, and it also provides the ability to scale the video to arbitrary smaller or larger sizes, for use in conjunction with subsequent 2-dimensional scaling where required for displaying video in a window, for example. In one embodiment, thedisplay feeder318 is adapted to supply two video scan lines concurrently, in which case the horizontal scaler in thefeeder318 is adapted to scale two lines concurrently, using identical parameters.

FIG. 5 is a block diagram depicting a clocking scheme fordecoding system300 according to an illustrative embodiment of the present invention. InFIG. 5, elements that are common toFIGS. 3 and 4 are given like reference numbers. Hardware accelerators block330 includesPVLD306,IQ308,inverse transform module309,pixel filter310,motion compensation module312 andfilter engine314. In an illustrative embodiment of the present invention, thecore processor302 runs at twice the frequency of the other processing modules. In another related illustrative embodiment,hardware accelerator block330 includesPVLD306,IQ308, andinverse transform module309, while one instance of thefilter engine module311 implementspixel filter310 andmotion compensation312, and yet another instance of thefilter module314 implementsloop filter313 and postfilter315, noting thatFE311 andFE314 receive both 243 MHz and 121.5 MHz clocks. In an exemplary embodiment, the core processor runs at 243 MHz and the individual modules at half this rate, i.e., 121.5 MHz. An elegant, flexible and efficient clock strategy is achieved by generating two internal clocks in an exact 2:1 relationship to each other. The system clock signal (CLK_IN)332 is used as input to the phase locked loop element (PLL)320, which is a closed-loop feedback control system that locks to a particular phase of the system clock to produce a stable signal with little jitter. ThePLL element320 generates a IX clock (targeting, e.g., 121.5 MHz) for thehardware accelerators330,bridge304 and the coreprocessor bus interface303, while generating a 2X clock (targeting, e.g., 243 MHz) for thecore processor302 and the coreprocessor bus interface303.

Referring again toFIGS. 3 and 4, for typical video formats such as MPEG-2, picture level processing, from the sequence level down to the slice level, including the sequence headers, picture headers, time stamps, and buffer management, are performed directly and sequentially by thecore processor302. ThePVLD306 assists the core processor when a bit-field in a header is to be decoded. Picture level processing does not overlap with macroblock level decoding.

The macroblock level decoding is the main video decoding process. It occurs within a direct execution loop. In an illustrative embodiment of the present invention, hardware blocksPVLD306,IQ308,inverse transform module309,pixel filter310, motion compensation module312 (and, depending on which decoding algorithm is being executed, possibly loop filter313) are all involved in the decoding loop. Thecore processor302 controls the loop by polling the status of each of the hardware blocks involved.

Still another aspect of the present invention is directed to a method of decoding a digital media data stream. Pursuant to this method, media data of a first encoding/decoding format is received. At least one external decoding function, such as variable-length decoding or inverse quantization, e.g., is configured based on the first encoding/decoding format. Media data of the first encoding/decoding format is decoded using the at least one external decoding function. Media data of a second encoding/decoding, format is received. The at least one external decoding function is configured based on the second encoding/decoding format. Then media data of the second encoding/decoding format is decoded using the at least one external decoding function.

In an illustrative embodiment of the present invention, the actions of the various hardware blocks are arranged in an execution pipeline comprising a plurality of stages. As used in the present application, the term “stage” can refer to all of the decoding functions performed during a given time slot, or it can refer to a functional step, or group of functional steps, in the decoding process. The pipeline scheme aims to achieve maximum throughput in defined worst case decoding scenarios. Pursuant to this objective, it is important to utilize the core processor efficiently.FIG. 6 is a chart representing a decoding pipeline according to an illustrative embodiment of the present invention. The number of decoding functions in the pipeline may vary depending on the target applications. Due to the selection of hardware elements that comprise the pipeline, the pipeline architecture of the present invention can accommodate, at least, substantially any existing or future compression algorithms that fall into the general class of block-oriented algorithms.

The rows ofFIG. 6 represent the decoding functions performed as part of the pipeline according to an exemplary embodiment.Variable length decoding600 is performed byPVLD306. Run length/inverse scan/IQ/mismatch602 are functions performed byIQ module308.Inverse transform operations604 are performed by theinverse transform module309. Pixel filter reference fetch606 andpixel filter reconstruction608 are performed bypixel filter310.Motion compensation reconstruction610 is performed bymotion compensation module312. The columns ofFIG. 6 represent the pipeline stages. The designations MB_i, MB_i+2, etc. represent the i^thmacroblock in a data stream, the i+1^stmacroblock in the data stream, the i+2^ndmacroblock, and so on. The pipeline scheme supports one pipeline stage per module, wherein any hardware module that depends on the result of another module is arranged in a following MB pipeline stage. In an illustrative embodiment, the pipeline scheme can support more than one pipeline stage per module.

At any given stage in the pipeline, while a given function is being performed on a given macroblock, the next macroblock in the data stream is being worked on by the previous function in the pipeline. Thus, at stage x612 in the pipeline represented inFIG. 6,variable length decoding600 is performed on MBi. Explodedview620 of the variablelength decoding function600 demonstrates how functions are divided between thecore processor302 and thePVLD306 during this stage, according to one embodiment of the present invention. Explodedview620 shows that during stage x612, thecore processor302 decodes the macmblock header of MB_i. ThePVLD306 assists thecore processor302 in the decoding of macroblock headers. Thecore processor302 also reconstructs the motion vectors of MB_i, calculates the address of the pixel filter reference fetch for MB_i, performs pipeline flow control and checks the status ofIQ module308,inverse transform module309,pixel filter310 andmotion compensator312 during stage x612. The hardware blocks operate concurrently with thecore processor302 while decoding a series of macroblocks. Thecore processor302 controls the pipeline, initiates the decoding of each macroblock, and controls the operation of each of the hardware accelerators. The core processor firmware checks the status of each of the hardware blocks to determine completion of previously assigned tasks and checks the buffer availability before advancing the pipeline. Each block will then process the corresponding next macroblock. ThePVLD306 also decodes the macro block coefficients of Mbi during stage x. Block coefficient VLC decoding is not started until thecore processor302 decodes the whole macro block header. Note that the functions listed in explodedview620 are performed during each stage of the pipeline ofFIG. 6, even though, for simplicity's sake, they are only exploded out with respect to stage x612.

At the next stage x+1614, theinverse quantizer308 works on MB_i(function602) whilevariable length decoding600 is performed on the next macroblock, MB_i+1. In stage x+1614, the data that theinverse quantizer308 works on are the quantized transform coefficients of MB_iextracted from the data stream by thePVLD306 during stage x612. In an exemplary embodiment of the present invention, also during stage x+1614, the pixel filter reference data is fetched for MB_i(function606) using the pixel filter reference fetch address calculated by thecore processor302 during stage x612.

Then, at stage x+2616, theinverse transform module309 performsinverse transform operations604 on the MB_itransform coefficients that were output by theinverse quantizer308 during stage x+1. Also during stage x+2, thepixel filter310 performspixel filtering608 for MB_iusing the pixel filter reference data fetched in stage x+1614 and the motion vectors reconstructed by thecore processor302 in stage x612. Additionally at stage x+2616, theinverse quantizer308 works on MB_i+1(function602), the pixel filter reference data is fetched for MB_i+1(function606), andvariable length decoding600 is performed on MB_i+2.

At stage x+3618, themotion compensation module312 performsmotion compensation reconstruction610 on MB_iusing decoded difference pixel information produced by the inverse transform module309 (function604) and pixel prediction data produced by the pixel filter310 (function608) in stage x+2616. Also during stage x+3618, theinverse transform module309 performsinverse transform operations604 on MB_i+hthepixel filter310 performspixel filtering608 for MB_i+1, theinverse quantizer308 works on MBi+2 (function602), the pixel filter reference data is fetched for MB_i+2(function606), andvariable length decoding600 is performed on MB_i+3. While the pipeline ofFIG. 6 shows just four pipeline stages, in an illustrative embodiment of the present invention, the pipeline includes as many stages as is needed to decode a complete incoming data stream.

In an alternative embodiment of the present invention, the functions of two or more hardware modules are combined into one pipeline stage and the macroblock data is processed by all the modules in that stage sequentially. For example, in an exemplary embodiment, inverse transform operations for a given macroblock are performed during the same pipeline stage as IQ operations. In this embodiment, theinverse transform module309 waits idle until theinverse quantizer308 finishes and theinverse quantizer308 becomes idle when the inverse transform operations start. This embodiment will have a longer processing time for the “packed” pipeline stage, and therefore such embodiments may have lower throughput. The benefits of the packed stage embodiment include fewer pipeline stages, fewer buffers and possibly simpler control for the pipeline.

The above-described macroblock-level pipeline advances stage-by-stage. Conceptually, the pipeline advances after all the tasks in the current stage are completed. The time elapsed in one macroblock pipeline stage will be referred to herein as the macroblock (MB) time. In the general case of decoding, the MB time is not a constant and varies from stage to stage according to various factors, such as the amount of processing time required by a given acceleration module to complete processing of a given block of data in a given stage. It depends on the encoded bitstream characteristics and is determined by the bottleneck module, which is the one that finishes last in that stage. Any module, including thecore processor302 itself, could be the bottleneck from stage to stage and it is not pre-determined at the beginning of each stage.

However, for a given encoding/decoding algorithm, each module, including thecore processor302, has a defined and predetermined task or group of tasks to complete. The macroblock time for each module is substantially constant for a given decoding standard. Therefore, in an illustrative embodiment of the present invention, the hardware acceleration pipeline is optimized by hardware balancing each module in the pipeline according to the compression format of the data stream.

The main video decoding operations occur within a direct execution loop that also includes polling of the accelerator functions. The coprocessor/accelerators operate concurrently with the core processor while decoding a series of macro blocks. Thecore processor302 controls the pipeline, initiates the decoding of each macro block, and controls the operation of each of the accelerators. The core processor also does a lot of actual decoding, as described in previous paragraphs. Upon completion of each macroblock processing stage in the core processor, firmware checks the status of each of the accelerators to determine completion of previously assigned tasks. In the event that the firmware gets to this point before an accelerator module has completed its required tasks, the firmware polls for completion. This is appropriate, since the pipeline cannot proceed efficiently until all of the pipeline elements have completed the current stage, and an interrupt driven scheme would be less efficient for this purpose. In an alternative embodiment, thecore processor302 is interrupted by the coprocessor or hardware accelerators when an exceptional occurrence is detected, such as an error in the processing task. In another alternative embodiment, the coprocessor or hardware accelerators interrupt the core processor when they complete their assigned tasks.

Eachhardware module306,308,309,310,312,313,315 is independently controllable by thecore processor302. Thecore processor302 drives a hardware module by issuing a certain start command after checking the module's status. In one embodiment, thecore processor302 issues the start command by setting up a register in the hardware module.

FIG. 7 is a flowchart representing a macroblock decoding loop according to an illustrative embodiment of the present invention.FIG. 7 depicts the decoding of one video picture, starting at the macro block level. In an illustrative embodiment of the present invention, the loop of macroblock level decoding pipeline control is fully synchronous. Atstep700, thecore processor302 retrieves a macroblock to be decoded fromsystem memory110. Atstep710, the core processor starts all the hardware modules for which input data is available. The criteria for starting all modules depends on an exemplary pipeline control mechanism illustrated inFIG. 6. Atstep720, thecore processor302 decodes the macroblock header with the help of thePVLD306. Atstep730, when the macroblock header is decoded, thecore processor302 commands thePVLD306 for block coefficient decoding. Atstep740, thecore processor302 calculates motion vectors and memory addresses, such as the pixel filter reference fetch address, controls buffer rotation and performs other housekeeping tasks. Atstep750, thecore processor302 checks to see whether the acceleration modules have completed their respective tasks. Atdecision box760, if all of the acceleration modules have completed their respective tasks, control passes todecision box770. If, atdecision box760, one or more of the acceleration modules have not finished their tasks, thecore processor302 continues polling the acceleration modules until they have all completed their tasks, as shown bystep750 anddecision box760. Atdecision box770, if the picture is decoded, the process is complete. If the picture is not decoded, thecore processor302 retrieves the next macroblock and the process continues as shown bystep700. In an illustrative embodiment of the present invention, when the current picture has been decoded, the incoming macroblock data of the next picture in the video sequence is decoded according to the process ofFIG. 7.

In general, thecore processor302 interprets the bits decoded (with the help of the PVLD306) for the appropriate headers and sets up and coordinates the actions of the hardware modules. More specifically, all header information, from the sequence level down to the macroblock level, is requested by thecore processor302. Thecore processor302 also controls and coordinates the actions of each hardware module. The core processor configures the hardware modules to operate in accordance with the encoding/decoding format of the data stream being decoded by providing operating parameters to the hardware modules. The parameters include but are not limited to (using MPEG2 as an example) the cbp (coded block pattern) used by thePVLD306 to control the decoding of the transform block coefficients, the quantization scale used by theIQ module308 to perform inverse quantization, motion vectors used by thepixel filter309 andmotion compensation module310 to reconstruct the macroblocks, and the working buffer address(es) indecoder memory316.

Eachhardware module306,308,309,310,312,313,315 performs the specific processing as instructed by thecore processor302 and sets up its status properly in a status register as the task is being executed and when it is done. Each of the modules has or shares a status register that is polled by the core processor to determine the module's status. In an alternative embodiment, each module issues an interrupt signal to the core processor so that in addition to polling the status registers, the core processor can be informed asynchronously of exceptional events like errors in the bitstream. Each hardware module is assigned a set of macroblock buffers indecoder memory316 for processing purposes. In an illustrative embodiment, each hardware module signals the busy/available status of the working buffer(s) associated with it so that thecore processor302 can properly coordinate the processing pipeline.

In an exemplary embodiment of the present invention, thehardware accelerator modules306,308,309,319,312,313,314,315 generally do not communicate with each other directly. The accelerators work on assigned areas ofdecoder memory316 and produce results that are written back todecoder memory316, in some cases to the same area ofdecoder memory316 as the input to the accelerator, or results are written back to main memory. In one embodiment of the present invention, when the incoming bitstream is of a format that includes a relatively large amount of data, or of a relatively complex encoding/decoding format, the accelerators in some cases may bypass thedecoder memory316 and pass data between themselves directly.

Software codecs from other sources, such as proprietary codecs, are ported to thedecoding system300 by analyzing the code to isolate those functions that are amenable to acceleration, such as variable-length decoding, run-length coding, inverse scanning, inverse quantization, transform, pixel filter, motion compensation, de-blocking filter, and display format conversion, and replacing those functions with equivalent functions that use the hardware accelerators in thedecoding system300. In an exemplary embodiment of the present invention,modules310,312 and313,315 are implemented in a programmable SIMD/RISC filter engine module (311 and314 respectively) that allows execution of a wide range of decoding algorithms, even ones that have not yet been specified in by any standards body. Software representing all other video decoding tasks is compiled to run directly on the core processor.

In an illustrative embodiment of the present invention, some functions are interrupt driven, particularly the management of the display, i.e., telling the display module which picture buffer to display from at each field time, setting display parameters that depend on the picture type (e.g. field or frame), and performing synchronization functions. Thedecoding system300 of the present invention provides flexible configurability and programmability to handle different video stream formats.FIG. 8 is a flowchart representing a method of decoding a digital video data stream or set of streams containing more than one video data format, according to an illustrative embodiment of the present invention. Atstep800, video data of a first encoding/decoding format is received. Atstep810, at least one external decoding function, such as variable-length decoding or inverse quantization: is configured based on the first encoding/decoding format. Atstep820, video data of the first encoding/decoding format is decoded using the at least one external decoding function. In an illustrative embodiment of the present invention, a full picture, or a least a full row, is processed before changing formats and before changing streams. Atstep830, video data of a second encoding/decoding format is received. Atstep840, at least one external decoding function is configured based on the second encoding/decoding format. Then, at step850, video data of the second encoding/decoding format is decoded using the at least one external decoding function. In an exemplary embodiment, the at least one decoding function is performed by one or more ofhardware accelerators306,308,309,310,312,313,314 and315. The hardware accelerators are programmed or configured by thecore processor302 to operate according to the appropriate encoding/decoding format. As is described above with respect to the individual hardware accelerators ofFIGS. 3 and 4, in one illustrative embodiment the programming for different decoding formats is done through register read/write. The core processor programs registers in each module to modify the operational behavior of the module.

In another illustrative embodiment, some or all of the hardware accelerators comprise programmable processors which are configured to operate according to different encoding/decoding formats by changing the software executed by those processors, in addition to programming registers as appropriate to the design. Although a preferred embodiment of the present invention has been described, it should not be construed to limit the scope of the appended claims. For example, the present invention is applicable to any type of media, including audio, in addition to the video media illustratively described herein. Those skilled in the art will understand that various modifications may be made to the described embodiment. Moreover, to those skilled in the various arts, the invention itself herein will suggest solutions to other tasks and adaptations for other applications. It is therefore desired that the present embodiments be considered in all respects as illustrative and not restrictive, reference being made to the appended claims rather than the foregoing description to indicate the scope of the invention.

Claims (31)

What is claimed is:

1. A method of decoding a digital media data stream, comprising:

(a) receiving media data of a first encoding/decoding format;

(b) configuring at least one external decoding function based on the first encoding/decoding format;

(c) decoding media data of the first encoding/decoding format using the at least one external decoding function;

(d) receiving media data of a second encoding/decoding format;

(e) configuring the at least one external decoding function based on the second encoding/decoding format; and

(f) decoding media data of the second encoding/decoding format using the at least one external decoding function,

wherein decoding media data using the at least one external decoding function in operations (c) and (f) comprises at least one configurable hardware module performing the at least one external decoding function, and wherein configuring the at least one external decoding function in operations (b) and (e) comprises configuring the at least one configurable hardware module,

wherein the at least one configurable hardware module is a plurality of configurable hardware modules, and wherein each of the plurality of configurable hardware modules performs at least one decoding function,

wherein at least one of the plurality of configurable hardware modules does not include a processor.

2. The method ofclaim 1, wherein the digital media data stream is a video stream and the media data is video data.

3. The method ofclaim 2, wherein the at least one external decoding function is an entropy decoding function.

4. The method ofclaim 2, wherein the at least one external decoding function is an inverse quantization function.

5. The method ofclaim 2, wherein the at least one external decoding function is an inverse transform operation.

6. The method ofclaim 2, wherein the at least one external decoding function is a pixel filtering function.

7. The method ofclaim 2, wherein the at least one external decoding function is a motion compensation function.

8. The method ofclaim 2, wherein the at least one external decoding function is a de-blocking operation.

9. A video decoding method, comprising:

receiving a first video macroblock encoded in a first encoding format;

configuring a first external decoding function based on the first encoding format;

decoding the first video macroblock using the first external decoding function;

receiving a second video macroblock encoded in a second encoding format;

configuring a second external decoding function based on the second encoding format; and

decoding the second video macroblock using the second external decoding function.

10. The method ofclaim 9, wherein the first external decoding function is an entropy decoding function.

11. The method ofclaim 9, wherein the first external decoding function is an inverse quantization function.

12. The method ofclaim 9, wherein the first external decoding function is an inverse transform operation.

13. The method ofclaim 9, wherein the first external decoding function is a pixel filtering function.

14. The method ofclaim 9, wherein the first external decoding function is a motion compensation function.

15. The method ofclaim 9, wherein the first external decoding function is a de-blocking operation.

16. The method ofclaim 9, wherein the second external decoding function is one selected from the group consisting of:

an entropy decoding function;

an inverse quantization function;

an inverse transform operation;

a pixel filtering function;

a motion compensation function; and

a de-blocking operation.

17. A video decoding method, comprising:

determining a data format for a video data stream;

configuring a programmable entropy decoder to perform entropy decoding based on the determined data format;

configuring an inverse quantizer to perform an inverse quantization based on the determined data format;

configuring an inverse transform accelerator to perform an inverse transform operations based on the determined data format;

configuring a pixel filter to perform a pixel filtering based on the determined data format;

configuring a motion compensator to perform a motion compensation based on the determined data format; and

configuring a de-blocking filter to perform a de-blocking operation based on the determined data format.

18. The method ofclaim 17, wherein the video data stream comprises a digital video image.

19. The method ofclaim 18, wherein the digital video image comprises macroblocks.

20. The method ofclaim 17, further comprising determining a second data format for a second video data stream; and configuring at least one of the programmable entropy decoder, inverse quantizer, inverse transform accelerator, pixel filter, motion compensator, and de-blocking filter to perform operations based on the determined second data format.

21. The method of claim 1, wherein the plurality of configurable hardware modules comprises two or more configurable hardware modules selected from the group consisting of:

an inverse quantizer adapted to perform inverse quantization on the digital media data stream;

an inverse transform accelerator adapted to perform inverse transform operations on the digital media data stream;

a pixel filter adapted to perform pixel filtering on the digital media data stream;

a motion compensator adapted to perform motion compensation on the digital media data stream; and

a de-blocking filter adapted to perform a de-blocking operation on the digital media data stream.

22. The method of claim 1, wherein the plurality of configurable hardware modules comprises four or more configurable hardware modules selected from the group consisting of:

an inverse quantizer adapted to perform inverse quantization on the digital media data stream;

an inverse transform accelerator adapted to perform inverse transform operations on the digital media data stream;

a pixel filter adapted to perform pixel filtering on the digital media data stream;

a motion compensator adapted to perform motion compensation on the digital media data stream; and

a de-blocking filter adapted to perform a de-blocking operation on the digital media data stream.

23. The method of claim 1, wherein the plurality of configurable hardware modules comprises:

an inverse quantizer adapted to perform inverse quantization on the digital media data stream;

an inverse transform accelerator adapted to perform inverse transform operations on the digital media data stream;

a pixel filter adapted to perform pixel filtering on the digital media data stream;

a motion compensator adapted to perform motion compensation on the digital media data stream; and

a de-blocking filter adapted to perform a de-blocking operation on the digital media data stream.

24. The method of claim 1, wherein the plurality of configurable hardware modules comprises three or more configurable hardware modules selected from the group consisting of:

an inverse quantizer adapted to perform inverse quantization on the digital media data stream;

an inverse transform accelerator adapted to perform inverse transform operations on the digital media data stream;

a pixel filter adapted to perform pixel filtering on the digital media data stream;

a motion compensator adapted to perform motion compensation on the digital media data stream; and

a de-blocking filter adapted to perform a de-blocking operation on the digital media data stream.

25. A method of decoding a digital media data stream, comprising:

(a) receiving media data of a first encoding/decoding format;

(b) configuring at least one external decoding function based on the first encoding/decoding format;

(c) decoding media data of the first encoding/decoding format using the at least one external decoding function;

(d) receiving media data of a second encoding/decoding format;

(e) configuring the at least one external decoding function based on the second encoding/decoding format; and

(f) decoding media data of the second encoding/decoding format using the at least one external decoding function,

wherein decoding media data using the at least one external decoding function in operations (c) and (f) comprises at least one configurable hardware module performing the at least one external decoding function, and wherein configuring the at least one external decoding function in operations (b) and (e) comprises configuring the at least one configurable hardware module,

wherein the at least one configurable hardware module is a plurality of configurable hardware modules, and wherein each of the plurality of configurable hardware modules performs at least one decoding function,

wherein none of the plurality of configurable hardware modules includes a processor.

26. The method of claim 24, wherein at least one of the plurality of configurable hardware modules is a hardware accelerator.

27. A method of decoding a digital media data stream, comprising:

(a) receiving media data of a first encoding/decoding format;

(b) configuring at least one external decoding function based on the first encoding/decoding format;

(c) decoding media data of the first encoding/decoding format using the at least one external decoding function;

(d) receiving media data of a second encoding/decoding format;

(e) configuring the at least one external decoding function based on the second encoding/decoding format; and

(f) decoding media data of the second encoding/decoding format using the at least one external decoding function,

wherein decoding media data using the at least one external decoding function in operations (c) and (f) comprises at least one configurable hardware module performing the at least one external decoding function, and wherein configuring the at least one external decoding function in operations (b) and (e) comprises configuring the at least one configurable hardware module,

wherein the at least one configurable hardware module is a plurality of configurable hardware modules, and wherein each of the plurality of configurable hardware modules performs at least one decoding function,

wherein each of the configurable hardware modules is separate from others of the plurality of configurable hardware modules.

28. The method of claim 24, wherein the plurality of configurable hardware modules runs in parallel according to a processing pipeline.

29. The method of claim 28, further comprising, dictating, by a core decoding processor, the processing pipeline.

30. The method of claim 24, wherein a core decoding processor programs a register for at least one of the configurable hardware modules.

31. A method of decoding a digital media data stream, comprising:

(a) receiving media data of a first encoding/decoding format;

(b) configuring at least one external decoding function based on the first encoding/decoding format;

(c) decoding media data of the first encoding/decoding format using the at least one external decoding function;

(d) receiving media data of a second encoding/decoding format;

(e) configuring the at least one external decoding function based on the second encoding/decoding format; and

(f) decoding media data of the second encoding/decoding format using the at least one external decoding function,

wherein decoding media data using the at least one external decoding function in operations (c) and (f) comprises at least one configurable hardware module performing the at least one external decoding function, and wherein configuring the at least one external decoding function in operations (b) and (e) comprises configuring the at least one configurable hardware module,

wherein the at least one configurable hardware module is a plurality of configurable hardware modules, and wherein each of the plurality of configurable hardware modules performs at least one decoding function,

wherein each of the plurality of configurable hardware modules is independently controlled by a core decoding processor,

wherein the core decoding processor independently controls each of the plurality of configurable hardware modules by programming a register for each of the plurality of configurable hardware modules.

Images