US20040240548A1 - Video encoding device - Google Patents

Abstract

A video encoding system includes a video source providing a multiple frame video signal, a compressed data interface, a host interface and a video encoding device. The video encoding device includes a video input processor, a global controller, a motion estimation processor, a digital signal processor and a bit-stream processor. The video input processor receives the video signal. The global controller controls the global operation of the video encoding device. The motion estimation processor is connected to the global controller. The digital signal processor is connected to the global controller and to the motion estimation processor. The bit-stream processor is connected to the digital signal processor, the global controller and the compressed data interface. The global controller stores encoding commands received from the host interface thereby programming the video input processor, the motion estimation processor, the digital signal processor and the bit-stream processor.

Description

CROSS REFERENCE TO RELATED APPLICATIONS
• This application a continuation application of U.S. Ser. No. 09/010.859, filed Jan. 22, 1998, which is incorporated in its entirety by reference herein.[0001]
FIELD OF THE INVENTION
• The present invention relates to video encoding devices in general and to single chip video encoding devices, in particular.[0002]
BACKGROUND OF THE INVENTION
• Methods for encoding an audio-visual signal are known in the art. According to these methods, a video signal is digitized, analyzed and encoded in a compressed manner. These methods are implemented in computer systems, either in software, hardware or a combined software-hardware form.[0003]
• Most hardware encoding systems consist of a set of semiconductor circuits, which are arranged on a large circuit board. State of the art encoding systems include a single semiconductor circuit, which is based on a high power processor.[0004]
• Reference is now made to FIG. 1, which is a schematic illustration of a video encoding circuit, referenced[0005]10, which is known in the art.
• [0006]Circuit10 includes amotion estimation processor12, a motionestimation memory unit14 connected to themotion estimation processor12, a Reduced Instruction Set Computer (RISC)processor16 connected to themotion estimation processor12 and animage buffer18, connected toRISC processor16.
• [0007]RISC processor16 transfers a portion of video signal fromimage buffer18 tomemory unit14.Motion estimation processor12 analyzes the motion of the video signal.Motion estimation processor12 utilizesmemory unit14 as a storage area for the video signal portion which is currently processed by it. When themotion estimation processor12 completed analyzing the motion of a video signal portion, it transfers the results of the motion estimation analysis to theRISC processor16.
• The[0008]RISC processor16 performs all other processing and encoding tasks which the video signal has to undergo, such as discrete COSINE transform (DCT), quantization, entropy encoding, bit-stream production and the like. TheRISC processor16 utilizes theimage buffer18 as a storage area for the video signal portion which is currently processed by it, and as a temporary storage for its computational purposes.
• It will be appreciated by-those skilled in the art that such encoding systems have several disadvantages. For example, one disadvantage of[0009]circuit10 is that each of theprocessing units12 and16 have a separate storage area. Accordingly, each of the processed portions of video signal, such as and ISO/IEC 13818 (MPEG-2) macro-blocks, have to be transferred to bothmemory unit14 andimage buffer18.RISC processor16 has to accessimage buffer18 for the same data, each time this data is required. Such Retrieval of large data blocks, many times, greatly increases data traffic volume over the encoding system data transmission lines.
• Another disadvantage is that[0010]circuit10 is able to execute all processing and encoding tasks in a serial manner, thereby capable of processing only a single macro-block at a time, requiring high operational processor frequencies.Circuit10 receives a macro-block, processes it and produces an encoded bit-stream. Internally, the RISCprocessor16 operates in the same manner.
• Hence, as long as the[0011]RISC processor10 hasn't completed transmitting the encoded bit-stream of a selected macro-block, it cannot receive the next macro-block.
• It will be appreciated by those skilled in the art that the operational frequency of[0012]circuit10 has a direct affect over the heat produced by it, thereby requiring large cooling elements as well as massive cooling devices such as fans and the like.
• It will be appreciated by those skilled in the art that such circuit structure requires that input-output (I\O) operations have to be performed extremely fast, thereby greatly increasing the storage memory bandwidth requirements.[0013]
• Another disadvantage of such systems is that all processing and encoding procedures (excluding motion estimation) are executed by the same RISC processor. In this case, the same circuit performs various types of computations, which makes the utilization of the processor's hardware resources very inefficient.[0014]
• Methods for estimating motion in a video signal are known in the art. According to these methods a frame is compared with previous frames. The difference between the frames is used to estimate a level of motion. These methods analyze a frame and map it, thereby indicating areas in frame which have no motion over previous frames and areas in the frame which are assigned with a motion level.[0015]
• According to one such like method each pixel in the search area is analyzed. This method requires a vast number of estimation operations and is thereby extremely resource consuming. This method is also called a full exhaustive search.[0016]
• According to another method, known in the art, the search area is scanned in a center weighted manner, which can be logarithmic, and the like, whereby the center of the search area is scanned thoroughly at full resolution and the rest of the search area is scanned at lower resolution. Areas which detected as having some motion, in the low resolution search, are scanned again in full resolution. This reduces the overall number of estimation operations.[0017]
• Reference is now made to FIG. 2, which is a schematic illustration of a Digital Signal Processing (DSP) processor, referenced[0018]50, which is known in the art.
• DSP[0019]processor50 is of a single instruction multiple data (SIMD) type machine. It includes a plurality of identical processing units (P.U.)52,5660,64,68 and72, and a random access memory (RAM)61.RAM61 is divided intosegments54,58,62,66,70 and74.
• Each memory segment is exclusively assigned and connected to a processing unit, whereas[0020]RAM segment units54,58,62,66,70 and74 are assigned to and connected to processing units (P.U.)52,56,60,64,68 and72, respectively.
• This structure has several disadvantages. One disadvantage of such machine is that the same operation is performed by all of the processing units at same time.[0021]
• Another disadvantage of the SIMD machine is that the data is not shared among the processing units. For example,[0022]processing unit56 can access data contained inRAM segment66 viaprocessing unit64 only. It cannot do so directly. It will be appreciated by those skilled in the art that such a configuration is inefficient.
• A further disadvantage is that individual operations that vary for different data items cannot be efficiently performed by an SIMD machine. The programming of such operations into the processing units, is very difficult. Such individual operations can be only performed in a serial manner, while masking all irrelevant data, resulting in shutting off most of the processing units. The utilization of the hardware resources in an SIMD machine during such programming operations is very low, and performance of the machine are dramatically decreased.[0023]
• Another disadvantage relates to the interconnection structure between the processing units. It will be appreciated that, a processing unit within an SIMD machine is connected to a limited number of neighboring processing units. Hence communication between such a processing unit and a processing unit riot connected thereto, is often a complex operation.[0024]
• Bit-stream processing and generation, in a conventional encoding circuit, is performed by a general purpose processor. Bit-stream generation requires some specific operations, which can not be performed efficiently by a general purpose processor. In order to perform such special operation, a general purpose processor uses a small portion of its processing resources, while shutting off rest of them. Therefore, the disadvantage is that the resources of such processor are not utilized efficiently.[0025]
SUMMARY OF THE PRESENT INVENTION
• It is an object of the present invention to provide a novel device for encoding an audio-visual signal, which overcomes the disadvantages of the prior art.[0026]
• There is therefore provided, in accordance with a preferred embodiment of the present invention, a video encoding system which includes a video source providing a multiple frame video signal, a compressed data interface, a host interface and a video encoding device. The video encoding device includes a video input processor, a global controller, a motion estimation processor, a digital signal processor and a bit-stream processor. The video input processor receives the video signal. The global controller controls the global operation of the video encoding device. The motion estimation processor is connected to the global controller. The digital signal processor is connected to the global controller and to the motion estimation processor. The bit-stream processor is connected to the digital signal processor, the global controller and the compressed data interface. The global controller stores encoding commands received from the host interface thereby programming the video input processor, the motion estimation processor, the digital signal processor and the bit-stream processor.[0027]
• There is also provided, in accordance with a preferred embodiment of the present invention, a video camera including an optical assembly, a light sensitive device, detecting light via the optical assembly, thereby producing a video signal and a video encoding device, connected to the light sensitive device. The video encoding device includes a video source providing a multiple frame video signal, a compressed data interface, a host interface and a video encoding device. The video encoding device includes a video input processor, a global controller, a motion estimation processor, a digital signal processor and a bit-stream processor. The video input processor receives the video signal. The global controller controls the global operation of the video encoding device. The motion estimation processor is connected to the global controller. The digital signal processor is connected to the global controller and to the motion estimation processor. The bit-stream processor is connected to the digital signal processor, the global controller and the compressed data interface. The global controller stores encoding commands received from the host interface thereby programming the video input processor, the motion estimation processor, the digital signal processor and the bit-stream processor.[0028]
BRIEF DESCRIPTION OF THE DRAWINGS
• The present invention will be understood and appreciated more fully from the following detailed description taken in conjunction with the drawings in which:[0029]
• FIG. 1 is a schematic illustration of an encoding circuit, known in the art;[0030]
• FIG. 2 is a schematic illustration of an DSP unit, which is known in the art;[0031]
• FIG. 3 is a schematic illustration of a video encoding device, constructed and operative in accordance with a preferred embodiment of the invention;[0032]
• FIG. 4 is a schematic illustration of a video encoding device, constructed and operative in accordance with another preferred embodiment of the invention;[0033]
• FIG. 5 is a schematic illustration in detail of the motion estimation processor, of the video encoding device of the device of FIG. 3, constructed and operative in accordance with a further preferred embodiment of the invention;[0034]
• FIG. 6 is a schematic illustration of a video frame, including three resolution representation of a portion of this frame;[0035]
• FIG. 7 is a schematic illustration in detail of the digital signal processor, of the video encoding device of FIG. 3, constructed and operative in accordance with another preferred embodiment of the invention;[0036]
• FIG. 8 is a schematic illustration in detail of a digital signal processor, constructed and operative in accordance with a further preferred embodiment of the invention;[0037]
• FIG. 9 is a schematic illustration in detail of the video input processor, of the encoding device of FIG. 3, constructed and operative in accordance with a further preferred embodiment of the invention;[0038]
• FIG. 10 is a schematic illustration in detail of the bit-stream processor, of the encoding device of FIG. 3, constructed and operative in accordance with another preferred embodiment of the invention;[0039]
• FIG. 11 is a schematic illustration in detail of the global controller of the encoding device of FIG. 3, constructed and operative in accordance with another preferred embodiment of the invention; and[0040]
• FIG. 12 is a schematic illustration in detail of the encoding device of FIG. 3, incorporated in a video camera, constructed and operative in accordance with a further preferred embodiment of the invention.[0041]
DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS
• The present invention overcomes the disadvantages of the prior art by providing a novel approach to video compression processing and a novel structure for a device, according to this approach.[0042]
• The device according to the invention is a massively parallel digital video processor designed, for the purpose of real-time video encoding, like MPEG. This device can be incorporated in a single chip, and installed in digital camcorders, recordable digital video disk (DVD), PC and workstation multimedia, educational and training systems, video conferencing, broadcast equipment, security, content creation/authoring/video editing equipment, and the like.[0043]
• Reference is now made to FIG. 3, which is a schematic illustration of a video encoding device, generally referenced[0044]100, constructed and operative in accordance with a preferred embodiment of the invention.
• [0045]Device100 includes avideo input processor106, amotion estimation processor108, adigital signal processor104, a bit-stream processor102 and aglobal controller110.
• The[0046]video input processor106 is connected to theglobal controller110. Themotion estimation processor108 is connected to theglobal controller110 and to thedigital signal processor104. Thedigital signal processor104 is connected to theglobal controller110 and to the bit-stream processor102, which is also connected to theglobal controller110.
• The bit-[0047]stream processor102 is further connected to acompressed data interface128. Theglobal controller110 is further connected to ahost interface120 and to amemory unit122. The input of thevideo input processor106 is further connected to a digital video source (not shown) via adigital video interface124. Such a host is typically a user interface which is operative to receive commands, operational parameters, and the like, from a user or a supervising system and also to provide to the user information received fromdevice100.
• [0048]Device100 operates in two modes: a programming mode and an operational mode.Device100 is operative to run according to both modes at the same time. In the programming mode, an external host transfers the data and control parameters to theglobal controller110, via thehost interface120.
• The[0049]global controller110 can transfer the data and control signals to thevideo input processor106,motion estimation processor108,digital signal processor104 and bit-stream processor102.
• In the operational mode, the[0050]video input processor106 captures motion video signal from an external video source via the digitizedvideo interface124.Video input processor106 also performs preprocessing of the video signal, such as spatial filtering, noise reduction, image quality improvement, image size adjustment, and the like, color format conversion, and the like, thereby producing preprocessed video data.
• [0051]Video input processor106 accumulates the preprocessed video data into data blocks and transfers them to theglobal controller110.Global controller110 stores the data blocks inmemory unit122. In the present example, the device operates under MPEG-2 video compression standard. Hence, a data block represents an MPEG-2 macro-block, which is a sixteen by sixteen [16×16] matrix of luminance pixels and two, four or eight, eight by eight [8×8] matrices of chrominance pixels, as defined by the MPEG-2 standard. A reference frame represents a picture which is compared versus current picture during the motion estimation.
• The[0052]global controller110 retrieves a current picture macro-block and reference picture macro-blocks from thememory unit122 and loads them to themotion estimation processor108.Motion estimation processor108 compares the current picture macro-block with the respective reference frame macro-blocks, thereby producing an estimation of the motion of the current picture macro-block.
• The[0053]motion estimation processor108 uses this estimation to remove temporal redundancy of the video signal, as will be described in detail hereinbelow. Themotion estimation processor108 transfers the resulting motion estimation data to theglobal controller110.Motion estimation processor108 also transfers the current picture macro-block and the corresponding reference frames macro-blocks to thedigital signal processor104.
• Digital signal processor[0054]104 (DSP) executes procedures which are intended to remove the spatial redundancy of the video signal, thereby producing a sequence of compression commands, as will be described in detail hereinbelow. This sequence of compression commands includes instruction as to which frame of the original video signal is to be compressed into an I-frame, a B-frame or a P-frame, and according to which reference frames.
• Then, the[0055]digital signal processor104 transfers the sequence of compressed data to the bit-stream processor102 and to theglobal controller110. The bit-stream processor102 performs a series of encoding procedures, such as entropy encoding, and the like, as will be described in detail hereinbelow.
• The bit-[0056]stream processor102 compresses data into an MPEG-2 standard format data, in accordance with the sequence of compression commands.
• Then, the bit-[0057]stream processor102 transfers the MPEG-2 standard format data to thecompressed data interface128. It will be noted thatcompressed data interface128 can be connected to any data receptacle element such as a storage unit or a communication transmission line.
• [0058]Global controller110 controls and schedules thevideo input processor106 themotion estimation processor108, thedigital signal processor104 and the bit-stream processor102.
• [0059]Global controller110 also governs the data transfer among themotion estimation processor108,digital signal processor104 and the bit-stream processor102.
• The[0060]global controller110 also connects between the external host andvideo input processor106,motion estimation processor108,digital signal processor104 and bit-stream processor102. In the operational mode, an external host can access the register file408 (FIG. 10) ofglobal controller110 for read and/or write operations.
• According to one aspect of the invention, in operational mode, macro-blocks are fed into[0061]device100, in a horizontal or vertical raster scan manner, from the top left macro-block through to the right bottom macro-block, of a specified frame.Device100 processes a number of successive macro-blocks, at the same time. For example, while the bit-stream processor102 processes the i-th macro-block,digital signal processor104 processes the i+1-th macro-block andmotion estimation processor108 processes the i+2-th through i+4-th macro-blocks.
• According to another example, while the bit-[0062]stream processor102 processes the i-th macro-block,digital signal processor104 processes the k-th macro-block andmotion estimation processor108 processes the j-th through j+m-th macro-blocks, wherein i<k<j and m≧1.
• [0063]Device100 overcomes a disadvantage of the prior art by usingmemory unit122 as a shared storage area which is accessible to all of its internal units, viaglobal controller110. In the present example, all access tostorage unit122 is provided viaglobal controller110.
• Reference is now made to FIG. 4, which is a schematic illustration of a video encoding device, generally referenced[0064]200, constructed and operative in accordance with another preferred embodiment of the invention.
• [0065]Device200 is generally similar todevice100 and includes avideo input processor206, amotion estimation processor208, adigital signal processor204, a bit-stream processor202 and aglobal controller210.
• [0066]Device200 is also connected to acompressed data interface228, amemory unit222, a digital video source (not shown) via adigital video interface224 and ahost interface220. Indevice200, all of the internal components are connected directly tomemory unit222.
• Accordingly,[0067]video input processor206,motion estimation processor208,digital signal processor204, bit-stream processor202 andglobal controller210 can, each, access any storage address withinmemory unit222, directly, thereby performing any I\O operation.
• It will be noted that a shared memory structure according to the invention can include a combination of the examples disclosed in FIGS. 3 and 4, whereby some components are directly connected to memory unit and the rest are connected to the memory unit via a mediating element, such as[0068]global controller110.
• Referring back to FIG. 3, it is noted that according to another aspect of the invention,[0069]memory unit110 can partitioned into many sub-areas, whereby each of the internal units ofdevice100, is granted an access level which is selected from a list of access levels such as read-write, read only, write only, no access and the like. It will be appreciated by those skilled that such a structure provides tremendous flexibility, whereby the amount of memory assigned to a selected internal unit can be increased or decreased dynamically, in real-time.
• According to another aspect of the present invention,[0070]device100 performs different processing and encoding procedures in parallel, by processing a number of successive macro-blocks simultaneously. Hence, a selected macro-block is permitted to be present in thedevice100 for an extended period of time, with comparison to device10 (FIG. 1), thereby greatly reducing the operational frequency ofdevice100, by factor of at least five.
• Reference is now made to FIGS. 5 and 6. FIG. 5 is a schematic illustration in detail of[0071]motion estimation processor108, constructed and operative in accordance with a further preferred embodiment of the invention. FIG. 6 is a schematic illustration of a video frame, generally referenced170, including three resolution representation of a portion of this frame, generally referenced A, B and C.
• [0072]Motion estimation processor108 includes alow resolution processor150, afull resolution processor152, ahyper resolution processor154 and a local-controller158. The frame input of themotion estimation processor108 is connected to the inputs of thelow resolution processor150, thefull resolution processor152, the hyper resolution processor.
• The frame output of[0073]motion estimation processor108 is connected to thehyper resolution processor154. The local-controller158 is connected to thelow resolution processor150, thefull resolution processor152 and the hyper resolution processor and the control port of themotion estimation processor108, which is connected to global controller110 (FIG. 3).
• Referring now to FIG. 6,[0074]frame170 is a [6×6] matrix which includes a digital representation of a video frame Fi. The currently scanned block is block B, which is a [4×4] matrix from pixel B:(1,1) to pixel B:(4,4).Frame170 is provided toprocessors150,152 and154 (FIG. 5).
• Referring back to FIG. 5,[0075]motion estimation processor108 can operate in two modes: a programming mode and an operational mode. The motion estimation processor can operate according to both modes at the same time.
• In the programming mode, the global controller[0076]110 (FIG. 3) provides control parameters as well as data parameters, to micro-controller158, via the control port.
• Local-[0077]controller158 controls and synchronizesprocessors150,152 and154. According to a preferred embodiment of the present invention,processors150,152 and154 operate either in a parallel or in a serial manner.
• In an example of parallel operation, the[0078]low resolution processor150 processes i-th macro-block, while thefull resolution processor152 processes I-1-th macro-block, whereas thehyper resolution processor154 processes1-2-th macro-block, all at the same time.
• In an example of serial operation, the[0079]low resolution processor150 processes i-th macro-block, while bothfull resolution processor152 and thehyper resolution processor154 process I+1-th macro-block in a serial manner.
• [0080]Low resolution processor150 operates as follows. Theglobal controller110 loads the current picture macro-block and the reference pictures data blocks into thelow resolution processor150. Thelow resolution processor150 performs a resolution reduction, resulting in decreasing the amount of image data. Thelow resolution processor150 can perform the resolution reduction by different methods, like decimation, low pass filtering, non-linear filtering, and the like.
• Reference is now made to FIG. 6. In the present example, low resolution processor[0081]150 (FIG. 5) generates a low resolution block A from block B. Block A is a [2×2] matrix, wherein pixel A:(1,1) is a combined representation of pixels B:(1,1), B:(1,2), B:(2,1) and B:(2,2), pixel A:(2,1) is a combined representation of pixels B:(3,1), B:(3,2), B:(4,1) and B:(4,2), pixel A:(1,2) is a combined representation of pixels B:(1,3), B:(1,4), B:(2,3) and B:(2,4) and pixel A:(2,2) is a combined representation of pixels B:(3,3), B:(3,4), B:(4,3) and B:(4,4).
• It will be noted that such pixel combination can be performed in many ways such as calculating the average value of the combined pixels, selecting the dominant one, and the like.[0082]
• The resolution reduction of the present example is at a ratio of 1:4. It will be noted that[0083]low resolution processor150 can perform a resolution reduction at any ratio desired.
• After low resolution processor completes the resolution reduction, then, it performs a search procedure. The[0084]low resolution processor150 can performs different types of search, like full exhaustive search, telescopic search, and the like, thereby producing low resolution motion analysis. After the search is completed, the global controller110 (FIG. 3) reads the low resolution motion analysis data from thelow resolution processor150, via thelocal controller158.
• [0085]Full resolution processor152 operates as follows: Theglobal controller110 loads the current picture block (referenced B in FIG. 6) and the reference pictures data block into thefull resolution processor152, according to the low resolution motion analysis. Then, thefull resolution processor152 performs a search procedure.
• The[0086]full resolution processor152 can perform different types of search, like full exhaustive search, telescopic search, and the like, thereby producing full resolution motion analysis. After the search is completed, theglobal controller110 reads the full resolution motion analysis data from thefull resolution processor152, via thelocal controller158.
• [0087]Full resolution processor152 andhyper resolution processor154 can have a shared storage area. This aspect of the invention reduces the memory requirements of the encoding system.
• [0088]Hyper resolution processor154 operates as follows: Theglobal controller110 loads the current picture macro-block and the reference pictures data blocks into thehyper resolution processor154. Thehyper resolution processor154 multiplies the resolution of the image data, enabling a motion prediction with a resolution higher than a single pixel step.
• [0089]Hyper resolution processor154 can perform the resolution multiplication by different methods, like zero order interpolation, first order interpolation, and the like.
• With reference to FIG. 6, in the present example, hyper resolution processor[0090]154 (FIG. 5) generates a hyper resolution block C from block B. Block C is a [9×9] matrix, wherein pixels C:(1,1), C:(3,1), C:(5,1), C:(7,1), C:(1,3), C:(3,3), C:(5,3), C:(7,3), C:(1,5), C:(3,5), C:(5,5), C:(7,5), C:(1,7), C:(3,7), C:(5,7) and C:(7,7) are equal to pixels B:(1,1), B:(2,1), B:(3,1), B:(4,1), B:(1,2), B:(2,2), B:(3,2), B:(4,2), B:(1,3), B:(2,3), B:(3,3), B:(4,3), B:(1,4), B:(2,4), B:(3,4) and B:(4,4), respectively.
• Each other pixel in block C: can be an interpolation of selected B: pixels adjacent to its respective place therein. For example, pixel C:(2,2) is an interpolation of pixels B:(1,1), B:(1,2), B:(2,1) and B:(2,2). Pixel C:(2,1) is an interpolation of pixels B:(1,1), and B:(2,1). Pixel C:(1,0) is an interpolation of pixels B:(1,0), and B:(1,1).[0091]
• The resolution enhancement of the present example is at a ratio of 4:1. It will be noted that[0092]hyper resolution processor154 can perform a resolution enhancement at any ratio desired.
• After the[0093]hyper resolution processor154 completes the resolution multiplication, it performs a search procedure. Thehyper resolution processor154 can perform different types of search, like full exhaustive search, telescopic search, and the like, thereby producing hyper resolution motion analysis. After the search is completed, theglobal controller110 reads the hyper resolution motion analysis data, from thehyper resolution processor154, via thelocal controller158.
• A motion estimation processor according to another embodiment of the invention can include as many resolution processors as desired, wherein some resolution processors are low resolution processors, at a variety of low resolutions and other resolution processors are hyper resolution processors, at a variety of hyper resolutions.[0094]
• Reference is now made to FIG. 7, which is a schematic illustration in detail of[0095]digital signal processor104, of FIG. 3, constructed and operative in accordance with another preferred embodiment of the invention.
• [0096]Digital signal processor104 includes amaster controller250, a random access unit (RAM)270 andN processing units252,254,256,258,260 and262. Eachprocessing unit252,254,256,258,260 and262 is connected to themaster controller250 and to theRAM270.
• The[0097]processing units252,254,256,258,260 and262 are further connected tomotion estimation processor108 and to the global controller110 (FIG. 3), for retrieving macro block data therefrom.
• [0098]Digital signal processor104 can operate in two modes: a programming mode and an operational mode.Digital signal processor104 can operate according to both modes at the same time. In the programming mode, the global controller110 (FIG. 3) transfers data and control parameters to/frommaster controller250. Themaster controller250 can independently program eachprocessing unit252,254,256,258,260 and262.
• In the operational mode, the[0099]master controller250 and all processingunits252,254,256,258,260 and262, operate in parallel. The motion estimation processor108 (FIG. 3) transfers the current macro-block and the reference macro-blocks data to theprocessing units252,254,256,258,260 and262 of thedigital signal processor104. In the present example, the motion estimation processor108 (FIG. 3) provides luminance macro-blocks and theglobal controller110 provides chrominance macro-blocks retrieved frommemory unit122.
• The[0100]global controller110 transfers the appropriate data (like motion vectors, macro-block type, perdition type, and the like) to themaster controller250. Themaster controller250 performs special processing procedures such as like rate control, DCT type selection, macro-block type selection and the like.
• The[0101]master controller250 distributes control instructions to theprocessing units252,254,256,258,260 and262, and receives processed data from each of these processing units. Processingunits252,254,256,258,260 and262 perform processing procedures on large data blocks, such as discrete cosine transform (DCT), inverse discrete cosine transform (IDCT), quantization, inverse quantization, frame type decisions, and the like.
• Each of these processing units processes different data blocks. Each processing unit can access the data blocks associated with other processing units, in[0102]RAM270. All processing unit can execute different operations in parallel. The processing units transfer the processed coefficient data to the bit-stream processor102 (FIG. 3). Themaster controller250 of thedigital signal processor104 transfers the appropriate data (like macro-block type, DCT type, quantizer scale, etc.) to the global controller110 (FIG. 3).
• The present invention overcomes the disadvantages of the prior art by configuring the[0103]master controller250 so as to perform individual processing tasks on some data items while theprocessing units252,254,256,258,260 and262, simultaneously perform massive processing tasks on large data blocks.
• According to the present example, the[0104]master controller250 temporarily assigns a storage area inRAM270, to each of theprocessing unit252,254,256,258,260 and262.
• Hence, each[0105]processing unit252,254,256,258,260 and262 can access the data which is associated with the other processing units. It will be appreciated by those skilled in the art that such a structure greatly enhances the efficiency of processing and data transfer operations inDSP104. Such parallel access structure of the processing units also allows very fast and efficient data transfer to and from thedigital signal processor104.
• Reference is now made to FIG. 8, which is a schematic illustration in detail of a digital signal processor, generally referenced[0106]144, constructed and operative in accordance with a further preferred embodiment of the invention.
• [0107]Digital signal processor144 is generally similar todigital signal processor104, with a slightly different memory structure.Digital signal processor144 includes amaster controller288,N processing units271,274,276,280,282 and286 and N/2 random access units (RAM)272,278 and284.
• Each RAM unit is connected to two processing unit. Processing[0108]units271 and274 are connected to RAMunit272. Processingunits276 and280 are connected to RAMunit278. Processingunits282 and286 are connected to RAMunit284.
• Each processing unit is able to access any address in the RAM unit connected thereto.[0109]
• According to FIGS. 7 and 8 it will be appreciated that the invention is not limited to any shared memory structure between processing units.[0110]
• Reference is now made to FIG. 9, which is a schematic illustration in detail of[0111]video input processor106, of FIG. 3, constructed and operative in accordance with a further preferred embodiment of the invention.
• [0112]Video input processor106 includes avideo capture unit230, avideo preprocessor232 and atemporary video storage236. The inputs of thevideo capture unit230, thevideo preprocessor232 and thetemporary video storage236 are connected to the video input processor106 (FIG. 3). The input of thevideo capture unit230 is connected tovideo input processor106. Thevideo capture unit230 is connected to thevideo preprocessor232. Thevideo preprocessor232 is connected to thetemporary video storage236. The output of thetemporary video storage236 is connected to theglobal controller110.
• [0113]Video input processor106 operates in two modes: programming and operational.Video input processor106 is operative to run according to both modes at the same time. In the programming mode, the global controller110 (FIG. 3) loads data and control parameters to thevideo capture unit230, to thevideo preprocessor232 and to thetemporary video storage236.
• In the operational mode, the[0114]video capture unit230 acquires the input video signal. Thevideo capture unit230 is self synchronized with the input video signal, according to its format (NTSC, PAL, SECAM, and the like), programmable resolution (DI, SIF, QSIF, and the like), and the like.
• The[0115]video capture unit230 also provides video synchronization signals (like a new frame start, a new field start, etc.) to theglobal controller110.
• The[0116]video preprocessor232 performs a series of video processing procedures to enhance the captured video signal. The video processing procedures can include a color format conversion, size reduction, noise reduction, edge sharpening, image quality improvement, and the like.
• The[0117]temporary video storage236 accumulates the processed video signal and provides a “data ready” signal to the global controller110 (FIG. 3). Theglobal controller110 reads the accumulated image data from thetemporary video storage236. Theglobal controller110 also provides control signals to thevideo input processor106.
• The usage of the[0118]temporary video storage236 allows to efficiently adjust the data rates of an external video signal and the internal data transfer. Thevideo input processor106 can accumulate the processed video signal in a real time variable rate whereas theglobal controller110 can transfer the accumulated data to thememory unit122 in a burst. This greatly reduces the memory bandwidth requirements, and makes the usage of amemory unit122 more efficient.
• Reference is now made to FIG. 10, which is a schematic illustration in detail of bit-[0119]stream processor102, of FIG. 3, constructed and operative in accordance with a further preferred embodiment of the invention.
• Bit-[0120]stream processor102 includes alocal processing unit356, anentropy encoder358, atemporary coefficient storage360, a bit-stream formatter352, anoutput buffer354 and amicro-controller350. The input of the bit-stream processor102 is connected to the input of thetemporary coefficient storage360 and of thelocal processing unit356. Thetemporary coefficient storage360 is connected to theentropy encoder358. Theentropy encoder358 is further connected to thelocal processing unit356 and to the bit-stream formatter352. Thelocal processing unit356 is further connected to the output of the bit-stream processor102. The bit-stream formatter352 is connected to theoutput buffer354. The output of theoutput buffer354 is connected to the output of the bit-stream processor102. The input of themicro-controller158 is connected to the input of the bit-stream processor102. Themicro-controller158 of the bit-stream processor102 is connected to the processing unit, theentropy encoder358,temporary coefficient storage360, bit-stream formatter352 andoutput buffer354.
• The bit-[0121]stream processor102 can operate in two modes: programming and operational. Bit-stream processor102 is operative to run according to both modes at the same time.
• In the programming mode, the global controller[0122]110 (FIG. 3) loads the data and control parameters to micro-controller350 and tolocal processing unit356. The digital signal processor104 (FIG. 3) loads the processed coefficients to thetemporary coefficient storage360.
• In the operational mode, the[0123]entropy encoder358 loads the data from thelocal processing unit356 registers and thetemporary coefficient storage360. Theentropy encoder358 performs a series of encoding procedures, like zigzag/alternate scan, run-length encoding, variable length encoding of data, and the like, thereby producing encoded data.
• The[0124]local processing unit356 performs arithmetic and logical operations required to support the entropy encoding. Thelocal processing unit356 also provides a temporary storage for the data loaded from the global controller110 (FIG. 3) in the programming mode.
• The bit-[0125]stream formatter352 reads the encoded data from theentropy encoder358 and formats it into a standard bit-stream. Theoutput buffer354 provides a temporary storage to the bit-stream data. Themicro-controller350 provides the control and synchronization signals to thelocal processing unit356, theentropy encoder358, thetemporary coefficient storage360, the bit-stream formatter352 and theoutput buffer354.
• The[0126]global controller110 can put the bit-stream processor102 into programming or operational mode by loading an appropriate control signal to themicro-controller158 of the bit-stream processor102.
• Entropy encoding and computational operations in the bit-[0127]stream processor102 are performed in parallel, by operatingentropy encoder358 andprocessing unit356, simultaneously. This allows a very efficient utilization of the bit-stream processor102 resources.
• Reference is now made to FIG. 11, which is a schematic illustration in detail of[0128]global controller110, of FIG. 3, constructed and operative in accordance with another preferred embodiment of the invention.
• [0129]Global controller110 includes amemory controller400, anexchange unit406, aregister file408, aprocessing unit410, ahost storage404 and amicro-controller402. The input of themicro-controller402 is connected to the input of theglobal controller110. Themicro-controller402 is connected to thememory controller400,exchange unit406,register file408, processing unit,host storage404 and to output ofglobal controller110. The external storage interface of theglobal controller110 is connected to thememory controller400. The input/output of thememory controller400 is connected to the input/output of theglobal controller110 respectively. Theexchange unit406 is connected to theregister file408. Theregister file408 is connected to theprocessing unit410 and to thehost storage404 of theglobal controller110. Thehost storage404 is connected to the host interface120 (FIG. 3).
• With further reference to FIG. 3, the[0130]global controller110 schedules, synchronizes and controlsmotion estimation processor108,digital signal processor104, bit-stream processor102 andvideo input processor106. Theglobal controller110 controls the internal data transfer ofdevice100 and the external data transfer fromdevice100 to external devices such asmemory unit122, an external host, and the like. Theglobal controller110 can also initializes and performs a variety of testing procedures onmotion estimation processor108,digital signal processor104, bit-stream processor102 andvideo input processor106 and alsomemory unit122. Theglobal controller110 is a massively parallel processor, capable of simultaneous execution of computational operations, internal data transfer and external data transfer.
• The[0131]global controller110 operates in two modes: programming and operational.Global controller110 is operative to run according to both modes at the same time.
• According to the programming mode, an external host loads the data and control parameters to the[0132]host storage404. The external host instruct theglobal controller110 to perform according to the operational mode by loading a predetermined control signal into thehost storage404.
• According to the operational mode, the[0133]micro-controller402 is synchronized to an external video source by the video synchronization signal which comes from the video input processor106 (FIG. 3).
• The[0134]micro-controller402 provides control and synchronization signals to themotion estimation processor108, thedigital signal processor104, the bit-stream processor102 and thevideo input processor106. Themicro-controller402 can also instruct each of these units to perform in accordance with a programming mode or an operational mode.
• According to the operational mode, the[0135]global controller110 loads a new frame from thevideo input processor106 to thememory controller400. Thememory controller400 transfers this data to the external storage. Thememory controller400 also reads the current and reference frames macro-blocks from the external storage and transfers them to the motion estimation processor108 (FIG. 3).
• The[0136]memory controller400 provides the control signals, addresses and the like to memory unit122 (FIG. 3).
• According to the operational mode, the[0137]exchange unit406 of theglobal controller110 reads and writes different data items to and from themotion estimation processor108, thedigital signal processor104, the bit-stream processor102 and thevideo input processor106.
• The[0138]exchange unit406 transfers the data to and from theregister file408. In order to support the simultaneous parallel processing of multiple macro-blocks indevice100,register file408 maintains a memory structure such as a stack, which contains the sets of parameters associated with each macro-block.
• This memory structure can be divided into multiple stacks of variable depth. The[0139]processing unit410 can read the data from theregister file408, perform various arithmetic and logical operations, and store the processed data back intoregister file408.
• The[0140]register file408 can access thehost storage404 to retrieve the data which an external host loaded into thehost storage404 during the programming mode.
• The[0141]register file408 can also transfer the data to thehost storage404, such that an external host can access the data during both programming and operational modes.
• Reference is now made to FIG. 12, which is a schematic illustration of[0142]encoding device100, incorporated in a video camera, generally referenced500, constructed and operative in accordance with another preferred embodiment of the invention.
• [0143]Camera500 includes anoptical assembly502, a charge coupled device (CCD)510, ahost interface504, arandom access memory506, acommunication interface520, twostorage units508 and516 andencoding device100.
• [0144]Device100 is connected to charge coupleddevice510,host interface504, acommunication interface520,random access memory506 andstorage units508 and516.
• [0145]Host interface504 includes adisplay514 and akeyboard512 and can be used to display the status ofencoding device100 as well as to receive instructions from a user.
• [0146]Storage unit508 is a tape based storage device.Storage unit516 is a disk based storage device, such as a magnetic hard drive, an optical storage device, a magneto-optical storage device and the like. It will be noted that other types of storage devices can also be used for this purpose, like semiconductor based memory units such as flash memory, RAM and the like.
• [0147]CCD510 converts light, arriving from theoptical assembly502, representing an image, into an electrical signal.CCD510 is preferably a digital light sensitive device which can be replaced by an analog light sensitive device, followed by an analog to digital converter, for converting an analog video signal into a digital video signal.
• Then,[0148]CCD510 provides the digital video signal tovideo input processor106, ofencoding device100. Theencoding device100 encodes the digital video signal, produces an encoded video signal and provides it at the output of bit-stream processor102. During the encoding process, theencoding device100 uses arandom access memory506 as a temporary storage area for video data as well as analysis data, produced by its inner components.
• [0149]Encoding device100 provides the encoded video signal tostorage devices508 and516, and tocommunication interface520.
• It will be appreciated that any of the[0150]storage units508 and516 as well as thecommunication interface520 can provide digital video signal as input fordevice100. In the present example,storage device516 is also connected todevice100 via thevideo input processor106.
• It will be noted that the present invention provides an encoding device which can easily be adapted to ISO/IEC 11172 (MPEG-1) as well as other encoding standards which are similar to MPEG such as ISO/IEC H.320, H.261 and H.263, as well as different motion JPEG methods.[0151]
• It will be appreciated by persons skilled in the art that the present invention is not limited to what has been particularly shown and described hereinabove. Rather the scope of the present invention is defined by the claims which follow.[0152]

Claims (10)

1-31. (Canceled)

32. A motion estimation device comprising:

a motion estimation processor;

a global controller; and

a plurality of resolution processors connected to the global controller, the plurality of resolution processors analyzing development of a digital video signal in time and performing motion analysis upon the digital video signal, wherein the global controller is adapted to control the plurality of resolution processors and interface with the motion estimation processor.

33. The motion estimation device according toclaim 32, wherein the plurality of resolution processors comprise at least one low resolution processor adapted to produce low resolution motion analysis.

34. The motion estimation device according toclaim 33, wherein the at least one low resolution processor is adapted to reduce resolution of a selected frame before producing low motion analysis.

35. The motion estimation device according toclaim 32, wherein the plurality of resolution processors comprise at least one full resolution processor adapted to produce full resolution motion analysis.

36. The motion estimation device according toclaim 32, wherein the plurality of resolution processors comprise at least one hyper resolution processor adapted to produce hyper resolution motion analysis.

37. The motion estimation device according toclaim 36, wherein the at least one hyper resolution processor is adapted to enhance resolution of a selected frame before producing hyper resolution motion analysis.

38. The motion estimation device according toclaim 32, wherein the plurality of resolution processors are at least adapted to acquire and store motion analysis information in at least one storage unit.

39. The motion estimation device according toclaim 32, wherein the plurality of resolution processors are each at least adapted to operate according to a same corresponding program command and a different corresponding program command.

40. The motion estimation device according toclaim 32, wherein the plurality of resolution processors are each at least adapted to operate up one of a same portion and different portions of information comprising a digital video signal.

Images