US20220335657A1 - Image Processing Device and Method - Google Patents

Abstract

The present invention relates to an image processing device and method enabling noise removal to be performed according to images and bit rates. A low-pass filter setting unit93 sets, from filter coefficients stored in a built-in filter coefficient memory94, a filter coefficient corresponding to intra prediction mode information and a quantization parameter. A neighboring image setting unit81 uses the filter coefficient set by the low-pass filter setting unit93 to subject neighboring pixel values of a current block from frame memory72 to filtering processing. A prediction image generating unit82 performs intra prediction using the neighboring pixel values subjected to filtering processing, from the neighboring image setting unit81, and generates a prediction image. The present invention can be applied to an image encoding device which encodes with the H.264/AVC format, for example.

Description

CROSS REFERENCE TO PRIOR APPLICATION
• This application is a continuation of U.S. application Ser. No. 16/805,115 filed Feb. 28, 2020, which is a continuation of U.S. application Ser. No. 15/707,509 filed Sep. 18, 2017, now U.S. Pat. No. 10,614,593, which is a continuation of U.S. application Ser. No. 14/483,325 filed Sep. 11, 2014, now U.S. Pat. No. 9,830,716, which is a continuation of U.S. application Ser. No. 13/380,478 filed Dec. 22, 2011, now U.S. Pat. No. 8,885,956, which is a National Stage patent application of Patent Cooperation Treaty Application No. PCT/JP2010/060605 filed Jun. 23, 2010 under 35 U.S.C. § 371, which claims priority to Japanese Patent Application Nos. 2009-244753 filed Oct. 23, 2009 and 2009-156563 filed on Jul. 1, 2009, which are all hereby incorporated by reference in their entirety.
TECHNICAL FIELD
• The present invention relates to an image processing device and method, and specifically relates to an image processing device and method which enable prediction efficiency to be improved.
BACKGROUND ART
• In recent years, there have come into widespread use devices which subject an image to compression encoding by employing an encoding format for handling image information as digital signals, and taking advantage of redundancy peculiar to the image information with transmission and storage of high effective information taken as an object at that time to compress the image by orthogonal transform such as discrete cosine transform or the like and motion compensation. Examples of this encoding method include MPEG (Moving Picture Experts Group) and so forth.
• In particular, MPEG2 (ISO/IEC 13818-2) is defined as a general-purpose image encoding format, and is a standard encompassing both of interlaced scanning images and sequential-scanning images, and standard resolution images and high definition images. For example, MPEG2 has widely been employed now by broad range of applications for professional usage and for consumer usage. By employing the MPEG2 compression format, a code amount (bit rate) of 4 through 8 Mbps is allocated in the event of an interlaced scanning image of standard resolution having 720×480 pixels, for example. Also, by employing the MPEG2 compression format, a code amount (bit rate) of 18 through 22 Mbps is allocated in the event of an interlaced scanning image of high resolution having 1920×1088 pixels, for example. Thus, a high compression rate and excellent image quality can be realized.
• With MPEG2, high image quality encoding adapted to broadcasting usage is principally taken as a object, but a lower code amount (bit rate) than the code amount of MPEG1, i.e., an encoding format having a higher compression rate is not handled. According to spread of personal digital assistants, it has been expected that needs for such an encoding format will be increased from now on, and in response to this, standardization of the MPEG4 encoding format has been performed. With regard to an image encoding format, the specification thereof was confirmed as international standard as ISO/IEC 14496-2 in December in 1998.
• Further, in recent years, standardization of a standard serving as H.26L (ITU-T Q6/16 VCEG) has progressed with image encoding, originally intended for television conference usage. With H.26L, it has been known that as compared to a conventional encoding format such as MPEG2 or MPEG4, though greater computation amount is requested for encoding and decoding thereof, higher encoding efficiency is realized. Also, currently, as part of activity of MPEG4, standardization for also taking advantage of a function that is not supported by H.26L with this H.26L taken as a base, to realize higher encoding efficiency, has been performed as Joint Model of Enhanced-Compression Video Coding. As a schedule of standardization, H.264 and MPEG-4 Part 10 (Advanced Video Coding, hereafter referred to as H.264/AVC) become an international standard in March, 2003.
• Further, as an extension thereof, standardization of FRExt (Fidelity Range Extension) including a coding tool necessary for business use such as RGB, 4:2:2, or 4:4:4, 8×8DCT and quantization matrix stipulated by MPEG-2 has been completed in February, 2005. Thus, H.264/AVC has become a encoding format capable of suitably expressing even film noise included in movies, and has been employed for wide ranging applications such as Blu-Ray Disc (registered trademark) and so forth.
• However, nowadays, needs for further high-compression encoding have been increased, such as intending to compress an image having around 4000×2000 pixels, which is quadruple of a high-vision image. Alternatively, needs for further high-compression encoding have been increased, such as intending to distribute a high-vision image within an environment with limited transmission capacity like the Internet. Therefore, with the above-mentioned VCEG (=Video Coding Expert Group) under the control of ITU-T, studies relating to improvement of encoding efficiency have continuously been performed.
• Now, one factor that can be given why the H.264/AVC format realizes high encoding efficiency as compared to the conventional MPEG2 format or the like is employing an intra prediction method.
• With the intra prediction method, the intra prediction modes of nine kinds of 4×4 pixel and 8×8 pixel block units, and four kinds of 16×16 pixel macro block units are determined regarding luminance signals. The intra prediction modes of four kinds of 8×8 pixel block units are determined regarding color difference signals. The intra prediction modes for color difference signals may be set independently from the intra prediction modes for luminance signals.
• There are particular patterns for each intra prediction mode regarding how residual following such intra prediction is manifested.
• As a method to eliminate such redundancy and further raise encoding efficiency, NPL 1 proposes the following method.
• That is to say, intra image encoding processing is performed by normal H.264/AVC format using training signals in offline processing beforehand, orthogonal transform such as Karhunen-Loeve transform or the like is performed for each intra prediction mode as to each block, and optimal transform coefficients are calculated.
• Then, in the actual encoding processing, processing using orthogonal transform coefficients optimized for each mode by the aforementioned Karhunen-Loeve transform are used instead of the orthogonal transform stipulated by the H.264/AVC format.
• Also, NPL 2 proposes a method of combining the aforementioned intra prediction with inter prediction.
• That is to say, withNPL 2, difference information is generated as to motion vector information obtained in inter prediction not only for a current block but also for neighboring pixel values around the current block. Performing intra prediction between the difference information relating to the current block, and the difference information relating to neighboring pixels, generated in this way, generates second order difference information. The generated second order difference information is then subjected to orthogonal transform and quantization, and output downstream along with a compressed image.
• Thus, encoding efficiency is further improved.
• Also, as described above, the macro block size is 16×16 pixels with the H.264/AVC format. However, a macro block size of 16×16 pixels is not optimal for large image frames such as UHD (Ultra High Definition; 4000×2000 pixels) which will be handled by next-generation encoding methods.
• Accordingly, NPL 3 and so forth propose enlarging the macro block size to a size of 32×32 pixels, for example.
CITATION LISTNon Patent Literature
• NPL 1: “Improved Intra Coding”, VCEG-AF15, ITU-Telecommunications Standardization Sector STUDY GROUPQuestion 6 Video coding Experts Group (VCEG), 20-21 Apr. 2007
• NPL 2: “Second Order Prediction (SOP) in P Slice”, Sijia Chen, Jinpeng Wang, Shangwen Li and, Lu Yu, VCEG-AD09, ITU-Telecommunications Standardization Sector STUDY GROUPQuestion 6 Video coding Experts Group (VCEG), 16-18 Jul. 2008
• NPL 3: “Video Coding Using Extended Block Sizes”, VCEG-AD09, ITU-Telecommunications Standardization Sector STUDY GROUPQuestion 16—Contribution 123, January 2009
SUMMARY OF INVENTIONTechnical Problem
• Now, with the H.264/AVC format, low-pass filter processing of pixel values of neighboring pixels is performed before performing intra prediction in increments of blocks of 8×8 pixels describe above. Thus, noise included in the neighboring pixels is removed, and correlation is raised, so higher encoding efficiency can be realized.
• However, regardless the fact that the degree of included noise differs according to the input images, quantization parameter values, intra prediction modes, and so forth, the low-pass filter for removing the noise has been fixed with the H.264/AVC format. That is to say, this low-pass filter has not been optimal in accordance with input images, quantization parameter values, intra prediction modes, and so forth.
• Also, with the H.264/AVC format, the noise removal of pixel values of neighboring pixels described above is only performed with the intra prediction mode in increments of blocks of 8×8 pixels, and has not been applied for other modes.
• The present invention has been made in light of this situation, and realizes noise removal in accordance with images and bit rates, thereby improving prediction efficiency.
Solution to Problem
• An image processing device according to a first aspect of the present invention includes: decoding means configured to decode an image of a current block which is to be the object of decoding processing; filter setting means configured to set, in accordance with the current block, a coefficient to be used for filtering processing to which neighboring pixels of the current block are to be subjected to, used for intra prediction of the current block, in accordance with an encoding parameter; and intra prediction means configured to subject the neighboring pixels to the filtering processing using the coefficient set by the filter setting means, and performing intra prediction of the current block.
• The encoding parameter may include a mode of intra prediction of the current block, or a quantization parameter of the current block; the decoding means may decode the mode of intra prediction of the current block or the quantization parameter of the current block; and the filter setting means may set the coefficient in accordance with the mode of intra prediction decoded by the decoding means, or the quantization parameter decoded by the decoding means.
• The image processing device may further include: filter coefficient storage means configured to store the coefficient; wherein the coefficient is calculated so as to obtain, with a learning image, the smallest residual between a learning block which is the object of encoding processing, and a prediction image obtained by performing intra prediction for the learning block corresponding to a mode of intra prediction of the learning block or a quantization parameter of the learning block, and is stored in the filter coefficient storage means; and wherein the filter setting means set, as the coefficient, that from the coefficients stored in the filter coefficient storage means which corresponds to the mode of intra prediction of the current block, or the quantization parameter of the current block.
• The filter coefficient storage means may hold the coefficient as an n-bit (wherein n is an integer) value in accordance with a register length of a processor.
• The decoding means may decode the coefficient, which has been calculated at an encoding side using the neighboring pixels prior to being subjected to the filtering processing such that the residual as to a prediction image obtained by intra prediction being performed regarding the current block is smallest, and which has been set corresponding to the current block, and to the mode of intra prediction of the current block or the quantization parameter of the current block; and the filter setting means may set, as the coefficient, that from the coefficients decoded by the decoding means which corresponds to the mode of intra prediction of the current block, or the quantization parameter of the current block.
• The coefficient may be configured of a filter coefficient and offset value.
• The filter coefficient may be configured of three taps.
• The filter coefficient may have symmetry centered on a coefficient corresponding to zero-phase.
• The decoding means may decode the mode of intra prediction of the current block; and the filter setting means may take, of modes of intra prediction, a vertical mode or horizontal mode as a first class, and other modes as a second class, and, in the event that the mode of intra prediction of the current block belongs to the first class, set the coefficient corresponding to the first class, and in the event that the mode of intra prediction of the current block which has been decoded belongs to the second class, set the coefficient corresponding to the second class.
• The image processing device may further include: receiving means configured to receive flag information indicating whether or not to perform the filtering processing as to the neighboring pixels; wherein the filter setting means set whether or not to perform the filtering processing as to the neighboring pixels, based on flag information received by the receiving means.
• The receiving means may receive the flag information in increments of macro blocks; and the filter setting means may set whether or not to perform the filter processing as to the neighboring pixels, in increments of macro blocks, based on flag information received by the receiving means.
• The receiving means may receive the flag information in increments of blocks; and the filter setting means may set whether or not to perform the filter processing as to the neighboring pixels, in increments of blocks, based on flag information received by the receiving means.
• An image processing method according to the first aspect of the present invention includes the steps of: an image processing device performing decoding of an image of a current block which is to be the object of decoding processing; setting of, in accordance with the current block, a coefficient to be used for filtering processing to which neighboring pixels of the current block are to be subjected to, used for intra prediction of the current block, in accordance with an encoding parameter; and subjecting the neighboring pixels to the filtering processing using the coefficient that has been set, and performing intra prediction of the current block.
• An image processing device according to a second aspect of the present invention includes: filter setting means configured to set a coefficient to be used for filtering processing to which neighboring pixels of a current block which is to be the object of decoding processing are to be subjected to, used for intra prediction of the current block, in accordance with an encoding parameter; intra prediction means configured to subject the neighboring pixels to the filtering processing using the coefficient set by the filter setting means, and performing intra prediction of the current block; and encoding means configured to encode an image of the current block.
• The encoding parameter may include a mode of intra prediction of the current block, or a quantization parameter of the current block; and the filter setting means may set the coefficient in accordance with the mode of intra prediction of the current block, or the quantization parameter of the current block; and the encoding means may encode the corresponding mode of intra prediction of the current block or quantization parameter of the current block.
• The image processing device may further include: filter coefficient storage means configured to store the coefficient; wherein the coefficient is calculated so as to obtain, with a learning image, the smallest residual between a learning block which is the object of encoding processing, and a prediction image obtained by performing intra prediction for the learning block corresponding to a mode of intra prediction of the learning block or a quantization parameter of the learning block, and is stored in the filter coefficient storage means; and wherein the filter setting means set, as the coefficient, that from the coefficients stored in the filter coefficient storage means which corresponds to the mode of intra prediction of the current block, or the quantization parameter of the current block.
• The image processing device may further include: filter coefficient calculating means configured to calculate the coefficient such that the residual between the current block, and a prediction image obtained by intra prediction being performed regarding the current block, using the neighboring pixels prior to being subjected to the filtering processing in accordance with the mode of intra prediction of the current block or the quantization parameter of the current block is smallest; wherein the filter setting means set, as the coefficient, that from the coefficients calculated by the filter coefficient calculating means which corresponds to the mode of intra prediction of the current block, or the quantization parameter of the current block; and wherein the encoding means further encode the coefficient.
• The coefficient may be configured of a filter coefficient and offset value.
• The filter setting means may set whether or not to perform the filtering processing as to the neighboring pixels; and the encoding means may encode flag information indicating whether or not to perform the filtering processing set by the filter setting means.
• An image processing method according to the second aspect of the present invention includes the steps of: an image processing device performing setting of a coefficient to be used for filtering processing to which neighboring pixels of a current block which is to be the object of decoding processing are to be subjected to, used for intra prediction of the current block, in accordance with an encoding parameter; subjecting the neighboring pixels to the filtering processing using the coefficient that has been set, and performing intra prediction of the current block; and encoding an image of the current block.
• With the first aspect of the present invention, an image of a current block which is to be the object of decoding processing is decoded, a coefficient is set, which is to be used for filtering processing to which neighboring pixels of the current block are to be subjected to which are used for intra prediction of the current block, in accordance with an encoding parameter. The neighboring pixels are then subjected to the filtering processing using the coefficient that has been set, and intra prediction of the current block is performed.
• With the second aspect of the present invention, an a coefficient is set, to be used for filtering processing to which neighboring pixels of a current block which is to be the object of decoding processing are to be subjected to which are used for intra prediction of the current block, in accordance with an encoding parameter, the neighboring pixels are subjected to the filtering processing using the coefficient that has been set, intra prediction of the current block is performed, and an image of the current block is encoded.
• Note that the above-described image processing devices may be stand-alone devices, or may be internal blocks making up one image encoding device or image decoding device.
Advantageous Effects of Invention
• According to the first invention, images can be decoded. Also, according to the second invention, noise removal can be performed in accordance with the image and the bit rate.
• According to the second invention, images can be encoded. Also, according to the first invention, noise removal can be performed in accordance with the image and the bit rate.
BRIEF DESCRIPTION OF DRAWINGS
• FIG. 1 is a block diagram illustrating the configuration of an embodiment of an image encoding device to which the present invention has been applied.
• FIG. 2 is a diagram for describing processing sequence in the event of a 16×16 pixel intra prediction mode.
• FIG. 3 is a diagram illustrating the kinds of 4×4 pixel intra prediction modes for luminance signals.
• FIG. 4 is a diagram illustrating the kinds of 4×4 pixel intra prediction modes for luminance signals.
• FIG. 5 is a diagram for describing the direction of 4×4 pixel intra prediction.
• FIG. 6 is a diagram for describing 4×4 pixel intra prediction.
• FIG. 7 is a diagram for describing encoding of the 4×4 pixel intra prediction modes for luminance signals.
• FIG. 8 is a diagram illustrating the kinds of 8×8 pixel intra prediction modes for luminance signals.
• FIG. 9 is a diagram illustrating the kinds of 8×8 pixel intra prediction modes for luminance signals.
• FIG. 10 is a diagram illustrating the kinds of 16×16 pixel intra prediction modes for luminance signals.
• FIG. 11 is a diagram illustrating the kinds of 16×16 pixel intra prediction modes for luminance signals.
• FIG. 12 is a diagram for describing 16×16 pixel intra prediction.
• FIG. 13 is a diagram illustrating the kinds of intra prediction modes for color difference signals.
• FIG. 14 is a block diagram illustrating a configuration example of an intra prediction unit and neighboring pixel interpolation filter switching unit inFIG. 1.
• FIG. 15 is a diagram for describing calculation of filter coefficients.
• FIG. 16 is a flowchart for describing the encoding processing of the image encoding device inFIG. 1.
• FIG. 17 is a flowchart for describing the prediction processing in step S21 inFIG. 16.
• FIG. 18 is a flowchart for describing the intra prediction processing in step S31 inFIG. 17.
• FIG. 19 is a flowchart for describing the inter motion prediction processing in step S32 inFIG. 17.
• FIG. 20 is a block diagram illustrating another configuration example of an intra prediction unit and neighboring pixel interpolation filter switching unit inFIG. 1.
• FIG. 21 is a flowchart for describing another example of the intra prediction processing in step S31 inFIG. 17.
• FIG. 22 is a block diagram illustrating the configuration of an embodiment of an image decoding device to which the present invention has been applied.
• FIG. 23 is a block diagram illustrating a configuration example of an intra prediction unit and neighboring pixel interpolation filter switching unit inFIG. 22.
• FIG. 24 is a flowchart for describing the decoding processing of the image decoding device inFIG. 22.
• FIG. 25 is a flowchart for describing the prediction processing in step S138 inFIG. 24.
• FIG. 26 is a block diagram illustrating another configuration example of an intra prediction unit and neighboring pixel interpolation filter switching unit inFIG. 22.
• FIG. 27 is a flowchart for describing another example of the prediction processing in step S138 inFIG. 24.
• FIG. 28 is a block diagram illustrating the configuration of an embodiment of a learning device to which the present invention has been applied.
• FIG. 29 is a block diagram illustrating a configuration example of an intra prediction unit and neighboring pixel interpolation filter calculating unit inFIG. 28.
• FIG. 30 is a flowchart for describing intra prediction processing with the learning device inFIG. 28.
• FIG. 31 is a block diagram illustrating the configuration of another embodiment of an image encoding device to which the present invention has been applied.
• FIG. 32 is a diagram for describing second order prediction processing.
• FIG. 33 is a block diagram illustrating the configuration of another embodiment of an image decoding device to which the present invention has been applied.
• FIG. 34 is a block diagram illustrating the configuration of yet another embodiment of an image encoding device to which the present invention has been applied.
• FIG. 35 is a block diagram illustrating a configuration example of an intra prediction unit inFIG. 34.
• FIG. 36 is a flowchart for describing another example of the intra prediction processing in step S31 inFIG. 17.
• FIG. 37 is a flowchart for describing yet another example of the intra prediction processing in step S31 inFIG. 17.
• FIG. 38 is a flowchart for describing another example of the intra prediction processing in step S31 inFIG. 17.
• FIG. 39 is a block diagram illustrating the configuration of yet another embodiment of an image decoding device to which the present invention has been applied.
• FIG. 40 is a block diagram illustrating a configuration example of an intra prediction unit and neighboring pixel interpolation filter control unit inFIG. 39.
• FIG. 41 is a flowchart for describing yet another example of the prediction processing in step S138 inFIG. 24.
• FIG. 42 is a block diagram illustrating the configuration of another embodiment of an image encoding device to which the present invention has been applied.
• FIG. 43 is a block diagram illustrating the configuration of another embodiment of an image decoding device to which the present invention has been applied.
• FIG. 44 is a diagram illustrating an example of an extended block size.
• FIG. 45 is a block diagram illustrating a configuration example of the hardware of a computer.
• FIG. 46 is a block diagram illustrating a principal configuration example of a television receiver to which the present invention has been applied.
• FIG. 47 is a block diagram illustrating a principal configuration example of a cellular phone to which the present invention has been applied.
• FIG. 48 is a block diagram illustrating a principal configuration example of a hard disk recorder to which the present invention has been applied.
• FIG. 49 is a block diagram illustrating a principal configuration example of a camera to which the present invention has been applied.
DESCRIPTION OF EMBODIMENTS
• Hereinafter, embodiments of the present invention will be described with reference to the drawings. Note that description will proceed in the following order.
• 1. First Embodiment (neighboring pixel interpolation filter switching: example of intra prediction)
• 2. Second Embodiment (neighboring pixel interpolation filter switching: example of second order prediction)
• 3. Third Embodiment (neighboring pixel interpolation filter on/off control: example of intra prediction)
• 4. Fourth Embodiment (neighboring pixel interpolation filter on/off control: example of second order prediction)
1. First Embodiment[Configuration Example of Image Encoding Device]
• FIG. 1 represents the configuration of an embodiment of an image encoding device serving as an image processing device to which the present invention has been applied.
• This image encoding device51 subjects an image to compression encoding using, for example, the H.264 and MPEG-4 Part 10 (Advanced Video Coding) (hereafter, described as H.264/AVC) format.
• With the example inFIG. 1, the image encoding device51 is configured of an A/D conversion unit61, ascreen rearranging buffer62, acomputing unit63, anorthogonal transform unit64, aquantization unit65, alossless encoding unit66, an storingbuffer67, aninverse quantization unit68, an inverseorthogonal transform unit69, acomputing unit70, adeblocking filter71,frame memory72, aswitch73, anintra prediction unit74, a neighboring pixel interpolationfilter switching unit75, a motion prediction/compensation unit76, a predictionimage selecting unit77, and arate control unit78.
• The A/D conversion unit61 converts an input image from analog to digital, and outputs to thescreen rearranging buffer62 for storing. Thescreen rearranging buffer62 rearranges the images of frames in the stored order for display into the order of frames for encoding according to GOP (Group of Picture).
• Thecomputing unit63 subtracts from the image read out from thescreen rearranging buffer62 the prediction image from theintra prediction unit74 selected by the predictionimage selecting unit77 or the prediction image from the motion prediction/compensation unit76, and outputs difference information thereof to theorthogonal transform unit64. Theorthogonal transform unit64 subjects the difference information from thecomputing unit63 to orthogonal transform, such as discrete cosine transform, Karhunen-Loeve transform, or the like, and outputs a transform coefficient thereof. Thequantization unit65 quantizes the transform coefficient that theorthogonal transform unit64 outputs.
• The quantized transform coefficient that is the output of thequantization unit65 is input to thelossless encoding unit66, where it is subjected to lossless encoding, such as variable length coding, arithmetic coding, or the like, and compressed.
• Thelossless encoding unit66 obtains information indicating intra prediction and so forth from theintra prediction unit74, and obtains information indicating an inter prediction mode, and so forth from the motion prediction/compensation unit76. Note that the information indicating intra prediction will also be referred to as intra prediction mode information hereinafter. Also, the information indicating inter prediction will also be referred to as inter prediction mode information hereinafter.
• Thelossless encoding unit66 encodes the quantized transform coefficient, and also encodes the information indicating intra prediction, the information indicating an inter prediction mode, quantization parameters, and so forth, and takes these as part of header information in the compressed image. Thelossless encoding unit66 supplies the encoded data to the storingbuffer67 for storage.
• For example, with thelossless encoding unit66, lossless encoding processing, such as variable length coding, arithmetic coding, or the like, is performed. Examples of the variable length coding include CAVLC (Context-Adaptive Variable Length Coding) determined by the H.264/AVC format. Examples of the arithmetic coding include CABAC (Context-Adaptive Binary Arithmetic Coding).
• The storingbuffer67 outputs the data supplied from thelossless encoding unit66 to, for example, a storage device or transmission path or the like downstream not shown in the drawing, as a compressed image encoded by the H.264/AVC format.
• Also, the quantized transform coefficient output from thequantization unit65 is also input to theinverse quantization unit68, subjected to inverse quantization, and then subjected to further inverse orthogonal transform at the inverseorthogonal transform unit69. The output subjected to inverse orthogonal transform is added to the prediction image supplied from the predictionimage selecting unit77 by thecomputing unit70, and changed into a locally decoded image. Thedeblocking filter71 removes block distortion from the decoded image, and then supplies to theframe memory72 for storage. An image before the deblocking filter processing is performed by thedeblocking filter71 is also supplied to theframe memory72 for storage.
• Theswitch73 outputs the reference images stored in theframe memory72 to the motion prediction/compensation unit76 orintra prediction unit74.
• With this image encoding device51, the I picture, B picture, and P picture from thescreen rearranging buffer62 are supplied to theintra prediction unit74 as an image to be subjected to intra prediction (also referred to as intra processing), for example. Also, the B picture and P picture read out from thescreen rearranging buffer62 are supplied to the motion prediction/compensation unit76 as an image to be subjected to inter prediction (also referred to as inter processing).
• Theintra prediction unit74 performs intra prediction processing of all of the candidate intra prediction modes based on the image to be subjected to intra prediction read out from thescreen rearranging buffer62, and the reference image supplied from theframe memory72 to generate a prediction image.
• Prior to the intra prediction processing, theintra prediction unit74 performs filtering professing on neighboring pixels which are pixels used for intra prediction of each current block, and neighboring the current block with a predetermined positional relation. This filtering processing uses a filter coefficient set by a neighboring pixel interpolationfilter switching unit75, in accordance with the intra prediction mode supplied from theintra prediction unit74, and so forth. That is to say, theintra prediction unit74 uses, for intra prediction processing of all candidate intra prediction modes, neighboring pixels subjected to filtering processing with filter coefficients set by the neighboring pixel interpolationfilter switching unit75.
• Theintra prediction unit74 calculates a cost function value as to the intra prediction mode where the prediction image has been generated, and selects the intra prediction mode where the calculated cost function value gives the minimum value, as the optimal intra prediction mode. Theintra prediction unit74 supplies the prediction image generated in the optimal intra prediction mode, and the cost function value calculated regarding the corresponding optimal intra prediction mode, to the predictionimage selecting unit77.
• In the event that the prediction image generated in the optimal intra prediction mode has been selected by the predictionimage selecting unit77, theintra prediction unit74 supplies information indicating the optimal intra prediction mode to thelossless encoding unit66. In the event that the information has been transmitted from theintra prediction unit74, thelossless encoding unit66 encodes this information, and takes this as part of the header information in the compressed image.
• The neighboring pixel interpolationfilter switching unit75 stores the filter coefficients corresponding to quantization parameters and intra prediction modes obtained by performing learning at alearning device251 inFIG. 28 which will be described later, using a training image.
• The neighboring pixel interpolationfilter switching unit75 is supplied with quantization parameters from therate control unit78 and intra prediction mode information from theintra prediction unit74. The neighboring pixel interpolationfilter switching unit75 sets filter coefficients corresponding to the quantization parameters from therate control unit78 and the intra prediction mode from theintra prediction unit74. The neighboring pixel interpolationfilter switching unit75 supplies the set filter coefficients to theintra prediction unit74.
• Note that the neighboring pixel interpolationfilter switching unit75 may perform learning and storing of filter coefficients corresponding to just one, rather than both, of quantization parameters and intra prediction modes.
• Also, while the neighboring pixel interpolationfilter switching unit75 stores filter coefficients learned offline beforehand, filter coefficients may be calculated online instead. In this case, filter coefficients set by the neighboring pixel interpolationfilter switching unit75 are output to thelossless encoding unit66 to be sent to the decoding side, as indicated by the dotted arrow.
• The motion prediction/compensation unit76 performs motion prediction and compensation processing regarding all of the candidate inter prediction modes. Specifically, the motion prediction/compensation unit76 is supplied with the image to be subjected to inter processing read out from thescreen rearranging buffer62, and the reference image from theframe memory72 via theswitch73. The motion prediction/compensation unit76 detects the motion vectors of all of the candidate inter prediction modes based on the image to be subjected to inter processing and the reference image, subjects the reference image to compensation processing based on the motion vectors, and generates a prediction image.
• Also, the motion prediction/compensation unit76 calculates a cost function value as to all of the candidate inter prediction modes. The motion prediction/compensation unit76 determines, of the calculated cost function values, the prediction mode that provides the minimum value to be the optimal inter prediction mode.
• The motion prediction/compensation unit76 supplies the prediction image generated in the optimal inter prediction mode, and the cost function value thereof to the predictionimage selecting unit77. In the event that the prediction image generated in the optimal inter prediction mode by the predictionimage selecting unit77 has been selected, the motion prediction/compensation unit76 outputs information indicating the optimal inter prediction mode (inter prediction mode information) to thelossless encoding unit66.
• Note that the motion vector information, flag information, reference frame information, and so forth are output to thelossless encoding unit66 according to need. Thelossless encoding unit66 also subjects the information from the motion prediction/compensation unit76 to lossless encoding processing such as variable length coding, arithmetic coding, or the like, and inserts into the header portion of the compressed image.
• The predictionimage selecting unit77 determines the optimal prediction mode from the optimal intra prediction mode and the optimal inter prediction mode based on the cost function values output from theintra prediction unit74 or motion prediction/compensation unit76. The predictionimage selecting unit77 then selects the prediction image in the determined optimal prediction mode, and supplies to thecomputing units63 and70. At this time, the predictionimage selecting unit77 supplies the selection information of the prediction image to theintra prediction unit74 or motion prediction/compensation unit76.
• Therate control unit78 controls the rate of the quantization operation of thequantization unit65 with quantization parameters, based on a compressed image stored in the storingbuffer67, so as not to cause overflow or underflow.
• The quantization parameter used for rate control at thequantization unit65 is supplied to thelossless encoding unit66, subjected to lossless encoding processing, and inserted to the header portion of the compressed image. This quantization parameter is supplied to the neighboring pixel interpolationfilter switching unit75, and used for setting filter coefficients used for filter processing to be applied to neighboring pixels.
[Description of Intra Prediction Processing According to H.264/AVC Format]
• First, the intra prediction modes determined by the H.264/AVC format will be described.
• First, the intra prediction modes as to luminance signals will be described. With the intra prediction modes for luminance signals, three systems of anintra 4×4 prediction mode, anintra 8×8 prediction mode, and an intra 16×16 prediction mode are determined. These are modes for determining block units, and are set for each macro block. Also, an intra prediction mode may be set to color difference signals independently from luminance signals for each macro block.
• Further, in the event of theintra 4×4 prediction mode, one prediction mode can be set out of the nine kinds of prediction modes for each 4×4 pixel current block. In the event of theintra 8×8 prediction mode, one prediction mode can be set out of the nine kinds of prediction modes for each 8×8 pixel current block. Also, in the event of the intra 16×16 prediction mode, one prediction mode can be set to a 16×16 pixel current macro block out of the four kinds of prediction modes.
• Note that, hereafter, theintra 4×4 prediction mode, intra 8×8 prediction mode, and intra 16×16 prediction mode will also be referred to as 4×4 pixel intra prediction mode, 8×8 pixel intra prediction mode, and 16×16 pixel intra prediction mode as appropriate, respectively.
• With the example inFIG. 2, numerals −1 through 25 appended to the blocks represent the bit stream sequence (processing sequence on the decoding side) of the blocks thereof. Note that, with regard to luminance signals, a macro block is divided into 4×4 pixels, and DCT of 4×4 pixels is performed. Only in the event of the intra 16×16 prediction mode, as shown in a block of −1, the DC components of the blocks are collected, a 4×4 matrix is generated, and this is further subjected to orthogonal transform.
• On the other hand, with regard to color difference signals, after a macro block is divided into 4×4 pixels, and DCT of 4×4 pixels is performed, as shown in theblocks 16 and 17, the DC components of the blocks are collected, a 2×2 matrix is generated, and this is further subjected to orthogonal transform.
• Note that, with regard to theintra 8×8 prediction mode, this may be applied to only a case where the current macro block is subjected to 8×8 orthogonal transform with a high profile or a profile beyond this.
• FIG. 3 andFIG. 4 are diagrams showing nine kinds of 4×4 pixel intra prediction modes (Intra_4×4_pred_mode) for luminance signals. The eight kinds of modes other than themode 2 showing average value (DC) prediction correspond to directions indicated withnumbers 0, 1, 3 through 8 inFIG. 5, respectively.
• The nine kinds of intra_4×4_pred_mode will be described with reference toFIG. 6. With the example inFIG. 6, pixels a through p represent the pixels of the current block to be subjected to intra processing, and pixel values A through M represent the pixel values of pixels belonging to a neighboring block. Specifically, the pixels a through p are an image to be processed read out from thescreen rearranging buffer62, and the pixel values A through M are the pixel values of a decoded image to be read out from theframe memory72 and referenced.
• In the case of the intra prediction modes shown inFIG. 3 andFIG. 4, the prediction pixel values of the pixels a through p are generated as follows using the pixel values A through M of the pixels belonging to a neighboring block. Here, that a pixel value is “available” represents that the pixel value is available without a reason such that the pixel is positioned in the edge of the image frame, or has not been encoded yet. On the other hand, that a pixel value is “unavailable” represents that the pixel value is unavailable due to a reason such that the pixel is positioned in the edge of the image frame, or has not been encoded yet.
• Themode 0 is a Vertical Prediction mode, and is applied to only a case where the pixel values A through D are “available”. In this case, the prediction pixel values of the pixels a through p are generated as with the following Expression (1).
• 
  Prediction pixel values of pixelsa,e,i, andm=A
• 
  Prediction pixel values of pixelsb,f,j, andn=B
• 
  Prediction pixel values of pixelsc,g,k, ando=C
• 
  Prediction pixel values of pixelsd,h,l, andp=D  (1)
• Themode 1 is a Horizontal Prediction mode, and is applied to only a case where the pixel values I through L are “available”. In this case, the prediction pixel values of the pixels a through p are generated as with the following Expression (2).
• 
  Prediction pixel values of pixelsa,b,c, andd=I
• 
  Prediction pixel values of pixelse,f,g, andh=J
• 
  Prediction pixel values of pixelsi,j,k, andl=K
• 
  Prediction pixel values of pixelsm,n,o, andp=L  (2)
• Themode 2 is a DC Prediction mode, and the prediction pixel value is generated like Expression (3) when the pixel values A, B, C, D, I, J, K, and L are all “available”.
• 
  (A+B+C+D+I+J+K+L+4)>>3  (3)
• Also, when the pixel values A, B, C, and D are all “unavailable”, the prediction pixel value is generated like Expression (4).
• 
  (I+J+K+L+2)>>2  (4)
• Also, when the pixel values I, J, K, and L are all “unavailable”, the prediction pixel value is generated like Expression (5).
• 
  (A+B+C+D+2)>>2  (5)
• Note that, when the pixel values A, B, C, D, I, J, K, and L are all “unavailable”, 128 is employed as the prediction pixel value.
• Themode 3 is a Diagonal_Down_Left Prediction mode, and is applied to only a case where the pixel values A, B, C, D, I, J, K, L, and M are “available”. In this case, the prediction pixel values of the pixels a through p are generated as with the following Expression (6).
• 
  Prediction pixel value of pixela=(A+2B+C+2)>>2
• 
  Prediction pixel values of pixelsbande=(B+2C+D+2)>>2
• 
  Prediction pixel values of pixelsc,f, andi=(C+2D+E+2)>>2
• 
  Prediction pixel values of pixelsd,g,j, andm=(D+2E+F+2)>>2
• 
  Prediction pixel values of pixelsh,k, andn=(E+2F+G+2)>>2
• 
  Prediction pixel values of pixelslando=(F+2G+H+2)>>2
• 
  Prediction pixel value of pixelp=(G+3H+2)>>2  (6)
• Themode 4 is a Diagonal_Down_Right Prediction mode, and is applied to only a case where the pixel values A, B, C, D, I, J, K, L, and M are “available”. In this case, the prediction pixel values of the pixels a through p are generated as with the following Expression (7).
• 
  Prediction pixel value of pixelm=(J+2K+L+2)>>2
• 
  Prediction pixel values of pixelsiandn=(I+2J+K+2)>>2
• 
  Prediction pixel values of pixelse,j, ando=(M+2I+J+2)>>2
• 
  Prediction pixel values of pixelsa,f,k, andp=(A+2M+I+2)>>2
• 
  Prediction pixel values of pixelsb,g, andl=(M+2A+B+2)>>2
• 
  Prediction pixel values of pixelscandh=(A+2B+C+2)>>2
• 
  Prediction pixel value of pixeld=(B+2C+D+2)>>2  (7)
• Themode 5 is a Diagonal_Vertical_Right Prediction mode, and is applied to only a case where the pixel values A, B, C, D, I, J, K, L, and M are “available”. In this case, the prediction pixel values of the pixels a through p are generated as with the following Expression (8).
• 
  Prediction pixel values of pixelsaandj=(M+A+1)>>1
• 
  Prediction pixel values of pixelsbandk=(A+B+1)>>1
• 
  Prediction pixel values of pixelscandl=(B+C+1)>>1
• 
  Prediction pixel value of pixeld=(C+D+1)>>1
• 
  Prediction pixel values of pixelseandn=(I+2M+A+2)>>2
• 
  Prediction pixel values of pixelsfando=(M+2A+B+2)>>2
• 
  Prediction pixel values of pixelsgandp=(A+2B+C+2)>>2
• 
  Prediction pixel value of pixelh=(B+2C+D+2)>>2
• 
  Prediction pixel value of pixeli=(M+2I+J+2)>>2
• 
  Prediction pixel value of pixelm=(I+2J+K+2)>>2  (8)
• Themode 6 is a Horizontal_Down Prediction mode, and is applied to only a case where the pixel values A, B, C, D, I, J, K, L, and M are “available”. In this case, the prediction pixel values of the pixels a through p are generated as with the following Expression (9).
• 
  Prediction pixel values of pixelsaandg=(M+I+1)>>1
• 
  Prediction pixel values of pixelsbandh=(I+2M+A+2)>>2
• 
  Prediction pixel value of pixelc=(M+2A+B+2)>>2
• 
  Prediction pixel value of pixeld=(A+2B+C+2)>>2
• 
  Prediction pixel values of pixelseandk=(I+J+1)>>1
• 
  Prediction pixel values of pixelsfandl=(M+2I+J+2)>>2
• 
  Prediction pixel values of pixelsiando=(J+K+1)>>1
• 
  Prediction pixel values of pixelsjandp=(I+2J+K+2)>>2
• 
  Prediction pixel value of pixelm=(K+L+1)>>1
• 
  Prediction pixel value of pixeln=(J+2K+L+2)>>2  (9)
• Themode 7 is a Vertical_Left Prediction mode, and is applied to only a case where the pixel values A, B, C, D, I, J, K, L, and M are “available”. In this case, the prediction pixel values of the pixels a through p are generated as with the following Expression (10).
• 
  Prediction pixel value of pixela=(A+B+1)>>1
• 
  Prediction pixel values of pixelsbandi=(B+C+1)>>1
• 
  Prediction pixel values of pixelscandj=(C+D+1)>>1
• 
  Prediction pixel values of pixelsdandk=(D+E+1)>>1
• 
  Prediction pixel value of pixell=(E+F+1)>>1
• 
  Prediction pixel value of pixele=(A+2B+C+2)>>2
• 
  Prediction pixel values of pixelsfandm=(B+2C+D+2)>>2
• 
  Prediction pixel values of pixelsgandn=(C+2D+E+2)>>2
• 
  Prediction pixel values of pixelshando=(D+2E+F+2)>>2
• 
  Prediction pixel value of pixelp=(E+2F+G+2)>>2  (10)
• Themode 8 is a Horizontal_Up Prediction mode, and is applied to only a case where the pixel values A, B, C, D, I, J, K, L, and M are “available”. In this case, the prediction pixel values of the pixels a through p are generated as with the following Expression (11).
• 
  Prediction pixel value of pixela=(I+J+1)>>1
• 
  Prediction pixel value of pixelb=(I+2J+K+2)>>2
• 
  Prediction pixel values of pixelscande=(J+K+1)>>1
• 
  Prediction pixel values of pixelsdandf=(J+2K+L+2)>>2
• 
  Prediction pixel values of pixelsgandi=(K+L+1)>>1
• 
  Prediction pixel values of pixelshandj=(K+3L+2)>>2
• 
  Prediction pixel values of pixelsk,l,m,n,o, andp=L  (11)
• Next, the encoding format of the 4×4 pixel intra prediction mode (Intra_4×4 pred_mode) for luminance signals will be described with reference toFIG. 7. With the example inFIG. 7, an current block C serving as an encoding object, which is made up of 4×4 pixels, is shown, and a block A and a block B, which neighbor the current block C and are made up of 4×4 pixels, are shown.
• In this case, it can be conceived that the Intra_4×4_pred_mode in the current block C, and the Intra_4×4_pred_mode in the block A and block B have high correlation. Encoding processing is performed as follows using this correlation, whereby higher encoding efficiency can be realized.
• Specifically, with the example inFIG. 7, the Intra_4×4_pred_mode in the block A and block B are taken as Intra_4×4_pred_modeA and Intra_4×4_pred_modeB respectively, and MostProbableMode is defined as the following Expression (12).
• 
  MostProbableMode=Min(Intra_4×4_pred_modeA,Intra_4×4_pred_modeB)  (12)
• That is to say, of the block A and block B, one to which a smaller mode number is assigned is taken as MostProbableMode.
• Two values called prev_intra4×4_pred_mode_flag[luma4×4BlkIdx] and rem_intra4×4_pred_mode[luma4×4BlkIdx] are defined within a bit stream as parameters as to the current block C, and decoding processing is performed by processing based on the pseudo-code shown in the following Expression (13), whereby the values of Intra_4×4_pred_mode and Intra4×4PredMode[luma4×4BlkIdx] as to the block C can be obtained.
• 
  If(prev_intra4×4_pred_mode_flag[luma4×4BlkIdx])
• 
  Intra4×4PredMode[luma4×4BlkIdx]=MostProbableMode
• 
  else
• 
  if(rem_intra4×4_pred_mode[luma4×4BlkIdx]<MostProbableMode)
• 
  Intra4×4PredMode[luma4×4BlkIdx]=rem_intra4×4_pred_mode[luma4×4BlkIdx]
• 
  else
• 
  Intra4×4PredMode[luma4×4BlkIdx]=rem_intra4×4_pred_mode[luma4×4BlkIdx]+1  (13)
• Next, the 8×8 pixel intra prediction mode will be described.FIG. 8 andFIG. 9 are diagrams showing the nine kinds of 8×8 pixel intra prediction modes (intra_8×8_pred_mode) for luminance signals.
• Let us say that the pixel values in the current 8×8 block are taken as p[x, y](0.ltoreq.x.ltoreq.7; 0.ltoreq.y.ltoreq.7), and the pixel values of a neighboring block are represented like p[−1, −1], . . . , p[−1, 15], p[−1, 0], . . . , [p−1, 7].
• With regard to the 8×8 pixel intra prediction modes, neighboring pixels are subjected to low-pass filtering processing prior to generating a prediction value. Now, let us say that pixel values before low-pass filtering processing are represented with p[−1, −1], . . . , p[−1, 15], p[−1, 0], . . . , p[−1, 7], and pixel values after the processing are represented with p′[−1, −1], . . . , p′[−1, 15], p′[−1, 0], . . . , p′[−1, 7].
• First, p′[0, −1] is calculated as with the following Expression (14) in the event that p[−1, −1] is “available”, and calculated as with the following Expression (15) in the event of “not available”.
• 
  p′[0,−1]=(p[−1,−1]+2*p[0,−1]+p[1,−1]+2)>>2  (14)
• 
  p′[0,−1]=(3*p[0,−1]+p[1,−1]+2)>>2  (15)
• p′[x, −1] (x=0, . . . , 7) is calculated as with the following Expression (16).
• 
  p′(x,−1)=(p[x−1,−1)+2*p[x,−1]+p(x+1,−1]+2)>>2  (16)
• p′[x, −1] (x=8, . . . , 15) is calculated as with the following Expression (17) in the event that p[x, −1] (x=8, . . . , 15) is “available”.
• 
  p′[x,−1]=(p[x−1,−1]+2*p[x,−1]+p[x+1,−1]+2)>>2
• 
  p′[15,−1]=(p[14,−1]+3*p[15,−1]+2)>>2  (17)
• p′[−1, −1] is calculated as follows in the event that p[−1, −1] is “available”. Specifically, p′[−1, −1] is calculated like Expression (18) in the event that both of p[0, −1] and p[−1, 0] are “available”, and calculated like Expression (19) in the event that p[−1, 0] is “unavailable”. Also, p′[−1, −1] is calculated like Expression (20) in the event that p[0, −1] is “unavailable”.
• 
  p′[−1,−1]=(p[0,−1]+2*p[−1,−1]+p[−1,0]+2)>>2  (18)
• 
  p′[−1,−1]=(3*p[−1,−1]+p[0,−1]+2)>>2  (19)
• 
  p′[−1,−1]=(3*p[−1,−1]+p[−1,0]+2)>>2  (20)
• p′[−1, y] (y=0, . . . , 7) is calculated as follows when p[−1, y] (y=0, . . . , 7) is “available”. Specifically, first, in the event that p[−1, −1] is “available”, p′[−1, 0] is calculated as with the following Expression (21), and in the event of “unavailable”, calculated like Expression (22).
• 
  p′[−1,0]=(p[−1,−1]+2*p[−1,0]+p[−1,1]+2)>>2  (21)
• 
  p′[−1,0]=(3*p[−1,0]+p[−1,1]+2)>>2  (22)
• Also, p′[−1, y] (y=1, . . . , 6) is calculated as with the following Expression (23), and p′[−1, 7] is calculated like Expression (24).
• 
  p[−1,y]=(p[−1,y−1]+2*p[−1,y]+p[−1,y+1]+2)>>2  (23)
• 
  p′[−1,7]=(p[−1,6]+3*p[−1,7]+2)>>2  (24)
• Prediction values in the intra prediction modes shown inFIG. 8 andFIG. 9 are generated as follows using p′ thus calculated.
• Themode 0 is a Vertical Prediction mode, and is applied only when p[x, −1] (x=0, . . . , 7) is “available”. A prediction value pred8×8_L[x, y] is generated as with the following Expression (25).
• 
  pred8×8_L[x,y]=p′[x,−1]x,y=0, . . . ,7  (25)
• Themode 1 is a Horizontal Prediction mode, and is applied only when p[−1, y] (y=0, . . . , 7) is “available”. The prediction value pred8×8_L[x, y] is generated as with the following Expression (26).
• 
  pred8×8_L[x,y]=p′[−1,y]x,y=0, . . . ,7  (26)
• Themode 2 is a DC Prediction mode, and the prediction value pred8×8_L[x, y] is generated as follows. Specifically, in the event that both of p[x, −1] (x=0, . . . , 7) and p[−1, y] (y=0, . . . , 7) are “available”, the prediction value pred8×8_L[x, y] is generated as with the following Expression (27).
• $\begin{array}{cc}\left[\mathrm{Mathematical}\mathrm{Expression}1\right]& \\ \mathrm{Pred}8\times 8_L\left[x,y\right]=\left(\text{?}{P}^{\prime }\left[\text{?}\right]+\text{?}{P}^{\prime }\left[-1,y\right]+8\right)>>4& \left(27\right)\end{array}$$\text{?}\text{indicates text missing or illegible when filed}$
• In the event that p[x, −1] (x=0, . . . , 7) is “available”, but p[−1, y] (y=0, . . . , 7) is “unavailable”, the prediction value pred8×8_L[x, y] is generated as with the following Expression (28).
• $\begin{array}{cc}\left[\mathrm{Mathematical}\mathrm{Expression}2\right]& \\ \mathrm{Pred}8\times 8_L\left[x,y\right]=\left(\text{?}{P}^{\prime }\left[\text{?}\right]+4\right)>>3& \left(28\right)\end{array}$$\text{?}\text{indicates text missing or illegible when filed}$
• In the event that p[x, −1] (x=0, . . . , 7) is “unavailable”, but p[−1, y] (y=0, . . . , 7) is “available”, the prediction value pred8×8_L[x, y] is generated as with the following Expression (29).
• $\left[\mathrm{Mathe}\mathrm{matical}\mathrm{Expression}4\right]\begin{array}{cc}\mathrm{Pred}8\times 8_L\left[x,y\right]=\left(\underset{\text{?}}{\sum ^7}{P}^{\prime }\left[-1,y\right]+4\right)\gg 3& \left(29\right)\end{array}\text{?}\text{indicates text missing or illegible when filed}$
• In the event that both of p[x, −1] (x=0, . . . , 7) and p[−1, y] (y=0, . . . , 7) are “unavailable”, the prediction value pred8×8_L[x, y] is generated as with the following Expression (30).
• 
  pred8×8_L[x,y]=128  (30)
• Here, Expression (30) represents a case of 8-bit input.
• Themode 3 is a Diagonal_Down_Left_prediction mode, and the prediction value pred8×8_L[x, y] is generated as follows. Specifically, the Diagonal_Down_Left_prediction mode is applied only when p[x, −1], x=0, . . . , 15, is “available”, and the prediction pixel value with x=7 and y=7 is generated as with the following Expression (31), and other prediction pixel values are generated as with the following Expression (32).
• 
  pred8×8_L[x,y]=(p′[14,−1]+3*p[15,−1]+2)>>2  (31)
• 
  red8×8_L[x,y]=(p′[x+y,−1]+2*p′[x+y+1,−1]+p′[x+y+2,−1]+2)>>2  (32)
• Themode 4 is a Diagnonal_Down_Right_prediction mode, and the prediction value pred8×8_L[x, y] is generated as follows. Specifically, the Diagnonal_Down_Right_prediction mode is applied only when p[x, −1], x=0, . . . , 7 and p[−1, y], y=0, . . . , 7 are “available”, the prediction pixel value with x>y is generated as with the following Expression (33), and the prediction pixel value with x<y is generated as with the following Expression (34). Also, the prediction pixel value with x=y is generated as with the following Expression (35).
• 
  pred8×8_L[x,y]=(p′[x−y−2,−1]+2*p′[x−y−1,−1]+p′[x−y,−1]+2)>>2  (33)
• 
  pred8×8_L[x,y]=(p′[−1,y−x−2]+2*p′[−1,y−x−1]+p′[−1,y−x]+2)>>2  (34)
• 
  pred8×8_L[x,y]=(p′[0,−1]+2*p′[−1,−1]+p′[−1,0]+2)>>2  (35)
• Themode 5 is a Vertical_Right_prediction mode, and the prediction value pred8×8_L[x, y] is generated as follows. Specifically, the Vertical_Right_prediction mode is applied only when p[x, −1], x=0, . . . , 7 and p[−1, y], y=−1, . . . , 7 are “available”. Now, zVR is defined as with the following Expression (36).
• 
  zVR=2*x−y  (36)
• At this time, in the event that zVR is 0, 2, 4, 6, 8, 10, 12, or 14, the pixel prediction value is generated as with the following Expression (37), and in the event that zVR is 1, 3, 5, 7, 9, 11, or 13, the pixel prediction value is generated as with the following Expression (38).
• 
  pred8×8_L[x,y]=(p′[x−(y>>1)−1,−1]+p′[x−(y>>1),−1]+−1)>>1  (37)
• 
  pred8×8_L[x,y]=(p′[x−(y>>1)−2,−1]1+2*p′[x−(y>>1)−1,−1]+p′[x−(y>>1),−1]+2)>>2  (38)
• Also, in the event that zVR is −1, the pixel prediction value is generated as with the following Expression (39), and in the cases other than this, specifically, in the event that zVR is −2, −3, −4, −5, −6, or −7, the pixel prediction value is generated as with the following Expression (40).
• 
  pred8×8_L[x,y]=(p′[−1,0]+2*p′[−1,−1]+p′[0,−1]+2)>>2  (39)
• 
  pred8×8_L[x,y]=(p′[−1,y−2*x−1]+2*p′[−1,y−2*x−2]+p′[−1,y−2*x−3]+2)>>2  (40)
• Themode 6 is a Horizontal_Down prediction mode, and the prediction value pred8×8_L[x, y] is generated as follows. Specifically, the Horizontal_Down prediction mode is applied only when p[x, −1], x=0, . . . , 7 and p[−1, y], y=−1, . . . , 7 are “available”. Now, zVR is defined as with the following Expression (41).
• 
  zHD=2*y−x  (41)
• At this time, in the event that zHD is 0, 2, 4, 6, 8, 10, 12, or 14, the prediction pixel value is generated as with the following Expression (42), and in the event that zHD is 1, 3, 5, 7, 9, 11, or 13, the prediction pixel value is generated as with the following Expression (43).
• 
  pred8×8_L[x,y]=(p′[−1,y−(x>>1)−1]+p′[−1,y−(x>>1)+1]>>1  (42)
• 
  pred8×8_L[x,y]=(p′[−1,y−(x>>1)−2]+2*p′[−1,y−(x>>1)−1]+p′[−1,y−(x>>1)]+2)>>2  (43)
• Also, in the event that zHD is −1, the prediction pixel value is generated as with the following Expression (44), and in the event that zHD is other than this, specifically, in the event that zHD is −2, −3, −4, −5, −6, or −7, the prediction pixel value is generated as with the following Expression (45).
• 
  pred8×8_L[x,y]=(p′[−1,0]+2*p′[−1,−1]+p′([0,−1]+2)>>2  (44)
• 
  pred8×8_L[x,y]=(p′[x−2*Y−1,−1]+2*p′[x−2*y−2,−1]+p′[x−2*y−3,−1]+2)>>2  (45)
• Themode 7 is a Vertical_Left_prediction mode, and the prediction value pred8×8_L[x, y] is generated as follows. Specifically, the Vertical_Left_prediction mode is applied only when p[x, −1], x=0, . . . , 15, is “available”, in the case that y=0, 2, 4, or 6, the prediction pixel value is generated as with the following Expression (46), and in the cases other than this, i.e., in the case that y=1, 3, 5, or 7, the prediction pixel value is generated as with the following Expression (47).
• 
  pred8×8_L[x,y]=(p′[x+(y>>1),−1]+p′[x(y>>1)+1,1]+1)>>1  (46)
• 
  pred8×8_L[x,y]=(p′[x+(y>>1),−1]+2*p′[x+(y>>1)+1,−1]+p′[x+(y>>1)+2,−1]+2)>>2  (47)
• Themode 8 is a Horizontal_Up_prediction mode, and the prediction value pred8×8_L[x, y] is generated as follows. Specifically, the Horizontal_Up_prediction mode is applied only when p[−1, y], y=0, . . . , 7, is “available”. Hereafter, zHU is defined as with the following Expression . . . (48).
• 
  zHU=x+2*y  (48)
• In the event that the value of zHU is 0, 2, 4, 6, 8, 10, or 12, the prediction pixel value is generated as with the following Expression (49), and in the event that the value of zHU is 1, 3, 5, 7, 9, or 11, the prediction pixel value is generated as with the following Expression (50).
• 
  pred8×8_L[x,y]=(p′[−1,y+(x>>1)]+p′[−1,y+(x>>1)+1]+1)>>1  (49)
• 
  pred8×8_L[x,y]=(p′[−1,y+(x>>1)]  (50)
• Also, in the event that the value of zHU is 13, the prediction pixel value is generated as with the following Expression (51), and in the cases other than this, i.e., in the event that the value of zHU is greater than 13, the prediction pixel value is generated as with the following Expression (52).
• 
  pred8×8_L[x,y]=(p′[−1,6]+3*p′[−1,7]+2>>2  (51)
• 
  pred8×8_L[x,y]=p′[−1,7]  (52)
• Next, the 16×16 pixel intra prediction mode will be described.FIG. 10 andFIG. 11 are diagrams showing the four kinds of the 16×16 pixel intra prediction modes for luminance signals (Intra_16×16_pred_mode).
• The four types of intra prediction modes will be described with reference toFIG. 12. With the example inFIG. 12, an current macro block A to be subjected to intra processing is shown, and P(x, y); x, y=−1, 0, . . . , 15 represents the pixel value of a pixel neighboring the current macro block A.
• Themode 0 is a Vertical Prediction mode, and is applied only when P(x, −1); x, y=−1, 0, . . . , 15 is “available”. In this case, the prediction pixel value Pred(x, y) of each pixel of the current macro block A is generated as with the following Expression (53).
• 
  Pred(x,y)=P(x,−1);x,y=0, . . . ,15  (53)
• Themode 1 is a Horizontal Prediction mode, and is applied only when P(−1, y); x, y=−1, 0, . . . , 15 is “available”. In this case, the prediction pixel value Pred(x, y) of each pixel of the current macro block A is generated as with the following Expression (54).
• 
  Pred(x,y)=P(−1,y);x,y=0, . . . ,15  (54)
• Themode 2 is a DC Prediction mode, and in the case that all of P(x, −1) and P(−1, y); x, y=−1, 0, . . . , 15 are “available”, the prediction pixel value Pred(x, y) of each pixel of the current macro block A is generated as with the following Expression (55).
• $\begin{array}{cc}\left[\mathrm{Mathematical}\mathrm{Expression}4\right]& \\ \mathrm{Pred}\left(x,y\right)=\left[\sum _{\text{?}}^{15}P\left(x^{\prime },-1\right)+\sum _{\text{?}}^{15}P\left(-1,y^{\prime }\right)+16\right]\gg 5\mathrm{with}x,y=0,\dots ,15\text{?}\text{indicates text missing or illegible when filed}& \left(55\right)\end{array}$
• Also, in the event that P(x, −1); x, y=−1, 0, . . . , 15 is “unavailable”, the prediction pixel value Pred(x, y) of each pixel of the current macro block A is generated as with the following Expression (56).
• $\begin{array}{cc}\left[\mathrm{Mathematical}\mathrm{Expression}5\right]& \\ \mathrm{Pred}\left(x,y\right)=\left[\sum _{\text{?}}^{15}P\left(-1,y^{\prime }\right)+8\right]\gg 4\mathrm{with}x,y=0,\dots ,15\text{?}\text{indicates text missing or illegible when filed}& \left(56\right)\end{array}$
• In the event that P(−1, y); x, y=−1, 0, . . . , 15 is “unavailable”, the prediction pixel value Pred(x, y) of each pixel of the current macro block A is generated as with the following Expression (57).
• $\begin{array}{cc}\left[\mathrm{Mathematical}\mathrm{Expression}6\right]& \\ \mathrm{Pred}\left(x,y\right)=\left[\sum _{\text{?}}^{15}P\left(x^{\prime },-1\right)+8\right]\gg 4\mathrm{with}x,y=0,\dots ,15\text{?}\text{indicates text missing or illegible when filed}& \left(57\right)\end{array}$
• In the event that all of P(x, −1) and P(−1, y); x, y=−1, 0, . . . , 15 are “unavailable”, 128 is employed as the prediction pixel value.
• Themode 3 is a Plane Prediction mode, and is applied only when all of P(x, −1) and P(−1, y); x, y=−1, 0, . . . , 15 are “available”. In this case, the prediction pixel value Pred(x, y) of each pixel of the current macro block A is generated as with the following Expression (58).
• $\begin{array}{cc}\left[\mathrm{Mathematical}\mathrm{Expression}7\right]& \\ \mathrm{Pred}\left(x,y\right)=\mathrm{Clip}1\left(\left(a+b*\left(x-7\right)+c*\left(y-7\right)+16\right)\gg 5\right)a=16*\left(P\left(-1,15\right)+P\left(15,-1\right)\right)b=\left(5*H+32\right)\gg 6c=\left(5*V+32\right)\gg 6H=\sum _{\text{?}}^{\text{?}}x*\left(P\left(7+x,-1\right)-P\left(7-x,-1\right)\right)V=\sum _{\text{?}}^{\text{?}}y*\left(P\left(-1,7+y\right)-P\left(-1,7-y\right)\right)\text{?}\text{indicates text missing or illegible when filed}& \left(58\right)\end{array}$
• Next, the intra prediction modes as to color difference signals will be described.FIG. 13 is a diagram showing the four kinds of intra prediction modes for color difference signals (Intra_chroma_pred_mode). The intra prediction modes for color difference signals may be set independently from the intra prediction modes for luminance signals. The intra prediction modes as to color difference signals conform to the above-mentioned 16×16 pixel intra prediction modes for luminance signals.
• However, the 16×16 pixel intra prediction modes for luminance signals take a 16×16 pixel block as the object, but on the other hand, the intra prediction modes as to color difference signals take an 8×8 pixel block as the object. Further, as shown in the above-mentionedFIG. 10 andFIG. 13, mode numbers between both do not correspond.
• Now, let us conform to the definitions of the pixel values of the current block A in the 16×16 pixel intra prediction mode for the luminance signal described above with reference toFIG. 12, and a neighboring pixel value. For example, let us say that the pixel value of a pixel neighboring the current macro block A (8×8 pixels in the event of color difference signal) to be subjected to intra processing is taken as P(x, y); x, y=−1, 0, . . . , 7.
• Themode 0 is a DC Prediction mode, and in the event that all of P(x, −1) and P(−1, y); x, y=−1, 0, . . . , 7 are “available”, the prediction pixel value Pred(x, y) of each pixel of the current macro block A is generated as with the following Expression (59).
• $\begin{array}{cc}\left[\mathrm{Mathematical}\mathrm{Expression}8\right]& \\ \mathrm{Pred}\left(x,y\right)=[\sum _{n=0}^{7}P\left(-1,n\right)+P\left(n,-1\right)))+8)\gg 4\mathrm{with}x,y=0,\dots ,7& \left(59\right)\end{array}$
• Also, in the event that P(−1, y); x, y=−1, 0, . . . , 7 is “unavailable”, the prediction pixel value Pred(x, y) of each pixel of the current macro block A is generated as with the following Expression (60).
• $\begin{array}{cc}\left[\mathrm{Mathematical}\mathrm{Expression}9\right]& \\ \mathrm{Pred}\left(x,y\right)=\left[\sum _{n=0}^{7}P\left(n,-1\right)+4\right]\gg 3\mathrm{with}x,y=0,\dots ,7& \left(60\right)\end{array}$
• Also, in the event that P(x, −1); x, y=−1, 0, . . . , 7 is “unavailable”, the prediction pixel value Pred(x, y) of each pixel of the current macro block A is generated as with the following Expression (61).
• $\begin{array}{cc}\left[\mathrm{Mathematical}\mathrm{Expression}10\right]& \\ \mathrm{Pred}\left(x,y\right)=\left[\sum _{n=0}^{7}P\left(-1,n\right)+4\right]\gg 3\mathrm{with}x,y=0,\dots ,7& \left(61\right)\end{array}$
• Themode 1 is a Horizontal Prediction mode, and is applied only when P(−1, y); x, y=−1, 0, . . . , 7 is “available”. In this case, the prediction pixel value Pred(x, y) of each pixel of the current macro block A is generated as with the following Expression (62).
• 
  Pred(x,y)=P(−1,y);x,y=0, . . . ,7  (62)
• Themode 2 is a Vertical Prediction mode, and is applied only when P(x, −1); x, y=−1, 0, . . . , 7 is “available”. In this case, the prediction pixel value Pred(x, y) of each pixel of the current macro block A is generated as with the following Expression (63).
• 
  Pred(x,y)=P(x,−1);x,y=0, . . . ,7  (63)
• Themode 3 is a Plane Prediction mode, and is applied only when P(x, −1) and P(−1, y); x, y=−1, 0, . . . , 7 are “available”. In this case, the prediction pixel value Pred(x, y) of each pixel of the current macro block A is generated as with the following Expression (64).
• $\begin{array}{cc}\left[\mathrm{Mathematical}\mathrm{Expression}11\right]& \\ \mathrm{Pred}\left(x,y\right)=\mathrm{Clip}1\left(a+b*\left(x-3\right)+c*\left(y-3\right)+16\right)\gg 5;x,y=0,\dots ,7a=16*\left(P\left(-1,7\right)+P\left(7,-1\right)\right)b=\left(17*H+16\right)\gg 5c=\left(17*V+16\right)\gg 5H=\sum _{x=1}^{d}x*\left[P\left(3+x,-1\right)-P\left(3-x,-1\right)\right]V=\sum _{y=1}^{*}y*\left[P\left(-1,3+y\right)-P\left(-1,3-y\right)\right]& \left(64\right)\end{array}$
• As described above, the intra prediction modes for luminance signals include the nine kinds of prediction modes of 4×4 pixel and 8×8 pixel block units, and the four kinds of prediction modes of 16×16 pixel macro block units. The modes of these block units are set for each macro block unit. The intra prediction modes for color difference signals include the four kinds of prediction modes of 8×8 pixel block units. The intra prediction modes for color difference signals may be set independently from the intra prediction modes for luminance signals.
• Also, with regard to the 4×4 pixel intra prediction modes (intra 4×4 prediction modes), and the 8×8 pixel intra prediction modes (intra 8×8 prediction modes) for luminance signals, one intra prediction mode is set for each 4×4 pixel and 8×8 pixel luminance signal block. With regard to the 16×16 pixel intra prediction mode (intra 16×16 prediction mode) for luminance signals and the intra prediction modes for color difference signals, one prediction mode is set as to one macro block.
• Note that the kinds of prediction modes correspond to directions indicated with the above-mentionednumbers 0, 1, 3 through 8 inFIG. 5. Theprediction mode 2 is average value prediction.
• As described above, with intra prediction according to the H.264/AVC format, filtering processing of pixel values of neighboring pixels is performed with determined filter coefficients only before performing intra prediction in block increments of 8×8 pixels as described above with Expression (14) through Expression (24). In contrast, with the image encoding device51, filtering processing of pixel values of neighboring pixels is performed with filter coefficients set according to the block to be predicted, prior to performing intra prediction of all intra prediction modes.
• [Configuration Example of Intra Prediction Unit and Neighboring Pixel Interpolation Filter Switching Unit]
• FIG. 14 is a block diagram illustrating a detailed configuration example of theintra prediction unit74 and neighboring pixel interpolationfilter switching unit75 shown inFIG. 1.
• In the case of the example inFIG. 14, theintra prediction unit74 is configured of a neighboringimage setting unit81, a predictionimage generating unit82, and an optimal predictionmode determining unit83.
• The neighboring pixel interpolationfilter switching unit75 is configured of aprediction mode buffer91, aquantization parameter buffer92, and a low-passfilter setting unit93. Note that the low-passfilter setting unit93 has built-infilter coefficient memory94.
• The neighboringimage setting unit81 is supplied with neighboring pixel values of a current block for intra prediction from theframe memory72. While illustrating of theswitch73 is omitted in the case ofFIG. 14, in reality, supply is performed from theframe memory72 to the neighboringimage setting unit81 via theswitch73. Note that in the case of intra prediction, pixel values not subjected to deblocking filtering by thedeblocking filter71 are used as neighboring pixel values.
• The neighboringimage setting unit81 uses the filter coefficient set by the low-passfilter setting unit93 to perform filter processing of the neighboring pixel values of the current block from theframe memory72, and supplies the neighboring pixel values subjected to filter processing to the predictionimage generating unit82.
• The predictionimage generating unit82 supplies information of which mode the intra prediction mode currently being processed is, to theprediction mode buffer91. The predictionimage generating unit82 uses the neighboring pixel values subjected to filter processing from the neighboringimage setting unit81 to perform intra prediction of the current block in the intra prediction mode supplied to theprediction mode buffer91, and generates a prediction image. The generated prediction image is supplied to the optimal predictionmode determining unit83 along with the intra prediction mode information.
• The optimal predictionmode determining unit83 is supplied with the image to be subjected to intra prediction that is read out from thescreen rearranging buffer62, the prediction image generated by the predictionimage generating unit82, and the intra prediction mode information thereof.
• The optimal predictionmode determining unit83 uses the supplied information to calculate a cost function value for the intra prediction mode regarding which the prediction image has been generated, using the supplied information, and decides the intra prediction mode yielding the smallest value of the calculated cost function value to be the optimal intra prediction mode. The optimal predictionmode determining unit83 outputs the prediction image of the optimal intra prediction mode and the corresponding cost function value to the predictionimage selecting unit77.
• Also, in the event a prediction image generated in the optimal intra prediction mode is selected by the predictionimage selecting unit77, the optimal predictionmode determining unit83 supplies information indicating the optimal intra prediction mode to thelossless encoding unit66.
• Theprediction mode buffer91 stores intra prediction mode information from the predictionimage generating unit82. Thequantization parameter buffer92 stores the quantization parameter from therate control unit78.
• The low-passfilter setting unit93 reads out the intra prediction mode information of the current block from theprediction mode buffer91, and reads out the quantization parameter corresponding to the current block from thequantization parameter buffer92. The low-passfilter setting unit93 sets, from the filter coefficients stored in the built-infilter coefficient memory94, filter coefficients corresponding to this information, and supplies the set filter coefficients to the neighboringimage setting unit81.
• Thefilter coefficient memory94 stores filter coefficients corresponding to quantization parameters and intra prediction modes obtained by learning using a training image at alearning device251 inFIG. 28 which will be described later. The filter coefficients are calculated and stored as described next, for each slice, for example.
• [Description of Calculation of Optimal Filter Coefficients]
• Next, a calculation method of optimal filter coefficients, used for this filter processing of neighboring pixels will be described with reference toFIG. 15. Note that in the example inFIG. 15, an example of performing vertical prediction (vertical Prediction) for a current block of 4×4 pixels is shown, but the following description is of a case applicable to any intra prediction mode.
• With the intra prediction in block increments of 8×8 pixels as described above with Expression (14) through Expression (24), a 3-tap filter coefficient of (1, 2, 1) is defined as a low-pass filter for neighboring pixels, but we will consider {c₀, c₁, c₂} is a common form of a 3-tap. Further, with the present invention, a fourth parameter c₃is also introduced as an offset value.
• Note that while this 3-tap filter is described as being settable for each slice in the following description, this is not restricted to this, and may be settable for the entire sequence, or for each COP, for example.
• In the example inFIG. 15, a_km(0.ltoreq.k, m.ltoreq.3) is the pixel value of a pixel included in the current block, and bm(−1.ltoreq.m.ltoreq.4) is a pixel value of a neighboring pixel used for vertical prediction.
• First, the b′m(0.ltoreq.m.ltoreq.3) shown in the following Expression (65) is generated by 3-tap filtering processing being performed on the neighboring pixel value b_m.
• 
  [Mathematical Expression 12]
• 
  b′_m=c₀*b_m−1+c₁*b_m+c₂*b_m+1+c₃(0≤m≤3)  (65)
• That is to say, in the event that filtering processing is performed, and a filter coefficient is used, we will say that a corresponding offset value is also used as shown in Expression (65), even without particularly mentioning the following. On other words, filter coefficients and offset values are coefficients used for filtering processing. In the same way, in the event that a filter coefficient is sent to the decoding side encoded, we will say that the corresponding offset value is also sent encoded.
• Now, if we say that the prediction pixel value when the intra prediction mode is n is p_ij(b′_m, n); 0.ltoreq.i, j.ltoreq.3, the following Expression (66) holds regarding the intra prediction pixel value, since the prediction pixel is generated by a linear expression, as described above with reference toFIG. 2 throughFIG. 14.
• $\begin{array}{cc}\left[\mathrm{Mathematical}\mathrm{Expression}13\right]& \\ P_{\mathrm{ij}}\left(b_m^{\prime },n\right)=P_{\mathrm{ij}}\left({\mathrm{<figure-callout\; label="c"\; filenames="US20220335657A1-20221020-D00003.png,US20220335657A1-20221020-D00004.png"\; state="\{\{state\}\}">c</figure-callout>}}_0*b_{m-1}+{\mathrm{<figure-callout\; label="c"\; filenames="US20220335657A1-20221020-D00000.png,US20220335657A1-20221020-D00003.png"\; state="\{\{state\}\}">c</figure-callout>}}_1\ne b_m+{\mathrm{<figure-callout\; label="c"\; filenames="US20220335657A1-20221020-D00002.png,US20220335657A1-20221020-D00003.png"\; state="\{\{state\}\}">c</figure-callout>}}_2\ne b_{m+1}+{\mathrm{<figure-callout\; label="c"\; filenames="US20220335657A1-20221020-D00003.png,US20220335657A1-20221020-D00004.png"\; state="\{\{state\}\}">c</figure-callout>}}_3,n\right)={\mathrm{<figure-callout\; label="c"\; filenames="US20220335657A1-20221020-D00003.png,US20220335657A1-20221020-D00004.png"\; state="\{\{state\}\}">c</figure-callout>}}_0*P_{\mathrm{ij}}\left(b_{m-1},n\right)+{\mathrm{<figure-callout\; label="c"\; filenames="US20220335657A1-20221020-D00000.png,US20220335657A1-20221020-D00003.png"\; state="\{\{state\}\}">c</figure-callout>}}_1*p_{\mathrm{ij}}\left(b_m,n\right)+{\mathrm{<figure-callout\; label="c"\; filenames="US20220335657A1-20221020-D00002.png,US20220335657A1-20221020-D00003.png"\; state="\{\{state\}\}">c</figure-callout>}}_2*p_{\mathrm{ij}}\left(b_{m+1},n\right)+c_3& \left(66\right)\end{array}$
• At this time, the prediction squared error as to the current block .OMEGA. with a_ijas the original image pixel value is as shown in the following Expression (67).
• $\begin{array}{cc}\left[\mathrm{Mathematical}\mathrm{Expression}14\right]& \\ \mathrm{Err}\left(\Omega \right)=\sum _{i=0}^3\sum _{j=0}^3{\left(a_{\mathrm{ij}}-p_{\mathrm{ij}}\left(b_m^{\prime },n\right)\right)}^2=\sum _{i=0}^3\sum _{j=0}^3{\left(a_{km}-\left\{{\mathrm{<figure-callout\; label="c"\; filenames="US20220335657A1-20221020-D00003.png,US20220335657A1-20221020-D00004.png"\; state="\{\{state\}\}">c</figure-callout>}}_0*p_{\mathrm{ij}}\left(b_{m-1},n\right)+{\mathrm{<figure-callout\; label="c"\; filenames="US20220335657A1-20221020-D00000.png,US20220335657A1-20221020-D00003.png"\; state="\{\{state\}\}">c</figure-callout>}}_1*p_{\mathrm{ij}}\left(b_m,n\right)+{\mathrm{<figure-callout\; label="c"\; filenames="US20220335657A1-20221020-D00002.png,US20220335657A1-20221020-D00003.png"\; state="\{\{state\}\}">c</figure-callout>}}_2*p_{\mathrm{ij}}\left(b_{m+1},n\right)+c_3\right\}\right)}^2& \left(67\right)\end{array}$
• Now, if we represent with .PHI. a set of intra blocks encoded with the intra prediction mode n in the current slice, the summation of the prediction squared error relating to the blocks belonging to .PHI. is represented with the following Expression (68).
• $\begin{array}{cc}\left[\mathrm{Mathematical}\mathrm{Expression}15\right]& \\ \mathrm{Err}\left(\Omega \in \Phi \right)=\sum _{\Omega \in \Phi }\sum _{i=0}^{\text{?}}\sum _{j=0}^{\text{?}}{\left(a_{km}-\left\{{\mathrm{<figure-callout\; label="c"\; filenames="US20220335657A1-20221020-D00003.png,US20220335657A1-20221020-D00004.png"\; state="\{\{state\}\}">c</figure-callout>}}_0*p_{\mathrm{ij}}\left(b_{m-1},n\right)+{\mathrm{<figure-callout\; label="c"\; filenames="US20220335657A1-20221020-D00000.png,US20220335657A1-20221020-D00003.png"\; state="\{\{state\}\}">c</figure-callout>}}_1*p_{\mathrm{ij}}\left(b_m,n\right)+{\mathrm{<figure-callout\; label="c"\; filenames="US20220335657A1-20221020-D00002.png,US20220335657A1-20221020-D00003.png"\; state="\{\{state\}\}">c</figure-callout>}}_2*p_{\mathrm{ij}}\left(b_{m+1},n\right)+c_3\right\}\right)}^2\text{?}\text{indicates text missing or illegible when filed}& \left(68\right)\end{array}$
• In the above Expression (68), we will consider Err(.OMEGA..epsilon..PHI.) to be a function of c₀, c₁, c₂, c₃, i.e., Err(.OMEGA..epsilon..PHI.; c₀, c₁, c₂, c₃), so a c₀, c₁, c₂, c₃which minimizes Err(.OMEGA..epsilon..PHI.; c₀, c₁, c₂, c₃) will be an optimal filer coefficient value in the current slice. That is to say, it is sufficient to obtain a c₀, c₁, c₂, c₃where the following Expression (69) holds.
• $\begin{array}{cc}\left[\mathrm{Mathematical}\mathrm{Expression}16\right]& \\ \frac{\mathrm{Err}\left(\Omega \in \Phi \right)}{{\mathrm{<figure-callout\; label="c"\; filenames="US20220335657A1-20221020-D00003.png,US20220335657A1-20221020-D00004.png"\; state="\{\{state\}\}">c</figure-callout>}}_0}=0;\frac{\mathrm{Err}\left(\Omega \in \Phi \right)}{{\mathrm{<figure-callout\; label="c"\; filenames="US20220335657A1-20221020-D00000.png,US20220335657A1-20221020-D00003.png"\; state="\{\{state\}\}">c</figure-callout>}}_1}=0;\frac{\mathrm{Err}\left(\Omega \in \Phi \right)}{{\mathrm{<figure-callout\; label="c"\; filenames="US20220335657A1-20221020-D00002.png,US20220335657A1-20221020-D00003.png"\; state="\{\{state\}\}">c</figure-callout>}}_2}=0;\frac{\mathrm{Err}\left(\Omega \in \Phi \right)}{{\mathrm{<figure-callout\; label="c"\; filenames="US20220335657A1-20221020-D00003.png,US20220335657A1-20221020-D00004.png"\; state="\{\{state\}\}">c</figure-callout>}}_3}=0;& \left(69\right)\end{array}$
• That is to say, the simultaneous equation shown in the following Expression (70) is obtained from Expression (69).
• $\begin{array}{cc}\left[\mathrm{Mathematical}\mathrm{Expression}17\right]& \\ 0=\frac{\mathrm{ssd}}{c_0}=\mathrm{\Sigma \Sigma \Sigma }((-2\times a_{\mathrm{ij}}\times p_{\mathrm{ij}}\left(b_{k-1}^{\prime }\right)\times \left(a_{\mathrm{ij}}-\left(c_0{p}_{\mathrm{ij}}\left(b_{k-1}^{\prime }\right)+c_1{p}_{\mathrm{ij}}\left(b_k^{\prime }\right)+c_2{p}_{\mathrm{ij}}\left(b_{k+j}^{\prime }\right)+c_3\right)\right)0=\frac{\mathrm{ssd}}{c_1}=\mathrm{\Sigma \Sigma \Sigma }((-2\times a_{\mathrm{ij}}\times p_{\mathrm{ij}}\left(b_k^{\prime }\right)\times \left(a_{\mathrm{ij}}-\left(c_0{p}_{\mathrm{ij}}\left(b_{k-1}^{\prime }\right)+c_1{p}_{\mathrm{ij}}\left(b_k^{\prime }\right)+c_2{p}_{\mathrm{ij}}\left(b_{k+j}^{\prime }\right)+c_3\right)\right)0=\frac{\mathrm{ssd}}{c_2}=\mathrm{\Sigma \Sigma \Sigma }((-2\times a_{\mathrm{ij}}\times p_{\mathrm{ij}}\left(b_{k+1}^{\prime }\right)\times \left(a_{\mathrm{ij}}-\left(c_0{p}_{\mathrm{ij}}\left(b_{k-1}^{\prime }\right)+c_1{p}_{\mathrm{ij}}\left(b_k^{\prime }\right)+c_2{p}_{\mathrm{ij}}\left(b_{k+j}^{\prime }\right)+c_3\right)\right)0=\frac{\mathrm{ssd}}{c_3}=\mathrm{\Sigma \Sigma \Sigma }((-2\times a_{\mathrm{ij}}\times \left(a_{\mathrm{ij}}-\left(c_0{p}_{\mathrm{ij}}\left(b_{k-1}^{\prime }\right)+c_1{p}_{\mathrm{ij}}\left(b_k^{\prime }\right)+c_2{p}_{\mathrm{ij}}\left(b_{k+1}^{\prime }\right)+c_3\right)\right)& \left(70\right)\end{array}$
• Reformulating this Expression (70) using a matrix yields Expression (71).
• $\begin{array}{cc}\left[\mathrm{Mathematical}\mathrm{Expression}18\right]& \\ \left[\begin{array}{cccc}\mathrm{\Sigma \Sigma \Sigma }{p}_{\mathrm{ij}}\left(b_{k-1}\right)p_{\mathrm{ij}}\left(b_{k-j}\right)& \mathrm{\Sigma \Sigma \Sigma }{p}_{\mathrm{ij}}\left(b_{k-1}\right)p_{\mathrm{ij}}\left(b_k\right)& \mathrm{\Sigma \Sigma \Sigma }{p}_{\mathrm{ij}}\left(b_{k+1}\right)p_{\mathrm{ij}}\left(b_{k-1}\right)& \mathrm{\Sigma \Sigma \Sigma }{p}_{\mathrm{ij}}\left(b_{k-1}\right)\\ \mathrm{\Sigma \Sigma \Sigma }{p}_{\mathrm{ij}}\left(b_k\right)p_{\mathrm{ij}}\left(b_{k-j}\right)& \mathrm{\Sigma \Sigma \Sigma }{p}_{\mathrm{ij}}\left(b_k\right)p_{\mathrm{ij}}\left(b_k\right)& \mathrm{\Sigma \Sigma \Sigma }{p}_{\mathrm{ij}}\left(b_{k+1}\right)p_{\mathrm{ij}}\left(b_k\right)& \mathrm{\Sigma \Sigma \Sigma }{p}_{\mathrm{ij}}\left(b_k\right)\\ \mathrm{\Sigma \Sigma \Sigma }{p}_{\mathrm{ij}}\left(b_{k+1}\right)p_{\mathrm{ij}}\left(b_{k-j}\right)& \mathrm{\Sigma \Sigma \Sigma }{p}_{\mathrm{ij}}\left(b_{k+1}\right)p_{\mathrm{ij}}\left(b_k\right)& \mathrm{\Sigma \Sigma \Sigma }{p}_{\mathrm{ij}}\left(b_{k-1}\right)p_{\mathrm{ij}}\left(b_{k+1}\right)& \mathrm{\Sigma \Sigma \Sigma }{p}_{\mathrm{ij}}\left(b_{k+1}\right)\\ \mathrm{\Sigma \Sigma \Sigma }{p}_{\mathrm{ij}}\left(b_{k-1}\right)& \mathrm{\Sigma \Sigma \Sigma }{p}_{\mathrm{ij}}\left(b_k\right)& \mathrm{\Sigma \Sigma \Sigma }{p}_{\mathrm{ij}}\left(b_{k-1}\right)& \mathrm{\Sigma \Sigma \Sigma }\text{?}\end{array}\right]\left[\begin{array}{c}{c}_0\\ c_1\\ c_2\\ c_3\end{array}\right]=\left[\begin{array}{c}\mathrm{\Sigma \Sigma \Sigma }{a}_{\mathrm{ij}}{p}_{\mathrm{ij}}\left(b_{k+1}\right)\\ \mathrm{\Sigma \Sigma \Sigma }{a}_{\mathrm{ij}}{p}_{\mathrm{ij}}\left(b_k\right)\\ \mathrm{\Sigma \Sigma \Sigma }{a}_{\mathrm{ij}}{p}_{\mathrm{ij}}\left(b_{k-1}\right)\\ \mathrm{\Sigma \Sigma \Sigma }{a}_{\mathrm{ij}}\end{array}\right]\text{?}\text{indicates text missing or illegible when filed}& \left(71\right)\end{array}$
• Solving this Expression (71) enables the optimal filter coefficient and offset value {c₀, c₁, c₂, c₃} to be obtained as to the current slice.
• Note that optimal filter coefficients and offset values {c₀, c₁, c₂, c₃} are obtained as floating-point values by solving the simultaneous equation in Expression (70), but in the case of the image encoding device51 inFIG. 1, and the correspondingimage decoding device151 inFIG. 22, these are rounded off to 8-bit coefficients, for example.
• That is to say, even if the filter coefficients are floating-point, thefilter coefficient memory94 holds these as n-bit (where n is an integer) values in accordance with the register length of the processor, for example.
• By applying the same method as that described above to other intra prediction methods as well, optimal filter coefficients can also be obtained for the other intra prediction methods. Also, optimal filter coefficients can also be obtained for not only theintra 4×4 prediction mode, but also for theintra 8×8 prediction mode, intra 16×16 prediction mode, and intra prediction mode for color difference signals, by the same method.
• While one filter coefficient is obtained for each intra prediction mode in the above description, this is not restricted to this, and an arrangement may be made where just one filter coefficient is obtained for all intra prediction modes. Particularly, with the intra prediction mode described above with reference toFIG. 2 throughFIG. 14, prediction pixel values are used as they are for Vertical (vertical) and Horizontal (horizontal) modes, while some sort of averaging processing or weighted-averaging processing is performed to generate prediction pixels for other modes, so the properties thereof differ. Accordingly, performing two types of class classification, of Vertical and Horizontal modes, and other modes, and calculating filter coefficients for each class, can realize further improvement of encoding efficiency. Also, for luminance signals, there may be one for theintra 4×4 prediction mode, one for theintra 8×8 prediction mode, and one for the intra 16×16 prediction mode, for example. For color difference signals, filter coefficients may be obtained separately for Cb/Cr, for example.
• Also, in the above description, the three taps of {c₀, c₁, c₂} have been used for the filer coefficients for low-pass filter processing, but this is not restricted to 3-tap, and filters of any number of taps may be used. That is to say, filter coefficients+offset values of the number of taps are obtained. However, as the number of taps increases, the order of simultaneous equations to be solved increases.
• Further, an arrangement may be made wherein filter coefficients which differ according to the image frame, such as CIF (Common Intermediate Format)/QCIF (Quarter CIF), SD (Standard Definition), HD (High Definition), or the like are prepared, and applied.
• Also, with the above-described method, filter coefficients are calculated by minimizing the intra prediction residual (prediction squared error). However, the filter coefficient calculation method is not restricted to this, and in the event that there is the need to send filter coefficients to the decoding side, optimization including bits for sending the filer coefficients may be performed as well.
• Further, with the above-described filter coefficients, we assume symmetry of coefficients, as shown in the following Expression (72).
• 
  C0=C2  (72)
• That is to say, filter coefficients are calculating so as to have symmetry as to a center coefficient corresponding to zero-phase, as with {c₀, c₁, c₂}. Accordingly, the three simultaneous equations shown in the above Expression (70) can be reduced to two. As a result the amount of computation can be reduced.
• By setting filter coefficients suitable for the input image, and performing low-pass filer processing as to neighboring pixels in an adaptive manner, using the above method, enables encoding to performed using prediction images suitable for the image, quantization parameters, and prediction mode, whereby encoding efficiency can be improved.
• Two methods can be conceived regarding the above-described optimal filter coefficient calculation. One method is offline processing where, prior to performing encoding processing, image signals for training are used to calculate filer coefficients beforehand optimizing the entire image signals. Learning processing which is this offline processing will be described later with reference toFIG. 28, and filter coefficients and offset values calculated by the learning processing are stored in thefilter coefficient memory94 inFIG. 14.
• The second method is online processing where optimal filter coefficients are successively calculated for each slice. In this case, the calculated filter coefficients and offset values are sent to the decoding side. An example of a case of performing online processing which is the second method will be described later with reference toFIG. 20.
[Description of Encoding Processing of Image Encoding Device]
• Next, the encoding processing of the image encoding device51 inFIG. 1 will be described with reference to the flowchart inFIG. 16.
• In step S11, the A/D conversion unit61 converts an input image from analog to digital. In step S12, thescreen rearranging buffer62 stores the image supplied from the A/D conversion unit61, and performs rearranging from the sequence for displaying the pictures to the sequence for encoding.
• In step S13, thecomputing unit63 computes difference between an image rearranged in step S12 and the prediction image. The prediction image is supplied to thecomputing unit63 from the motion prediction/compensation unit76 in the event of performing inter prediction, and from theintra prediction unit74 in the event of performing intra prediction, via the predictionimage selecting unit77.
• The difference data is smaller in the data amount as compared to the original image data. Accordingly, the data amount can be compressed as compared to the case of encoding the original image without change.
• In step S14, theorthogonal transform unit64 subjects the difference information supplied from thecomputing unit63 to orthogonal transform. Specifically, orthogonal transform, such as discrete cosine transform, Karhunen-Loeve transform, or the like, is performed, and a transform coefficient is output. In step S15, thequantization unit65 quantizes the transform coefficient. At the time of this quantization, a rate is controlled such that later-described processing in step S25 will be described.
• The difference information thus quantized is locally decoded as follows. Specifically, in step S16, theinverse quantization unit68 subjects the transform coefficient quantized by thequantization unit65 to inverse quantization using a property corresponding to the property of thequantization unit65. In step S17, the inverseorthogonal transform unit69 subjects the transform coefficient subjected to inverse quantization by theinverse quantization unit68 to inverse orthogonal transform using a property corresponding to the property of theorthogonal transform unit64.
• In step S18, thecomputing unit70 adds the prediction image input via the predictionimage selecting unit77 to the locally decoded difference information, and generates a locally decoded image (the image corresponding to the input to the computing unit63). In step S19, thedeblocking filter71 subjects the image output from thecomputing unit70 to filtering. Thus, block distortion is removed. In step S20, theframe memory72 stores the image subjected to filtering. Note that an image not subjected to filtering processing by thedeblocking filter71 is also supplied from thecomputing unit70 to theframe memory72 for storing.
• In step S21, theintra prediction unit74 and motion prediction/compensation unit76 each perform image prediction processing. Specifically, in step S21, theintra prediction unit74 performs intra prediction processing in the intra prediction mode. The motion prediction/compensation unit76 performs motion prediction and compensation processing in the inter prediction mode.
• The details of the prediction processing in step S21 will be described later with reference toFIG. 17, but according to this processing, the prediction processes in all of the candidate prediction modes are performed, and the cost function values in all of the candidate prediction modes are calculated. The optimal intra prediction mode is then selected based on the calculated cost function values, and the prediction image generated by the intra prediction in the optimal intra prediction mode, and the cost function value thereof are supplied to the predictionimage selecting unit77.
• Note that at this time, before intra prediction processing, theintra prediction unit74 performs filtering processing as to the neighboring pixels used for inter prediction of the current blocks, using the filter coefficients set by the neighboring pixel interpolationfilter switching unit75. Intra prediction is then performed at theintra prediction unit74 using the neighboring pixels subjected to filter processing, and the prediction image is generated.
• In step S22, the predictionimage selecting unit77 determines one of the optimal intra prediction mode and the optimal inter prediction mode to be the optimal prediction mode based on the cost function values output from theintra prediction unit74 and the motion prediction/compensation unit76. The predictionimage selecting unit77 then selects the prediction image in the determined optimal prediction mode, and supplies to thecomputing units63 and70. This prediction image is, as described above, used for calculations in steps S13 and S18.
• Note that the selection information of this prediction image is supplied to theintra prediction unit74 or motion prediction/compensation unit76. In the event that the prediction image in the optimal intra prediction mode has been selected, theintra prediction unit74 supplies information indicating the optimal intra prediction mode (i.e., intra prediction mode information) to thelossless encoding unit66.
• In the event that the prediction image in the optimal inter prediction mode has been selected, the motion prediction/compensation unit76 outputs information indicating the optimal inter prediction mode, and according to need, information according to the optimal inter prediction mode to thelossless encoding unit66. Examples of the information according to the optimal inter prediction mode include motion vector information, flag information, and reference frame information. That is to say, when a prediction image according to the inter prediction mode as the optimal inter prediction mode is selected, the motion prediction/compensation unit76 outputs inter prediction mode information, motion vector information, and reference frame information, to thelossless encoding unit66.
• In step S23, thelossless encoding unit66 encodes the quantized transform coefficient output from thequantization unit65. Specifically, the difference image is subjected to lossless encoding such as variable length coding, arithmetic coding, or the like, and compressed. At this time, the optimal intra prediction mode information from theintra prediction unit74, or the information according to the optimal inter prediction mode from the motion prediction/compensation unit76, and quantization parameters from therate control unit78 and so forth, input to thelossless encoding unit66 in step S22, are also encoded, and added to the header information.
• In step S24, the storingbuffer67 stores the difference image as the compressed image. The compressed image stored in the storingbuffer67 is read out as appropriate, and transmitted to the decoding side via the transmission path.
• In step S25, therate control unit78 controls the rate of the quantization operation of thequantization unit65 by quantization parameters, based on the compressed image stored in the storingbuffer67, so as not to cause overflow or underflow.
• The quantization parameter used for rate control at thequantization unit65 is supplied to thelossless encoding unit66, subjected to lossless encoding processing in the above-described step S23, and inserted into the header portion of the compressed image. Also, this quantization parameter is supplied to the neighboring pixel interpolationfilter switching unit75, and used for setting of filter coefficients used in the filtering processing to be performed on neighboring pixels, which is performed prior to the intra prediction.
[Description of Prediction Processing]
• Next, the prediction processing in step S21 inFIG. 16 will be described with reference to the flowchart inFIG. 17.
• In the event that the image to be processed, supplied from thescreen rearranging buffer62, is an image in a block to be subjected to intra processing, the decoded image to be referenced is read out from theframe memory72, and supplied to theintra prediction unit74 via theswitch73.
• In step S31, theintra prediction unit74 performs intra prediction of the pixels of the block to be processed, in all candidate intra prediction modes, using the supplied image. Note that pixels not subjected to deblocking filtering by thedeblocking filer71 are used as decoded pixels to be referenced.
• The details of the intra prediction processing in step S31 will be described with reference toFIG. 18, but according to this processing, an optimal filter coefficient is set, and filtering processing is performed as to neighboring pixels using the set filter coefficient. The neighboring pixels on which filer processing has been performed has been performed are then used to perform intra prediction, thereby generating a prediction image.
• The above-described processing is performed on all candidate intra prediction modes, a cost function value is calculated as to all of the candidate intra prediction modes, and the optimal intra prediction mode is decided based on the calculated cost function values. The prediction image generated thereby and the cost function value of the optimal intra prediction mode are supplied to the predictionimage selecting unit77.
• In the event that the image to be processed supplied from thescreen rearranging buffer62 is an image to be subjected to inter processing, the image to be referenced is read out from theframe memory72, and supplied to the motion prediction/compensation unit76 via theswitch73. In step S32, based on these images, the motion prediction/compensation unit76 performs inter motion prediction processing. That is to say, the motion prediction/compensation unit76 references the image supplied from theframe memory72 to perform the motion prediction processing in all of the candidate inter prediction modes.
• The details of the inter motion prediction processing in step S32 will be described later with reference toFIG. 19, and according to this processing, the motion prediction processing in all of the candidate inter prediction modes is performed, and a cost function value is calculated as to all of the candidate inter prediction modes.
• In step S33, the motion prediction/compensation unit76 compares the cost function values as to the inter prediction modes calculated in step S32, and determines the prediction mode that provides the minimum value, to be the optimal inter prediction mode. The motion prediction/compensation unit76 then supplies the prediction image generated in the optimal inter prediction mode, and the cost function value thereof to the predictionimage selecting unit77.
[Description of Intra Prediction Processing]
• Next, the intra prediction processing in step S31 inFIG. 17 will be described with reference to the flowchart inFIG. 18. Note that with the example inFIG. 18, description will be made regarding a case of a luminance signal as an example.
• In step S25 in the aboveFIG. 16, therate control unit78 supplies a quantization parameter for the current block. In step S41, thequantization parameter buffer92 obtains the quantization parameter for the current block from therate control unit78, and stores this.
• In step S42, the predictionimage generating unit82 selects one intra prediction mode from the intra prediction modes of 4×4 pixels, 8×8 pixels, and 16×16 pixels. The selected intra prediction mode information is stored in theprediction mode buffer91.
• The low-passfilter setting unit93 reads out the intra prediction mode information from theprediction mode buffer91, and reads out the quantization parameter value from thequantization parameter buffer92. In step S43, the low-passfilter setting unit93 then sets, from the filter coefficients calculated for each slice that are stored in thefilter coefficient memory94, a filter coefficient corresponding to the intra prediction mode and quantization parameter. The set filter coefficient is supplied to the neighboringimage setting unit81.
• In step S44, the neighboringimage setting unit81 uses the set filter coefficient to perform filtering processing on the neighboring pixel values of the current block, and supplies the neighboring pixel values subjected to filtering processing to the predictionimage generating unit82.
• In step S43, the predictionimage generating unit82 uses the neighboring pixel values subjected to filtering processing to perform intra prediction of the current block with the intra prediction mode selected in step S42, and generates a prediction image.
• The optimal predictionmode determining unit83 is supplied with the image to be subjected to intra prediction that has been read out from thescreen rearranging buffer62, the prediction image generated by the predictionimage generating unit82, and the intra prediction mode information thereof.
• In step S46, the optimal predictionmode determining unit83 uses the supplied information to calculate a cost function value for the intra prediction mode with which the prediction image has been generated. Here, calculation of a cost function value is performed based on one of the techniques of a High Complexity mode or Low Complexity mode. These modes are determined in JM (Joint Model) that is reference software in the H.264/AVC format.
• Specifically, in the High Complexity mode, tentatively, up to encoding processing is performed as to all of the candidate prediction modes as the processing in step S45. A cost function value represented with the following Expression (73) is calculated as to the prediction modes, and a prediction mode that provides the minimum value thereof is selected as the optimal prediction mode.
• 
  Cost(Mode)=D+λ*R  (73)
• D represents the difference (distortion) between the original image and a decoded image, R represents a generated code amount including an orthogonal transform coefficient, and λ represents a LaGrange multiplier to be provided as a function of a quantization parameter QP.
• On the other hand, in the Low Complexity mode, a prediction image is generated, and up to header bits of motion vector information, prediction mode information, flag information, and so forth are calculated as to all of the candidate prediction modes as the processing in step S45. A cost function value represented with the following Expression (74) is calculated as to the prediction modes, and a prediction mode that provides the minimum value thereof is selected as the optimal prediction mode.
• 
  Cost(Mode)=D+QPtoQuant(QP)+Header_Bit  (74)
• D represents difference (distortion) between the original image and a decoded image, Header_Bit represents header bits as to a prediction mode, and QPtoQuant is a function provided as a function of the quantization parameter QP.
• In the Low Complexity mode, a prediction image is only generated as to all of the prediction modes, and there is no need to perform encoding processing and decoding processing, and accordingly, calculation amount can be reduced.
• In step S47, the optimal predictionmode determining unit83 determines whether or not processing has ended for all intra prediction modes. That is to say, in step S47, determination is made regarding whether or not the processing of steps S42 through S46 has been performed all of the intra prediction modes of 4×4 pixels, 8×8 pixels, and 16×16 pixels.
• In the event that determination is made in step S47 that processing has not ended for all intra prediction modes, the processing returns to step S42, and subsequent processing is repeated.
• In the event that determination is made in step S47 that processing has ended for all intra prediction modes, the processing advances to step S48. In step S48, the optimal predictionmode determining unit83 determines the intra prediction mode of which the calculated cost function value is the minimal value, to be the optimal intra prediction mode.
• The prediction image of the optimal intra prediction mode, and the corresponding cost function value, are supplied to the predictionimage selecting unit77.
• In the event that the prediction image generated in the optimal intra prediction mode is selected by the predictionimage selecting unit77, the optimal predictionmode determining unit83 supplies information indicating the optimal intra prediction mode to thelossless encoding unit66. This is then encoded at thelossless encoding unit66 and added to the header information of the compressed image (step S23 in the aboveFIG. 16).
• Note that the filter coefficients calculated by the learning processing that are stored in thefilter coefficient memory94 are also similarly stored in theimage decoding device151 inFIG. 22 which will be described later, whereby the set filter coefficients do not have to be added to the header information of the compressed image and sent.
• Accordingly, in the case of H.264/AVC, there are 51 quantization parameters, there are nine intra prediction modes for 4×4 pixels and 8×8 pixels, and when these combinations are considered, an immense number of filter coefficients, which is 51×9=459, becomes necessary. Information relating to such an immense number of filter coefficients does not have to be sent to the decoding side, so processing can be realized without increasing the overhead for coefficient information.
[Description of Inter Motion Prediction Processing]
• Next, the inter motion prediction processing in step S32 inFIG. 17 will be described with reference to the flowchart inFIG. 19.
• In step S61, the motion prediction/compensation unit76 determines a motion vector and a reference image as to each of the eight kinds of the inter prediction modes made up of 16×16 pixels through 4×4 pixels. That is to say, a motion vector and a reference image are each determined as to the block to be processed in each of the inter prediction modes.
• In step S62, the motion prediction/compensation unit76 subjects the reference image to motion prediction and compensation processing based on the motion vector determined in step S61 regarding each of the eight kinds of the inter prediction modes made up of 16×16 pixels through 4×4 pixels. According to this motion prediction and compensation processing, a prediction image in each of the inter prediction modes is generated.
• In step S63, the motion prediction/compensation unit76 generates motion vector information to be added to the compressed image regarding the motion vector determined as to each of the eight kinds of inter prediction modes made up of 16×16 pixels through 4×4 pixels.
• The generated motion vector information is also used at the time of calculation of a cost function value in the next step S64, and output, in the event that the corresponding prediction image has ultimately been selected by the predictionimage selecting unit77, to thelossless encoding unit66 along with the prediction mode information and reference frame information.
• In step S64, the motion prediction/compensation unit76 calculates the cost function value shown in the above-mentioned Expression (73) or Expression (74) as to each of the eight kinds of the inter prediction modes made up of 16×16 pixels through 4×4 pixels. The cost function values calculated here are used at the time of determining the optimal inter prediction mode in step S34 inFIG. 17 described above.
• Next, as the second method for calculating an optimal filter coefficient, an example of a case where online processing is performed in which optimal filter coefficients are successively calculated for each slice, will be described with reference toFIG. 20.
• Now, in this case, there is the need to send, to the decoding side, the filter coefficients calculated at the encoding side for each slice, and sending filter coefficients having been broken down into a great number of cases leads to deterioration in encoding efficiency. Accordingly, just one filter coefficient is sent for a slice, or just one is sent for each prediction mode for each block size, or just one is sent for prediction mode type, such as horizontal prediction, vertical prediction, and so on.
• Also, in the case of the above offline processing, description has been made regarding an example of using intra prediction mode and quantization parameters, as parameters for calculating filter coefficients. On the other hand, in the case of online processing, a great number of parameters for calculating filter coefficients increases the processing amount, so an example will be described withFIG. 20 with regard to an example of using only the intra prediction mode for parameters. While description will be omitted, just quantization parameters may be used, or both parameters may be used, as a matter of course.
• [Other Configuration Example of Intra Prediction Unit and Neighboring Pixel Interpolation Filter Switching Unit]
• FIG. 20 is a block diagram illustrating another configuration example of theintra prediction unit74 and neighboring pixel interpolationfilter switching unit75 and in the case of performing online processing on each slice to successively calculate optimal filter coefficients.
• In the case of the example inFIG. 20, aswitch101 is introduced between theintra prediction unit74 and the neighboring pixel interpolationfilter switching unit75, and unlike the case shown inFIG. 14, by switching theswitch101 on and off, theintra prediction unit74 performs intra prediction twice. That is to say, with theintra prediction unit74, intra prediction defined in H.264/AVC is performed in the state that theswitch101 is off, and filter coefficients suitable for the intra prediction are calculated. In the state that theswitch101 is on, intra prediction is performed with the filter coefficient set by the neighboring pixel interpolationfilter switching unit75 out of the calculated filter coefficients.
• Theintra prediction unit74 inFIG. 20 is configured of a neighboringimage setting unit111, a predictionimage generating unit112, and an optimal predictionmode determining unit113.
• The neighboring pixel interpolationfilter switching unit75 is configured of aprediction mode buffer121, an optimalfilter calculating unit122, and a low-passfilter setting unit123.
• The neighboringimage setting unit111 is supplied with the neighboring pixel values of all current blocks of the current slice for intra prediction from theframe memory72. In the case inFIG. 20 as well, illustrating of theswitch73 is omitted. Note that in the case of intra prediction, pixel values not subjected to deblocking filtering by thedeblocking filter71 are used as neighboring pixel values.
• In the event that theswitch101 is in an oft state, the neighboringimage setting unit111 subjects the neighboring pixel values of the current block from theframe memory72 to filtering processing using filter coefficients only for intra prediction modes defined in H.264/AVC, and supplies this to the predictionimage generating unit112. That is to say, only in the case of the 8×8 pixel prediction mode described above with Expression (14) through Expression (24) are the neighboring pixel values subjected to filtering processing subjected to the predictionimage generating unit112. In all other cases, the neighboring pixel values of the current block from theframe memory72 are supplied to the predictionimage generating unit112 as they are.
• In the event that theswitch101 is in the on state, the neighboringimage setting unit111 is supplied with filter coefficients from the low-passfilter setting unit123. Accordingly, the neighboringimage setting unit111 subjects the neighboring pixel values of the current block from theframe memory72 to filtering processing using the filter coefficients set by the low-passfilter setting unit123, and supplies the neighboring pixel values subjected to filtering processing to the predictionimage generating unit112.
• The predictionimage generating unit112 uses the neighboring pixel values from the neighboringimage setting unit111 to perform intra prediction of the current block in all intra prediction modes, and generates prediction images. The generated prediction images are supplied to the optimal predictionmode determining unit113 along with the intra prediction mode information.
• The optimal predictionmode determining unit113 is supplied with the image for intra prediction that has been read out from thescreen rearranging buffer62, the prediction images generated by the predictionimage generating unit112, and the intra prediction mode information thereof.
• The optimal predictionmode determining unit113 uses the supplied information to calculate cost function values for the intra prediction modes by which the prediction images have been generated, and determines the intra prediction mode yielding the smallest value for the calculated cost function values to be the optimal intra prediction mode.
• In the event that theswitch101 is in the off state, the optimal predictionmode determining unit113 supplies the information of the optimal intra prediction mode to theprediction mode buffer121. In the event that theswitch101 is in the on state, the optimal predictionmode determining unit113 supplies the prediction image of the optimal intra prediction mode, and the corresponding cost function value, to the predictionimage selecting unit77.
• Also, in the event the prediction image generated in the optimal intra prediction mode is selected by the predictionimage selecting unit77, the optimal predictionmode determining unit113 supplies information indicating the optimal intra prediction mode to thelossless encoding unit66.
• Theprediction mode buffer121 stores the intra prediction mode information from the optimal predictionmode determining unit113.
• The optimalfilter calculating unit122 is supplied with the image for intra prediction that has been read out from thescreen rearranging buffer62, and the neighboring pixel values of the current block from theframe memory72. The optimalfilter calculating unit122 reads out the intra prediction mode for each block included in the current slice from theprediction mode buffer121. The optimalfilter calculating unit122 then uses this information to calculate optimal filter coefficients for the intra prediction modes of the current slice as described above with reference toFIG. 15, and supplies the calculated filter coefficients to the low-passfilter setting unit123.
• The low-passfilter setting unit123 sets, from the filter coefficients of the current slice that have been calculated, a filter coefficient for the current block, turns the terminal of theswitch101 on, and supplies the set filter coefficient to the neighboringimage setting unit111. Also, the low-passfilter setting unit123 supplies the filter coefficient for the current slice to thelossless encoding unit66.
[Other Description of Intra Prediction Processing]
• Next, the intra prediction processing which theintra prediction unit74 and neighboring pixel interpolationfilter switching unit75 inFIG. 20 perform will be described with reference to the flowchart inFIG. 21. Note that this intra prediction processing is another example of the intra prediction processing of step S31 inFIG. 17.
• First, theswitch101 is in an off state. The neighboringimage setting unit111 is supplied with neighboring pixel values of all current blocks of the current slice to be subjected to intra prediction, from theframe memory72. The neighboringimage setting unit111 performs filtering processing on the neighboring pixel values of the current block from theframe memory72 using the filter coefficients only for the 8×8 pixel intra prediction mode defined in H.264/AVC, and supplies to the predictionimage generating unit112. That is to say, in the case of other intra prediction modes, the neighboring pixel values of the current block from theframe memory72 are supplied to the predictionimage generating unit112 as they are.
• In step S101, the predictionimage generating unit112 performs intra prediction processing on all blocks included in the current slice. That is to say, the predictionimage generating unit112 uses the neighboring pixel values of the current block from the neighboringimage setting unit111 to perform intra prediction in each intra prediction mode, and generates prediction images.
• The optimal predictionmode determining unit113 is supplied with the image to be subjected to intra prediction that has been read out from thescreen rearranging buffer62, the prediction images generated by the predictionimage generating unit112, and the intra prediction mode information thereof.
• In step S102, the optimal predictionmode determining unit113 uses the supplied information to calculate the cost function values in Expression (73) or Expression (74) described above for all intra prediction modes regarding which a prediction image has been generated.
• In step S103, the optimal predictionmode determining unit113 determines the intra prediction mode where the cost function in Expression (73) or Expression (74) is the smallest to be the optimal intra prediction mode, and supplies the information of the determined intra prediction mode to theprediction mode buffer121.
• The optimalfilter calculating unit122 is supplied with the image to be subjected to intra prediction that has been read out from thescreen rearranging buffer62, and the neighboring pixel values of the current block from theframe memory72. The optimalfilter calculating unit122 reads out the intra prediction mode for each block included in the current slice from theprediction mode buffer121.
• In step S104, the optimalfilter calculating unit122 uses this information to calculate a filter coefficient which minimizes the residual of the overall current slice as the optimal filter coefficient for each of the intra prediction modes of the current slice. The filter coefficients calculated as described above with reference toFIG. 15 are supplied to the low-passfilter setting unit123.
• The low-passfilter setting unit123 sets, from the calculated filter coefficients for the current slice, a filter coefficient corresponding to the current block, turns the terminal of theswitch101 on, and supplies the set filter coefficient to the neighboringimage setting unit111.
• In step S105, the neighboringimage setting unit111 uses the filter coefficient set by the low-passfilter setting unit123 to perform filtering processing on the neighboring pixel values of the current block from theframe memory72.
• The neighboring pixel values subjected to filter processing are supplied to the predictionimage generating unit112. in step S106, the predictionimage generating unit112 performs intra prediction again on all blocks included in the current slice using the neighboring pixel values subjected to filtering processing, generating a prediction image. The generated prediction image is supplied to the optimal predictionmode determining unit113 along with the intra prediction mode information.
• In the event that theswitch101 is on, the optimal predictionmode determining unit113 supplies the prediction image of the optimal intra prediction mode, and the corresponding cost function value, to the predictionimage selecting unit77.
• In step S22 inFIG. 16 described above, the predictionimage selecting unit77 determines one of the optimal intra prediction mode and optimal inter prediction mode to be the optimal prediction mode, and supplies selection information of the prediction image.
• In step S107, the optimal predictionmode determining unit113 determines whether or not the prediction image of the optimal intra prediction mode has been selected, in accordance with the selection information of the prediction image. In the event that determination is made in step S107 that the prediction image of the optimal intra prediction mode has been selected, the processing advances to step S108.
• In step S108, the optimal predictionmode determining unit113 supplies the intra prediction mode information to thelossless encoding unit66. Note that, in the event that filter coefficients have not yet been supplied regarding the current slice, the filter coefficient from the optimalfilter calculating unit122 is also supplied to thelossless encoding unit66.
• In the event that determination is made in step S107 that the prediction image of the optimal intra prediction mode has not been selected, the intra prediction processing ends.
• Note that a further optimized filter coefficient can be obtained by repeating the processing of the above-described steps S104 through S106.
• The encoded compressed image is transmitted via a predetermined transmission path, and decoded by the image decoding device.
[Configuration Example of Image Decoding Device]
• FIG. 22 represents the configuration of an embodiment of an image decoding device serving as the image processing device to which the present invention has been applied.
• Animage decoding device151 is configured of an storingbuffer161, alossless decoding unit162, aninverse quantization unit163, an inverseorthogonal transform unit164, acomputing unit165, adeblocking filter166, ascreen rearranging buffer167, a D/A conversion unit168,frame memory169, aswitch170, anintra prediction unit171, a neighboring pixel interpolationfilter switching unit172, a motion prediction/compensation unit173, and aswitch174.
• The storingbuffer161 stores a transmitted compressed image. Thelossless decoding unit162 decodes information supplied from the storingbuffer161 and encoded by thelossless encoding unit66 inFIG. 1 using a format corresponding to the encoding format of thelossless encoding unit66. Theinverse quantization unit163 subjects the image decoded by thelossless decoding unit162 to inverse quantization using a format corresponding to the quantization format of thequantization unit65 inFIG. 1. The inverseorthogonal transform unit164 subjects the output of theinverse quantization unit163 to inverse orthogonal transform using a format corresponding to the orthogonal transform format of theorthogonal transform unit64 inFIG. 1.
• The output subjected to inverse orthogonal transform is decoded by being added with the prediction image supplied from theswitch174 by thecomputing unit165. Thedeblocking filter166 removes the block distortion of the decoded image, then supplies to theframe memory169 for storage, and also outputs to thescreen rearranging buffer167.
• Thescreen rearranging buffer167 performs rearranging of images. Specifically, the sequence of frames rearranged for encoding sequence by thescreen rearranging buffer62 inFIG. 1 is rearranged to the original display sequence. The D/A conversion unit168 converts the image supplied from thescreen rearranging buffer167 from digital to analog, and outputs to an unshown display for display.
• Theswitch170 reads out an image to be subjected to inter processing and an image to be referenced from theframe memory169, outputs to the motion prediction/compensation unit173, and also reads out an image to be used for intra prediction from theframe memory169, and supplies to theintra prediction unit171.
• Information indicating the intra prediction mode obtained by decoding the header information is supplied from thelossless decoding unit162 to theintra prediction unit171. Theintra prediction unit171 generates a prediction image by performing filtering processing and intra prediction on the neighboring pixel values using the filter coefficient set by the neighboring pixel interpolationfilter switching unit172, based on this information, and outputs the generated prediction image to theswitch174.
• The neighboring pixel interpolationfilter switching unit172 is supplied with at least one of information indicating the intra prediction mode obtained by decoding the header information and information of quantization parameter, in accordance with the encoding at the image encoding device51, from thelossless decoding unit162. In the same way as with the neighboring pixel interpolationfilter switching unit75 inFIG. 1, the neighboring pixel interpolationfilter switching unit172 stores filter coefficients corresponding to at least one of the quantization parameter and intra prediction mode, obtained by learning at thelearning device251 inFIG. 28 which will be described later.
• The neighboring pixel interpolationfilter switching unit172 sets a filter coefficient corresponding to at least one of the quantization parameter and intra prediction mode from thelossless decoding unit162. The neighboring pixel interpolationfilter switching unit172 supplies the set filter coefficient to theintra prediction unit74.
• Note that with the neighboring pixel interpolationfilter switching unit172, filer coefficients learned offline beforehand are stored. Note however, in the event that filter coefficients are calculated online with the neighboring pixel interpolationfilter switching unit75 inFIG. 1, these filter coefficients are transmitted hereto for each slice, for example. In this case, the neighboring pixel interpolationfilter switching unit172 uses the filter coefficients decoded by thelossless decoding unit162.
• Information obtained by decoding the header information (prediction mode information, motion vector information, and reference frame information) is supplied from thelossless decoding unit162 to the motion prediction/compensation unit173. In the event that information indicating the inter prediction mode has been supplied, the motion prediction/compensation unit173 subjects the image to motion prediction and compensation processing based on the motion vector information and reference frame information to generate a prediction image. The motion prediction/compensation unit173 outputs the prediction image generated in the inter prediction mode to theswitch174.
• Theswitch174 selects the prediction image generated by the motion prediction/compensation unit173 orintra prediction unit171, and supplies to thecomputing unit165.
• Note that with the image encoding device51 inFIG. 1, intra prediction processing is performed as to all intra prediction modes, for prediction mode determination based on cost function. On the other hand, with theimage decoding device151, intra prediction processing is performed based on only the information of the intra prediction mode sent thereto encoded.
[Configuration Example of Intra Prediction Unit and Neighboring Pixel Interpolation Filter Switching Unit]
• FIG. 23 is a block diagram illustrating a detailed configuration example of the intra prediction unit and neighboring pixel interpolation filter switching unit. Note that the functional blocks inFIG. 23 correspond to the functional blocks in the case of offline processing with the image encoding device51 shown inFIG. 14.
• In the case of the example inFIG. 23, theintra prediction unit171 is configured of a predictionimage generating unit181 and a neighboringpixel setting unit182. The neighboring pixel interpolationfilter switching unit172 is configured of aprediction mode buffer191, aquantization parameter buffer192, and a low-passfilter setting unit193. The low-passfilter setting unit193 hasfilter coefficient memory194 built in.
• The predictionimage generating unit181 is supplied with intra prediction mode information from thelossless decoding unit162 and neighboring pixel values subject to filtering processing from the neighboringpixel setting unit182. The predictionimage generating unit181 uses the neighboring pixel values supplied thereto to perform intra prediction with the intra prediction mode from thelossless decoding unit162, generates a prediction image, and supplies the generated prediction image to theswitch174.
• The neighboringpixel setting unit182 is supplied with the neighboring pixel values of the current block to be subjected to intra prediction, from theframe memory169. In the case ofFIG. 23, illustration of theswitch170 is omitted, but actually, the neighboring pixel values are supplied to the neighboringpixel setting unit182 from theframe memory169 via theswitch170.
• The neighboringpixel setting unit182 uses filter coefficients set by the low-passfilter setting unit193, to subject the neighboring pixel values of the current block from theframe memory169 to filtering processing, and supplies the neighboring pixel values subjected to filtering processing to the predictionimage generating unit181.
• Theprediction mode buffer191 stores the intra prediction mode information from thelossless decoding unit162. Thequantization parameter buffer192 stores the quantization parameter from thelossless decoding unit162.
• The low-passfilter setting unit193 reads out the intra prediction mode information of the current block from theprediction mode buffer191, and reads out the quantization parameter corresponding to the current block from thequantization parameter buffer192. The low-passfilter setting unit193 sets, from the filter coefficients stored in the built-infilter coefficient memory194, a filter coefficient corresponding to this information, and supplies the set filter coefficient to the neighboringpixel setting unit182.
• Thefilter coefficient memory194 stores filter coefficients corresponding to the quantization parameter and intra prediction mode obtained by learning at thelearning device251 inFIG. 28 to be described later, in the same way as with thefilter coefficient memory94 inFIG. 14.
• The filter coefficients are calculated and stored as described above with reference toFIG. 15, for example, for each slice. Note that with thefilter coefficient memory194 as well, the filter coefficients are held as n-bit (where n is an integer) values in accordance with the register length of the processor.
[Description of Decoding Processing of Image Decoding Device]
• Next, the decoding processing that theimage decoding device151 executes will be described with reference to the flowchart inFIG. 24.
• In step S131, the storingbuffer161 stores the transmitted image. In step S132, thelossless decoding unit162 decodes the compressed image supplied from the storingbuffer161. Specifically, the I picture, P picture, and B picture encoded by thelossless encoding unit66 inFIG. 1 are decoded.
• At this time, the motion vector information, reference frame information, prediction mode information (information indicating the intra prediction mode or inter prediction mode), quantization parameter information, flag information, and so forth are also decoded.
• Specifically, in the event that the prediction mode information is intra prediction mode information, the prediction mode information is supplied to theintra prediction unit171 and neighboring pixel interpolationfilter switching unit172. Also, in the event that quantization parameter information has been decoded, this also is supplied to the neighboring pixel interpolationfilter switching unit172. In the event that the prediction mode information is inter prediction mode information, motion vector information and reference frame information corresponding to the prediction mode information are supplied to the motion prediction/compensation unit173.
• In step S133, theinverse quantization unit163 inversely quantizes the transform coefficient decoded by thelossless decoding unit162 using a property corresponding to the property of thequantization unit65 inFIG. 1. In step S134, the inverseorthogonal transform unit164 subjects the transform coefficient inversely quantized by theinverse quantization unit163 to inverse orthogonal transform using a property corresponding to the property of theorthogonal transform unit64 inFIG. 1. This means that difference information corresponding to the input of theorthogonal transform unit64 inFIG. 1 (the output of the computing unit63) has been decoded.
• In step S135, thecomputing unit165 adds the prediction image selected in the processing in later-described step S141 and input via theswitch174, to the difference information. Thus, the original image is decoded. In step S136, thedeblocking filter166 subjects the image output from thecomputing unit165 to filtering. Thus, block distortion is removed. In step S137, theframe memory169 stores the image subjected to filtering.
• In step S138, theintra prediction unit171 and motion prediction/compensation unit173 perform the corresponding image prediction processing in response to the prediction mode information supplied from thelossless decoding unit162.
• Specifically, in the event that the intra prediction mode information has been supplied from thelossless decoding unit162, theintra prediction unit171 performs the intra prediction processing in the intra prediction mode. At this time, theintra prediction unit171 performs filtering processing and intra prediction processing of neighboring pixels using the filter coefficients set by the neighboring pixel interpolationfilter switching unit172.
• The details of the prediction processing in step S138 will be described later with reference toFIG. 25, but according to this processing, the prediction image generated by theintra prediction unit171 or the prediction image generated by the motion prediction/compensation unit173 is supplied to theswitch174.
• In step S139, theswitch174 selects the prediction image. Specifically, the prediction image generated by theintra prediction unit171 or the prediction image generated by the motion prediction/compensation unit173 is supplied. Accordingly, the supplied prediction image is selected, supplied to thecomputing unit165, and in step S134, as described above, added to the output of the inverseorthogonal transform unit164.
• In step S140, thescreen rearranging buffer167 performs rearranging. Specifically, the sequence of frames rearranged for encoding by thescreen rearranging buffer62 of the image encoding device51 is rearranged in the original display sequence.
• In step S141, the D/A conversion unit168 converts the image from thescreen rearranging buffer167 from digital to analog. This image is output to an unshown display, and the image is displayed.
[Description of Prediction Processing]
• Next, the prediction processing in step S138 inFIG. 24 will be described with reference to the flowchart inFIG. 25.
• In step S171, the predictionimage generating unit181 determines whether or not the current block has been subjected to intra encoding. Upon the intra prediction mode information being supplied from thelossless decoding unit162 to the predictionimage generating unit181, in step S171 the predictionimage generating unit181 determines that the current block has been subjected to intra encoding, and the processing proceeds to step S172.
• In step S172, the predictionimage generating unit181 receives and obtains the intra prediction mode information from thelossless decoding unit162. At this time, intra prediction mode information is also supplied to theprediction mode buffer191 and stored.
• Also, upon the quantization parameter information from thelossless decoding unit162 being supplied to thequantization parameter buffer192, in step S173 thequantization parameter buffer192 obtains and stores the quantization parameter.
• The low-passfilter setting unit193 reads out the intra prediction mode information of the current block from theprediction mode buffer191, and reads out the quantization parameter as to the current block from thequantization parameter buffer192. In step S174, the low-passfilter setting unit193 sets, from the filter coefficients for each slice stored in the built-infilter coefficient memory194, a filter coefficient for the neighboring pixels corresponding to this information. The set filter coefficient is supplied to the neighboringpixel setting unit182.
• In step S175 the neighboringpixel setting unit182 uses the filter coefficient set by the low-passfilter setting unit193 to subject the neighboring pixel values of the current block from theframe memory169 to filtering processing, and supplies the neighboring pixel values subjected to filtering processing to the predictionimage generating unit181.
• The predictionimage generating unit181 uses the neighboring pixel values supplied from the neighboringpixel setting unit182 to perform intra prediction in the intra prediction mode obtained in step S172, and generates a prediction image. The generated prediction image is supplied to theswitch174.
• On the other hand, in the event that determination is made in step S171 that intra encoding has not been performed, the processing proceeds to step S177.
• In the event that the image to be processed is an image to be subjected to inter processing, the inter prediction mode information, reference frame information, and motion vector information are supplied from thelossless decoding unit162 to the motion prediction/compensation unit173. In step S177, the motion prediction/compensation unit173 obtains the inter prediction mode information, reference frame information, motion vector information, and so forth from thelossless decoding unit162.
• In step S178, the motion prediction/compensation unit173 then performs inter motion prediction. Specifically, in the event that the image to be processed is an image to be subjected to inter prediction processing, a necessary image is read out from theframe memory169, and supplied to the motion prediction/compensation unit173 via theswitch170. In step S177, the motion prediction/compensation unit173 performs motion prediction in the inter prediction mode to generate a prediction image based on the motion vector obtained in step S176. The generated prediction image is output to theswitch174.
[Other Configuration Example of Intra Prediction Unit and Neighboring Pixel Interpolation Filter Switching Unit]
• FIG. 26 is a block diagram illustrating a detailed configuration example of the intra prediction unit and neighboring pixel interpolation filter switching unit. Note that the functional blocks inFIG. 26 correspond to the functional blocks in the case of online processing with the image encoding device51 shown inFIG. 20.
• In the case of the example inFIG. 26, theintra prediction unit171 is configured of the predictionimage generating unit181 and neighboringpixel setting unit182 inFIG. 23. The neighboring pixel interpolationfilter switching unit172 is configured of theprediction mode buffer191 inFIG. 23, and aninterpolation filter buffer201 and a low-passfilter setting unit202. Note that in the example inFIG. 26, portions corresponding to the case inFIG. 23 are denoted with corresponding reference numerals, and basically perform the same processing, so description thereof will be omitted.
• In the case inFIG. 26, filter coefficients calculated regarding the current slice are encoded and sent from the image encoding device51. Accordingly, thelossless decoding unit162 decodes this along with other information, and supplies to theinterpolation filter buffer201 of the neighboring pixel interpolationfilter switching unit172.
• Theinterpolation filter buffer201 obtains the filter coefficient for the current slice from thelossless decoding unit162 and stores it.
• The low-passfilter setting unit202 reads out the intra prediction mode information of the current block from theprediction mode buffer191. The low-passfilter setting unit202 reads out the filter coefficient corresponding to the intra prediction mode that has been read out, from the filter coefficients of the current slice stored in theinterpolation filter buffer201, and sets this as the filter coefficient for the current block. The set filter coefficient is supplied to the neighboringpixel setting unit182.
[Other Description of Prediction Processing]
• Next, prediction processing in the case of theintra prediction unit171 and the neighboring pixel interpolationfilter switching unit172 will be described with reference to the flowchart inFIG. 27. Note that this intra prediction processing is another example of the prediction processing in step S138 inFIG. 24. Also, the processing of steps S181, S182, and S185 through S188 inFIG. 27 basically perform the same processing as steps S171, S172, and S175 through S178 inFIG. 25, so detailed description thereof will be omitted.
• In step S181, the predictionimage generating unit181 determines whether or not the current block is intra encoded. Upon intra prediction mode information being supplied from thelossless decoding unit162 to the predictionimage generating unit181, in step S181 the predictionimage generating unit181 determines that the current block is intra encoded, and the processing advances to step S182.
• In step S182, the predictionimage generating unit181 receives and obtains the intra prediction mode information from thelossless decoding unit162. At this time, this intra prediction mode information is also supplied to theprediction mode buffer191 and stored.
• Also, upon the information of filter coefficients for the current slice being supplied from thelossless decoding unit162 to theinterpolation filter buffer201, theinterpolation filter buffer201 obtains the filter coefficients for the current slice in step S183, and stores. Note that the filter coefficients are supplied for each slice.
• The low-passfilter setting unit202 reads out the intra prediction mode information for the current block from theprediction mode buffer191. In step S184, the low-passfilter setting unit202 sets, out of the filter coefficients of the current slice stored in theinterpolation filter buffer201, a filter coefficient for the neighboring pixels, corresponding to the intra prediction mode of the current block. The set filter coefficient is supplied to the neighboringpixel setting unit182.
• In step S185, the neighboringpixel setting unit182 uses the filter coefficient set by the low-passfilter setting unit202 to subject the neighboring pixel values of the current block from theframe memory169 to filtering processing, and supplies the neighboring pixel values subjected to filtering processing to the predictionimage generating unit181.
• In step S186, the predictionimage generating unit181 uses the neighboring pixel values supplied from the neighboringpixel setting unit182 to perform intra prediction with the intra prediction mode obtained in step S172, and generates a prediction image. The generated prediction image is supplied to theswitch174.
• On the other hand, in the event that determination is made in step S181 that this is not intra encoded, the processing advances to step S187.
• In step S187, the motion prediction/compensation unit173 obtains inter prediction mode information, reference frame information, motion vector information, and so forth, from thelossless decoding unit162.
• In step S188, the motion prediction/compensation unit173 performs inter motion prediction. Due to this processing, a generated prediction image is output to theswitch174.
• Thus, with the image encoding device51 inFIG. 1 and theimage decoding device151 inFIG. 22, filtering processing is performed on neighboring pixels used for intra prediction using filter coefficients adaptively set for the image, prior to intra processing. For example, filter coefficients are set in accordance to intra prediction mode or quantization parameter.
• Accordingly, noise removal corresponding to images and bit rates can be performed. As a result, prediction efficiency can be improved.
• FIG. 28 represents the configuration of one embodiment of a learning device to which the present invention has been applied. In the example inFIG. 28, alearning device251 performs learning processing of filter coefficients using training image signals.
• Note that training image signals are test images for obtaining filter coefficients, and a standard sequence used for standardization of image compression encoding, obtainable at www.vqeg.org, for example, may be used. Alternatively, the input image corresponding to each application may also be used. For example, in the event that the input is camera signals, learning may be performed using baseband signals imaged using a CCD or CMOS sensor.
• Thelearning device251 inFIG. 28 has in common with the image encoding device51 inFIG. 1 the point of having the A/D conversion unit61,screen rearranging buffer62, computingunit63,orthogonal transform unit64,quantization unit65,lossless encoding unit66, storingbuffer67,inverse quantization unit68, inverseorthogonal transform unit69, computingunit70,deblocking filter71,frame memory72,switch73,intra prediction unit74, motion prediction/compensation unit76, predictionimage selecting unit77, andrate control unit78.
• Also, thelearning device251 differs from the image encoding device51 inFIG. 1 with regard to the points of using training image signals for signals which are used, and including a neighboring pixel interpolationfilter calculating unit261 instead of the neighboring pixel interpolationfilter switching unit75.
• Specifically, with thelearning device251, only blocks included in I pictures are used to perform learning. Alternatively, with thelearning device251, only blocks within intra macro blocks included in B pictures and P pictures are used to perform learning. The former needs less computing amount for learning than the latter. Also, in the case of the former, coefficients obtained for blocks obtained included in I pictures may be applied just to blocks included in I pictures, or may be applied to intra macro blocks included in B pictures and P pictures.
• That is to say, with thelearning device251, learning by just intra prediction with theintra prediction unit74 is performed. Accordingly, we will say that the motion prediction/compensation unit76 does not actually function.
• Further, the neighboring pixel interpolationfilter calculating unit261 inFIG. 29 has in common with the neighboring pixel interpolationfilter switching unit75 inFIG. 20 the point of having theprediction mode buffer121, optimalfilter calculating unit122, and low-passfilter setting unit123.
• On the other hand, the neighboring pixel interpolationfilter calculating unit261 inFIG. 29 differs from the neighboring pixel interpolationfilter switching unit75 inFIG. 20 with regard to the points that a filtercoefficient storage unit271 has been added, and quantization parameters from therate control unit78 being supplied to the optimalfilter calculating unit122.
• Specifically, in the example inFIG. 29, in the same way as with the case of the example inFIG. 20, aswitch101 is provided between theintra prediction unit74 and the neighboring pixel interpolationfilter switching unit75, and theintra prediction unit74 performs intra prediction twice in accordance with on/off of theswitch101.
• That is to say, with theintra prediction unit74, in the off state of theswitch101, intra prediction defined in H.264/AVC is performed, and filter coefficients optimal for the intra prediction mode and quantization parameter are calculated for each slice. The calculated filter coefficients for each slice are stored in the filtercoefficient storage unit271. In the on state of theswitch101, then, intra prediction is performed with a filter condition set by the neighboring pixel interpolationfilter switching unit75 out of the filter coefficients for each slice that have been calculated.
• The filter coefficients stored in this filtercoefficient storage unit271 are stored in the filter coefficient memory94 (FIG. 14) of the image encoding device51 inFIG. 1 and the filter coefficient memory194 (FIG. 23) of theimage decoding device151 inFIG. 22, via storage medium or network or the like.
[Description of Intra Prediction Processing in Learning Processing]
• Next, intra prediction processing which thelearning device251 inFIG. 28 performs as one process of learning processing will be described with reference to the flowchart inFIG. 30. Note that thelearning device251 performs basically the same processing for learning processing as the encoding processing inFIG. 17, except that the prediction processing in step S21 has been replaced with intra prediction processing in step S30.
• Also, steps S201 through S203 and S206 through S209 inFIG. 30 basically perform the same processing as steps S101 through S103 and S105 through S108, so redundant description thereof will be omitted. That is to say, in step S204 inFIG. 30, the optimalfilter calculating unit122 calculates a filter coefficient for each inter prediction mode and corresponding quantization parameter of the current slice which minimizes the residual of the entire slice, as optimal filter coefficients. The calculated filter coefficients are supplied to the filtercoefficient storage unit271.
• In step S205, the filtercoefficient storage unit271 stores the filter coefficients supplied from the optimalfilter calculating unit122.
• The low-passfilter setting unit123 sets, from the filter coefficients of the current slice stored in the filtercoefficient storage unit271, a filter coefficient corresponding to the current block, turns the terminal of theswitch101 on, and supplies the set filter coefficient to the neighboringimage setting unit111.
• Accordingly, in step S206, the set filter coefficient is used to perform filtering processing on the neighboring pixel values of the current block, from theframe memory72.
• Of course, in the same way as with the example inFIG. 21, repeating the processing of steps S204 through S207 described above enables further optimized filter coefficients to be obtained.
• As described above, with thelearning device251, processing the same as with actually-used encoding processing is performed using training image signals, and filter coefficients calculated thereby are to the filtercoefficient storage unit271. Accordingly, optimal filter coefficients can be obtained.
• The filter coefficients stored in this filtercoefficient storage unit271 are stored in the filter coefficient memory94 (FIG. 14) of the image encoding device51 inFIG. 1 and the filter coefficient memory194 (FIG. 23) of theimage decoding device151 inFIG. 22, via storage medium or network or the like.
• Also, with the image encoding device51, as described above, coefficients obtained for blocks obtained included in I pictures (or intra macro blocks included in B pictures and P pictures) may be applied just to blocks included in I pictures. Alternatively, may be applied to intra macro blocks included in B pictures and P pictures, not only blocks included in I pictures.
• Accordingly, high encoding efficient can be realized with the image encoding device51 inFIG. 1 and theimage decoding device151 inFIG. 22.
• Note that with the above-describedorthogonal transform unit64 and inverseorthogonal transform unit69 of the image encoding device51 inFIG. 1 and the inverseorthogonal transform unit164 of theimage decoding device151 inFIG. 22, orthogonal transform/inverse orthogonal transform defined in H.264/AVC is performed. Alternatively, an arrangement may be made wherein theorthogonal transform unit64 and inverseorthogonal transform unit69 of the image encoding device51 inFIG. 1 and the inverseorthogonal transform unit164 of theimage decoding device151 inFIG. 22 perform the orthogonal transform/inverse orthogonal transform proposed inNPL 1.
• Accordingly, the encoding efficiency of the format proposed inNPL 1 can be further improved.
• Note that while an example of performing intra prediction has been described in the above description, the present invention can also be applied to intra prediction in second order prediction proposed inNPL 2 as well.
2. Second Embodiment[Other Configuration Example of Image Encoding Device]
• FIG. 31 shows the configuration of another embodiment of an image encoding device serving as an image processing device to which the present invention has been applied.
• Animage encoding device351 has in common with the image encoding device51 inFIG. 1 the point of having the A/D conversion unit61,screen rearranging buffer62, computingunit63,orthogonal transform unit64,quantization unit65,lossless encoding unit66, storingbuffer67,inverse quantization unit68, inverseorthogonal transform unit69, computingunit70,deblocking filter71,frame memory72,switch73,intra prediction unit74, motion prediction/compensation unit76, predictionimage selecting unit77, andrate control unit78.
• Also, theimage encoding device351 differs from the image encoding device51 inFIG. 1 with regard to the points that the neighboring pixel interpolationfilter switching unit75 is omitted, and a secondorder prediction unit361 and a neighboring pixel interpolationfilter switching unit362 have been added.
• That is to say, with the example inFIG. 31, theintra prediction unit74 performs H.264/AVC intra prediction.
• On the other hand, the motion prediction/compensation unit76 detects motion vectors for all candidate inter prediction modes, based on the image for inter processing and a reference image, subjects the reference image to compensation processing based on the motion vectors, and generates a prediction image.
• The motion prediction/compensation unit76 supplies the secondorder prediction unit361 with detected motion vector information, information of an image for inter processing (address, etc.), and first order residual which is the difference between the image for inter prediction and the generated prediction image.
• The motion prediction/compensation unit76 determines an optimal intra production mode in the second order prediction by comparing the second order residuals from the secondorder prediction unit361. Also, the motion prediction/compensation unit76 determines whether to encode the second order residual or to encode the first order residual, by comparing the second order residual with the first order residual. Note that this processing is performed on all candidate inter prediction modes.
• The motion prediction/compensation unit76 calculates cost function values for all candidate inter prediction modes. At this time, of the first order residual and second order residual, the residual determined for each inter prediction mode is used to calculate the cost function value. The motion prediction/compensation unit76 determines the prediction mode of which yields the smallest value of the calculated cost function values to be the optimal inter prediction mode.
• The motion prediction/compensation unit76 supplies the prediction image generated in the optimal inter prediction mode (or difference between image for inter and second order residual), and the cost function value thereof to the predictionimage selecting unit77. In the event the prediction image generated by the predictionimage selecting unit77 in the optimal inter prediction mode has been selected, the motion prediction/compensation unit76 outputs information indicating the optimal inter prediction mode to thelossless encoding unit66.
• At this time, motion vector information, reference frame information, a second order prediction flag indicating that second order prediction is to be performed, information of the intra prediction mode in second order prediction, and so forth, are also output to thelossless encoding unit66. Thelossless encoding unit66 also subjects the information from the motion prediction/compensation unit76 to lossless encoding processing such as variable length coding, arithmetic coding, or the like, and inserts into the header portion of the compressed image.
• Based on the motion vector information from the motion prediction/compensation unit76 and information of the image to be subjected to inter processing, the secondorder prediction unit361 reads out current neighboring pixels neighboring the current block which is to be subjected to inter processing, from theframe memory72. Also, the secondorder prediction unit361 reads out reference neighboring pixels neighboring the reference block correlated with the current block by motion vector information, from theframe memory72.
• The secondorder prediction unit361 performs second order prediction processing. Second order prediction processing is processing wherein intra prediction is performed between the first order residual and the difference between the current neighboring pixels and reference neighboring pixels, thereby generating information of second order difference (second order residual).
• Now, second order prediction processing will be described with reference toFIG. 32.
• With the example inFIG. 32, a current frame and reference frame as shown, with a current block A shown in the current frame.
• In the event that a motion vector mv(mv_x, mv_y) is obtained in the reference frame and current frame regarding the current block A, difference information (residual) between the current block A and the block correlated with the current bloc A by the motion vector my is calculated.
• With the second order prediction system, not only difference information relating to the current block A, but also difference information between a neighboring pixel group R neighboring the current block A and a neighboring pixel group R1 correlated with the neighboring pixel group R by a motion vector my, is calculated.
• That is to say, the coordinates of the neighboring pixel group R are obtained from the upper left coordinates (x, y) of the current block A. Also, the coordinates of the neighboring pixel group R1 are obtained from the upper left coordinates (x+mv_x, y+mv_y) of the block correlated with the current block A by the motion vector my. Difference information of the neighboring pixel groups is calculated from these coordinate values.
• With the second order prediction system, intra prediction according to the H.264/AVC format is performed between the difference information relating to eh current block thus calculated, and difference information relating to the reference pixels, thereby generating second order different information. The generated second order difference information is subjected to orthogonal transform and quantization, encoded along with the compressed image, and sent to the decoding side.
• Prior to this second order prediction, the secondorder prediction unit361 uses the filter coefficients set by the neighboring pixel interpolationfilter switching unit362 to performing filtering processing on the difference between the current neighboring pixels used of r intra prediction and the reference neighboring pixels. The secondorder prediction unit361 then uses the filtered difference between the current neighboring pixels and reference neighboring pixels subjected to filtering processing to perform second order prediction processing, and outputs the second order difference information (second order residual) to the motion prediction/compensation unit76.
• That is to say, the secondorder prediction unit361 is configured including theintra prediction unit74 shown inFIG. 14, and so forth.
• The neighboring pixel interpolationfilter switching unit362 is configured basically in the same way as the neighboring pixel interpolationfilter switching unit75 inFIG. 1, and perform the same processing. That is to say, the neighboring pixel interpolationfilter switching unit362 sets filter coefficients according to intra prediction mode information from the secondorder prediction unit361 and quantization parameters from therate control unit78, and supplies the set filter coefficients to the secondorder prediction unit361.
• Note that the encoding processing of theimage encoding device351 inFIG. 31 differs from the encoding processing inFIG. 16 performed by the image encoding device51 inFIG. 1 only regarding the following intra processing and motion prediction processing, and other processing is basically the same, so description thereof will be omitted.
• That is to say, with theimage encoding device351 inFIG. 31, intra prediction according to the H.264/AVC format is performed as intra processing. Also, as motion prediction processing, the filter coefficients set by the neighboring pixel interpolationfilter switching unit362 are used at the time of motion prediction processing, thereby generating second order difference information. That with the better second order difference information is selected from the first order difference information and second order difference information, and the optimal inter prediction mode is determined by the cost function values being compared.
• An image decoding device which receives a compressed image encoded by thisimage encoding device351 and decodes this will be described with reference toFIG. 33.
[Other Example of Image Decoding Device]
• FIG. 33 represents the configuration of another embodiment of an image decoding device serving as the image processing device to which the present invention has been applied.
• Animage decoding device401 has in common with theimage decoding device151 inFIG. 22 the point of including the storingbuffer161,lossless decoding unit162,inverse quantization unit163, inverseorthogonal transform unit164, computingunit165,deblocking filter166,screen rearranging buffer167, D/Aconversion unit168,frame memory169,switch170,intra prediction unit171, motion prediction/compensation unit173, andswitch174.
• Also, theimage decoding device401 differs from theimage decoding device151 inFIG. 22 regarding the points of the neighboring pixel interpolationfilter switching unit172 having been omitted, and a secondorder prediction unit411 and a neighboring pixel interpolationfilter switching unit412 having been added.
• That is to say, information indicating the intra prediction mode obtained by decoding header information is supplied from thelossless decoding unit162 to theintra prediction unit171. Based on this information, theintra prediction unit171 generates a prediction image, and outputs the generated prediction image to theswitch174.
• Of the information obtained by decoding header information, the prediction mode information, motion vector information, reference frame information, and so forth, are supplied from thelossless decoding unit162 to the motion prediction/compensation unit173. Also, in the event that second order prediction processing has been applied to the current block, a second order prediction flag indicating that second order prediction is to be performed, and intra mode information for second order prediction, are also supplied from thelossless decoding unit162 to the motion prediction/compensation unit173.
• In the event of having determined that second order prediction processing is applied, the motion prediction/compensation unit173 controls the secondorder prediction unit411 so that second order prediction is performed in the intra prediction mode which the intra prediction mode information for second order prediction indicates.
• The motion prediction/compensation unit173 subjects the image to motion prediction and compensation processing based on the motion vector information and reference frame information, and generates a prediction image. That is to say, a prediction image of the current block is generated using the pixel values of the reference block correlated to the current block, within the reference block. The motion prediction/compensation unit173 then adds the generated prediction image and the prediction difference values from the secondorder prediction unit411, and outputs to these to theswitch174.
• The secondorder prediction unit411 performs second order prediction using the difference between the current neighboring pixels and reference neighboring pixels read out from theframe memory169. Prior to this second order prediction, the secondorder prediction unit411 uses the filter coefficients set by the neighboring pixel interpolationfilter switching unit412 to perform filtering processing on the difference of the current neighboring pixels and reference neighboring pixels. The secondorder prediction unit411 then uses the difference of the current neighboring pixels and reference neighboring pixels subjected to filtering processing to perform second order prediction processing, and outputs the obtained second order difference information (second order residual) to the motion prediction/compensation unit173.
• That is to say, the secondorder prediction unit411 is configured including theintra prediction unit171 shown inFIG. 26.
• The neighboring pixel interpolationfilter switching unit412 is configured basically the same way as with the neighboring pixel interpolationfilter switching unit172. That is to say, the neighboring pixel interpolationfilter switching unit412 sets filter coefficients corresponding to at least one of the quantization parameters from thelossless decoding unit162 and the intra prediction mode. The neighboring pixel interpolationfilter switching unit412 supplies the set filter coefficients to the secondorder prediction unit411.
• Note that the encoding processing of theimage decoding device401 inFIG. 33 only differs from the decoding processing inFIG. 24 of theimage decoding device151 inFIG. 22 except for the following intra processing and motion prediction processing, and other processing is basically the same processing, so description thereof will be omitted.
• That is to say, with theimage decoding device401 inFIG. 33, intra prediction according to the H.264/AVC format is performed as intra processing. Also, as motion prediction processing, second order prediction (intra prediction) is performed at the time of motion prediction processing using the filter coefficients set by the neighboring pixel interpolationfilter switching unit412, and second order difference information is generated.
• The present invention can also be applied to intra prediction in second order prediction processing such as described above.
• Note that in the above description, an example has been described of performing filtering processing on neighboring pixels used for intra prediction, using filter coefficients adaptively set to an image, prior to the intra prediction.
• Now, noise included in neighboring pixels used for the intra prediction system differs depending on encoding conditions, such as the contents of the image, quantization parameters, and so forth. Accordingly, there are blocks regarding which encoding efficiency improves by performing filtering processing performed in the H.264/AVC format for example, and blocks where this is not true.
• Regardless of this, blanket filtering processing has been performed on all blocks at the time of performing 8×8 block-based intra processing on macro blocks with the H.264/AVC format, so there have occurred blocks where encoding efficiency is lowered thereby.
• Accordingly, an example of a case of performing on/off of filtering processing as to neighboring pixels sued for intra prediction will be described next.
3. Third Embodiment[Other Configuration Example of Image Encoding Device]
• FIG. 34 shows the configuration of another embodiment of an image encoding device serving as an image processing device to which the present invention has been applied.
• Animage encoding device451 has in common with the image encoding device51 inFIG. 1 the point of having the A/D conversion unit61,screen rearranging buffer62, computingunit63,orthogonal transform unit64,quantization unit65,lossless encoding unit66, storingbuffer67,inverse quantization unit68, inverseorthogonal transform unit69, computingunit70,deblocking filter71,frame memory72,switch73,intra prediction unit74, motion prediction/compensation unit76, predictionimage selecting unit77, andrate control unit78.
• Also, theimage encoding device451 differs from the image encoding device51 inFIG. 1 with regard to the point that the neighboring pixel interpolationfilter switching unit75 has been replaced with a neighboring pixel interpolationfilter control unit461.
• That is to say, the neighboring pixel interpolationfilter control unit461 performs control of on or off of the blanket filtering processing which has been performed on neighboring pixels of all blocks at the time of performing 8×8 block-based intra processing on macro blocks with the H.264/AVC format. Note that while filtering processing has only been performed with 8×8 block-based intra processing with the H.264/AVC format, this is performed onintra 4×4 and intra 16×16 as well with the neighboring pixel interpolationfilter control unit461.
• The on/off control signals from the neighboring pixel interpolationfilter control unit461 are supplied to theintra prediction unit74.
• Theintra prediction unit74 performs intra prediction processing for all candidate intra prediction modes, based on the image to be subjected to intra prediction that has been read out from thescreen rearranging buffer62 and the reference image supplied from theframe memory72. At this time, at theintra prediction unit74, prior to intra prediction, intra prediction is performed with the filter processing turned on or off in accordance with control signals from the neighboring pixel interpolationfilter control unit461, and the one with a smaller calculated cost function value as a result is employed.
• Further, theintra prediction unit74 generates a flag indicating on or off of the filter processing. This flag information is supplied to thelossless encoding unit66 along with information indicating the optimal intra prediction mode, in the event that the prediction image generated with the optimal intra prediction mode is selected by the predictionimage selecting unit77.
[Configuration Example of Intra Prediction Unit]
• FIG. 35 is a block diagram illustrating a detailed configuration example of theintra prediction unit74 inFIG. 34. In the case of the example inFIG. 35, theintra prediction unit74 is configured of a predictionimage generating unit471, a cost functionvalue generating unit472, and an mode and on/offflag generating unit473.
• The predictionimage generating unit471 is supplied with neighboring pixel values of the current block of intra prediction from theframe memory72. In the case inFIG. 35, theswitch73 is omitted from the illustration, but actually the neighboring pixel values are supplied from theframe memory72 to the predictionimage generating unit471 via theswitch73. Note that in the case of intra prediction, pixel values not subjected to deblocking filtering by thedeblocking filter71 are used as neighboring pixel values.
• Based on control signals from the neighboring pixel interpolationfilter control unit461, the predictionimage generating unit471 performs intra prediction in all candidate intra prediction modes, performing filtering processing on the neighboring pixel values, or not performing filtering processing, thereby generating prediction images. The on/off control by the neighboring pixel interpolationfilter control unit461 indicated by the control signals is performed in increments of blocks or increments of macro blocks as described later with reference toFIG. 36 throughFIG. 38.
• The predictionimage generating unit471 supplies the generated prediction image pixel values, and the intra prediction mode information thereof, to the cost functionvalue generating unit472.
• The cost functionvalue generating unit472 is supplied with original image pixel values from thescreen rearranging buffer62. The cost functionvalue generating unit472 uses the original image pixel values and prediction image pixel values to calculate cost function values for each intra prediction mode, for cases of filtering processing being on and off. The cost functionvalue generating unit472 supplies the calculated cost function values, prediction image pixel values, and intra prediction mode information, to the mode and on/offflag generating unit473.
• The mode and on/offflag generating unit473 uses the cost function values from the cost functionvalue generating unit472 to determine the optimal intra prediction mode, and which of on and off to set the filter processing to, and generates on/off flag information indicating whether the filter processing is on or off.
• The mode and on/offflag generating unit473 supplies the prediction image pixel values of the optima intra prediction mode to the predictionimage selecting unit77. In the event that a prediction image of the optimal intra prediction mode is selected by the predictionimage selecting unit77, the mode and on/offflag generating unit473 supplies information indicating the optimal intra prediction mode and the corresponding on/off flag information to thelossless encoding unit66.
• Note that processing performed by theimage encoding device451 is basically the same as the processing of the image encoding device51 inFIG. 1 except for the intra prediction described next, so redundant description thereof will be omitted.
• Next, the intra prediction processing performed by theintra prediction unit74 inFIG. 34 in the event of on or off control being performed in increments of blocks will be described with reference to the flowchart inFIG. 36. Note that this processing is another example of the intra prediction processing in step S31 inFIG. 17, and that in the example inFIG. 36, an example ofintra 4×4 will be described. Also, in the following filter processing on/off may also be referred to simply as filter on/off.
• In step S401, the cost functionvalue generating unit472 generates cost function values for the current block, for each of the nine types of intra prediction modes shown inFIG. 3 orFIG. 4.
• That is to say, the predictionimage generating unit471 is supplied with neighboring pixel values of the current block for intra prediction from theframe memory72. The predictionimage generating unit471 performs intra prediction on each of the nine types of intra prediction modes shown inFIG. 3 orFIG. 4, and generates prediction images for the current block.
• At this time, control signals to the effect that filtering processing is not to be performed as to the neighboring pixels is supplied from the neighboring pixel interpolationfilter control unit461, and the predictionimage generating unit471 performs intra prediction without performing filtering processing on the neighboring pixels. Note that here, an arrangement may be made wherein control signals to the effect that filtering processing is to be performed on the neighboring pixels are supplied. Note however, different control is not made regarding the nine modes, such as performing on Vertical but not performing on Horizontal; rather, the same control regarding performing or not performing is made for the nine modes. Also note here that not performing filtering processing on all modes allows for less computation amount.
• The predictionimage generating unit471 supplies the generated prediction image pixel values and the intra prediction mode information to the cost functionvalue generating unit472. The cost functionvalue generating unit472 uses the original image pixel values from thescreen rearranging buffer62 and prediction image pixel values to calculate the cost function values shown in the above-described Expression (73) or Expression (74) for each intra prediction mode in the case of filtering being off. The cost functionvalue generating unit472 supplies the calculated cost function values, prediction image pixel values, and intra prediction mode information, to the mode and on/offflag generating unit473.
• In step S402, the mode and on/offflag generating unit473 uses the cost function values from the cost functionvalue generating unit472 to select an optimal intra prediction mode for the current block. The selected intra prediction mode information is supplied to the neighboring pixel interpolationfilter control unit461.
• In step S403, the neighboring pixel interpolationfilter control unit461 causes the cost functionvalue generating unit472 to generate cost function values for the selected intra prediction mode with the filtering on and off. Note that in step S401, the cost function value for filtering being off has been generated, so actually in step S403, the const function value with filtering on is generated.
• That is to say, the neighboring pixel interpolationfilter control unit461 supplies filtering on control signals and the selected intra prediction mode information to the predictionimage generating unit471. The predictionimage generating unit471 performs filtering processing as to the neighboring pixels used for the selected intra prediction mode, performs intra prediction in the selected intra prediction mode, and generates a prediction image of the current block.
• The predictionimage generating unit471 supplies the generated prediction image pixel values and the selected intra prediction mode information to the cost functionvalue generating unit472. the cost functionvalue generating unit472 uses the original image pixel values from thescreen rearranging buffer62 and prediction image pixel values to calculate the cost function values shown in the above-described Expression (73) or Expression (74) for the selected intra prediction mode in the case of filtering being on. The cost functionvalue generating unit472 supplies the calculated cost function values and prediction image pixel values to the mode and on/offflag generating unit473.
• In step S404, the mode and on/offflag generating unit473 determines filtering on/off for the current block by comparing the cost function values with filtering on and off for the selected intra prediction mode. That is to say, in the event that the cost function value for filtering on is smaller, filtering on is determined for the current block and in the event that the cost function value for filtering off is smaller, filtering off is determined for the current block. The mode and on/offflag generating unit473 then supplies the determined prediction image values to the predictionimage selecting unit77.
• In step S405, the mode and on/offflag generating unit473 generates an on/off flag indicating the on or off decided upon in step S404 for the current block. For example, in the case of filtering on, the on/off filter value is 1. In the case of filtering off, the on/off filter value is 0.
• In the event that a prediction image in the intra prediction mode has been selected in step S22 inFIG. 16 described above, the generated on/off flag information is supplied to thelossless encoding unit66 along with the information indicating the optimal intra prediction mode. The supplied information is encoded in step S23 inFIG. 16, added to the header of the compressed image, and sent to the decoding side.
• Next, another example of intra prediction processing of theintra prediction unit74 inFIG. 34 in the case of on or off control being performed in increments of blocks will be described with reference to the flowchart inFIG. 37. In the case of the example inFIG. 37 as well, an example ofintra 4×4 will be described.
• In step S421, the cost functionvalue generating unit472 generates cost function values for the current block according to filtering being on and off, for each of the intra prediction modes.
• That is to say, the predictionimage generating unit471 is supplied with the neighboring pixel values of the current block for intra prediction from theframe memory72. The predictionimage generating unit471 performs intra processing on each of the nine types of intra processing modes shown inFIG. 3 orFIG. 4, and generates prediction images of the current block.
• At this time, first, control signals to the effect that filtering processing is not to be performed as to the neighboring pixels are supplied from the neighboring pixel interpolationfilter control unit461, and the predictionimage generating unit471 performs intra prediction on each of the intra prediction modes without performing filtering processing as to the neighboring pixels, and generates prediction images. Further, control signals to the effect that filtering processing is to be performed as to the neighboring pixels are supplied from the neighboring pixel interpolationfilter control unit461, and the predictionimage generating unit471 performs intra prediction on each of the intra prediction modes having performed filtering processing as to the neighboring pixels, and generates prediction images.
• The predictionimage generating unit471 supplies the information of each of the intra prediction modes with filtering on and off, and the corresponding prediction image pixel values, to the cost functionvalue generating unit472. The cost functionvalue generating unit472 uses the original image pixel values from thescreen rearranging buffer62 and prediction image pixel values to calculate the cost function values shown in the above-described Expression (73) or Expression (74) for each intra prediction mode in each case of filtering being off and being on. The cost functionvalue generating unit472 supplies the calculated cost function values, prediction image pixel values, and intra prediction mode information, in each case of filtering being off and being on, to the mode and on/offflag generating unit473.
• In step S422, the mode and on/offflag generating unit473 uses the cost function values from the cost functionvalue generating unit472 to determine whether filtering should be on or off for the current block with each intra prediction mode.
• In step S423, the mode and on/offflag generating unit473 selects the optimal intra prediction mode for the current block out from the intra prediction modes regarding which determination has been made for filtering on or off.
• In step S424, the mode and on/offflag generating unit473 generates on/off flag information indicating the state of the filter for the selected intra prediction mode (on or off). In the event that a prediction image in the intra prediction mode has been selected in step S22 inFIG. 16 described above, the generated on/off flag information is supplied to thelossless encoding unit66 along with the information indicating the optimal intra prediction mode. The supplied information is encoded in step S23 inFIG. 16, added to the header of the compressed image, and sent to the decoding side.
• Next, the intra prediction processing performed by theintra prediction unit74 inFIG. 34 in the event of on or off control being performed in increments of macro blocks will be described with reference to the flowchart inFIG. 38. Note that this processing is another example of the intra prediction processing in step S31 inFIG. 17, and that an example ofintra 4×4 will be described in the example inFIG. 38 as well.
• In step S451, the neighboring pixel interpolationfilter control unit461 fixes the filtering as to the entire macro block to off or on. In this case, the neighboring pixel interpolationfilter control unit461 fixes the filtering to off, and supplies control signals for filtering off to the predictionimage generating unit471. Fixing of filtering may be either on or off, but fixing to off can be realized with less calculation amount.
• In step S452, theintra prediction unit74 determines the intra prediction mode for each block. That is to say, the predictionimage generating unit471 is supplied with neighboring pixel values of the current block for intra prediction from theframe memory72. The predictionimage generating unit471 performs intra processing on each of the nine types of intra processing modes shown inFIG. 3 orFIG. 4, and generates prediction images of the current block.
• At this time, first, control signals to the effect that filtering processing is not to be performed as to the neighboring pixels are supplied from the neighboring pixel interpolationfilter control unit461, and the predictionimage generating unit471 performs intra prediction on each of the intra prediction modes without performing filtering processing as to the neighboring pixels, and generates prediction images. The predictionimage generating unit471 supplies the generated prediction image pixel values and the intra prediction mode information thereof to the cost functionvalue generating unit472.
• The cost functionvalue generating unit472 uses the original image pixel values from thescreen rearranging buffer62 and prediction image pixel values to calculate the cost function values shown in the above-described Expression (73) or Expression (74) for each intra prediction mode in each case of filtering being off. The cost functionvalue generating unit472 supplies the calculated cost function values, prediction image pixel values, and intra prediction mode information, in each case of filtering being off to the mode and on/offflag generating unit473.
• The mode and on/offflag generating unit473 uses the cost function values from the cost functionvalue generating unit472 to determine an optimal intra prediction mode for each block. The decoded intra prediction mode information is supplied to the neighboring pixel interpolationfilter control unit461.
• In step S453, the neighboring pixel interpolationfilter control unit461 causes the cost functionvalue generating unit472 to generate cost function values for filtering on and off for the entire macro block. Note that the cost function values for the optimal intra prediction mode for each block within the macro block (i.e., the entire macro block) with filtering off has been generated in step S452. Accordingly, actually in step S453, the cost function values for the entire macro block with filtering on are generated.
• That is to say, the neighboring pixel interpolationfilter control unit461 supplies filtering on control signals, and information of the intra prediction mode determined for each of the blocks, to the predictionimage generating unit471. The predictionimage generating unit471 performs filtering processing as to the neighboring pixel values used in the determined intra prediction mode, performs intra prediction with the determined intra prediction mode, and generates a prediction image for the current block.
• The predictionimage generating unit471 supplies the generated prediction image pixel values and determined intra prediction mode information to the cost functionvalue generating unit472. The cost functionvalue generating unit472 uses the original image pixel values from thescreen rearranging buffer62 and prediction image pixel values to calculate the cost function values shown in the above-described Expression (73) or Expression (74) for the determined intra prediction mode in each case of filtering being on. The cost functionvalue generating unit472 supplies the calculated cost function values, prediction image pixel values, and intra prediction mode information, in each case of filtering being on and off, to the mode and on/offflag generating unit473.
• In step S454, the mode and on/offflag generating unit473 compares the cost function value of all blocks within the macro block in the cases of filtering on and off from the cost functionvalue generating unit472, and determines which of filtering on/off to apply to the entire macro block.
• In step S455, the mode and on/offflag generating unit473 generates an on/off flag indicating the on or off decided in step S454, as to the entire macro block. The generated on/off flag information is supplied to thelossless encoding unit66 for each macro block. The supplied information is encoded in step S23 inFIG. 16, added to the header of the compressed image, and sent to the decoding side.
• As described above, control of filtering on/off (on or off) can be performed in increments of blocks, or can be performed in increments of macro blocks. Note that while the prediction precision of intra prediction processing can be improved by controlling on/off in increments of blocks, the amount of information necessary for transmitting the flag information for each block increases. Conversely, with the case of controlling in increments of macro blocks, the improvement in prediction precision is lower than that of performing in increments of blocks, but one flag information per macro block is sufficient, is increase in flag information amount can be reduced.
• While an example of luminance signals has been described in the above description, this can be used for intra prediction regarding color difference signals as well. Also, filter coefficients in the filtering processing to be controlled are not restricted to the three-tap {1,2,1}//4 in the H.264/AVC format, and this can be applied to any coefficients of any tap length set with the image encoding device51 inFIG. 1.
• That is to say, in the case of filtering on, filtering processing with the filter coefficients set by the image encoding device51 inFIG. 1 may also be performed.
• An image decoding device which receives a compressed image encoded by theimage encoding device451 and decodes this will be described with reference toFIG. 39.
[Other Example of Image Decoding Device]
• FIG. 39 illustrates the configuration of another embodiment of an image decoding device as an image processing device to which the present invention has been applied.
• An image decoding device501 has in common with theimage decoding device151 inFIG. 22 the point of including the storingbuffer161,lossless decoding unit162,inverse quantization unit163, inverseorthogonal transform unit164, computingunit165,deblocking filter166,screen rearranging buffer167, D/Aconversion unit168,frame memory169,switch170,intra prediction unit171, motion prediction/compensation unit173, andswitch174.
• Also, the image decoding device501 differs from theimage decoding device151 inFIG. 22 regarding the point of the neighboring pixel interpolationfilter switching unit172 having been replaced with a neighboring pixel interpolationfilter control unit511.
• That is to say, information indicating the intra prediction mode obtained by decoding header information is supplied from thelossless decoding unit162 to theintra prediction unit171. Based on this information, theintra prediction unit171 generates a prediction image, and outputs the generated prediction image to theswitch174. At this time. Prior to intra prediction, theintra prediction unit171 performs (or does not perform) filtering processing as to the neighboring pixel values in accordance with control signals from the neighboring pixel interpolationfilter control unit511.
• The neighboring pixel interpolationfilter control unit511 is supplied with on/off flag information for each macro block or each block from thelossless decoding unit162, in accordance with the encoding at theimage encoding device451. The neighboring pixel interpolationfilter control unit511 supplies theintra prediction unit171 with control signals to the effect that filtering processing is to be performed or not performed, in accordance with the supplied on/off flag information.
• Note that with theimage encoding device451 inFIG. 34, both cases of filtering being on and being off are tested, and intra prediction processing is performed having selected the one which yields higher encoding efficiency by cost function values. On the other hand, with the image decoding device501, filtering on or off is controlled based on the on/off flag information sent encoded, and intra prediction processing is performed.
[Configuration Example of Intra Prediction Unit and Neighboring Pixel Interpolation Filter Control Unit]
• FIG. 40 is a block diagram illustrating a detailed configuration example of the intra prediction unit and neighboring pixel interpolation filter control unit.
• In the case of the example inFIG. 40, theintra prediction unit171 is configured of aprediction mode buffer521 and predictionimage generating unit522. The neighboring pixel interpolationfilter control unit511 is configured of aflag buffer531 and controlsignal generating unit532.
• Theprediction mode buffer521 is supplied with the intra prediction mode information from thelossless decoding unit162. The predictionimage generating unit522 is supplied with neighboring pixel values of the current block for intra prediction from theframe memory169. In the case ofFIG. 40 as well, theswitch170 is omitted from the illustration, but actually, the neighboring pixel values are supplied from theframe memory169 to the predictionimage generating unit522 via theswitch170.
• The predictionimage generating unit522 reads out the intra prediction mode information for the current block from theprediction mode buffer521, performs intra prediction on the current block in the intra prediction mode that has been read out, and generates a prediction image. Prior to this intra prediction, the predictionimage generating unit522 performs filtering processing on the prediction image pixel values from theframe memory169 in accordance to the control signals from the controlsignal generating unit532.
• Theflag buffer531 is supplied with on/off flag information from thelossless decoding unit162 for each macro block or each block. The controlsignal generating unit532 reads out the corresponding on/off flag from theflag buffer531, generates control signals indicating whether to or not to perform filtering processing for each block, and supplies the generated control signals to the predictionimage generating unit522.
• Note that the processing performed by the image decoding device501 is basically the same as the processing of theimage decoding device151 inFIG. 22 except for the prediction processing described next, so redundant description thereof will be omitted.
[Description of Prediction Processing]
• Next, prediction processing of the image decoding device501 inFIG. 39 will be described with reference to the flowchart inFIG. 41. Note that this intra prediction processing is another example of the prediction processing in step S138 inFIG. 24.
• In step S501, the predictionimage generating unit522 determines whether or not the current block is intra encoded. Intra prediction mode information is supplied from thelossless decoding unit162 to theprediction mode buffer521, which is read by the predictionimage generating unit522. Accordingly, in step S501, the predictionimage generating unit522 determines that the current block is intra encoded, and the processing advances to step S502.
• In step S502, the predictionimage generating unit522 obtains the intra prediction mode information of theprediction mode buffer521.
• Also, upon on/off flag information from thelossless decoding unit162 being supplied to theflag buffer531, theflag buffer531 obtains an prediction image pixel values flag in step S503, and stores this.
• The controlsignal generating unit532 reads out the on/off flag corresponding to the current flag from theflag buffer531, and in step S504 determines whether or not the on/off flag is 1. In the event that determination is made in step S504 that the on/off flag is 1, i.e., that the filtering processing is on, the controlsignal generating unit532 supplies control signals to the predictionimage generating unit522 to cause filtering processing to be performed.
• In accordance with the control signals, in step S505 the predictionimage generating unit522 subjects the neighboring pixels to filtering processing using filter coefficients. In step S506, the predictionimage generating unit522 performs intra prediction using the neighboring pixel values subjected to filtering processing, and generates a prediction image.
• On the other hand, in the event that determination is made in step S504 that the on/off flag is not 1, i.e., that the filtering processing is off, the filtering processing in step S505 is skipped, and the processing advances to step S506.
• In step S506, the predictionimage generating unit522 uses the prediction image pixel values from theframe memory169 to perform intra prediction, and generates a prediction image.
• The prediction image generated in step S506 is supplied to theswitch174.
• On the other hand, in the event that determination is made in step S501 that intra encoding is not performed, the processing advances to step S507.
• In step S507, the motion prediction/compensation unit173 performs inter motion prediction. That is to say, in the event that the image to be processed is an image for inter prediction processing, the necessary image is read out from theframe memory169, and supplied to the motion prediction/compensation unit173 via theswitch170. In step S508, the motion prediction/compensation unit173 performs motion prediction in the inter prediction mode based on the motion vector obtained in step S507, and generates a prediction image. The generated prediction image is output to theswitch174.
• As described above, with theimage encoding device451 and image decoding device501, on and off of filtering processing as to the neighboring pixels used for intra prediction is controlled, and filtering processing is not performed for blocks where encoding efficiency deteriorates. Accordingly, the encoding efficiency can be improved.
• Note that while an example of performing intra prediction has been described in the above description, control of filtering processing on and off can be applied to the intra prediction in the second order prediction described above with reference toFIG. 32.
4. Fourth Embodiment[Other Configuration Example of Image Encoding Device]
• FIG. 42 shows the configuration of another embodiment of an image encoding device serving as an image processing device to which the present invention has been applied.
• Animage encoding device551 has in common with theimage encoding device451 inFIG. 34 the point of having the A/D conversion unit61,screen rearranging buffer62, computingunit63,orthogonal transform unit64,quantization unit65,lossless encoding unit66, storingbuffer67,inverse quantization unit68, inverseorthogonal transform unit69, computingunit70,deblocking filter71,frame memory72,switch73,intra prediction unit74, motion prediction/compensation unit76, predictionimage selecting unit77, andrate control unit78.
• Also, theimage encoding device551 differs from theimage encoding device451 inFIG. 34 with regard to the points that the neighboring pixel interpolationfilter control unit461 has been omitted and that the secondorder prediction unit361 inFIG. 31 and a neighboring pixel interpolationfilter control unit561 have been added.
• That is to say, with the example inFIG. 42, theintra prediction unit74 performs intra prediction according to H.264/AVC.
• On the other hand, the motion prediction/compensation unit76 detects motion vectors for all candidate inter prediction modes, based on the image for inter processing and a reference image, subjects the reference image to compensation processing based on the motion vectors, and generates a prediction image.
• The motion prediction/compensation unit76 supplies the secondorder prediction unit361 with detected motion vector information, information of an image for inter processing (address, etc.), and first order residual which is the difference between the image for inter prediction and the generated prediction image.
• The motion prediction/compensation unit76 determines an optimal intra production mode in the second order prediction by comparing the second order residuals from the secondorder prediction unit361. Also, the motion prediction/compensation unit76 determines whether to encode the second order residual or to encode the first order residual, by comparing the second order residual with the first order residual. Note that this processing is performed on all candidate inter prediction modes.
• The motion prediction/compensation unit76 calculates cost function values for all candidate inter prediction modes. At this time, of the first order residual and second order residual, the residual determined for each inter prediction mode is used to calculate the cost function value. The motion prediction/compensation unit76 determines the prediction mode of which yields the smallest value of the calculated cost function values to be the optimal inter prediction mode.
• The motion prediction/compensation unit76 supplies the prediction image generated in the optimal inter prediction mode (or difference between image for inter and second order residual), and the cost function value thereof to the predictionimage selecting unit77. In the event that the prediction image generated by the predictionimage selecting unit77 in the optimal inter prediction mode has been selected, the motion prediction/compensation unit76 outputs information indicating the optimal inter prediction mode to thelossless encoding unit66.
• At this time, motion vector information, reference frame information, a second order prediction flag indicating that second order prediction is to be performed, information of the intra prediction mode in second order prediction, and so forth, are also output to thelossless encoding unit66. Thelossless encoding unit66 also performs lossless encoding processing such as variable length encoding, arithmetic encoding, and so forth, on the information from the motion prediction/compensation unit76, and inserts this to the header portion of the compressed image.
• Based on the motion vector information from the motion prediction/compensation unit76 and information of the image to be subjected to inter processing, the secondorder prediction unit361 reads out current neighboring pixels neighboring the current block which is to be subjected to inter processing, from theframe memory72. Also, the secondorder prediction unit361 reads out reference neighboring pixels neighboring the reference block correlated with the current block by motion vector information, from theframe memory72.
• The secondorder prediction unit361 performs second order prediction processing described above with reference toFIG. 32. Second order prediction processing is processing wherein intra prediction is performed between the first order residual and the difference between the current neighboring pixels and reference neighboring pixels, thereby generating information of second order difference (second order residual).
• Note however, that prior to this second order prediction, the secondorder prediction unit361 inFIG. 42 performs (or does not perform) filtering processing as to the difference between current neighboring pixels and reference neighboring pixels used for inter prediction, in accordance with control signals from the neighboring pixel interpolationfilter control unit561. The secondorder prediction unit361 then uses the filtered (or unfiltered) difference between current neighboring pixels and reference neighboring pixels to perform second order prediction processing, and outputs the obtained second order difference information (second order residual) to the motion prediction/compensation unit76. At this time, the secondorder prediction unit361 also outputs on/off flag information indicating whether or not to perform filtering processing to the motion prediction/compensation unit76.
• That is to say, the secondorder prediction unit361 includes theintra prediction unit74 shown inFIG. 35.
• The neighboring pixel interpolationfilter control unit561 is configured basically the same way as with the neighboring pixel interpolationfilter control unit461, and performs the same processing. That is to say, the neighboring pixel interpolationfilter control unit561 supplies control signals effecting control whether or not to perform filtering processing in increments of blocks or increments of macro blocks to the secondorder prediction unit361.
• Note that processing performed by theimage encoding device551 inFIG. 42 is basically the same as the processing of theimage encoding device451 inFIG. 34 (i.e., the encoding processing inFIG. 16) except for the following intra processing and motion prediction processing, so description thereof will be omitted.
• That is to say, with theimage encoding device551 inFIG. 42, intra prediction according to the H.264/AVC format is performed as intra processing. Also, as motion prediction processing, filtering processing is controlled in accordance with control signals from the neighboring pixel interpolationfilter control unit561 at the time of motion prediction processing, whereby filtered (or unfiltered) second order difference information is generated. Of the first order difference information and second order difference information, that with better encoding efficiency is selected, and cost function values are compared, whereby an optimal inter prediction mode is determined.
• An image decoding device which receives a compressed image encoded by thisimage encoding device551 and decodes this will be described with reference toFIG. 43.
[Other Configuration Example of Image Decoding Device]
• FIG. 43 illustrates the configuration of another embodiment of an image decoding device as an image processing device to which the present invention has been applied.
• Animage decoding device601 has in common with the image decoding device501 inFIG. 39 the point of including the storingbuffer161,lossless decoding unit162,inverse quantization unit163, inverseorthogonal transform unit164, computingunit165,deblocking filter166,screen rearranging buffer167, D/Aconversion unit168,frame memory169,switch170,intra prediction unit171, motion prediction/compensation unit173, andswitch174.
• Also, theimage decoding device601 differs from the image decoding device501 inFIG. 39 regarding the points that the neighboring pixel interpolationfilter control unit511 has been omitted, and that the secondorder prediction unit411 inFIG. 33 and a neighboring pixel interpolationfilter control unit611 have been added.
• That is to say, information indicating the intra prediction mode obtained by decoding header information is supplied from thelossless decoding unit162 to theintra prediction unit171. Based on this information, theintra prediction unit171 generates a prediction image, and outputs the generated prediction image to theswitch174.
• Of the information obtained by decoding the header information, the prediction mode information, motion vector information, reference frame information, and so forth, are supplied from thelossless decoding unit162 to the motion prediction/compensation unit173. Also, in the event that second order prediction processing has been applied to the current block, a second order prediction flag and intra prediction mode information for the second order prediction are also supplied from thelossless decoding unit162 to the motion prediction/compensation unit173.
• In the event of having determined that second order prediction processing has been applied, the motion prediction/compensation unit173 controls the secondorder prediction unit411 so as to perform second order prediction with the intra prediction mode which the intra prediction mode information of second order prediction indicates.
• The motion prediction/compensation unit173 performs motion prediction and compensation processing on the image based on the motion vector information and reference frame information, and generates a prediction image. That is to say, a prediction image of the current block is generated using the pixel values of the reference block correlated to the current block, within the reference block. The motion prediction/compensation unit173 then adds the generated prediction image and the prediction difference values from the secondorder prediction unit411, and outputs to these to theswitch174.
• The secondorder prediction unit411 performs second order prediction using the difference between the current neighboring pixels and reference neighboring pixels read out from theframe memory169. Note however, in the event that control signals effecting control to perform filtering processing have been received from the neighboring pixel interpolationfilter control unit611, the secondorder prediction unit411 performs filtering processing on the difference of the current neighboring pixels and reference neighboring pixels prior to this second order prediction. The secondorder prediction unit411 then uses the difference of the current neighboring pixels and reference neighboring pixels subjected to filtering processing to perform second order prediction processing, and outputs the obtained second order difference information (second order residual) to the motion prediction/compensation unit173.
• Note that in the event that control signals effecting control not to perform filtering processing have been received from the neighboring pixel interpolationfilter control unit611, the secondorder prediction unit411 does not perform filtering processing, and performs second order processing using the difference of the current neighboring pixels and reference neighboring pixels.
• That is to say, the secondorder prediction unit411 is configured including theintra prediction unit171 shown inFIG. 40.
• The neighboring pixel interpolationfilter control unit611 is configured basically in the same way as the neighboring pixel interpolationfilter control unit511 inFIG. 39, and basically performs the same processing. That is to say, the neighboring pixel interpolationfilter control unit611 is supplied with, of the information obtained by decoding the header information, the on/off flag information, from thelossless decoding unit162. The neighboring pixel interpolationfilter control unit611 supplies control signals to cause the secondorder prediction unit411 to perform or not perform filtering processing of the neighboring pixels, in accordance with the on/off flag information.
• Note that processing performed by theimage decoding device601 inFIG. 43 is basically the same as the processing of the image decoding device501 inFIG. 39 (i.e., the decoding processing inFIG. 24) except for the following intra processing and motion prediction processing, so description thereof will be omitted.
• That is to say, with theimage decoding device601 inFIG. 43, intra prediction according to the H.264/AVC format is performed as intra processing. Also, as motion prediction processing, filtering processing is controlled in accordance with control signals from the neighboring pixel interpolationfilter control unit611 at the time of motion prediction processing, second order prediction (intra prediction) is performed, and second order difference information is generated.
• The on and off control of filtering processing can also be applied to intra prediction with second order prediction processing such as described above, as well.
• Note that while description has been made with the above description regarding a case that the size of macro blocks is 16×16 pixels, but the present invention can be applied to the extended macro block sizes described inPTL 3 as well.
• FIG. 44 is a diagram illustrating an example of an extended macro block size. WithPTL 3, the macro block size is extended up to 32×32 pixels.
• Macro blocks made up of 32×32 pixels divided into blocks (partitions) of 32×32 pixels, 32×16 pixels, 16×32 pixels, and 16×16 pixels are shown from the left in order on the upper tier inFIG. 44. Blocks made up of 16×16 pixels divided into blocks of 16×16 pixels, 16×8 pixels, 8×16 pixels, and 8×8 pixels are shown from the left in order on the middle tier inFIG. 44. Also, blocks made up of 8×8 pixels divided into blocks of 8×8 pixels, 8×4 pixels, 4×8 pixels, and 4×4 pixels are shown from the left in order on the lower tier inFIG. 44.
• In other words, the macro blocks of 32×32 pixels may be processed with blocks of 32×32 pixels, 32×16 pixels, 16×32 pixels, and 16×16 pixels shown on the upper tier inFIG. 44.
• Also, the blocks of 16×16 pixels shown on the right side on the upper tier may be processed with blocks of 16×16 pixels, 16×8 pixels, 8×16 pixels, and 8×8 pixels shown on the middle tier in the same way as with the H.264/AVC format.
• Further, the blocks of 8×8 pixels shown on the right side on the middle tier may be processed with blocks of 8×8 pixels, 8×4 pixels, 4×8 pixels, and 4×4 pixels shown on the lower tier in the same way as with the H.264/AVC format.
• With the extended macro block sizes, by employing such a hierarchical structure, regarding a 16×16 pixel block or less, a greater block is defined as a superset thereof while maintaining compatibility with the H.264/AVC format.
• The filter coefficient setting, calculation, and filter processing on/off control, according to the present invention, may also be applied to the proposed macro block sizes extended as described above.
• Description has been made so far with the H.264/AVC format employed as a encoding format, but the present invention is not restricted to this, and another encoding format/decoding format for performing prediction using neighboring pixels (e.g., intra prediction or second order prediction) may be employed.
• Note that the present invention may be applied to an image encoding device and an image decoding device used at the time of receiving image information (bit streams) compressed by orthogonal transform such as discrete cosine transform or the like and motion compensation via a network medium such as satellite broadcasting, a cable television, the Internet, a cellular phone, or the like, for example, as with MPEG, H.26x, or the like. Also, the present invention may be applied to an image encoding device and an image decoding device used at the time of processing image information on storage media such as an optical disc, a magnetic disk, and flash memory. Further, the present invention may be applied to a motion prediction compensation device included in such an image encoding device and an image decoding device.
• The above-mentioned series of processing may be executed by hardware, or may be executed by software. In the event of executing the series of processing by software, a program making up the software thereof is installed in a computer. Here, examples of the computer include a computer built into dedicated hardware, and a general-purpose personal computer whereby various functions can be executed by installing various programs therein.
• FIG. 45 is a block diagram illustrating a configuration example of the hardware of a computer which executes the above-mentioned series of processing using a program.
• With the computer, a CPU (Central Processing Unit)801, ROM (Read Only Memory)802, and RAM (Random Access Memory)803 are mutually connected by abus804.
• Further, an input/output interface805 is connected to thebus804. Aninput unit806, anoutput unit807, astorage unit808, acommunication unit809, and adrive810 are connected to the input/output interface805.
• Theinput unit806 is made up of a keyboard, a mouse, a microphone, and so forth. Theoutput unit807 is made up of a display, a speaker, and so forth. Thestorage unit808 is made up of a hard disk, nonvolatile memory, and so forth. Thecommunication unit809 is made up of a network interface and so forth. Thedrive810 drives aremovable medium811 such as a magnetic disk, an optical disc, a magneto-optical disk, semiconductor memory, or the like.
• With the computer thus configured, for example, theCPU801 loads a program stored in thestorage unit808 to theRAM803 via the input/output interface805 andbus804, and executes the program, and accordingly, the above-mentioned series of processing is performed.
• The program that the computer (CPU801) executes may be provided by being recorded in theremovable medium811 serving as a package medium or the like. Also, the program may be provided via a cable or wireless transmission medium such as a local area network, the Internet, or digital broadcasting.
• With the computer, the program may be installed in thestorage unit808 via the input/output interface805 by mounting theremovable medium811 on thedrive810. Also, the program may be received at thecommunication unit809 via a cable or wireless transmission medium, and installed in thestorage unit808. Additionally, the program may be installed in theROM802 orstorage unit808 beforehand.
• Note that the program that the computer executes may be a program wherein the processing is performed in the time sequence along the sequence described in the present Specification, or may be a program wherein the processing is performed in parallel or at necessary timing such as when call-up is performed.
• The embodiments of the present invention are not restricted to the above-mentioned embodiment, and various modifications may be made without departing from the essence of the present invention.
• For example, the above-mentionedimage encoding devices51,351,451, and551, and theimage decoding devices151,401,501, and601 may be applied to optional electronic devices. Hereafter, an example thereof will be described.
• FIG. 46 is a block diagram illustrating a principal configuration example of a television receiver using the image decoding device to which the present invention has been applied.
• Atelevision receiver1300 shown inFIG. 46 includes aterrestrial tuner1313, avideo decoder1315, a videosignal processing circuit1318, agraphics generating circuit1319, apanel driving circuit1320, and adisplay panel1321.
• Theterrestrial tuner1313 receives the broadcast wave signals of a terrestrial analog broadcast via an antenna, demodulates, obtains video signals, and supplies these to thevideo decoder1315. Thevideo decoder1315 subjects the video signals supplied from theterrestrial tuner1313 to decoding processing, and supplies the obtained digital component signals to the videosignal processing circuit1318.
• The videosignal processing circuit1318 subjects the video data supplied from thevideo decoder1315 to predetermined processing such as noise removal or the like, and supplies the obtained video data to thegraphics generating circuit1319.
• Thegraphics generating circuit1319 generates the video data of a program to be displayed on adisplay panel1321, or image data due to processing based on an application to be supplied via a network, or the like, and supplies the generated video data or image data to thepanel driving circuit1320. Also, thegraphics generating circuit1319 also performs processing such as supplying video data obtained by generating video data (graphics) for the user displaying a screen used for selection of an item or the like, and superimposing this on the video data of a program, to thepanel driving circuit1320 as appropriate.
• Thepanel driving circuit1320 drives thedisplay panel1321 based on the data supplied from thegraphics generating circuit1319 to display the video of a program, or the above-mentioned various screens on thedisplay panel1321.
• Thedisplay panel1321 is made up of an LCD (Liquid Crystal Display) and so forth, and displays the video of a program or the like in accordance with the control by thepanel driving circuit1320.
• Also, thetelevision receiver1300 also includes an audio A/D (Analog/Digital)conversion circuit1314, an audiosignal processing circuit1322, an echo cancellation/audio synthesizing circuit1323, anaudio amplifier circuit1324, and aspeaker1325.
• Theterrestrial tuner1313 demodulates the received broadcast wave signal, thereby obtaining not only a video signal but also an audio signal. Theterrestrial tuner1313 supplies the obtained audio signal to the audio A/D conversion circuit1314.
• The audio A/D conversion circuit1314 subjects the audio signal supplied from theterrestrial tuner1313 to A/D conversion processing, and supplies the obtained digital audio signal to the audiosignal processing circuit1322.
• The audiosignal processing circuit1322 subjects the audio data supplied from the audio A/D conversion circuit1314 to predetermined processing such as noise removal or the like, and supplies the obtained audio data to the echo cancellation/audio synthesizing circuit1323.
• The echo cancellation/audio synthesizing circuit1323 supplies the audio data supplied from the audiosignal processing circuit1322 to theaudio amplifier circuit1324.
• Theaudio amplifier circuit1324 subjects the audio data supplied from the echo cancellation/audio synthesizing circuit1323 to D/A conversion processing, subjects to amplifier processing to adjust to predetermined volume, and then outputs the audio from thespeaker1325.
• Further, thetelevision receiver1300 also includes adigital tuner1316, and anMPEG decoder1317.
• Thedigital tuner1316 receives the broadcast wave signals of a digital broadcast (terrestrial digital broadcast, BS (Broadcasting Satellite)/CS (Communications Satellite) digital broadcast) via the antenna, demodulates to obtain MPEG-TS (Moving Picture Experts Group-Transport Stream), and supplies this to theMPEG decoder1317.
• TheMPEG decoder1317 descrambles the scrambling given to the MPEG-TS supplied from thedigital tuner1316, and extracts a stream including the data of a program serving as a playback object (viewing object). TheMPEG decoder1317 decodes an audio packet making up the extracted stream, supplies the obtained audio data to the audiosignal processing circuit1322, and also decodes a video packet making up the stream, and supplies the obtained video data to the videosignal processing circuit1318. Also, theMPEG decoder1317 supplies EPG (Electronic Program Guide) data extracted from the MPEG-TS to aCPU1332 via an unshown path.
• Thetelevision receiver1300 uses the above-mentionedimage decoding devices151,401,501, or601, as theMPEG decoder1317 for decoding video packets in this way. Accordingly, in the same way as with theimage encoding devices151 and401, theMPEG decoder1317 switches filter coefficients according to the quantization parameter and prediction mode, and performs filtering processing of neighboring pixels, prior to intra prediction. Alternatively, in the same way as with theimage encoding devices501 and601, theMPEG decoder1317 controls whether or not to perform filtering processing of neighboring pixels performed prior to intra prediction, based on the on/off flag. Thus, encoding efficiency can be improved.
• The video data supplied from theMPEG decoder1317 is, in the same way as with the case of the video data supplied from thevideo decoder1315, subjected to predetermined processing at the videosignal processing circuit1318. The video data subjected to predetermined processing is then superimposed on the generated video data and so forth at thegraphics generating circuit1319 as appropriate, supplied to thedisplay panel1321 via thepanel driving circuit1320, and the image thereof is displayed thereon.
• The audio data supplied from theMPEG decoder1317 is, in the same way as with the case of the audio data supplied from the audio A/D conversion circuit1314, subjected to predetermined processing at the audiosignal processing circuit1322. The audio data subjected to predetermined processing is then supplied to theaudio amplifier circuit1324 via the echo cancellation/audio synthesizing circuit1323, and subjected to D/A conversion processing and amplifier processing. As a result thereof, the audio adjusted in predetermined volume is output from thespeaker1325.
• Also, thetelevision receiver1300 also includes amicrophone1326, and an A/D conversion circuit1327.
• The A/D conversion circuit1327 receives the user's audio signal collected by themicrophone1326 provided to thetelevision receiver1300 serving as for audio conversation. The A/D conversion circuit1327 subjects the received audio signal to A/D conversion processing, and supplies the obtained digital audio data to the echo cancellation/audio synthesizing circuit1323.
• In the event that the user (user A)'s audio data of thetelevision receiver1300 has been supplied from the A/D conversion circuit1327, the echo cancellation/audio synthesizing circuit1323 perform echo cancellation with the user A's audio data taken as a object. After echo cancellation, the echo cancellation/audio synthesizing circuit1323 outputs audio data obtained by synthesizing the user A's audio data and other audio data, or the like from thespeaker1325 via theaudio amplifier circuit1324.
• Further, thetelevision receiver1300 also includes anaudio codec1328, aninternal bus1329, SDRAM (Synchronous Dynamic Random Access Memory)1330,flash memory1331, aCPU1332, a USB (Universal Serial Bus) I/F1333, and a network I/F1334.
• The A/D conversion circuit1327 receives the user's audio signal collected by themicrophone1326 provided to thetelevision receiver1300 serving as for audio conversation. The A/D conversion circuit1327 subjects the received audio signal to A/D conversion processing, and supplies the obtained digital audio data to theaudio codec1328.
• Theaudio codec1328 converts the audio data supplied from the A/D conversion circuit1327 into the data of a predetermined format for transmission via a network, and supplies to the network I/F1334 via theinternal bus1329.
• The network I/F1334 is connected to the network via a cable mounted on anetwork terminal1335. The network I/F1334 transmits the audio data supplied from theaudio codec1328 to another device connected to the network thereof, for example. Also, the network I/F1334 receives, via thenetwork terminal1335, the audio data transmitted from another device connected thereto via the network, and supplies this to theaudio codec1328 via theinternal bus1329, for example.
• Theaudio codec1328 converts the audio data supplied from the network I/F1334 into the data of a predetermined format, and supplies this to the echo cancellation/audio synthesizing circuit1323.
• The echo cancellation/audio synthesizing circuit1323 performs echo cancellation with the audio data supplied from theaudio codec1328 taken as a object, and outputs the data of audio obtained by synthesizing the audio data and other audio data, or the like, from thespeaker1325 via theaudio amplifier circuit1324.
• TheSDRAM1330 stores various types of data necessary for theCPU1332 performing processing.
• Theflash memory1331 stores a program to be executed by theCPU1332. The program stored in theflash memory1331 is read out by theCPU1332 at predetermined timing such as when activating thetelevision receiver1300, or the like. EPG data obtained via a digital broadcast, data obtained from a predetermined server via the network, and so forth are also stored in theflash memory1331.
• For example, MPEG-TS including the content data obtained from a predetermined server via the network by the control of theCPU1332 is stored in theflash memory1331. Theflash memory1331 supplies the MPEG-TS thereof to theMPEG decoder1317 via theinternal bus1329 by the control of theCPU1332, for example.
• TheMPEG decoder1317 processes the MPEG-TS thereof in the same way as with the case of the MPEG-TS supplied from thedigital tuner1316. In this way, thetelevision receiver1300 receives the content data made up of video, audio, and so forth via the network, decodes using theMPEG decoder1317, whereby video thereof can be displayed, and audio thereof can be output.
• Also, thetelevision receiver1300 also includes alight reception unit1337 for receiving the infrared signal transmitted from aremote controller1351.
• Thelight reception unit1337 receives infrared rays from theremote controller1351, and outputs a control code representing the content of the user's operation obtained by demodulation, to theCPU1332.
• TheCPU1332 executes the program stored in theflash memory1331 to control the entire operation of thetelevision receiver1300 according to the control code supplied from thelight reception unit1337, and so forth. TheCPU1332, and the units of thetelevision receiver1300 are connected via an unshown path.
• The USB I/F1333 performs transmission/reception of data as to an external device of thetelevision receiver1300 which is connected via a USB cable mounted on aUSB terminal1336. The network I/F1334 connects to the network via a cable mounted on thenetwork terminal1335, also performs transmission/reception of data other than audio data as to various devices connected to the network.
• Thetelevision receiver1300 uses theimage decoding devices151,401,501, or601, as theMPEG decoder1317, whereby encoding efficiency can be improved. As a result thereof, thetelevision receiver1300 can obtain a decoded image with higher precision from broadcast wave signals received via the antenna, or the content data obtained via the network, at higher speeds, and display this.
• FIG. 47 is a block diagram illustrating a principal configuration example of a cellular phone using the image encoding device and image decoding device to which the present invention has been applied.
• Acellular phone1400 shown inFIG. 47 includes amain control unit1450 configured so as to integrally control the units, a powersupply circuit unit1451, an operationinput control unit1452, animage encoder1453, a camera I/F unit1454, anLCD control unit1455, animage decoder1456, a multiplexing/separating unit1457, a recording/playback unit1462, a modulation/demodulation circuit unit1458, and anaudio codec1459. These are mutually connected via abus1460.
• Also, thecellular phone1400 includesoperation keys1419, a CCD (Charge Coupled Devices)camera1416, aliquid crystal display1418, astorage unit1423, a transmission/reception circuit unit1463, anantenna1414, a microphone (MIC)1421, and aspeaker1417.
• Upon a call being ended and a power key being turned on by the user's operation, the powersupply circuit unit1451 activates thecellular phone1400 in an operational state by supplying power to the units from a battery pack.
• Thecellular phone1400 performs various operations, such as transmission/reception of an audio signal, transmission/reception of an e-mail and image data, image shooting, data recoding, and so forth, in various modes such as a voice call mode, a data communication mode, and so forth, based on the control of themain control unit1450 made up of a CPU, ROM, RAM, and so forth.
• For example, in the voice call mode, thecellular phone1400 converts the audio signal collected by the microphone (mike)1421 into digital audio data by theaudio codec1459, subjects this to spectrum spread processing at the modulation/demodulation circuit unit1458, and subjects this to digital/analog conversion processing and frequency conversion processing at the transmission/reception circuit unit1463. Thecellular phone1400 transmits the signal for transmission obtained by the conversion processing thereof to an unshown base station via theantenna1414. The signal for transmission (audio signal) transmitted to the base station is supplied to the communication partner's cellular phone via the public telephone network.
• Also, for example, in the voice call mode, thecellular phone1400 amplifies the reception signal received at theantenna1414, at the transmission/reception circuit unit1463, further subjects to frequency conversion processing and analog/digital conversion processing, subjects to spectrum inverse spread processing at the modulation/demodulation circuit unit1458, and converts into an analog audio signal by theaudio codec1459. Thecellular phone1400 outputs the converted and obtained analog audio signal thereof from thespeaker1417.
• Further, for example, in the event of transmitting an e-mail in the data communication mode, thecellular phone1400 accepts the text data of the e-mail input by the operation of theoperation keys1419 at the operationinput control unit1452. Thecellular phone1400 processes the text data thereof at themain control unit1450, and displays on theliquid crystal display1418 via theLCD control unit1455 as an image.
• Also, thecellular phone1400 generates e-mail data at themain control unit1450 based on the text data accepted by the operationinput control unit1452, the user's instructions, and so forth. Thecellular phone1400 subjects the e-mail data thereof to spectrum spread processing at the modulation/demodulation circuit unit1458, and subjects to digital/analog conversion processing and frequency conversion processing at the transmission/reception circuit unit1463. Thecellular phone1400 transmits the signal for transmission obtained by the conversion processing thereof to an unshown base station via theantenna1414. The signal for transmission (e-mail) transmitted to the base station is supplied to a predetermined destination via the network, mail server, and so forth.
• Also, for example, in the event of receiving an e-mail in the data communication mode, thecellular phone1400 receives the signal transmitted from the base station via theantenna1414 with the transmission/reception circuit unit1463, amplifies, and further subjects to frequency conversion processing and analog/digital conversion processing. Thecellular phone1400 subjects the reception signal thereof to spectrum inverse spread processing at the modulation/demodulation circuit unit1458 to restore the original e-mail data. Thecellular phone1400 displays the restored e-mail data on theliquid crystal display1418 via theLCD control unit1455.
• Note that thecellular phone1400 may record (store) the received e-mail data in thestorage unit1423 via the recording/playback unit1462.
• Thisstorage unit1423 is an optional rewritable recording medium. Thestorage unit1423 may be semiconductor memory such as RAM, built-in flash memory, or the like, may be a hard disk, or may be a removable medium such as a magnetic disk, a magneto-optical disk, an optical disc, USB memory, a memory card, or the like. It goes without saying that thestorage unit1423 may be other than these.
• Further, for example, in the event of transmitting image data in the data communication mode, thecellular phone1400 generates image data by imaging at theCCD camera1416. TheCCD camera1416 includes a CCD serving as an optical device such as a lens, diaphragm, and so forth, and serving as a photoelectric conversion device, which images a subject, converts the intensity of received light into an electrical signal, and generates the image data of an image of the subject. The image data thereof is subjected to compression encoding at theimage encoder1453 using a predetermined encoding format, for example, such as MPEG2, MPEG4, or the like, via the camera I/F unit1454, and accordingly, the image data thereof is converted into encoded image data.
• Thecellular phone1400 employs the above-mentionedimage encoding devices51,351,451, and551, as theimage encoder1453 for performing such processing. Accordingly, in the same way as with theimage encoding devices51 and351, theimage encoder1453 sets filter coefficients according to the quantization parameter and prediction mode, and performs filtering processing of neighboring pixels, prior to intra prediction. Alternatively, in the same way as with theimage encoding devices451 and551, theimage encoder1453 controls whether or not to perform filtering processing of neighboring pixels performed prior to intra prediction. Thus, encoding efficiency can be improved.
• Note that, at this time simultaneously, thecellular phone1400 converts the audio collected at the microphone (mike)1421, while shooting with theCCD camera1416, from analog to digital at theaudio codec1459, and further encodes this.
• Thecellular phone1400 multiplexes the encoded image data supplied from theimage encoder1453, and the digital audio data supplied from theaudio codec1459 at the multiplexing/separating unit1457 using a predetermined method. Thecellular phone1400 subjects the multiplexed data obtained as a result thereof to spectrum spread processing at the modulation/demodulation circuit unit1458, and subjects to digital/analog conversion processing and frequency conversion processing at the transmission/reception circuit unit1463. Thecellular phone1400 transmits the signal for transmission obtained by the conversion processing thereof to an unshown base station via theantenna1414. The signal for transmission (image data) transmitted to the base station is supplied to the communication partner via the network or the like.
• Note that in the event that image data is not transmitted, thecellular phone1400 may also display the image data generated at theCCD camera1416 on theliquid crystal display1418 via theLCD control unit1455 instead of theimage encoder1453.
• Also, for example, in the event of receiving the data of a moving image file linked to a simple website or the like in the data communication mode, thecellular phone1400 receives the signal transmitted from the base station at the transmission/reception circuit unit1463 via theantenna1414, amplifies, and further subjects to frequency conversion processing and analog/digital conversion processing. Thecellular phone1400 subjects the received signal to spectrum inverse spread processing at the modulation/demodulation circuit unit1458 to restore the original multiplexed data. Thecellular phone1400 separates the multiplexed data thereof at the multiplexing/separating unit1457 into encoded image data and audio data.
• Thecellular phone1400 decodes the encoded image data at theimage decoder1456 using the decoding format corresponding to a predetermined encoding format such as MPEG2, MPEG4, or the like, thereby generating playback moving image data, and displays this on theliquid crystal display1418 via theLCD control unit1455. Thus, moving image data included in a moving image file linked to a simple website is displayed on theliquid crystal display1418, for example.
• Thecellular phone1400 employs the above-mentionedimage decoding device151,401,501, or601 as theimage decoder1456 for performing such processing. Accordingly, in the same way as with theimage decoding devices151 and401, theimage decoder1456 switches filter coefficients according to the quantization parameter and prediction mode, and performs filtering processing of neighboring pixels, prior to intra prediction. Alternatively, in the same way as with theimage decoding devices501 and601, theimage decoder1456 controls whether or not to perform filtering processing of neighboring pixels performed prior to intra prediction, based on the on/off flag. Thus, encoding efficiency can be improved.
• At this time, simultaneously, thecellular phone1400 converts the digital audio data into an analog audio signal at theaudio codec1459, and outputs this from thespeaker1417. Thus, audio data included in a moving image file linked to a simple website is played, for example.
• Note that, in the same way as with the case of e-mail, thecellular phone1400 may record (store) the received data linked to a simple website or the like in thestorage unit1423 via the recording/playback unit1462.
• Also, thecellular phone1400 analyzes the imaged two-dimensional code obtained by theCCD camera1416 at themain control unit1450, whereby information recorded in the two-dimensional code can be obtained.
• Further, thecellular phone1400 can communicate with an external device at the infrared communication unit1481 using infrared rays.
• Thecellular phone1400 employs theimage encoding device51,351,451, or551 as theimage encoder1453, whereby the encoding efficiency of encoded data to be generated by encoding the image data generated at theCCD camera1416 can be improved, for example. As a result, thecellular phone1400 can provide encoded data (image data) with excellent encoding efficiency to another device.
• Also, thecellular phone1400 employs theimage decoding device151,401,501, or601 as theimage decoder1456, whereby a prediction image with high precision can be generated. As a result thereof, thecellular phone1400 can obtain a decoded image with higher precision from a moving image file linked to a simple website, and display this, for example.
• Note that description has been made so far wherein thecellular phone1400 employs theCCD camera1416, but thecellular phone1400 may employ an image sensor (CMOS image sensor) using CMOS (Complementary Metal Oxide Semiconductor) instead of thisCCD camera1416. In this case as well, thecellular phone1400 can image a subject and generate the image data of an image of the subject in the same way as with the case of employing theCCD camera1416.
• Also, description has been made so far regarding thecellular phone1400, but theimage encoding devices51,351,451, and551, and theimage decoding devices151,401,501, and601 may be applied to any kind of device in the same way as with the case of thecellular phone1400 as long as it is a device having the same imaging function and communication function as those of thecellular phone1400, for example, such as a PDA (Personal Digital Assistants), smart phone, UMPC (Ultra Mobile Personal Computer), net book, notebook-sized personal computer, or the like.
• FIG. 48 is a block diagram illustrating a principal configuration example of a hard disk recorder which employs the image encoding device and image decoding device to which the present invention has been applied.
• A hard disk recorder (HDD recorder)1500 shown inFIG. 48 is a device which stores, in a built-in hard disk, audio data and video data of a broadcast program included in broadcast wave signals (television signals) received by a tuner and transmitted from a satellite or a terrestrial antenna or the like, and provides the stored data to the user at timing according to the user's instructions.
• Thehard disk recorder1500 can extract audio data and video data from broadcast wave signals, decode these as appropriate, and store in the built-in hard disk, for example. Also, thehard disk recorder1500 can also obtain audio data and video data from another device via the network, decode these as appropriate, and store in the built-in hard disk, for example.
• Further, thehard disk recorder1500 decodes audio data and video data recorded in the built-in hard disk, supplies to amonitor1560, and displays an image thereof on the screen of themonitor1560, for example. Also, thehard disk recorder1500 can output audio thereof from the speaker of themonitor1560.
• Thehard disk recorder1500 decodes audio data and video data extracted from the broadcast wave signals obtained via the tuner, or the audio data and video data obtained from another device via the network, supplies to themonitor1560, and displays an image thereof on the screen of themonitor1560, for example. Also, thehard disk recorder1500 can output audio thereof from the speaker of themonitor1560.
• It goes without saying that operations other than these may be performed.
• As shown inFIG. 48, thehard disk recorder1500 includes areception unit1521, ademodulation unit1522, ademultiplexer1523, anaudio decoder1524, avideo decoder1525, and arecorder control unit1526. Thehard disk recorder1500 further includesEPG data memory1527,program memory1528,work memory1529, adisplay converter1530, an OSD (On Screen Display)control unit1531, adisplay control unit1532, a recording/playback unit1533, a D/A converter1534, and acommunication unit1535.
• Also, thedisplay converter1530 includes avideo encoder1541. The recording/playback unit1533 includes anencoder1551 and adecoder1552.
• Thereception unit1521 receives the infrared signal from the remote controller (not shown), converts into an electrical signal, and outputs to therecorder control unit1526. Therecorder control unit1526 is configured of, for example, a microprocessor and so forth, and executes various types of processing in accordance with the program stored in theprogram memory1528. At this time, therecorder control unit1526 uses thework memory1529 according to need.
• Thecommunication unit1535, which is connected to the network, performs communication processing with another device via the network. For example, thecommunication unit1535 is controlled by therecorder control unit1526 to communicate with a tuner (not shown), and to principally output a channel selection control signal to the tuner.
• Thedemodulation unit1522 demodulates the signal supplied from the tuner, and outputs to thedemultiplexer1523. Thedemultiplexer1523 separates the data supplied from thedemodulation unit1522 into audio data, video data, and EPG data, and outputs to theaudio decoder1524,video decoder1525, andrecorder control unit1526, respectively.
• Theaudio decoder1524 decodes the input audio data, for example, using the MPEG format, and outputs to the recording/playback unit1533. Thevideo decoder1525 decodes the input video data, for example, using the MPEG format, and outputs to thedisplay converter1530. Therecorder control unit1526 supplies the input EPG data to theEPG data memory1527 for storing.
• Thedisplay converter1530 encodes the video data supplied from thevideo decoder1525 orrecorder control unit1526 into, for example, the video data conforming to the NTSC (National Television Standards Committee) format using thevideo encoder1541, and outputs to the recording/playback unit1533. Also, thedisplay converter1530 converts the size of the screen of the video data supplied from thevideo decoder1525 orrecorder control unit1526 into the size corresponding to the size of themonitor1560. Thedisplay converter1530 further converts the video data of which the screen size has been converted into the video data conforming to the NTSC format using thevideo encoder1541, converts into an analog signal, and outputs to thedisplay control unit1532.
• Thedisplay control unit1532 superimposes, under the control of therecorder control unit1526, the OSD signal output from the OSD (On Screen Display)control unit1531 on the video signal input from thedisplay converter1530, and outputs to the display of themonitor1560 for display.
• Also, the audio data output from theaudio decoder1524 has been converted into an analog signal using the D/A converter1534, and supplied to themonitor1560. Themonitor1560 outputs this audio signal from a built-in speaker.
• The recording/playback unit1533 includes a hard disk as a recording medium in which video data, audio data, and so forth are recorded.
• The recording/playback unit1533 encodes the audio data supplied from theaudio decoder1524 by theencoder1551 using the MPEG format, for example. Also, the recording/playback unit1533 encodes the video data supplied from thevideo encoder1541 of thedisplay converter1530 by theencoder1551 using the MPEG format. The recording/playback unit1533 synthesizes the encoded data of the audio data thereof, and the encoded data of the video data thereof using the multiplexer. The recording/playback unit1533 amplifies the synthesized data by channel coding, and writes the data thereof in the hard disk via a recording head.
• The recording/playback unit1533 plays the data recorded in the hard disk via a playback head, amplifies, and separates into audio data and video data using the demultiplexer. The recording/playback unit1533 decodes the audio data and video data by thedecoder1552 using the MPEG format. The recording/playback unit1533 converts the decoded audio data from digital to analog, and outputs to the speaker of themonitor1560. Also, the recording/playback unit1533 converts the decoded video data from digital to analog, and outputs to the display of themonitor1560.
• Therecorder control unit1526 reads out the latest EPG data from theEPG data memory1527 based on the user's instructions indicated by the infrared signal from the remote controller which is received via thereception unit1521, and supplies to theOSD control unit1531. TheOSD control unit1531 generates image data corresponding to the input EPG data, and outputs to thedisplay control unit1532. Thedisplay control unit1532 outputs the video data input from theOSD control unit1531 to the display of themonitor1560 for display. Thus, EPG (Electronic Program Guide) is displayed on the display of themonitor1560.
• Also, thehard disk recorder1500 can obtain various types of data such as video data, audio data, EPG data, and so forth supplied from another device via the network such as the Internet or the like.
• Thecommunication unit1535 is controlled by therecorder control unit1526 to obtain encoded data such as video data, audio data, EPG data, and so forth transmitted from another device via the network, and to supply this to therecorder control unit1526. Therecorder control unit1526 supplies the encoded data of the obtained video data and audio data to the recording/playback unit1533, and stores in the hard disk, for example. At this time, therecorder control unit1526 and recording/playback unit1533 may perform processing such as re-encoding or the like according to need.
• Also, therecorder control unit1526 decodes the encoded data of the obtained video data and audio data, and supplies the obtained video data to thedisplay converter1530. Thedisplay converter1530 processes, in the same way as the video data supplied from thevideo decoder1525, the video data supplied from therecorder control unit1526, supplies to themonitor1560 via thedisplay control unit1532 for displaying an image thereof.
• Alternatively, an arrangement may be made wherein in accordance with this image display, therecorder control unit1526 supplies the decoded audio data to themonitor1560 via the D/A converter1534, and outputs audio thereof from the speaker.
• Further, therecorder control unit1526 decodes the encoded data of the obtained EPG data, and supplies the decoded EPG data to theEPG data memory1527.
• Thehard disk recorder1500 thus configured employs theimage decoding device151,401,501, or601 as thevideo decoder1525,decoder1552, and a decoder housed in therecorder control unit1526. Accordingly, in the same way as with theimage decoding devices151 and401, thevideo decoder1525,decoder1552, and decoder housed in therecorder control unit1526 switch filter coefficients according to the quantization parameter and prediction mode, and perform filtering processing of neighboring pixels, prior to intra prediction. Alternatively, in the same way as with theimage decoding devices501 and601, thevideo decoder1525,decoder1552, and decoder housed in therecorder control unit1526 control whether or not to perform filtering processing of neighboring pixels performed prior to intra prediction, based on the on/off flag. Thus, encoding efficiency can be improved.
• Accordingly, thehard disk recorder1500 can generate a prediction image with high precision. As a result thereof, thehard disk recorder1500 can obtain a decoded image with higher precision, for example, from the encoded data of video data received via the tuner, the encoded data of video data read out from the hard disk of the recording/playback unit1533, or the encoded data of video data obtained via the network, and display on themonitor1560.
• Also, thehard disk recorder1500 employs theimage encoding device51,351,451, or551 as theencoder1551. Accordingly, in the same way as with theimage encoding devices51 and351, theencoder1551 sets filter coefficients according to the quantization parameter and prediction mode, and performs filtering processing of neighboring pixels, prior to intra prediction. Alternatively, in the same way as with theimage encoding devices451 and551, theencoder1551 controls whether or not to perform filtering processing of neighboring pixels performed prior to intra prediction. Thus, encoding efficiency can be improved.
• Accordingly, thehard disk recorder1500 can improve the encoding efficiency of encoded data to be recorded in the hard disk, for example. As a result thereof, thehard disk recorder1500 can use the storage region of the hard disk in a more effective manner.
• Note that description has been made so far regarding thehard disk recorder1500 for recording video data and audio data in the hard disk, but it goes without saying that any kind of recording medium may be employed. For example, even with a recorder to which a recording medium other than a hard disk, such as flash memory, optical disc, a video tape, or the like, is applied, in the same way as with the case of the above-mentionedhard disk recorder1500, theimage encoding devices51,351,451, and551, and theimage decoding devices151,401,501, and601 can be applied thereto.
• FIG. 49 is a block diagram illustrating a principal configuration example of a camera employing the image decoding device and image encoding device to which the present invention has been applied.
• Acamera1600 shown inFIG. 49 images a subject, displays an image of the subject on anLCD1616, and records this in arecording medium1633 as image data.
• Alens block1611 inputs light (i.e., video of a subject) to a CCD/CMOS1612. The CCD/CMOS1612 is an image sensor employing a CCD or CMOS, converts the intensity of received light into an electrical signal, and supplies to a camerasignal processing unit1613.
• The camerasignal processing unit1613 converts the electrical signal supplied from the CCD/CMOS1612 into color difference signals of Y, Cr, and Cb, and supplies to an imagesignal processing unit1614. The imagesignal processing unit1614 subjects, under the control of acontroller1621, the image signal supplied from the camerasignal processing unit1613 to predetermined image processing, or encodes the image signal thereof by anencoder1641 using the MPEG format for example. The imagesignal processing unit1614 supplies encoded data generated by encoding an image signal, to adecoder1615. Further, the imagesignal processing unit1614 obtains data for display generated at an on-screen display (OSD)1620, and supplies this to thedecoder1615.
• With the above-mentioned processing, the camerasignal processing unit1613 appropriately takes advantage of DRAM (Dynamic Random Access Memory)1618 connected via abus1617 to hold image data, encoded data encoded from the image data thereof, and so forth in theDRAM1618 thereof according to need.
• Thedecoder1615 decodes the encoded data supplied from the imagesignal processing unit1614, and supplies obtained image data (decoded image data) to theLCD1616. Also, thedecoder1615 supplies the data for display supplied from the imagesignal processing unit1614 to theLCD1616. TheLCD1616 synthesizes the image of the decoded image data, and the image of the data for display, supplied from thedecoder1615 as appropriate, and displays a synthesizing image thereof.
• The on-screen display1620 outputs, under the control of thecontroller1621, data for display such as a menu screen or icon or the like made up of a symbol, characters, or a figure to the imagesignal processing unit1614 via thebus1617.
• Based on a signal indicating the content commanded by the user using anoperating unit1622, thecontroller1621 executes various types of processing, and also controls the imagesignal processing unit1614,DRAM1618,external interface1619, on-screen display1620, media drive1623, and so forth via thebus1617. A program, data, and so forth necessary for thecontroller1621 executing various types of processing are stored inFLASH ROM1624.
• For example, thecontroller1621 can encode image data stored in theDRAM1618, or decode encoded data stored in theDRAM1618 instead of the imagesignal processing unit1614 anddecoder1615. At this time, thecontroller1621 may perform encoding and decoding processing using the same format as the encoding and decoding format of the imagesignal processing unit1614 anddecoder1615, or may perform encoding and decoding processing using a format that neither the imagesignal processing unit1614 nor thedecoder1615 can handle.
• Also, for example, in the event that start of image printing has been instructed from theoperating unit1622, thecontroller1621 reads out image data from theDRAM1618, and supplies this to aprinter1634 connected to theexternal interface1619 via thebus1617 for printing.
• Further, for example, in the event that image recording has been instructed from theoperating unit1622, thecontroller1621 reads out encoded data from theDRAM1618, and supplies this to arecording medium1633 mounted on the media drive1623 via thebus1617 for storing.
• Therecording medium1633 is an optional readable/writable removable medium, for example, such as a magnetic disk, a magneto-optical disk, an optical disc, semiconductor memory, or the like. It goes without saying that therecording medium1633 is also optional regarding the type of a removable medium, and accordingly may be a tape device, or may be a disc, or may be a memory card. It goes without saying that therecoding medium1633 may be a non-contact IC card or the like.
• Alternatively, the media drive1623 and therecording medium1633 may be configured so as to be integrated into a non-transportability recording medium, for example, such as a built-in hard disk drive, SSD (Solid State Drive), or the like.
• Theexternal interface1619 is configured of, for example, a USB input/output terminal and so forth, and is connected to theprinter1634 in the event of performing printing of an image. Also, adrive1631 is connected to theexternal interface1619 according to need, on which the removable medium1632 such as a magnetic disk, optical disc, or magneto-optical disk is mounted as appropriate, and a computer program read out therefrom is installed in theFLASH ROM1624 according to need.
• Further, theexternal interface1619 includes a network interface to be connected to a predetermined network such as a LAN, the Internet, or the like. For example, in accordance with the instructions from theoperating unit1622, thecontroller1621 can read out encoded data from theDRAM1618, and supply this from theexternal interface1619 to another device connected via the network. Also, thecontroller1621 can obtain, via theexternal interface1619, encoded data or image data supplied from another device via the network, and hold this in theDRAM1618, or supply this to the imagesignal processing unit1614.
• Thecamera1600 thus configured employs theimage decoding device151,401,501, or601 as thedecoder1615. Accordingly, in the same way as with theimage decoding devices151 and401, thedecoder1615 switches filter coefficients according to the quantization parameter and prediction mode, and performs filtering processing of neighboring pixels, prior to intra prediction. Alternatively, in the same way as with theimage decoding devices501 and601, thedecoder1615 controls whether or not to perform filtering processing of neighboring pixels performed prior to intra prediction, based on the on/off flag. Thus, encoding efficiency can be improved.
• Accordingly, thecamera1600 can generate a prediction image with high precision. As a result thereof, thecamera1600 can obtain a decoded image with higher precision, for example, from the image data generated at the CCD/CMOS1612, the encoded data of video data read out from theDRAM1618 orrecording medium1633, or the encoded data of video data obtained via the network, and display on theLCD1616.
• Also, thecamera1600 employs theimage encoding device51,351,451, or551, as theencoder1641. Accordingly, in the same way as with theimage encoding devices51 and351, theencoder1641 sets filter coefficients according to the quantization parameter and prediction mode, and performs filtering processing of neighboring pixels, prior to intra prediction. Alternatively, in the same way as with theimage encoding devices451 and551, theencoder1641 controls whether or not to perform filtering processing of neighboring pixels performed prior to intra prediction. Thus, encoding efficiency can be improved.
• Accordingly, thecamera1600 can improve encoding efficiency of encoded data to be recorded in the hard disk, for example. As a result thereof, thecamera1600 can use the storage region of theDRAM1618 orrecording medium1633 in a more effective manner.
• Note that the decoding method of theimage decoding devices151,401,501, and601 may be applied to the decoding processing that thecontroller1621 performs. Similarly, the encoding method of theimage encoding devices51,351,451, and551 may be applied to the encoding processing that thecontroller1621 performs.
• Also, the image data that thecamera1600 images may be a moving image, or may be a still image.
• It goes without saying that theimage encoding devices51,351,451, and551, and theimage decoding devices151,401,501, and601 may be applied to a device or system other than the above-mentioned devices.
REFERENCE SIGNS LIST
• • 51 image encoding device
  • 66 lossless encoding unit
  • 74 intra prediction unit
  • 75 neighboring pixel interpolation filter switching unit
  • 81 neighboring pixel setting unit
  • 82 prediction image generating unit
  • 83 optimal prediction mode determining unit
  • 91 prediction mode buffer
  • 92 quantization parameter buffer
  • 93 low-pass filter setting unit
  • 94 frame coefficient memory
  • 111 neighboring image setting unit
  • 112 prediction image generating unit
  • 113 optimal prediction mode determining unit
  • 121 prediction mode buffet
  • 122 optimal filter calculating unit
  • 93 low-pass filter setting unit
  • 151 image decoding device
  • 162 lossless decoding unit
  • 171 intra prediction unit
  • 172 neighboring pixel interpolation filter switching unit
  • 181 prediction image generating unit
  • 182 neighboring pixel setting unit
  • 191 prediction mode buffer
  • 192 quantization parameter buffer
  • 193 low-pass filter setting unit
  • 194 filter coefficient memory
  • 202 low-pass filter setting unit
  • 251 learning device
  • 261 neighboring interpolation filter calculating unit
  • 271 filter coefficient storage unit
  • 351 image encoding device
  • 361 second order prediction unit
  • 362 neighboring pixel interpolation filter switching unit
  • 401 image decoding device
  • 411 second order prediction unit
  • 412 neighboring pixel interpolation filter switching unit
  • 451 image encoding device
  • 461 neighboring pixel interpolation filter control unit
  • 501 image decoding device
  • 511 neighboring pixel interpolation filter control unit
  • 551 image encoding device
  • 561 neighboring pixel interpolation filter control unit
  • 601 image decoding device
  • 611 neighboring pixel interpolation filter control unit

Claims (9)

What is claimed is:

1. An image encoding method, comprising:

determining an intra prediction mode of a block to be encoded;

determining a block-level control flag value for the intra prediction mode of the block, wherein the block-level control flag value indicates whether filter processing is to be turned on or off as to neighboring pixels that are located adjacent to the block and not included within the block; and

in response to the block-level control flag value indicating the filter process is to be turned on:

filtering the neighboring pixels; and

generating a prediction image for the block using the intra prediction mode and the filtered neighboring pixels, wherein the filtered neighboring pixels are used as reference pixels to generate the prediction image.

2. The image encoding method according toclaim 1, wherein the block-level control flag value is determined in increments of a block size of the block, wherein the size of the block is selected from a predetermined set of sizes of 4×4, 8×8, 16×16, and 32×32.

3. The image encoding method according toclaim 2, further comprising setting filter coefficients according to the block size of the block.

4. An image processing apparatus comprising:

a processor; and

a memory storing instructions which when executed by the processor, cause the image processing apparatus to:

determine an intra prediction mode of a block to be encoded;

determine a block-level control flag value for the intra prediction mode of the block, wherein the block-level control flag value indicates whether filter processing is to be turned on or off as to neighboring pixels that are located adjacent to the block and not included within the block; and

when the block-level control flag value indicates the filter process is to be turned on:

filter the neighboring pixels; and

generate a prediction image for the block using the intra prediction mode and the filtered neighboring pixels, wherein the filtered neighboring pixels are used as reference pixels to generate the prediction image.

5. The image processing apparatus according toclaim 4, wherein the block-level control flag value is determined in increments of a block size of the block, wherein the size of the block is selected from a predetermined set of sizes of 4×4, 8×8, 16×16, and 32×32.

6. The image processing apparatus according toclaim 5, wherein the instructions when executed by the processor, further cause the image processing apparatus to: set filter coefficients according to the block size of the block.

7. A non-transitory computer-readable medium comprising computer instructions, which when executed by a processor causes a device to perform:

determining an intra prediction mode of a block to be encoded;

determining a block-level control flag value for the intra prediction mode of the block, wherein the block-level control flag value indicates whether filter processing is to be turned on or off as to neighboring pixels that are located adjacent to the block and not included within the block; and

in response to the block-level control flag value indicating the filter process is to be turned on:

filtering the neighboring pixels; and

generating a prediction image for the block using the intra prediction mode and the filtered neighboring pixels, wherein the filtered neighboring pixels are used as reference pixels to generate the prediction image.

8. The non-transitory computer-readable medium according toclaim 7, wherein the block-level control flag value is determined in increments of a block size of the block, wherein the size of the block is selected from a predetermined set of sizes of 4×4, 8×8, 16×16, and 32×32.

9. The non-transitory computer-readable medium according toclaim 8, further comprising instructions which when executed by the processor, cause the device to perform: setting filter coefficients according to the block size of the block.

Images