This is a continuation of application Ser. No. 08/324,481, filed Oct. 18, 1994 which was abandoned upon the filing hereof, and which, in turn, is a continuation of application Ser. No. 07/972,564, filed Nov. 6, 1992 now abandoned.
BACKGROUND OF THE INVENTIONThe present invention relates to high-efficiency coding and decoding apparatus which are used in recording, transfer and display apparatus which perform digital signal processing, and which perform efficient coding and decoding, and in particular, to coding and decoding apparatus which perform inter-image processing of moving image signals have small deterioration of image quality even when there are transmission errors.
High-efficiency coding of moving image signals (moving images) can involve interframe predictive coding which uses the correlation between frames of image signals and uses a frame for which coding has been performed, to predict and code only the prediction error. In recent years, motion compensation predictive coding has become the general method used for prediction in accordance with motion of an image.
On the other hand, in coding in which a storage media is the object, intraframe independent coding is performed without interframe prediction for each of several frames and this enables random access and high-speed search.
In addition, there is also known a method such as MPEG (ISO-IEC) which uses skip prediction and pre- and post-prediction between skip predictions to raise the coding efficiency. With the MPEG method, differences in the method of prediction mean that a frame can be an I (intra) frame coded independently within a frame, a P (Prediction) frame which is skip predicted, or a B (Bi-directional) frame which is pre- and post-predicted.
The following is a description of a detailed configuration of a conventional coding apparatus.
FIG. 1 shows an example of the configuration of a coding apparatus of the MPEG type. Here, the frame types of I, P and B cause thechangeover switch 2, 4, 22 to be controlled by sync signals separated from input signals, and to be switched to the positions shown in the figure.
Image signals input from animage input terminal 1 are directly led to a predictive, subtracter 5 via thechangeover switches 2 and 4 in the case of I or P frames, while B frames are led to thepredictive subtracter 5 after having been delayed until there is pre- and post-I and P in aframe memory 3. In thepredictive subtracter 5, prediction signals arriving from anadaptive predictor 42 are subtracted from input signals and a prediction residual signal is output to become coded data compressed by coding in anintraframe encoder 6.
In theintraframe encoder 6, DCT (discrete cosine transform) is first preformed, and that conversion output is quantized, and given a variable length coding such as Huffman coding or the like. That compressed DCT information is applied to amultiplexer 40 and in the case of I and P frames, is led to anintraframe decoder 21 via thechangeover switch 22.
Theintraframe decoder 21 first decodes the variable length coding, and replaces the fixed-length codes with quantized representative values, and also performs reverse DCT to obtain the reproduced signals. In theintraframe decoder 21, the reproduced prediction error signals have the prediction signals added in aresidual adder 20 to produce the reproduced image signals. The reproduced image signals are stored in aframe memory 19 while the signal that have been stored in theframe memory 19 until that time are transferred to aframe memory 18.
The output of theframe memory 19 is given to amotion compensator 15 and amotion vector detector 17, and the output of theframe memory 18 is given to amotion compensator 14 and themotion vector detector 16.
For each block of 16×16 picture elements, themotion vector estimators 16 and 17 detect the motion vectors between the input signals and the signals given to theframe memories 18 and 19. The motion vector information is given to themotion compensators 14 and 15 and also to amultiplexer 40. Themotion compensators 14 and 15 spatially move reproduced image signals stored in theframe memories 18 and 19 by the motion vector portion given from the motion vector detector, and applies them to anadaptive predictor 42.
For the same block as the motion vector detection, theadaptive predictor 42 creates four types of prediction signals from the two signals (F and B) which have been motion compensated, and of those, the optimum prediction signals is decided from matching with the input signals which become the signals to be predicted.
The prediction mode used here is one of the four types of only the "F" (Front: prediction signals from the frame temporally prior) mode, only the "B" (Back: prediction signals from the frame temporarily later) mode, the "(F+B)/2" mode or the "0" mode, with the "0" mode being intraframe independent coding. The prediction mode is only the "0" mode for I-frames, the "F" and "0" modes for P-frames, or any of the four modes for B-frames.
Themultiplexer 40 recombines the DCT information which is the output of theintraframe encoder 6, the prediction mode information (MODE) which is the output of theadaptive predictor 42, the motion vector information (MVF and MVB) which is the output of themotion vector detector 16, for each block (macroblock: MB) for which the motion vector and the prediction mode have been determined, and outputs them via adata output terminal 12, to the side of a decoding apparatus. FIG. 6A shows the configuration of the data. Here, there is no transfer of the motion vector information not used in the prediction.
The following is a description of a conventional decoding apparatus.
FIG. 2 is a view showing the configuration of a decoding apparatus. Those portions which correspond to portions of the coding apparatus of FIG. 1 are shown with corresponding numerals. The coded data which is input from adata input terminal 30 is disassembled into each information by ademultiplexer 41 and the DCT information is applied to theintraframe decoder 21, the prediction mode information is applied to anadaptive predictor 43, and the motion vector information is applied to themotion compensators 14 and 15.
The DCT information is decoded by theintraframe decoder 21, and prediction signals are added at theresidual adder 20 to create the reproduced image signals.
In the case of B-frames, reproduced image signals are immediately outputted from a reproduced imagesignal output terminal 36 viachangeover switches 34 and 35, while I- and P-frames are stored in theframe memory 19. The signals which have been stored in theframe memory 19 up till that time are moved to theframe memory 18 and are outputted from the reproduced imagesignal output terminal 36 via thechangeover switch 35.
The output of theframe memory 19 is applied to themotion compensator 15 while the output of theframe memory 18 is applied to themotion compensator 14. Themotion compensators 14 and 15 spatially move the reproduction image signals stored in the frame memory, by the motion vector portion given from thedemultiplexer 41, and applies them to theadaptive predictor 43. Theadaptive predictor 43 makes the prediction signals from the prediction mode information given from thedemultiplexer 41 and outputs it to theresidual adder 20.
Here, the inter-image processing units are frames but the description is the same if they are fields of interlace signals.
When there is a coding error between the coding apparatus and its decoding apparatus during transfer or recording, normal demodulation does not occur and there is a deterioration in the image quality. Coding errors result in cell loss in ATM (asynchronous transfer mode) circuits when they occur in normal circuits and recording media, and this loss in cell units becomes a "dropout".
In this case, even for the case of the coding apparatus and the decoding apparatus shown as the conventional example, there is normally detection to the effect that a coding error has occurred and so with prediction residual errors, the prediction residue is not added and the reproduced image signals are the prediction signals only, to result in there being no particularly large deterioration. However, there is absolutely no decoding of a block if there is "dropout" of the motion vector, adjacent blocks and the like are used for interpolation within the same frame and there is no image deterioration as a result.
On the other hand, with a coding method which periodically has independent frames, the deterioration stops with the independent frames and so this method appears advantageous at first. However, coding errors in independent frames can only be compensated for spatially and so there the deterioration becomes large, and the image is influenced later. Furthermore, the amount of data for independent frames is larger than that for prediction frames and so when there are ten independent frames at once, the amount of data is about 40% of the overall amount of data, and the influence of coding errors becomes serious.
SUMMARY OF THE INVENTIONIn order to solve the problems described above, the object of the present invention is to provide a moving image coding apparatus and decoding apparatus which always makes a plural number of frames used in interimage processing and transfers information for the motion compensation and inter-image processing method, which detects errors by the decoding apparatus and switches to another frame without the use of a frame which had an error in the inter-image processing for each block, and which has no large image deterioration even if there is a coding error in the transfer path.
In order to attain this objective, as shown in FIG. 3, the present invention is a moving image coding apparatus which codes moving image signals in block units, and is a moving image coding apparatus which has means (apredictive subtractor 5, anadaptive predictor 13, and the like) for motion compensation inter-image processing between a plural number of frames (or fields) and means (memory 7, 8, 9 and 10 and selector 11) for combining coded information for each block and which has been coded by the processing means, for each type of coded information, into a plural number of block units (a plural number of macroblock units) and for then transferring it.
Furthermore, as shown in FIG. 4 for example, the present invention is a moving image decoding apparatus which comprises a decoding apparatus for moving image signals which have been coded in block units, and has means (error detector 38) for detecting transfer coding errors for each type of coded information, and means (adaptive predictor 38,variable adder 33, and the like) for performing motion compensation inter-image processing by changing the method of motion compensation inter-image processing for each block in accordance with errors in each type of coded information which have been detected, and which either does not use decoded signals of frames (or fields) having coding errors and limits use to only decoded signals of correct frames (or fields), or substitute them with decoded signals of other frames (or fields).
A decoding apparatus always makes a plural number of frames for use in the inter-image processing and recombines the information for that motion compensation and inter-image processing method for each type of information and so it is possible to lower the probability that a plural number of pieces of information of the same block will not be used even if error detection is performed in code units of a certain quantity.
In the decoding apparatus, there is the detection of transfer coding errors for each type of coded information and there is switching to another frame instead of using signals of frames having errors in the information for inter-image processing for each block. There is therefore very little image deterioration.
In the moving image coding apparatus and decoding apparatus of the present invention, the number of frames used for inter-image processing is always made a plural number, and the information for the motion compensation and inter-image processing method is recombined into each type of information and transferred, with errors for each type of information being detected and with inter-image processing for each block being switched to another frame without the use of signals of frames having errors in the information, thereby lowering the probability that a plural number of pieces of information in the same block will not be used even if error detection is performed in code units of a certain amount, and thereby enabling there to be little deterioration of the image quality.
By this, it is possible to not have a large amount of deterioration in the image quality even if there is a large number of errors in the transfer path. Accordingly, coding errors are permissible and it is not necessary to have a large amount of correction coding in the transfer coding, with the result that the amount of data can be reduced.
As has been described above, a moving image coding apparatus and decoding apparatus of the present invention has advantageous effects in its practical application.
BRIEF DESCRIPTION OF THE DRAWINGSIn the appended figures:
FIG. 1 is a block diagram showing an outline configuration of a conventional moving image signal coding apparatus;
FIG. 2 is a block diagram showing an outline configuration of a conventional moving image signal decoding apparatus;
FIG. 3 is a block diagram shown an outline configuration of a moving image signal coding apparatus according to a first embodiment of the present invention;
FIG. 4 is a block diagram showing an outline configuration of a moving image signal decoding apparatus according to a first embodiment of the present invention;
FIG. 5 is a block diagram showing an outline configuration of a moving image signal decoding apparatus according to a second embodiment of the present invention; and
FIGS. 6A and 6B are views showing the data configurations in a conventional coding/decoding apparatus and the coding/decoding apparatus according to the present invention.
DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTSThe following is a detailed description of the preferred embodiments of the moving image coding apparatus and decoding apparatus of the present invention, with reference to the appended drawings.
First is a description of the coding apparatus of a first embodiment of the present invention.
FIG. 3 is a block diagram showing an outline configuration of a moving image signal coding apparatus according to the first embodiment of the present invention. Those portions which correspond to portions of the coding apparatus of FIG. 1 are shown with corresponding numerals.
In FIG. 3, the coding processing is fundamentally the same, with the operation of the changeover switches 2, 4 and 22, thepredictive subtracter 5, theintraframe encoder 6, theintraframe decoder 21, theframe memories 18 and 19, themotion compensators 14 and 15, and themotion vector detectors 16 and 17 being the same (as shown by the dots and the like where there is coding in block units).
This coding apparatus differs from the conventional transfer method (FIG. 2) for each of the information of the DCT information which is the output of theintraframe encoder 6, the prediction mode information (MODE) which is the output of theadaptive predictor 42, and the motion vector information (MVF and MVB) which is the output of themotion vector estimators 16 and 17. More specifically,memories 7, 8, 9 and 10 and theselector 11 are configured so that each of the types of information is recombined in the memories and then transferred.
The DCT information which is the output of theintraframe encoder 6, the prediction mode information (MODE) which is the output of anadaptive predictor 13, and the motion vector information (MVF and MVB) which is the output of themotion vector estimators 16 and 17 are all respectively stored once in thememories 7, 8, 19 and 10. Then, at the time when from 30-300 macroblocks have been stored, there is output via thesequential data output 12 selected by theselector 11 and in the format shown in FIG. 6B. The size of the plural number of block units (macroblocks) for each type of the recombined coded information is sufficiently larger than the blocks for error detection, and can be set to a size smaller than the number of blocks of one frame. In this embodiment, dropout in cell units in an ATM circuit is for several macroblocks and the number of blocks of one frame is about 1350 macroblocks and so the size is set to 30-300 macroblocks as described above.
In addition, theadaptive predictor 13 is conventionally used only for the one frame (F-frames) for P frames but in the present embodiment, the configuration enables its use for a plural number of frames (F- and B-frames). Moreover, a motion vector is sent without the use of the predictive mode because of error correspondence in a decoder apparatus.
The following is a description of a decoder apparatus of the first embodiment.
FIG. 4 is a block diagram showing an outline configuration of a moving image signal decoding apparatus according to the first embodiment of the present invention. Those portions which correspond to portions of the coding apparatus of FIGS. 2 and 3 are shown by the same numerals. In FIG. 4, the decoding processing is fundamentally the same as that conventional example shown in FIG. 2, and the operation of the changeover switches 34 and 35, theintraframe decoder 21, theframe memories 18 and 19 and themotion compensators 14 and 15 are the same.
The differences with the conventional example (FIG. 2) is that there are thememories 7, 8, 9 and 10, theselector 31, anerror detector 38 and avariable adder 33 which changes the gain of the signals from the predictor and the intraframe decoding signals, with there being a different method of handling of each piece of information and different operation for anadaptive predictor 37 and thevariable adder 33.
More specifically, the coded data signals are transferred by the coding device and via thedata input terminal 30 and arrive at theselector 31 where they are separated into multiplexer DCT information, prediction mode information and motion vector information which are respectively stored in thememories 7, 8, 9 and 10.
Then, the DCT information which is stored in thememory 7 is applied to theintraframe decoder 21, the prediction mode information which is stored in thememory 8 is applied to theadaptive predictor 37, and the motion vector information which is stored in thememories 9 and 10 is applied to themotion compensators 14 and 15.
On the other hand, theerror detector 38 decodes the error detection code of the transmission path coding generated by the coding device, and judges the cell loss information for the ATM circuit to decide if there is an error in what type of information for the macroblock. The type of error changes the prediction mode according to the rules shown in Table 1.
In Table 1, C is a current decoding signal which is the output of the intraframe decoder. When there is dropout of mode information, the frame is which of B or F is temporally closer to C.
TABLE 1 ______________________________________ Dropout Information Image Used ______________________________________ DCT F, B mode C, B or F MVF C, B MVB C, F ______________________________________
When, as shown in Table 1 which is the error correspondence table, a plural number of images are used when a frame of information has no errors, and an error portion is discarded to use only a correct portion when any frame of information has errors. When there are errors in all of the frames which would have been used, there is also the use of frames which would not have been used. In other words, when an error is obtained in the motion vector MV of frame F even though only the frame F is used in the prediction mode, the intra-image prediction is performed by using the motion vector MV of the frame B which was not originally used. Conversely, under the condition where only the frame B is used in the prediction mode, when the error is obtained in the frame B, the motion vector of the frame F is used despite that the frame F was not originally used. In the present invention, both motion vectors of the frames B and F are transmitted together.
The operation of thevariable adder 33 is the same as theresidual adder 20 for the conventional example, and when there is dropout of the DCT information, the output of theintraframe decoder 21 is made zero, and the output of theadaptive predictor 37 is output as it is.
In this manner, according to this moving image coding apparatus and decoding apparatus, the number of frames used for inter-image processing is always made a plural number, and the information for the motion compensation and inter-image processing method is recombined into each type of information and transferred, with errors for each type of information being detected and with inter-image processing for each block having switching to another frame without the use of signals of frames having errors in the information, thereby lowering the probability that a plural number of pieces of information in the same block will not be used even if error detection is performed in coding units of a certain amount, and thereby enabling there to be little deterioration of the image quality.
The following is a description of a decoding apparatus of a second embodiment.
The second embodiment of the present invention is applicable to the apparatus "High-efficiency Coding Apparatus and Decoding Apparatus," disclosed in U.S. Ser. No. 07/873,949 filed on Apr. 24, 1992, the inventor of which is the inventor of the invention of the present application.
This embodiment has an improved coding efficiency while at the same time maintaining frame independence, and therefore even uses the inter-frame correlation for I-frames, and performs inter-frame image addition at the decoding apparatus, so that the coding method basically enables the image quality to be maintained even if the quantization is rough.
The difference with the first embodiment is the decoding apparatus, the configuration of which is shown in FIG. 5. The input data is coded by the coding apparatus shown in FIG. 3. The difference between the decoding apparatus of FIG. 5 and that of FIG. 4 is that there is amatching decider 32 which decides the matching of two images. More specifically, the configuration is such that the output of theintraframe decoder 21 and the output of the reproduced imagesignal output terminal 36 are both led to thematching decider 32 and thevariable adder 33.
In addition, the operation differs from the operation of the first embodiment in that the P- and B-frames are the same for only the I-frames. In the I-frames, thematching decider 32 checks the matching of the two images, and gives that much information to thevariable adder 33. In thevariable adder 33, the prediction signal from theadaptive predictor 37 is increased when there is good matching, while the current frame signal from theintraframe decoder 21 is increased when there is poor matching, and adding is then performed. Here, the sum of the gains of the respective signals is "1".
When there is an error in the signal from the intraframe decoder, the output of theintraframe decoder 21 is forcedly made "0" and only the output of theadaptive predictor 37 is used as the reproduced signals.
By this, it is possible to compensate for coding errors which have occurred in independent frames.
Moreover, I-frame quantization error compensation can use a coding apparatus in which the second embodiment of the present invention has been applied.