BACKGROUND OF THE INVENTION1. Field of the Invention
The present invention relates to a moving image signal encoding apparatus for performing compression encoding of a moving image signal for transmission of the moving image signal or recording of the same on a recording medium and a decoding apparatus for decoding the codes which have been transmitted or reproduced from a recording medium to obtain a reproduced image.
2. Description of Prior Art
Recently, in the moving image signal encoding apparatus and decoding apparatus, in consequence of the developments of television telephones and television conference systems, various compression encoding systems have been in practical use. As a procedure for curtailing the information amount to be used in these encoding systems there is a frame decimation. This is to curtail the information amount by decimating selected frames of the moving image signal in encoding. Since the movements of the reproduced images become unnatural by decimating the frames, frame interpolation is carried out in the decoding apparatus to obtain the reproduced images having smooth movements.
As one of the precedents to carry out frame interpolation with a moving image signal decoding apparatus there is a construction shown in U.S. Pat. No. 4,727,422. Hereinafter, the construction of the conventional moving image signal encoding apparatus and decoding apparatus is described.
The input of the moving image signal encoding apparatus is a moving image signal of 30 frames/sec. The inputted moving image signal, after decimation of the number of frames to 1/2, becomes a moving image signal of 15 frames/sec. These remaining frames of the moving image signal will be encoded. In the description hereinafter, these frames are called "encoded frames". Interframe motion vectors are obtained from the encoded frames. The motion vectors are obtained on a block by block basis. The motion vectors are used for frame interpolation in the decoding apparatus. The encoded frames and the motion vectors are respectively encoded, after which additional information is incorporated to obtain an output signal of the moving image signal encoding apparatus. The output signal is sent out to a transmission channel or recorded on a recording medium.
The moving image signal decoding apparatus is to decode the signal encoded by the moving image signal encoding apparatus and reproduce the moving image signal. By a signal receiving circuit, each code is received from the transmission channel or read out from the recording medium. The codes are decoded by respective decoding circuits to become the reproduced frames and the motion vectors. The frequency of the reproduced frames is 15 frames/sec. A frame interpolation circuit obtains interpolated frames each positioned between two reproduced frames. The frame interpolation is a motion compensating frame interpolation using a motion vector between the frames. By alternately outputting the interpolated frames and the reproduced frames, an output image signal of 30 frames/sec is obtained.
However, the above construction involves the problem to cause errors to the interpolated frames because there are no correct motion vectors in such cases that: (1) there are objects which move in different directions from each other in a block; (2) the background appears from the shade of a moving object or the background is hidden by a moving object; (3) the moving object changes in shape; and (4) there is a movement accompanied with rotation.
SUMMARY OF THE INVENTIONAn object of the present invention is to realize reduction in interpolation errors of frames in a moving image signal encoding apparatus which decimates frames in encoding and a moving image signal decoding apparatus which interpolates frames in decoding.
To achieve this object, a moving image signal encoding apparatus of the present invention comprises a frame decimator for extracting . .encoded.!. frames from an input moving image signal at specified intervals, a frame encoder for encoding said . .encoded.!. frames .Iadd.extracted by the decimator .Iaddend.to obtain frame codes; a frame interpolator for producing interpolated frames positioned between said . .encoded.!. .Iadd.extracted .Iaddend.frames from said frame codes; a motion estimator for evaluating errors of said interpolated frames, and a transmitter for transmitting said frame codes and output signals of said error evaluator as an output signal of the moving image signal encoding apparatus.
A moving image signal decoding apparatus of the present invention is to decode the signals transmitted from the aforementioned moving image signal encoding apparatus, and comprises: a receiver for extracting said frame codes and said error evaluator output signals from the input signal; a frame decoder for decoding said frame codes to obtain the reproduced frames, and a frame interpolator for producing interpolated frames positioned between said reproduced frames.
Preferably, said error evaluator includes a means for encoding the errors of said interpolated frames to error codes, and said moving image signal decoding apparatus includes a means for correcting the errors of said interpolated frames according to the error codes to obtain said reproduced frames and error-corrected interpolated frames in a specified sequence.
Alternatively, said error evaluator includes a means for obtaining the errors of said interpolated frames to obtain error codes and a means for producing a mode selection signal from the error codes for changing over an operation mode of the frame interpolator of the moving image signal decoding apparatus, and, in the moving image signal decoding apparatus, the frame interpolator includes a means for selecting whether to perform frame interpolation or to hold a preceding reproduced frame (to repeat the preceding reproduced frame) according to the mode selection signal, and outputting said reproduced frames and said interpolated frames in a specified sequence.
BRIEF DESCRIPTION OF THE DRAWINGSFIG. 1 is a block diagram showing a moving image signal encoding apparatus and a moving image signal decoding apparatus in accordance with a first embodiment of the present invention;
FIGS. 2(a)-2(b) are illustrative . .view to explain.!. .Iadd.views for explaining .Iaddend.a . .relation,.!. .Iadd.relationship .Iaddend.between frames;
FIG. 3 is a block diagram of a moving image signal encoding apparatus in accordance with a second embodiment of the present invention;
FIG. 4 is a block diagram of a moving image signal decoding apparatus in accordance with the second embodiment of the present invention;
FIG. 5 is a block diagram of an error evaluator and coder of a moving image signal decoding apparatus in accordance with a third embodiment of the present invention;
FIG. 6 is a view showing an interpolated frame divided into a plurality of blocks;
FIG. 7 is a block diagram of a moving image signal encoding apparatus in accordance with a fourth embodiment of the present invention;
FIG. 8 is a block diagram of a moving image signal decoding apparatus in accordance with the fourth embodiment of the present invention;
FIGS. 9(a)-9(b) are views showing a relationship between frames in accordance with the fourth embodiment of the present invention.
DESCRIPTION OF PREFERRED EMBODIMENTSHereinafter, the moving image signal encoding apparatus and decoding apparatus according to the embodiments of the present invention are explained with reference to the drawings.
FIG. 1 shows a block diagram of a moving image signal encoding apparatus and a decoding apparatus in accordance with a first embodiment of the present invention. In FIG. 1, the numeral 1 denotes the moving image signal encoding apparatus for encoding and transmitting a moving image signal, in whichelement 101 is an input moving image signal;element 102 is a frame decimator; 103 is a frame encoder; 104 is a local frame decoder; 105 is a frame interpolator; 106 is an error calculator and coder; 107 is a multiplexer and transmitter, andelement 108 is an output signal of the moving image signal encoding apparatus. Thenumeral 2 denotes the moving image signal decoding apparatus for reproducing the moving image signal, in whichelement 201 is an input signal of the moving image signal decoding apparatus;element 202 is a receiver and demultiplexer; 203 is a frame decoder; 204 is a frame interpolator; 205 is an error corrector; 206 is a selector, and 207 is an output signal of the moving image signal decoding apparatus.
The operations of the moving image signal encoding apparatus and decoding apparatus constituted as above are explained by way of FIG. 1.
To an input of the moving image signal encoding apparatus, a movingimage signal 101 to be encoded is inputted. Aframe decimator 102 decimates the frames of the input movingimage signal 101 by one-half. This operation is illustrated in FIG. 2(a). A, B, C, D are the continued frames of the input moving image signal, of which B and D are the frames to be decimated by theframe decimator 102, and A and C are the frames to be inputted to theframe encoder 103. In the following explanation, frames A and C are called the "encoded frames". Theframe encoder 103 encodes each . .encoded.!. frame to .Iadd.obtain .Iaddend.aframe code 1038. Theframe decoder 104 decodes the frame code to obtain a reproducedframe 1046. Theframe interpolator 105 synthesizes interpolatedframes 1051 each positioned between two reproduced frames. The relationship between the reproduced frames and the interpolated frames is explained in FIG. 2(b). A' and C' are the reproduced frames, and these correspond to the encoded frames A and C. The encoded frames A and C are encoded by theframe encoder 103 and then decoded by theframe decoder 104 to be the reproduced frames A' and C'. Bi and Di are the interpolated frames outputted by theframe interpolator 105. The error evaluator andcoder 106 encodes an error of each interpolator frame obtained as a difference between the interpolated frame and a corresponding frame of the input moving image signal, and outputs it as an interpolatedframe code 1066. The multiplexer andtransmitter 107 multiplexes the frame codes and the interpolated frame codes, and outputs the multiplexed result as anoutput signal 108 of the moving image signal encoding apparatus.
Next, the operations of the moving imagesignal decoding apparatus 2 are explained. The moving imagesignal decoding apparatus 2 is to decode the inputtedsignal 201 and output a reproduced moving image signal. The inputtedsignal 201 is theoutput signal 108 of the moving image signal encoding apparatus 1. The receiver anddemultiplexer 202 extracts theframe codes 2021 and the interpolatedframe codes 2022 from the inputted signal. These codes are respectively equal to theframe codes 1038 and the interpolatedframe codes 1066 of the moving image signal encoding apparatus 1. Theframe decoder 203 decodes the frame codes and outputs reproduced frames 2036. Theframe interpolator 204 synthesizes interpolatedframes 2041 each positioned between two reproduced frames. The relationship between the reproduced frames and the interpolated frames is the same as in the case of the moving image signal encoding apparatus 1 as shown in FIG. 2(b). Theerror corrector 205 corrects the errors of the interpolated frames by using the interpolatedframe codes 2022. Theselector 206 alternately selects the reproducedframes 2036 and the error-correctedinterpolated frames 2037 to obtain theoutput signal 207 of the moving image signal encoding apparatus. Adisplay apparatus 208 displays the reproduced image based on theoutput signal 207 of the moving imagesignal encoding apparatus 2.
FIG. 3 shows a block diagram of a moving image signal encoding apparatus in accordance with a second embodiment of the present invention. In FIG. 3,element 101 is an input moving image signal;element 102 is a frame decimator;element 103 is a frame encoder;element 1033 is a subtraction circuit;element 1035 is a DCT (discrete cosine transform) operation circuit;element 1037 is a quantizer;element 104 is a local frame decoder;element 1041 is a dequantizer;element 1043 is an inverse DCT (IDCT) operation circuit;element 1045 is an addition circuit;element 1047 is a frame memory;element 1048 is a motion compensator;element 105 is a frame interpolator;element 106 is an error evaluator and coder;element 1061 is a subtraction circuit;element 1063 is a DCT operation circuit, 1065 is a quantizer;element 107 is a multiplexer and transmitter;element 108 is an output signal of the moving image signal encoding apparatus, andelement 1091 is a motion estimator.
The operations of the moving image signal encoding apparatus constituted as above are explained by way of FIG. 3.
Themotion estimator 1091 estimates the motion of the input movingimage signal 101 and outputs amotion vector 1092.
Theframe decimator 102 decimates the frames of the input moving image signal by one-half.Iadd.. .Iaddend.The operation of theframe decimator 102 is the same as that of the first embodiment.
Theframe encoder 103 encodes the . .encoded.!.frames 1031 to .Iadd.obtain .Iaddend.frame codes 1038. The encoding method is an interframe coding. Thesubtraction circuit 1033 obtains a predictederror signal 1034 which is a differential value between the encodedframe 1031 and a predictedframe 1032 formed by the later-describedlocal decoder 104 TheDCT operation circuit 1035 transforms the predictederror signal 1034 to aDCT coefficient 1036. Thequantizer 1037 quantizes theDCT coefficient 1036 to obtain theframe code 1038.
Thelocal decoder 104 decodes theframe code 1038 to obtain a reproducedframe 1046 and the predictedframe 1032. Thedequantizer 1041 dequantizes theframe code 1038 to obtain a reproducedDCT coefficient 1042. The inverseDCT operation circuit 1043 inverse discrete cosine transforms the reproducedDCT coefficient 1042 to obtain a reproduced predictederror signal 1044. Theaddition circuit 1045 adds the reproduced predictederror signal 1044 and the predictedframe 1032 to obtain the reproducedframe 1046. Theframe memory 1047 stores the reproducedframe 1046. Themotion compensator 1048 carries out a motion compensation of the reproduced frame read out from theframe memory 1047 according to themotion vector 1092 to obtain the predictedframe 1032.
Theframe interpolator 105 synthesizes an interpolatedframe 1051 from themotion vector 1092 and the reproducedframe 1046. The relationship between the reproduced frames and the interpolated frames is the same as that explained in the first embodiment.
The error evaluator andcoder 106 encodes the error of the interpolatedframe 1051 to obtain an interpolatedframe code 1066. Thesubtraction circuit 1061 calculates a differential value between the interpolatedframe 1051 and a corresponding frame of the input movingimage signal 101 to obtain an interpolatedframe error signal 1062. The DCT (Discrete Cosine Transform)operation circuit 1063 transforms the interpolatedframe error signal 1062 to aDCT coefficient 1064. Thequantizer 1065 quantizes theDCT coefficient 1064 to obtain the interpolatedframe code 1066.
The multiplexer andtransmitter 107 multiplexes and outputs theframe code 1038, themotion vector 1092, and the interpolatedframe code 1066 as theoutput signal 108 of the moving image signal encoding apparatus.
FIG. 4 shows a block diagram of a moving image signal decoding apparatus in accordance with the second embodiment of the present invention. In FIG. 4,element 201 is an input signal of the moving image signal decoding apparatus;element 202 is a receiver and multiplexer;element 203 is a frame decoder;element 2031 is a dequantizer;element 2033 is an inverse DCT operation circuit. .,.!..Iadd.; element .Iaddend.2035 is an addition circuit. .,.!..Iadd.; element .Iaddend.2037 is a frame memory. .,.!..Iadd.; element .Iaddend.2038 is a motion compensator;element 204 is a frame interpolator;element 205 is an error corrector;element 2051 is a dequantizer;element 2053 is an inverse DCT operation circuit;element 2055 is an addition circuit;element 206 is a selector, andelement 207 is an output signal of the moving image signal decoding circuit.
The operations of the moving image signal decoding apparatus constituted as above are explained by way of FIG. 4.
Theinput signal 201 is an output signal of the moving image signal encoding apparatus of FIG. 3. The receiver anddemultiplexer 202 extracts aframe code 2021, an interpolatedframe code 2022, and amotion vector 2023 from theinput signal 201. These codes are equal to theframe code 1038, the interpolatedframe code 1066, and themotion vector 1092, respectively, of the moving image signal encoding apparatus in FIG. 3.
Theframe decoder 203 decodes theframe code 2021 to obtain a reproducedframe 2036. Thedequantizer 2031 dequantizes theframe code 2031 to obtain a reproducedDCT coefficient 2032. The inverseDCT operation circuit 2035 inverse discrete cosine transforms the reproducedDCT coefficient 2032 to obtain a reproduced predictederror signal 2034. Theaddition circuit 2035 adds the reproduced predictederror signal 2034 and a predictedframe 2039 formed by the later-describedmotion compensator 2038 and to obtain the reproducedframe 2036. Theframe memory 2037 stores the reproducedframe 2036. Themotion compensator 2032 carries out a motion compensation of the reproduced frame read out from theframe memory 2037 according to themotion vector 2023 to obtain the predictedframe 2039.
Theframe interpolator 204 synthesizes an interpolatedframe 2041 from themotion vector 2023 and the reproducedframe 2036. The relationship between the reproduced frame and the interpolated frame is the same as explained in the first embodiment.
Theerror corrector 205 corrects the error of the interpolatedframe 2041 by using the interpolatedframe code 2022. Thedequantizer 2051 dequantizes the interpolatedframe code 2022 to obtain a reproducedDCT coefficient 2052. The inverseDCT operation circuit 2053 inversely discrete cosine transforms the reproducedDCT coefficient 2052 to obtain an interpolatedframe error signal 2054. Theaddition circuit 2055 adds the reproduced interpolatedframe error signal 2054 and the interpolatedframe 2041 to obtain a reproduced interpolatedframe 2056. Theselector 206 alternately selects the reproducedframes 2036 and the reproduced interpolatedframes 2056 to obtain theoutput signal 207 of the moving image signal decoding apparatus and supplies theoutput signal 207 to thedisplay apparatus 208.
FIG. 5 shows a block diagram of an error evaluator and coder of a moving image signal encoding apparatus in accordance with a third embodiment of the present invention. The constructions of the other parts are the same as those in the second embodiment shown in FIG. 3. In FIG. 5,element 1051 is an input interpolated frame;element 101 is an input moving image signal of the moving image signal encoding apparatus;element 1061 is a subtraction circuit;element 1063 is a DCT operation circuit;element 1065 is a quantizer;element 1068 is an error calculator;element 10611 is a comparator,element 10610 is a reference level;element 10613 is a switch, andelement 1066 is an interpolated frame code.
The operations of the error evaluator and coder circuit constituted as above are explained by way of FIG. 5. Thesubtraction circuit 1061 obtains an interpolatedframe error signal 1062 which is a differential value between the input interpolatedframe 1051 and the input movingimage signal 101. TheDCT operation circuit 1063 transforms the interpolatedframe error signal 1062 to aDCT coefficient 1064. Thequantizer 1065 quantizes theDCT coefficient 1064 to obtain acode 10614. Theerror calculator 1068 obtains the value of the interpolatedframe error signal 1062 on a block by block by block basis to obtain anerror value 1069. This block is explained with reference to FIG. 6. In FIG. 6,element 3001 is interpolated frame, andelement 3002 is a block in this frame. The interpolated frame is divided at intervals of 8 image elements both vertically and horizontally to obtain each block. Thecomparator 1061 compares theerror value 1069 with a specifiedreference level 10610, and closes theswitch 1061 when theerror value 1069 exceeds thereference level 10610. When the switch is closed, theoutput code 10614 of thequantizer 1065 becomes the interpolatedframe code 1066 which is an output of the interpolated frame encoding circuit.
FIG. 7 shows a block diagram of a moving image signal encoding apparatus in accordance with a fourth embodiment of the present invention. In FIG. 7,element 101 is an input moving image signal;element 102 is a frame decimator;element 103 is a frame encoder;element 104 is a frame decoder;element 105 is a frame interpolator;element 107 is a transmitter and multiplexer;element 108 is an output signal of the moving image signal encoding apparatus;element 1091 is a motion estimator;element 1093 is an error evaluator, andelement 1094 is a selector.
The operations of the moving image signal encoding apparatus constituted as above are explained by way of FIG. 7.
Theframe decimator 102 decimates the frames of the input movingimage signal 101 by one-half. The operation of theframe decimator 102 is the same as that of the first embodiment.
Theframe encoder 103 encodes the . .encoded.!. frame to obtain theframe code 1038. Theframe decoder 104 decodes the . .encoded.!. frame .Iadd.code .Iaddend.to obtain the reproducedframe 1046. Theframe interpolator 105 produces the interpolatedframe 1051 from themotion vector 1092 and the reproducedframe 1046. The relationship between the reproduced frame and the interpolated frame is the same as that explained in the first embodiment.
Theerror evaluator 1093 obtains the error of the interpolatedframe 1051 and outputs amode selection signal 10931. When the error of the interpolated frame is smaller than a predetermined reference level, the mode selector signal becomes a code indicating a frame interpolation mode. When the error is larger than the reference level, the mode selection signal becomes a code indicating a previous value retaining mode. Theselector 1094 outputs as itsoutput 1096 themotion vector 1092 when the mode selection signal indicates the frame interpolation mode, and outputs a flag signal when the mode selection signal indicates the previous value retaining mode.
The multiplexer andtransmitter 107 multiplexes and outputs theframe code 1038 and theoutput 1096 of theselector 1094 as theoutput signal 108 of the moving image signal encoding apparatus.
FIG. 8 shows a block diagram of the decoding apparatus in accordance with the fourth embodiment of the present invention. In FIG. 8, .Iadd.element .Iaddend.201 is an input signal of the moving image signal decoding apparatus. .,.!..Iadd.; element .Iaddend.202 is a receiver and demultiplexer. .,.!..Iadd.; element .Iaddend.203 is a frame decoder. .,.!..Iadd.; element .Iaddend.204 is a frame interpolator. .,.!..Iadd.; element .Iaddend.206 is a selector, and .Iadd.element .Iaddend.207 is an output signal of the moving image signal decoding apparatus.
The operations of the moving image signal decoding apparatus constituted as above are explained by way of FIG. 8.
The moving imagesignal decoding apparatus 2 is to decode the inputtedsignal 201 and output the movingimage signal 207. The inputtedsignal 201 is theoutput signal 108 of the moving image signal encoding apparatus of FIG. 6. The receiver anddemultiplexer 202 extracts from the inputted signal 201 aframe code 2021 and asignal 2024 which is the motion vector or the flag signal.
When the motion vector is extracted from the receiver anddemultiplexer 202, the operation sequence is as follows. Theframe decoder 203 decodes theframe code 2021 to obtain the reproducedframe 2036. Theframe interpolator 204 interpolates a frame between the reproduced frames. Theselector 206 alternately selects the reproducedframes 2036 and the interpolatedframes 2087 to obtain theoutput signal 207 of the moving image signal decoding apparatus. In FIG. 9(a) the output signal of the moving image signal decoding apparatus is shown, in which A' and C' are the reproduced frames, and Bi' and Di' are the interpolated frames.
Further, when the flag signal is extracted as theoutput 2024 of the receiver anddemultiplexer 202, the operation sequence is as follows. Theframe decoder 203 decodes theframe code 2021 to obtain the reproducedframe 2036. Theframe interpolator 204 obtains the frame positioned between the reproduced frames by holding the preceding frame. The held previous reproduced frame is outputted as theoutput frame 2087. This operation is shown in FIG. 9(b). The frames A' and C' at the time t0 and t2 are the reproduced frames, and the frames A' and C' at the time t1 and t3 are those in which the frames A' and C' at the time t0 and t2 are respectively held for 1 frame period of time, i.e; the reproduced frames A' and C' are repeated. Theselector 206 alternately selects the reproducedframes 2036 and the output frames 2087 of the frominterpolator 204 to obtain theoutput signal 207 of the moving image signal decoding apparatus. Theoutput signal 207 is displayed as a reproduced image by thedisplay apparatus 208.