Rate-distortion optimization (RDO) is a method of improvingvideo quality invideo compression. The name refers to the optimization of the amount ofdistortion (loss of video quality) against the amount of data required to encode the video, therate. While it is primarily used by video encoders, rate-distortion optimization can be used to improve quality in any encoding situation (image, video, audio, or otherwise) where decisions have to be made that affect both file size and quality simultaneously.
The classical method of making encoding decisions is for the video encoder to choose the result which yields the highest quality output image. However, this has the disadvantage that the choice it makes might require more bits while giving comparatively little quality benefit. One common example of this problem is inmotion estimation,[1]and in particular regarding the use ofquarter pixel-precision motion estimation. Adding the extra precision to the motion of ablock during motion estimation might increase quality, but in some cases that extra quality isn't worth the extra bits necessary to encode the motion vector to a higher precision.
Rate-distortion optimization solves the aforementioned problem by acting as a video quality metric, measuring both the deviation from the source material and the bit cost for each possible decision outcome. The bits are mathematically measured by multiplying the bit cost by theLagrangian, a value representing the relationship between bit cost and quality for a particular quality level. The deviation from the source is usually measured as themean squared error, in order to maximize thePSNR video quality metric.
Calculating the bit cost is made more difficult by theentropy encoders in modern video codecs, requiring the rate-distortion optimization algorithm to pass each block of video to be tested to the entropy coder to measure its actual bit cost. InMPEG codecs, the full process consists of adiscrete cosine transform, followed byquantization and entropy encoding. Because of this, rate-distortion optimization is much slower than most other block-matching metrics, such as the simplesum of absolute differences (SAD) andsum of absolute transformed differences (SATD). As such it is usually used only for the final steps of themotion estimation process, such as deciding between different partition types inH.264/AVC.