| RFC 0000 | Video Codec Requirements and Evaluation | February 2020 |
| Filippov, et al. | Informational | [Page] |
This document provides requirements for a video codec designed mainly for use over the Internet. In addition, this document describes an evaluation methodology needed for measuring the compression efficiency to ensure whether the stated requirements are fulfilled or not.¶
This document is not an Internet Standards Track specification; it is published for informational purposes.¶
This document is a product of the Internet Engineering Task Force (IETF). It represents the consensus of the IETF community. It has received public review and has been approved for publication by the Internet Engineering Steering Group (IESG). Not all documents approved by the IESG are candidates for any level of Internet Standard; see Section 2 of RFC 7841.¶
Information about the current status of this document, any errata, and how to provide feedback on it may be obtained athttps://www.rfc-editor.org/info/rfc0000.¶
Copyright (c) 2020 IETF Trust and the persons identified as the document authors. All rights reserved.¶
This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (https://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Simplified BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Simplified BSD License.¶
This document presents the requirements for a video codec designed mainly foruse over the Internet. The requirements encompass a wide rangeof applications that use data transmission over the Internet, includingInternet video streaming, IPTV, peer-to-peer video conferencing, videosharing, screencasting, game streaming, and video monitoring /surveillance. For each application, typical resolutions, frame rates, andpicture-access modes are presented.Specific requirements related to data transmission over packet-loss networksare considered, as well. In this document, when we discuss data-protectiontechniques, we only refer to methods designed and implemented to protect datainside the video codec, since there are many existing techniques that protectgeneric data transmitted over networks with packet losses. From thetheoretical point of view, both packet-loss and bit-error robustness can bebeneficial for video codecs. In practice, packet losses are a more significantproblem than bit corruption in IP networks. It is worth noting that there isan evident interdependence between possible amount of delay and the necessityof error-robust video streams:¶
o If an amount of delay is not crucial for an application, then reliabletransport protocols such as TCP that retransmit undelivered packets can beused to guarantee correct decoding of transmitted data.¶
o If the amount of delay must be kept low, then either data transmissionshould be error free (e.g., by using managed networks), or the compressedvideo stream should be error resilient.¶
Thus, error resilience can be useful for delay-critical applications toprovide low delay in a packet-loss environment.¶
| Abbreviation | Meaning |
|---|---|
| AI | All-Intra (each picture is intra-coded) |
| BD-Rate | Bjontegaard Delta Rate |
| FIZD | just the First picture is Intra-coded, Zero structural Delay |
| GOP | Group of Picture |
| HBR | High Bitrate Range |
| HDR | High Dynamic Range |
| HRD | Hypothetical Reference Decoder |
| IPTV | Internet Protocol Television |
| LBR | Low Bitrate Range |
| MBR | Medium Bitrate Range |
| MOS | Mean Opinion Score |
| MS-SSIM | Multi-Scale Structural Similarity quality index |
| PAM | Picture Access Mode |
| PSNR | Peak Signal-to-Noise Ratio |
| QoS | Quality of Service |
| QP | Quantization Parameter |
| RA | Random Access |
| RAP | Random Access Period |
| RD | Rate-Distortion |
| SEI | Supplemental Enhancement Information |
| SNR | Signal-to-Noise Ratio |
| UGC | User-Generated Content |
| VDI | Virtual Desktop Infrastructure |
| VUI | Video Usability Information |
| WCG | Wide Color Gamut |
In this section, an overview of video codec applications that are currentlyavailable on the Internet market is presented. It is worth noting that thereare different use cases for each application that define a target platform,and hence there are different types of communication channels involved (e.g.,wired or wireless channels), which are characterized by different qualities ofservice as well as bandwidth; for instance, wired channels are considerablymore free from error than wireless channels and therefore require different QoSapproaches. The target platform, the channel bandwidth, and the channel qualitydetermine resolutions, frame rates, and quality (bitrates) for encoding ordecoding video streams. By default, color format YCbCr 4:2:0 is assumed forthe application scenarios listed below.¶
Typical content for this application is movies, TV shows, andanimation. Internet video streaming uses a variety of client devices and hasto operate under changing network conditions. For this reason, an adaptivestreaming model has been widely adopted. Video material is encoded atdifferent quality levels and different resolutions, which are then chosen by aclient depending on its capabilities and current network bandwidth. An examplecombination of resolutions and bitrates is shown inTable 2.¶
A video encoding pipeline in on-demand Internet video streaming typically operates as follows:¶
| Resolution * | Frame rate, fps | PAM |
|---|---|---|
| 4K, 3840x2160 | 24/1.001, 24, 25, 30/1.001, 30, 50, 60/1.001, 60, 100, 120/1.001, 120. The set of frame rates presented in this table is taken from Table 1 in[PARAMETER] | RA |
| 2K (1080p), 1920x1080 | RA | |
| 1080i, 1920x1080* | RA | |
| 720p, 1280x720 | RA | |
| 576p (EDTV), 720x576 | RA | |
| 576i (SDTV), 720x576* | RA | |
| 480p (EDTV), 720x480 | RA | |
| 480i (SDTV), 720x480* | RA | |
| 512x384 | RA | |
| QVGA, 320x240 | RA |
*NB: Interlaced content can be handled at the higher system level and not necessarily by using specialized video coding tools. It is included in this table only for the sake of completeness, as most video content today is in the progressive format.¶
The characteristics and requirements of this application scenario are as follows:¶
Support and efficient encoding of a wide range of content types and formats is required:¶
This is a service for delivering television content over IP-based networks. IPTV may be classified into two main groups based on the type of delivery, as follows:¶
In the IPTV scenario, traffic is transmitted over managed (QoS- based)networks. Typical content used in this application is news, movies, cartoons,series, TV shows, etc. One important requirement for both groups is randomaccess to pictures -- i.e., random access period (RAP) -- should be kept smallenough (approximately, 1-5 seconds). Optional requirements are as follows:¶
For this application, typical values of resolutions, frame rates, and RAPsare presented inTable 3.¶
| Resolution * | Frame rate, fps | PAM |
|---|---|---|
| 2160p (4K),3840x2160 | 24/1.001, 24, 25, 30/1.001, 30, 50, 60/1.001, 60, 100, 120/1.001, 120. The set of frame rates presented in this table is taken from Table 2 in[PARAMETER] | RA |
| 1080p, 1920x1080 | RA | |
| 1080i, 1920x1080* | RA | |
| 720p, 1280x720 | RA | |
| 576p (EDTV), 720x576 | RA | |
| 576i (SDTV), 720x576* | RA | |
| 480p (EDTV), 720x480 | RA | |
| 480i (SDTV), 720x480* | RA |
*NB: Interlaced content can be handled at the higher system level and not necessarily by using specialized video coding tools. It is included in this table only for the sake of completeness, as most video content today is in a progressive format.¶
This is a form of video connection over the Internet. This form allowsusers to establish connections to two or more people by two- way video andaudio transmission for communication in real time. For this application, bothstationary and mobile devices can be used. The main requirements are asfollows:¶
Support of resolution and quality (SNR) scalability is highlydesirable. For this application, typical values of resolutions, frame rates,and RAPs are presented inTable 4.¶
| Resolution | Frame rate, fps | PAM |
|---|---|---|
| 1080p, 1920x1080 | 15, 30 | FIZD |
| 720p, 1280x720 | 30, 60 | FIZD |
| 4CIF, 704x576 | 30, 60 | FIZD |
| 4SIF, 704x480 | 30, 60 | FIZD |
| VGA, 640x480 | 30, 60 | FIZD |
| 360p, 640x360 | 30, 60 | FIZD |
This is a service that allows people to upload and share video data (usinglive streaming or not) and watch those videos. It is also known as video hosting. Atypical User-Generated Content (UGC) scenario for this application is tocapture video using mobile cameras such as GoPros or cameras integrated intosmartphones (amateur video). The main requirements are as follows:¶
Support of resolution and quality (SNR) scalability is highlydesirable. For this application, typical values of resolutions, frame rates,and RAPs are presented inTable 5.¶
| Resolution | Frame rate, fps | PAM |
|---|---|---|
| 2160p (4K),3840x2160 | 24, 25, 30, 48, 50, 60 | RA |
| 1440p (2K),2560x1440 | 24, 25, 30, 48, 50, 60 | RA |
| 1080p, 1920x1080 | 24, 25, 30, 48, 50, 60 | RA |
| 720p, 1280x720 | 24, 25, 30, 48, 50, 60 | RA |
| 480p, 854x480 | 24, 25, 30, 48, 50, 60 | RA |
| 360p, 640x360 | 24, 25, 30, 48, 50, 60 | RA |
This is a service that allows users to record and distributecomputer-desktop screen output. This service requires efficient compression ofcomputer-generated content with high visual quality up to visually andmathematically (numerically) lossless[HEVC-EXT]. Currently, this applicationincludes business presentations (PowerPoint, Word documents, email messages,etc.), animation (cartoons), gaming content, and data visualization -- i.e., thetype of content that is characterized by fast motion, rotation, smooth shade,3D effect, highly saturated colors with full resolution, clear textures,sharp edges with distinct colors[HEVC-EXT]), virtual desktopinfrastructure (VDI),screen/desktop sharing and collaboration, supervisory control and dataacquisition (SCADA) display, automotive/navigation display, cloud gaming, factory automationdisplay, wireless display, display wall,digital operating room (DiOR), etc. For this application, an importantrequirement is the support of low-delay configurations with zero structuraldelay for a wide range of video formats (e.g., RGB) in addition to YCbCr 4:2:0and YCbCr 4:4:4[HEVC-EXT]. For this application, typical values of resolutions,frame rates, and RAPs are presented inTable 6.¶
| Resolution | Frame rate, fps | PAM |
|---|---|---|
| Input color format: RGB 4:4:4 | ||
| 5k, 5120x2880 | 15, 30, 60 | AI, RA, FIZD |
| 4k, 3840x2160 | 15, 30, 60 | AI, RA, FIZD |
| WQXGA, 2560x1600 | 15, 30, 60 | AI, RA, FIZD |
| WUXGA, 1920x1200 | 15, 30, 60 | AI, RA, FIZD |
| WSXGA+, 1680x1050 | 15, 30, 60 | AI, RA, FIZD |
| WXGA, 1280x800 | 15, 30, 60 | AI, RA, FIZD |
| XGA, 1024x768 | 15, 30, 60 | AI, RA, FIZD |
| SVGA, 800x600 | 15, 30, 60 | AI, RA, FIZD |
| VGA, 640x480 | 15, 30, 60 | AI, RA, FIZD |
| Input color format: YCbCr 4:4:4 | ||
| 5k, 5120x2880 | 15, 30, 60 | AI, RA, FIZD |
| 4k, 3840x2160 | 15, 30, 60 | AI, RA, FIZD |
| 1440p (2K), 2560x1440 | 15, 30, 60 | AI, RA, FIZD |
| 1080p, 1920x1080 | 15, 30, 60 | AI, RA, FIZD |
| 720p, 1280x720 | 15, 30, 60 | AI, RA, FIZD |
This is a service that provides game content over the Internet to differentlocal devices such as notebooks and gaming tablets. In this category ofapplications, the server renders 3D games in a cloud server and streams the game toany device with a wired or wireless broadband connection[GAME]. There are low-latency requirements for transmitting user interactions andreceiving game data with a turnaround delay of less than 100 ms. This allowsanyone to play (or resume) full-featured games from anywhere in the Internet[GAME]. An example of this application is Nvidia Grid[GAME]. Another category application is broadcast of video gamesplayed by people over the Internet in real time or for later viewing[GAME]. There are many companies, such as Twitch and YY in China, that enablegame broadcasting[GAME]. Games typically contain a lot ofsharp edges and large motion[GAME]. The main requirements areas follows:¶
Support of resolution and quality (SNR) scalability is highlydesirable. For this application, typical values of resolutions, frame rates,and RAPs are similar to ones presented inTable 4.¶
This is a type of live broadcasting over IP-based networks. Video streamsare sent to many receivers at the same time. A new receiver may connect to thestream at an arbitrary moment, so the random access period should be kept smallenough (approximately, ~1-5 seconds). Data are transmitted publicly in thecase of video monitoring and privately in the case of video surveillance,respectively. For IP cameras that have to capture, process, and encode videodata, complexity -- including computational and hardware complexity, as well asmemory bandwidth -- should be kept low to allow real-time processing. Inaddition, support of a high dynamic range and a monochrome mode (e.g., forinfrared cameras) as well as resolution and quality (SNR) scalability is anessential requirement for video surveillance. In some use cases, highvideo-signal fidelity is required even after lossy compression. Typical valuesofresolutions, frame rates, and RAPs for video monitoring / surveillanceapplications are presented inTable 7.¶
| Resolution | Frame rate, fps | PAM |
|---|---|---|
| 2160p (4K),3840x2160 | 12, 25, 30 | RA, FIZD |
| 5Mpixels, 2560x1920 | 12, 25, 30 | RA, FIZD |
| 1080p, 1920x1080 | 25, 30 | RA, FIZD |
| 1.3Mpixels, 1280x960 | 25, 30 | RA, FIZD |
| 720p, 1280x720 | 25, 30 | RA, FIZD |
| SVGA, 800x600 | 25, 30 | RA, FIZD |
Taking the requirements discussed above for specific video applications,this section proposes requirements for an Internet video codec.¶
The most basic requirement is coding efficiency -- i.e., compressionperformance on both "easy" and "difficult" content for applications and use cases inSection 2. The codec should provide higher coding efficiency overstate-of-the-art video codecs such as HEVC/H.265 and VP9 by at least 25%, inaccordance with the methodology described inSection 4.1 of this document. Forhigher resolutions, the coding efficiency improvements are expected to behigher than for lower resolutions.¶
Good-quality specification and well-defined profiles and levels arerequired to enable device interoperability and facilitate decoderimplementations. A profile consists of a subset of entire bitstream syntaxelements, and consequently it also defines the necessary tools for decoding aconforming bitstream of that profile. A level imposes a set of numericallimits to the values of some syntax elements. An example of codec levels to besupported is presented inTable 8. An actual leveldefinition should include constraints on features that impact the decodercomplexity. For example, these features might be as follows: maximum bitrate,line buffer size, memory usage, etc.¶
| Level | Example picture resolution at highest frame rate |
|---|---|
| 1 | 128x96(12,288*)@30.0 176x144(25,344*)@15.0 |
| 2 | 352x288(101,376*)@30.0 |
| 3 | 352x288(101,376*)@60.0 640x360(230,400*)@30.0 |
| 4 | 640x360(230,400*)@60.0 960x540(518,400*)@30.0 |
| 5 | 720x576(414,720*)@75.0 960x540(518,400*)@60.0 1280x720(921,600*)@30.0 |
| 6 | 1,280x720(921,600*)@68.0 2,048x1,080(2,211,840*)@30.0 |
| 7 | 1,280x720(921,600*)@120.0 |
| 8 | 1,920x1,080(2,073,600*)@120.0 3,840x2,160(8,294,400*)@30.0 4,096x2,160(8,847,360*)@30.0 |
| 9 | 1,920x1,080(2,073,600*)@250.0 4,096x2,160(8,847,360*)@60.0 |
| 10 | 1,920x1,080(2,073,600*)@300.0 4,096x2,160(8,847,360*)@120.0 |
| 11 | 3,840x2,160(8,294,400*)@120.0 8,192x4,320(35,389,440*)@30.0 |
| 12 | 3,840x2,160(8,294,400*)@250.0 8,192x4,320(35,389,440*)@60.0 |
| 13 | 3,840x2,160(8,294,400*)@300.0 8,192x4,320(35,389,440*)@120.0 |
*NB: The quantities of pixels are presented for applications in which a picture can have an arbitrary size (e.g., screencasting)¶
Bitstream syntax should allow extensibility and backwardcompatibility. New features can be supported easily by using metadata (such asSEI messages, VUI, and headers) without affecting the bitstream compatibilitywith legacy decoders. A newer version of the decoder shall be able to playbitstreams of an older version of the same or lower profile and level.¶
A bitstream should have a model that allows easy parsing andidentification of the sample components (such as Annex B of[ADVANCED] or[NAL]). In particular, informationneeded for packet handling(e.g., frame type) should not require parsing anything below the headerlevel.¶
Perceptual-quality tools (such as adaptive QP and quantization matrices) should be supported by the codec bitstream.¶
The codec specification shall define a buffer model such as hypothetical reference decoder (HRD).¶
Specifications providing integration with system and delivery layers should be developed.¶
Color sampling formats:¶
| Profile | Bit depths per color component | Color sampling formats |
|---|---|---|
| 1 | 8 and 10 | 4:0:0 and 4:2:0 |
| 2 | 8 and 10 | 4:0:0, 4:2:0 and 4:4:4 |
| 3 | 8, 10 and 12 | 4:0:0, 4:2:0, 4:2:2 and 4:4:4 |
Support of configurations with zero structural delay, also referred toas "low-delay" configurations.¶
Tools that enable parallel processing (e.g., slices, tiles, and wave-frontpropagation processing) at both encoder and decoder sides are highly desirablefor many applications.¶
Compression efficiency on noisy content, content with film grain, computergenerated content, and low resolution materials is desirable.¶
As shown inFigure 1, compression-performance testing isperformed in 3 overlapped ranges that encompass 10 different bitrate values:¶
Initially, for the codec selected as a reference one (e.g., HEVC or VP9), aset of 10 QP (quantization parameter) values should be specified as in[TESTING], and corresponding quality values should be calculated. InFigure 1, QP and quality values are denoted as QP0, QP1, QP2,..., QP8, QP9 andQ0, Q1, Q2,..., Q8, and Q9, respectively. To guarantee the overlaps of qualitylevels between the bitrate ranges of the reference and tested codecs, aquality-alignment procedure should be performed for each range's outermost(left- and rightmost) quality levels Qk of the reference codec (i.e., for Q0,Q3, Q6, and Q9) and the quality levels Q'k (i.e., Q'0, Q'3, Q'6, and Q'9) ofthe tested codec. Thus, these quality levels Q'k and, hence, the correspondingQP value QP'k (i.e., QP'0, QP'3, QP'6, and QP'9) of the tested codec should beselected using the following formulas:¶
Q'k = min { abs(Q'i - Qk) }, i in RQP'k = argmin { abs(Q'i(QP'i) - Qk(QPk)) }, i in R¶where R is the range of the QP indexes of the tested codec -- i.e., thecandidate Internet video codec. The inner quality levels (i.e., Q'1, Q'2, Q'4,Q'5, Q'7, and Q'8), as well as their corresponding QP values of each range(i.e., QP'1, QP'2, QP'4, QP'5, QP'7, and QP'8), should be as equidistantlyspaced as possible between the left- and rightmost quality levels withoutexplicitly mapping their values using the procedure described above.¶
QP'9 QP'8 QP'7 QP'6 QP'5 QP'4 QP'3 QP'2 QP'1 QP'0 <+----- ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ | Tested | | | | | | | | | | | codecQ'0 Q'1 Q'2 Q'3 Q'4 Q'5 Q'6 Q'7 Q'8 Q'9 <+----- ^ ^ ^ ^ | | | |Q0 Q1 Q2 Q3 Q4 Q5 Q6 Q7 Q8 Q9 <+----- ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ | Reference | | | | | | | | | | | codecQP9 QP8 QP7 QP6 QP5 QP4 QP3 QP2 QP1 QP0 <+-----+----------------+--------------+--------------+--------->^ ^ ^ ^ Bitrate|-------LBR------| |-----HBR------| ^ ^ |------MBR-----|
Since the QP mapping results may vary for different sequences, this qualityalignment procedure eventually needs to be performed separately for each qualityassessment index and each sequence used for codec-performance evaluation tofulfill the requirements described above.¶
To assess the quality of output (decoded) sequences, two indexes areseparately computed, PSNR[IMAGE] and MS-SSIM[IMAGE][MULTI-SCALE]. In the caseof the YCbCr color format, PSNR should be calculated for each color plane,whereas MS-SSIM is calculated for the luma channel only. In the case of the RGBcolor format, both metrics are computed for R, G, and B channels. Thus, foreach sequence, 30 RD-points for PSNR (i.e., three RD-curves, one for eachchannel) and 10 RD-points for MS-SSIM (i.e., one RD-curve, for luma channelonly) should be calculated in the case of YCbCr. If content is encoded as RGB,60 RD-points (30 for PSNR and 30 for MS-SSIM) should be calculated -- i.e., threeRD- curves (one for each channel) are computed for PSNR as well as threeRD-curves (one for each channel) for MS-SSIM.¶
Finally, to obtain an integral estimation, BD-rate savings[PSNR] should becomputed for each range and each quality index. In addition, average valuesover all 3 ranges should be provided for both PSNR and MS-SSIM. A list ofvideo sequences that should be used for testing are defined in[TESTING], as are the 10 QP valuesfor the reference codec. Testing processes should use theinformation on the codec applications presented in this document. As thereference for evaluation, state-of-the-art video codecs such as HEVC/H.265[HETEROGENEOUS][CODING] or VP9 must be used. The reference sourcecode of the HEVC/H.265 codeccan be found at[HEVC]. The HEVC/H.265 codec must be configuredaccording to[CONDITIONS]andTable 10.¶
| Intra-period, second | HEVC/H.265 encoding mode according to[CONDITIONS] |
|---|---|
| AI | Intra Main or Intra Main10 |
| RA | Random access Main or Random access Main10 |
| FIZD | Low delay Main or Low delay Main10 |
According to the coding efficiency requirement described in Section 3.1.1,BD-rate savings calculated for each color plane and averaged for all the videosequences used to test the NETVC codec should be, at least,¶
Since values of the two objective metrics (PSNR and MS-SSIM) are availablefor some color planes, each value should meet these coding efficiencyrequirements. That is, the final BD-rate saving denoted as S is calculated fora given color plane as follows:¶
S = min { S_psnr, S_ms-ssim },¶where S_psnr and S_ms-ssim are BD-rate savings calculated for the given color plane using PSNR and MS-SSIM metrics, respectively.¶
In addition to the objective quality measures defined above, subjectiveevaluation must also be performed for the final NETVC codec adoption. Forsubjective tests, the MOS-based evaluation procedure must be used as describedin Section 2.1 of[IMAGE]. For perception-oriented tools that primarily impact subjective quality, additional tests may also be individually assigned even for intermediate evaluation, subject to a decision of the NETVC WG.¶
This document itself does not address any security considerations. However,it is worth noting that a codec implementation (for both an encoder and adecoder) should take into consideration the worst-case computationalcomplexity, memory bandwidth, and physical memory size needed to process thepotentially untrusted input (e.g., the decoded pictures used as references).¶
This document has no IANA actions.¶
The authors would like to thank Mr. Paul Coverdale, Mr. VasilyRufitskiy, and Dr. Jianle Chen for many useful discussions on thisdocument and their help while preparing it, as well as Mr. Mo Zanaty,Dr. Minhua Zhou, Dr. Ali Begen, Mr. Thomas Daede, Mr. Adam Roach,Dr. Thomas Davies, Mr. Jonathan Lennox, Dr. Timothy Terriberry,Mr. Peter Thatcher, Dr. Jean-Marc Valin, Mr. Roman Danyliw, Mr. JackMoffitt, Mr. Greg Coppa, and Mr. Andrew Krupiczka for their valuablecomments on different revisions of this document.¶