What problem are we solving?¶

We would like to support pre-, and post-blending complex colortransformations in display controller hardware in order to allow forHW-supported HDR use-cases, as well as to provide support tocolor-managed applications, such as video or image editors.

It is possible to support an HDR output on HW supporting the Colorspaceand HDR Metadata drm_connector properties, but that requires thecompositor or application to render and compose the content into onefinal buffer intended for display. Doing so is costly.

Most modern display HW offers various 1D LUTs, 3D LUTs, matrices, and otheroperations to support color transformations. These operations are oftenimplemented in fixed-function HW and therefore much more power efficient thanperforming similar operations via shaders or CPU.

We would like to make use of this HW functionality to support complex colortransformations with no, or minimal CPU or shader load. The switch between HWfixed-function blocks and shaders/CPU must be seamless with no visibledifference when fallback to shaders/CPU is neceesary at any time.

How are other OSes solving this problem?¶

The most widely supported use-cases regard HDR content, whether video orgaming.

Most OSes will specify the source content format (color gamut, encoding transferfunction, and other metadata, such as max and average light levels) to a driver.Drivers will then program their fixed-function HW accordingly to map from asource content buffer’s space to a display’s space.

When fixed-function HW is not available the compositor will assemble a shader toask the GPU to perform the transformation from the source content format to thedisplay’s format.

A compositor’s mapping function and a driver’s mapping function are usuallyentirely separate concepts. On OSes where a HW vendor has no insight intoclosed-source compositor code such a vendor will tune their color managementcode to visually match the compositor’s. On other OSes, where both mappingfunctions are open to an implementer they will ensure both mappings match.

This results in mapping algorithm lock-in, meaning that no-one alone canexperiment with or introduce new mapping algorithms and achieveconsistent results regardless of which implementation path is taken.

Why is Linux different?¶

Unlike other OSes, where there is one compositor for one or more drivers, onLinux we have a many-to-many relationship. Many compositors; many drivers.In addition each compositor vendor or community has their own view of howcolor management should be done. This is what makes Linux so beautiful.

This means that a HW vendor can now no longer tune their driver to onecompositor, as tuning it to one could make it look fairly different fromanother compositor’s color mapping.

We need a better solution.

Descriptive API¶

An API that describes the source and destination colorspaces is a descriptiveAPI. It describes the input and output color spaces but does not describehow precisely they should be mapped. Such a mapping includes many minutedesign decision that can greatly affect the look of the final result.

It is not feasible to describe such mapping with enough detail to ensure thesame result from each implementation. In fact, these mappings are a very activeresearch area.

Prescriptive API¶

A prescriptive API describes not the source and destination colorspaces. Itinstead prescribes a recipe for how to manipulate pixel values to arrive at thedesired outcome.

This recipe is generally an ordered list of straight-forward operations,with clear mathematical definitions, such as 1D LUTs, 3D LUTs, matrices,or other operations that can be described in a precise manner.

The Color Pipeline API¶

HW color management pipelines can significantly differ between HWvendors in terms of availability, ordering, and capabilities of HWblocks. This makes a common definition of color management blocks andtheir ordering nigh impossible. Instead we are defining an API thatallows user space to discover the HW capabilities in a generic manner,agnostic of specific drivers and hardware.

drm_colorop Object¶

To support the definition of color pipelines we define the DRM coreobject type drm_colorop. Individual drm_colorop objects will be chainedvia the NEXT property of a drm_colorop to constitute a color pipeline.Each drm_colorop object is unique, i.e., even if multiple colorpipelines have the same operation they won’t share the same drm_coloropobject to describe that operation.

Note that drivers are not expected to map drm_colorop objects staticallyto specific HW blocks. The mapping of drm_colorop objects is entirely adriver-internal detail and can be as dynamic or static as a driver needsit to be. See more in the Driver Implementation Guide section below.

Each drm_colorop has three core properties:

TYPE: An enumeration property, defining the type of transformation, such as* enumerated curve* custom (uniform) 1D LUT* 3x3 matrix* 3x4 matrix* 3D LUT* etc.

Depending on the type of transformation other properties will describemore details.

BYPASS: A boolean property that can be used to easily put a block intobypass mode. The BYPASS property is not mandatory for a colorop, as longas the entire pipeline can get bypassed by setting the COLOR_PIPELINE ona plane to ‘0’.

NEXT: The ID of the next drm_colorop in a color pipeline, or 0 if thisdrm_colorop is the last in the chain.

An example of a drm_colorop object might look like one of these:

/* 1D enumerated curve */Color operation 42├─ "TYPE": immutable enum {1D enumerated curve, 1D LUT, 3x3 matrix, 3x4 matrix, 3D LUT, etc.} = 1D enumerated curve├─ "BYPASS": bool {true, false}├─ "CURVE_1D_TYPE": enum {sRGB EOTF, sRGB inverse EOTF, PQ EOTF, PQ inverse EOTF, …}└─ "NEXT": immutable color operation ID = 43/* custom 4k entry 1D LUT */Color operation 52├─ "TYPE": immutable enum {1D enumerated curve, 1D LUT, 3x3 matrix, 3x4 matrix, 3D LUT, etc.} = 1D LUT├─ "BYPASS": bool {true, false}├─ "SIZE": immutable range = 4096├─ "DATA": blob└─ "NEXT": immutable color operation ID = 0/* 17^3 3D LUT */Color operation 72├─ "TYPE": immutable enum {1D enumerated curve, 1D LUT, 3x3 matrix, 3x4 matrix, 3D LUT, etc.} = 3D LUT├─ "BYPASS": bool {true, false}├─ "SIZE": immutable range = 17├─ "DATA": blob└─ "NEXT": immutable color operation ID = 73

COLOR_PIPELINE Plane Property¶

Color Pipelines are created by a driver and advertised via a newCOLOR_PIPELINEenumproperty on each plane. Values of the propertyalways include object id 0, which is the default and means all colorprocessing is disabled. Additional values will be the object IDs of thefirst drm_colorop in a pipeline. A driver can create and advertise none,one, or more possible color pipelines. A DRM client will select a colorpipeline by setting the COLOR PIPELINE to the respective value.

NOTE: Many DRM clients will set enumeration properties via the stringvalue, often hard-coding it. Since this enumeration is generated basedon the colorop object IDs it is important to perform the Color PipelineDiscovery, described below, instead of hard-coding color pipelineassignment. Drivers might generate theenumstrings dynamically.Hard-coded strings might only work for specific drivers on a specificpieces of HW. Color Pipeline Discovery can work universally, as long asdrivers implement the required color operations.

The COLOR_PIPELINE property is only exposed when theDRM_CLIENT_CAP_PLANE_COLOR_PIPELINE is set. Drivers shall ignore anyexisting pre-blend color operations when this cap is set, such asCOLOR_RANGE and COLOR_ENCODING. If drivers want to support COLOR_RANGEor COLOR_ENCODING functionality when the color pipeline client cap isset, they are expected to expose colorops in the pipeline to allow forthe appropriate color transformation.

Setting of the COLOR_PIPELINE plane property or drm_colorop propertiesis only allowed for userspace that sets this client cap.

An example of a COLOR_PIPELINE property on a plane might look like this:

Plane 10├─ "TYPE": immutable enum {Overlay, Primary, Cursor} = Primary├─ …└─ "COLOR_PIPELINE": enum {0, 42, 52} = 0

Color Pipeline Discovery¶

A DRM client wanting color management on a drm_plane will:

1. Get the COLOR_PIPELINE property of the plane

2. iterate all COLOR_PIPELINEenumvalues

3. for eachenumvalue walk the color pipeline (via the NEXT pointers)and see if the available color operations are suitable for thedesired color management operations

If userspace encounters an unknown or unsuitable color operation duringdiscovery it does not need to reject the entire color pipeline outright,as long as the unknown or unsuitable colorop has a “BYPASS” property.Drivers will ensure that a bypassed block does not have any effect.

An example of chained properties to define an AMD pre-blending colorpipeline might look like this:

Plane 10├─ "TYPE" (immutable) = Primary└─ "COLOR_PIPELINE": enum {0, 44} = 0Color operation 44├─ "TYPE" (immutable) = 1D enumerated curve├─ "BYPASS": bool├─ "CURVE_1D_TYPE": enum {sRGB EOTF, PQ EOTF} = sRGB EOTF└─ "NEXT" (immutable) = 45Color operation 45├─ "TYPE" (immutable) = 3x4 Matrix├─ "BYPASS": bool├─ "DATA": blob└─ "NEXT" (immutable) = 46Color operation 46├─ "TYPE" (immutable) = 1D enumerated curve├─ "BYPASS": bool├─ "CURVE_1D_TYPE": enum {sRGB Inverse EOTF, PQ Inverse EOTF} = sRGB EOTF└─ "NEXT" (immutable) = 47Color operation 47├─ "TYPE" (immutable) = 1D LUT├─ "SIZE": immutable range = 4096├─ "DATA": blob└─ "NEXT" (immutable) = 48Color operation 48├─ "TYPE" (immutable) = 3D LUT├─ "DATA": blob└─ "NEXT" (immutable) = 49Color operation 49├─ "TYPE" (immutable) = 1D enumerated curve├─ "BYPASS": bool├─ "CURVE_1D_TYPE": enum {sRGB EOTF, PQ EOTF} = sRGB EOTF└─ "NEXT" (immutable) = 0

Color Pipeline Programming¶

Once a DRM client has found a suitable pipeline it will:

1. Set the COLOR_PIPELINEenumvalue to the one pointing at the firstdrm_colorop object of the desired pipeline

2. Set the properties for all drm_colorop objects in the pipeline to thedesired values, setting BYPASS to true for unused drm_colorop blocks,and false for enabled drm_colorop blocks

3. Perform (TEST_ONLY or not) atomic commit with all the other KMSstates it wishes to change

To configure the pipeline for an HDR10 PQ plane and blending in linearspace, a compositor might perform an atomic commit with the followingproperty values:

Plane 10└─ "COLOR_PIPELINE" = 42Color operation 42└─ "BYPASS" = trueColor operation 44└─ "BYPASS" = trueColor operation 45└─ "BYPASS" = trueColor operation 46└─ "BYPASS" = trueColor operation 47├─ "DATA" = Gamut mapping + tone mapping + night mode└─ "BYPASS" = falseColor operation 48├─ "CURVE_1D_TYPE" = PQ EOTF└─ "BYPASS" = false

Driver Implementer’s Guide¶

What does this all mean for driver implementations? As noted above thecolorops can map to HW directly but don’t need to do so. Here are somesuggestions on how to think about creating your color pipelines:

• Try to expose pipelines that use already defined colorops, even ifyour hardware pipeline is split differently. This allows existinguserspace to immediately take advantage of the hardware.

• Additionally, try to expose your actual hardware blocks as colorops.Define new colorop types where you believe it can offer significantbenefits if userspace learns to program them.

• Avoid defining new colorops for compound operations with very narrowscope. If you have a hardware block for a special operation thatcannot be split further, you can expose that as a new colorop type.However, try to not define colorops for “use cases”, especially ifthey require you to combine multiple hardware blocks.

• Design new colorops as prescriptive, not descriptive; by themathematical formula, not by the assumed input and output.

A defined colorop type must be deterministic. The exact behavior of thecolorop must be documented entirely, whether via a mathematical formulaor some other description. Its operation can depend only on itsproperties and input and nothing else, allowed error tolerancenotwithstanding.

Driver Forward/Backward Compatibility¶

As this is uAPI drivers can’t regress color pipelines that have beenintroduced for a given HW generation. New HW generations are free toabandon color pipelines advertised for previous generations.Nevertheless, it can be beneficial to carry support for existing colorpipelines forward as those will likely already have support in DRMclients.

Introducing new colorops to a pipeline is fine, as long as they can bebypassed or are purely informational. DRM clients implementing supportfor the pipeline can always skip unknown properties as long as they canbe confident that doing so will not cause unexpected results.

If a new colorop doesn’t fall into one of the above categories(bypassable or informational) the modified pipeline would be unusablefor user space. In this case a new pipeline should be defined.

References¶

1. https://lore.kernel.org/dri-devel/QMers3awXvNCQlyhWdTtsPwkp5ie9bze_hD5nAccFW7a_RXlWjYB7MoUW_8CKLT2bSQwIXVi5H6VULYIxCdgvryZoAoJnC5lZgyK1QWn488=@emersion.fr/