drm/i915 Intel GFX Driver¶
The drm/i915 driver supports all (with the exception of some very earlymodels) integrated GFX chipsets with both Intel display and renderingblocks. This excludes a set of SoC platforms with an SGX rendering unit,those have basic support through the gma500 drm driver.
Core Driver Infrastructure¶
This section covers core driver infrastructure used by both the displayand the GEM parts of the driver.
Runtime Power Management¶
The i915 driver supports dynamic enabling and disabling of entire hardwareblocks at runtime. This is especially important on the display side wheresoftware is supposed to control many power gates manually on recent hardware,since on the GT side a lot of the power management is done by the hardware.But even there some manual control at the device level is required.
Since i915 supports a diverse set of platforms with a unified codebase andhardware engineers just love to shuffle functionality around between powerdomains there’s a sizeable amount of indirection required. This file providesgeneric functions to the driver for grabbing and releasing references forabstract power domains. It then maps those to the actual power wellspresent for a given platform.
- intel_wakeref_t
intel_runtime_pm_get_raw(struct intel_runtime_pm * rpm)¶ grab a raw runtime pm reference
Parameters
structintel_runtime_pm*rpm- the intel_runtime_pm structure
Description
This is the unlocked version of intel_display_power_is_enabled() and shouldonly be used from error capture and recovery code where deadlocks arepossible.This function grabs a device-level runtime pm reference (mostly used forasynchronous PM management from display code) and ensures that it is poweredup. Raw references are not considered during wakelock assert checks.
Any runtime pm reference obtained by this function must have a symmetriccall tointel_runtime_pm_put_raw() to release the reference again.
Return
the wakeref cookie to pass tointel_runtime_pm_put_raw(), evaluatesas True if the wakeref was acquired, or False otherwise.
- intel_wakeref_t
intel_runtime_pm_get(struct intel_runtime_pm * rpm)¶ grab a runtime pm reference
Parameters
structintel_runtime_pm*rpm- the intel_runtime_pm structure
Description
This function grabs a device-level runtime pm reference (mostly used for GEMcode to ensure the GTT or GT is on) and ensures that it is powered up.
Any runtime pm reference obtained by this function must have a symmetriccall tointel_runtime_pm_put() to release the reference again.
Return
the wakeref cookie to pass tointel_runtime_pm_put()
- intel_wakeref_t
intel_runtime_pm_get_if_in_use(struct intel_runtime_pm * rpm)¶ grab a runtime pm reference if device in use
Parameters
structintel_runtime_pm*rpm- the intel_runtime_pm structure
Description
This function grabs a device-level runtime pm reference if the device isalready in use and ensures that it is powered up. It is illegal to tryand access the HW shouldintel_runtime_pm_get_if_in_use() report failure.
Any runtime pm reference obtained by this function must have a symmetriccall tointel_runtime_pm_put() to release the reference again.
Return
the wakeref cookie to pass tointel_runtime_pm_put(), evaluatesas True if the wakeref was acquired, or False otherwise.
- intel_wakeref_t
intel_runtime_pm_get_noresume(struct intel_runtime_pm * rpm)¶ grab a runtime pm reference
Parameters
structintel_runtime_pm*rpm- the intel_runtime_pm structure
Description
This function grabs a device-level runtime pm reference (mostly used for GEMcode to ensure the GTT or GT is on).
It will _not_ power up the device but instead only check that it’s poweredon. Therefore it is only valid to call this functions from contexts wherethe device is known to be powered up and where trying to power it up wouldresult in hilarity and deadlocks. That pretty much means only the systemsuspend/resume code where this is used to grab runtime pm references fordelayed setup down in work items.
Any runtime pm reference obtained by this function must have a symmetriccall tointel_runtime_pm_put() to release the reference again.
Return
the wakeref cookie to pass tointel_runtime_pm_put()
- void
intel_runtime_pm_put_raw(struct intel_runtime_pm * rpm, intel_wakeref_t wref)¶ release a raw runtime pm reference
Parameters
structintel_runtime_pm*rpm- the intel_runtime_pm structure
intel_wakeref_twref- wakeref acquired for the reference that is being released
Description
This function drops the device-level runtime pm reference obtained byintel_runtime_pm_get_raw() and might power down the correspondinghardware block right away if this is the last reference.
- void
intel_runtime_pm_put_unchecked(struct intel_runtime_pm * rpm)¶ release an unchecked runtime pm reference
Parameters
structintel_runtime_pm*rpm- the intel_runtime_pm structure
Description
This function drops the device-level runtime pm reference obtained byintel_runtime_pm_get() and might power down the correspondinghardware block right away if this is the last reference.
This function exists only for historical reasons and should be avoided innew code, as the correctness of its use cannot be checked. Always useintel_runtime_pm_put() instead.
- void
intel_runtime_pm_put(struct intel_runtime_pm * rpm, intel_wakeref_t wref)¶ release a runtime pm reference
Parameters
structintel_runtime_pm*rpm- the intel_runtime_pm structure
intel_wakeref_twref- wakeref acquired for the reference that is being released
Description
This function drops the device-level runtime pm reference obtained byintel_runtime_pm_get() and might power down the correspondinghardware block right away if this is the last reference.
- void
intel_runtime_pm_enable(struct intel_runtime_pm * rpm)¶ enable runtime pm
Parameters
structintel_runtime_pm*rpm- the intel_runtime_pm structure
Description
This function enables runtime pm at the end of the driver load sequence.
Note that this function does currently not enable runtime pm for thesubordinate display power domains. That is done byintel_power_domains_enable().
- void
intel_uncore_forcewake_get(struct intel_uncore * uncore, enum forcewake_domains fw_domains)¶ grab forcewake domain references
Parameters
structintel_uncore*uncore- the intel_uncore structure
enumforcewake_domainsfw_domains- forcewake domains to get reference on
Description
This function can be used get GT’s forcewake domain references.Normal register access will handle the forcewake domains automatically.However if some sequence requires the GT to not power down a particularforcewake domains this function should be called at the beginning of thesequence. And subsequently the reference should be dropped by symmetriccall to intel_unforce_forcewake_put(). Usually caller wants all the domainsto be kept awake so thefw_domains would be then FORCEWAKE_ALL.
- void
intel_uncore_forcewake_user_get(struct intel_uncore * uncore)¶ claim forcewake on behalf of userspace
Parameters
structintel_uncore*uncore- the intel_uncore structure
Description
This function is a wrapper aroundintel_uncore_forcewake_get() to acquirethe GT powerwell and in the process disable our debugging for theduration of userspace’s bypass.
- void
intel_uncore_forcewake_user_put(struct intel_uncore * uncore)¶ release forcewake on behalf of userspace
Parameters
structintel_uncore*uncore- the intel_uncore structure
Description
This function complementsintel_uncore_forcewake_user_get() and releasesthe GT powerwell taken on behalf of the userspace bypass.
- void
intel_uncore_forcewake_get__locked(struct intel_uncore * uncore, enum forcewake_domains fw_domains)¶ grab forcewake domain references
Parameters
structintel_uncore*uncore- the intel_uncore structure
enumforcewake_domainsfw_domains- forcewake domains to get reference on
Description
Seeintel_uncore_forcewake_get(). This variant places the onuson the caller to explicitly handle the dev_priv->uncore.lock spinlock.
- void
intel_uncore_forcewake_put(struct intel_uncore * uncore, enum forcewake_domains fw_domains)¶ release a forcewake domain reference
Parameters
structintel_uncore*uncore- the intel_uncore structure
enumforcewake_domainsfw_domains- forcewake domains to put references
Description
This function drops the device-level forcewakes for specifieddomains obtained byintel_uncore_forcewake_get().
- void
intel_uncore_forcewake_flush(struct intel_uncore * uncore, enum forcewake_domains fw_domains)¶ flush the delayed release
Parameters
structintel_uncore*uncore- the intel_uncore structure
enumforcewake_domainsfw_domains- forcewake domains to flush
- void
intel_uncore_forcewake_put__locked(struct intel_uncore * uncore, enum forcewake_domains fw_domains)¶ grab forcewake domain references
Parameters
structintel_uncore*uncore- the intel_uncore structure
enumforcewake_domainsfw_domains- forcewake domains to get reference on
Description
Seeintel_uncore_forcewake_put(). This variant places the onuson the caller to explicitly handle the dev_priv->uncore.lock spinlock.
- int
__intel_wait_for_register_fw(struct intel_uncore * uncore, i915_reg_t reg, u32 mask, u32 value, unsigned int fast_timeout_us, unsigned int slow_timeout_ms, u32 * out_value)¶ wait until register matches expected state
Parameters
structintel_uncore*uncore- the struct intel_uncore
i915_reg_treg- the register to read
u32mask- mask to apply to register value
u32value- expected value
unsignedintfast_timeout_us- fast timeout in microsecond for atomic/tight wait
unsignedintslow_timeout_ms- slow timeout in millisecond
u32*out_value- optional placeholder to hold registry value
Description
This routine waits until the target registerreg contains the expectedvalue after applying themask, i.e. it waits until
(I915_READ_FW(reg) & mask) == value
Otherwise, the wait will timeout afterslow_timeout_ms milliseconds.For atomic contextslow_timeout_ms must be zero andfast_timeout_usmust be not larger than 20,0000 microseconds.
Note that this routine assumes the caller holds forcewake asserted, it isnot suitable for very long waits. See intel_wait_for_register() if youwish to wait without holding forcewake for the duration (i.e. you expectthe wait to be slow).
Return
0 if the register matches the desired condition, or -ETIMEDOUT.
- int
__intel_wait_for_register(struct intel_uncore * uncore, i915_reg_t reg, u32 mask, u32 value, unsigned int fast_timeout_us, unsigned int slow_timeout_ms, u32 * out_value)¶ wait until register matches expected state
Parameters
structintel_uncore*uncore- the struct intel_uncore
i915_reg_treg- the register to read
u32mask- mask to apply to register value
u32value- expected value
unsignedintfast_timeout_us- fast timeout in microsecond for atomic/tight wait
unsignedintslow_timeout_ms- slow timeout in millisecond
u32*out_value- optional placeholder to hold registry value
Description
This routine waits until the target registerreg contains the expectedvalue after applying themask, i.e. it waits until
(I915_READ(reg) & mask) == value
Otherwise, the wait will timeout aftertimeout_ms milliseconds.
Return
0 if the register matches the desired condition, or -ETIMEDOUT.
- enum forcewake_domains
intel_uncore_forcewake_for_reg(struct intel_uncore * uncore, i915_reg_t reg, unsigned int op)¶ which forcewake domains are needed to access a register
Parameters
structintel_uncore*uncore- pointer to struct intel_uncore
i915_reg_treg- register in question
unsignedintop- operation bitmask of FW_REG_READ and/or FW_REG_WRITE
Description
Returns a set of forcewake domains required to be taken with for exampleintel_uncore_forcewake_get for the specified register to be accessible in thespecified mode (read, write or read/write) with raw mmio accessors.
NOTE
On Gen6 and Gen7 write forcewake domain (FORCEWAKE_RENDER) requires thecallers to do FIFO management on their own or risk losing writes.
Interrupt Handling¶
These functions provide the basic support for enabling and disabling theinterrupt handling support. There’s a lot more functionality in i915_irq.cand related files, but that will be described in separate chapters.
- void
intel_irq_init(struct drm_i915_private * dev_priv)¶ initializes irq support
Parameters
structdrm_i915_private*dev_priv- i915 device instance
Description
This function initializes all the irq support including work items, timersand all the vtables. It does not setup the interrupt itself though.
- void
intel_runtime_pm_disable_interrupts(struct drm_i915_private * dev_priv)¶ runtime interrupt disabling
Parameters
structdrm_i915_private*dev_priv- i915 device instance
Description
This function is used to disable interrupts at runtime, both in the runtimepm and the system suspend/resume code.
- void
intel_runtime_pm_enable_interrupts(struct drm_i915_private * dev_priv)¶ runtime interrupt enabling
Parameters
structdrm_i915_private*dev_priv- i915 device instance
Description
This function is used to enable interrupts at runtime, both in the runtimepm and the system suspend/resume code.
Intel GVT-g Guest Support(vGPU)¶
Intel GVT-g is a graphics virtualization technology which shares theGPU among multiple virtual machines on a time-sharing basis. Eachvirtual machine is presented a virtual GPU (vGPU), which has equivalentfeatures as the underlying physical GPU (pGPU), so i915 driver can runseamlessly in a virtual machine. This file provides vGPU specificoptimizations when running in a virtual machine, to reduce the complexityof vGPU emulation and to improve the overall performance.
A primary function introduced here is so-called “address space ballooning”technique. Intel GVT-g partitions global graphics memory among multiple VMs,so each VM can directly access a portion of the memory without hypervisor’sintervention, e.g. filling textures or queuing commands. However with thepartitioning an unmodified i915 driver would assume a smaller graphicsmemory starting from address ZERO, then requires vGPU emulation module totranslate the graphics address between ‘guest view’ and ‘host view’, forall registers and command opcodes which contain a graphics memory address.To reduce the complexity, Intel GVT-g introduces “address space ballooning”,by telling the exact partitioning knowledge to each guest i915 driver, whichthen reserves and prevents non-allocated portions from allocation. Thus vGPUemulation module only needs to scan and validate graphics addresses withoutcomplexity of address translation.
- void
intel_vgpu_detect(struct drm_i915_private * dev_priv)¶ detect virtual GPU
Parameters
structdrm_i915_private*dev_priv- i915 device private
Description
This function is called at the initialization stage, to detect whetherrunning on a vGPU.
- void
intel_vgt_deballoon(struct i915_ggtt * ggtt)¶ deballoon reserved graphics address trunks
Parameters
structi915_ggtt*ggtt- the global GGTT from which we reserved earlier
Description
This function is called to deallocate the ballooned-out graphic memory, whendriver is unloaded or when ballooning fails.
- int
intel_vgt_balloon(struct i915_ggtt * ggtt)¶ balloon out reserved graphics address trunks
Parameters
structi915_ggtt*ggtt- the global GGTT from which to reserve
Description
This function is called at the initialization stage, to balloon out thegraphic address space allocated to other vGPUs, by marking these spaces asreserved. The ballooning related knowledge(starting address and size ofthe mappable/unmappable graphic memory) is described in the vgt_if structurein a reserved mmio range.
To give an example, the drawing below depicts one typical scenario afterballooning. Here the vGPU1 has 2 pieces of graphic address spaces balloonedout each for the mappable and the non-mappable part. From the vGPU1 point ofview, the total size is the same as the physical one, with the start addressof its graphic space being zero. Yet there are some portions ballooned out(the shadow part, which are marked as reserved by drm allocator). From thehost point of view, the graphic address space is partitioned by multiplevGPUs in different VMs.
vGPU1 view Host view 0 ------> +-----------+ +-----------+ ^ |###########| | vGPU3 | | |###########| +-----------+ | |###########| | vGPU2 | | +-----------+ +-----------+ mappable GM | available | ==> | vGPU1 | | +-----------+ +-----------+ | |###########| | | v |###########| | Host | +=======+===========+ +===========+ ^ |###########| | vGPU3 | | |###########| +-----------+ | |###########| | vGPU2 | | +-----------+ +-----------+ unmappable GM | available | ==> | vGPU1 | | +-----------+ +-----------+ | |###########| | | | |###########| | Host | v |###########| | |total GM size ------> +-----------+ +-----------+
Return
zero on success, non-zero if configuration invalid or ballooning failed
Intel GVT-g Host Support(vGPU device model)¶
Intel GVT-g is a graphics virtualization technology which shares theGPU among multiple virtual machines on a time-sharing basis. Eachvirtual machine is presented a virtual GPU (vGPU), which has equivalentfeatures as the underlying physical GPU (pGPU), so i915 driver can runseamlessly in a virtual machine.
To virtualize GPU resources GVT-g driver depends on hypervisor technologye.g KVM/VFIO/mdev, Xen, etc. to provide resource access trapping capabilityand be virtualized within GVT-g device module. More architectural designdoc is available onhttps://01.org/group/2230/documentation-list.
- void
intel_gvt_sanitize_options(struct drm_i915_private * dev_priv)¶ sanitize GVT related options
Parameters
structdrm_i915_private*dev_priv- drm i915 private data
Description
This function is called at the i915 options sanitize stage.
- int
intel_gvt_init(struct drm_i915_private * dev_priv)¶ initialize GVT components
Parameters
structdrm_i915_private*dev_priv- drm i915 private data
Description
This function is called at the initialization stage to create a GVT device.
Return
Zero on success, negative error code if failed.
- void
intel_gvt_driver_remove(struct drm_i915_private * dev_priv)¶ cleanup GVT components when i915 driver is unbinding
Parameters
structdrm_i915_private*dev_priv- drm i915 private *
Description
This function is called at the i915 driver unloading stage, to shutdownGVT components and release the related resources.
Workarounds¶
This file is intended as a central place to implement most[1] of therequired workarounds for hardware to work as originally intended. They fallin five basic categories depending on how/when they are applied:
- Workarounds that touch registers that are saved/restored to/from the HWcontext image. The list is emitted (via Load Register Immediate commands)everytime a new context is created.
- GT workarounds. The list of these WAs is applied whenever these registersrevert to default values (on GPU reset, suspend/resume[2], etc..).
- Display workarounds. The list is applied during display clock-gatinginitialization.
- Workarounds that whitelist a privileged register, so that UMDs can managethem directly. This is just a special case of a MMMIO workaround (as wewrite the list of these to/be-whitelisted registers to some special HWregisters).
- Workaround batchbuffers, that get executed automatically by the hardwareon every HW context restore.
| [1] | Please notice that there are other WAs that, due to their nature,cannot be applied from a central place. Those are peppered around the restof the code, as needed. |
| [2] | Technically, some registers are powercontext saved & restored, so theysurvive a suspend/resume. In practice, writing them again is not toocostly and simplifies things. We can revisit this in the future. |
Layout¶
Keep things in this file ordered by WA type, as per the above (context, GT,display, register whitelist, batchbuffer). Then, inside each type, keep thefollowing order:
- Infrastructure functions and macros
- WAs per platform in standard gen/chrono order
- Public functions to init or apply the given workaround type.
Display Hardware Handling¶
This section covers everything related to the display hardware includingthe mode setting infrastructure, plane, sprite and cursor handling anddisplay, output probing and related topics.
Mode Setting Infrastructure¶
The i915 driver is thus far the only DRM driver which doesn’t use thecommon DRM helper code to implement mode setting sequences. Thus it hasits own tailor-made infrastructure for executing a display configurationchange.
Frontbuffer Tracking¶
Many features require us to track changes to the currently activefrontbuffer, especially rendering targeted at the frontbuffer.
To be able to do so we track frontbuffers using a bitmask for all possiblefrontbuffer slots throughintel_frontbuffer_track(). The functions in thisfile are then called when the contents of the frontbuffer are invalidated,when frontbuffer rendering has stopped again to flush out all the changesand when the frontbuffer is exchanged with a flip. Subsystems interested infrontbuffer changes (e.g. PSR, FBC, DRRS) should directly put their callbacksinto the relevant places and filter for the frontbuffer slots that they areinterested int.
On a high level there are two types of powersaving features. The first onework like a special cache (FBC and PSR) and are interested when they shouldstop caching and when to restart caching. This is done by placing callbacksinto the invalidate and the flush functions: At invalidate the caching mustbe stopped and at flush time it can be restarted. And maybe they need to knowwhen the frontbuffer changes (e.g. when the hw doesn’t initiate an invalidateand flush on its own) which can be achieved with placing callbacks into theflip functions.
The other type of display power saving feature only cares about busyness(e.g. DRRS). In that case all three (invalidate, flush and flip) indicatebusyness. There is no direct way to detect idleness. Instead an idle timerwork delayed work should be started from the flush and flip functions andcancelled as soon as busyness is detected.
- bool
intel_frontbuffer_invalidate(struct intel_frontbuffer * front, enum fb_op_origin origin)¶ invalidate frontbuffer object
Parameters
structintel_frontbuffer*front- GEM object to invalidate
enumfb_op_originorigin- which operation caused the invalidation
Description
This function gets called every time rendering on the given object starts andfrontbuffer caching (fbc, low refresh rate for DRRS, panel self refresh) mustbe invalidated. For ORIGIN_CS any subsequent invalidation will be delayeduntil the rendering completes or a flip on this frontbuffer plane isscheduled.
- void
intel_frontbuffer_flush(struct intel_frontbuffer * front, enum fb_op_origin origin)¶ flush frontbuffer object
Parameters
structintel_frontbuffer*front- GEM object to flush
enumfb_op_originorigin- which operation caused the flush
Description
This function gets called every time rendering on the given object hascompleted and frontbuffer caching can be started again.
- void
frontbuffer_flush(struct drm_i915_private * i915, unsigned int frontbuffer_bits, enum fb_op_origin origin)¶ flush frontbuffer
Parameters
structdrm_i915_private*i915- i915 device
unsignedintfrontbuffer_bits- frontbuffer plane tracking bits
enumfb_op_originorigin- which operation caused the flush
Description
This function gets called every time rendering on the given planes hascompleted and frontbuffer caching can be started again. Flushes will getdelayed if they’re blocked by some outstanding asynchronous rendering.
Can be called without any locks held.
- void
intel_frontbuffer_flip_prepare(struct drm_i915_private * i915, unsigned frontbuffer_bits)¶ prepare asynchronous frontbuffer flip
Parameters
structdrm_i915_private*i915- i915 device
unsignedfrontbuffer_bits- frontbuffer plane tracking bits
Description
This function gets called after scheduling a flip onobj. The actualfrontbuffer flushing will be delayed until completion is signalled withintel_frontbuffer_flip_complete. If an invalidate happens in between thisflush will be cancelled.
Can be called without any locks held.
- void
intel_frontbuffer_flip_complete(struct drm_i915_private * i915, unsigned frontbuffer_bits)¶ complete asynchronous frontbuffer flip
Parameters
structdrm_i915_private*i915- i915 device
unsignedfrontbuffer_bits- frontbuffer plane tracking bits
Description
This function gets called after the flip has been latched and will completeon the next vblank. It will execute the flush if it hasn’t been cancelled yet.
Can be called without any locks held.
- void
intel_frontbuffer_flip(struct drm_i915_private * i915, unsigned frontbuffer_bits)¶ synchronous frontbuffer flip
Parameters
structdrm_i915_private*i915- i915 device
unsignedfrontbuffer_bits- frontbuffer plane tracking bits
Description
This function gets called after scheduling a flip onobj. This is forsynchronous plane updates which will happen on the next vblank and which willnot get delayed by pending gpu rendering.
Can be called without any locks held.
- void
intel_frontbuffer_track(struct intel_frontbuffer * old, struct intel_frontbuffer * new, unsigned int frontbuffer_bits)¶ update frontbuffer tracking
Parameters
structintel_frontbuffer*old- current buffer for the frontbuffer slots
structintel_frontbuffer*new- new buffer for the frontbuffer slots
unsignedintfrontbuffer_bits- bitmask of frontbuffer slots
Description
This updates the frontbuffer tracking bitsfrontbuffer_bits by clearing themfromold and setting them innew. Bothold andnew can be NULL.
Display FIFO Underrun Reporting¶
The i915 driver checks for display fifo underruns using the interrupt signalsprovided by the hardware. This is enabled by default and fairly useful todebug display issues, especially watermark settings.
If an underrun is detected this is logged into dmesg. To avoid flooding logsand occupying the cpu underrun interrupts are disabled after the firstoccurrence until the next modeset on a given pipe.
Note that underrun detection on gmch platforms is a bit more ugly since thereis no interrupt (despite that the signalling bit is in the PIPESTAT pipeinterrupt register). Also on some other platforms underrun interrupts areshared, which means that if we detect an underrun we need to disable underrunreporting on all pipes.
The code also supports underrun detection on the PCH transcoder.
- bool
intel_set_cpu_fifo_underrun_reporting(struct drm_i915_private * dev_priv, enum pipe pipe, bool enable)¶ set cpu fifo underrrun reporting state
Parameters
structdrm_i915_private*dev_priv- i915 device instance
enumpipepipe- (CPU) pipe to set state for
boolenable- whether underruns should be reported or not
Description
This function sets the fifo underrun state forpipe. It is used in themodeset code to avoid false positives since on many platforms underruns areexpected when disabling or enabling the pipe.
Notice that on some platforms disabling underrun reports for one pipedisables for all due to shared interrupts. Actual reporting is still per-pipethough.
Returns the previous state of underrun reporting.
- bool
intel_set_pch_fifo_underrun_reporting(struct drm_i915_private * dev_priv, enum pipe pch_transcoder, bool enable)¶ set PCH fifo underrun reporting state
Parameters
structdrm_i915_private*dev_priv- i915 device instance
enumpipepch_transcoder- the PCH transcoder (same as pipe on IVB and older)
boolenable- whether underruns should be reported or not
Description
This function makes us disable or enable PCH fifo underruns for a specificPCH transcoder. Notice that on some PCHs (e.g. CPT/PPT), disabling FIFOunderrun reporting for one transcoder may also disable all the other PCHerror interruts for the other transcoders, due to the fact that there’s justone interrupt mask/enable bit for all the transcoders.
Returns the previous state of underrun reporting.
- void
intel_cpu_fifo_underrun_irq_handler(struct drm_i915_private * dev_priv, enum pipe pipe)¶ handle CPU fifo underrun interrupt
Parameters
structdrm_i915_private*dev_priv- i915 device instance
enumpipepipe- (CPU) pipe to set state for
Description
This handles a CPU fifo underrun interrupt, generating an underrun warninginto dmesg if underrun reporting is enabled and then disables the underruninterrupt to avoid an irq storm.
- void
intel_pch_fifo_underrun_irq_handler(struct drm_i915_private * dev_priv, enum pipe pch_transcoder)¶ handle PCH fifo underrun interrupt
Parameters
structdrm_i915_private*dev_priv- i915 device instance
enumpipepch_transcoder- the PCH transcoder (same as pipe on IVB and older)
Description
This handles a PCH fifo underrun interrupt, generating an underrun warninginto dmesg if underrun reporting is enabled and then disables the underruninterrupt to avoid an irq storm.
- void
intel_check_cpu_fifo_underruns(struct drm_i915_private * dev_priv)¶ check for CPU fifo underruns immediately
Parameters
structdrm_i915_private*dev_priv- i915 device instance
Description
Check for CPU fifo underruns immediately. Useful on IVB/HSW where the sharederror interrupt may have been disabled, and so CPU fifo underruns won’tnecessarily raise an interrupt, and on GMCH platforms where underruns neverraise an interrupt.
- void
intel_check_pch_fifo_underruns(struct drm_i915_private * dev_priv)¶ check for PCH fifo underruns immediately
Parameters
structdrm_i915_private*dev_priv- i915 device instance
Description
Check for PCH fifo underruns immediately. Useful on CPT/PPT where the sharederror interrupt may have been disabled, and so PCH fifo underruns won’tnecessarily raise an interrupt.
Plane Configuration¶
This section covers plane configuration and composition with the primaryplane, sprites, cursors and overlays. This includes the infrastructureto do atomic vsync’ed updates of all this state and also tightly coupledtopics like watermark setup and computation, framebuffer compression andpanel self refresh.
Atomic Plane Helpers¶
The functions here are used by the atomic plane helper functions toimplement legacy plane updates (i.e., drm_plane->update_plane() anddrm_plane->disable_plane()). This allows plane updates to use theatomic state infrastructure and perform plane updates as separateprepare/check/commit/cleanup steps.
- structdrm_plane_state *
intel_plane_duplicate_state(structdrm_plane * plane)¶ duplicate plane state
Parameters
structdrm_plane*plane- drm plane
Description
Allocates and returns a copy of the plane state (both common andIntel-specific) for the specified plane.
Return
The newly allocated plane state, or NULL on failure.
- void
intel_plane_destroy_state(structdrm_plane * plane, structdrm_plane_state * state)¶ destroy plane state
Parameters
structdrm_plane*plane- drm plane
structdrm_plane_state*state- state object to destroy
Description
Destroys the plane state (both common and Intel-specific) for thespecified plane.
Output Probing¶
This section covers output probing and related infrastructure like thehotplug interrupt storm detection and mitigation code. Note that thei915 driver still uses most of the common DRM helper code for outputprobing, so those sections fully apply.
Hotplug¶
Simply put, hotplug occurs when a display is connected to or disconnectedfrom the system. However, there may be adapters and docking stations andDisplay Port short pulses and MST devices involved, complicating matters.
Hotplug in i915 is handled in many different levels of abstraction.
The platform dependent interrupt handling code in i915_irq.c enables,disables, and does preliminary handling of the interrupts. The interrupthandlers gather the hotplug detect (HPD) information from relevant registersinto a platform independent mask of hotplug pins that have fired.
The platform independent interrupt handlerintel_hpd_irq_handler() inintel_hotplug.c does hotplug irq storm detection and mitigation, and passesfurther processing to appropriate bottom halves (Display Port specific andregular hotplug).
The Display Port work function i915_digport_work_func() calls intointel_dp_hpd_pulse() via hooks, which handles DP short pulses and DP MST longpulses, with failures and non-MST long pulses triggering regular hotplugprocessing on the connector.
The regular hotplug work function i915_hotplug_work_func() calls connectordetect hooks, and, if connector status changes, triggers sending of hotpluguevent to userspace viadrm_kms_helper_hotplug_event().
Finally, the userspace is responsible for triggering a modeset upon receivingthe hotplug uevent, disabling or enabling the crtc as needed.
The hotplug interrupt storm detection and mitigation code keeps track of thenumber of interrupts per hotplug pin per a period of time, and if the numberof interrupts exceeds a certain threshold, the interrupt is disabled for awhile before being re-enabled. The intention is to mitigate issues raisingfrom broken hardware triggering massive amounts of interrupts and grindingthe system to a halt.
Current implementation expects that hotplug interrupt storm will not beseen when display port sink is connected, hence on platforms whose DPcallback is handled by i915_digport_work_func reenabling of hpd is notperformed (it was never expected to be disabled in the first place ;) )this is specific to DP sinks handled by this routine and any other displaysuch as HDMI or DVI enabled on the same port will have proper logic sinceit will use i915_hotplug_work_func where this logic is handled.
- enum hpd_pin
intel_hpd_pin_default(struct drm_i915_private * dev_priv, enum port port)¶ return default pin associated with certain port.
Parameters
structdrm_i915_private*dev_priv- private driver data pointer
enumportport- the hpd port to get associated pin
Description
It is only valid and used by digital port encoder.
Return pin that is associatade withport and HDP_NONE if no pin ishard associated with thatport.
- bool
intel_hpd_irq_storm_detect(struct drm_i915_private * dev_priv, enum hpd_pin pin, bool long_hpd)¶ gather stats and detect HPD IRQ storm on a pin
Parameters
structdrm_i915_private*dev_priv- private driver data pointer
enumhpd_pinpin- the pin to gather stats on
boollong_hpd- whether the HPD IRQ was long or short
Description
Gather stats about HPD IRQs from the specifiedpin, and detect IRQstorms. Only the pin specific stats and state are changed, the caller isresponsible for further action.
The number of IRQs that are allowed withinHPD_STORM_DETECT_PERIOD isstored indev_priv->hotplug.hpd_storm_threshold which defaults toHPD_STORM_DEFAULT_THRESHOLD. Long IRQs count as +10 to this threshold, andshort IRQs count as +1. If this threshold is exceeded, it’s considered anIRQ storm and the IRQ state is set toHPD_MARK_DISABLED.
By default, most systems will only count long IRQs towardsdev_priv->hotplug.hpd_storm_threshold. However, some older systems alsosuffer from short IRQ storms and must also track these. Because short IRQstorms are naturally caused by sideband interactions with DP MST devices,short IRQ detection is only enabled for systems without DP MST support.Systems which are new enough to support DP MST are far less likely tosuffer from IRQ storms at all, so this is fine.
The HPD threshold can be controlled through i915_hpd_storm_ctl in debugfs,and should only be adjusted for automated hotplug testing.
Return true if an IRQ storm was detected onpin.
- void
intel_hpd_trigger_irq(struct intel_digital_port * dig_port)¶ trigger an hpd irq event for a port
Parameters
structintel_digital_port*dig_port- digital port
Description
Trigger an HPD interrupt event for the given port, emulating a short pulsegenerated by the sink, and schedule the dig port work to handle it.
- void
intel_hpd_irq_handler(struct drm_i915_private * dev_priv, u32 pin_mask, u32 long_mask)¶ main hotplug irq handler
Parameters
structdrm_i915_private*dev_priv- drm_i915_private
u32pin_mask- a mask of hpd pins that have triggered the irq
u32long_mask- a mask of hpd pins that may be long hpd pulses
Description
This is the main hotplug irq handler for all platforms. The platform specificirq handlers call the platform specific hotplug irq handlers, which read anddecode the appropriate registers into bitmasks about hpd pins that havetriggered (pin_mask), and which of those pins may be long pulses(long_mask). Thelong_mask is ignored if the port corresponding to the pinis not a digital port.
Here, we do hotplug irq storm detection and mitigation, and pass furtherprocessing to appropriate bottom halves.
- void
intel_hpd_init(struct drm_i915_private * dev_priv)¶ initializes and enables hpd support
Parameters
structdrm_i915_private*dev_priv- i915 device instance
Description
This function enables the hotplug support. It requires that interrupts havealready been enabled with intel_irq_init_hw(). From this point on hotplug andpoll request can run concurrently to other code, so locking rules must beobeyed.
This is a separate step from interrupt enabling to simplify the locking rulesin the driver load and resume code.
Also see:intel_hpd_poll_init(), which enables connector polling
- void
intel_hpd_poll_init(struct drm_i915_private * dev_priv)¶ enables/disables polling for connectors with hpd
Parameters
structdrm_i915_private*dev_priv- i915 device instance
Description
This function enables polling for all connectors, regardless of whether ornot they support hotplug detection. Under certain conditions HPD may not befunctional. On most Intel GPUs, this happens when we enter runtime suspend.On Valleyview and Cherryview systems, this also happens when we shut off allof the powerwells.
Since this function can get called in contexts where we’re already holdingdev->mode_config.mutex, we do the actual hotplug enabling in a seperateworker.
Also see:intel_hpd_init(), which restores hpd handling.
High Definition Audio¶
The graphics and audio drivers together support High Definition Audio overHDMI and Display Port. The audio programming sequences are divided into audiocodec and controller enable and disable sequences. The graphics driverhandles the audio codec sequences, while the audio driver handles the audiocontroller sequences.
The disable sequences must be performed before disabling the transcoder orport. The enable sequences may only be performed after enabling thetranscoder and port, and after completed link training. Therefore the audioenable/disable sequences are part of the modeset sequence.
The codec and controller sequences could be done either parallel or serial,but generally the ELDV/PD change in the codec sequence indicates to the audiodriver that the controller sequence should start. Indeed, most of theco-operation between the graphics and audio drivers is handled via audiorelated registers. (The notable exception is the power management, notcovered here.)
The structi915_audio_component is used to interact between the graphicsand audio drivers. The structi915_audio_component_opsops in it isdefined in graphics driver and called in audio driver. Thestructi915_audio_component_audio_opsaudio_ops is called from i915 driver.
- void
intel_audio_codec_enable(struct intel_encoder * encoder, const struct intel_crtc_state * crtc_state, const structdrm_connector_state * conn_state)¶ Enable the audio codec for HD audio
Parameters
structintel_encoder*encoder- encoder on which to enable audio
conststructintel_crtc_state*crtc_state- pointer to the current crtc state.
conststructdrm_connector_state*conn_state- pointer to the current connector state.
Description
The enable sequences may only be performed after enabling the transcoder andport, and after completed link training.
- void
intel_audio_codec_disable(struct intel_encoder * encoder, const struct intel_crtc_state * old_crtc_state, const structdrm_connector_state * old_conn_state)¶ Disable the audio codec for HD audio
Parameters
structintel_encoder*encoder- encoder on which to disable audio
conststructintel_crtc_state*old_crtc_state- pointer to the old crtc state.
conststructdrm_connector_state*old_conn_state- pointer to the old connector state.
Description
The disable sequences must be performed before disabling the transcoder orport.
- void
intel_init_audio_hooks(struct drm_i915_private * dev_priv)¶ Set up chip specific audio hooks
Parameters
structdrm_i915_private*dev_priv- device private
- void
i915_audio_component_init(struct drm_i915_private * dev_priv)¶ initialize and register the audio component
Parameters
structdrm_i915_private*dev_priv- i915 device instance
Description
This will register with the component framework a child component whichwill bind dynamically to the snd_hda_intel driver’s corresponding mastercomponent when the latter is registered. During binding the childinitializes an instance of struct i915_audio_component which it receivesfrom the master. The master can then start to use the interface defined bythis struct. Each side can break the binding at any point by deregisteringits own component after which each side’s component unbind callback iscalled.
We ignore any error during registration and continue with reducedfunctionality (i.e. without HDMI audio).
- void
i915_audio_component_cleanup(struct drm_i915_private * dev_priv)¶ deregister the audio component
Parameters
structdrm_i915_private*dev_priv- i915 device instance
Description
Deregisters the audio component, breaking any existing binding to thecorresponding snd_hda_intel driver’s master component.
- void
intel_audio_init(struct drm_i915_private * dev_priv)¶ Initialize the audio driver either using component framework or using lpe audio bridge
Parameters
structdrm_i915_private*dev_priv- the i915 drm device private data
- void
intel_audio_deinit(struct drm_i915_private * dev_priv)¶ deinitialize the audio driver
Parameters
structdrm_i915_private*dev_priv- the i915 drm device private data
- struct
i915_audio_component¶ Used for direct communication between i915 and hda drivers
Definition
struct i915_audio_component { struct drm_audio_component base; int aud_sample_rate[MAX_PORTS];};Members
base- the drm_audio_component base class
aud_sample_rate- the array of audio sample rate per port
Intel HDMI LPE Audio Support¶
Motivation:Atom platforms (e.g. valleyview and cherryTrail) integrates a DMA-basedinterface as an alternative to the traditional HDaudio path. While thismode is unrelated to the LPE aka SST audio engine, the documentation refersto this mode as LPE so we keep this notation for the sake of consistency.
The interface is handled by a separate standalone driver maintained in theALSA subsystem for simplicity. To minimize the interaction between the twosubsystems, a bridge is setup between the hdmi-lpe-audio and i915:1. Create a platform device to share MMIO/IRQ resources2. Make the platform device child of i915 device for runtime PM.3. Create IRQ chip to forward the LPE audio irqs.the hdmi-lpe-audio driver probes the lpe audio device and creates a newsound card
Threats:Due to the restriction in Linux platform device model, user need manuallyuninstall the hdmi-lpe-audio driver before uninstalling i915 module,otherwise we might run into use-after-free issues after i915 removes theplatform device: even though hdmi-lpe-audio driver is released, the modulesis still in “installed” status.
Implementation:The MMIO/REG platform resources are created according to the registersspecification.When forwarding LPE audio irqs, the flow control handler selection dependson the platform, for example on valleyview handle_simple_irq is enough.
- void
intel_lpe_audio_irq_handler(struct drm_i915_private * dev_priv)¶ forwards the LPE audio irq
Parameters
structdrm_i915_private*dev_priv- the i915 drm device private data
Description
the LPE Audio irq is forwarded to the irq handler registered by LPE audiodriver.
- int
intel_lpe_audio_init(struct drm_i915_private * dev_priv)¶ detect and setup the bridge between HDMI LPE Audio driver and i915
Parameters
structdrm_i915_private*dev_priv- the i915 drm device private data
Return
0 if successful. non-zero if detection orllocation/initialization fails
- void
intel_lpe_audio_teardown(struct drm_i915_private * dev_priv)¶ destroy the bridge between HDMI LPE audio driver and i915
Parameters
structdrm_i915_private*dev_priv- the i915 drm device private data
Description
release all the resources for LPE audio <-> i915 bridge.
- void
intel_lpe_audio_notify(struct drm_i915_private * dev_priv, enum pipe pipe, enum port port, const void * eld, int ls_clock, bool dp_output)¶ notify lpe audio event audio driver and i915
Parameters
structdrm_i915_private*dev_priv- the i915 drm device private data
enumpipepipe- pipe
enumportport- port
constvoid*eld- ELD data
intls_clock- Link symbol clock in kHz
booldp_output- Driving a DP output?
Description
Notify lpe audio driver of eld change.
Panel Self Refresh PSR (PSR/SRD)¶
Since Haswell Display controller supports Panel Self-Refresh on displaypanels witch have a remote frame buffer (RFB) implemented according to PSRspec in eDP1.3. PSR feature allows the display to go to lower standby stateswhen system is idle but display is on as it eliminates display refreshrequest to DDR memory completely as long as the frame buffer for thatdisplay is unchanged.
Panel Self Refresh must be supported by both Hardware (source) andPanel (sink).
PSR saves power by caching the framebuffer in the panel RFB, which allows usto power down the link and memory controller. For DSI panels the same ideais called “manual mode”.
The implementation uses the hardware-based PSR support which automaticallyenters/exits self-refresh mode. The hardware takes care of sending therequired DP aux message and could even retrain the link (that part isn’tenabled yet though). The hardware also keeps track of any frontbufferchanges to know when to exit self-refresh mode again. Unfortunately thatpart doesn’t work too well, hence why the i915 PSR support uses thesoftware frontbuffer tracking to make sure it doesn’t miss a screenupdate. For this integrationintel_psr_invalidate() andintel_psr_flush()get called by the frontbuffer tracking code. Note that because of lockingissues the self-refresh re-enable code is done from a work queue, whichmust be correctly synchronized/cancelled when shutting down the pipe.”
DC3CO (DC3 clock off)
On top of PSR2, GEN12 adds a intermediate power savings state that turnsclock off automatically during PSR2 idle state.The smaller overhead of DC3co entry/exit vs. the overhead of PSR2 deep sleepentry/exit allows the HW to enter a low-power state even when page flippingperiodically (for instance a 30fps video playback scenario).
Every time a flips occurs PSR2 will get out of deep sleep state(if it was),so DC3CO is enabled and tgl_dc3co_disable_work is schedule to run after 6frames, if no other flip occurs and the function above is executed, DC3CO isdisabled and PSR2 is configured to enter deep sleep, resetting again in caseof another flip.Front buffer modifications do not trigger DC3CO activation on purpose as itwould bring a lot of complexity and most of the moderns systems will onlyuse page flips.
- void
intel_psr_enable(struct intel_dp * intel_dp, const struct intel_crtc_state * crtc_state, const structdrm_connector_state * conn_state)¶ Enable PSR
Parameters
structintel_dp*intel_dp- Intel DP
conststructintel_crtc_state*crtc_state- new CRTC state
conststructdrm_connector_state*conn_state- new CONNECTOR state
Description
This function can only be called after the pipe is fully trained and enabled.
- void
intel_psr_disable(struct intel_dp * intel_dp, const struct intel_crtc_state * old_crtc_state)¶ Disable PSR
Parameters
structintel_dp*intel_dp- Intel DP
conststructintel_crtc_state*old_crtc_state- old CRTC state
Description
This function needs to be called before disabling pipe.
- void
intel_psr_update(struct intel_dp * intel_dp, const struct intel_crtc_state * crtc_state, const structdrm_connector_state * conn_state)¶ Update PSR state
Parameters
structintel_dp*intel_dp- Intel DP
conststructintel_crtc_state*crtc_state- new CRTC state
conststructdrm_connector_state*conn_state- new CONNECTOR state
Description
This functions will update PSR states, disabling, enabling or switching PSRversion when executing fastsets. For full modeset,intel_psr_disable() andintel_psr_enable() should be called instead.
- int
intel_psr_wait_for_idle(const struct intel_crtc_state * new_crtc_state, u32 * out_value)¶ wait for PSR1 to idle
Parameters
conststructintel_crtc_state*new_crtc_state- new CRTC state
u32*out_value- PSR status in case of failure
Description
This function is expected to be called from pipe_update_start() where it isnot expected to race with PSR enable or disable.
Return
0 on success or -ETIMEOUT if PSR status does not idle.
- void
intel_psr_invalidate(struct drm_i915_private * dev_priv, unsigned frontbuffer_bits, enum fb_op_origin origin)¶ Invalidade PSR
Parameters
structdrm_i915_private*dev_priv- i915 device
unsignedfrontbuffer_bits- frontbuffer plane tracking bits
enumfb_op_originorigin- which operation caused the invalidate
Description
Since the hardware frontbuffer tracking has gaps we need to integratewith the software frontbuffer tracking. This function gets called everytime frontbuffer rendering starts and a buffer gets dirtied. PSR must bedisabled if the frontbuffer mask contains a buffer relevant to PSR.
Dirty frontbuffers relevant to PSR are tracked in busy_frontbuffer_bits.”
- void
intel_psr_flush(struct drm_i915_private * dev_priv, unsigned frontbuffer_bits, enum fb_op_origin origin)¶ Flush PSR
Parameters
structdrm_i915_private*dev_priv- i915 device
unsignedfrontbuffer_bits- frontbuffer plane tracking bits
enumfb_op_originorigin- which operation caused the flush
Description
Since the hardware frontbuffer tracking has gaps we need to integratewith the software frontbuffer tracking. This function gets called everytime frontbuffer rendering has completed and flushed out to memory. PSRcan be enabled again if no other frontbuffer relevant to PSR is dirty.
Dirty frontbuffers relevant to PSR are tracked in busy_frontbuffer_bits.
- void
intel_psr_init(struct drm_i915_private * dev_priv)¶ Init basic PSR work and mutex.
Parameters
structdrm_i915_private*dev_priv- i915 device private
Description
This function is called only once at driver load to initialize basicPSR stuff.
Frame Buffer Compression (FBC)¶
FBC tries to save memory bandwidth (and so power consumption) bycompressing the amount of memory used by the display. It is totaltransparent to user space and completely handled in the kernel.
The benefits of FBC are mostly visible with solid backgrounds andvariation-less patterns. It comes from keeping the memory footprint smalland having fewer memory pages opened and accessed for refreshing the display.
i915 is responsible to reserve stolen memory for FBC and configure itsoffset on proper registers. The hardware takes care of allcompress/decompress. However there are many known cases where we have toforcibly disable it to allow proper screen updates.
- bool
intel_fbc_is_active(struct drm_i915_private * dev_priv)¶ Is FBC active?
Parameters
structdrm_i915_private*dev_priv- i915 device instance
Description
This function is used to verify the current state of FBC.
FIXME: This should be tracked in the plane config eventuallyinstead of queried at runtime for most callers.
- void
__intel_fbc_disable(struct drm_i915_private * dev_priv)¶ disable FBC
Parameters
structdrm_i915_private*dev_priv- i915 device instance
Description
This is the low level function that actually disables FBC. Callers shouldgrab the FBC lock.
- void
intel_fbc_choose_crtc(struct drm_i915_private * dev_priv, struct intel_atomic_state * state)¶ select a CRTC to enable FBC on
Parameters
structdrm_i915_private*dev_priv- i915 device instance
structintel_atomic_state*state- the atomic state structure
Description
This function looks at the proposed state for CRTCs and planes, then chooseswhich pipe is going to have FBC by setting intel_crtc_state->enable_fbc totrue.
Later, intel_fbc_enable is going to look for state->enable_fbc and then maybeenable FBC for the chosen CRTC. If it does, it will set dev_priv->fbc.crtc.
- void
intel_fbc_enable(struct intel_atomic_state * state, struct intel_crtc * crtc)¶
Parameters
structintel_atomic_state*state- corresponding
drm_crtc_stateforcrtc structintel_crtc*crtc- the CRTC
Description
This function checks if the given CRTC was chosen for FBC, then enables it ifpossible. Notice that it doesn’t activate FBC. It is valid to callintel_fbc_enable multiple times for the same pipe without anintel_fbc_disable in the middle, as long as it is deactivated.
- void
intel_fbc_disable(struct intel_crtc * crtc)¶ disable FBC if it’s associated with crtc
Parameters
structintel_crtc*crtc- the CRTC
Description
This function disables FBC if it’s associated with the provided CRTC.
- void
intel_fbc_global_disable(struct drm_i915_private * dev_priv)¶ globally disable FBC
Parameters
structdrm_i915_private*dev_priv- i915 device instance
Description
This function disables FBC regardless of which CRTC is associated with it.
- void
intel_fbc_handle_fifo_underrun_irq(struct drm_i915_private * dev_priv)¶ disable FBC when we get a FIFO underrun
Parameters
structdrm_i915_private*dev_priv- i915 device instance
Description
Without FBC, most underruns are harmless and don’t really cause too manyproblems, except for an annoying message on dmesg. With FBC, underruns canbecome black screens or even worse, especially when paired with badwatermarks. So in order for us to be on the safe side, completely disable FBCin case we ever detect a FIFO underrun on any pipe. An underrun on any pipealready suggests that watermarks may be bad, so try to be as safe aspossible.
This function is called from the IRQ handler.
- void
intel_fbc_init(struct drm_i915_private * dev_priv)¶ Initialize FBC
Parameters
structdrm_i915_private*dev_priv- the i915 device
Description
This function might be called during PM init process.
Display Refresh Rate Switching (DRRS)¶
Display Refresh Rate Switching (DRRS) is a power conservation featurewhich enables swtching between low and high refresh rates,dynamically, based on the usage scenario. This feature is applicablefor internal panels.
Indication that the panel supports DRRS is given by the panel EDID, whichwould list multiple refresh rates for one resolution.
DRRS is of 2 types - static and seamless.Static DRRS involves changing refresh rate (RR) by doing a full modeset(may appear as a blink on screen) and is used in dock-undock scenario.Seamless DRRS involves changing RR without any visual effect to the userand can be used during normal system usage. This is done by programmingcertain registers.
Support for static/seamless DRRS may be indicated in the VBT based oninputs from the panel spec.
DRRS saves power by switching to low RR based on usage scenarios.
The implementation is based on frontbuffer tracking implementation. Whenthere is a disturbance on the screen triggered by user activity or a periodicsystem activity, DRRS is disabled (RR is changed to high RR). When there isno movement on screen, after a timeout of 1 second, a switch to low RR ismade.
For integration with frontbuffer tracking code,intel_edp_drrs_invalidate()andintel_edp_drrs_flush() are called.
DRRS can be further extended to support other internal panels and alsothe scenario of video playback wherein RR is set based on the raterequested by userspace.
- void
intel_dp_set_drrs_state(struct drm_i915_private * dev_priv, const struct intel_crtc_state * crtc_state, int refresh_rate)¶ program registers for RR switch to take effect
Parameters
structdrm_i915_private*dev_priv- i915 device
conststructintel_crtc_state*crtc_state- a pointer to the active intel_crtc_state
intrefresh_rate- RR to be programmed
Description
This function gets called when refresh rate (RR) has to be changed fromone frequency to another. Switches can be between high and low RRsupported by the panel or to any other RR based on media playback (inthis case, RR value needs to be passed from user space).
The caller of this function needs to take a lock on dev_priv->drrs.
- void
intel_edp_drrs_enable(struct intel_dp * intel_dp, const struct intel_crtc_state * crtc_state)¶ init drrs struct if supported
Parameters
structintel_dp*intel_dp- DP struct
conststructintel_crtc_state*crtc_state- A pointer to the active crtc state.
Description
Initializes frontbuffer_bits and drrs.dp
- void
intel_edp_drrs_disable(struct intel_dp * intel_dp, const struct intel_crtc_state * old_crtc_state)¶ Disable DRRS
Parameters
structintel_dp*intel_dp- DP struct
conststructintel_crtc_state*old_crtc_state- Pointer to old crtc_state.
- void
intel_edp_drrs_invalidate(struct drm_i915_private * dev_priv, unsigned int frontbuffer_bits)¶ Disable Idleness DRRS
Parameters
structdrm_i915_private*dev_priv- i915 device
unsignedintfrontbuffer_bits- frontbuffer plane tracking bits
Description
This function gets called everytime rendering on the given planes start.Hence DRRS needs to be Upclocked, i.e. (LOW_RR -> HIGH_RR).
Dirty frontbuffers relevant to DRRS are tracked in busy_frontbuffer_bits.
- void
intel_edp_drrs_flush(struct drm_i915_private * dev_priv, unsigned int frontbuffer_bits)¶ Restart Idleness DRRS
Parameters
structdrm_i915_private*dev_priv- i915 device
unsignedintfrontbuffer_bits- frontbuffer plane tracking bits
Description
This function gets called every time rendering on the given planes hascompleted or flip on a crtc is completed. So DRRS should be upclocked(LOW_RR -> HIGH_RR). And also Idleness detection should be started again,if no other planes are dirty.
Dirty frontbuffers relevant to DRRS are tracked in busy_frontbuffer_bits.
- structdrm_display_mode *
intel_dp_drrs_init(struct intel_connector * connector, structdrm_display_mode * fixed_mode)¶ Init basic DRRS work and mutex.
Parameters
structintel_connector*connector- eDP connector
structdrm_display_mode*fixed_mode- preferred mode of panel
Description
This function is called only once at driver load to initialize basicDRRS stuff.
Return
Downclock mode if panel supports it, else return NULL.DRRS support is determined by the presence of downclock mode (apartfrom VBT setting).
DPIO¶
VLV, CHV and BXT have slightly peculiar display PHYs for driving DP/HDMIports. DPIO is the name given to such a display PHY. These PHYsdon’t follow the standard programming model using direct MMIOregisters, and instead their registers must be accessed trough IOSFsideband. VLV has one such PHY for driving ports B and C, and CHVadds another PHY for driving port D. Each PHY responds to specificIOSF-SB port.
Each display PHY is made up of one or two channels. Each channelhouses a common lane part which contains the PLL and other commonlogic. CH0 common lane also contains the IOSF-SB logic for theCommon Register Interface (CRI) ie. the DPIO registers. CRI clockmust be running when any DPIO registers are accessed.
In addition to having their own registers, the PHYs are alsocontrolled through some dedicated signals from the displaycontroller. These include PLL reference clock enable, PLL enable,and CRI clock selection, for example.
Eeach channel also has two splines (also called data lanes), andeach spline is made up of one Physical Access Coding Sub-Layer(PCS) block and two TX lanes. So each channel has two PCS blocksand four TX lanes. The TX lanes are used as DP lanes or TMDSdata/clock pairs depending on the output type.
Additionally the PHY also contains an AUX lane with AUX blocksfor each channel. This is used for DP AUX communication, butthis fact isn’t really relevant for the driver since AUX iscontrolled from the display controller side. No DPIO registersneed to be accessed during AUX communication,
Generally on VLV/CHV the common lane corresponds to the pipe andthe spline (PCS/TX) corresponds to the port.
For dual channel PHY (VLV/CHV):
pipe A == CMN/PLL/REF CH0
pipe B == CMN/PLL/REF CH1
port B == PCS/TX CH0
port C == PCS/TX CH1
This is especially important when we cross the streamsie. drive port B with pipe B, or port C with pipe A.
For single channel PHY (CHV):
pipe C == CMN/PLL/REF CH0
port D == PCS/TX CH0
On BXT the entire PHY channel corresponds to the port. That meansthe PLL is also now associated with the port rather than the pipe,and so the clock needs to be routed to the appropriate transcoder.Port A PLL is directly connected to transcoder EDP and port B/CPLLs can be routed to any transcoder A/B/C.
Note: DDI0 is digital port B, DD1 is digital port C, and DDI2 isdigital port D (CHV) or port A (BXT).
Dual channel PHY (VLV/CHV/BXT)---------------------------------| CH0 | CH1 || CMN/PLL/REF | CMN/PLL/REF ||---------------|---------------| Display PHY| PCS01 | PCS23 | PCS01 | PCS23 ||-------|-------|-------|-------||TX0|TX1|TX2|TX3|TX0|TX1|TX2|TX3|---------------------------------| DDI0 | DDI1 | DP/HDMI ports---------------------------------Single channel PHY (CHV/BXT)-----------------| CH0 || CMN/PLL/REF ||---------------| Display PHY| PCS01 | PCS23 ||-------|-------||TX0|TX1|TX2|TX3|-----------------| DDI2 | DP/HDMI port-----------------
CSR firmware support for DMC¶
Display Context Save and Restore (CSR) firmware support added from gen9onwards to drive newly added DMC (Display microcontroller) in displayengine to save and restore the state of display engine when it enter intolow-power state and comes back to normal.
- void
intel_csr_load_program(struct drm_i915_private * dev_priv)¶ write the firmware from memory to register.
Parameters
structdrm_i915_private*dev_priv- i915 drm device.
Description
CSR firmware is read from a .bin file and kept in internal memory one time.Everytime display comes back from low power state this function is called tocopy the firmware from internal memory to registers.
- void
intel_csr_ucode_init(struct drm_i915_private * dev_priv)¶ initialize the firmware loading.
Parameters
structdrm_i915_private*dev_priv- i915 drm device.
Description
This function is called at the time of loading the display driver to readfirmware from a .bin file and copied into a internal memory.
- void
intel_csr_ucode_suspend(struct drm_i915_private * dev_priv)¶ prepare CSR firmware before system suspend
Parameters
structdrm_i915_private*dev_priv- i915 drm device
Description
Prepare the DMC firmware before entering system suspend. This includesflushing pending work items and releasing any resources acquired duringinit.
- void
intel_csr_ucode_resume(struct drm_i915_private * dev_priv)¶ init CSR firmware during system resume
Parameters
structdrm_i915_private*dev_priv- i915 drm device
Description
Reinitialize the DMC firmware during system resume, reacquiring anyresources released inintel_csr_ucode_suspend().
- void
intel_csr_ucode_fini(struct drm_i915_private * dev_priv)¶ unload the CSR firmware.
Parameters
structdrm_i915_private*dev_priv- i915 drm device.
Description
Firmmware unloading includes freeing the internal memory and reset thefirmware loading status.
Video BIOS Table (VBT)¶
The Video BIOS Table, or VBT, provides platform and board specificconfiguration information to the driver that is not discoverable or availablethrough other means. The configuration is mostly related to displayhardware. The VBT is available via the ACPI OpRegion or, on older systems, inthe PCI ROM.
The VBT consists of a VBT Header (defined asstructvbt_header), a BDBHeader (structbdb_header), and a number of BIOS Data Blocks (BDB) thatcontain the actual configuration information. The VBT Header, and thus theVBT, begins with “$VBT” signature. The VBT Header contains the offset of theBDB Header. The data blocks are concatenated after the BDB Header. The datablocks have a 1-byte Block ID, 2-byte Block Size, and Block Size bytes ofdata. (Block 53, the MIPI Sequence Block is an exception.)
The driver parses the VBT during load. The relevant information is stored indriver private data for ease of use, and the actual VBT is not read afterthat.
- bool
intel_bios_is_valid_vbt(const void * buf, size_t size)¶ does the given buffer contain a valid VBT
Parameters
constvoid*buf- pointer to a buffer to validate
size_tsize- size of the buffer
Description
Returns true on valid VBT.
- void
intel_bios_init(struct drm_i915_private * dev_priv)¶ find VBT and initialize settings from the BIOS
Parameters
structdrm_i915_private*dev_priv- i915 device instance
Description
Parse and initialize settings from the Video BIOS Tables (VBT). If the VBTwas not found in ACPI OpRegion, try to find it in PCI ROM first. Alsoinitialize some defaults if the VBT is not present at all.
- void
intel_bios_driver_remove(struct drm_i915_private * dev_priv)¶ Free any resources allocated by
intel_bios_init()
Parameters
structdrm_i915_private*dev_priv- i915 device instance
- bool
intel_bios_is_tv_present(struct drm_i915_private * dev_priv)¶ is integrated TV present in VBT
Parameters
structdrm_i915_private*dev_priv- i915 device instance
Description
Return true if TV is present. If no child devices were parsed from VBT,assume TV is present.
- bool
intel_bios_is_lvds_present(struct drm_i915_private * dev_priv, u8 * i2c_pin)¶ is LVDS present in VBT
Parameters
structdrm_i915_private*dev_priv- i915 device instance
u8*i2c_pin- i2c pin for LVDS if present
Description
Return true if LVDS is present. If no child devices were parsed from VBT,assume LVDS is present.
- bool
intel_bios_is_port_present(struct drm_i915_private * dev_priv, enum port port)¶ is the specified digital port present
Parameters
structdrm_i915_private*dev_priv- i915 device instance
enumportport- port to check
Description
Return true if the device inport is present.
- bool
intel_bios_is_port_edp(struct drm_i915_private * dev_priv, enum port port)¶ is the device in given port eDP
Parameters
structdrm_i915_private*dev_priv- i915 device instance
enumportport- port to check
Description
Return true if the device inport is eDP.
- bool
intel_bios_is_dsi_present(struct drm_i915_private * dev_priv, enum port * port)¶ is DSI present in VBT
Parameters
structdrm_i915_private*dev_priv- i915 device instance
enumport*port- port for DSI if present
Description
Return true if DSI is present, and return the port inport.
- bool
intel_bios_is_port_hpd_inverted(const struct drm_i915_private * i915, enum port port)¶ is HPD inverted for
port
Parameters
conststructdrm_i915_private*i915- i915 device instance
enumportport- port to check
Description
Return true if HPD should be inverted forport.
- bool
intel_bios_is_lspcon_present(const struct drm_i915_private * i915, enum port port)¶ if LSPCON is attached on
port
Parameters
conststructdrm_i915_private*i915- i915 device instance
enumportport- port to check
Description
Return true if LSPCON is present on this port
- struct
vbt_header¶ VBT Header structure
Definition
struct vbt_header { u8 signature[20]; u16 version; u16 header_size; u16 vbt_size; u8 vbt_checksum; u8 reserved0; u32 bdb_offset; u32 aim_offset[4];};Members
signature- VBT signature, always starts with “$VBT”
version- Version of this structure
header_size- Size of this structure
vbt_size- Size of VBT (VBT Header, BDB Header and data blocks)
vbt_checksum- Checksum
reserved0- Reserved
bdb_offset- Offset of
structbdb_headerfrom beginning of VBT aim_offset- Offsets of add-in data blocks from beginning of VBT
- struct
bdb_header¶ BDB Header structure
Definition
struct bdb_header { u8 signature[16]; u16 version; u16 header_size; u16 bdb_size;};Members
signature- BDB signature “BIOS_DATA_BLOCK”
version- Version of the data block definitions
header_size- Size of this structure
bdb_size- Size of BDB (BDB Header and data blocks)
Display clocks¶
The display engine uses several different clocks to do its work. Thereare two main clocks involved that aren’t directly related to the actualpixel clock or any symbol/bit clock of the actual output port. Theseare the core display clock (CDCLK) and RAWCLK.
CDCLK clocks most of the display pipe logic, and thus its frequencymust be high enough to support the rate at which pixels are flowingthrough the pipes. Downscaling must also be accounted as that increasesthe effective pixel rate.
On several platforms the CDCLK frequency can be changed dynamicallyto minimize power consumption for a given display configuration.Typically changes to the CDCLK frequency require all the display pipesto be shut down while the frequency is being changed.
On SKL+ the DMC will toggle the CDCLK off/on during DC5/6 entry/exit.DMC will not change the active CDCLK frequency however, so that partwill still be performed by the driver directly.
RAWCLK is a fixed frequency clock, often used by various auxiliaryblocks such as AUX CH or backlight PWM. Hence the only thing wereally need to know about RAWCLK is its frequency so that variousdividers can be programmed correctly.
- void
intel_cdclk_init_hw(struct drm_i915_private * i915)¶ Initialize CDCLK hardware
Parameters
structdrm_i915_private*i915- i915 device
Description
Initialize CDCLK. This consists mainly of initializing dev_priv->cdclk.hw andsanitizing the state of the hardware if needed. This is generally done onlyduring the display core initialization sequence, after which the DMC willtake care of turning CDCLK off/on as needed.
- void
intel_cdclk_uninit_hw(struct drm_i915_private * i915)¶ Uninitialize CDCLK hardware
Parameters
structdrm_i915_private*i915- i915 device
Description
Uninitialize CDCLK. This is done only during the display coreuninitialization sequence.
- bool
intel_cdclk_needs_modeset(const struct intel_cdclk_config * a, const struct intel_cdclk_config * b)¶ Determine if changong between the CDCLK configurations requires a modeset on all pipes
Parameters
conststructintel_cdclk_config*a- first CDCLK configuration
conststructintel_cdclk_config*b- second CDCLK configuration
Return
True if changing between the two CDCLK configurationsrequires all pipes to be off, false if not.
- bool
intel_cdclk_can_cd2x_update(struct drm_i915_private * dev_priv, const struct intel_cdclk_config * a, const struct intel_cdclk_config * b)¶ Determine if changing between the two CDCLK configurations requires only a cd2x divider update
Parameters
structdrm_i915_private*dev_priv- i915 device
conststructintel_cdclk_config*a- first CDCLK configuration
conststructintel_cdclk_config*b- second CDCLK configuration
Return
True if changing between the two CDCLK configurationscan be done with just a cd2x divider update, false if not.
- bool
intel_cdclk_changed(const struct intel_cdclk_config * a, const struct intel_cdclk_config * b)¶ Determine if two CDCLK configurations are different
Parameters
conststructintel_cdclk_config*a- first CDCLK configuration
conststructintel_cdclk_config*b- second CDCLK configuration
Return
True if the CDCLK configurations don’t match, false if they do.
- void
intel_set_cdclk(struct drm_i915_private * dev_priv, const struct intel_cdclk_config * cdclk_config, enum pipe pipe)¶ Push the CDCLK configuration to the hardware
Parameters
structdrm_i915_private*dev_priv- i915 device
conststructintel_cdclk_config*cdclk_config- new CDCLK configuration
enumpipepipe- pipe with which to synchronize the update
Description
Program the hardware based on the passed in CDCLK state,if necessary.
- void
intel_set_cdclk_pre_plane_update(struct intel_atomic_state * state)¶ Push the CDCLK state to the hardware
Parameters
structintel_atomic_state*state- intel atomic state
Description
Program the hardware before updating the HW plane state based on thenew CDCLK state, if necessary.
- void
intel_set_cdclk_post_plane_update(struct intel_atomic_state * state)¶ Push the CDCLK state to the hardware
Parameters
structintel_atomic_state*state- intel atomic state
Description
Program the hardware after updating the HW plane state based on thenew CDCLK state, if necessary.
- void
intel_update_max_cdclk(struct drm_i915_private * dev_priv)¶ Determine the maximum support CDCLK frequency
Parameters
structdrm_i915_private*dev_priv- i915 device
Description
Determine the maximum CDCLK frequency the platform supports, and alsoderive the maximum dot clock frequency the maximum CDCLK frequencyallows.
- void
intel_update_cdclk(struct drm_i915_private * dev_priv)¶ Determine the current CDCLK frequency
Parameters
structdrm_i915_private*dev_priv- i915 device
Description
Determine the current CDCLK frequency.
- u32
intel_read_rawclk(struct drm_i915_private * dev_priv)¶ Determine the current RAWCLK frequency
Parameters
structdrm_i915_private*dev_priv- i915 device
Description
Determine the current RAWCLK frequency. RAWCLK is a fixedfrequency clock so this needs to done only once.
- void
intel_init_cdclk_hooks(struct drm_i915_private * dev_priv)¶ Initialize CDCLK related modesetting hooks
Parameters
structdrm_i915_private*dev_priv- i915 device
Display PLLs¶
Display PLLs used for driving outputs vary by platform. While some haveper-pipe or per-encoder dedicated PLLs, others allow the use of any PLLfrom a pool. In the latter scenario, it is possible that multiple pipesshare a PLL if their configurations match.
This file provides an abstraction over display PLLs. The functionintel_shared_dpll_init() initializes the PLLs for the given platform. Theusers of a PLL are tracked and that tracking is integrated with the atomicmodset interface. During an atomic operation, required PLLs can be reservedfor a given CRTC and encoder configuration by callingintel_reserve_shared_dplls() and previously reserved PLLs can be releasedwithintel_release_shared_dplls().Changes to the users are first staged in the atomic state, and then madeeffective by callingintel_shared_dpll_swap_state() during the atomiccommit phase.
- structintel_shared_dpll *
intel_get_shared_dpll_by_id(struct drm_i915_private * dev_priv, enumintel_dpll_id id)¶ get a DPLL given its id
Parameters
structdrm_i915_private*dev_priv- i915 device instance
enumintel_dpll_idid- pll id
Return
A pointer to the DPLL withid
- enumintel_dpll_id
intel_get_shared_dpll_id(struct drm_i915_private * dev_priv, structintel_shared_dpll * pll)¶ get the id of a DPLL
Parameters
structdrm_i915_private*dev_priv- i915 device instance
structintel_shared_dpll*pll- the DPLL
Return
The id ofpll
- void
intel_prepare_shared_dpll(const struct intel_crtc_state * crtc_state)¶ call a dpll’s prepare hook
Parameters
conststructintel_crtc_state*crtc_state- CRTC, and its state, which has a shared dpll
Description
This calls the PLL’s prepare hook if it has one and if the PLL is notalready enabled. The prepare hook is platform specific.
- void
intel_enable_shared_dpll(const struct intel_crtc_state * crtc_state)¶ enable a CRTC’s shared DPLL
Parameters
conststructintel_crtc_state*crtc_state- CRTC, and its state, which has a shared DPLL
Description
Enable the shared DPLL used bycrtc.
- void
intel_disable_shared_dpll(const struct intel_crtc_state * crtc_state)¶ disable a CRTC’s shared DPLL
Parameters
conststructintel_crtc_state*crtc_state- CRTC, and its state, which has a shared DPLL
Description
Disable the shared DPLL used bycrtc.
- void
intel_shared_dpll_swap_state(struct intel_atomic_state * state)¶ make atomic DPLL configuration effective
Parameters
structintel_atomic_state*state- atomic state
Description
This is the dpll version ofdrm_atomic_helper_swap_state() since thehelper does not handle driver-specific global state.
For consistency with atomic helpers this function does a complete swap,i.e. it also puts the current state intostate, even though there is noneed for that at this moment.
- void
icl_set_active_port_dpll(struct intel_crtc_state * crtc_state, enum icl_port_dpll_id port_dpll_id)¶ select the active port DPLL for a given CRTC
Parameters
structintel_crtc_state*crtc_state- state for the CRTC to select the DPLL for
enumicl_port_dpll_idport_dpll_id- the activeport_dpll_id to select
Description
Select the givenport_dpll_id instance from the DPLLs reserved for theCRTC.
- void
intel_shared_dpll_init(structdrm_device * dev)¶ Initialize shared DPLLs
Parameters
structdrm_device*dev- drm device
Description
Initialize shared DPLLs fordev.
- bool
intel_reserve_shared_dplls(struct intel_atomic_state * state, struct intel_crtc * crtc, struct intel_encoder * encoder)¶ reserve DPLLs for CRTC and encoder combination
Parameters
structintel_atomic_state*state- atomic state
structintel_crtc*crtc- CRTC to reserve DPLLs for
structintel_encoder*encoder- encoder
Description
This function reserves all required DPLLs for the given CRTC and encodercombination in the current atomic commitstate and the newcrtc atomicstate.
The new configuration in the atomic commitstate is made effective bycallingintel_shared_dpll_swap_state().
The reserved DPLLs should be released by callingintel_release_shared_dplls().
Return
True if all required DPLLs were successfully reserved.
- void
intel_release_shared_dplls(struct intel_atomic_state * state, struct intel_crtc * crtc)¶ end use of DPLLs by CRTC in atomic state
Parameters
structintel_atomic_state*state- atomic state
structintel_crtc*crtc- crtc from which the DPLLs are to be released
Description
This function releases all DPLLs reserved byintel_reserve_shared_dplls()from the current atomic commitstate and the oldcrtc atomic state.
The new configuration in the atomic commitstate is made effective bycallingintel_shared_dpll_swap_state().
- void
intel_update_active_dpll(struct intel_atomic_state * state, struct intel_crtc * crtc, struct intel_encoder * encoder)¶ update the active DPLL for a CRTC/encoder
Parameters
structintel_atomic_state*state- atomic state
structintel_crtc*crtc- the CRTC for which to update the active DPLL
structintel_encoder*encoder- encoder determining the type of port DPLL
Description
Update the active DPLL for the givencrtc/encoder incrtc’s atomic state,from the port DPLLs reserved previously byintel_reserve_shared_dplls(). TheDPLL selected will be based on the current mode of the encoder’s port.
- int
intel_dpll_get_freq(struct drm_i915_private * i915, const structintel_shared_dpll * pll)¶ calculate the DPLL’s output frequency
Parameters
structdrm_i915_private*i915- i915 device
conststructintel_shared_dpll*pll- DPLL for which to calculate the output frequency
Description
Return the output frequency corresponding topll’s current state.
- void
intel_dpll_dump_hw_state(struct drm_i915_private * dev_priv, const struct intel_dpll_hw_state * hw_state)¶ write hw_state to dmesg
Parameters
structdrm_i915_private*dev_priv- i915 drm device
conststructintel_dpll_hw_state*hw_state- hw state to be written to the log
Description
Write the relevant values inhw_state to dmesg using drm_dbg_kms.
- enum
intel_dpll_id¶ possible DPLL ids
Constants
DPLL_ID_PRIVATE- non-shared dpll in use
DPLL_ID_PCH_PLL_A- DPLL A in ILK, SNB and IVB
DPLL_ID_PCH_PLL_B- DPLL B in ILK, SNB and IVB
DPLL_ID_WRPLL1- HSW and BDW WRPLL1
DPLL_ID_WRPLL2- HSW and BDW WRPLL2
DPLL_ID_SPLL- HSW and BDW SPLL
DPLL_ID_LCPLL_810- HSW and BDW 0.81 GHz LCPLL
DPLL_ID_LCPLL_1350- HSW and BDW 1.35 GHz LCPLL
DPLL_ID_LCPLL_2700- HSW and BDW 2.7 GHz LCPLL
DPLL_ID_SKL_DPLL0- SKL and later DPLL0
DPLL_ID_SKL_DPLL1- SKL and later DPLL1
DPLL_ID_SKL_DPLL2- SKL and later DPLL2
DPLL_ID_SKL_DPLL3- SKL and later DPLL3
DPLL_ID_ICL_DPLL0- ICL/TGL combo PHY DPLL0
DPLL_ID_ICL_DPLL1- ICL/TGL combo PHY DPLL1
DPLL_ID_EHL_DPLL4- EHL combo PHY DPLL4
DPLL_ID_ICL_TBTPLL- ICL/TGL TBT PLL
DPLL_ID_ICL_MGPLL1- ICL MG PLL 1 port 1 (C),
- TGL TC PLL 1 port 1 (TC1)
DPLL_ID_ICL_MGPLL2- ICL MG PLL 1 port 2 (D)
- TGL TC PLL 1 port 2 (TC2)
DPLL_ID_ICL_MGPLL3- ICL MG PLL 1 port 3 (E)
- TGL TC PLL 1 port 3 (TC3)
DPLL_ID_ICL_MGPLL4- ICL MG PLL 1 port 4 (F)
- TGL TC PLL 1 port 4 (TC4)
DPLL_ID_TGL_MGPLL5- TGL TC PLL port 5 (TC5)
DPLL_ID_TGL_MGPLL6- TGL TC PLL port 6 (TC6)
Description
Enumeration of possible IDs for a DPLL. Real shared dpll ids must be >= 0.
- struct
intel_shared_dpll_state¶ hold the DPLL atomic state
Definition
struct intel_shared_dpll_state { unsigned crtc_mask; struct intel_dpll_hw_state hw_state;};Members
crtc_mask- mask of CRTC using this DPLL, active or not
hw_state- hardware configuration for the DPLL stored instruct
intel_dpll_hw_state.
Description
This structure holds an atomic state for the DPLL, that can representeither its current state (in structintel_shared_dpll) or a desiredfuture state which would be applied by an atomic mode set (stored ina structintel_atomic_state).
See alsointel_reserve_shared_dplls() andintel_release_shared_dplls().
- struct
intel_shared_dpll_funcs¶ platform specific hooks for managing DPLLs
Definition
struct intel_shared_dpll_funcs { void (*prepare)(struct drm_i915_private *dev_priv, struct intel_shared_dpll *pll); void (*enable)(struct drm_i915_private *dev_priv, struct intel_shared_dpll *pll); void (*disable)(struct drm_i915_private *dev_priv, struct intel_shared_dpll *pll); bool (*get_hw_state)(struct drm_i915_private *dev_priv,struct intel_shared_dpll *pll, struct intel_dpll_hw_state *hw_state); int (*get_freq)(struct drm_i915_private *i915, const struct intel_shared_dpll *pll);};Members
prepare- Optional hook to perform operations prior to enabling the PLL.Called from
intel_prepare_shared_dpll()function unless the PLLis already enabled. enable- Hook for enabling the pll, called from
intel_enable_shared_dpll()if the pll is not already enabled. disable- Hook for disabling the pll, called from
intel_disable_shared_dpll()only when it is safe to disable the pll, i.e., there are no moretracked users for it. get_hw_state- Hook for reading the values currently programmed to the DPLLregisters. This is used for initial hw state readout and stateverification after a mode set.
get_freq- Hook for calculating the pll’s output frequency based on itscurrent state.
- struct
dpll_info¶ display PLL platform specific info
Definition
struct dpll_info { const char *name; const struct intel_shared_dpll_funcs *funcs; enum intel_dpll_id id;#define INTEL_DPLL_ALWAYS_ON (1 << 0); u32 flags;};Members
name- DPLL name; used for logging
funcs- platform specific hooks
id- unique indentifier for this DPLL; should match the index in thedev_priv->shared_dplls array
flags- INTEL_DPLL_ALWAYS_ON
- Inform the state checker that the DPLL is kept enabled even ifnot in use by any CRTC.
- struct
intel_shared_dpll¶ display PLL with tracked state and users
Definition
struct intel_shared_dpll { struct intel_shared_dpll_state state; unsigned active_mask; bool on; const struct dpll_info *info; intel_wakeref_t wakeref;};Members
state- Store the state for the pll, including its hw stateand CRTCs using it.
active_mask- mask of active CRTCs (i.e. DPMS on) using this DPLL
on- is the PLL actually active? Disabled during modeset
info- platform specific info
wakeref- In some platforms a device-level runtime pm reference mayneed to be grabbed to disable DC states while this DPLL is enabled
Display State Buffer¶
A DSB (Display State Buffer) is a queue of MMIO instructions in the memorywhich can be offloaded to DSB HW in Display Controller. DSB HW is a DMAengine that can be programmed to download the DSB from memory.It allows driver to batch submit display HW programming. This helps toreduce loading time and CPU activity, thereby making the context switchfaster. DSB Support added from Gen12 Intel graphics based platform.
DSB’s can access only the pipe, plane, and transcoder Data Island Packetregisters.
DSB HW can support only register writes (both indexed and direct MMIOwrites). There are no registers reads possible with DSB HW engine.
- void
intel_dsb_indexed_reg_write(const struct intel_crtc_state * crtc_state, i915_reg_t reg, u32 val)¶ Write to the DSB context for auto increment register.
Parameters
conststructintel_crtc_state*crtc_state- intel_crtc_state structure
i915_reg_treg- register address.
u32val- value.
Description
This function is used for writing register-value pair in commandbuffer of DSB for auto-increment register. During command buffer overflow,a warning is thrown and rest all erroneous condition register programmingis done through mmio write.
- void
intel_dsb_reg_write(const struct intel_crtc_state * crtc_state, i915_reg_t reg, u32 val)¶ Write to the DSB context for normal register.
Parameters
conststructintel_crtc_state*crtc_state- intel_crtc_state structure
i915_reg_treg- register address.
u32val- value.
Description
This function is used for writing register-value pair in commandbuffer of DSB. During command buffer overflow, a warning is thrownand rest all erroneous condition register programming is donethrough mmio write.
- void
intel_dsb_commit(const struct intel_crtc_state * crtc_state)¶ Trigger workload execution of DSB.
Parameters
conststructintel_crtc_state*crtc_state- intel_crtc_state structure
Description
This function is used to do actual write to hardware using DSB.On errors, fall back to MMIO. Also this function help to reset the context.
- void
intel_dsb_prepare(struct intel_crtc_state * crtc_state)¶ Allocate, pin and map the DSB command buffer.
Parameters
structintel_crtc_state*crtc_state- intel_crtc_state structure to prepare associated dsb instance.
Description
This function prepare the command buffer which is used to store dsbinstructions with data.
- void
intel_dsb_cleanup(struct intel_crtc_state * crtc_state)¶ To cleanup DSB context.
Parameters
structintel_crtc_state*crtc_state- intel_crtc_state structure to cleanup associated dsb instance.
Description
This function cleanup the DSB context by unpinning and releasingthe VMA object associated with it.
Memory Management and Command Submission¶
This sections covers all things related to the GEM implementation in thei915 driver.
Intel GPU Basics¶
An Intel GPU has multiple engines. There are several engine types.
- RCS engine is for rendering 3D and performing compute, this is namedI915_EXEC_RENDER in user space.
- BCS is a blitting (copy) engine, this is namedI915_EXEC_BLT in userspace.
- VCS is a video encode and decode engine, this is namedI915_EXEC_BSDin user space
- VECS is video enhancement engine, this is namedI915_EXEC_VEBOX in userspace.
- The enumerationI915_EXEC_DEFAULT does not refer to specific engine;instead it is to be used by user space to specify a default renderingengine (for 3D) that may or may not be the same as RCS.
The Intel GPU family is a family of integrated GPU’s using UnifiedMemory Access. For having the GPU “do work”, user space will feed theGPU batch buffers via one of the ioctlsDRM_IOCTL_I915_GEM_EXECBUFFER2orDRM_IOCTL_I915_GEM_EXECBUFFER2_WR. Most such batchbuffers willinstruct the GPU to perform work (for example rendering) and that workneeds memory from which to read and memory to which to write. All memoryis encapsulated within GEM buffer objects (usually created with the ioctlDRM_IOCTL_I915_GEM_CREATE). An ioctl providing a batchbuffer for the GPUto create will also list all GEM buffer objects that the batchbuffer readsand/or writes. For implementation details of memory management seeGEM BO Management Implementation Details.
The i915 driver allows user space to create a context via the ioctlDRM_IOCTL_I915_GEM_CONTEXT_CREATE which is identified by a 32-bitinteger. Such a context should be viewed by user-space as -loosely-analogous to the idea of a CPU process of an operating system. The i915driver guarantees that commands issued to a fixed context are to beexecuted so that writes of a previously issued command are seen byreads of following commands. Actions issued between different contexts(even if from the same file descriptor) are NOT given that guaranteeand the only way to synchronize across contexts (even from the samefile descriptor) is through the use of fences. At least as far back asGen4, also have that a context carries with it a GPU HW context;the HW context is essentially (most of atleast) the state of a GPU.In addition to the ordering guarantees, the kernel will restore GPUstate via HW context when commands are issued to a context, this savesuser space the need to restore (most of atleast) the GPU state at thestart of each batchbuffer. The non-deprecated ioctls to submit batchbufferwork can pass that ID (in the lower bits of drm_i915_gem_execbuffer2::rsvd1)to identify what context to use with the command.
The GPU has its own memory management and address space. The kerneldriver maintains the memory translation table for the GPU. For olderGPUs (i.e. those before Gen8), there is a single global such translationtable, a global Graphics Translation Table (GTT). For newer generationGPUs each context has its own translation table, called Per-ProcessGraphics Translation Table (PPGTT). Of important note, is that althoughPPGTT is named per-process it is actually per context. When user spacesubmits a batchbuffer, the kernel walks the list of GEM buffer objectsused by the batchbuffer and guarantees that not only is the memory ofeach such GEM buffer object resident but it is also present in the(PP)GTT. If the GEM buffer object is not yet placed in the (PP)GTT,then it is given an address. Two consequences of this are: the kernelneeds to edit the batchbuffer submitted to write the correct value ofthe GPU address when a GEM BO is assigned a GPU address and the kernelmight evict a different GEM BO from the (PP)GTT to make address roomfor another GEM BO. Consequently, the ioctls submitting a batchbufferfor execution also include a list of all locations within buffers thatrefer to GPU-addresses so that the kernel can edit the buffer correctly.This process is dubbed relocation.
Locking Guidelines¶
Note
This is a description of how the locking should be afterrefactoring is done. Does not necessarily reflect what the lockinglooks like while WIP.
All locking rules and interface contracts with cross-driver interfaces(dma-buf, dma_fence) need to be followed.
No struct_mutex anywhere in the code
dma_resv will be the outermost lock (when needed) and ww_acquire_ctxis to be hoisted at highest level and passed down within i915_gem_ctxin the call chain
While holding lru/memory manager (buddy, drm_mm, whatever) lockssystem memory allocations are not allowed
- Enforce this by priming lockdep (with fs_reclaim). If weallocate memory while holding these looks we get a rehashof the shrinker vs. struct_mutex saga, and that would bereal bad.
Do not nest different lru/memory manager locks within each other.Take them in turn to update memory allocations, relying on the object’sdma_resv ww_mutex to serialize against other operations.
The suggestion for lru/memory managers locks is that they are smallenough to be spinlocks.
All features need to come with exhaustive kernel selftests and/orIGT tests when appropriate
All LMEM uAPI paths need to be fully restartable (_interruptible()for all locks/waits/sleeps)
- Error handling validation through signal injection.Still the best strategy we have for validating GEM uAPIcorner cases.Must be excessively used in the IGT, and we need to checkthat we really have full path coverage of all error cases.
- -EDEADLK handling with ww_mutex
GEM BO Management Implementation Details¶
A VMA represents a GEM BO that is bound into an address space. Therefore, aVMA’s presence cannot be guaranteed before binding, or after unbinding theobject into/from the address space.
To make things as simple as possible (ie. no refcounting), a VMA’s lifetimewill always be <= an objects lifetime. So object refcounting should cover us.
Buffer Object Eviction¶
This section documents the interface functions for evicting bufferobjects to make space available in the virtual gpu address spaces. Notethat this is mostly orthogonal to shrinking buffer objects caches, whichhas the goal to make main memory (shared with the gpu through theunified memory architecture) available.
- int
i915_gem_evict_something(struct i915_address_space * vm, u64 min_size, u64 alignment, unsigned long color, u64 start, u64 end, unsigned flags)¶ Evict vmas to make room for binding a new one
Parameters
structi915_address_space*vm- address space to evict from
u64min_size- size of the desired free space
u64alignment- alignment constraint of the desired free space
unsignedlongcolor- color for the desired space
u64start- start (inclusive) of the range from which to evict objects
u64end- end (exclusive) of the range from which to evict objects
unsignedflags- additional flags to control the eviction algorithm
Description
This function will try to evict vmas until a free space satisfying therequirements is found. Callers must check first whether any such hole existsalready before calling this function.
This function is used by the object/vma binding code.
Since this function is only used to free up virtual address space it onlyignores pinned vmas, and not object where the backing storage itself ispinned. Hence obj->pages_pin_count does not protect against eviction.
To clarify: This is for freeing up virtual address space, not for freeingmemory in e.g. the shrinker.
- int
i915_gem_evict_for_node(struct i915_address_space * vm, structdrm_mm_node * target, unsigned int flags)¶ Evict vmas to make room for binding a new one
Parameters
structi915_address_space*vm- address space to evict from
structdrm_mm_node*target- range (and color) to evict for
unsignedintflags- additional flags to control the eviction algorithm
Description
This function will try to evict vmas that overlap the target node.
To clarify: This is for freeing up virtual address space, not for freeingmemory in e.g. the shrinker.
- int
i915_gem_evict_vm(struct i915_address_space * vm)¶ Evict all idle vmas from a vm
Parameters
structi915_address_space*vm- Address space to cleanse
Description
This function evicts all vmas from a vm.
This is used by the execbuf code as a last-ditch effort to defragment theaddress space.
To clarify: This is for freeing up virtual address space, not for freeingmemory in e.g. the shrinker.
Buffer Object Memory Shrinking¶
This section documents the interface function for shrinking memory usageof buffer object caches. Shrinking is used to make main memoryavailable. Note that this is mostly orthogonal to evicting bufferobjects, which has the goal to make space in gpu virtual address spaces.
- unsigned long
i915_gem_shrink(struct drm_i915_private * i915, unsigned long target, unsigned long * nr_scanned, unsigned int shrink)¶ Shrink buffer object caches
Parameters
structdrm_i915_private*i915- i915 device
unsignedlongtarget- amount of memory to make available, in pages
unsignedlong*nr_scanned- optional output for number of pages scanned (incremental)
unsignedintshrink- control flags for selecting cache types
Description
This function is the main interface to the shrinker. It will try to releaseup totarget pages of main memory backing storage from buffer objects.Selection of the specific caches can be done withflags. This is e.g. usefulwhen purgeable objects should be removed from caches preferentially.
Note that it’s not guaranteed that released amount is actually available asfree system memory - the pages might still be in-used to due to other reasons(like cpu mmaps) or the mm core has reused them before we could grab them.Therefore code that needs to explicitly shrink buffer objects caches (e.g. toavoid deadlocks in memory reclaim) must fall back toi915_gem_shrink_all().
Also note that any kind of pinning (both per-vma address space pins andbacking storage pins at the buffer object level) result in the shrinker codehaving to skip the object.
Return
The number of pages of backing storage actually released.
- unsigned long
i915_gem_shrink_all(struct drm_i915_private * i915)¶ Shrink buffer object caches completely
Parameters
structdrm_i915_private*i915- i915 device
Description
This is a simple wraper aroundi915_gem_shrink() to aggressively shrink allcaches completely. It also first waits for and retires all outstandingrequests to also be able to release backing storage for active objects.
This should only be used in code to intentionally quiescent the gpu or as alast-ditch effort when memory seems to have run out.
Return
The number of pages of backing storage actually released.
Batchbuffer Parsing¶
Motivation:Certain OpenGL features (e.g. transform feedback, performance monitoring)require userspace code to submit batches containing commands such asMI_LOAD_REGISTER_IMM to access various registers. Unfortunately, somegenerations of the hardware will noop these commands in “unsecure” batches(which includes all userspace batches submitted via i915) even though thecommands may be safe and represent the intended programming model of thedevice.
The software command parser is similar in operation to the command parsingdone in hardware for unsecure batches. However, the software parser allowssome operations that would be noop’d by hardware, if the parser determinesthe operation is safe, and submits the batch as “secure” to prevent hardwareparsing.
Threats:At a high level, the hardware (and software) checks attempt to preventgranting userspace undue privileges. There are three categories of privilege.
First, commands which are explicitly defined as privileged or which shouldonly be used by the kernel driver. The parser rejects such commands
Second, commands which access registers. To support correct/enhanceduserspace functionality, particularly certain OpenGL extensions, the parserprovides a whitelist of registers which userspace may safely access
Third, commands which access privileged memory (i.e. GGTT, HWS page, etc).The parser always rejects such commands.
The majority of the problematic commands fall in the MI_* range, with only afew specific commands on each engine (e.g. PIPE_CONTROL and MI_FLUSH_DW).
Implementation:Each engine maintains tables of commands and registers which the parseruses in scanning batch buffers submitted to that engine.
Since the set of commands that the parser must check for is significantlysmaller than the number of commands supported, the parser tables contain onlythose commands required by the parser. This generally works because commandopcode ranges have standard command length encodings. So for commands thatthe parser does not need to check, it can easily skip them. This isimplemented via a per-engine length decoding vfunc.
Unfortunately, there are a number of commands that do not follow the standardlength encoding for their opcode range, primarily amongst the MI_* commands.To handle this, the parser provides a way to define explicit “skip” entriesin the per-engine command tables.
Other command table entries map fairly directly to high level categoriesmentioned above: rejected, register whitelist. The parser implements a numberof checks, including the privileged memory checks, via a general bitmaskingmechanism.
- void
intel_engine_init_cmd_parser(struct intel_engine_cs * engine)¶ set cmd parser related fields for an engine
Parameters
structintel_engine_cs*engine- the engine to initialize
Description
Optionally initializes fields related to batch buffer command parsing in thestruct intel_engine_cs based on whether the platform requires softwarecommand parsing.
- void
intel_engine_cleanup_cmd_parser(struct intel_engine_cs * engine)¶ clean up cmd parser related fields
Parameters
structintel_engine_cs*engine- the engine to clean up
Description
Releases any resources related to command parsing that may have beeninitialized for the specified engine.
- int
intel_engine_cmd_parser(struct intel_engine_cs * engine, struct i915_vma * batch, u32 batch_offset, u32 batch_length, struct i915_vma * shadow, bool trampoline)¶ parse a batch buffer for privilege violations
Parameters
structintel_engine_cs*engine- the engine on which the batch is to execute
structi915_vma*batch- the batch buffer in question
u32batch_offset- byte offset in the batch at which execution starts
u32batch_length- length of the commands in batch_obj
structi915_vma*shadow- validated copy of the batch buffer in question
booltrampoline- whether to emit a conditional trampoline at the end of the batch
Description
Parses the specified batch buffer looking for privilege violations asdescribed in the overview.
Return
non-zero if the parser finds violations or otherwise fails; -EACCESif the batch appears legal but should use hardware parsing
- int
i915_cmd_parser_get_version(struct drm_i915_private * dev_priv)¶ get the cmd parser version number
Parameters
structdrm_i915_private*dev_priv- i915 device private
Description
The cmd parser maintains a simple increasing integer version number suitablefor passing to userspace clients to determine what operations are permitted.
Return
the current version number of the cmd parser
User Batchbuffer Execution¶
Userspace submits commands to be executed on the GPU as an instructionstream within a GEM object we call a batchbuffer. This instructions mayrefer to other GEM objects containing auxiliary state such as kernels,samplers, render targets and even secondary batchbuffers. Userspace doesnot know where in the GPU memory these objects reside and so before thebatchbuffer is passed to the GPU for execution, those addresses in thebatchbuffer and auxiliary objects are updated. This is known as relocation,or patching. To try and avoid having to relocate each object on the nextexecution, userspace is told the location of those objects in this pass,but this remains just a hint as the kernel may choose a new location forany object in the future.
At the level of talking to the hardware, submitting a batchbuffer for theGPU to execute is to add content to a buffer from which the HWcommand streamer is reading.
- Add a command to load the HW context. For Logical Ring Contexts, i.e.Execlists, this command is not placed on the same buffer as theremaining items.
- Add a command to invalidate caches to the buffer.
- Add a batchbuffer start command to the buffer; the start command isessentially a token together with the GPU address of the batchbufferto be executed.
- Add a pipeline flush to the buffer.
- Add a memory write command to the buffer to record when the GPUis done executing the batchbuffer. The memory write writes theglobal sequence number of the request,
i915_request::global_seqno;the i915 driver uses the current value in the register to determineif the GPU has completed the batchbuffer. - Add a user interrupt command to the buffer. This command instructsthe GPU to issue an interrupt when the command, pipeline flush andmemory write are completed.
- Inform the hardware of the additional commands added to the buffer(by updating the tail pointer).
Processing an execbuf ioctl is conceptually split up into a few phases.
- Validation - Ensure all the pointers, handles and flags are valid.
- Reservation - Assign GPU address space for every object
- Relocation - Update any addresses to point to the final locations
- Serialisation - Order the request with respect to its dependencies
- Construction - Construct a request to execute the batchbuffer
- Submission (at some point in the future execution)
Reserving resources for the execbuf is the most complicated phase. Weneither want to have to migrate the object in the address space, nor dowe want to have to update any relocations pointing to this object. Ideally,we want to leave the object where it is and for all the existing relocationsto match. If the object is given a new address, or if userspace thinks theobject is elsewhere, we have to parse all the relocation entries and updatethe addresses. Userspace can set the I915_EXEC_NORELOC flag to hint thatall the target addresses in all of its objects match the value in therelocation entries and that they all match the presumed offsets given by thelist of execbuffer objects. Using this knowledge, we know that if we haven’tmoved any buffers, all the relocation entries are valid and we can skipthe update. (If userspace is wrong, the likely outcome is an impromptu GPUhang.) The requirement for using I915_EXEC_NO_RELOC are:
The addresses written in the objects must match the correspondingreloc.presumed_offset which in turn must match the correspondingexecobject.offset.
Any render targets written to in the batch must be flagged withEXEC_OBJECT_WRITE.
To avoid stalling, execobject.offset should match the currentaddress of that object within the active context.
The reservation is done is multiple phases. First we try and keep anyobject already bound in its current location - so as long as meets theconstraints imposed by the new execbuffer. Any object left unbound after thefirst pass is then fitted into any available idle space. If an object doesnot fit, all objects are removed from the reservation and the process rerunafter sorting the objects into a priority order (more difficult to fitobjects are tried first). Failing that, the entire VM is cleared and we tryto fit the execbuf once last time before concluding that it simply will notfit.
A small complication to all of this is that we allow userspace not only tospecify an alignment and a size for the object in the address space, butwe also allow userspace to specify the exact offset. This objects aresimpler to place (the location is known a priori) all we have to do is makesure the space is available.
Once all the objects are in place, patching up the buried pointers to pointto the final locations is a fairly simple job of walking over the relocationentry arrays, looking up the right address and rewriting the value intothe object. Simple! … The relocation entries are stored in user memoryand so to access them we have to copy them into a local buffer. That copyhas to avoid taking any pagefaults as they may lead back to a GEM objectrequiring the struct_mutex (i.e. recursive deadlock). So once again we splitthe relocation into multiple passes. First we try to do everything within anatomic context (avoid the pagefaults) which requires that we never wait. Ifwe detect that we may wait, or if we need to fault, then we have to fallbackto a slower path. The slowpath has to drop the mutex. (Can you hear alarmbells yet?) Dropping the mutex means that we lose all the state we havebuilt up so far for the execbuf and we must reset any global data. However,we do leave the objects pinned in their final locations - which is apotential issue for concurrent execbufs. Once we have left the mutex, we canallocate and copy all the relocation entries into a large array at ourleisure, reacquire the mutex, reclaim all the objects and other state andthen proceed to update any incorrect addresses with the objects.
As we process the relocation entries, we maintain a record of whether theobject is being written to. Using NORELOC, we expect userspace to providethis information instead. We also check whether we can skip the relocationby comparing the expected value inside the relocation entry with the target’sfinal address. If they differ, we have to map the current object and rewritethe 4 or 8 byte pointer within.
Serialising an execbuf is quite simple according to the rules of the GEMABI. Execution within each context is ordered by the order of submission.Writes to any GEM object are in order of submission and are exclusive. Readsfrom a GEM object are unordered with respect to other reads, but ordered bywrites. A write submitted after a read cannot occur before the read, andsimilarly any read submitted after a write cannot occur before the write.Writes are ordered between engines such that only one write occurs at anytime (completing any reads beforehand) - using semaphores where availableand CPU serialisation otherwise. Other GEM access obey the same rules, anywrite (either via mmaps using set-domain, or via pwrite) must flush all GPUreads before starting, and any read (either using set-domain or pread) mustflush all GPU writes before starting. (Note we only employ a barrier before,we currently rely on userspace not concurrently starting a new executionwhilst reading or writing to an object. This may be an advantage or notdepending on how much you trust userspace not to shoot themselves in thefoot.) Serialisation may just result in the request being inserted intoa DAG awaiting its turn, but most simple is to wait on the CPU untilall dependencies are resolved.
After all of that, is just a matter of closing the request and handing it tothe hardware (well, leaving it in a queue to be executed). However, we alsooffer the ability for batchbuffers to be run with elevated privileges sothat they access otherwise hidden registers. (Used to adjust L3 cache etc.)Before any batch is given extra privileges we first must check that itcontains no nefarious instructions, we check that each instruction is fromour whitelist and all registers are also from an allowed list. We firstcopy the user’s batchbuffer to a shadow (so that the user doesn’t haveaccess to it, either by the CPU or GPU as we scan it) and then parse eachinstruction. If everything is ok, we set a flag telling the hardware to runthe batchbuffer in trusted mode, otherwise the ioctl is rejected.
Logical Rings, Logical Ring Contexts and Execlists¶
Motivation:GEN8 brings an expansion of the HW contexts: “Logical Ring Contexts”.These expanded contexts enable a number of new abilities, especially“Execlists” (also implemented in this file).
One of the main differences with the legacy HW contexts is that logicalring contexts incorporate many more things to the context’s state, likePDPs or ringbuffer control registers:
The reason why PDPs are included in the context is straightforward: asPPGTTs (per-process GTTs) are actually per-context, having the PDPscontained there mean you don’t need to do a ppgtt->switch_mm yourself,instead, the GPU will do it for you on the context switch.
But, what about the ringbuffer control registers (head, tail, etc..)?shouldn’t we just need a set of those per engine command streamer? This iswhere the name “Logical Rings” starts to make sense: by virtualizing therings, the engine cs shifts to a new “ring buffer” with every contextswitch. When you want to submit a workload to the GPU you: A) choose yourcontext, B) find its appropriate virtualized ring, C) write commands to itand then, finally, D) tell the GPU to switch to that context.
Instead of the legacy MI_SET_CONTEXT, the way you tell the GPU to switchto a contexts is via a context execution list, ergo “Execlists”.
LRC implementation:Regarding the creation of contexts, we have:
- One global default context.
- One local default context for each opened fd.
- One local extra context for each context create ioctl call.
Now that ringbuffers belong per-context (and not per-engine, like before)and that contexts are uniquely tied to a given engine (and not reusable,like before) we need:
- One ringbuffer per-engine inside each context.
- One backing object per-engine inside each context.
The global default context starts its life with these new objects fullyallocated and populated. The local default context for each opened fd ismore complex, because we don’t know at creation time which engine is goingto use them. To handle this, we have implemented a deferred creation of LRcontexts:
The local context starts its life as a hollow or blank holder, that onlygets populated for a given engine once we receive an execbuffer. If lateron we receive another execbuffer ioctl for the same context but a differentengine, we allocate/populate a new ringbuffer and context backing object andso on.
Finally, regarding local contexts created using the ioctl call: as they areonly allowed with the render ring, we can allocate & populate them rightaway (no need to defer anything, at least for now).
Execlists implementation:Execlists are the new method by which, on gen8+ hardware, workloads aresubmitted for execution (as opposed to the legacy, ringbuffer-based, method).This method works as follows:
When a request is committed, its commands (the BB start and any leading ortrailing commands, like the seqno breadcrumbs) are placed in the ringbufferfor the appropriate context. The tail pointer in the hardware context is notupdated at this time, but instead, kept by the driver in the ringbufferstructure. A structure representing this request is added to a request queuefor the appropriate engine: this structure contains a copy of the context’stail after the request was written to the ring buffer and a pointer to thecontext itself.
If the engine’s request queue was empty before the request was added, thequeue is processed immediately. Otherwise the queue will be processed duringa context switch interrupt. In any case, elements on the queue will get sent(in pairs) to the GPU’s ExecLists Submit Port (ELSP, for short) with aglobally unique 20-bits submission ID.
When execution of a request completes, the GPU updates the context statusbuffer with a context complete event and generates a context switch interrupt.During the interrupt handling, the driver examines the events in the buffer:for each context complete event, if the announced ID matches that on the headof the request queue, then that request is retired and removed from the queue.
After processing, if any requests were retired and the queue is not emptythen a new execution list can be submitted. The two requests at the front ofthe queue are next to be submitted but since a context may not occur twice inan execution list, if subsequent requests have the same ID as the first thenthe two requests must be combined. This is done simply by discarding requestsat the head of the queue until either only one requests is left (in which casewe use a NULL second context) or the first two requests have unique IDs.
By always executing the first two requests in the queue the driver ensuresthat the GPU is kept as busy as possible. In the case where a single contextcompletes but a second context is still executing, the request for this secondcontext will be at the head of the queue when we remove the first one. Thisrequest will then be resubmitted along with a new request for a different context,which will cause the hardware to continue executing the second request and queuethe new request (the GPU detects the condition of a context getting preemptedwith the same context and optimizes the context switch flow by not doingpreemption, but just sampling the new tail pointer).
Global GTT views¶
Background and previous state
Historically objects could exists (be bound) in global GTT space only assingular instances with a view representing all of the object’s backing pagesin a linear fashion. This view will be called a normal view.
To support multiple views of the same object, where the number of mappedpages is not equal to the backing store, or where the layout of the pagesis not linear, concept of a GGTT view was added.
One example of an alternative view is a stereo display driven by a singleimage. In this case we would have a framebuffer looking like this(2x2 pages):
1234
Above would represent a normal GGTT view as normally mapped for GPU or CPUrendering. In contrast, fed to the display engine would be an alternativeview which could look something like this:
12123434
In this example both the size and layout of pages in the alternative view isdifferent from the normal view.
Implementation and usage
GGTT views are implemented using VMAs and are distinguished via enumi915_ggtt_view_type and struct i915_ggtt_view.
A new flavour of core GEM functions which work with GGTT bound objects wereadded with the _ggtt_ infix, and sometimes with _view postfix to avoidrenaming in large amounts of code. They take the struct i915_ggtt_viewparameter encapsulating all metadata required to implement a view.
As a helper for callers which are only interested in the normal view,globally const i915_ggtt_view_normal singleton instance exists. All old coreGEM API functions, the ones not taking the view parameter, are operating on,or with the normal GGTT view.
Code wanting to add or use a new GGTT view needs to:
- Add a new enum with a suitable name.
- Extend the metadata in the i915_ggtt_view structure if required.
- Add support to i915_get_vma_pages().
New views are required to build a scatter-gather table from within thei915_get_vma_pages function. This table is stored in the vma.ggtt_view andexists for the lifetime of an VMA.
Core API is designed to have copy semantics which means that passed instruct i915_ggtt_view does not need to be persistent (left around aftercalling the core API functions).
- int
i915_gem_gtt_reserve(struct i915_address_space * vm, structdrm_mm_node * node, u64 size, u64 offset, unsigned long color, unsigned int flags)¶ reserve a node in an address_space (GTT)
Parameters
structi915_address_space*vm- the
structi915_address_space structdrm_mm_node*node- the
structdrm_mm_node(typically i915_vma.mode) u64size- how much space to allocate inside the GTT,must be #I915_GTT_PAGE_SIZE aligned
u64offset- where to insert inside the GTT,must be #I915_GTT_MIN_ALIGNMENT aligned, and the node(offset +size) must fit within the address space
unsignedlongcolor- color to apply to node, if this node is not from a VMA,color must be #I915_COLOR_UNEVICTABLE
unsignedintflags- control search and eviction behaviour
Description
i915_gem_gtt_reserve() tries to insert thenode at the exactoffset insidethe address space (usingsize andcolor). If thenode does not fit, ittries to evict any overlapping nodes from the GTT, including anyneighbouring nodes if the colors do not match (to ensure guard pages betweendiffering domains). Seei915_gem_evict_for_node() for the gory detailson the eviction algorithm. #PIN_NONBLOCK may used to prevent waiting onevicting active overlapping objects, and any overlapping node that is pinnedor marked as unevictable will also result in failure.
Return
0 on success, -ENOSPC if no suitable hole is found, -EINTR ifasked to wait for eviction and interrupted.
- int
i915_gem_gtt_insert(struct i915_address_space * vm, structdrm_mm_node * node, u64 size, u64 alignment, unsigned long color, u64 start, u64 end, unsigned int flags)¶ insert a node into an address_space (GTT)
Parameters
structi915_address_space*vm- the
structi915_address_space structdrm_mm_node*node- the
structdrm_mm_node(typically i915_vma.node) u64size- how much space to allocate inside the GTT,must be #I915_GTT_PAGE_SIZE aligned
u64alignment- required alignment of starting offset, may be 0 butif specified, this must be a power-of-two and at least#I915_GTT_MIN_ALIGNMENT
unsignedlongcolor- color to apply to node
u64start- start of any range restriction inside GTT (0 for all),must be #I915_GTT_PAGE_SIZE aligned
u64end- end of any range restriction inside GTT (U64_MAX for all),must be #I915_GTT_PAGE_SIZE aligned if not U64_MAX
unsignedintflags- control search and eviction behaviour
Description
i915_gem_gtt_insert() first searches for an available hole into whichis can insert the node. The hole address is aligned toalignment anditssize must then fit entirely within the [start,end] bounds. Thenodes on either side of the hole must matchcolor, or else a guard pagewill be inserted between the two nodes (or the node evicted). If nosuitable hole is found, first a victim is randomly selected and testedfor eviction, otherwise then the LRU list of objects within the GTTis scanned to find the first set of replacement nodes to create the hole.Those old overlapping nodes are evicted from the GTT (and so must berebound before any future use). Any node that is currently pinned cannotbe evicted (see i915_vma_pin()). Similar if the node’s VMA is currentlyactive and #PIN_NONBLOCK is specified, that node is also skipped whensearching for an eviction candidate. Seei915_gem_evict_something() forthe gory details on the eviction algorithm.
Return
0 on success, -ENOSPC if no suitable hole is found, -EINTR ifasked to wait for eviction and interrupted.
GTT Fences and Swizzling¶
- void
i915_vma_revoke_fence(struct i915_vma * vma)¶ force-remove fence for a VMA
Parameters
structi915_vma*vma- vma to map linearly (not through a fence reg)
Description
This function force-removes any fence from the given object, which is usefulif the kernel wants to do untiled GTT access.
- int
i915_vma_pin_fence(struct i915_vma * vma)¶ set up fencing for a vma
Parameters
structi915_vma*vma- vma to map through a fence reg
Description
When mapping objects through the GTT, userspace wants to be able to writeto them without having to worry about swizzling if the object is tiled.This function walks the fence regs looking for a free one forobj,stealing one if it can’t find any.
It then sets up the reg based on the object’s properties: address, pitchand tiling format.
For an untiled surface, this removes any existing fence.
0 on success, negative error code on failure.
Return
- struct i915_fence_reg *
i915_reserve_fence(struct i915_ggtt * ggtt)¶ Reserve a fence for vGPU
Parameters
structi915_ggtt*ggtt- Global GTT
Description
This function walks the fence regs looking for a free one and removeit from the fence_list. It is used to reserve fence for vGPU to use.
- void
i915_unreserve_fence(struct i915_fence_reg * fence)¶ Reclaim a reserved fence
Parameters
structi915_fence_reg*fence- the fence reg
Description
This function add a reserved fence register from vGPU to the fence_list.
- void
intel_ggtt_restore_fences(struct i915_ggtt * ggtt)¶ restore fence state
Parameters
structi915_ggtt*ggtt- Global GTT
Description
Restore the hw fence state to match the software tracking again, to be calledafter a gpu reset and on resume. Note that on runtime suspend we only cancelthe fences, to be reacquired by the user later.
- void
detect_bit_6_swizzle(struct i915_ggtt * ggtt)¶ detect bit 6 swizzling pattern
Parameters
structi915_ggtt*ggtt- Global GGTT
Description
Detects bit 6 swizzling of address lookup between IGD access and CPUaccess through main memory.
- void
i915_gem_object_do_bit_17_swizzle(struct drm_i915_gem_object * obj, struct sg_table * pages)¶ fixup bit 17 swizzling
Parameters
structdrm_i915_gem_object*obj- i915 GEM buffer object
structsg_table*pages- the scattergather list of physical pages
Description
This function fixes up the swizzling in case any page frame number for thisobject has changed in bit 17 since that state has been saved withi915_gem_object_save_bit_17_swizzle().
This is called when pinning backing storage again, since the kernel is freeto move unpinned backing storage around (either by directly moving pages orby swapping them out and back in again).
- void
i915_gem_object_save_bit_17_swizzle(struct drm_i915_gem_object * obj, struct sg_table * pages)¶ save bit 17 swizzling
Parameters
structdrm_i915_gem_object*obj- i915 GEM buffer object
structsg_table*pages- the scattergather list of physical pages
Description
This function saves the bit 17 of each page frame number so that swizzlingcan be fixed up later on withi915_gem_object_do_bit_17_swizzle(). This mustbe called before the backing storage can be unpinned.
Global GTT Fence Handling¶
Important to avoid confusions: “fences” in the i915 driver are not executionfences used to track command completion but hardware detiler objects whichwrap a given range of the global GTT. Each platform has only a fairly limitedset of these objects.
Fences are used to detile GTT memory mappings. They’re also connected to thehardware frontbuffer render tracking and hence interact with frontbuffercompression. Furthermore on older platforms fences are required for tiledobjects used by the display engine. They can also be used by the renderengine - they’re required for blitter commands and are optional for rendercommands. But on gen4+ both display (with the exception of fbc) and renderinghave their own tiling state bits and don’t need fences.
Also note that fences only support X and Y tiling and hence can’t be used forthe fancier new tiling formats like W, Ys and Yf.
Finally note that because fences are such a restricted resource they’redynamically associated with objects. Furthermore fence state is committed tothe hardware lazily to avoid unnecessary stalls on gen2/3. Therefore code mustexplicitly call i915_gem_object_get_fence() to synchronize fencing statusfor cpu access. Also note that some code wants an unfenced view, for thosecases the fence can be removed forcefully with i915_gem_object_put_fence().
Internally these functions will synchronize with userspace access by removingCPU ptes into GTT mmaps (not the GTT ptes themselves) as needed.
Hardware Tiling and Swizzling Details¶
The idea behind tiling is to increase cache hit rates by rearrangingpixel data so that a group of pixel accesses are in the same cacheline.Performance improvement from doing this on the back/depth buffer are onthe order of 30%.
Intel architectures make this somewhat more complicated, though, byadjustments made to addressing of data when the memory is in interleavedmode (matched pairs of DIMMS) to improve memory bandwidth.For interleaved memory, the CPU sends every sequential 64 bytesto an alternate memory channel so it can get the bandwidth from both.
The GPU also rearranges its accesses for increased bandwidth to interleavedmemory, and it matches what the CPU does for non-tiled. However, when tiledit does it a little differently, since one walks addresses not just in theX direction but also Y. So, along with alternating channels when bit6 of the address flips, it also alternates when other bits flip – Bits 9(every 512 bytes, an X tile scanline) and 10 (every two X tile scanlines)are common to both the 915 and 965-class hardware.
The CPU also sometimes XORs in higher bits as well, to improvebandwidth doing strided access like we do so frequently in graphics. Thisis called “Channel XOR Randomization” in the MCH documentation. The resultis that the CPU is XORing in either bit 11 or bit 17 to bit 6 of its addressdecode.
All of this bit 6 XORing has an effect on our memory management,as we need to make sure that the 3d driver can correctly address objectcontents.
If we don’t have interleaved memory, all tiling is safe and no swizzling isrequired.
When bit 17 is XORed in, we simply refuse to tile at all. Bit17 is not just a page offset, so as we page an object out and back in,individual pages in it will have different bit 17 addresses, resulting ineach 64 bytes being swapped with its neighbor!
Otherwise, if interleaved, we have to tell the 3d driver what the addressswizzling it needs to do is, since it’s writing with the CPU to the pages(bit 6 and potentially bit 11 XORed in), and the GPU is reading from thepages (bit 6, 9, and 10 XORed in), resulting in a cumulative bit swizzlingrequired by the CPU of XORing in bit 6, 9, 10, and potentially 11, in orderto match what the GPU expects.
Object Tiling IOCTLs¶
- u32
i915_gem_fence_size(struct drm_i915_private * i915, u32 size, unsigned int tiling, unsigned int stride)¶ required global GTT size for a fence
Parameters
structdrm_i915_private*i915- i915 device
u32size- object size
unsignedinttiling- tiling mode
unsignedintstride- tiling stride
Description
Return the required global GTT size for a fence (view of a tiled object),taking into account potential fence register mapping.
- u32
i915_gem_fence_alignment(struct drm_i915_private * i915, u32 size, unsigned int tiling, unsigned int stride)¶ required global GTT alignment for a fence
Parameters
structdrm_i915_private*i915- i915 device
u32size- object size
unsignedinttiling- tiling mode
unsignedintstride- tiling stride
Description
Return the required global GTT alignment for a fence (a view of a tiledobject), taking into account potential fence register mapping.
- int
i915_gem_set_tiling_ioctl(structdrm_device * dev, void * data, structdrm_file * file)¶ IOCTL handler to set tiling mode
Parameters
structdrm_device*dev- DRM device
void*data- data pointer for the ioctl
structdrm_file*file- DRM file for the ioctl call
Description
Sets the tiling mode of an object, returning the required swizzling ofbit 6 of addresses in the object.
Called by the user via ioctl.
Return
Zero on success, negative errno on failure.
- int
i915_gem_get_tiling_ioctl(structdrm_device * dev, void * data, structdrm_file * file)¶ IOCTL handler to get tiling mode
Parameters
structdrm_device*dev- DRM device
void*data- data pointer for the ioctl
structdrm_file*file- DRM file for the ioctl call
Description
Returns the current tiling mode and required bit 6 swizzling for the object.
Called by the user via ioctl.
Return
Zero on success, negative errno on failure.
i915_gem_set_tiling_ioctl() andi915_gem_get_tiling_ioctl() is the userspaceinterface to declare fence register requirements.
In principle GEM doesn’t care at all about the internal data layout of anobject, and hence it also doesn’t care about tiling or swizzling. There’s twoexceptions:
- For X and Y tiling the hardware provides detilers for CPU access, so calledfences. Since there’s only a limited amount of them the kernel must managethese, and therefore userspace must tell the kernel the object tiling if itwants to use fences for detiling.
- On gen3 and gen4 platforms have a swizzling pattern for tiled objects whichdepends upon the physical page frame number. When swapping such objects thepage frame number might change and the kernel must be able to fix this upand hence now the tiling. Note that on a subset of platforms withasymmetric memory channel population the swizzling pattern changes in anunknown way, and for those the kernel simply forbids swapping completely.
Since neither of this applies for new tiling layouts on modern platforms likeW, Ys and Yf tiling GEM only allows object tiling to be set to X or Y tiled.Anything else can be handled in userspace entirely without the kernel’sinvovlement.
Microcontrollers¶
Starting from gen9, three microcontrollers are available on the HW: thegraphics microcontroller (GuC), the HEVC/H.265 microcontroller (HuC) and thedisplay microcontroller (DMC). The driver is responsible for loading thefirmwares on the microcontrollers; the GuC and HuC firmwares are transferredto WOPCM using the DMA engine, while the DMC firmware is written through MMIO.
WOPCM¶
WOPCM Layout¶
The layout of the WOPCM will be fixed after writing to GuC WOPCM size andoffset registers whose values are calculated and determined by HuC/GuCfirmware size and set of hardware requirements/restrictions as shown below:
+=========> +====================+ <== WOPCM Top ^ | HW contexts RSVD | | +===> +====================+ <== GuC WOPCM Top | ^ | | | | | | | | | | | GuC | | | WOPCM | | | Size +--------------------+WOPCM | | GuC FW RSVD | | | +--------------------+ | | | GuC Stack RSVD | | | +------------------- + | v | GuC WOPCM RSVD | | +===> +====================+ <== GuC WOPCM base | | WOPCM RSVD | | +------------------- + <== HuC Firmware Top v | HuC FW | +=========> +====================+ <== WOPCM Base
GuC accessible WOPCM starts at GuC WOPCM base and ends at GuC WOPCM top.The top part of the WOPCM is reserved for hardware contexts (e.g. RC6context).
GuC¶
The GuC is a microcontroller inside the GT HW, introduced in gen9. The GuC isdesigned to offload some of the functionality usually performed by the hostdriver; currently the main operations it can take care of are:
- Authentication of the HuC, which is required to fully enable HuC usage.
- Low latency graphics context scheduling (a.k.a. GuC submission).
- GT Power management.
The enable_guc module parameter can be used to select which of thoseoperations to enable within GuC. Note that not all the operations aresupported on all gen9+ platforms.
Enabling the GuC is not mandatory and therefore the firmware is only loadedif at least one of the operations is selected. However, not loading the GuCmight result in the loss of some features that do require the GuC (currentlyjust the HuC, but more are expected to land in the future).
GuC Firmware Layout¶
The GuC/HuC firmware layout looks like this:
+======================================================================+| Firmware blob |+===============+===============+============+============+============+| CSS header | uCode | RSA key | modulus | exponent |+===============+===============+============+============+============+ <-header size-> <---header size continued -----------> <--- size -----------------------------------------------------------> <-key size-> <-mod size-> <-exp size->
The firmware may or may not have modulus key and exponent data. The header,uCode and RSA signature are must-have components that will be used by driver.Length of each components, which is all in dwords, can be found in header.In the case that modulus and exponent are not present in fw, a.k.a truncatedimage, the length value still appears in header.
Driver will do some basic fw size validation based on the following rules:
- Header, uCode and RSA are must-have components.
- All firmware components, if they present, are in the sequence illustratedin the layout table above.
- Length info of each component can be found in header, in dwords.
- Modulus and exponent key are not required by driver. They may not appearin fw. So driver will load a truncated firmware in this case.
GuC Memory Management¶
GuC can’t allocate any memory for its own usage, so all the allocations mustbe handled by the host driver. GuC accesses the memory via the GGTT, with theexception of the top and bottom parts of the 4GB address space, which areinstead re-mapped by the GuC HW to memory location of the FW itself (WOPCM)or other parts of the HW. The driver must take care not to place objects thatthe GuC is going to access in these reserved ranges. The layout of the GuCaddress space is shown below:
+===========> +====================+ <== FFFF_FFFF ^ | Reserved | | +====================+ <== GUC_GGTT_TOP | | | | | DRAM | GuC | |Address +===> +====================+ <== GuC ggtt_pin_bias Space ^ | | | | | | | GuC | GuC | | WOPCM | WOPCM | | Size | | | | | | v v | | +=======+===> +====================+ <== 0000_0000
The lower part of GuC Address Space [0, ggtt_pin_bias) is mapped to GuC WOPCMwhile upper part of GuC Address Space [ggtt_pin_bias, GUC_GGTT_TOP) is mappedto DRAM. The value of the GuC ggtt_pin_bias is the GuC WOPCM size.
- struct i915_vma *
intel_guc_allocate_vma(struct intel_guc * guc, u32 size)¶ Allocate a GGTT VMA for GuC usage
Parameters
structintel_guc*guc- the guc
u32size- size of area to allocate (both virtual space and memory)
Description
This is a wrapper to create an object for use with the GuC. In order touse it inside the GuC, an object needs to be pinned lifetime, so we allocateboth some backing storage and a range inside the Global GTT. We must pinit in the GGTT somewhere other than than [0, GUC ggtt_pin_bias) because thatrange is reserved inside GuC.
Return
A i915_vma if successful, otherwise an ERR_PTR.
GuC-specific firmware loader¶
- int
intel_guc_fw_upload(struct intel_guc * guc)¶ load GuC uCode to device
Parameters
structintel_guc*guc- intel_guc structure
Description
Called from intel_uc_init_hw() during driver load, resume from sleep andafter a GPU reset.
The firmware image should have already been fetched into memory, so onlycheck that fetch succeeded, and then transfer the image to the h/w.
Return
non-zero code on error
GuC-based command submission¶
IMPORTANT NOTE: GuC submission is currently not supported in i915. The GuCfirmware is moving to an updated submission interface and we plan toturn submission back on when that lands. The below documentation (and relatedcode) matches the old submission model and will be updated as part of theupgrade to the new flow.
GuC stage descriptor:During initialization, the driver allocates a static pool of 1024 suchdescriptors, and shares them with the GuC. Currently, we only use onedescriptor. This stage descriptor lets the GuC know about the workqueue andprocess descriptor. Theoretically, it also lets the GuC know about our HWcontexts (context ID, etc…), but we actually employ a kind of submissionwhere the GuC uses the LRCA sent via the work item instead. This is calleda “proxy” submission.
The Scratch registers:There are 16 MMIO-based registers start from 0xC180. The kernel driver writesa value to the action register (SOFT_SCRATCH_0) along with any data. It thentriggers an interrupt on the GuC via another register write (0xC4C8).Firmware writes a success/fail code back to the action register afterprocesses the request. The kernel driver polls waiting for this update andthen proceeds.
Work Items:There are several types of work items that the host may place into aworkqueue, each with its own requirements and limitations. Currently onlyWQ_TYPE_INORDER is needed to support legacy submission via GuC, whichrepresents in-order queue. The kernel driver packs ring tail pointer and anELSP context descriptor dword into Work Item.See guc_add_request()
HuC¶
The HuC is a dedicated microcontroller for usage in media HEVC (HighEfficiency Video Coding) operations. Userspace can directly use the firmwarecapabilities by adding HuC specific commands to batch buffers.
The kernel driver is only responsible for loading the HuC firmware andtriggering its security authentication, which is performed by the GuC. ForThe GuC to correctly perform the authentication, the HuC binary must beloaded before the GuC one. Loading the HuC is optional; however, not usingthe HuC might negatively impact power usage and/or performance of mediaworkloads, depending on the use-cases.
Seehttps://github.com/intel/media-driver for the latest details on HuCfunctionality.
- int
intel_huc_auth(struct intel_huc * huc)¶ Authenticate HuC uCode
Parameters
structintel_huc*huc- intel_huc structure
Description
Called after HuC and GuC firmware loading during intel_uc_init_hw().
This function invokes the GuC action to authenticate the HuC firmware,passing the offset of the RSA signature to intel_guc_auth_huc(). It thenwaits for up to 50ms for firmware verification ACK.
HuC Memory Management¶
Similarly to the GuC, the HuC can’t do any memory allocations on its own,with the difference being that the allocations for HuC usage are handled bythe userspace driver instead of the kernel one. The HuC accesses the memoryvia the PPGTT belonging to the context loaded on the VCS executing theHuC-specific commands.
HuC Firmware Layout¶
The HuC FW layout is the same as the GuC one, seeGuC Firmware Layout
Tracing¶
This sections covers all things related to the tracepoints implementedin the i915 driver.
i915_ppgtt_create and i915_ppgtt_release¶
With full ppgtt enabled each process using drm will allocate at least onetranslation table. With these traces it is possible to keep track of theallocation and of the lifetime of the tables; this can be used duringtesting/debug to verify that we are not leaking ppgtts.These traces identify the ppgtt through the vm pointer, which is also printedby the i915_vma_bind and i915_vma_unbind tracepoints.
i915_context_create and i915_context_free¶
These tracepoints are used to track creation and deletion of contexts.If full ppgtt is enabled, they also print the address of the vm assigned tothe context.
Perf¶
Overview¶
Gen graphics supports a large number of performance counters that can helpdriver and application developers understand and optimize their use of theGPU.
This i915 perf interface enables userspace to configure and open a filedescriptor representing a stream of GPU metrics which can then be read() asa stream of sample records.
The interface is particularly suited to exposing buffered metrics that arecaptured by DMA from the GPU, unsynchronized with and unrelated to the CPU.
Streams representing a single context are accessible to applications with acorresponding drm file descriptor, such that OpenGL can use the interfacewithout special privileges. Access to system-wide metrics requires rootprivileges by default, unless changed via the dev.i915.perf_event_paranoidsysctl option.
Comparison with Core Perf¶
The interface was initially inspired by the core Perf infrastructure butsome notable differences are:
i915 perf file descriptors represent a “stream” instead of an “event”; wherea perf event primarily corresponds to a single 64bit value, while a streammight sample sets of tightly-coupled counters, depending on theconfiguration. For example the Gen OA unit isn’t designed to supportorthogonal configurations of individual counters; it’s configured for a setof related counters. Samples for an i915 perf stream capturing OA metricswill include a set of counter values packed in a compact HW specific format.The OA unit supports a number of different packing formats which can beselected by the user opening the stream. Perf has support for groupingevents, but each event in the group is configured, validated andauthenticated individually with separate system calls.
i915 perf stream configurations are provided as an array of u64 (key,value)pairs, instead of a fixed struct with multiple miscellaneous config members,interleaved with event-type specific members.
i915 perf doesn’t support exposing metrics via an mmap’d circular buffer.The supported metrics are being written to memory by the GPU unsynchronizedwith the CPU, using HW specific packing formats for counter sets. Sometimesthe constraints on HW configuration require reports to be filtered before itwould be acceptable to expose them to unprivileged applications - to hidethe metrics of other processes/contexts. For these use cases a read() basedinterface is a good fit, and provides an opportunity to filter data as itgets copied from the GPU mapped buffers to userspace buffers.
Issues hit with first prototype based on Core Perf¶
The first prototype of this driver was based on the core perfinfrastructure, and while we did make that mostly work, with some changes toperf, we found we were breaking or working around too many assumptions bakedinto perf’s currently cpu centric design.
In the end we didn’t see a clear benefit to making perf’s implementation andinterface more complex by changing design assumptions while we knew we stillwouldn’t be able to use any existing perf based userspace tools.
Also considering the Gen specific nature of the Observability hardware andhow userspace will sometimes need to combine i915 perf OA metrics withside-band OA data captured via MI_REPORT_PERF_COUNT commands; we’reexpecting the interface to be used by a platform specific userspace such asOpenGL or tools. This is to say; we aren’t inherently missing out on havinga standard vendor/architecture agnostic interface by not using perf.
For posterity, in case we might re-visit trying to adapt core perf to bebetter suited to exposing i915 metrics these were the main pain points wehit:
The perf based OA PMU driver broke some significant design assumptions:
Existing perf pmus are used for profiling work on a cpu and we wereintroducing the idea of _IS_DEVICE pmus with different securityimplications, the need to fake cpu-related data (such as user/kernelregisters) to fit with perf’s current design, and adding _DEVICE recordsas a way to forward device-specific status records.
The OA unit writes reports of counters into a circular buffer, withoutinvolvement from the CPU, making our PMU driver the first of a kind.
Given the way we were periodically forward data from the GPU-mapped, OAbuffer to perf’s buffer, those bursts of sample writes looked to perf likewe were sampling too fast and so we had to subvert its throttling checks.
Perf supports groups of counters and allows those to be read viatransactions internally but transactions currently seem designed to beexplicitly initiated from the cpu (say in response to a userspace read())and while we could pull a report out of the OA buffer we can’ttrigger a report from the cpu on demand.
Related to being report based; the OA counters are configured in HW as aset while perf generally expects counter configurations to be orthogonal.Although counters can be associated with a group leader as they areopened, there’s no clear precedent for being able to provide group-wideconfiguration attributes (for example we want to let userspace choose theOA unit report format used to capture all counters in a set, or specify aGPU context to filter metrics on). We avoided using perf’s groupingfeature and forwarded OA reports to userspace via perf’s ‘raw’ samplefield. This suited our userspace well considering how coupled the countersare when dealing with normalizing. It would be inconvenient to splitcounters up into separate events, only to require userspace to recombinethem. For Mesa it’s also convenient to be forwarded raw, periodic reportsfor combining with the side-band raw reports it captures usingMI_REPORT_PERF_COUNT commands.
As a side note on perf’s grouping feature; there was also some concernthat using PERF_FORMAT_GROUP as a way to pack together counter valueswould quite drastically inflate our sample sizes, which would likelylower the effective sampling resolutions we could use when the availablememory bandwidth is limited.
With the OA unit’s report formats, counters are packed together as 32or 40bit values, with the largest report size being 256 bytes.
PERF_FORMAT_GROUP values are 64bit, but there doesn’t appear to be adocumented ordering to the values, implying PERF_FORMAT_ID must also beused to add a 64bit ID before each value; giving 16 bytes per counter.
Related to counter orthogonality; we can’t time share the OA unit, whileevent scheduling is a central design idea within perf for allowinguserspace to open + enable more events than can be configured in HW at anyone time. The OA unit is not designed to allow re-configuration while inuse. We can’t reconfigure the OA unit without losing internal OA unitstate which we can’t access explicitly to save and restore. Reconfiguringthe OA unit is also relatively slow, involving ~100 register writes. Fromuserspace Mesa also depends on a stable OA configuration when emittingMI_REPORT_PERF_COUNT commands and importantly the OA unit can’t bedisabled while there are outstanding MI_RPC commands lest we hang thecommand streamer.
The contents of sample records aren’t extensible by device drivers (i.e.the sample_type bits). As an example; Sourab Gupta had been looking toattach GPU timestamps to our OA samples. We were shoehorning OA reportsinto sample records by using the ‘raw’ field, but it’s tricky to pack morethan one thing into this field because events/core.c currently only lets apmu give a single raw data pointer plus len which will be copied into thering buffer. To include more than the OA report we’d have to copy thereport into an intermediate larger buffer. I’d been considering allowing avector of data+len values to be specified for copying the raw data, butit felt like a kludge to being using the raw field for this purpose.
It felt like our perf based PMU was making some technical compromisesjust for the sake of using perf:
perf_event_open() requires events to either relate to a pid or a specificcpu core, while our device pmu related to neither. Events opened with apid will be automatically enabled/disabled according to the scheduling ofthat process - so not appropriate for us. When an event is related to acpu id, perf ensures pmu methods will be invoked via an inter processinterrupt on that core. To avoid invasive changes our userspace opened OAperf events for a specific cpu. This was workable but it meant themajority of the OA driver ran in atomic context, including all OA reportforwarding, which wasn’t really necessary in our case and seems to makeour locking requirements somewhat complex as we handled the interactionwith the rest of the i915 driver.
i915 Driver Entry Points¶
This section covers the entrypoints exported outside of i915_perf.c tointegrate with drm/i915 and to handle theDRM_I915_PERF_OPEN ioctl.
- void
i915_perf_init(struct drm_i915_private * i915)¶ initialize i915-perf state on module bind
Parameters
structdrm_i915_private*i915- i915 device instance
Description
Initializes i915-perf state without exposing anything to userspace.
Note
i915-perf initialization is split into an ‘init’ and ‘register’phase with thei915_perf_register() exposing state to userspace.
- void
i915_perf_fini(struct drm_i915_private * i915)¶ Counter part to
i915_perf_init()
Parameters
structdrm_i915_private*i915- i915 device instance
- void
i915_perf_register(struct drm_i915_private * i915)¶ exposes i915-perf to userspace
Parameters
structdrm_i915_private*i915- i915 device instance
Description
In particular OA metric sets are advertised under a sysfs metrics/directory allowing userspace to enumerate valid IDs that can beused to open an i915-perf stream.
- void
i915_perf_unregister(struct drm_i915_private * i915)¶ hide i915-perf from userspace
Parameters
structdrm_i915_private*i915- i915 device instance
Description
i915-perf state cleanup is split up into an ‘unregister’ and‘deinit’ phase where the interface is first hidden fromuserspace byi915_perf_unregister() before cleaning upremaining state ini915_perf_fini().
- int
i915_perf_open_ioctl(structdrm_device * dev, void * data, structdrm_file * file)¶ DRM ioctl() for userspace to open a stream FD
Parameters
structdrm_device*dev- drm device
void*data- ioctl data copied from userspace (unvalidated)
structdrm_file*file- drm file
Description
Validates the stream open parameters given by userspace including flagsand an array of u64 key, value pair properties.
Very little is assumed up front about the nature of the stream beingopened (for instance we don’t assume it’s for periodic OA unit metrics). Ani915-perf stream is expected to be a suitable interface for other forms ofbuffered data written by the GPU besides periodic OA metrics.
Note we copy the properties from userspace outside of the i915 perfmutex to avoid an awkward lockdep with mmap_lock.
Most of the implementation details are handled byi915_perf_open_ioctl_locked() after taking theperf->lockmutex for serializing with any non-file-operation driver hooks.
Return
A newly opened i915 Perf stream file descriptor or negativeerror code on failure.
- int
i915_perf_release(struct inode * inode, struct file * file)¶ handles userspace close() of a stream file
Parameters
structinode*inode- anonymous inode associated with file
structfile*file- An i915 perf stream file
Description
Cleans up any resources associated with an open i915 perf stream file.
NB: close() can’t really fail from the userspace point of view.
Return
zero on success or a negative error code.
- int
i915_perf_add_config_ioctl(structdrm_device * dev, void * data, structdrm_file * file)¶ DRM ioctl() for userspace to add a new OA config
Parameters
structdrm_device*dev- drm device
void*data- ioctl data (pointer to struct drm_i915_perf_oa_config) copied fromuserspace (unvalidated)
structdrm_file*file- drm file
Description
Validates the submitted OA register to be saved into a new OA config thatcan then be used for programming the OA unit and its NOA network.
Return
A new allocated config number to be used with the perf open ioctlor a negative error code on failure.
- int
i915_perf_remove_config_ioctl(structdrm_device * dev, void * data, structdrm_file * file)¶ DRM ioctl() for userspace to remove an OA config
Parameters
structdrm_device*dev- drm device
void*data- ioctl data (pointer to u64 integer) copied from userspace
structdrm_file*file- drm file
Description
Configs can be removed while being used, the will stop appearing in sysfsand their content will be freed when the stream using the config is closed.
Return
0 on success or a negative error code on failure.
i915 Perf Stream¶
This section covers the stream-semantics-agnostic structures and functionsfor representing an i915 perf stream FD and associated file operations.
- struct
i915_perf_stream¶ state for a single open stream FD
Definition
struct i915_perf_stream { struct i915_perf *perf; struct intel_uncore *uncore; struct intel_engine_cs *engine; u32 sample_flags; int sample_size; struct i915_gem_context *ctx; bool enabled; bool hold_preemption; const struct i915_perf_stream_ops *ops; struct i915_oa_config *oa_config; struct llist_head oa_config_bos; struct intel_context *pinned_ctx; u32 specific_ctx_id; u32 specific_ctx_id_mask; struct hrtimer poll_check_timer; wait_queue_head_t poll_wq; bool pollin; bool periodic; int period_exponent; struct { struct i915_vma *vma; u8 *vaddr; u32 last_ctx_id; int format; int format_size; int size_exponent; spinlock_t ptr_lock; u32 aging_tail; u64 aging_timestamp; u32 head; u32 tail; } oa_buffer; struct i915_vma *noa_wait; u64 poll_oa_period;};Members
perf- i915_perf backpointer
uncore- mmio access path
engine- Engine associated with this performance stream.
sample_flags- Flags representing theDRM_I915_PERF_PROP_SAMPLE_*properties given when opening a stream, representing the contentsof a single sample as read() by userspace.
sample_size- Considering the configured contents of a samplecombined with the required header size, this is the total sizeof a single sample record.
ctxNULLif measuring system-wide across all contexts or aspecific context that is being monitored.enabled- Whether the stream is currently enabled, consideringwhether the stream was opened in a disabled state and basedonI915_PERF_IOCTL_ENABLE andI915_PERF_IOCTL_DISABLE calls.
hold_preemption- Whether preemption is put on hold for commandsubmissions done on thectx. This is useful for some drivers thatcannot easily post process the OA buffer context to subtract deltaof performance counters not associated withctx.
ops- The callbacks providing the implementation of this specifictype of configured stream.
oa_config- The OA configuration used by the stream.
oa_config_bos- A list of struct i915_oa_config_bo allocated lazilyeach timeoa_config changes.
pinned_ctx- The OA context specific information.
specific_ctx_id- The id of the specific context.
specific_ctx_id_mask- The mask used to masking specific_ctx_id bits.
poll_check_timer- High resolution timer that will periodicallycheck for data in the circular OA buffer for notifying userspace(e.g. during a read() or poll()).
poll_wq- The wait queue that hrtimer callback wakes when itsees data ready to read in the circular OA buffer.
pollin- Whether there is data available to read.
periodic- Whether periodic sampling is currently enabled.
period_exponent- The OA unit sampling frequency is derived from this.
oa_buffer- State of the OA buffer.
noa_wait- A batch buffer doing a wait on the GPU for the NOA logic to bereprogrammed.
poll_oa_period- The period in nanoseconds at which the OAbuffer should be checked for available data.
- struct
i915_perf_stream_ops¶ the OPs to support a specific stream type
Definition
struct i915_perf_stream_ops { void (*enable)(struct i915_perf_stream *stream); void (*disable)(struct i915_perf_stream *stream); void (*poll_wait)(struct i915_perf_stream *stream,struct file *file, poll_table *wait); int (*wait_unlocked)(struct i915_perf_stream *stream); int (*read)(struct i915_perf_stream *stream,char __user *buf,size_t count, size_t *offset); void (*destroy)(struct i915_perf_stream *stream);};Members
enable- Enables the collection of HW samples, either in response toI915_PERF_IOCTL_ENABLE or implicitly called when stream is openedwithoutI915_PERF_FLAG_DISABLED.
disable- Disables the collection of HW samples, either in responsetoI915_PERF_IOCTL_DISABLE or implicitly called before destroyingthe stream.
poll_wait- Call poll_wait, passing a wait queue that will be wokenonce there is something ready to read() for the stream
wait_unlocked- For handling a blocking read, wait until there issomething to ready to read() for the stream. E.g. wait on the samewait queue that would be passed to poll_wait().
readCopy buffered metrics as records to userspacebuf: the userspace, destination buffercount: the number of bytes to copy, requested by userspaceoffset: zero at the start of the read, updated as the readproceeds, it represents how many bytes have been copied so far andthe buffer offset for copying the next record.
Copy as many buffered i915 perf samples and records for this streamto userspace as will fit in the given buffer.
Only write complete records; returning -
ENOSPCif there isn’t roomfor a complete record.Return any error condition that results in a short read such as-
ENOSPCor -EFAULT, even though these may be squashed beforereturning to userspace.destroyCleanup any stream specific resources.
The stream will always be disabled before this is called.
- int
read_properties_unlocked(struct i915_perf * perf, u64 __user * uprops, u32 n_props, structperf_open_properties * props)¶ validate + copy userspace stream open properties
Parameters
structi915_perf*perf- i915 perf instance
u64__user*uprops- The array of u64 key value pairs given by userspace
u32n_props- The number of key value pairs expected inuprops
structperf_open_properties*props- The stream configuration built up while validating properties
Description
Note this function only validates properties in isolation it doesn’tvalidate that the combination of properties makes sense or that allproperties necessary for a particular kind of stream have been set.
Note that there currently aren’t any ordering requirements for properties sowe shouldn’t validate or assume anything about ordering here. This doesn’trule out defining new properties with ordering requirements in the future.
- int
i915_perf_open_ioctl_locked(struct i915_perf * perf, struct drm_i915_perf_open_param * param, structperf_open_properties * props, structdrm_file * file)¶ DRM ioctl() for userspace to open a stream FD
Parameters
structi915_perf*perf- i915 perf instance
structdrm_i915_perf_open_param*param- The open parameters passed to ‘DRM_I915_PERF_OPEN`
structperf_open_properties*props- individually validated u64 property value pairs
structdrm_file*file- drm file
Description
See i915_perf_ioctl_open() for interface details.
Implements further stream config validation and stream initialization onbehalf ofi915_perf_open_ioctl() with theperf->lock mutextaken to serialize with any non-file-operation driver hooks.
In the case where userspace is interested in OA unit metrics then furtherconfig validation and stream initialization details will be handled byi915_oa_stream_init(). The code here should only validate config state thatwill be relevant to all stream types / backends.
Note
at this point theprops have only been validated in isolation andit’s still necessary to validate that the combination of properties makessense.
Return
zero on success or a negative error code.
- void
i915_perf_destroy_locked(structi915_perf_stream * stream)¶ destroy an i915 perf stream
Parameters
structi915_perf_stream*stream- An i915 perf stream
Description
Frees all resources associated with the given i915 perfstream, disablingany associated data capture in the process.
Note
Theperf->lock mutex has been taken to serializewith any non-file-operation driver hooks.
- ssize_t
i915_perf_read(struct file * file, char __user * buf, size_t count, loff_t * ppos)¶ handles read() FOP for i915 perf stream FDs
Parameters
structfile*file- An i915 perf stream file
char__user*buf- destination buffer given by userspace
size_tcount- the number of bytes userspace wants to read
loff_t*ppos- (inout) file seek position (unused)
Description
The entry point for handling a read() on a stream file descriptor fromuserspace. Most of the work is left to the i915_perf_read_locked() andi915_perf_stream_ops->read but to save having stream implementations (ofwhich we might have multiple later) we handle blocking read here.
We can also consistently treat trying to read from a disabled streamas an IO error so implementations can assume the stream is enabledwhile reading.
Return
The number of bytes copied or a negative error code on failure.
- long
i915_perf_ioctl(struct file * file, unsigned int cmd, unsigned long arg)¶ support ioctl() usage with i915 perf stream FDs
Parameters
structfile*file- An i915 perf stream file
unsignedintcmd- the ioctl request
unsignedlongarg- the ioctl data
Description
Implementation deferred toi915_perf_ioctl_locked().
Return
zero on success or a negative error code. Returns -EINVAL foran unknown ioctl request.
- void
i915_perf_enable_locked(structi915_perf_stream * stream)¶ handleI915_PERF_IOCTL_ENABLE ioctl
Parameters
structi915_perf_stream*stream- A disabled i915 perf stream
Description
[Re]enables the associated capture of data for this stream.
If a stream was previously enabled then there’s currently no intentionto provide userspace any guarantee about the preservation of previouslybuffered data.
- void
i915_perf_disable_locked(structi915_perf_stream * stream)¶ handleI915_PERF_IOCTL_DISABLE ioctl
Parameters
structi915_perf_stream*stream- An enabled i915 perf stream
Description
Disables the associated capture of data for this stream.
The intention is that disabling an re-enabling a stream will ideally becheaper than destroying and re-opening a stream with the same configuration,though there are no formal guarantees about what state or buffered datamust be retained between disabling and re-enabling a stream.
Note
while a stream is disabled it’s considered an error for userspaceto attempt to read from the stream (-EIO).
- __poll_t
i915_perf_poll(struct file * file, poll_table * wait)¶ call poll_wait() with a suitable wait queue for stream
Parameters
structfile*file- An i915 perf stream file
poll_table*wait- poll() state table
Description
For handling userspace polling on an i915 perf stream, this ensurespoll_wait() gets called with a wait queue that will be woken for new streamdata.
Note
Implementation deferred toi915_perf_poll_locked()
Return
any poll events that are ready without sleeping
- __poll_t
i915_perf_poll_locked(structi915_perf_stream * stream, struct file * file, poll_table * wait)¶ poll_wait() with a suitable wait queue for stream
Parameters
structi915_perf_stream*stream- An i915 perf stream
structfile*file- An i915 perf stream file
poll_table*wait- poll() state table
Description
For handling userspace polling on an i915 perf stream, this calls through toi915_perf_stream_ops->poll_wait to call poll_wait() with a wait queue thatwill be woken for new stream data.
Note
Theperf->lock mutex has been taken to serializewith any non-file-operation driver hooks.
Return
any poll events that are ready without sleeping
i915 Perf Observation Architecture Stream¶
- struct
i915_oa_ops¶ Gen specific implementation of an OA unit stream
Definition
struct i915_oa_ops { bool (*is_valid_b_counter_reg)(struct i915_perf *perf, u32 addr); bool (*is_valid_mux_reg)(struct i915_perf *perf, u32 addr); bool (*is_valid_flex_reg)(struct i915_perf *perf, u32 addr); int (*enable_metric_set)(struct i915_perf_stream *stream, struct i915_active *active); void (*disable_metric_set)(struct i915_perf_stream *stream); void (*oa_enable)(struct i915_perf_stream *stream); void (*oa_disable)(struct i915_perf_stream *stream); int (*read)(struct i915_perf_stream *stream,char __user *buf,size_t count, size_t *offset); u32 (*oa_hw_tail_read)(struct i915_perf_stream *stream);};Members
is_valid_b_counter_reg- Validates register’s address forprogramming boolean counters for a particular platform.
is_valid_mux_reg- Validates register’s address for programming muxfor a particular platform.
is_valid_flex_reg- Validates register’s address for programmingflex EU filtering for a particular platform.
enable_metric_set- Selects and applies any MUX configuration to setup the Boolean and Custom (B/C) counters that are part of thecounter reports being sampled. May apply system constraints such asdisabling EU clock gating as required.
disable_metric_set- Remove system constraints associated with usingthe OA unit.
oa_enable- Enable periodic sampling
oa_disable- Disable periodic sampling
read- Copy data from the circular OA buffer into a given userspacebuffer.
oa_hw_tail_readread the OA tail pointer register
In particular this enables us to share all the fiddly code forhandling the OA unit tail pointer race that affects multiplegenerations.
- int
i915_oa_stream_init(structi915_perf_stream * stream, struct drm_i915_perf_open_param * param, structperf_open_properties * props)¶ validate combined props for OA stream and init
Parameters
structi915_perf_stream*stream- An i915 perf stream
structdrm_i915_perf_open_param*param- The open parameters passed toDRM_I915_PERF_OPEN
structperf_open_properties*props- The property state that configures stream (individually validated)
Description
Whileread_properties_unlocked() validates properties in isolation itdoesn’t ensure that the combination necessarily makes sense.
At this point it has been determined that userspace wants a stream ofOA metrics, but still we need to further validate the combinedproperties are OK.
If the configuration makes sense then we can allocate memory fora circular OA buffer and apply the requested metric set configuration.
Return
zero on success or a negative error code.
- int
i915_oa_read(structi915_perf_stream * stream, char __user * buf, size_t count, size_t * offset)¶ just calls through to
i915_oa_ops->read
Parameters
structi915_perf_stream*stream- An i915-perf stream opened for OA metrics
char__user*buf- destination buffer given by userspace
size_tcount- the number of bytes userspace wants to read
size_t*offset- (inout): the current position for writing intobuf
Description
Updatesoffset according to the number of bytes successfully copied intothe userspace buffer.
Return
zero on success or a negative error code
- void
i915_oa_stream_enable(structi915_perf_stream * stream)¶ handleI915_PERF_IOCTL_ENABLE for OA stream
Parameters
structi915_perf_stream*stream- An i915 perf stream opened for OA metrics
Description
[Re]enables hardware periodic sampling according to the period configuredwhen opening the stream. This also starts a hrtimer that will periodicallycheck for data in the circular OA buffer for notifying userspace (e.g.during a read() or poll()).
- void
i915_oa_stream_disable(structi915_perf_stream * stream)¶ handleI915_PERF_IOCTL_DISABLE for OA stream
Parameters
structi915_perf_stream*stream- An i915 perf stream opened for OA metrics
Description
Stops the OA unit from periodically writing counter reports into thecircular OA buffer. This also stops the hrtimer that periodically checks fordata in the circular OA buffer, for notifying userspace.
- int
i915_oa_wait_unlocked(structi915_perf_stream * stream)¶ handles blocking IO until OA data available
Parameters
structi915_perf_stream*stream- An i915-perf stream opened for OA metrics
Description
Called when userspace tries to read() from a blocking stream FD openedfor OA metrics. It waits until the hrtimer callback finds a non-emptyOA buffer and wakes us.
Note
it’s acceptable to have this return with some false positivessince any subsequent read handling will return -EAGAIN if there isn’treally data ready for userspace yet.
Return
zero on success or a negative error code
- void
i915_oa_poll_wait(structi915_perf_stream * stream, struct file * file, poll_table * wait)¶ call poll_wait() for an OA stream poll()
Parameters
structi915_perf_stream*stream- An i915-perf stream opened for OA metrics
structfile*file- An i915 perf stream file
poll_table*wait- poll() state table
Description
For handling userspace polling on an i915 perf stream opened for OA metrics,this starts a poll_wait with the wait queue that our hrtimer callback wakeswhen it sees data ready to read in the circular OA buffer.
All i915 Perf Internals¶
This section simply includes all currently documented i915 perf internals, inno particular order, but may include some more minor utilities or platformspecific details than found in the more high-level sections.
- struct
perf_open_properties¶ for validated properties given to open a stream
Definition
struct perf_open_properties { u32 sample_flags; u64 single_context:1; u64 hold_preemption:1; u64 ctx_handle; int metrics_set; int oa_format; bool oa_periodic; int oa_period_exponent; struct intel_engine_cs *engine; bool has_sseu; struct intel_sseu sseu; u64 poll_oa_period;};Members
sample_flags- DRM_I915_PERF_PROP_SAMPLE_* properties are tracked as flags
single_context- Whether a single or all gpu contexts should be monitored
hold_preemption- Whether the preemption is disabled for the filteredcontext
ctx_handle- A gem ctx handle for use withsingle_context
metrics_set- An ID for an OA unit metric set advertised via sysfs
oa_format- An OA unit HW report format
oa_periodic- Whether to enable periodic OA unit sampling
oa_period_exponent- The OA unit sampling period is derived from this
engine- The engine (typically rcs0) being monitored by the OA unit
has_sseu- Whethersseu was specified by userspace
sseu- internal SSEU configuration computed either from the userspacespecified configuration in the opening parameters or a default value(see get_default_sseu_config())
poll_oa_period- The period in nanoseconds at which the CPU will check for OAdata availability
Description
Asread_properties_unlocked() enumerates and validates the properties givento open a stream of metrics the configuration is built up in the structurewhich starts out zero initialized.
- bool
oa_buffer_check_unlocked(structi915_perf_stream * stream)¶ check for data and update tail ptr state
Parameters
structi915_perf_stream*stream- i915 stream instance
Description
This is either called via fops (for blocking reads in user ctx) or the pollcheck hrtimer (atomic ctx) to check the OA buffer tail pointer and checkif there is data available for userspace to read.
This function is central to providing a workaround for the OA unit tailpointer having a race with respect to what data is visible to the CPU.It is responsible for reading tail pointers from the hardware and givingthe pointers time to ‘age’ before they are made available for reading.(See description of OA_TAIL_MARGIN_NSEC above for further details.)
Besides returning true when there is data available to read() this functionalso updates the tail, aging_tail and aging_timestamp in the oa_bufferobject.
Note
It’s safe to read OA config state here unlocked, assuming that this isonly called while the stream is enabled, while the global OA configurationcan’t be modified.
Return
true if the OA buffer contains data, elsefalse
- int
append_oa_status(structi915_perf_stream * stream, char __user * buf, size_t count, size_t * offset, enum drm_i915_perf_record_type type)¶ Appends a status record to a userspace read() buffer.
Parameters
structi915_perf_stream*stream- An i915-perf stream opened for OA metrics
char__user*buf- destination buffer given by userspace
size_tcount- the number of bytes userspace wants to read
size_t*offset- (inout): the current position for writing intobuf
enumdrm_i915_perf_record_typetype- The kind of status to report to userspace
Description
Writes a status record (such asDRM_I915_PERF_RECORD_OA_REPORT_LOST)into the userspace read() buffer.
Thebufoffset will only be updated on success.
Return
0 on success, negative error code on failure.
- int
append_oa_sample(structi915_perf_stream * stream, char __user * buf, size_t count, size_t * offset, const u8 * report)¶ Copies single OA report into userspace read() buffer.
Parameters
structi915_perf_stream*stream- An i915-perf stream opened for OA metrics
char__user*buf- destination buffer given by userspace
size_tcount- the number of bytes userspace wants to read
size_t*offset- (inout): the current position for writing intobuf
constu8*report- A single OA report to (optionally) include as part of the sample
Description
The contents of a sample are configured throughDRM_I915_PERF_PROP_SAMPLE_*properties when opening a stream, tracked asstream->sample_flags. Thisfunction copies the requested components of a single sample to the givenread()buf.
Thebufoffset will only be updated on success.
Return
0 on success, negative error code on failure.
- int
gen8_append_oa_reports(structi915_perf_stream * stream, char __user * buf, size_t count, size_t * offset)¶
Parameters
structi915_perf_stream*stream- An i915-perf stream opened for OA metrics
char__user*buf- destination buffer given by userspace
size_tcount- the number of bytes userspace wants to read
size_t*offset- (inout): the current position for writing intobuf
Description
Notably any error condition resulting in a short read (-ENOSPC or-EFAULT) will be returned even though one or more records mayhave been successfully copied. In this case it’s up to the callerto decide if the error should be squashed before returning touserspace.
Note
reports are consumed from the head, and appended to thetail, so the tail chases the head?… If you think that’s madand back-to-front you’re not alone, but this follows theGen PRM naming convention.
Return
0 on success, negative error code on failure.
- int
gen8_oa_read(structi915_perf_stream * stream, char __user * buf, size_t count, size_t * offset)¶ copy status records then buffered OA reports
Parameters
structi915_perf_stream*stream- An i915-perf stream opened for OA metrics
char__user*buf- destination buffer given by userspace
size_tcount- the number of bytes userspace wants to read
size_t*offset- (inout): the current position for writing intobuf
Description
Checks OA unit status registers and if necessary appends correspondingstatus records for userspace (such as for a buffer full condition) and theninitiate appending any buffered OA reports.
Updatesoffset according to the number of bytes successfully copied intothe userspace buffer.
NB: some data may be successfully copied to the userspace buffereven if an error is returned, and this is reflected in theupdatedoffset.
Return
zero on success or a negative error code
- int
gen7_append_oa_reports(structi915_perf_stream * stream, char __user * buf, size_t count, size_t * offset)¶
Parameters
structi915_perf_stream*stream- An i915-perf stream opened for OA metrics
char__user*buf- destination buffer given by userspace
size_tcount- the number of bytes userspace wants to read
size_t*offset- (inout): the current position for writing intobuf
Description
Notably any error condition resulting in a short read (-ENOSPC or-EFAULT) will be returned even though one or more records mayhave been successfully copied. In this case it’s up to the callerto decide if the error should be squashed before returning touserspace.
Note
reports are consumed from the head, and appended to thetail, so the tail chases the head?… If you think that’s madand back-to-front you’re not alone, but this follows theGen PRM naming convention.
Return
0 on success, negative error code on failure.
- int
gen7_oa_read(structi915_perf_stream * stream, char __user * buf, size_t count, size_t * offset)¶ copy status records then buffered OA reports
Parameters
structi915_perf_stream*stream- An i915-perf stream opened for OA metrics
char__user*buf- destination buffer given by userspace
size_tcount- the number of bytes userspace wants to read
size_t*offset- (inout): the current position for writing intobuf
Description
Checks Gen 7 specific OA unit status registers and if necessary appendscorresponding status records for userspace (such as for a buffer fullcondition) and then initiate appending any buffered OA reports.
Updatesoffset according to the number of bytes successfully copied intothe userspace buffer.
Return
zero on success or a negative error code
- int
i915_oa_wait_unlocked(structi915_perf_stream * stream) handles blocking IO until OA data available
Parameters
structi915_perf_stream*stream- An i915-perf stream opened for OA metrics
Description
Called when userspace tries to read() from a blocking stream FD openedfor OA metrics. It waits until the hrtimer callback finds a non-emptyOA buffer and wakes us.
Note
it’s acceptable to have this return with some false positivessince any subsequent read handling will return -EAGAIN if there isn’treally data ready for userspace yet.
Return
zero on success or a negative error code
- void
i915_oa_poll_wait(structi915_perf_stream * stream, struct file * file, poll_table * wait) call poll_wait() for an OA stream poll()
Parameters
structi915_perf_stream*stream- An i915-perf stream opened for OA metrics
structfile*file- An i915 perf stream file
poll_table*wait- poll() state table
Description
For handling userspace polling on an i915 perf stream opened for OA metrics,this starts a poll_wait with the wait queue that our hrtimer callback wakeswhen it sees data ready to read in the circular OA buffer.
- int
i915_oa_read(structi915_perf_stream * stream, char __user * buf, size_t count, size_t * offset) just calls through to
i915_oa_ops->read
Parameters
structi915_perf_stream*stream- An i915-perf stream opened for OA metrics
char__user*buf- destination buffer given by userspace
size_tcount- the number of bytes userspace wants to read
size_t*offset- (inout): the current position for writing intobuf
Description
Updatesoffset according to the number of bytes successfully copied intothe userspace buffer.
Return
zero on success or a negative error code
- int
oa_get_render_ctx_id(structi915_perf_stream * stream)¶ determine and hold ctx hw id
Parameters
structi915_perf_stream*stream- An i915-perf stream opened for OA metrics
Description
Determine the render context hw id, and ensure it remains fixed for thelifetime of the stream. This ensures that we don’t have to worry aboutupdating the context ID in OACONTROL on the fly.
Return
zero on success or a negative error code
- void
oa_put_render_ctx_id(structi915_perf_stream * stream)¶ counterpart to oa_get_render_ctx_id releases hold
Parameters
structi915_perf_stream*stream- An i915-perf stream opened for OA metrics
Description
In case anything needed doing to ensure the context HW ID would remain validfor the lifetime of the stream, then that can be undone here.
- void
i915_oa_stream_enable(structi915_perf_stream * stream) handleI915_PERF_IOCTL_ENABLE for OA stream
Parameters
structi915_perf_stream*stream- An i915 perf stream opened for OA metrics
Description
[Re]enables hardware periodic sampling according to the period configuredwhen opening the stream. This also starts a hrtimer that will periodicallycheck for data in the circular OA buffer for notifying userspace (e.g.during a read() or poll()).
- void
i915_oa_stream_disable(structi915_perf_stream * stream) handleI915_PERF_IOCTL_DISABLE for OA stream
Parameters
structi915_perf_stream*stream- An i915 perf stream opened for OA metrics
Description
Stops the OA unit from periodically writing counter reports into thecircular OA buffer. This also stops the hrtimer that periodically checks fordata in the circular OA buffer, for notifying userspace.
- int
i915_oa_stream_init(structi915_perf_stream * stream, struct drm_i915_perf_open_param * param, structperf_open_properties * props) validate combined props for OA stream and init
Parameters
structi915_perf_stream*stream- An i915 perf stream
structdrm_i915_perf_open_param*param- The open parameters passed toDRM_I915_PERF_OPEN
structperf_open_properties*props- The property state that configures stream (individually validated)
Description
Whileread_properties_unlocked() validates properties in isolation itdoesn’t ensure that the combination necessarily makes sense.
At this point it has been determined that userspace wants a stream ofOA metrics, but still we need to further validate the combinedproperties are OK.
If the configuration makes sense then we can allocate memory fora circular OA buffer and apply the requested metric set configuration.
Return
zero on success or a negative error code.
- ssize_t
i915_perf_read(struct file * file, char __user * buf, size_t count, loff_t * ppos) handles read() FOP for i915 perf stream FDs
Parameters
structfile*file- An i915 perf stream file
char__user*buf- destination buffer given by userspace
size_tcount- the number of bytes userspace wants to read
loff_t*ppos- (inout) file seek position (unused)
Description
The entry point for handling a read() on a stream file descriptor fromuserspace. Most of the work is left to the i915_perf_read_locked() andi915_perf_stream_ops->read but to save having stream implementations (ofwhich we might have multiple later) we handle blocking read here.
We can also consistently treat trying to read from a disabled streamas an IO error so implementations can assume the stream is enabledwhile reading.
Return
The number of bytes copied or a negative error code on failure.
- __poll_t
i915_perf_poll_locked(structi915_perf_stream * stream, struct file * file, poll_table * wait) poll_wait() with a suitable wait queue for stream
Parameters
structi915_perf_stream*stream- An i915 perf stream
structfile*file- An i915 perf stream file
poll_table*wait- poll() state table
Description
For handling userspace polling on an i915 perf stream, this calls through toi915_perf_stream_ops->poll_wait to call poll_wait() with a wait queue thatwill be woken for new stream data.
Note
Theperf->lock mutex has been taken to serializewith any non-file-operation driver hooks.
Return
any poll events that are ready without sleeping
- __poll_t
i915_perf_poll(struct file * file, poll_table * wait) call poll_wait() with a suitable wait queue for stream
Parameters
structfile*file- An i915 perf stream file
poll_table*wait- poll() state table
Description
For handling userspace polling on an i915 perf stream, this ensurespoll_wait() gets called with a wait queue that will be woken for new streamdata.
Note
Implementation deferred toi915_perf_poll_locked()
Return
any poll events that are ready without sleeping
- void
i915_perf_enable_locked(structi915_perf_stream * stream) handleI915_PERF_IOCTL_ENABLE ioctl
Parameters
structi915_perf_stream*stream- A disabled i915 perf stream
Description
[Re]enables the associated capture of data for this stream.
If a stream was previously enabled then there’s currently no intentionto provide userspace any guarantee about the preservation of previouslybuffered data.
- void
i915_perf_disable_locked(structi915_perf_stream * stream) handleI915_PERF_IOCTL_DISABLE ioctl
Parameters
structi915_perf_stream*stream- An enabled i915 perf stream
Description
Disables the associated capture of data for this stream.
The intention is that disabling an re-enabling a stream will ideally becheaper than destroying and re-opening a stream with the same configuration,though there are no formal guarantees about what state or buffered datamust be retained between disabling and re-enabling a stream.
Note
while a stream is disabled it’s considered an error for userspaceto attempt to read from the stream (-EIO).
- long
i915_perf_ioctl_locked(structi915_perf_stream * stream, unsigned int cmd, unsigned long arg)¶ support ioctl() usage with i915 perf stream FDs
Parameters
structi915_perf_stream*stream- An i915 perf stream
unsignedintcmd- the ioctl request
unsignedlongarg- the ioctl data
Note
Theperf->lock mutex has been taken to serializewith any non-file-operation driver hooks.
Return
zero on success or a negative error code. Returns -EINVAL foran unknown ioctl request.
- long
i915_perf_ioctl(struct file * file, unsigned int cmd, unsigned long arg) support ioctl() usage with i915 perf stream FDs
Parameters
structfile*file- An i915 perf stream file
unsignedintcmd- the ioctl request
unsignedlongarg- the ioctl data
Description
Implementation deferred toi915_perf_ioctl_locked().
Return
zero on success or a negative error code. Returns -EINVAL foran unknown ioctl request.
- void
i915_perf_destroy_locked(structi915_perf_stream * stream) destroy an i915 perf stream
Parameters
structi915_perf_stream*stream- An i915 perf stream
Description
Frees all resources associated with the given i915 perfstream, disablingany associated data capture in the process.
Note
Theperf->lock mutex has been taken to serializewith any non-file-operation driver hooks.
- int
i915_perf_release(struct inode * inode, struct file * file) handles userspace close() of a stream file
Parameters
structinode*inode- anonymous inode associated with file
structfile*file- An i915 perf stream file
Description
Cleans up any resources associated with an open i915 perf stream file.
NB: close() can’t really fail from the userspace point of view.
Return
zero on success or a negative error code.
- int
i915_perf_open_ioctl_locked(struct i915_perf * perf, struct drm_i915_perf_open_param * param, structperf_open_properties * props, structdrm_file * file) DRM ioctl() for userspace to open a stream FD
Parameters
structi915_perf*perf- i915 perf instance
structdrm_i915_perf_open_param*param- The open parameters passed to ‘DRM_I915_PERF_OPEN`
structperf_open_properties*props- individually validated u64 property value pairs
structdrm_file*file- drm file
Description
See i915_perf_ioctl_open() for interface details.
Implements further stream config validation and stream initialization onbehalf ofi915_perf_open_ioctl() with theperf->lock mutextaken to serialize with any non-file-operation driver hooks.
In the case where userspace is interested in OA unit metrics then furtherconfig validation and stream initialization details will be handled byi915_oa_stream_init(). The code here should only validate config state thatwill be relevant to all stream types / backends.
Note
at this point theprops have only been validated in isolation andit’s still necessary to validate that the combination of properties makessense.
Return
zero on success or a negative error code.
- int
read_properties_unlocked(struct i915_perf * perf, u64 __user * uprops, u32 n_props, structperf_open_properties * props) validate + copy userspace stream open properties
Parameters
structi915_perf*perf- i915 perf instance
u64__user*uprops- The array of u64 key value pairs given by userspace
u32n_props- The number of key value pairs expected inuprops
structperf_open_properties*props- The stream configuration built up while validating properties
Description
Note this function only validates properties in isolation it doesn’tvalidate that the combination of properties makes sense or that allproperties necessary for a particular kind of stream have been set.
Note that there currently aren’t any ordering requirements for properties sowe shouldn’t validate or assume anything about ordering here. This doesn’trule out defining new properties with ordering requirements in the future.
- int
i915_perf_open_ioctl(structdrm_device * dev, void * data, structdrm_file * file) DRM ioctl() for userspace to open a stream FD
Parameters
structdrm_device*dev- drm device
void*data- ioctl data copied from userspace (unvalidated)
structdrm_file*file- drm file
Description
Validates the stream open parameters given by userspace including flagsand an array of u64 key, value pair properties.
Very little is assumed up front about the nature of the stream beingopened (for instance we don’t assume it’s for periodic OA unit metrics). Ani915-perf stream is expected to be a suitable interface for other forms ofbuffered data written by the GPU besides periodic OA metrics.
Note we copy the properties from userspace outside of the i915 perfmutex to avoid an awkward lockdep with mmap_lock.
Most of the implementation details are handled byi915_perf_open_ioctl_locked() after taking theperf->lockmutex for serializing with any non-file-operation driver hooks.
Return
A newly opened i915 Perf stream file descriptor or negativeerror code on failure.
- void
i915_perf_register(struct drm_i915_private * i915) exposes i915-perf to userspace
Parameters
structdrm_i915_private*i915- i915 device instance
Description
In particular OA metric sets are advertised under a sysfs metrics/directory allowing userspace to enumerate valid IDs that can beused to open an i915-perf stream.
- void
i915_perf_unregister(struct drm_i915_private * i915) hide i915-perf from userspace
Parameters
structdrm_i915_private*i915- i915 device instance
Description
i915-perf state cleanup is split up into an ‘unregister’ and‘deinit’ phase where the interface is first hidden fromuserspace byi915_perf_unregister() before cleaning upremaining state ini915_perf_fini().
- int
i915_perf_add_config_ioctl(structdrm_device * dev, void * data, structdrm_file * file) DRM ioctl() for userspace to add a new OA config
Parameters
structdrm_device*dev- drm device
void*data- ioctl data (pointer to struct drm_i915_perf_oa_config) copied fromuserspace (unvalidated)
structdrm_file*file- drm file
Description
Validates the submitted OA register to be saved into a new OA config thatcan then be used for programming the OA unit and its NOA network.
Return
A new allocated config number to be used with the perf open ioctlor a negative error code on failure.
- int
i915_perf_remove_config_ioctl(structdrm_device * dev, void * data, structdrm_file * file) DRM ioctl() for userspace to remove an OA config
Parameters
structdrm_device*dev- drm device
void*data- ioctl data (pointer to u64 integer) copied from userspace
structdrm_file*file- drm file
Description
Configs can be removed while being used, the will stop appearing in sysfsand their content will be freed when the stream using the config is closed.
Return
0 on success or a negative error code on failure.
- void
i915_perf_init(struct drm_i915_private * i915) initialize i915-perf state on module bind
Parameters
structdrm_i915_private*i915- i915 device instance
Description
Initializes i915-perf state without exposing anything to userspace.
Note
i915-perf initialization is split into an ‘init’ and ‘register’phase with thei915_perf_register() exposing state to userspace.
- void
i915_perf_fini(struct drm_i915_private * i915) Counter part to
i915_perf_init()
Parameters
structdrm_i915_private*i915- i915 device instance
- int
i915_perf_ioctl_version(void)¶ Version of the i915-perf subsystem
Parameters
void- no arguments
Description
This version number is used by userspace to detect available features.
Style¶
The drm/i915 driver codebase has some style rules in addition to (and, in somecases, deviating from) the kernel coding style.
Register macro definition style¶
The style guide fori915_reg.h.
Follow the style described here for new macros, and while changing existingmacros. Donot mass change existing definitions just to update the style.
File Layout¶
Keep helper macros near the top. For example, _PIPE() and friends.
Prefix macros that generally should not be used outside of this file withunderscore ‘_’. For example, _PIPE() and friends, single instances ofregisters that are defined solely for the use by function-like macros.
Avoid using the underscore prefixed macros outside of this file. There areexceptions, but keep them to a minimum.
There are two basic types of register definitions: Single registers andregister groups. Register groups are registers which have two or moreinstances, for example one per pipe, port, transcoder, etc. Register groupsshould be defined using function-like macros.
For single registers, define the register offset first, followed by registercontents.
For register groups, define the register instance offsets first, prefixedwith underscore, followed by a function-like macro choosing the rightinstance based on the parameter, followed by register contents.
Define the register contents (i.e. bit and bit field macros) from mostsignificant to least significant bit. Indent the register content macrosusing two extra spaces between#define and the macro name.
Define bit fields usingREG_GENMASK(h,l). Define bit field contentsusingREG_FIELD_PREP(mask,value). This will define the values alreadyshifted in place, so they can be directly OR’d together. For convenience,function-like macros may be used to define bit fields, but do note that themacros may be needed to read as well as write the register contents.
Define bits usingREG_BIT(N). Donot add_BIT suffix to the name.
Group the register and its contents together without blank lines, separatefrom other registers and their contents with one blank line.
Indent macro values from macro names using TABs. Align values vertically. Usebraces in macro values as needed to avoid unintended precedence after macrosubstitution. Use spaces in macro values according to kernel codingstyle. Use lower case in hexadecimal values.
Naming¶
Try to name registers according to the specs. If the register name changes inthe specs from platform to another, stick to the original name.
Try to re-use existing register macro definitions. Only add new macros fornew register offsets, or when the register contents have changed enough towarrant a full redefinition.
When a register macro changes for a new platform, prefix the new macro usingthe platform acronym or generation. For example,SKL_ orGEN8_. Theprefix signifies the start platform/generation using the register.
When a bit (field) macro changes or gets added for a new platform, whileretaining the existing register macro, add a platform acronym or generationsuffix to the name. For example,_SKL or_GEN8.
Examples¶
(Note that the values in the example are indented using spaces instead ofTABs to avoid misalignment in generated documentation. Use TABs in thedefinitions.):
#define _FOO_A 0xf000#define _FOO_B 0xf001#define FOO(pipe) _MMIO_PIPE(pipe, _FOO_A, _FOO_B)#define FOO_ENABLE REG_BIT(31)#define FOO_MODE_MASK REG_GENMASK(19, 16)#define FOO_MODE_BAR REG_FIELD_PREP(FOO_MODE_MASK, 0)#define FOO_MODE_BAZ REG_FIELD_PREP(FOO_MODE_MASK, 1)#define FOO_MODE_QUX_SNB REG_FIELD_PREP(FOO_MODE_MASK, 2)#define BAR _MMIO(0xb000)#define GEN8_BAR _MMIO(0xb888)