Device Initialization (devinit)

The devinit process is complex and subject to change. This document provides a high-leveloverview using the Ampere GPU family as an example. The goal is to provide a conceptualoverview of the process to aid in understanding the corresponding kernel code.

Device initialization (devinit) is a crucial sequence of register read/write operationsthat occur after a GPU reset. The devinit sequence is essential for properly configuringthe GPU hardware before it can be used.

The devinit engine is an interpreter program that typically runs on the PMU (Power ManagementUnit) microcontroller of the GPU. This interpreter executes a “script” of initializationcommands. The devinit engine itself is part of the VBIOS ROM in the same ROM image as theFWSEC (Firmware Security) image (seeFWSEC (Firmware Security) andVBIOS) and it runs before thenova-core driver is even loaded. On an Ampere GPU, the devinit ucode is separate from theFWSEC ucode. It is launched by FWSEC, which runs on the GSP in ‘heavy-secure’ mode, whiledevinit runs on the PMU in ‘light-secure’ mode.

Key Functions of devinit

devinit performs several critical tasks:

  1. Programming VRAM memory controller timings

  2. Power sequencing

  3. Clock and PLL (Phase-Locked Loop) configuration

  4. Thermal management

Low-level Firmware Initialization Flow

Upon reset, several microcontrollers on the GPU (such as PMU, SEC2, GSP, etc.) run GPUfirmware (gfw) code to set up the GPU and its core parameters. Most of the GPU isconsidered unusable until this initialization process completes.

These low-level GPU firmware components are typically:

  1. Located in the VBIOS ROM in the same ROM partition (seeVBIOS andFWSEC (Firmware Security)).

  2. Executed in sequence on different microcontrollers:

  • The devinit engine typically but not necessarily runs on the PMU.

  • On an Ampere GPU, the FWSEC typically runs on the GSP (GPU System Processor) inheavy-secure mode.

Before the driver can proceed with further initialization, it must wait for a signalindicating that core initialization is complete (known as GFW_BOOT). This signal isasserted by the FWSEC running on the GSP in heavy-secure mode.

Runtime Considerations

It’s important to note that the devinit sequence also needs to run during suspend/resumeoperations at runtime, not just during initial boot, as it is critical to power management.

Security and Access Control

The initialization process involves careful privilege management. For example, beforeaccessing certain completion status registers, the driver must check privilege levelmasks. Some registers are only accessible after secure firmware (FWSEC) lowers theprivilege level to allow CPU (LS/low-secure) access. This is the case, for example,when receiving the GFW_BOOT signal.