Booting AArch64 Linux

Author: Will Deacon <will.deacon@arm.com>

Date : 07 September 2012

This document is based on the ARM booting document by Russell King andis relevant to all public releases of the AArch64 Linux kernel.

The AArch64 exception model is made up of a number of exception levels(EL0 - EL3), with EL0 and EL1 having a secure and a non-securecounterpart. EL2 is the hypervisor level and exists only in non-securemode. EL3 is the highest priority level and exists only in secure mode.

For the purposes of this document, we will use the termboot loadersimply to define all software that executes on the CPU(s) before controlis passed to the Linux kernel. This may include secure monitor andhypervisor code, or it may just be a handful of instructions forpreparing a minimal boot environment.

Essentially, the boot loader should provide (as a minimum) thefollowing:

  1. Setup and initialise the RAM
  2. Setup the device tree
  3. Decompress the kernel image
  4. Call the kernel image

1. Setup and initialise RAM

Requirement: MANDATORY

The boot loader is expected to find and initialise all RAM that thekernel will use for volatile data storage in the system. It performsthis in a machine dependent manner. (It may use internal algorithmsto automatically locate and size all RAM, or it may use knowledge ofthe RAM in the machine, or any other method the boot loader designersees fit.)

2. Setup the device tree

Requirement: MANDATORY

The device tree blob (dtb) must be placed on an 8-byte boundary and mustnot exceed 2 megabytes in size. Since the dtb will be mapped cacheableusing blocks of up to 2 megabytes in size, it must not be placed withinany 2M region which must be mapped with any specific attributes.

NOTE: versions prior to v4.2 also require that the DTB be placed withinthe 512 MB region starting at text_offset bytes below the kernel Image.

3. Decompress the kernel image

Requirement: OPTIONAL

The AArch64 kernel does not currently provide a decompressor andtherefore requires decompression (gzip etc.) to be performed by the bootloader if a compressed Image target (e.g. Image.gz) is used. Forbootloaders that do not implement this requirement, the uncompressedImage target is available instead.

4. Call the kernel image

Requirement: MANDATORY

The decompressed kernel image contains a 64-byte header as follows:

u32 code0;                    /* Executable code */u32 code1;                    /* Executable code */u64 text_offset;              /* Image load offset, little endian */u64 image_size;               /* Effective Image size, little endian */u64 flags;                    /* kernel flags, little endian */u64 res2      = 0;            /* reserved */u64 res3      = 0;            /* reserved */u64 res4      = 0;            /* reserved */u32 magic     = 0x644d5241;   /* Magic number, little endian, "ARM\x64" */u32 res5;                     /* reserved (used for PE COFF offset) */

Header notes:

  • As of v3.17, all fields are little endian unless stated otherwise.

  • code0/code1 are responsible for branching to stext.

  • when booting through EFI, code0/code1 are initially skipped.res5 is an offset to the PE header and the PE header has the EFIentry point (efi_stub_entry). When the stub has done its work, itjumps to code0 to resume the normal boot process.

  • Prior to v3.17, the endianness of text_offset was not specified. Inthese cases image_size is zero and text_offset is 0x80000 in theendianness of the kernel. Where image_size is non-zero image_size islittle-endian and must be respected. Where image_size is zero,text_offset can be assumed to be 0x80000.

  • The flags field (introduced in v3.17) is a little-endian 64-bit fieldcomposed as follows:

    Bit 0Kernel endianness. 1 if BE, 0 if LE.
    Bit 1-2

    Kernel Page size.

    • 0 - Unspecified.
    • 1 - 4K
    • 2 - 16K
    • 3 - 64K
    Bit 3

    Kernel physical placement

    0
    2MB aligned base should be as close as possibleto the base of DRAM, since memory below it is notaccessible via the linear mapping
    1
    2MB aligned base may be anywhere in physicalmemory
    Bits 4-63Reserved.
  • When image_size is zero, a bootloader should attempt to keep as muchmemory as possible free for use by the kernel immediately after theend of the kernel image. The amount of space required will varydepending on selected features, and is effectively unbound.

The Image must be placed text_offset bytes from a 2MB aligned baseaddress anywhere in usable system RAM and called there. The regionbetween the 2 MB aligned base address and the start of the image has nospecial significance to the kernel, and may be used for other purposes.At least image_size bytes from the start of the image must be free foruse by the kernel.NOTE: versions prior to v4.6 cannot make use of memory below thephysical offset of the Image so it is recommended that the Image beplaced as close as possible to the start of system RAM.

If an initrd/initramfs is passed to the kernel at boot, it must resideentirely within a 1 GB aligned physical memory window of up to 32 GB insize that fully covers the kernel Image as well.

Any memory described to the kernel (even that below the start of theimage) which is not marked as reserved from the kernel (e.g., with amemreserve region in the device tree) will be considered as available tothe kernel.

Before jumping into the kernel, the following conditions must be met:

  • Quiesce all DMA capable devices so that memory does not getcorrupted by bogus network packets or disk data. This will saveyou many hours of debug.

  • Primary CPU general-purpose register settings:

    • x0 = physical address of device tree blob (dtb) in system RAM.
    • x1 = 0 (reserved for future use)
    • x2 = 0 (reserved for future use)
    • x3 = 0 (reserved for future use)
  • CPU mode

    All forms of interrupts must be masked in PSTATE.DAIF (Debug, SError,IRQ and FIQ).The CPU must be in either EL2 (RECOMMENDED in order to have access tothe virtualisation extensions) or non-secure EL1.

  • Caches, MMUs

    The MMU must be off.

    The instruction cache may be on or off, and must not hold any staleentries corresponding to the loaded kernel image.

    The address range corresponding to the loaded kernel image must becleaned to the PoC. In the presence of a system cache or othercoherent masters with caches enabled, this will typically requirecache maintenance by VA rather than set/way operations.System caches which respect the architected cache maintenance by VAoperations must be configured and may be enabled.System caches which do not respect architected cache maintenance by VAoperations (not recommended) must be configured and disabled.

  • Architected timers

    CNTFRQ must be programmed with the timer frequency and CNTVOFF mustbe programmed with a consistent value on all CPUs. If entering thekernel at EL1, CNTHCTL_EL2 must have EL1PCTEN (bit 0) set whereavailable.

  • Coherency

    All CPUs to be booted by the kernel must be part of the same coherencydomain on entry to the kernel. This may require IMPLEMENTATION DEFINEDinitialisation to enable the receiving of maintenance operations oneach CPU.

  • System registers

    All writable architected system registers at the exception level wherethe kernel image will be entered must be initialised by software at ahigher exception level to prevent execution in an UNKNOWN state.

    • SCR_EL3.FIQ must have the same value across all CPUs the kernel isexecuting on.
    • The value of SCR_EL3.FIQ must be the same as the one present at boottime whenever the kernel is executing.

    For systems with a GICv3 interrupt controller to be used in v3 mode:- If EL3 is present:

    • ICC_SRE_EL3.Enable (bit 3) must be initialiased to 0b1.
    • ICC_SRE_EL3.SRE (bit 0) must be initialised to 0b1.
    • ICC_CTLR_EL3.PMHE (bit 6) must be set to the same value acrossall CPUs the kernel is executing on, and must stay constantfor the lifetime of the kernel.
    • If the kernel is entered at EL1:

      • ICC.SRE_EL2.Enable (bit 3) must be initialised to 0b1
      • ICC_SRE_EL2.SRE (bit 0) must be initialised to 0b1.
    • The DT or ACPI tables must describe a GICv3 interrupt controller.

    For systems with a GICv3 interrupt controller to be used incompatibility (v2) mode:

    • If EL3 is present:

      ICC_SRE_EL3.SRE (bit 0) must be initialised to 0b0.

    • If the kernel is entered at EL1:

      ICC_SRE_EL2.SRE (bit 0) must be initialised to 0b0.

    • The DT or ACPI tables must describe a GICv2 interrupt controller.

    For CPUs with pointer authentication functionality:

    • If EL3 is present:
      • SCR_EL3.APK (bit 16) must be initialised to 0b1
      • SCR_EL3.API (bit 17) must be initialised to 0b1
    • If the kernel is entered at EL1:
      • HCR_EL2.APK (bit 40) must be initialised to 0b1
      • HCR_EL2.API (bit 41) must be initialised to 0b1

    For CPUs with Activity Monitors Unit v1 (AMUv1) extension present:

    • If EL3 is present:
      • CPTR_EL3.TAM (bit 30) must be initialised to 0b0
      • CPTR_EL2.TAM (bit 30) must be initialised to 0b0
      • AMCNTENSET0_EL0 must be initialised to 0b1111
      • AMCNTENSET1_EL0 must be initialised to a platform specific valuehaving 0b1 set for the corresponding bit for each of the auxiliarycounters present.
    • If the kernel is entered at EL1:
      • AMCNTENSET0_EL0 must be initialised to 0b1111
      • AMCNTENSET1_EL0 must be initialised to a platform specific valuehaving 0b1 set for the corresponding bit for each of the auxiliarycounters present.

The requirements described above for CPU mode, caches, MMUs, architectedtimers, coherency and system registers apply to all CPUs. All CPUs mustenter the kernel in the same exception level.

The boot loader is expected to enter the kernel on each CPU in thefollowing manner:

  • The primary CPU must jump directly to the first instruction of thekernel image. The device tree blob passed by this CPU must containan ‘enable-method’ property for each cpu node. The supportedenable-methods are described below.

    It is expected that the bootloader will generate these device treeproperties and insert them into the blob prior to kernel entry.

  • CPUs with a “spin-table” enable-method must have a ‘cpu-release-addr’property in their cpu node. This property identifies anaturally-aligned 64-bit zero-initalised memory location.

    These CPUs should spin outside of the kernel in a reserved area ofmemory (communicated to the kernel by a /memreserve/ region in thedevice tree) polling their cpu-release-addr location, which must becontained in the reserved region. A wfe instruction may be insertedto reduce the overhead of the busy-loop and a sev will be issued bythe primary CPU. When a read of the location pointed to by thecpu-release-addr returns a non-zero value, the CPU must jump to thisvalue. The value will be written as a single 64-bit little-endianvalue, so CPUs must convert the read value to their native endiannessbefore jumping to it.

  • CPUs with a “psci” enable method should remain outside ofthe kernel (i.e. outside of the regions of memory described to thekernel in the memory node, or in a reserved area of memory describedto the kernel by a /memreserve/ region in the device tree). Thekernel will issue CPU_ON calls as described in ARM document number ARMDEN 0022A (“Power State Coordination Interface System Software on ARMprocessors”) to bring CPUs into the kernel.

    The device tree should contain a ‘psci’ node, as described inDocumentation/devicetree/bindings/arm/psci.yaml.

  • Secondary CPU general-purpose register settings

    • x0 = 0 (reserved for future use)
    • x1 = 0 (reserved for future use)
    • x2 = 0 (reserved for future use)
    • x3 = 0 (reserved for future use)