7.35.Raspberry Pi 3

TheRaspberry Pi 3 is an inexpensive single-board computer that contains fourArm Cortex-A53 cores.

The following instructions explain how to use this port of the TF-A with thedefault distribution ofRaspbian because that’s the distribution officiallysupported by the Raspberry Pi Foundation. At the moment of writing this, theofficially supported kernel is a AArch32 kernel. This doesn’t mean that thisport of TF-A can’t boot a AArch64 kernel. TheLinux tree fork maintained bythe Foundation can be compiled for AArch64 by following the steps inAArch64 kernel build instructions.

IMPORTANT NOTE: This port isn’t secure. All of the memory used is DRAM,which is available from both the Non-secure and Secure worlds. This portshouldn’t be considered more than a prototype to play with and implementelements like PSCI to support the Linux kernel.

7.35.1.Design

The SoC used by the Raspberry Pi 3 is the Broadcom BCM2837. It is a SoC with aVideoCore IV that acts as primary processor (and loads everything from the SDcard) and is located between all Arm cores and the DRAM. Check theRaspberry Pi3 documentation for more information.

This explains why it is possible to change the execution state (AArch64/AArch32)depending on a few files on the SD card. We only care about the cases in whichthe cores boot in AArch64 mode.

The rules are simple:

  • If a file calledkernel8.img is located on theboot partition of theSD card, it will load it and execute in EL2 in AArch64. Basically, it executesadefault AArch64 stub at address0x0 that jumps to the kernel.

  • If there is also a file calledarmstub8.bin, it will load it at address0x0 (instead of the default stub) and execute it in EL3 in AArch64. Allthe cores are powered on at the same time and start at address0x0.

This means that we can use the default AArch32 kernel provided in the officialRaspbian distribution by renaming it tokernel8.img, while TF-A andanything else we need is inarmstub8.bin. This way we can forget about thedefault bootstrap code. When using a AArch64 kernel, it is only needed to makesure that the name on the SD card iskernel8.img.

Ideally, we want to load the kernel and have all cores available, which meansthat we need to make the secondary cores work in the way the kernel expects, asexplained inSecondary cores. In practice, a small bootstrap is neededbetween TF-A and the kernel.

To get the most out of a AArch32 kernel, we want to boot it in Hypervisor modein AArch32. This means that BL33 can’t be in EL2 in AArch64 mode. Thearchitecture specifies that AArch32 Hypervisor mode isn’t present when AArch64is used for EL2. When using a AArch64 kernel, it should simply start in EL2.

7.35.1.1.Placement of images

The filearmstub8.bin contains BL1 and the FIP. It is needed to add paddingbetween them so that the addresses they are loaded to match the ones specifiedwhen compiling TF-A. This is done automatically by the build system.

The device tree block is loaded by the VideoCore loader from an appropriatefile, but we can specify the address it is loaded to inconfig.txt.

The filekernel8.img contains a kernel image that is loaded to the addressspecified inconfig.txt. TheLinux kernel tree has information about howa AArch32 Linux kernel image is loaded inDocumentation/arm/Booting:

ThezImagemayalsobeplacedinsystemRAMandcalledthere.Thekernelshouldbeplacedinthefirst128MiBofRAM.Itisrecommendedthatitisloadedabove32MiBinordertoavoidtheneedtorelocatepriortodecompression,whichwillmakethebootprocessslightlyfaster.

There are no similar restrictions for AArch64 kernels, as specified in the fileDocumentation/arm64/booting.txt.

This means that we need to avoid the first 128 MiB of RAM when placing theTF-A images (and specially the first 32 MiB, as they are directly used toplace the uncompressed AArch32 kernel image. This way, both AArch32 andAArch64 kernels can be placed at the same address.

In the end, the images look like the following diagram when placed in memory.All addresses are Physical Addresses from the point of view of the Arm cores.Again, note that this is all just part of the same DRAM that goes from0x00000000 to0x3F000000, it just has different names to simulate a realsecure platform!

0x00000000+-----------------+|ROM|BL10x00020000+-----------------+|FIP|0x00200000+-----------------+|||...|||0x01000000+-----------------+|DTB|(LoadedbytheVideoCore)+-----------------+|||...|||0x02000000+-----------------+|Kernel|(LoadedbytheVideoCore)+-----------------+|||...|||0x10000000+-----------------+|SecureSRAM|BL2,BL310x10100000+-----------------+|SecureDRAM|BL32(Securepayload)0x11000000+-----------------+|Non-secureDRAM|BL33+-----------------+|||...|||0x3F000000+-----------------+|I/O|0x40000000+-----------------+

The area between0x10000000 and0x11000000 has to be manually protectedso that the kernel doesn’t use it. The current port tries to modify the live DTBto add a memreserve region that reserves the previously mentioned area.

If this is not possible, the user may manually addmemmap=16M$256M to thecommand line passed to the kernel incmdline.txt. See theSetup SD cardinstructions to see how to do it. This system is strongly discouraged.

The last 16 MiB of DRAM can only be accessed by the VideoCore, that hasdifferent mappings than the Arm cores in which the I/O addresses don’t overlapthe DRAM. The memory reserved to be used by the VideoCore is always placed atthe end of the DRAM, so this space isn’t wasted.

Considering the 128 MiB allocated to the GPU and the 16 MiB allocated forTF-A, there are 880 MiB available for Linux.

7.35.1.2.Boot sequence

The boot sequence of TF-A is the usual one except when booting an AArch32kernel. In that case, BL33 is booted in AArch32 Hypervisor mode so that itcan jump to the kernel in the same mode and let it take over that privilegelevel. If BL33 was running in EL2 in AArch64 (as in the default bootflow ofTF-A) it could only jump to the kernel in AArch32 in Supervisor mode.

TheLinux kernel tree has instructions on how to jump to the Linux kernelinDocumentation/arm/Booting andDocumentation/arm64/booting.txt. Thebootstrap should take care of this.

This port support a direct boot of the Linux kernel from the firmware (as a BL33image). Alternatively, U-Boot or other bootloaders may be used.

7.35.1.3.Secondary cores

This port of the Trusted Firmware-A supportsPSCI_CPU_ON,PSCI_SYSTEM_RESET andPSCI_SYSTEM_OFF. The last one doesn’t really turnthe system off, it simply reboots it and asks the VideoCore firmware to keep itin a low power mode permanently.

The kernel used byRaspbian doesn’t have support for PSCI, so it is needed touse mailboxes to trap the secondary cores until they are ready to jump to thekernel. This mailbox is located at a different address in the AArch32 defaultkernel than in the AArch64 kernel.

Kernels with PSCI support can use the PSCI calls instead for a cleaner boot.

Also, this port of TF-A has another Trusted Mailbox in Shared BL RAM. Duringcold boot, all secondary cores wait in a loop until they are given given anaddress to jump to in this Mailbox (bl31_warm_entrypoint).

Once BL31 has finished and the primary core has jumped to the BL33 payload, ithas to callPSCI_CPU_ON to release the secondary CPUs from the wait loop.The payload then makes them wait in another waitloop listening from messagesfrom the kernel. When the primary CPU jumps into the kernel, it will send anaddress to the mailbox so that the secondary CPUs jump to it and are recognisedby the kernel.

7.35.2.Build Instructions

To boot a AArch64 kernel, only the AArch64 toolchain is required.

To boot a AArch32 kernel, both AArch64 and AArch32 toolchains are required. TheAArch32 toolchain is needed for the AArch32 bootstrap needed to load a 32-bitkernel.

The build system concatenates BL1 and the FIP so that the addresses match theones in the memory map. The resulting file isarmstub8.bin, located in thebuild folder (e.g.build/rpi3/debug/armstub8.bin). To know how to use thisfile, follow the instructions inSetup SD card.

The following build options are supported:

  • RPI3_BL33_IN_AARCH32: This port can load a AArch64 or AArch32 BL33 image.By default this option is 0, which means that TF-A will jump to BL33 in EL2in AArch64 mode. If set to 1, it will jump to BL33 in Hypervisor in AArch32mode.

  • PRELOADED_BL33_BASE: Used to specify the address of a BL33 binary that hasbeen preloaded by any other system than using the firmware.BL33 isn’tneeded in the build command line if this option is used. Specially usefulbecause the filekernel8.img can be loaded anywhere by modifying the fileconfig.txt. It doesn’t have to contain a kernel, it could have anyarbitrary payload.

  • RPI3_DIRECT_LINUX_BOOT: Disabled by default. Set to 1 to enable the directboot of the Linux kernel from the firmware. OptionRPI3_PRELOADED_DTB_BASEis mandatory when the direct Linux kernel boot is used. OptionsPRELOADED_BL33_BASE will most likely be needed as well because it isunlikely that the kernel image will fit in the space reserved for BL33 images.This option can be combined withRPI3_BL33_IN_AARCH32 in order to boot a32-bit kernel. The only thing this option does is to set the arguments inregisters x0-x3 or r0-r2 as expected by the kernel.

  • RPI3_PRELOADED_DTB_BASE: Auxiliary build option needed when usingRPI3_DIRECT_LINUX_BOOT=1. This option allows to specify the location of aDTB in memory.

  • RPI3_RUNTIME_UART: Indicates whether the UART should be used at runtimeor disabled.-1 (default) disables the runtime UART. Any other valueenables the default UART (currently UART1) for runtime messages.

  • RPI3_USE_UEFI_MAP: Set to 1 to build ATF with the altername memorymapping required for an UEFI firmware payload. These changes are neededto be able to run Windows on ARM64. This option, which is disabled bydefault, results in the following memory mappings:

0x00000000+-----------------+|ROM|BL10x00010000+-----------------+|DTB|(LoadedbytheVideoCore)0x00020000+-----------------+|FIP|0x00030000+-----------------+|||UEFIPAYLOAD|||0x00200000+-----------------+|SecureSRAM|BL2,BL310x00300000+-----------------+|SecureDRAM|BL32(Securepayload)0x00400000+-----------------+|||||Non-secureDRAM|BL33||||0x01000000+-----------------+|||...|||0x3F000000+-----------------+|I/O|

  • BL32: This port can load and run OP-TEE. The OP-TEE image is optional.Please use the code fromhere.Build the Trusted Firmware with optionBL32=tee-header_v2.binBL32_EXTRA1=tee-pager_v2.bin BL32_EXTRA2=tee-pageable_v2.binto put the binaries into the FIP.

    Warning

    If OP-TEE is used it may be needed to add the following options to theLinux command line so that the USB driver doesn’t use FIQs:dwc_otg.fiq_enable=0dwc_otg.fiq_fsm_enable=0dwc_otg.nak_holdoff=0.This will unfortunately reduce the performance of the USB driver. It isneeded when using Raspbian, for example.

  • TRUSTED_BOARD_BOOT: This port supports TBB. Set this option to 1 to enableit. In order to use TBB, you might want to setGENERATE_COT=1 to let thecontents of the FIP automatically signed by the build process. The ROT keywill be generated and output torot_key.pem in the build directory. It isable to set ROT_KEY to your own key in PEM format. Also in order to build,you need to clone mbed TLS fromhere.MBEDTLS_DIR must point at the mbed TLS source directory.

  • ENABLE_STACK_PROTECTOR: Disabled by default. It uses the hardware RNG ofthe board.

The following is not currently supported:

  • AArch32 for TF-A itself.

  • EL3_PAYLOAD_BASE: The reason is that you can already load anything to anyaddress by changing the filearmstub8.bin, so there’s no point in usingTF-A in this case.

7.35.2.1.Building the firmware for kernels that don’t support PSCI

This is the case for the 32-bit image of Raspbian, for example. 64-bit kernelsalways support PSCI, but they may not know that the system understands PSCI dueto an incorrect DTB file.

First, clone and compile the 32-bit version of theRaspberry Pi 3 TF-Abootstrap. Choose the one needed for the architecture of your kernel.

Then compile TF-A. For a 32-bit kernel, use the following command line:

CROSS_COMPILE=aarch64-linux-gnu-makePLAT=rpi3\RPI3_BL33_IN_AARCH32=1\BL33=../rpi3-arm-tf-bootstrap/aarch32/el2-bootstrap.bin

For a 64-bit kernel, use this other command line:

CROSS_COMPILE=aarch64-linux-gnu-makePLAT=rpi3\BL33=../rpi3-arm-tf-bootstrap/aarch64/el2-bootstrap.bin

However, enabling PSCI support in a 64-bit kernel is really easy. In therepositoryRaspberry Pi 3 TF-A bootstrap there is a patch that can be appliedto the Linux kernel tree maintained by the Raspberry Pi foundation. It modifesthe DTS to tell the kernel to use PSCI. Once this patch is applied, follow theinstructions inAArch64 kernel build instructions to get a working 64-bitkernel image and supporting files.

7.35.2.2.Building the firmware for kernels that support PSCI

For a 64-bit kernel:

CROSS_COMPILE=aarch64-linux-gnu-makePLAT=rpi3\PRELOADED_BL33_BASE=0x02000000\RPI3_PRELOADED_DTB_BASE=0x01000000\RPI3_DIRECT_LINUX_BOOT=1

For a 32-bit kernel:

CROSS_COMPILE=aarch64-linux-gnu-makePLAT=rpi3\PRELOADED_BL33_BASE=0x02000000\RPI3_PRELOADED_DTB_BASE=0x01000000\RPI3_DIRECT_LINUX_BOOT=1\RPI3_BL33_IN_AARCH32=1

7.35.3.AArch64 kernel build instructions

The following instructions show how to install and run a AArch64 kernel byusing a SD card with the defaultRaspbian install as base. Skip them if youwant to use the default 32-bit kernel.

Note that this system won’t be fully 64-bit because all the tools in thefilesystem are 32-bit binaries, but it’s a quick way to get it working, and itallows the user to run 64-bit binaries in addition to 32-bit binaries.

  1. Clone theLinux tree fork maintained by the Raspberry Pi Foundation. Tospeed things up, do a shallow clone of the desired branch.

gitclone--depth=1-brpi-4.18.yhttps://github.com/raspberrypi/linuxcdlinux

  1. Configure and compile the kernel. Adapt the number after-j so that it is1.5 times the number of CPUs in your computer. This may take some time tofinish.

makeARCH=arm64CROSS_COMPILE=aarch64-linux-gnu-bcmrpi3_defconfigmake-j6ARCH=arm64CROSS_COMPILE=aarch64-linux-gnu-

  1. Copy the kernel image and the device tree to the SD card. Replace the pathby the corresponding path in your computers to theboot partition of theSD card.

cparch/arm64/boot/Image/path/to/boot/kernel8.imgcparch/arm64/boot/dts/broadcom/bcm2710-rpi-3-b.dtb/path/to/boot/cparch/arm64/boot/dts/broadcom/bcm2710-rpi-3-b-plus.dtb/path/to/boot/

  1. Install the kernel modules. Replace the path by the corresponding path to thefilesystem partition of the SD card on your computer.

makeARCH=arm64CROSS_COMPILE=aarch64-linux-gnu-\INSTALL_MOD_PATH=/path/to/filesystemmodules_install

  1. Follow the instructions inSetup SD card except for the step of renamingthe existingkernel7.img (we have already copied a AArch64 kernel).

7.35.4.Setup SD card

The instructions assume that you have an SD card with a fresh install ofRaspbian (or that, at least, theboot partition is untouched, or nearlyuntouched). They have been tested with the image available in 2018-03-13.

  1. Insert the SD card and open theboot partition.

  2. Renamekernel7.img tokernel8.img. This tricks the VideoCorebootloader into booting the Arm cores in AArch64 mode, like TF-A needs,even though the kernel is not compiled for AArch64.

  3. Copyarmstub8.bin here. Whenkernel8.img is available, The VideoCorebootloader will look for a file calledarmstub8.bin and load it ataddress0x0 instead of a predefined one.

  4. To enable the serial port “Mini UART” in Linux, opencmdline.txt and addconsole=serial0,115200console=tty1.

  5. Openconfig.txt and add the following lines at the end (enable_uart=1is only needed to enable debugging through the Mini UART):

enable_uart=1kernel_address=0x02000000device_tree_address=0x01000000

If you connect a serial cable to the Mini UART and your computer, and connectto it (for example, withscreen/dev/ttyUSB0115200) you should see sometext. In the case of an AArch32 kernel, you should see something like this:

NOTICE:BootingTrustedFirmwareNOTICE:BL1:v1.4(release):v1.4-329-g61e94684-dirtyNOTICE:BL1:Built:00:09:25,Nov62017NOTICE:BL1:BootingBL2NOTICE:BL2:v1.4(release):v1.4-329-g61e94684-dirtyNOTICE:BL2:Built:00:09:25,Nov62017NOTICE:BL1:BootingBL31NOTICE:BL31:v1.4(release):v1.4-329-g61e94684-dirtyNOTICE:BL31:Built:00:09:25,Nov62017[0.266484]bcm2835-aux-uart3f215040.serial:couldnotgetclk:-517RaspbianGNU/Linux9raspberrypittyS0raspberrypilogin:

Just enter your credentials, everything should work as expected. Note that theHDMI output won’t show any text during boot.