Security levels and how to use them¶

The interface presented here is not meant for end users. Instead thereshould be a userspace tool that handles all the low-level details, keepsa database of the authorized devices and prompts users for new connections.

More details about the sysfs interface for Thunderbolt devices can befound inABI file testing/sysfs-bus-thunderbolt.

Those users who just want to connect any device without any sort ofmanual work can add following line to/etc/udev/rules.d/99-local.rules:

ACTION=="add", SUBSYSTEM=="thunderbolt", ATTR{authorized}=="0", ATTR{authorized}="1"

This will authorize all devices automatically when they appear. However,keep in mind that this bypasses the security levels and makes the systemvulnerable to DMA attacks.

Starting with Intel Falcon Ridge Thunderbolt controller there are 4security levels available. Intel Titan Ridge added one more security level(usbonly). The reason for these is the fact that the connected devices canbe DMA masters and thus read contents of the host memory without CPU and OSknowing about it. There are ways to prevent this by setting up an IOMMU butit is not always available for various reasons.

Some USB4 systems have a BIOS setting to disable PCIe tunneling. This istreated as another security level (nopcie).

The security levels are as follows:

none

All devices are automatically connected by the firmware. No userapproval is needed. In BIOS settings this is typically calledLegacy mode.

user

User is asked whether the device is allowed to be connected.Based on the device identification information available through/sys/bus/thunderbolt/devices, the user then can make the decision.In BIOS settings this is typically calledUnique ID.

secure

User is asked whether the device is allowed to be connected. Inaddition to UUID the device (if it supports secure connect) is senta challenge that should match the expected one based on a random keywritten to thekey sysfs attribute. In BIOS settings this istypically calledOne time saved key.

dponly

The firmware automatically creates tunnels for Display Port andUSB. No PCIe tunneling is done. In BIOS settings this istypically calledDisplay Port Only.

usbonly

The firmware automatically creates tunnels for the USB controller andDisplay Port in a dock. All PCIe links downstream of the dock areremoved.

nopcie

PCIe tunneling is disabled/forbidden from the BIOS. Available in someUSB4 systems.

The current security level can be read from/sys/bus/thunderbolt/devices/domainX/security wheredomainX isthe Thunderbolt domain the host controller manages. There is typicallyone domain per Thunderbolt host controller.

If the security level reads asuser orsecure the connecteddevice must be authorized by the user before PCIe tunnels are created(e.g the PCIe device appears).

Each Thunderbolt device plugged in will appear in sysfs under/sys/bus/thunderbolt/devices. The device directory carriesinformation that can be used to identify the particular device,including its name and UUID.

Authorizing devices when security level isuser orsecure¶

When a device is plugged in it will appear in sysfs as follows:

/sys/bus/thunderbolt/devices/0-1/authorized   - 0/sys/bus/thunderbolt/devices/0-1/device       - 0x8004/sys/bus/thunderbolt/devices/0-1/device_name  - Thunderbolt to FireWire Adapter/sys/bus/thunderbolt/devices/0-1/vendor       - 0x1/sys/bus/thunderbolt/devices/0-1/vendor_name  - Apple, Inc./sys/bus/thunderbolt/devices/0-1/unique_id    - e0376f00-0300-0100-ffff-ffffffffffff

Theauthorized attribute reads 0 which means no PCIe tunnels arecreated yet. The user can authorize the device by simply entering:

# echo 1 > /sys/bus/thunderbolt/devices/0-1/authorized

This will create the PCIe tunnels and the device is now connected.

If the device supports secure connect, and the domain security level isset tosecure, it has an additional attributekey which can holda random 32-byte value used for authorization and challenging the device infuture connects:

/sys/bus/thunderbolt/devices/0-3/authorized   - 0/sys/bus/thunderbolt/devices/0-3/device       - 0x305/sys/bus/thunderbolt/devices/0-3/device_name  - AKiTiO Thunder3 PCIe Box/sys/bus/thunderbolt/devices/0-3/key          -/sys/bus/thunderbolt/devices/0-3/vendor       - 0x41/sys/bus/thunderbolt/devices/0-3/vendor_name  - inXtron/sys/bus/thunderbolt/devices/0-3/unique_id    - dc010000-0000-8508-a22d-32ca6421cb16

Notice the key is empty by default.

If the user does not want to use secure connect they can justecho1to theauthorized attribute and the PCIe tunnels will be created inthe same way as in theuser security level.

If the user wants to use secure connect, the first time the device isplugged a key needs to be created and sent to the device:

# key=$(openssl rand -hex 32)# echo $key > /sys/bus/thunderbolt/devices/0-3/key# echo 1 > /sys/bus/thunderbolt/devices/0-3/authorized

Now the device is connected (PCIe tunnels are created) and in additionthe key is stored on the device NVM.

Next time the device is plugged in the user can verify (challenge) thedevice using the same key:

# echo $key > /sys/bus/thunderbolt/devices/0-3/key# echo 2 > /sys/bus/thunderbolt/devices/0-3/authorized

If the challenge the device returns back matches the one we expect basedon the key, the device is connected and the PCIe tunnels are created.However, if the challenge fails no tunnels are created and error isreturned to the user.

If the user still wants to connect the device they can either approvethe device without a key or write a new key and write 1 to theauthorized file to get the new key stored on the device NVM.

De-authorizing devices¶

It is possible to de-authorize devices by writing0 to theirauthorized attribute. This requires support from the connectionmanager implementation and can be checked by reading domaindeauthorization attribute. If it reads1 then the feature issupported.

When a device is de-authorized the PCIe tunnel from the parent devicePCIe downstream (or root) port to the device PCIe upstream port is torndown. This is essentially the same thing as PCIe hot-remove and the PCIetoplogy in question will not be accessible anymore until the device isauthorized again. If there is storage such as NVMe or similar involved,there is a risk for data loss if the filesystem on that storage is notproperly shut down. You have been warned!

DMA protection utilizing IOMMU¶

Recent systems from 2018 and forward with Thunderbolt ports may nativelysupport IOMMU. This means that Thunderbolt security is handled by an IOMMUso connected devices cannot access memory regions outside of what isallocated for them by drivers. When Linux is running on such system itautomatically enables IOMMU if not enabled by the user already. Thesesystems can be identified by reading1 from/sys/bus/thunderbolt/devices/domainX/iommu_dma_protection attribute.

The driver does not do anything special in this case but because DMAprotection is handled by the IOMMU, security levels (if set) areredundant. For this reason some systems ship with security level set tonone. Other systems have security level set touser in order tosupport downgrade to older OS, so users who want to automaticallyauthorize devices when IOMMU DMA protection is enabled can use thefollowingudev rule:

ACTION=="add", SUBSYSTEM=="thunderbolt", ATTRS{iommu_dma_protection}=="1", ATTR{authorized}=="0", ATTR{authorized}="1"

Upgrading NVM on Thunderbolt device, host or retimer¶

Since most of the functionality is handled in firmware running on ahost controller or a device, it is important that the firmware can beupgraded to the latest where possible bugs in it have been fixed.Typically OEMs provide this firmware from their support site.

Currently, recommended method of updating firmware is through “fwupd” tool.It uses LVFS (Linux Vendor Firmware Service) portal by default to get thelatest firmware from hardware vendors and updates connected devices if foundcompatible. For details refer to:https://github.com/fwupd/fwupd.

Before you upgrade firmware on a device, host or retimer, please makesure it is a suitable upgrade. Failing to do that may render the devicein a state where it cannot be used properly anymore without specialtools!

Host NVM upgrade on Apple Macs is not supported.

Fwupd is installed by default. If you don’t have it on your system, simplyuse your distro package manager to get it.

To see possible updates through fwupd, you need to plug in a Thunderboltdevice so that the host controller appears. It does not matter whichdevice is connected (unless you are upgrading NVM on a device - then youneed to connect that particular device).

Note an OEM-specific method to power the controller up (“force power”) maybe available for your system in which case there is no need to plug in aThunderbolt device.

Updating firmware using fwupd is straightforward - refer to officialreadme on fwupd github.

If firmware image is written successfully, the device shortly disappears.Once it comes back, the driver notices it and initiates a full powercycle. After a while device appears again and this time it should befully functional.

Device of interest should display new version under “Current version”and “Update State: Success” in fwupd’s interface.

Upgrading firmware manually¶

If possible, use fwupd to updated the firmware. However, if your device OEMhas not uploaded the firmware to LVFS, but it is available for downloadfrom their side, you can use method below to directly upgrade thefirmware.

Manual firmware update can be done with ‘dd’ tool. To update firmwareusing this method, you need to write it to the non-active parts of NVMof the host or device. Example on how to update Intel NUC6i7KYK(Skull Canyon) Thunderbolt controller NVM:

# dd if=KYK_TBT_FW_0018.bin of=/sys/bus/thunderbolt/devices/0-0/nvm_non_active0/nvmem

Once the operation completes we can trigger NVM authentication andupgrade process as follows:

# echo 1 > /sys/bus/thunderbolt/devices/0-0/nvm_authenticate

If no errors are returned, device should behave as described in previoussection.

We can verify that the new NVM firmware is active by running the followingcommands:

# cat /sys/bus/thunderbolt/devices/0-0/nvm_authenticate0x0# cat /sys/bus/thunderbolt/devices/0-0/nvm_version18.0

Ifnvm_authenticate contains anything other than 0x0 it is the errorcode from the last authentication cycle, which means the authenticationof the NVM image failed.

Note names of the NVMem devicesnvm_activeN andnvm_non_activeNdepend on the order they are registered in the NVMem subsystem. N inthe name is the identifier added by the NVMem subsystem.

Upgrading on-board retimer NVM when there is no cable connected¶

If the platform supports, it may be possible to upgrade the retimer NVMfirmware even when there is nothing connected to the USB4ports. When this is the case theusb4_portX devices have two specialattributes:offline andrescan. The way to upgrade the firmwareis to first put the USB4 port into offline mode:

# echo 1 > /sys/bus/thunderbolt/devices/0-0/usb4_port1/offline

This step makes sure the port does not respond to any hotplug events,and also ensures the retimers are powered on. The next step is to scanfor the retimers:

# echo 1 > /sys/bus/thunderbolt/devices/0-0/usb4_port1/rescan

This enumerates and adds the on-board retimers. Now retimer NVM can beupgraded in the same way than with cable connected (see previoussection). However, the retimer is not disconnected as we are offlinemode) so after writing1 tonvm_authenticate one should wait for5 or more seconds before running rescan again:

# echo 1 > /sys/bus/thunderbolt/devices/0-0/usb4_port1/rescan

This point if everything went fine, the port can be put back tofunctional state again:

# echo 0 > /sys/bus/thunderbolt/devices/0-0/usb4_port1/offline

Upgrading NVM when host controller is in safe mode¶

If the existing NVM is not properly authenticated (or is missing) thehost controller goes into safe mode which means that the only availablefunctionality is flashing a new NVM image. When in this mode, readingnvm_version fails withENODATA and the device identificationinformation is missing.

To recover from this mode, one needs to flash a valid NVM image to thehost controller in the same way it is done in the previous chapter.

Tunneling events¶

The driver sendsKOBJ_CHANGE events to userspace when there is atunneling change in thethunderbolt_domain. The notification carriesfollowing environment variables:

TUNNEL_EVENT=<EVENT>TUNNEL_DETAILS=0:12 <-> 1:20 (USB3)

Possible values for<EVENT> are:

activated

The tunnel was activated (created).

changed

There is a change in this tunnel. For example bandwidth allocation waschanged.

deactivated

The tunnel was torn down.

low bandwidth

The tunnel is not getting optimal bandwidth.

insufficient bandwidth

There is not enough bandwidth for the current tunnel requirements.

TheTUNNEL_DETAILS is only provided if the tunnel is known. Forexample, in case of Firmware Connection Manager this is missing or doesnot provide full tunnel information. In case of Software Connection Managerthis includes full tunnel details. The format currently matches what thedriver uses when logging. This may change over time.

Networking over Thunderbolt cable¶

Thunderbolt technology allows software communication between two hostsconnected by a Thunderbolt cable.

It is possible to tunnel any kind of traffic over a Thunderbolt link butcurrently we only support Apple ThunderboltIP protocol.

If the other host is running Windows or macOS, the only thing you need todo is to connect a Thunderbolt cable between the two hosts; thethunderbolt-net driver is loaded automatically. If the other host isalso Linux you should loadthunderbolt-net manually on one host (itdoes not matter which one):

# modprobe thunderbolt-net

This triggers module load on the other host automatically. If the driveris built-in to the kernel image, there is no need to do anything.

The driver will create one virtual ethernet interface per Thunderboltport which are named likethunderbolt0 and so on. From this pointyou can either use standard userspace tools likeifconfig toconfigure the interface or let your GUI handle it automatically.

Forcing power¶

Many OEMs include a method that can be used to force the power of aThunderbolt controller to an “On” state even if nothing is connected.If supported by your machine this will be exposed by the WMI bus witha sysfs attribute called “force_power”, seeABI file testing/sysfs-platform-intel-wmi-thunderbolt for details.

Note: it’s currently not possible to query the force power state of a platform.