22.1.3.1.TDX Memory Policy¶

TDX reports a list of “Convertible Memory Region” (CMR) to tell thekernel which memory is TDX compatible. The kernel needs to build a listof memory regions (out of CMRs) as “TDX-usable” memory and pass thoseregions to the TDX module. Once this is done, those “TDX-usable” memoryregions are fixed during module’s lifetime.

To keep things simple, currently the kernel simply guarantees all pagesin the page allocator are TDX memory. Specifically, the kernel uses allsystem memory in the core-mm “at the time of TDX module initialization”as TDX memory, and in the meantime, refuses to online any non-TDX-memoryin the memory hotplug.

22.1.3.2.Physical Memory Hotplug¶

Note TDX assumes convertible memory is always physically present duringmachine’s runtime. A non-buggy BIOS should never support hot-removal ofany convertible memory. This implementation doesn’t handle ACPI memoryremoval but depends on the BIOS to behave correctly.

22.1.3.3.CPU Hotplug¶

TDX module requires the per-cpu initialization SEAMCALL must be done onone cpu before any other SEAMCALLs can be made on that cpu. The kernelprovidestdx_cpu_enable() to let the user of TDX to do it when the userwants to use a new cpu for TDX task.

TDX doesn’t support physical (ACPI) CPU hotplug. During machine boot,TDX verifies all boot-time present logical CPUs are TDX compatible beforeenabling TDX. A non-buggy BIOS should never support hot-add/removal ofphysical CPU. Currently the kernel doesn’t handle physical CPU hotplug,but depends on the BIOS to behave correctly.

Note TDX works with CPU logical online/offline, thus the kernel stillallows to offline logical CPU and online it again.

22.1.3.4.Erratum¶

The first few generations of TDX hardware have an erratum. A partialwrite to a TDX private memory cacheline will silently “poison” theline. Subsequent reads will consume the poison and generate a machinecheck.

A partial write is a memory write where a write transaction of less thancacheline lands at the memory controller. The CPU does these vianon-temporal write instructions (like MOVNTI), or through UC/WC memorymappings. Devices can also do partial writes via DMA.

Theoretically, a kernel bug could do partial write to TDX private memoryand trigger unexpected machine check. What’s more, the machine checkcode will present these as “Hardware error” when they were, in fact, asoftware-triggered issue. But in the end, this issue is hard to trigger.

If the platform has such erratum, the kernel prints additional message inmachine check handler to tell user the machine check may be caused bykernel bug on TDX private memory.

22.1.3.5.Kexec¶

Currently kexec doesn’t work on the TDX platforms with the aforementionederratum. It fails when loading the kexec kernel image. Otherwise itworks normally.

22.1.3.6.Interaction vs S3 and deeper states¶

TDX cannot survive from S3 and deeper states. The hardware resets anddisables TDX completely when platform goes to S3 and deeper. Both TDXguests and the TDX module get destroyed permanently.

The kernel uses S3 for suspend-to-ram, and use S4 and deeper states forhibernation. Currently, for simplicity, the kernel chooses to make TDXmutually exclusive with S3 and hibernation.

The kernel disables TDX during early boot when hibernation support isavailable:

[..] virt/tdx: initialization failed: Hibernation support is enabled

Add ‘nohibernate’ kernel command line to disable hibernation in order touse TDX.

ACPI S3 is disabled during kernel early boot if TDX is enabled. The userneeds to turn off TDX in the BIOS in order to use S3.

22.2.1.1.Instruction-based #VE¶

• Port I/O (INS, OUTS, IN, OUT)

• HLT

• MONITOR, MWAIT

• WBINVD, INVD

• VMCALL

• RDMSR*,WRMSR*

• CPUID*

22.2.1.2.Instruction-based #GP¶

• All VMX instructions: INVEPT, INVVPID, VMCLEAR, VMFUNC, VMLAUNCH,VMPTRLD, VMPTRST, VMREAD, VMRESUME, VMWRITE, VMXOFF, VMXON

• ENCLS, ENCLU

• GETSEC

• RSM

• ENQCMD

• RDMSR*,WRMSR*

22.2.1.3.RDMSR/WRMSR Behavior¶

MSR access behavior falls into three categories:

• #GP generated

• #VE generated

• “Just works”

In general, the #GP MSRs should not be used in guests. Their use likelyindicates a bug in the guest. The guest may try to handle the #GP with ahypercall but it is unlikely to succeed.

The #VE MSRs are typically able to be handled by the hypervisor. Guestscan make a hypercall to the hypervisor to handle the #VE.

The “just works” MSRs do not need any special guest handling. They mightbe implemented by directly passing through the MSR to the hardware or bytrapping and handling in the TDX module. Other than possibly being slow,these MSRs appear to function just as they would on bare metal.

22.2.1.4.CPUID Behavior¶

For some CPUID leaves and sub-leaves, the virtualized bit fields of CPUIDreturn values (in guest EAX/EBX/ECX/EDX) are configurable by thehypervisor. For such cases, the Intel TDX module architecture defines twovirtualization types:

• Bit fields for which the hypervisor controls the value seen by the guestTD.

• Bit fields for which the hypervisor configures the value such that theguest TD either sees their native value or a value of 0. For these bitfields, the hypervisor can mask off the native values, but it can notturnon values.

A #VE is generated for CPUID leaves and sub-leaves that the TDX module doesnot know how to handle. The guest kernel may ask the hypervisor for thevalue with a hypercall.

22.2.2.1.#VE on Shared Memory¶

Access to shared mappings can cause a #VE. The hypervisor ultimatelycontrols whether a shared memory access causes a #VE, so the guest must becareful to only reference shared pages it can safely handle a #VE. Forinstance, the guest should be careful not to access shared memory in the#VE handler before it reads the #VE info structure (TDG.VP.VEINFO.GET).

Shared mapping content is entirely controlled by the hypervisor. The guestshould only use shared mappings for communicating with the hypervisor.Shared mappings must never be used for sensitive memory content like kernelstacks. A good rule of thumb is that hypervisor-shared memory should betreated the same as memory mapped to userspace. Both the hypervisor anduserspace are completely untrusted.

MMIO for virtual devices is implemented as shared memory. The guest mustbe careful not to access device MMIO regions unless it is also prepared tohandle a #VE.

22.2.2.2.#VE on Private Pages¶

An access to private mappings can also cause a #VE. Since all kernelmemory is also private memory, the kernel might theoretically need tohandle a #VE on arbitrary kernel memory accesses. This is not feasible, soTDX guests ensure that all guest memory has been “accepted” before memoryis used by the kernel.

A modest amount of memory (typically 512M) is pre-accepted by the firmwarebefore the kernel runs to ensure that the kernel can start up withoutbeing subjected to a #VE.

The hypervisor is permitted to unilaterally move accepted pages to a“blocked” state. However, if it does this, page access will not generate a#VE. It will, instead, cause a “TD Exit” where the hypervisor is requiredto handle the exception.