33.2.1.Enclave Page Types¶

SGX Enclave Control Structure (SECS)

Enclave’s address range, attributes and other global data are definedby this structure.

Regular (REG)

Regular EPC pages contain the code and data of an enclave.

Thread Control Structure (TCS)

Thread Control Structure pages define the entry points to an enclave andtrack the execution state of an enclave thread.

Version Array (VA)

Version Array pages contain 512 slots, each of which can contain a versionnumber for a page evicted from the EPC.

33.2.2.Enclave Page Cache Map¶

The processor tracks EPC pages in a hardware metadata structure called theEnclave Page Cache Map (EPCM). The EPCM contains an entry for each EPC pagewhich describes the owning enclave, access rights and page type among the otherthings.

EPCM permissions are separate from the normal page tables. This prevents thekernel from, for instance, allowing writes to data which an enclave wishes toremain read-only. EPCM permissions may only impose additional restrictions ontop of normal x86 page permissions.

For all intents and purposes, the SGX architecture allows the processor toinvalidate all EPCM entries at will. This requires that software be prepared tohandle an EPCM fault at any time. In practice, this can happen on events likepower transitions when the ephemeral key that encrypts enclave memory is lost.

33.3.1.Enclave build functions¶

In addition to the traditional compiler and linker build process, SGX has aseparate enclave “build” process. Enclaves must be built before they can beexecuted (entered). The first step in building an enclave is opening the/dev/sgx_enclave device. Since enclave memory is protected from directaccess, special privileged instructions are then used to copy data into enclavepages and establish enclave page permissions.

longsgx_ioc_enclave_create(structsgx_encl*encl,void__user*arg)¶

handler forSGX_IOC_ENCLAVE_CREATE

longsgx_ioc_enclave_add_pages(structsgx_encl*encl,void__user*arg)¶

The handler forSGX_IOC_ENCLAVE_ADD_PAGES

longsgx_ioc_enclave_init(structsgx_encl*encl,void__user*arg)¶

handler forSGX_IOC_ENCLAVE_INIT

longsgx_ioc_enclave_provision(structsgx_encl*encl,void__user*arg)¶

handler forSGX_IOC_ENCLAVE_PROVISION

33.3.2.Enclave runtime management¶

Systems supporting SGX2 additionally support changes to initializedenclaves: modifying enclave page permissions and type, and dynamicallyadding and removing of enclave pages. When an enclave accesses an addresswithin its address range that does not have a backing page then a newregular page will be dynamically added to the enclave. The enclave isstill required to run EACCEPT on the new page before it can be used.

longsgx_ioc_enclave_restrict_permissions(structsgx_encl*encl,void__user*arg)¶

handler forSGX_IOC_ENCLAVE_RESTRICT_PERMISSIONS

longsgx_ioc_enclave_modify_types(structsgx_encl*encl,void__user*arg)¶

handler forSGX_IOC_ENCLAVE_MODIFY_TYPES

longsgx_ioc_enclave_remove_pages(structsgx_encl*encl,void__user*arg)¶

handler forSGX_IOC_ENCLAVE_REMOVE_PAGES

33.3.3.Enclave vDSO¶

Entering an enclave can only be done through SGX-specific EENTER and ERESUMEfunctions, and is a non-trivial process. Because of the complexity oftransitioning to and from an enclave, enclaves typically utilize a library tohandle the actual transitions. This is roughly analogous to how glibcimplementations are used by most applications to wrap system calls.

Another crucial characteristic of enclaves is that they can generate exceptionsas part of their normal operation that need to be handled in the enclave or areunique to SGX.

Instead of the traditional signal mechanism to handle these exceptions, SGXcan leverage special exception fixup provided by the vDSO. The kernel-providedvDSO function wraps low-level transitions to/from the enclave like EENTER andERESUME. The vDSO function intercepts exceptions that would otherwise generatea signal and return the fault information directly to its caller. This avoidsthe need to juggle signal handlers.

vdso_sgx_enter_enclave_t¶

Typedef: Prototype for__vdso_sgx_enter_enclave(), a vDSO function to enter an SGX enclave.

Syntax

intvdso_sgx_enter_enclave_t(unsignedlongrdi,unsignedlongrsi,unsignedlongrdx,unsignedintfunction,unsignedlongr8,unsignedlongr9,structsgx_enclave_run*run)

33.4.1.EPC sanitization¶

ksgxd is started when SGX initializes. Enclave memory is typically readyfor use when the processor powers on or resets. However, if SGX has been inuse since the reset, enclave pages may be in an inconsistent state. This mightoccur after a crash andkexec() cycle, for instance. At boot, ksgxdreinitializes all enclave pages so that they can be allocated and re-used.

The sanitization is done by going through EPC address space and applying theEREMOVE function to each physical page. Some enclave pages like SECS pages havehardware dependencies on other pages which prevents EREMOVE from functioning.Executing two EREMOVE passes removes the dependencies.

33.4.2.Page reclaimer¶

Similar to the core kswapd, ksgxd, is responsible for managing theovercommitment of enclave memory. If the system runs out of enclave memory,ksgxd “swaps” enclave memory to normal memory.

33.7.1.Shared Library¶

Sensitive data and the code that acts on it is partitioned from the applicationinto a separate library. The library is then linked as a DSO which can be loadedinto an enclave. The application can then make individual function calls intothe enclave through special SGX instructions. A run-time within the enclave isconfigured to marshal function parameters into and out of the enclave and tocall the correct library function.

33.7.2.Application Container¶

An application may be loaded into a container enclave which is speciallyconfigured with a library OS and run-time which permits the application to run.The enclave run-time and library OS work together to execute the applicationwhen a thread enters the enclave.

33.8.1.EPC leaks¶

When EPC page leaks happen, a WARNING like this is shown in dmesg:

“EREMOVE returned ... and an EPC page was leaked. SGX may become unusable...”

This is effectively a kernel use-after-free of an EPC page, and dueto the way SGX works, the bug is detected at freeing. Rather thanadding the page back to the pool of available EPC pages, the kernelintentionally leaks the page to avoid additional errors in the future.

When this happens, the kernel will likely soon leak more EPC pages, andSGX will likely become unusable because the memory available to SGX islimited. However, while this may be fatal to SGX, the rest of the kernelis unlikely to be impacted and should continue to work.

As a result, when this happens, user should stop running any newSGX workloads, (or just any new workloads), and migrate all valuableworkloads. Although a machine reboot can recover all EPC memory, the bugshould be reported to Linux developers.