Creating a userfaultfd¶

There are two ways to create a new userfaultfd, each of which provide ways torestrict access to this functionality (since historically userfaultfds whichhandle kernel page faults have been a useful tool for exploiting the kernel).

The first way, supported since userfaultfd was introduced, is theuserfaultfd(2) syscall. Access to this is controlled in several ways:

• Any user can always create a userfaultfd which traps userspace page faultsonly. Such a userfaultfd can be created using the userfaultfd(2) syscallwith the flag UFFD_USER_MODE_ONLY.

• In order to also trap kernel page faults for the address space, either theprocess needs the CAP_SYS_PTRACE capability, or the system must havevm.unprivileged_userfaultfd set to 1. By default, vm.unprivileged_userfaultfdis set to 0.

The second way, added to the kernel more recently, is by opening/dev/userfaultfd and issuing a USERFAULTFD_IOC_NEW ioctl to it. This methodyields equivalent userfaultfds to the userfaultfd(2) syscall.

Unlike userfaultfd(2), access to /dev/userfaultfd is controlled via normalfilesystem permissions (user/group/mode), which gives fine grained access touserfaultfd specifically, without also granting other unrelated privileges atthe same time (as e.g. granting CAP_SYS_PTRACE would do). Users who have accessto /dev/userfaultfd can always create userfaultfds that trap kernel page faults;vm.unprivileged_userfaultfd is not considered.

Initializing a userfaultfd¶

When first opened theuserfaultfd must be enabled invoking theUFFDIO_API ioctl specifying auffdio_api.api value set toUFFD_API (ora later API version) which will specify theread/POLLIN protocoluserland intends to speak on theUFFD and theuffdio_api.featuresuserland requires. TheUFFDIO_API ioctl if successful (i.e. if therequesteduffdio_api.api is spoken also by the running kernel and therequested features are going to be enabled) will return intouffdio_api.features anduffdio_api.ioctls two 64bit bitmasks ofrespectively all the available features of the read(2) protocol andthe generic ioctl available.

Theuffdio_api.features bitmask returned by theUFFDIO_API ioctldefines what memory types are supported by theuserfaultfd and whatevents, except page fault notifications, may be generated:

• TheUFFD_FEATURE_EVENT_* flags indicate that various other eventsother than page faults are supported. These events are described in moredetail below in theNon-cooperative userfaultfd section.

• UFFD_FEATURE_MISSING_HUGETLBFS andUFFD_FEATURE_MISSING_SHMEMindicate that the kernel supportsUFFDIO_REGISTER_MODE_MISSINGregistrations for hugetlbfs and shared memory (covering all shmem APIs,i.e. tmpfs,IPCSHM,/dev/zero,MAP_SHARED,memfd_create,etc) virtual memory areas, respectively.

• UFFD_FEATURE_MINOR_HUGETLBFS indicates that the kernel supportsUFFDIO_REGISTER_MODE_MINOR registration for hugetlbfs virtual memoryareas.UFFD_FEATURE_MINOR_SHMEM is the analogous feature indicatingsupport for shmem virtual memory areas.

• UFFD_FEATURE_MOVE indicates that the kernel supports moving anexisting page contents from userspace.

The userland application should set the feature flags it intends to usewhen invoking theUFFDIO_API ioctl, to request that those features beenabled if supported.

Once theuserfaultfd API has been enabled theUFFDIO_REGISTERioctl should be invoked (if present in the returneduffdio_api.ioctlsbitmask) to register a memory range in theuserfaultfd by setting theuffdio_register structure accordingly. Theuffdio_register.modebitmask will specify to the kernel which kind of faults to track forthe range. TheUFFDIO_REGISTER ioctl will return theuffdio_register.ioctls bitmask of ioctls that are suitable to resolveuserfaults on the range registered. Not all ioctls will necessarily besupported for all memory types (e.g. anonymous memory vs. shmem vs.hugetlbfs), or all types of intercepted faults.

Userland can use theuffdio_register.ioctls to manage the virtualaddress space in the background (to add or potentially also removememory from theuserfaultfd registered range). This means a userfaultcould be triggering just before userland maps in the background theuser-faulted page.

Resolving Userfaults¶

There are three basic ways to resolve userfaults:

• UFFDIO_COPY atomically copies some existing page contents fromuserspace.

• UFFDIO_ZEROPAGE atomically zeros the new page.

• UFFDIO_CONTINUE maps an existing, previously-populated page.

These operations are atomic in the sense that they guarantee nothing cansee a half-populated page, since readers will keep userfaulting until theoperation has finished.

By default, these wake up userfaults blocked on the range in question.They support aUFFDIO_*_MODE_DONTWAKEmode flag, which indicatesthat waking will be done separately at some later time.

Which ioctl to choose depends on the kind of page fault, and what we’dlike to do to resolve it:

• ForUFFDIO_REGISTER_MODE_MISSING faults, the fault needs to beresolved by either providing a new page (UFFDIO_COPY), or mappingthe zero page (UFFDIO_ZEROPAGE). By default, the kernel would mapthe zero page for a missing fault. With userfaultfd, userspace candecide what content to provide before the faulting thread continues.

• ForUFFDIO_REGISTER_MODE_MINOR faults, there is an existing page (inthe page cache). Userspace has the option of modifying the page’scontents before resolving the fault. Once the contents are correct(modified or not), userspace asks the kernel to map the page and let thefaulting thread continue withUFFDIO_CONTINUE.

Notes:

• You can tell which kind of fault occurred by examiningpagefault.flags within theuffd_msg, checking for theUFFD_PAGEFAULT_FLAG_* flags.

• None of the page-delivering ioctls default to the range that youregistered with. You must fill in all fields for the appropriateioctlstructincluding the range.

• You get the address of the access that triggered the missing pageevent out of astructuffd_msg that you read in the thread from theuffd. You can supply as many pages as you want with these IOCTLs.Keep in mind that unless you used DONTWAKE then the first of any ofthose IOCTLs wakes up the faulting thread.

• Be sure to test for all errors including(pollfd[0].revents&POLLERR). This can happen, e.g. when rangessupplied were incorrect.

Write Protect Notifications¶

This is equivalent to (but faster than) using mprotect and a SIGSEGVsignal handler.

Firstly you need to register a range withUFFDIO_REGISTER_MODE_WP.Instead of using mprotect(2) you useioctl(uffd,UFFDIO_WRITEPROTECT,struct*uffdio_writeprotect)whilemode=UFFDIO_WRITEPROTECT_MODE_WPin thestructpassed in. The range does not default to and does nothave to be identical to the range you registered with. You can writeprotect as many ranges as you like (inside the registered range).Then, in the thread reading from uffd thestructwill havemsg.arg.pagefault.flags&UFFD_PAGEFAULT_FLAG_WP set. Now you sendioctl(uffd,UFFDIO_WRITEPROTECT,struct*uffdio_writeprotect)again whilepagefault.mode does not haveUFFDIO_WRITEPROTECT_MODE_WPset. This wakes up the thread which will continue to run with writes. Thisallows you to do the bookkeeping about the write in the uffd readingthread before the ioctl.

If you registered with bothUFFDIO_REGISTER_MODE_MISSING andUFFDIO_REGISTER_MODE_WP then you need to think about the sequence inwhich you supply a page and undo write protect. Note that there is adifference between writes into a WP area and into a !WP area. Theformer will haveUFFD_PAGEFAULT_FLAG_WP set, the latterUFFD_PAGEFAULT_FLAG_WRITE. The latter did not fail on protection butyou still need to supply a page whenUFFDIO_REGISTER_MODE_MISSING wasused.

Userfaultfd write-protect mode currently behave differently on none ptes(when e.g. page is missing) over different types of memories.

For anonymous memory,ioctl(UFFDIO_WRITEPROTECT) will ignore none ptes(e.g. when pages are missing and not populated). For file-backed memorieslike shmem and hugetlbfs, none ptes will be write protected just like apresent pte. In other words, there will be a userfaultfd write faultmessage generated when writing to a missing page on file typed memories,as long as the page range was write-protected before. Such a message willnot be generated on anonymous memories by default.

If the application wants to be able to write protect none ptes on anonymousmemory, one can pre-populate the memory with e.g. MADV_POPULATE_READ. Onnewer kernels, one can also detect the feature UFFD_FEATURE_WP_UNPOPULATEDand set the feature bit in advance to make sure none ptes will also bewrite protected even upon anonymous memory.

When usingUFFDIO_REGISTER_MODE_WP in combination with eitherUFFDIO_REGISTER_MODE_MISSING orUFFDIO_REGISTER_MODE_MINOR, whenresolving missing / minor faults withUFFDIO_COPY orUFFDIO_CONTINUErespectively, it may be desirable for the new page / mapping to bewrite-protected (so future writes will also result in a WP fault). These ioctlssupport a mode flag (UFFDIO_COPY_MODE_WP orUFFDIO_CONTINUE_MODE_WPrespectively) to configure the mapping this way.

If the userfaultfd context hasUFFD_FEATURE_WP_ASYNC feature bit set,any vma registered with write-protection will work in async mode ratherthan the default sync mode.

In async mode, there will be no message generated when a write operationhappens, meanwhile the write-protection will be resolved automatically bythe kernel. It can be seen as a more accurate version of soft-dirtytracking and it can be different in a few ways:

• The dirty result will not be affected by vma changes (e.g. vmamerging) because the dirty is only tracked by the pte.

• It supports range operations by default, so one can enable tracking onany range of memory as long as page aligned.

• Dirty information will not get lost if the pte was zapped due tovarious reasons (e.g. during split of a shmem transparent huge page).

• Due to a reverted meaning of soft-dirty (page clean when uffd-wp bitset; dirty when uffd-wp bit cleared), it has different semantics onsome of the memory operations. For example:MADV_DONTNEED onanonymous (orMADV_REMOVE on a file mapping) will be treated asdirtying of memory by dropping uffd-wp bit during the procedure.

The user app can collect the “written/dirty” status by looking up theuffd-wp bit for the pages being interested in /proc/pagemap.

The page will not be under track of uffd-wp async mode until the page isexplicitly write-protected byioctl(UFFDIO_WRITEPROTECT) with the modeflagUFFDIO_WRITEPROTECT_MODE_WP set. Trying to resolve a page faultthat was tracked by async mode userfaultfd-wp is invalid.

When userfaultfd-wp async mode is used alone, it can be applied to allkinds of memory.

Memory Poisioning Emulation¶

In response to a fault (either missing or minor), an action userspace cantake to “resolve” it is to issue aUFFDIO_POISON. This will cause anyfuture faulters to either get a SIGBUS, or in KVM’s case the guest willreceive an MCE as if there were hardware memory poisoning.

This is used to emulate hardware memory poisoning. Imagine a VM running on amachine which experiences a real hardware memory error. Later, we live migratethe VM to another physical machine. Since we want the migration to betransparent to the guest, we want that same address range to act as if it wasstill poisoned, even though it’s on a new physical host which ostensiblydoesn’t have a memory error in the exact same spot.