Design principles¶

• “graceful fallback”: mm components which don’t have transparent hugepageknowledge fall back to breaking huge pmd mapping into table of ptes and,if necessary, split a transparent hugepage. Therefore these componentscan continue working on the regular pages or regular pte mappings.
• if a hugepage allocation fails because of memory fragmentation,regular pages should be gracefully allocated instead and mixed inthe same vma without any failure or significant delay and withoutuserland noticing
• if some task quits and more hugepages become available (eitherimmediately in the buddy or through the VM), guest physical memorybacked by regular pages should be relocated on hugepagesautomatically (with khugepaged)
• it doesn’t require memory reservation and in turn it uses hugepageswhenever possible (the only possible reservation here is kernelcore=to avoid unmovable pages to fragment all the memory but such a tweakis not specific to transparent hugepage support and it’s a genericfeature that applies to all dynamic high order allocations in thekernel)

get_user_pages and follow_page¶

get_user_pages and follow_page if run on a hugepage, will return thehead or tail pages as usual (exactly as they would do onhugetlbfs). Most GUP users will only care about the actual physicaladdress of the page and its temporary pinning to release after the I/Ois complete, so they won’t ever notice the fact the page is huge. Butif any driver is going to mangle over the page structure of the tailpage (like for checking page->mapping or other bits that are relevantfor the head page and not the tail page), it should be updated to jumpto check head page instead. Taking a reference on any head/tail page wouldprevent the page from being split by anyone.

In case you can’t handle compound pages if they’re returned byfollow_page, the FOLL_SPLIT bit can be specified as a parameter tofollow_page, so that it will split the hugepages before returningthem.

Graceful fallback¶

Code walking pagetables but unaware about huge pmds can simply callsplit_huge_pmd(vma, pmd, addr) where the pmd is the one returned bypmd_offset. It’s trivial to make the code transparent hugepage awareby just grepping for “pmd_offset” and adding split_huge_pmd wheremissing after pmd_offset returns the pmd. Thanks to the gracefulfallback design, with a one liner change, you can avoid to writehundreds if not thousands of lines of complex code to make your codehugepage aware.

If you’re not walking pagetables but you run into a physical hugepagethat you can’t handle natively in your code, you can split it bycalling split_huge_page(page). This is what the Linux VM does beforeit tries to swapout the hugepage for example. split_huge_page() can failif the page is pinned and you must handle this correctly.

Example to make mremap.c transparent hugepage aware with a one linerchange:

diff --git a/mm/mremap.c b/mm/mremap.c--- a/mm/mremap.c+++ b/mm/mremap.c@@ -41,6 +41,7 @@ static pmd_t *get_old_pmd(struct mm_stru                return NULL;        pmd = pmd_offset(pud, addr);+       split_huge_pmd(vma, pmd, addr);        if (pmd_none_or_clear_bad(pmd))                return NULL;

Locking in hugepage aware code¶

We want as much code as possible hugepage aware, as callingsplit_huge_page() or split_huge_pmd() has a cost.

To make pagetable walks huge pmd aware, all you need to do is to callpmd_trans_huge() on the pmd returned by pmd_offset. You must hold themmap_lock in read (or write) mode to be sure a huge pmd cannot becreated from under you by khugepaged (khugepaged collapse_huge_pagetakes the mmap_lock in write mode in addition to the anon_vma lock). Ifpmd_trans_huge returns false, you just fallback in the old codepaths. If instead pmd_trans_huge returns true, you have to take thepage table lock (pmd_lock()) and re-run pmd_trans_huge. Taking thepage table lock will prevent the huge pmd being converted into aregular pmd from under you (split_huge_pmd can run in parallel to thepagetable walk). If the second pmd_trans_huge returns false, youshould just drop the page table lock and fallback to the old code asbefore. Otherwise, you can proceed to process the huge pmd and thehugepage natively. Once finished, you can drop the page table lock.

Refcounts and transparent huge pages¶

Refcounting on THP is mostly consistent with refcounting on other compoundpages:

• get_page()/put_page() and GUP operate on head page’s ->_refcount.
• ->_refcount in tail pages is always zero: get_page_unless_zero() neversucceeds on tail pages.
• map/unmap of the pages with PTE entry increment/decrement ->_mapcounton relevant sub-page of the compound page.
• map/unmap of the whole compound page is accounted for in compound_mapcount(stored in first tail page). For file huge pages, we also increment->_mapcount of all sub-pages in order to have race-free detection oflast unmap of subpages.

PageDoubleMap() indicates that the page ispossibly mapped with PTEs.

For anonymous pages, PageDoubleMap() also indicates ->_mapcount in allsubpages is offset up by one. This additional reference is required toget race-free detection of unmap of subpages when we have them mapped withboth PMDs and PTEs.

This optimization is required to lower the overhead of per-subpage mapcounttracking. The alternative is to alter ->_mapcount in all subpages on eachmap/unmap of the whole compound page.

For anonymous pages, we set PG_double_map when a PMD of the page is splitfor the first time, but still have a PMD mapping. The additional referencesgo away with the last compound_mapcount.

File pages get PG_double_map set on the first map of the page with PTE andgoes away when the page gets evicted from the page cache.

split_huge_page internally has to distribute the refcounts in the headpage to the tail pages before clearing all PG_head/tail bits from the pagestructures. It can be done easily for refcounts taken by page tableentries, but we don’t have enough information on how to distribute anyadditional pins (i.e. from get_user_pages). split_huge_page() fails anyrequests to split pinned huge pages: it expects page count to be equal tothe sum of mapcount of all sub-pages plus one (split_huge_page caller musthave a reference to the head page).

split_huge_page uses migration entries to stabilize page->_refcount andpage->_mapcount of anonymous pages. File pages just get unmapped.

We are safe against physical memory scanners too: the only legitimate waya scanner can get a reference to a page is get_page_unless_zero().

All tail pages have zero ->_refcount until atomic_add(). This prevents thescanner from getting a reference to the tail page up to that point. After theatomic_add() we don’t care about the ->_refcount value. We already know howmany references should be uncharged from the head page.

For head page get_page_unless_zero() will succeed and we don’t mind. It’sclear where references should go after split: it will stay on the head page.

Note that split_huge_pmd() doesn’t have any limitations on refcounting:pmd can be split at any point and never fails.

Partial unmap and deferred_split_huge_page()¶

Unmapping part of THP (with munmap() or other way) is not going to freememory immediately. Instead, we detect that a subpage of THP is not in usein page_remove_rmap() and queue the THP for splitting if memory pressurecomes. Splitting will free up unused subpages.

Splitting the page right away is not an option due to locking context inthe place where we can detect partial unmap. It also might becounterproductive since in many cases partial unmap happens during exit(2) ifa THP crosses a VMA boundary.

The function deferred_split_huge_page() is used to queue a page for splitting.The splitting itself will happen when we get memory pressure via shrinkerinterface.