Design principles¶

• “graceful fallback”: mm components which don’t have transparent hugepageknowledge fall back to breaking huge pmd mapping into table of ptes and,if necessary, split a transparent hugepage. Therefore these componentscan continue working on the regular pages or regular pte mappings.

• if a hugepage allocation fails because of memory fragmentation,regular pages should be gracefully allocated instead and mixed inthe same vma without any failure or significant delay and withoutuserland noticing

• if some task quits and more hugepages become available (eitherimmediately in the buddy or through the VM), guest physical memorybacked by regular pages should be relocated on hugepagesautomatically (with khugepaged)

• it doesn’t require memory reservation and in turn it uses hugepageswhenever possible (the only possible reservation here is kernelcore=to avoid unmovable pages to fragment all the memory but such a tweakis not specific to transparent hugepage support and it’s a genericfeature that applies to all dynamic high order allocations in thekernel)

get_user_pages and pin_user_pages¶

get_user_pages and pin_user_pages if run on a hugepage, will return thehead or tail pages as usual (exactly as they would do onhugetlbfs). Most GUP users will only care about the actual physicaladdress of the page and its temporary pinning to release after the I/Ois complete, so they won’t ever notice the fact the page is huge. Butif any driver is going to mangle over the page structure of the tailpage (like for checking page->mapping or other bits that are relevantfor the head page and not the tail page), it should be updated to jumpto check head page instead. Taking a reference on any head/tail page wouldprevent the page from being split by anyone.

Graceful fallback¶

Code walking pagetables but unaware about huge pmds can simply callsplit_huge_pmd(vma, pmd, addr) where the pmd is the one returned bypmd_offset. It’s trivial to make the code transparent hugepage awareby just grepping for “pmd_offset” and adding split_huge_pmd wheremissing after pmd_offset returns the pmd. Thanks to the gracefulfallback design, with a one liner change, you can avoid to writehundreds if not thousands of lines of complex code to make your codehugepage aware.

If you’re not walking pagetables but you run into a physical hugepagethat you can’t handle natively in your code, you can split it bycalling split_huge_page(page). This is what the Linux VM does beforeit tries to swapout the hugepage for example.split_huge_page() can failif the page is pinned and you must handle this correctly.

Example to make mremap.c transparent hugepage aware with a one linerchange:

diff --git a/mm/mremap.c b/mm/mremap.c--- a/mm/mremap.c+++ b/mm/mremap.c@@ -41,6 +41,7 @@ static pmd_t *get_old_pmd(struct mm_stru                return NULL;        pmd = pmd_offset(pud, addr);+       split_huge_pmd(vma, pmd, addr);        if (pmd_none_or_clear_bad(pmd))                return NULL;

Locking in hugepage aware code¶

We want as much code as possible hugepage aware, as callingsplit_huge_page() orsplit_huge_pmd() has a cost.

To make pagetable walks huge pmd aware, all you need to do is to callpmd_trans_huge() on the pmd returned by pmd_offset. You must hold themmap_lock in read (or write) mode to be sure a huge pmd cannot becreated from under you by khugepaged (khugepaged collapse_huge_pagetakes the mmap_lock in write mode in addition to the anon_vma lock). Ifpmd_trans_huge returns false, you just fallback in the old codepaths. If instead pmd_trans_huge returns true, you have to take thepage table lock (pmd_lock()) and re-run pmd_trans_huge. Taking thepage table lock will prevent the huge pmd being converted into aregular pmd from under you (split_huge_pmd can run in parallel to thepagetable walk). If the second pmd_trans_huge returns false, youshould just drop the page table lock and fallback to the old code asbefore. Otherwise, you can proceed to process the huge pmd and thehugepage natively. Once finished, you can drop the page table lock.

Refcounts and transparent huge pages¶

Refcounting on THP is mostly consistent with refcounting on other compoundpages:

• get_page()/put_page() and GUP operate on the folio->_refcount.

• ->_refcount in tail pages is always zero:get_page_unless_zero() neversucceeds on tail pages.

• map/unmap of a PMD entry for the whole THP increment/decrementfolio->_entire_mapcount and folio->_large_mapcount.

  We also maintain the two slots for tracking MM owners (MM ID andcorresponding mapcount), and the current status (“maybe mapped shared” vs.“mapped exclusively”).

  With CONFIG_PAGE_MAPCOUNT, we also increment/decrementfolio->_nr_pages_mapped by ENTIRELY_MAPPED when _entire_mapcount goesfrom -1 to 0 or 0 to -1.

• map/unmap of individual pages with PTE entry increment/decrementfolio->_large_mapcount.

  We also maintain the two slots for tracking MM owners (MM ID andcorresponding mapcount), and the current status (“maybe mapped shared” vs.“mapped exclusively”).

  With CONFIG_PAGE_MAPCOUNT, we also increment/decrementpage->_mapcount and increment/decrement folio->_nr_pages_mapped whenpage->_mapcount goes from -1 to 0 or 0 to -1 as this counts the numberof pages mapped by PTE.

split_huge_page internally has to distribute the refcounts in the headpage to the tail pages before clearing all PG_head/tail bits from the pagestructures. It can be done easily for refcounts taken by page tableentries, but we don’t have enough information on how to distribute anyadditional pins (i.e. from get_user_pages).split_huge_page() fails anyrequests to split pinned huge pages: it expects page count to be equal tothe sum of mapcount of all sub-pages plus one (split_huge_page caller musthave a reference to the head page).

split_huge_page uses migration entries to stabilize page->_refcount andpage->_mapcount of anonymous pages. File pages just get unmapped.

We are safe against physical memory scanners too: the only legitimate waya scanner can get a reference to a page isget_page_unless_zero().

All tail pages have zero ->_refcount untilatomic_add(). This prevents thescanner from getting a reference to the tail page up to that point. After theatomic_add() we don’t care about the ->_refcount value. We already know howmany references should be uncharged from the head page.

For head pageget_page_unless_zero() will succeed and we don’t mind. It’sclear where references should go after split: it will stay on the head page.

Note thatsplit_huge_pmd() doesn’t have any limitations on refcounting:pmd can be split at any point and never fails.

Partial unmap and deferred_split_folio() (anon THP only)¶

Unmapping part of THP (withmunmap() or other way) is not going to freememory immediately. Instead, we detect that a subpage of THP is not in usein folio_remove_rmap_*() and queue the THP for splitting if memory pressurecomes. Splitting will free up unused subpages.

Splitting the page right away is not an option due to locking context inthe place where we can detect partial unmap. It also might becounterproductive since in many cases partial unmap happens during exit(2) ifa THP crosses a VMA boundary.

The functiondeferred_split_folio() is used to queue a folio for splitting.The splitting itself will happen when we get memory pressure via shrinkerinterface.

With CONFIG_PAGE_MAPCOUNT, we reliably detect partial mappings based onfolio->_nr_pages_mapped.

With CONFIG_NO_PAGE_MAPCOUNT, we detect partial mappings based on theaverage per-page mapcount in a THP: if the average is < 1, an anon THP iscertainly partially mapped. As long as only a single process maps a THP,this detection is reliable. With long-running child processes, there canbe scenarios where partial mappings can currently not be detected, andmight need asynchronous detection during memory reclaim in the future.