In kernel use of migrate_pages()¶

1. Remove pages from the LRU.

   Lists of pages to be migrated are generated by scanning overpages and moving them into lists. This is done bycalling isolate_lru_page().Calling isolate_lru_page increases the references to the pageso that it cannot vanish while the page migration occurs.It also prevents the swapper or other scans to encounterthe page.

2. We need to have a function of type new_page_t that can bepassed to migrate_pages(). This function should figure outhow to allocate the correct new page given the old page.

3. The migrate_pages() function is called which attemptsto do the migration. It will call the function to allocatethe new page for each page that is considered formoving.

How migrate_pages() works¶

migrate_pages() does several passes over its list of pages. A page is movedif all references to a page are removable at the time. The page hasalready been removed from the LRU via isolate_lru_page() and the refcountis increased so that the page cannot be freed while page migration occurs.

Steps:

1. Lock the page to be migrated
2. Ensure that writeback is complete.
3. Lock the new page that we want to move to. It is locked so that accesses tothis (not yet uptodate) page immediately lock while the move is in progress.
4. All the page table references to the page are converted to migrationentries. This decreases the mapcount of a page. If the resultingmapcount is not zero then we do not migrate the page. All user spaceprocesses that attempt to access the page will now wait on the page lock.
5. The i_pages lock is taken. This will cause all processes tryingto access the page via the mapping to block on the spinlock.
6. The refcount of the page is examined and we back out if references remainotherwise we know that we are the only one referencing this page.
7. The radix tree is checked and if it does not contain the pointer to thispage then we back out because someone else modified the radix tree.
8. The new page is prepped with some settings from the old page so thataccesses to the new page will discover a page with the correct settings.
9. The radix tree is changed to point to the new page.
10. The reference count of the old page is dropped because the address spacereference is gone. A reference to the new page is established becausethe new page is referenced by the address space.
11. The i_pages lock is dropped. With that lookups in the mappingbecome possible again. Processes will move from spinning on the lockto sleeping on the locked new page.
12. The page contents are copied to the new page.
13. The remaining page flags are copied to the new page.
14. The old page flags are cleared to indicate that the page doesnot provide any information anymore.
15. Queued up writeback on the new page is triggered.
16. If migration entries were page then replace them with real ptes. Doingso will enable access for user space processes not already waiting forthe page lock.

19. The page locks are dropped from the old and new page.Processes waiting on the page lock will redo their page faultsand will reach the new page.
20. The new page is moved to the LRU and can be scanned by the swapperetc again.

Non-LRU page migration¶

Although original migration aimed for reducing the latency of memory accessfor NUMA, compaction who want to create high-order page is also main customer.

Current problem of the implementation is that it is designed to migrate onlyLRU pages. However, there are potential non-lru pages which can be migratedin drivers, for example, zsmalloc, virtio-balloon pages.

For virtio-balloon pages, some parts of migration code path have been hookedup and added virtio-balloon specific functions to intercept migration logics.It’s too specific to a driver so other drivers who want to make their pagesmovable would have to add own specific hooks in migration path.

To overclome the problem, VM supports non-LRU page migration which providesgeneric functions for non-LRU movable pages without driver specific hooksmigration path.

If a driver want to make own pages movable, it should define three functionswhich are function pointers of struct address_space_operations.

1. bool(*isolate_page)(structpage*page,isolate_mode_tmode);

   What VM expects on isolate_page function of driver is to returntrueif driver isolates page successfully. On returing true, VM marks the pageas PG_isolated so concurrent isolation in several CPUs skip the pagefor isolation. If a driver cannot isolate the page, it should returnfalse.

   Once page is successfully isolated, VM uses page.lru fields so drivershouldn’t expect to preserve values in that fields.

2.int(*migratepage)(structaddress_space*mapping,|structpage*newpage,structpage*oldpage,enummigrate_mode);

After isolation, VM calls migratepage of driver with isolated page.The function of migratepage is to move content of the old page to new pageand set up fields of struct page newpage. Keep in mind that you shouldindicate to the VM the oldpage is no longer movable via __ClearPageMovable()under page_lock if you migrated the oldpage successfully and returnsMIGRATEPAGE_SUCCESS. If driver cannot migrate the page at the moment, drivercan return -EAGAIN. On -EAGAIN, VM will retry page migration in a short timebecause VM interprets -EAGAIN as “temporal migration failure”. On returningany error except -EAGAIN, VM will give up the page migration without retryingin this time.

Driver shouldn’t touch page.lru field VM using in the functions.

3. void(*putback_page)(structpage*);

   If migration fails on isolated page, VM should return the isolated pageto the driver so VM calls driver’s putback_page with migration failed page.In this function, driver should put the isolated page back to the own datastructure.

4. non-lru movable page flags

   There are two page flags for supporting non-lru movable page.

   • PG_movable

     Driver should use the below function to make page movable under page_lock:

     void __SetPageMovable(struct page *page, struct address_space *mapping)

     It needs argument of address_space for registering migrationfamily functions which will be called by VM. Exactly speaking,PG_movable is not a real flag of struct page. Rather than, VMreuses page->mapping’s lower bits to represent it.

::

#define PAGE_MAPPING_MOVABLE 0x2page->mapping = page->mapping | PAGE_MAPPING_MOVABLE;

so driver shouldn’t access page->mapping directly. Instead, driver shoulduse page_mapping which mask off the low two bits of page->mapping underpage lock so it can get right struct address_space.

For testing of non-lru movable page, VM supports __PageMovable function.However, it doesn’t guarantee to identify non-lru movable page becausepage->mapping field is unified with other variables in struct page.As well, if driver releases the page after isolation by VM, page->mappingdoesn’t have stable value although it has PAGE_MAPPING_MOVABLE(Look at __ClearPageMovable). But __PageMovable is cheap to catch whetherpage is LRU or non-lru movable once the page has been isolated. BecauseLRU pages never can have PAGE_MAPPING_MOVABLE in page->mapping. It is alsogood for just peeking to test non-lru movable pages before more expensivechecking with lock_page in pfn scanning to select victim.

For guaranteeing non-lru movable page, VM provides PageMovable function.Unlike __PageMovable, PageMovable functions validates page->mapping andmapping->a_ops->isolate_page under lock_page. The lock_page prevents suddendestroying of page->mapping.

Driver using __SetPageMovable should clear the flag via __ClearMovablePageunder page_lock before the releasing the page.

• PG_isolated

  To prevent concurrent isolation among several CPUs, VM marks isolated pageas PG_isolated under lock_page. So if a CPU encounters PG_isolated non-lrumovable page, it can skip it. Driver doesn’t need to manipulate the flagbecause VM will set/clear it automatically. Keep in mind that if driversees PG_isolated page, it means the page have been isolated by VM so itshouldn’t touch page.lru field.PG_isolated is alias with PG_reclaim flag so driver shouldn’t use the flagfor own purpose.

Monitoring Migration¶

The following events (counters) can be used to monitor page migration.

1. PGMIGRATE_SUCCESS: Normal page migration success. Each count means that apage was migrated. If the page was a non-THP page, then this counter isincreased by one. If the page was a THP, then this counter is increased bythe number of THP subpages. For example, migration of a single 2MB THP thathas 4KB-size base pages (subpages) will cause this counter to increase by512.
2. PGMIGRATE_FAIL: Normal page migration failure. Same counting rules as for_SUCCESS, above: this will be increased by the number of subpages, if it wasa THP.
3. THP_MIGRATION_SUCCESS: A THP was migrated without being split.
4. THP_MIGRATION_FAIL: A THP could not be migrated nor it could be split.
5. THP_MIGRATION_SPLIT: A THP was migrated, but not as such: first, the THP hadto be split. After splitting, a migration retry was used for it’s sub-pages.

THP_MIGRATION_* events also update the appropriate PGMIGRATE_SUCCESS orPGMIGRATE_FAIL events. For example, a THP migration failure will cause bothTHP_MIGRATION_FAIL and PGMIGRATE_FAIL to increase.

Christoph Lameter, May 8, 2006.Minchan Kim, Mar 28, 2016.