Memory Management APIs¶
User Space Memory Access¶
- get_user¶
get_user(x,ptr)
Get a simple variable from user space.
Parameters
xVariable to store result.
ptrSource address, in user space.
Context
User context only. This function may sleep if pagefaults areenabled.
Description
This macro copies a single simple variable from user space to kernelspace. It supports simple types like char and int, but not largerdata types like structures or arrays.
ptr must have pointer-to-simple-variable type, and the result ofdereferencingptr must be assignable tox without a cast.
Return
zero on success, or -EFAULT on error.On error, the variablex is set to zero.
- __get_user¶
__get_user(x,ptr)
Get a simple variable from user space, with less checking.
Parameters
xVariable to store result.
ptrSource address, in user space.
Context
User context only. This function may sleep if pagefaults areenabled.
Description
This macro copies a single simple variable from user space to kernelspace. It supports simple types like char and int, but not largerdata types like structures or arrays.
ptr must have pointer-to-simple-variable type, and the result ofdereferencingptr must be assignable tox without a cast.
Caller must check the pointer with access_ok() before calling thisfunction.
Return
zero on success, or -EFAULT on error.On error, the variablex is set to zero.
- put_user¶
put_user(x,ptr)
Write a simple value into user space.
Parameters
xValue to copy to user space.
ptrDestination address, in user space.
Context
User context only. This function may sleep if pagefaults areenabled.
Description
This macro copies a single simple value from kernel space to userspace. It supports simple types like char and int, but not largerdata types like structures or arrays.
ptr must have pointer-to-simple-variable type, andx must be assignableto the result of dereferencingptr.
Return
zero on success, or -EFAULT on error.
- __put_user¶
__put_user(x,ptr)
Write a simple value into user space, with less checking.
Parameters
xValue to copy to user space.
ptrDestination address, in user space.
Context
User context only. This function may sleep if pagefaults areenabled.
Description
This macro copies a single simple value from kernel space to userspace. It supports simple types like char and int, but not largerdata types like structures or arrays.
ptr must have pointer-to-simple-variable type, andx must be assignableto the result of dereferencingptr.
Caller must check the pointer with access_ok() before calling thisfunction.
Return
zero on success, or -EFAULT on error.
- unsignedlongclear_user(void__user*to,unsignedlongn)¶
Zero a block of memory in user space.
Parameters
void__user*toDestination address, in user space.
unsignedlongnNumber of bytes to zero.
Description
Zero a block of memory in user space.
Return
number of bytes that could not be cleared.On success, this will be zero.
- unsignedlong__clear_user(void__user*to,unsignedlongn)¶
Zero a block of memory in user space, with less checking.
Parameters
void__user*toDestination address, in user space.
unsignedlongnNumber of bytes to zero.
Description
Zero a block of memory in user space. Caller must checkthe specified block with access_ok() before calling this function.
Return
number of bytes that could not be cleared.On success, this will be zero.
- intget_user_pages_fast(unsignedlongstart,intnr_pages,unsignedintgup_flags,structpage**pages)¶
pin user pages in memory
Parameters
unsignedlongstartstarting user address
intnr_pagesnumber of pages from start to pin
unsignedintgup_flagsflags modifying pin behaviour
structpage**pagesarray that receives pointers to the pages pinned.Should be at least nr_pages long.
Description
Attempt to pin user pages in memory without taking mm->mmap_lock.If not successful, it will fall back to taking the lock andcalling get_user_pages().
Returns number of pages pinned. This may be fewer than the number requested.If nr_pages is 0 or negative, returns 0. If no pages were pinned, returns-errno.
Memory Allocation Controls¶
Page mobility and placement hints¶
These flags provide hints about how mobile the page is. Pages with similarmobility are placed within the same pageblocks to minimise problems dueto external fragmentation.
__GFP_MOVABLE (also a zone modifier) indicates that the page can bemoved by page migration during memory compaction or can be reclaimed.
__GFP_RECLAIMABLE is used for slab allocations that specifySLAB_RECLAIM_ACCOUNT and whose pages can be freed via shrinkers.
__GFP_WRITE indicates the caller intends to dirty the page. Where possible,these pages will be spread between local zones to avoid all the dirtypages being in one zone (fair zone allocation policy).
__GFP_HARDWALL enforces the cpuset memory allocation policy.
__GFP_THISNODE forces the allocation to be satisfied from the requestednode with no fallbacks or placement policy enforcements.
__GFP_ACCOUNT causes the allocation to be accounted to kmemcg.
__GFP_NO_OBJ_EXT causes slab allocation to have no object extension.
Watermark modifiers -- controls access to emergency reserves¶
__GFP_HIGH indicates that the caller is high-priority and that grantingthe request is necessary before the system can make forward progress.For example creating an IO context to clean pages and requestsfrom atomic context.
__GFP_MEMALLOC allows access to all memory. This should only be used whenthe caller guarantees the allocation will allow more memory to be freedvery shortly e.g. process exiting or swapping. Users either shouldbe the MM or co-ordinating closely with the VM (e.g. swap over NFS).Users of this flag have to be extremely careful to not deplete the reservecompletely and implement a throttling mechanism which controls theconsumption of the reserve based on the amount of freed memory.Usage of a pre-allocated pool (e.g. mempool) should be always consideredbefore using this flag.
__GFP_NOMEMALLOC is used to explicitly forbid access to emergency reserves.This takes precedence over the__GFP_MEMALLOC flag if both are set.
Reclaim modifiers¶
Please note that all the following flags are only applicable to sleepableallocations (e.g.GFP_NOWAIT andGFP_ATOMIC will ignore them).
__GFP_IO can start physical IO.
__GFP_FS can call down to the low-level FS. Clearing the flag avoids theallocator recursing into the filesystem which might already be holdinglocks.
__GFP_DIRECT_RECLAIM indicates that the caller may enter direct reclaim.This flag can be cleared to avoid unnecessary delays when a fallbackoption is available.
__GFP_KSWAPD_RECLAIM indicates that the caller wants to wake kswapd whenthe low watermark is reached and have it reclaim pages until the highwatermark is reached. A caller may wish to clear this flag when fallbackoptions are available and the reclaim is likely to disrupt the system. Thecanonical example is THP allocation where a fallback is cheap butreclaim/compaction may cause indirect stalls.
__GFP_RECLAIM is shorthand to allow/forbid both direct and kswapd reclaim.
The default allocator behavior depends on the request size. We have a conceptof so-called costly allocations (with order >PAGE_ALLOC_COSTLY_ORDER).!costly allocations are too essential to fail so they are implicitlynon-failing by default (with some exceptions like OOM victims might fail sothe caller still has to check for failures) while costly requests try to benot disruptive and back off even without invoking the OOM killer.The following three modifiers might be used to override some of theseimplicit rules. Please note that all of them must be used along with__GFP_DIRECT_RECLAIM flag.
__GFP_NORETRY: The VM implementation will try only very lightweightmemory direct reclaim to get some memory under memory pressure (thusit can sleep). It will avoid disruptive actions like OOM killer. Thecaller must handle the failure which is quite likely to happen underheavy memory pressure. The flag is suitable when failure can easily behandled at small cost, such as reduced throughput.
__GFP_RETRY_MAYFAIL: The VM implementation will retry memory reclaimprocedures that have previously failed if there is some indicationthat progress has been made elsewhere. It can wait for othertasks to attempt high-level approaches to freeing memory such ascompaction (which removes fragmentation) and page-out.There is still a definite limit to the number of retries, but it isa larger limit than with__GFP_NORETRY.Allocations with this flag may fail, but only when there isgenuinely little unused memory. While these allocations do notdirectly trigger the OOM killer, their failure indicates thatthe system is likely to need to use the OOM killer soon. Thecaller must handle failure, but can reasonably do so by failinga higher-level request, or completing it only in a much lessefficient manner.If the allocation does fail, and the caller is in a position tofree some non-essential memory, doing so could benefit the systemas a whole.
__GFP_NOFAIL: The VM implementation _must_ retry infinitely: the callercannot handle allocation failures. The allocation could blockindefinitely but will never return with failure. Testing forfailure is pointless.It _must_ be blockable and used together with __GFP_DIRECT_RECLAIM.It should _never_ be used in non-sleepable contexts.New users should be evaluated carefully (and the flag should beused only when there is no reasonable failure policy) but it isdefinitely preferable to use the flag rather than opencode endlessloop around allocator.Allocating pages from the buddy with __GFP_NOFAIL and order > 1 isnot supported. Please consider using kvmalloc() instead.
Useful GFP flag combinations¶
Useful GFP flag combinations that are commonly used. It is recommendedthat subsystems start with one of these combinations and then set/clear__GFP_FOO flags as necessary.
GFP_ATOMIC users can not sleep and need the allocation to succeed. A lowerwatermark is applied to allow access to “atomic reserves”.The current implementation doesn’t support NMI and few other strictnon-preemptive contexts (e.g. raw_spin_lock). The same applies toGFP_NOWAIT.
GFP_KERNEL is typical for kernel-internal allocations. The caller requiresZONE_NORMAL or a lower zone for direct access but can direct reclaim.
GFP_KERNEL_ACCOUNT is the same as GFP_KERNEL, except the allocation isaccounted to kmemcg.
GFP_NOWAIT is for kernel allocations that should not stall for directreclaim, start physical IO or use any filesystem callback. It is verylikely to fail to allocate memory, even for very small allocations.
GFP_NOIO will use direct reclaim to discard clean pages or slab pagesthat do not require the starting of any physical IO.Please try to avoid using this flag directly and instead usememalloc_noio_{save,restore} to mark the whole scope which cannotperform any IO with a short explanation why. All allocation requestswill inherit GFP_NOIO implicitly.
GFP_NOFS will use direct reclaim but will not use any filesystem interfaces.Please try to avoid using this flag directly and instead usememalloc_nofs_{save,restore} to mark the whole scope which cannot/shouldn’trecurse into the FS layer with a short explanation why. All allocationrequests will inherit GFP_NOFS implicitly.
GFP_USER is for userspace allocations that also need to be directlyaccessibly by the kernel or hardware. It is typically used by hardwarefor buffers that are mapped to userspace (e.g. graphics) that hardwarestill must DMA to. cpuset limits are enforced for these allocations.
GFP_DMA exists for historical reasons and should be avoided where possible.The flags indicates that the caller requires that the lowest zone beused (ZONE_DMA or 16M on x86-64). Ideally, this would be removed butit would require careful auditing as some users really require it andothers use the flag to avoid lowmem reserves inZONE_DMA and treat thelowest zone as a type of emergency reserve.
GFP_DMA32 is similar toGFP_DMA except that the caller requires a 32-bitaddress. Note that kmalloc(..., GFP_DMA32) does not return DMA32 memorybecause the DMA32 kmalloc cache array is not implemented.(Reason: there is no such user in kernel).
GFP_HIGHUSER is for userspace allocations that may be mapped to userspace,do not need to be directly accessible by the kernel but that cannotmove once in use. An example may be a hardware allocation that mapsdata directly into userspace but has no addressing limitations.
GFP_HIGHUSER_MOVABLE is for userspace allocations that the kernel does notneed direct access to but can usekmap() when access is required. Theyare expected to be movable via page reclaim or page migration. Typically,pages on the LRU would also be allocated withGFP_HIGHUSER_MOVABLE.
GFP_TRANSHUGE andGFP_TRANSHUGE_LIGHT are used for THP allocations. Theyare compound allocations that will generally fail quickly if memory is notavailable and will not wake kswapd/kcompactd on failure. The _LIGHTversion does not attempt reclaim/compaction at all and is by default usedin page fault path, while the non-light is used by khugepaged.
The Slab Cache¶
- SLAB_HWCACHE_ALIGN¶
SLAB_HWCACHE_ALIGN
Align objects on cache line boundaries.
Description
Sufficiently large objects are aligned on cache line boundary. For objectsize smaller than a half of cache line size, the alignment is on the half ofcache line size. In general, if object size is smaller than 1/2^n of cacheline size, the alignment is adjusted to 1/2^n.
If explicit alignment is also requested by the respective
structkmem_cache_argsfield, the greater of both is alignments is applied.
- SLAB_TYPESAFE_BY_RCU¶
SLAB_TYPESAFE_BY_RCU
WARNING READ THIS!
Description
This delays freeing the SLAB page by a grace period, it does _NOT_delay object freeing. This means that if you do
kmem_cache_free()that memory location is free to be reused at any time. Thus it maybe possible to see another object there in the same RCU grace period.This feature only ensures the memory location backing the objectstays valid, the trick to using this is relying on an independentobject validation pass. Something like:
begin: rcu_read_lock(); obj = lockless_lookup(key); if (obj) { if (!try_get_ref(obj)) // might fail for free objects rcu_read_unlock(); goto begin; if (obj->key != key) { // not the object we expected put_ref(obj); rcu_read_unlock(); goto begin; } }rcu_read_unlock();This is useful if we need to approach a kernel structure obliquely,from its address obtained without the usual locking. We can lockthe structure to stabilize it and check it’s still at the given address,only if we can be sure that the memory has not been meanwhile reusedfor some other kind of object (which our subsystem’s lock might corrupt).
rcu_read_lock before reading the address, then rcu_read_unlock aftertaking the spinlock within the structure expected at that address.
Note that object identity check has to be doneafter acquiring areference, therefore user has to ensure proper ordering for loads.Similarly, when initializing objects allocated with SLAB_TYPESAFE_BY_RCU,the newly allocated object has to be fully initializedbefore itsrefcount gets initialized and proper ordering for stores is required.refcount_{add|inc}_not_zero_acquire() and
refcount_set_release()aredesigned with the proper fences required for reference counting objectsallocated with SLAB_TYPESAFE_BY_RCU.Note that it is not possible to acquire a lock within a structureallocated with SLAB_TYPESAFE_BY_RCU without first acquiring a referenceas described above. The reason is that SLAB_TYPESAFE_BY_RCU pagesare not zeroed before being given to the slab, which means that anylocks must be initialized after each and every kmem_struct_alloc().Alternatively, make the ctor passed to
kmem_cache_create()initializethe locks at page-allocation time, as is done in __i915_request_ctor(),sighand_ctor(), and anon_vma_ctor(). Such a ctor permits readersto safely acquire those ctor-initialized locks underrcu_read_lock()protection.Note that SLAB_TYPESAFE_BY_RCU was originally named SLAB_DESTROY_BY_RCU.
- SLAB_ACCOUNT¶
SLAB_ACCOUNT
Account allocations to memcg.
Description
All object allocations from this cache will be memcg accounted, regardless of__GFP_ACCOUNT being or not being passed to individual allocations.
- SLAB_RECLAIM_ACCOUNT¶
SLAB_RECLAIM_ACCOUNT
Objects are reclaimable.
Description
Use this flag for caches that have an associated shrinker. As a result, slabpages are allocated with __GFP_RECLAIMABLE, which affects grouping pages bymobility, and are accounted in SReclaimable counter in /proc/meminfo
- structkmem_cache_args¶
Less common arguments for
kmem_cache_create()
Definition:
struct kmem_cache_args { unsigned int align; unsigned int useroffset; unsigned int usersize; unsigned int freeptr_offset; bool use_freeptr_offset; void (*ctor)(void *);};Members
alignThe required alignment for the objects.
0means no specific alignment is requested.useroffsetUsercopy region offset.
0is a valid offset, whenusersize is non-0usersizeUsercopy region size.
0means no usercopy region is specified.freeptr_offsetCustom offset for the free pointerin
SLAB_TYPESAFE_BY_RCUcachesBy default
SLAB_TYPESAFE_BY_RCUcaches place the free pointeroutside of the object. This might cause the object to grow in size.Cache creators that have a reason to avoid this can specify a customfree pointer offset in their struct where the free pointer will beplaced.Note that placing the free pointer inside the object requires thecaller to ensure that no fields are invalidated that are required toguard against object recycling (See
SLAB_TYPESAFE_BY_RCUfordetails).Using
0as a value forfreeptr_offset is valid. Iffreeptr_offsetis specified,use_freeptr_offsetmust be settrue.Note thatctor currently isn’t supported with custom free pointersas actor requires an external free pointer.
use_freeptr_offsetWhether afreeptr_offset is used.
ctorA constructor for the objects.
The constructor is invoked for each object in a newly allocated slabpage. It is the cache user’s responsibility to free object in thesame state as after calling the constructor, or deal appropriatelywith any differences between a freshly constructed and a reallocatedobject.
NULLmeans no constructor.
Description
Any uninitialized fields of the structure are interpreted as unused. Theexception isfreeptr_offset where0 is a valid value, souse_freeptr_offset must be also set totrue in order to interpret the fieldas used. Foruseroffset0 is also valid, but only with non-0usersize.
WhenNULL args is passed tokmem_cache_create(), it is equivalent to allfields unused.
- structkmem_cache*kmem_cache_create_usercopy(constchar*name,unsignedintsize,unsignedintalign,slab_flags_tflags,unsignedintuseroffset,unsignedintusersize,void(*ctor)(void*))¶
Create a kmem cache with a region suitable for copying to userspace.
Parameters
constchar*nameA string which is used in /proc/slabinfo to identify this cache.
unsignedintsizeThe size of objects to be created in this cache.
unsignedintalignThe required alignment for the objects.
slab_flags_tflagsSLAB flags
unsignedintuseroffsetUsercopy region offset
unsignedintusersizeUsercopy region size
void(*ctor)(void*)A constructor for the objects, or
NULL.
Description
This is a legacy wrapper, new code should use either KMEM_CACHE_USERCOPY()if whitelisting a single field is sufficient, orkmem_cache_create() withthe necessary parameters passed via the args parameter (seestructkmem_cache_args)
Return
a pointer to the cache on success, NULL on failure.
- kmem_cache_create¶
kmem_cache_create(__name,__object_size,__args,...)
Create a kmem cache.
Parameters
__nameA string which is used in /proc/slabinfo to identify this cache.
__object_sizeThe size of objects to be created in this cache.
__argsOptional arguments, see
structkmem_cache_args. PassingNULLmeans defaults will be used for all the arguments....variable arguments
Description
This is currently implemented as a macro using_Generic() to calleither the new variant of the function, or a legacy one.
The new variant has 4 parameters:kmem_cache_create(name,object_size,args,flags)
See__kmem_cache_create_args() which implements this.
The legacy variant has 5 parameters:kmem_cache_create(name,object_size,align,flags,ctor)
The align and ctor parameters map to the respective fields ofstructkmem_cache_args
Context
Cannot be called within a interrupt, but can be interrupted.
Return
a pointer to the cache on success, NULL on failure.
- size_tksize(constvoid*objp)¶
Report actual allocation size of associated object
Parameters
constvoid*objpPointer returned from a prior
kmalloc()-family allocation.
Description
This should not be used for writing beyond the originally requestedallocation size. Either use krealloc() or round up the allocation sizewithkmalloc_size_roundup() prior to allocation. If this is used toaccess beyond the originally requested allocation size, UBSAN_BOUNDSand/or FORTIFY_SOURCE may trip, since they only know about theoriginally allocated size via the __alloc_size attribute.
- void*kmem_cache_alloc(structkmem_cache*cachep,gfp_tflags)¶
Allocate an object
Parameters
structkmem_cache*cachepThe cache to allocate from.
gfp_tflagsSee
kmalloc().
Description
Allocate an object from this cache.See kmem_cache_zalloc() for a shortcut of adding __GFP_ZERO to flags.
Return
pointer to the new object orNULL in case of error
- boolkmem_cache_charge(void*objp,gfp_tgfpflags)¶
memcg charge an already allocated slab memory
Parameters
void*objpaddress of the slab object to memcg charge
gfp_tgfpflagsdescribe the allocation context
Description
kmem_cache_charge allows charging a slab object to the current memcg,primarily in cases where charging at allocation time might not be possiblebecause the target memcg is not known (i.e. softirq context)
The objp should be pointer returned by the slab allocator functions likekmalloc (with __GFP_ACCOUNT in flags) or kmem_cache_alloc. The memcg chargebehavior can be controlled through gfpflags parameter, which affects how thenecessary internal metadata can be allocated. Including __GFP_NOFAIL denotesthat overcharging is requested instead of failure, but is not applied for theinternal metadata allocation.
There are several cases where it will return true even if the charging wasnot done:More specifically:
For !CONFIG_MEMCG or cgroup_disable=memory systems.
Already charged slab objects.
For slab objects from KMALLOC_NORMAL caches - allocated by
kmalloc()without __GFP_ACCOUNTAllocating internal metadata has failed
Return
true if charge was successful otherwise false.
- void*kmalloc(size_tsize,gfp_tflags)¶
allocate kernel memory
Parameters
size_tsizehow many bytes of memory are required.
gfp_tflagsdescribe the allocation context
Description
kmalloc is the normal method of allocating memoryfor objects smaller than page size in the kernel.
The allocated object address is aligned to at least ARCH_KMALLOC_MINALIGNbytes. Forsize of power of two bytes, the alignment is also guaranteedto be at least to the size. For other sizes, the alignment is guaranteed tobe at least the largest power-of-two divisor ofsize.
Theflags argument may be one of the GFP flags defined atinclude/linux/gfp_types.h and described atDocumentation/core-api/mm-api.rst
The recommended usage of theflags is described atDocumentation/core-api/memory-allocation.rst
Below is a brief outline of the most useful GFP flags
GFP_KERNELAllocate normal kernel ram. May sleep.
GFP_NOWAITAllocation will not sleep.
GFP_ATOMICAllocation will not sleep. May use emergency pools.
Also it is possible to set different flags by OR’ingin one or more of the following additionalflags:
__GFP_ZEROZero the allocated memory before returning. Also see
kzalloc().__GFP_HIGHThis allocation has high priority and may use emergency pools.
__GFP_NOFAILIndicate that this allocation is in no way allowed to fail(think twice before using).
__GFP_NORETRYIf memory is not immediately available,then give up at once.
__GFP_NOWARNIf allocation fails, don’t issue any warnings.
__GFP_RETRY_MAYFAILTry really hard to succeed the allocation but faileventually.
- void*kmalloc_array(size_tn,size_tsize,gfp_tflags)¶
allocate memory for an array.
Parameters
size_tnnumber of elements.
size_tsizeelement size.
gfp_tflagsthe type of memory to allocate (see kmalloc).
- void*krealloc_array(void*p,size_tnew_n,size_tnew_size,gfp_tflags)¶
reallocate memory for an array.
Parameters
void*ppointer to the memory chunk to reallocate
size_tnew_nnew number of elements to alloc
size_tnew_sizenew size of a single member of the array
gfp_tflagsthe type of memory to allocate (see kmalloc)
Description
If __GFP_ZERO logic is requested, callers must ensure that, starting with theinitial memory allocation, every subsequent call to this API for the samememory allocation is flagged with __GFP_ZERO. Otherwise, it is possible that__GFP_ZERO is not fully honored by this API.
See krealloc_noprof() for further details.
In any case, the contents of the object pointed to are preserved up to thelesser of the new and old sizes.
- kcalloc¶
kcalloc(n,size,flags)
allocate memory for an array. The memory is set to zero.
Parameters
nnumber of elements.
sizeelement size.
flagsthe type of memory to allocate (see kmalloc).
- void*kzalloc(size_tsize,gfp_tflags)¶
allocate memory. The memory is set to zero.
Parameters
size_tsizehow many bytes of memory are required.
gfp_tflagsthe type of memory to allocate (see kmalloc).
- size_tkmalloc_size_roundup(size_tsize)¶
Report allocation bucket size for the given size
Parameters
size_tsizeNumber of bytes to round up from.
Description
This returns the number of bytes that would be available in akmalloc()allocation ofsize bytes. For example, a 126 byte request would berounded up to the next sized kmalloc bucket, 128 bytes. (This is strictlyfor the general-purposekmalloc()-based allocations, and is not for thepre-sizedkmem_cache_alloc()-based allocations.)
Use this tokmalloc() the full bucket size ahead of time instead of usingksize() to query the size after an allocation.
- voidkmem_cache_free(structkmem_cache*s,void*x)¶
Deallocate an object
Parameters
structkmem_cache*sThe cache the allocation was from.
void*xThe previously allocated object.
Description
Free an object which was previously allocated from thiscache.
- voidkfree(constvoid*object)¶
free previously allocated memory
Parameters
constvoid*objectpointer returned by
kmalloc()orkmem_cache_alloc()
Description
Ifobject is NULL, no operation is performed.
- voidkvfree(constvoid*addr)¶
Free memory.
Parameters
constvoid*addrPointer to allocated memory.
Description
kvfree frees memory allocated by any of vmalloc(),kmalloc() or kvmalloc().It is slightly more efficient to usekfree() orvfree() if you are certainthat you know which one to use.
Context
Either preemptible task context or not-NMI interrupt.
- voidkvfree_sensitive(constvoid*addr,size_tlen)¶
Free a data object containing sensitive information.
Parameters
constvoid*addraddress of the data object to be freed.
size_tlenlength of the data object.
Description
Use the specialmemzero_explicit() function to clear the content of akvmalloc’ed object containing sensitive data to make sure that thecompiler won’t optimize out the data clearing.
- structkmem_cache*__kmem_cache_create_args(constchar*name,unsignedintobject_size,structkmem_cache_args*args,slab_flags_tflags)¶
Create a kmem cache.
Parameters
constchar*nameA string which is used in /proc/slabinfo to identify this cache.
unsignedintobject_sizeThe size of objects to be created in this cache.
structkmem_cache_args*argsAdditional arguments for the cache creation (see
structkmem_cache_args).slab_flags_tflagsSee the desriptions of individual flags. The common ones are listedin the description below.
Description
Not to be called directly, use thekmem_cache_create() wrapper with the sameparameters.
Commonly usedflags:
SLAB_ACCOUNT - Account allocations to memcg.
SLAB_HWCACHE_ALIGN - Align objects on cache line boundaries.
SLAB_RECLAIM_ACCOUNT - Objects are reclaimable.
SLAB_TYPESAFE_BY_RCU - Slab page (not individual objects) freeing delayedby a grace period - see the full description before using.
Context
Cannot be called within a interrupt, but can be interrupted.
Return
a pointer to the cache on success, NULL on failure.
- kmem_buckets*kmem_buckets_create(constchar*name,slab_flags_tflags,unsignedintuseroffset,unsignedintusersize,void(*ctor)(void*))¶
Create a set of caches that handle dynamic sized allocations via kmem_buckets_alloc()
Parameters
constchar*nameA prefix string which is used in /proc/slabinfo to identify thiscache. The individual caches with have their sizes as the suffix.
slab_flags_tflagsSLAB flags (see
kmem_cache_create()for details).unsignedintuseroffsetStarting offset within an allocation that may be copiedto/from userspace.
unsignedintusersizeHow many bytes, starting atuseroffset, may be copiedto/from userspace.
void(*ctor)(void*)A constructor for the objects, run when new allocations are made.
Description
Cannot be called within an interrupt, but can be interrupted.
Return
a pointer to the cache on success, NULL on failure. WhenCONFIG_SLAB_BUCKETS is not enabled, ZERO_SIZE_PTR is returned, andsubsequent calls to kmem_buckets_alloc() will fall back tokmalloc().(i.e. callers only need to check for NULL on failure.)
- intkmem_cache_shrink(structkmem_cache*cachep)¶
Shrink a cache.
Parameters
structkmem_cache*cachepThe cache to shrink.
Description
Releases as many slabs as possible for a cache.To help debugging, a zero exit status indicates all slabs were released.
Return
0 if all slabs were released, non-zero otherwise
- boolkmem_dump_obj(void*object)¶
Print available slab provenance information
Parameters
void*objectslab object for which to find provenance information.
Description
This function usespr_cont(), so that the caller is expected to haveprinted out whatever preamble is appropriate. The provenance informationdepends on the type of object and on how much debugging is enabled.For a slab-cache object, the fact that it is a slab object is printed,and, if available, the slab name, return address, and stack trace fromthe allocation and last free path of that object.
Return
true if the pointer is to a not-yet-freed object fromkmalloc() orkmem_cache_alloc(), eithertrue orfalse if the pointeris to an already-freed object, andfalse otherwise.
- voidkfree_sensitive(constvoid*p)¶
Clear sensitive information in memory before freeing
Parameters
constvoid*pobject to free memory of
Description
The memory of the objectp points to is zeroed before freed.Ifp isNULL,kfree_sensitive() does nothing.
Note
this function zeroes the whole allocated buffer which can be a gooddeal bigger than the requested buffer size passed tokmalloc(). So becareful when using this function in performance sensitive code.
- voidkvfree_rcu_barrier(void)¶
Wait until all in-flight kvfree_rcu() complete.
Parameters
voidno arguments
Description
Note that a single argument of kvfree_rcu() call has a slow path thattriggerssynchronize_rcu() following by freeing a pointer. It is donebefore the return from the function. Therefore for any single-argumentcall that will result in akfree() to a cache that is to be destroyedduring module exit, it is developer’s responsibility to ensure that allsuch calls have returned before the call to kmem_cache_destroy().
- voidkfree_const(constvoid*x)¶
conditionally free memory
Parameters
constvoid*xpointer to the memory
Description
Function calls kfree only ifx is not in .rodata section.
Virtually Contiguous Mappings¶
- voidvm_unmap_aliases(void)¶
unmap outstanding lazy aliases in the vmap layer
Parameters
voidno arguments
Description
The vmap/vmalloc layer lazily flushes kernel virtual mappings primarilyto amortize TLB flushing overheads. What this means is that any page youhave now, may, in a former life, have been mapped into kernel virtualaddress by the vmap layer and so there might be some CPUs with TLB entriesstill referencing that page (additional to the regular 1:1 kernel mapping).
vm_unmap_aliases flushes all such lazy mappings. After it returns, we canbe sure that none of the pages we have control over will have any aliasesfrom the vmap layer.
- voidvm_unmap_ram(constvoid*mem,unsignedintcount)¶
unmap linear kernel address space set up by vm_map_ram
Parameters
constvoid*memthe pointer returned by vm_map_ram
unsignedintcountthe count passed to that vm_map_ram call (cannot unmap partial)
- void*vm_map_ram(structpage**pages,unsignedintcount,intnode)¶
map pages linearly into kernel virtual address (vmalloc space)
Parameters
structpage**pagesan array of pointers to the pages to be mapped
unsignedintcountnumber of pages
intnodeprefer to allocate data structures on this node
Description
If you use this function for less than VMAP_MAX_ALLOC pages, it could befaster than vmap so it’s good. But if you mix long-life and short-lifeobjects withvm_map_ram(), it could consume lots of address space throughfragmentation (especially on a 32bit machine). You could see failures inthe end. Please use this function for short-lived objects.
Return
a pointer to the address that has been mapped, orNULL on failure
- voidvfree(constvoid*addr)¶
Release memory allocated by vmalloc()
Parameters
constvoid*addrMemory base address
Description
Free the virtually continuous memory area starting ataddr, as obtainedfrom one of the vmalloc() family of APIs. This will usually also free thephysical memory underlying the virtual allocation, but that memory isreference counted, so it will not be freed until the last user goes away.
Ifaddr is NULL, no operation is performed.
Context
May sleep if callednot from interrupt context.Must not be called in NMI context (strictly speaking, it could beif we have CONFIG_ARCH_HAVE_NMI_SAFE_CMPXCHG, but making the callingconventions forvfree() arch-dependent would be a really bad idea).
Parameters
constvoid*addrmemory base address
Description
Free the virtually contiguous memory area starting ataddr,which was created from the page array passed tovmap().
Must not be called in interrupt context.
- void*vmap(structpage**pages,unsignedintcount,unsignedlongflags,pgprot_tprot)¶
map an array of pages into virtually contiguous space
Parameters
structpage**pagesarray of page pointers
unsignedintcountnumber of pages to map
unsignedlongflagsvm_area->flags
pgprot_tprotpage protection for the mapping
Description
Mapscount pages frompages into contiguous kernel virtual space.Ifflags containsVM_MAP_PUT_PAGES the ownership of the pages array itself(which must be kmalloc or vmalloc memory) and one reference per pages in itare transferred from the caller tovmap(), and will be freed / dropped whenvfree() is called on the return value.
Return
the address of the area orNULL on failure
- void*vmap_pfn(unsignedlong*pfns,unsignedintcount,pgprot_tprot)¶
map an array of PFNs into virtually contiguous space
Parameters
unsignedlong*pfnsarray of PFNs
unsignedintcountnumber of pages to map
pgprot_tprotpage protection for the mapping
Description
Mapscount PFNs frompfns into contiguous kernel virtual space and returnsthe start address of the mapping.
- intremap_vmalloc_range(structvm_area_struct*vma,void*addr,unsignedlongpgoff)¶
map vmalloc pages to userspace
Parameters
structvm_area_struct*vmavma to cover (map full range of vma)
void*addrvmalloc memory
unsignedlongpgoffnumber of pages into addr before first page to map
Return
0 for success, -Exxx on failure
Description
This function checks that addr is a valid vmalloc’ed area, andthat it is big enough to cover the vma. Will return failure ifthat criteria isn’t met.
Similar toremap_pfn_range() (see mm/memory.c)
File Mapping and Page Cache¶
Filemap¶
- intfilemap_fdatawrite_wbc(structaddress_space*mapping,structwriteback_control*wbc)¶
start writeback on mapping dirty pages in range
Parameters
structaddress_space*mappingaddress space structure to write
structwriteback_control*wbcthe writeback_control controlling the writeout
Description
Call writepages on the mapping using the provided wbc to control thewriteout.
Return
0 on success, negative error code otherwise.
- intfilemap_fdatawrite_range_kick(structaddress_space*mapping,loff_tstart,loff_tend)¶
start writeback on a range
Parameters
structaddress_space*mappingtarget address_space
loff_tstartindex to start writeback on
loff_tendlast (inclusive) index for writeback
Description
This is a non-integrity writeback helper, to start writing back foliosfor the indicated range.
Return
0 on success, negative error code otherwise.
- intfilemap_flush(structaddress_space*mapping)¶
mostly a non-blocking flush
Parameters
structaddress_space*mappingtarget address_space
Description
This is a mostly non-blocking flush. Not suitable for data-integritypurposes - I/O may not be started against all dirty pages.
Return
0 on success, negative error code otherwise.
- boolfilemap_range_has_page(structaddress_space*mapping,loff_tstart_byte,loff_tend_byte)¶
check if a page exists in range.
Parameters
structaddress_space*mappingaddress space within which to check
loff_tstart_byteoffset in bytes where the range starts
loff_tend_byteoffset in bytes where the range ends (inclusive)
Description
Find at least one page in the range supplied, usually used to check ifdirect writing in this range will trigger a writeback.
Return
true if at least one page exists in the specified range,false otherwise.
- intfilemap_fdatawait_range(structaddress_space*mapping,loff_tstart_byte,loff_tend_byte)¶
wait for writeback to complete
Parameters
structaddress_space*mappingaddress space structure to wait for
loff_tstart_byteoffset in bytes where the range starts
loff_tend_byteoffset in bytes where the range ends (inclusive)
Description
Walk the list of under-writeback pages of the given address spacein the given range and wait for all of them. Check error status ofthe address space and return it.
Since the error status of the address space is cleared by this function,callers are responsible for checking the return value and handling and/orreporting the error.
Return
error status of the address space.
- intfilemap_fdatawait_range_keep_errors(structaddress_space*mapping,loff_tstart_byte,loff_tend_byte)¶
wait for writeback to complete
Parameters
structaddress_space*mappingaddress space structure to wait for
loff_tstart_byteoffset in bytes where the range starts
loff_tend_byteoffset in bytes where the range ends (inclusive)
Description
Walk the list of under-writeback pages of the given address space in thegiven range and wait for all of them. Unlikefilemap_fdatawait_range(),this function does not clear error status of the address space.
Use this function if callers don’t handle errors themselves. Expectedcall sites are system-wide / filesystem-wide data flushers: e.g. sync(2),fsfreeze(8)
- intfile_fdatawait_range(structfile*file,loff_tstart_byte,loff_tend_byte)¶
wait for writeback to complete
Parameters
structfile*filefile pointing to address space structure to wait for
loff_tstart_byteoffset in bytes where the range starts
loff_tend_byteoffset in bytes where the range ends (inclusive)
Description
Walk the list of under-writeback pages of the address space that filerefers to, in the given range and wait for all of them. Check errorstatus of the address space vs. the file->f_wb_err cursor and return it.
Since the error status of the file is advanced by this function,callers are responsible for checking the return value and handling and/orreporting the error.
Return
error status of the address space vs. the file->f_wb_err cursor.
- intfilemap_fdatawait_keep_errors(structaddress_space*mapping)¶
wait for writeback without clearing errors
Parameters
structaddress_space*mappingaddress space structure to wait for
Description
Walk the list of under-writeback pages of the given address spaceand wait for all of them. Unlike filemap_fdatawait(), this functiondoes not clear error status of the address space.
Use this function if callers don’t handle errors themselves. Expectedcall sites are system-wide / filesystem-wide data flushers: e.g. sync(2),fsfreeze(8)
Return
error status of the address space.
- intfilemap_write_and_wait_range(structaddress_space*mapping,loff_tlstart,loff_tlend)¶
write out & wait on a file range
Parameters
structaddress_space*mappingthe address_space for the pages
loff_tlstartoffset in bytes where the range starts
loff_tlendoffset in bytes where the range ends (inclusive)
Description
Write out and wait upon file offsets lstart->lend, inclusive.
Note thatlend is inclusive (describes the last byte to be written) sothat this function can be used to write to the very end-of-file (end = -1).
Return
error status of the address space.
- intfile_check_and_advance_wb_err(structfile*file)¶
report wb error (if any) that was previously and advance wb_err to current one
Parameters
structfile*filestructfileon which the error is being reported
Description
When userland calls fsync (or something like nfsd does the equivalent), wewant to report any writeback errors that occurred since the last fsync (orsince the file was opened if there haven’t been any).
Grab the wb_err from the mapping. If it matches what we have in the file,then just quickly return 0. The file is all caught up.
If it doesn’t match, then take the mapping value, set the “seen” flag init and try to swap it into place. If it works, or another task beat usto it with the new value, then update the f_wb_err and return the errorportion. The error at this point must be reported via proper channels(a’la fsync, or NFS COMMIT operation, etc.).
While we handle mapping->wb_err with atomic operations, the f_wb_errvalue is protected by the f_lock since we must ensure that it reflectsthe latest value swapped in for this file descriptor.
Return
0 on success, negative error code otherwise.
- intfile_write_and_wait_range(structfile*file,loff_tlstart,loff_tlend)¶
write out & wait on a file range
Parameters
structfile*filefile pointing to address_space with pages
loff_tlstartoffset in bytes where the range starts
loff_tlendoffset in bytes where the range ends (inclusive)
Description
Write out and wait upon file offsets lstart->lend, inclusive.
Note thatlend is inclusive (describes the last byte to be written) sothat this function can be used to write to the very end-of-file (end = -1).
After writing out and waiting on the data, we check and advance thef_wb_err cursor to the latest value, and return any errors detected there.
Return
0 on success, negative error code otherwise.
- voidreplace_page_cache_folio(structfolio*old,structfolio*new)¶
replace a pagecache folio with a new one
Parameters
structfolio*oldfolio to be replaced
structfolio*newfolio to replace with
Description
This function replaces a folio in the pagecache with a new one. Onsuccess it acquires the pagecache reference for the new folio anddrops it for the old folio. Both the old and new folios must belocked. This function does not add the new folio to the LRU, thecaller must do that.
The remove + add is atomic. This function cannot fail.
Parameters
structfolio*folioThe folio.
Description
Unlocks the folio and wakes up any thread sleeping on the page lock.
Context
May be called from interrupt or process context. May not becalled from NMI context.
Parameters
structfolio*folioThe folio.
boolsuccessTrue if all reads completed successfully.
Description
When all reads against a folio have completed, filesystems shouldcall this function to let the pagecache know that no more readsare outstanding. This will unlock the folio and wake up any threadsleeping on the lock. The folio will also be marked uptodate if allreads succeeded.
Context
May be called from interrupt or process context. May not becalled from NMI context.
Parameters
structfolio*folioThe folio.
Description
Clear the PG_private_2 bit on a folio and wake up any sleepers waiting forit. The folio reference held for PG_private_2 being set is released.
This is, for example, used when a netfs folio is being written to a localdisk cache, thereby allowing writes to the cache for the same folio to beserialised.
Parameters
structfolio*folioThe folio to wait on.
Description
Wait for PG_private_2 to be cleared on a folio.
Parameters
structfolio*folioThe folio to wait on.
Description
Wait for PG_private_2 to be cleared on a folio or until a fatal signal isreceived by the calling task.
Return
0 if successful.
-EINTR if a fatal signal was encountered.
Parameters
structfolio*folioThe folio.
Description
The folio must actually be under writeback.
Context
May be called from process or interrupt context.
Parameters
structfolio*folioThe folio to lock
- pgoff_tpage_cache_next_miss(structaddress_space*mapping,pgoff_tindex,unsignedlongmax_scan)¶
Find the next gap in the page cache.
Parameters
structaddress_space*mappingMapping.
pgoff_tindexIndex.
unsignedlongmax_scanMaximum range to search.
Description
Search the range [index, min(index + max_scan - 1, ULONG_MAX)] for thegap with the lowest index.
This function may be called under the rcu_read_lock. However, this willnot atomically search a snapshot of the cache at a single point in time.For example, if a gap is created at index 5, then subsequently a gap iscreated at index 10, page_cache_next_miss covering both indices mayreturn 10 if called under the rcu_read_lock.
Return
The index of the gap if found, otherwise an index outside therange specified (in which case ‘return - index >= max_scan’ will be true).In the rare case of index wrap-around, 0 will be returned.
- pgoff_tpage_cache_prev_miss(structaddress_space*mapping,pgoff_tindex,unsignedlongmax_scan)¶
Find the previous gap in the page cache.
Parameters
structaddress_space*mappingMapping.
pgoff_tindexIndex.
unsignedlongmax_scanMaximum range to search.
Description
Search the range [max(index - max_scan + 1, 0), index] for thegap with the highest index.
This function may be called under the rcu_read_lock. However, this willnot atomically search a snapshot of the cache at a single point in time.For example, if a gap is created at index 10, then subsequently a gap iscreated at index 5,page_cache_prev_miss() covering both indices mayreturn 5 if called under the rcu_read_lock.
Return
The index of the gap if found, otherwise an index outside therange specified (in which case ‘index - return >= max_scan’ will be true).In the rare case of wrap-around, ULONG_MAX will be returned.
- structfolio*__filemap_get_folio(structaddress_space*mapping,pgoff_tindex,fgf_tfgp_flags,gfp_tgfp)¶
Find and get a reference to a folio.
Parameters
structaddress_space*mappingThe address_space to search.
pgoff_tindexThe page index.
fgf_tfgp_flagsFGPflags modify how the folio is returned.gfp_tgfpMemory allocation flags to use if
FGP_CREATis specified.
Description
Looks up the page cache entry atmapping &index.
IfFGP_LOCK orFGP_CREAT are specified then the function may sleep evenif theGFP flags specified forFGP_CREAT are atomic.
If this function returns a folio, it is returned with an increased refcount.
Return
The found folio or anERR_PTR() otherwise.
- unsignedfilemap_get_folios(structaddress_space*mapping,pgoff_t*start,pgoff_tend,structfolio_batch*fbatch)¶
Get a batch of folios
Parameters
structaddress_space*mappingThe address_space to search
pgoff_t*startThe starting page index
pgoff_tendThe final page index (inclusive)
structfolio_batch*fbatchThe batch to fill.
Description
Search for and return a batch of folios in the mapping starting atindexstart and up to indexend (inclusive). The folios are returnedinfbatch with an elevated reference count.
Return
The number of folios which were found.We also updatestart to index the next folio for the traversal.
- unsignedfilemap_get_folios_contig(structaddress_space*mapping,pgoff_t*start,pgoff_tend,structfolio_batch*fbatch)¶
Get a batch of contiguous folios
Parameters
structaddress_space*mappingThe address_space to search
pgoff_t*startThe starting page index
pgoff_tendThe final page index (inclusive)
structfolio_batch*fbatchThe batch to fill
Description
filemap_get_folios_contig() works exactly likefilemap_get_folios(),except the returned folios are guaranteed to be contiguous. This maynot return all contiguous folios if the batch gets filled up.
Return
The number of folios found.Also updatestart to be positioned for traversal of the next folio.
- unsignedfilemap_get_folios_tag(structaddress_space*mapping,pgoff_t*start,pgoff_tend,xa_mark_ttag,structfolio_batch*fbatch)¶
Get a batch of folios matchingtag
Parameters
structaddress_space*mappingThe address_space to search
pgoff_t*startThe starting page index
pgoff_tendThe final page index (inclusive)
xa_mark_ttagThe tag index
structfolio_batch*fbatchThe batch to fill
Description
The first folio may start beforestart; if it does, it will containstart. The final folio may extend beyondend; if it does, it willcontainend. The folios have ascending indices. There may be gapsbetween the folios if there are indices which have no folio in thepage cache. If folios are added to or removed from the page cachewhile this is running, they may or may not be found by this call.Only returns folios that are tagged withtag.
Return
The number of folios found.Also updatestart to index the next folio for traversal.
- ssize_tfilemap_read(structkiocb*iocb,structiov_iter*iter,ssize_talready_read)¶
Read data from the page cache.
Parameters
structkiocb*iocbThe iocb to read.
structiov_iter*iterDestination for the data.
ssize_talready_readNumber of bytes already read by the caller.
Description
Copies data from the page cache. If the data is not currently present,uses the readahead and read_folio address_space operations to fetch it.
Return
Total number of bytes copied, including those already read bythe caller. If an error happens before any bytes are copied, returnsa negative error number.
- ssize_tgeneric_file_read_iter(structkiocb*iocb,structiov_iter*iter)¶
generic filesystem read routine
Parameters
structkiocb*iocbkernel I/O control block
structiov_iter*iterdestination for the data read
Description
This is the “read_iter()” routine for all filesystemsthat can use the page cache directly.
The IOCB_NOWAIT flag in iocb->ki_flags indicates that -EAGAIN shallbe returned when no data can be read without waiting for I/O requeststo complete; it doesn’t prevent readahead.
The IOCB_NOIO flag in iocb->ki_flags indicates that no new I/Orequests shall be made for the read or for readahead. When no datacan be read, -EAGAIN shall be returned. When readahead would betriggered, a partial, possibly empty read shall be returned.
Return
number of bytes copied, even for partial reads
negative error code (or 0 if IOCB_NOIO) if nothing was read
- ssize_tfilemap_splice_read(structfile*in,loff_t*ppos,structpipe_inode_info*pipe,size_tlen,unsignedintflags)¶
Splice data from a file’s pagecache into a pipe
Parameters
structfile*inThe file to read from
loff_t*pposPointer to the file position to read from
structpipe_inode_info*pipeThe pipe to splice into
size_tlenThe amount to splice
unsignedintflagsThe SPLICE_F_* flags
Description
This function gets folios from a file’s pagecache and splices them into thepipe. Readahead will be called as necessary to fill more folios. This maybe used for blockdevs also.
Return
On success, the number of bytes read will be returned and*pposwill be updated if appropriate; 0 will be returned if there is no more datato be read; -EAGAIN will be returned if the pipe had no space, and someother negative error code will be returned on error. A short read may occurif the pipe has insufficient space, we reach the end of the data or we hit ahole.
- vm_fault_tfilemap_fault(structvm_fault*vmf)¶
read in file data for page fault handling
Parameters
structvm_fault*vmfstruct vm_fault containing details of the fault
Description
filemap_fault() is invoked via the vma operations vector for amapped memory region to read in file data during a page fault.
The goto’s are kind of ugly, but this streamlines the normal case of havingit in the page cache, and handles the special cases reasonably withouthaving a lot of duplicated code.
vma->vm_mm->mmap_lock must be held on entry.
If our return value has VM_FAULT_RETRY set, it’s because the mmap_lockmay be dropped before doing I/O or by lock_folio_maybe_drop_mmap().
If our return value does not have VM_FAULT_RETRY set, the mmap_lockhas not been released.
We never return with VM_FAULT_RETRY and a bit from VM_FAULT_ERROR set.
Return
bitwise-OR ofVM_FAULT_ codes.
- structfolio*read_cache_folio(structaddress_space*mapping,pgoff_tindex,filler_tfiller,structfile*file)¶
Read into page cache, fill it if needed.
Parameters
structaddress_space*mappingThe address_space to read from.
pgoff_tindexThe index to read.
filler_tfillerFunction to perform the read, or NULL to use aops->read_folio().
structfile*filePassed to filler function, may be NULL if not required.
Description
Read one page into the page cache. If it succeeds, the folio returnedwill containindex, but it may not be the first page of the folio.
If the filler function returns an error, it will be returned to thecaller.
Context
May sleep. Expects mapping->invalidate_lock to be held.
Return
An uptodate folio on success,ERR_PTR() on failure.
- structfolio*mapping_read_folio_gfp(structaddress_space*mapping,pgoff_tindex,gfp_tgfp)¶
Read into page cache, using specified allocation flags.
Parameters
structaddress_space*mappingThe address_space for the folio.
pgoff_tindexThe index that the allocated folio will contain.
gfp_tgfpThe page allocator flags to use if allocating.
Description
This is the same as “read_cache_folio(mapping, index, NULL, NULL)”, but withany new memory allocations done using the specified allocation flags.
The most likely error from this function is EIO, but ENOMEM ispossible and so is EINTR. If ->read_folio returns another error,that will be returned to the caller.
The function expects mapping->invalidate_lock to be already held.
Return
Uptodate folio on success,ERR_PTR() on failure.
- structpage*read_cache_page_gfp(structaddress_space*mapping,pgoff_tindex,gfp_tgfp)¶
read into page cache, using specified page allocation flags.
Parameters
structaddress_space*mappingthe page’s address_space
pgoff_tindexthe page index
gfp_tgfpthe page allocator flags to use if allocating
Description
This is the same as “read_mapping_page(mapping, index, NULL)”, but withany new page allocations done using the specified allocation flags.
If the page does not get brought uptodate, return -EIO.
The function expects mapping->invalidate_lock to be already held.
Return
up to date page on success,ERR_PTR() on failure.
- ssize_t__generic_file_write_iter(structkiocb*iocb,structiov_iter*from)¶
write data to a file
Parameters
structkiocb*iocbIO state structure (file, offset, etc.)
structiov_iter*fromiov_iter with data to write
Description
This function does all the work needed for actually writing data to afile. It does all basic checks, removes SUID from the file, updatesmodification times and calls proper subroutines depending on whether wedo direct IO or a standard buffered write.
It expects i_rwsem to be grabbed unless we work on a block device or similarobject which does not need locking at all.
This function doesnot take care of syncing data in case of O_SYNC write.A caller has to handle it. This is mainly due to the fact that we want toavoid syncing under i_rwsem.
Return
number of bytes written, even for truncated writes
negative error code if no data has been written at all
- ssize_tgeneric_file_write_iter(structkiocb*iocb,structiov_iter*from)¶
write data to a file
Parameters
structkiocb*iocbIO state structure
structiov_iter*fromiov_iter with data to write
Description
This is a wrapper around__generic_file_write_iter() to be used by mostfilesystems. It takes care of syncing the file in case of O_SYNC fileand acquires i_rwsem as needed.
Return
negative error code if no data has been written at all of
vfs_fsync_range()failed for a synchronous writenumber of bytes written, even for truncated writes
Parameters
structfolio*folioThe folio which the kernel is trying to free.
gfp_tgfpMemory allocation flags (and I/O mode).
Description
The address_space is trying to release any data attached to a folio(presumably at folio->private).
This will also be called if the private_2 flag is set on a page,indicating that the folio has other metadata associated with it.
Thegfp argument specifies whether I/O may be performed to releasethis page (__GFP_IO), and whether the call may block(__GFP_RECLAIM & __GFP_FS).
Return
true if the release was successful, otherwisefalse.
- intfilemap_invalidate_inode(structinode*inode,boolflush,loff_tstart,loff_tend)¶
Invalidate/forcibly write back a range of an inode’s pagecache
Parameters
structinode*inodeThe inode to flush
boolflushSet to write back rather than simply invalidate.
loff_tstartFirst byte to in range.
loff_tendLast byte in range (inclusive), or LLONG_MAX for everything from startonwards.
Description
Invalidate all the folios on an inode that contribute to the specifiedrange, possibly writing them back first. Whilst the operation isundertaken, the invalidate lock is held to prevent new folios from beinginstalled.
Readahead¶
Readahead is used to read content into the page cache before it isexplicitly requested by the application. Readahead only everattempts to read folios that are not yet in the page cache. If afolio is present but not up-to-date, readahead will not try to readit. In that case a simple ->read_folio() will be requested.
Readahead is triggered when an application read request (whether asystem call or a page fault) finds that the requested folio is not inthe page cache, or that it is in the page cache and has thereadahead flag set. This flag indicates that the folio was readas part of a previous readahead request and now that it has beenaccessed, it is time for the next readahead.
Each readahead request is partly synchronous read, and partly asyncreadahead. This is reflected in thestructfile_ra_state whichcontains ->size being the total number of pages, and ->async_sizewhich is the number of pages in the async section. The readaheadflag will be set on the first folio in this async section to triggera subsequent readahead. Once a series of sequential reads has beenestablished, there should be no need for a synchronous component andall readahead request will be fully asynchronous.
When either of the triggers causes a readahead, three numbers needto be determined: the start of the region to read, the size of theregion, and the size of the async tail.
The start of the region is simply the first page address at or afterthe accessed address, which is not currently populated in the pagecache. This is found with a simple search in the page cache.
The size of the async tail is determined by subtracting the size thatwas explicitly requested from the determined request size, unlessthis would be less than zero - then zero is used. NOTE THISCALCULATION IS WRONG WHEN THE START OF THE REGION IS NOT THE ACCESSEDPAGE. ALSO THIS CALCULATION IS NOT USED CONSISTENTLY.
The size of the region is normally determined from the size of theprevious readahead which loaded the preceding pages. This may bediscovered from thestructfile_ra_state for simple sequential reads,or from examining the state of the page cache when multiplesequential reads are interleaved. Specifically: where the readaheadwas triggered by the readahead flag, the size of the previousreadahead is assumed to be the number of pages from the triggeringpage to the start of the new readahead. In these cases, the size ofthe previous readahead is scaled, often doubled, for the newreadahead, though see get_next_ra_size() for details.
If the size of the previous read cannot be determined, the number ofpreceding pages in the page cache is used to estimate the size ofa previous read. This estimate could easily be misled by randomreads being coincidentally adjacent, so it is ignored unless it islarger than the current request, and it is not scaled up, unless itis at the start of file.
In general readahead is accelerated at the start of the file, asreads from there are often sequential. There are other minoradjustments to the readahead size in various special cases and theseare best discovered by reading the code.
The above calculation, based on the previous readahead size,determines the size of the readahead, to which any requested readsize may be added.
Readahead requests are sent to the filesystem using the ->readahead()address space operation, for whichmpage_readahead() is a canonicalimplementation. ->readahead() should normally initiate reads on allfolios, but may fail to read any or all folios without causing an I/Oerror. The page cache reading code will issue a ->read_folio() requestfor any folio which ->readahead() did not read, and only an errorfrom this will be final.
->readahead() will generally callreadahead_folio() repeatedly to geteach folio from those prepared for readahead. It may fail to read afolio by:
not calling
readahead_folio()sufficiently many times, effectivelyignoring some folios, as might be appropriate if the path tostorage is congested.failing to actually submit a read request for a given folio,possibly due to insufficient resources, or
getting an error during subsequent processing of a request.
In the last two cases, the folio should be unlocked by the filesystemto indicate that the read attempt has failed. In the first case thefolio will be unlocked by the VFS.
Those folios not in the finalasync_size of the request should beconsidered to be important and ->readahead() should not fail them dueto congestion or temporary resource unavailability, but should waitfor necessary resources (e.g. memory or indexing information) tobecome available. Folios in the finalasync_size may beconsidered less urgent and failure to read them is more acceptable.In this case it is best to use filemap_remove_folio() to remove thefolios from the page cache as is automatically done for folios thatwere not fetched withreadahead_folio(). This will allow asubsequent synchronous readahead request to try them again. If theyare left in the page cache, then they will be read individually using->read_folio() which may be less efficient.
- voidpage_cache_ra_unbounded(structreadahead_control*ractl,unsignedlongnr_to_read,unsignedlonglookahead_size)¶
Start unchecked readahead.
Parameters
structreadahead_control*ractlReadahead control.
unsignedlongnr_to_readThe number of pages to read.
unsignedlonglookahead_sizeWhere to start the next readahead.
Description
This function is for filesystems to call when they want to startreadahead beyond a file’s stated i_size. This is almost certainlynot the function you want to call. Usepage_cache_async_readahead()orpage_cache_sync_readahead() instead.
Context
File is referenced by caller. Mutexes may be held by caller.May sleep, but will not reenter filesystem to reclaim memory.
- voidreadahead_expand(structreadahead_control*ractl,loff_tnew_start,size_tnew_len)¶
Expand a readahead request
Parameters
structreadahead_control*ractlThe request to be expanded
loff_tnew_startThe revised start
size_tnew_lenThe revised size of the request
Description
Attempt to expand a readahead request outwards from the current size to thespecified size by inserting locked pages before and after the current windowto increase the size to the new window. This may involve the insertion ofTHPs, in which case the window may get expanded even beyond what wasrequested.
The algorithm will stop if it encounters a conflicting page already in thepagecache and leave a smaller expansion than requested.
The caller must check for this by examining the revisedractl object for adifferent expansion than was requested.
Writeback¶
- intbalance_dirty_pages_ratelimited_flags(structaddress_space*mapping,unsignedintflags)¶
Balance dirty memory state.
Parameters
structaddress_space*mappingaddress_space which was dirtied.
unsignedintflagsBDP flags.
Description
Processes which are dirtying memory should call in here once for each pagewhich was newly dirtied. The function will periodically check the system’sdirty state and will initiate writeback if needed.
Seebalance_dirty_pages_ratelimited() for details.
Return
Ifflags contains BDP_ASYNC, it may return -EAGAIN toindicate that memory is out of balance and the caller must waitfor I/O to complete. Otherwise, it will return 0 to indicatethat either memory was already in balance, or it was able to sleepuntil the amount of dirty memory returned to balance.
- voidbalance_dirty_pages_ratelimited(structaddress_space*mapping)¶
balance dirty memory state.
Parameters
structaddress_space*mappingaddress_space which was dirtied.
Description
Processes which are dirtying memory should call in here once for each pagewhich was newly dirtied. The function will periodically check the system’sdirty state and will initiate writeback if needed.
Once we’re over the dirty memory limit we decrease the ratelimitingby a lot, to prevent individual processes from overshooting the limitby (ratelimit_pages) each.
- voidtag_pages_for_writeback(structaddress_space*mapping,pgoff_tstart,pgoff_tend)¶
tag pages to be written by writeback
Parameters
structaddress_space*mappingaddress space structure to write
pgoff_tstartstarting page index
pgoff_tendending page index (inclusive)
Description
This function scans the page range fromstart toend (inclusive) and tagsall pages that have DIRTY tag set with a special TOWRITE tag. The callercan then use the TOWRITE tag to identify pages eligible for writeback.This mechanism is used to avoid livelocking of writeback by a processsteadily creating new dirty pages in the file (thus it is important for thisfunction to be quick so that it can tag pages faster than a dirtying processcan create them).
- structfolio*writeback_iter(structaddress_space*mapping,structwriteback_control*wbc,structfolio*folio,int*error)¶
iterate folio of a mapping for writeback
Parameters
structaddress_space*mappingaddress space structure to write
structwriteback_control*wbcwriteback context
structfolio*foliopreviously iterated folio (
NULLto start)int*errorin-out pointer for writeback errors (see below)
Description
This function returns the next folio for the writeback operation described bywbc onmapping and should be called in a while loop in the ->writepagesimplementation.
To start the writeback operation,NULL is passed in thefolio argument, andfor every subsequent iteration the folio returned previously should be passedback in.
If there was an error in the per-folio writeback inside thewriteback_iter()loop,error should be set to the error value.
Once the writeback described inwbc has finished, this function will returnNULL and if there was an error in any iteration restore it toerror.
Note
callers should not manually break out of the loop using break or gotobut must keep callingwriteback_iter() until it returnsNULL.
Return
the folio to write orNULL if the loop is done.
- intwrite_cache_pages(structaddress_space*mapping,structwriteback_control*wbc,writepage_twritepage,void*data)¶
walk the list of dirty pages of the given address space and write all of them.
Parameters
structaddress_space*mappingaddress space structure to write
structwriteback_control*wbcsubtract the number of written pages from*wbc->nr_to_write
writepage_twritepagefunction called for each page
void*datadata passed to writepage function
Return
0 on success, negative error code otherwise
Note
please usewriteback_iter() instead.
- boolfilemap_dirty_folio(structaddress_space*mapping,structfolio*folio)¶
Mark a folio dirty for filesystems which do not use buffer_heads.
Parameters
structaddress_space*mappingAddress space this folio belongs to.
structfolio*folioFolio to be marked as dirty.
Description
Filesystems which do not use buffer heads should call this functionfrom their dirty_folio address space operation. It ignores thecontents of folio_get_private(), so if the filesystem marks individualblocks as dirty, the filesystem should handle that itself.
This is also sometimes used by filesystems which use buffer_heads whena single buffer is being dirtied: we want to set the folio dirty inthat case, but not all the buffers. This is a “bottom-up” dirtying,whereasblock_dirty_folio() is a “top-down” dirtying.
The caller must ensure this doesn’t race with truncation. Most willsimply hold the folio lock, but e.g. zap_pte_range() calls with thefolio mapped and the pte lock held, which also locks out truncation.
- boolfolio_redirty_for_writepage(structwriteback_control*wbc,structfolio*folio)¶
Decline to write a dirty folio.
Parameters
structwriteback_control*wbcThe writeback control.
structfolio*folioThe folio.
Description
When a writepage implementation decides that it doesn’t want to writefolio for some reason, it should call this function, unlockfolio andreturn 0.
Return
True if we redirtied the folio. False if someone else dirtiedit first.
Parameters
structfolio*folioThe folio.
Description
The folio may not be truncated while this function is running.Holding the folio lock is sufficient to prevent truncation, but somecallers cannot acquire a sleeping lock. These callers instead holdthe page table lock for a page table which contains at least one pagein this folio. Truncation will block on the page table lock as itunmaps pages before removing the folio from its mapping.
Return
True if the folio was newly dirtied, false if it was already dirty.
Parameters
structfolio*folioThe folio to wait for.
Description
If the folio is currently being written back to storage, wait for theI/O to complete.
Context
Sleeps. Must be called in process context and withno spinlocks held. Caller should hold a reference on the folio.If the folio is not locked, writeback may start again after writebackhas finished.
Parameters
structfolio*folioThe folio to wait for.
Description
If the folio is currently being written back to storage, wait for theI/O to complete or a fatal signal to arrive.
Context
Sleeps. Must be called in process context and withno spinlocks held. Caller should hold a reference on the folio.If the folio is not locked, writeback may start again after writebackhas finished.
Return
0 on success, -EINTR if we get a fatal signal while waiting.
Parameters
structfolio*folioThe folio to wait on.
Description
This function determines if the given folio is related to a backingdevice that requires folio contents to be held stable during writeback.If so, then it will wait for any pending writeback to complete.
Context
Sleeps. Must be called in process context and withno spinlocks held. Caller should hold a reference on the folio.If the folio is not locked, writeback may start again after writebackhas finished.
Truncate¶
- voidfolio_invalidate(structfolio*folio,size_toffset,size_tlength)¶
Invalidate part or all of a folio.
Parameters
structfolio*folioThe folio which is affected.
size_toffsetstart of the range to invalidate
size_tlengthlength of the range to invalidate
Description
folio_invalidate() is called when all or part of the folio has becomeinvalidated by a truncate operation.
folio_invalidate() does not have to release all buffers, but it mustensure that no dirty buffer is left outsideoffset and that no I/Ois underway against any of the blocks which are outside the truncationpoint. Because the caller is about to free (and possibly reuse) thoseblocks on-disk.
- voidtruncate_inode_pages_range(structaddress_space*mapping,loff_tlstart,loff_tlend)¶
truncate range of pages specified by start & end byte offsets
Parameters
structaddress_space*mappingmapping to truncate
loff_tlstartoffset from which to truncate
loff_tlendoffset to which to truncate (inclusive)
Description
Truncate the page cache, removing the pages that are betweenspecified offsets (and zeroing out partial pagesif lstart or lend + 1 is not page aligned).
Truncate takes two passes - the first pass is nonblocking. It will notblock on page locks and it will not block on writeback. The second passwill wait. This is to prevent as much IO as possible in the affected region.The first pass will remove most pages, so the search cost of the second passis low.
We pass down the cache-hot hint to the page freeing code. Even if themapping is large, it is probably the case that the final pages are the mostrecently touched, and freeing happens in ascending file offset order.
Note that since ->invalidate_folio() accepts range to invalidatetruncate_inode_pages_range is able to handle cases where lend + 1 is notpage aligned properly.
- voidtruncate_inode_pages(structaddress_space*mapping,loff_tlstart)¶
truncateall the pages from an offset
Parameters
structaddress_space*mappingmapping to truncate
loff_tlstartoffset from which to truncate
Description
Called under (and serialised by) inode->i_rwsem andmapping->invalidate_lock.
Note
When this function returns, there can be a page in the process ofdeletion (inside __filemap_remove_folio()) in the specified range. Thusmapping->nrpages can be non-zero when this function returns even aftertruncation of the whole mapping.
- voidtruncate_inode_pages_final(structaddress_space*mapping)¶
truncateall pages before inode dies
Parameters
structaddress_space*mappingmapping to truncate
Description
Called under (and serialized by) inode->i_rwsem.
Filesystems have to use this in the .evict_inode path to inform theVM that this is the final truncate and the inode is going away.
- unsignedlonginvalidate_mapping_pages(structaddress_space*mapping,pgoff_tstart,pgoff_tend)¶
Invalidate all clean, unlocked cache of one inode
Parameters
structaddress_space*mappingthe address_space which holds the cache to invalidate
pgoff_tstartthe offset ‘from’ which to invalidate
pgoff_tendthe offset ‘to’ which to invalidate (inclusive)
Description
This function removes pages that are clean, unmapped and unlocked,as well as shadow entries. It will not block on IO activity.
If you want to remove all the pages of one inode, regardless oftheir use and writeback state, usetruncate_inode_pages().
Return
The number of indices that had their contents invalidated
- intinvalidate_inode_pages2_range(structaddress_space*mapping,pgoff_tstart,pgoff_tend)¶
remove range of pages from an address_space
Parameters
structaddress_space*mappingthe address_space
pgoff_tstartthe page offset ‘from’ which to invalidate
pgoff_tendthe page offset ‘to’ which to invalidate (inclusive)
Description
Any pages which are found to be mapped into pagetables are unmapped prior toinvalidation.
Return
-EBUSY if any pages could not be invalidated.
- intinvalidate_inode_pages2(structaddress_space*mapping)¶
remove all pages from an address_space
Parameters
structaddress_space*mappingthe address_space
Description
Any pages which are found to be mapped into pagetables are unmapped prior toinvalidation.
Return
-EBUSY if any pages could not be invalidated.
- voidtruncate_pagecache(structinode*inode,loff_tnewsize)¶
unmap and remove pagecache that has been truncated
Parameters
structinode*inodeinode
loff_tnewsizenew file size
Description
inode’s new i_size must already be written before truncate_pagecacheis called.
This function should typically be called before the filesystemreleases resources associated with the freed range (eg. deallocatesblocks). This way, pagecache will always stay logically coherentwith on-disk format, and the filesystem would not have to deal withsituations such as writepage being called for a page that has alreadyhad its underlying blocks deallocated.
- voidtruncate_setsize(structinode*inode,loff_tnewsize)¶
update inode and pagecache for a new file size
Parameters
structinode*inodeinode
loff_tnewsizenew file size
Description
truncate_setsize updates i_size and performs pagecache truncation (ifnecessary) tonewsize. It will be typically be called from the filesystem’ssetattr function when ATTR_SIZE is passed in.
Must be called with a lock serializing truncates and writes (generallyi_rwsem but e.g. xfs uses a different lock) and before all filesystemspecific block truncation has been performed.
- voidpagecache_isize_extended(structinode*inode,loff_tfrom,loff_tto)¶
update pagecache after extension of i_size
Parameters
structinode*inodeinode for which i_size was extended
loff_tfromoriginal inode size
loff_ttonew inode size
Description
Handle extension of inode size either caused by extending truncate orby write starting after current i_size. We mark the page straddlingcurrent i_size RO so that page_mkwrite() is called on the firstwrite access to the page. The filesystem will update its per-blockinformation before user writes to the page via mmap after the i_sizehas been changed.
The function must be called after i_size is updated so that page faultcoming after we unlock the folio will already see the new i_size.The function must be called while we still hold i_rwsem - this not onlymakes sure i_size is stable but also that userspace cannot observe newi_size value before we are prepared to store mmap writes at new inode size.
- voidtruncate_pagecache_range(structinode*inode,loff_tlstart,loff_tlend)¶
unmap and remove pagecache that is hole-punched
Parameters
structinode*inodeinode
loff_tlstartoffset of beginning of hole
loff_tlendoffset of last byte of hole
Description
This function should typically be called before the filesystemreleases resources associated with the freed range (eg. deallocatesblocks). This way, pagecache will always stay logically coherentwith on-disk format, and the filesystem would not have to deal withsituations such as writepage being called for a page that has alreadyhad its underlying blocks deallocated.
- voidfilemap_set_wb_err(structaddress_space*mapping,interr)¶
set a writeback error on an address_space
Parameters
structaddress_space*mappingmapping in which to set writeback error
interrerror to be set in mapping
Description
When writeback fails in some way, we must record that error so thatuserspace can be informed when fsync and the like are called. We endeavorto report errors on any file that was open at the time of the error. Someinternal callers also need to know when writeback errors have occurred.
When a writeback error occurs, most filesystems will want to callfilemap_set_wb_err to record the error in the mapping so that it will beautomatically reported whenever fsync is called on the file.
- intfilemap_check_wb_err(structaddress_space*mapping,errseq_tsince)¶
has an error occurred since the mark was sampled?
Parameters
structaddress_space*mappingmapping to check for writeback errors
errseq_tsincepreviously-sampled errseq_t
Description
Grab the errseq_t value from the mapping, and see if it has changed “since”the given value was sampled.
If it has then report the latest error set, otherwise return 0.
- errseq_tfilemap_sample_wb_err(structaddress_space*mapping)¶
sample the current errseq_t to test for later errors
Parameters
structaddress_space*mappingmapping to be sampled
Description
Writeback errors are always reported relative to a particular sample pointin the past. This function provides those sample points.
Parameters
structfile*filefile pointer to be sampled
Description
Grab the most current superblock-level errseq_t value for the givenstructfile.
- voidmapping_set_error(structaddress_space*mapping,interror)¶
record a writeback error in the address_space
Parameters
structaddress_space*mappingthe mapping in which an error should be set
interrorthe error to set in the mapping
Description
When writeback fails in some way, we must record that error so thatuserspace can be informed when fsync and the like are called. We endeavorto report errors on any file that was open at the time of the error. Someinternal callers also need to know when writeback errors have occurred.
When a writeback error occurs, most filesystems will want to callmapping_set_error to record the error in the mapping so that it can bereported when the application calls fsync(2).
- voidmapping_set_large_folios(structaddress_space*mapping)¶
Indicate the file supports large folios.
Parameters
structaddress_space*mappingThe address space of the file.
Description
The filesystem should call this function in its inode constructor toindicate that the VFS can use large folios to cache the contents ofthe file.
Context
This should not be called while the inode is active as itis non-atomic.
- pgoff_tmapping_align_index(structaddress_space*mapping,pgoff_tindex)¶
Align index for this mapping.
Parameters
structaddress_space*mappingThe address_space.
pgoff_tindexThe page index.
Description
The index of a folio must be naturally aligned. If you are adding anew folio to the page cache and need to know what index to give it,call this function.
- structaddress_space*folio_flush_mapping(structfolio*folio)¶
Find the file mapping this folio belongs to.
Parameters
structfolio*folioThe folio.
Description
For folios which are in the page cache, return the mapping that thispage belongs to. Anonymous folios return NULL, even if they’re inthe swap cache. Other kinds of folio also return NULL.
This is ONLY used by architecture cache flushing code. If you aren’twriting cache flushing code, you want eitherfolio_mapping() orfolio_file_mapping().
Parameters
structfolio*folioThe folio.
Description
For folios which are in the page cache, return the inode that this foliobelongs to.
Do not call this for folios which aren’t in the page cache.
Parameters
structfolio*folioFolio to attach data to.
void*dataData to attach to folio.
Description
Attaching private data to a folio increments the page’s reference count.The data must be detached before the folio will be freed.
Parameters
structfolio*folioFolio to change the data on.
void*dataData to set on the folio.
Description
Change the private data attached to a folio and return the olddata. The page must previously have had data attached and the datamust be detached before the folio will be freed.
Return
Data that was previously attached to the folio.
Parameters
structfolio*folioFolio to detach data from.
Description
Removes the data that was previously attached to the folio and decrementsthe refcount on the page.
Return
Data that was attached to the folio.
- typefgf_t¶
Flags for getting folios from the page cache.
Description
Most users of the page cache will not need to use these flags;there are convenience functions such asfilemap_get_folio() andfilemap_lock_folio(). For users which need more control over exactlywhat is done with the folios, these flags to__filemap_get_folio()are available.
FGP_ACCESSED- The folio will be marked accessed.FGP_LOCK- The folio is returned locked.FGP_CREAT- If no folio is present then a new folio is allocated,added to the page cache and the VM’s LRU list. The folio isreturned locked.FGP_FOR_MMAP- The caller wants to do its own locking dance if thefolio is already in cache. If the folio was allocated, unlock itbefore returning so the caller can do the same dance.FGP_WRITE- The folio will be written to by the caller.FGP_NOFS- __GFP_FS will get cleared in gfp.FGP_NOWAIT- Don’t block on the folio lock.FGP_STABLE- Wait for the folio to be stable (finished writeback)FGP_DONTCACHE- Uncached buffered IOFGP_WRITEBEGIN- The flags to use in a filesystem write_begin()implementation.
Parameters
size_tsizeThe suggested size of the folio to create.
Description
The caller of__filemap_get_folio() can use this to suggest a preferredsize for the folio that is created. If there is already a folio atthe index, it will be returned, no matter what its size. If a foliois freshly created, it may be of a different size than requesteddue to alignment constraints, memory pressure, or the presence ofother folios at nearby indices.
- structfolio*filemap_get_folio(structaddress_space*mapping,pgoff_tindex)¶
Find and get a folio.
Parameters
structaddress_space*mappingThe address_space to search.
pgoff_tindexThe page index.
Description
Looks up the page cache entry atmapping &index. If a folio ispresent, it is returned with an increased refcount.
Return
A folio or ERR_PTR(-ENOENT) if there is no folio in the cache forthis index. Will not return a shadow, swap or DAX entry.
- structfolio*filemap_lock_folio(structaddress_space*mapping,pgoff_tindex)¶
Find and lock a folio.
Parameters
structaddress_space*mappingThe address_space to search.
pgoff_tindexThe page index.
Description
Looks up the page cache entry atmapping &index. If a folio ispresent, it is returned locked with an increased refcount.
Context
May sleep.
Return
A folio or ERR_PTR(-ENOENT) if there is no folio in the cache forthis index. Will not return a shadow, swap or DAX entry.
- structfolio*filemap_grab_folio(structaddress_space*mapping,pgoff_tindex)¶
grab a folio from the page cache
Parameters
structaddress_space*mappingThe address space to search
pgoff_tindexThe page index
Description
Looks up the page cache entry atmapping &index. If no folio is found,a new folio is created. The folio is locked, marked as accessed, andreturned.
Return
A found or created folio. ERR_PTR(-ENOMEM) if no folio is foundand failed to create a folio.
- structpage*find_get_page(structaddress_space*mapping,pgoff_toffset)¶
find and get a page reference
Parameters
structaddress_space*mappingthe address_space to search
pgoff_toffsetthe page index
Description
Looks up the page cache slot atmapping &offset. If there is apage cache page, it is returned with an increased refcount.
Otherwise,NULL is returned.
- structpage*find_lock_page(structaddress_space*mapping,pgoff_tindex)¶
locate, pin and lock a pagecache page
Parameters
structaddress_space*mappingthe address_space to search
pgoff_tindexthe page index
Description
Looks up the page cache entry atmapping &index. If there is apage cache page, it is returned locked and with an increasedrefcount.
Context
May sleep.
Return
A struct page orNULL if there is no page in the cache for thisindex.
- structpage*find_or_create_page(structaddress_space*mapping,pgoff_tindex,gfp_tgfp_mask)¶
locate or add a pagecache page
Parameters
structaddress_space*mappingthe page’s address_space
pgoff_tindexthe page’s index into the mapping
gfp_tgfp_maskpage allocation mode
Description
Looks up the page cache slot atmapping &offset. If there is apage cache page, it is returned locked and with an increasedrefcount.
If the page is not present, a new page is allocated usinggfp_maskand added to the page cache and the VM’s LRU list. The page isreturned locked and with an increased refcount.
On memory exhaustion,NULL is returned.
find_or_create_page() may sleep, even ifgfp_flags specifies anatomic allocation!
- structpage*grab_cache_page_nowait(structaddress_space*mapping,pgoff_tindex)¶
returns locked page at given index in given cache
Parameters
structaddress_space*mappingtarget address_space
pgoff_tindexthe page index
Description
Same as grab_cache_page(), but do not wait if the page is unavailable.This is intended for speculative data generators, where the data canbe regenerated if the page couldn’t be grabbed. This routine shouldbe safe to call while holding the lock for another page.
Clear __GFP_FS when allocating the page to avoid recursion into the fsand deadlock against the caller’s locked page.
Parameters
structfolio*folioThe current folio.
Return
The index of the folio which follows this folio in the file.
Parameters
structfolio*folioThe folio which contains this index.
pgoff_tindexThe index we want to look up.
Description
Sometimes after looking up a folio in the page cache, we need toobtain the specific page for an index (eg a page fault).
Return
The page containing the file data for this index.
Parameters
structfolio*folioThe folio.
pgoff_tindexThe page index within the file.
Context
The caller should have the folio locked and ensuree.g., shmem did not move this folio to the swap cache.
Return
true or false.
- pgoff_tpage_pgoff(conststructfolio*folio,conststructpage*page)¶
Calculate the logical page offset of this page.
Parameters
conststructfolio*folioThe folio containing this page.
conststructpage*pageThe page which we need the offset of.
Description
For file pages, this is the offset from the beginning of the filein units of PAGE_SIZE. For anonymous pages, this is the offset fromthe beginning of the anon_vma in units of PAGE_SIZE. This willreturn nonsense for KSM pages.
Context
Caller must have a reference on the folio or otherwiseprevent it from being split or freed.
Return
The offset in units of PAGE_SIZE.
Parameters
conststructfolio*folioThe folio.
Parameters
structfolio*folioThe folio to attempt to lock.
Description
Sometimes it is undesirable to wait for a folio to be unlocked (egwhen the locks are being taken in the wrong order, or if makingprogress through a batch of folios is more important than processingthem in order). Usuallyfolio_lock() is the correct function to call.
Context
Any context.
Return
Whether the lock was successfully acquired.
Parameters
structfolio*folioThe folio to lock.
Description
The folio lock protects against many things, probably more than itshould. It is primarily held while a folio is being brought uptodate,either from its backing file or from swap. It is also held while afolio is being truncated from its address_space, so holding the lockis sufficient to keep folio->mapping stable.
The folio lock is also held while write() is modifying the page toprovide POSIX atomicity guarantees (as long as the write does notcross a page boundary). Other modifications to the data in the foliodo not hold the folio lock and can race with writes, eg DMA and storesto mapped pages.
Context
May sleep. If you need to acquire the locks of two ormore folios, they must be in order of ascending index, if they arein the same address_space. If they are in different address_spaces,acquire the lock of the folio which belongs to the address_space whichhas the lowest address in memory first.
Parameters
structpage*pageThe page to lock.
Description
Seefolio_lock() for a description of what the lock protects.This is a legacy function and new code should probably usefolio_lock()instead.
Context
May sleep. Pages in the same folio share a lock, so do notattempt to lock two pages which share a folio.
Parameters
structfolio*folioThe folio to lock.
Description
Attempts to lock the folio, likefolio_lock(), except that the sleepto acquire the lock is interruptible by a fatal signal.
Context
May sleep; seefolio_lock().
Return
0 if the lock was acquired; -EINTR if a fatal signal was received.
- boolfilemap_range_needs_writeback(structaddress_space*mapping,loff_tstart_byte,loff_tend_byte)¶
check if range potentially needs writeback
Parameters
structaddress_space*mappingaddress space within which to check
loff_tstart_byteoffset in bytes where the range starts
loff_tend_byteoffset in bytes where the range ends (inclusive)
Description
Find at least one page in the range supplied, usually used to check ifdirect writing in this range will trigger a writeback. Used by O_DIRECTread/write with IOCB_NOWAIT, to see if the caller needs to dofilemap_write_and_wait_range() before proceeding.
Return
true if the caller should dofilemap_write_and_wait_range() beforedoing O_DIRECT to a page in this range,false otherwise.
- structreadahead_control¶
Describes a readahead request.
Definition:
struct readahead_control { struct file *file; struct address_space *mapping; struct file_ra_state *ra;};Members
fileThe file, used primarily by network filesystems for authentication.May be NULL if invoked internally by the filesystem.
mappingReadahead this filesystem object.
raFile readahead state. May be NULL.
Description
A readahead request is for consecutive pages. Filesystems whichimplement the ->readahead method should callreadahead_folio() or__readahead_batch() in a loop and attempt to start reads into eachfolio in the request.
Most of the fields in this struct are private and should be accessedby the functions below.
- voidpage_cache_sync_readahead(structaddress_space*mapping,structfile_ra_state*ra,structfile*file,pgoff_tindex,unsignedlongreq_count)¶
generic file readahead
Parameters
structaddress_space*mappingaddress_space which holds the pagecache and I/O vectors
structfile_ra_state*rafile_ra_state which holds the readahead state
structfile*fileUsed by the filesystem for authentication.
pgoff_tindexIndex of first page to be read.
unsignedlongreq_countTotal number of pages being read by the caller.
Description
page_cache_sync_readahead() should be called when a cache miss happened:it will submit the read. The readahead logic may decide to piggyback morepages onto the read request if access patterns suggest it will improveperformance.
- voidpage_cache_async_readahead(structaddress_space*mapping,structfile_ra_state*ra,structfile*file,structfolio*folio,unsignedlongreq_count)¶
file readahead for marked pages
Parameters
structaddress_space*mappingaddress_space which holds the pagecache and I/O vectors
structfile_ra_state*rafile_ra_state which holds the readahead state
structfile*fileUsed by the filesystem for authentication.
structfolio*folioThe folio which triggered the readahead call.
unsignedlongreq_countTotal number of pages being read by the caller.
Description
page_cache_async_readahead() should be called when a page is used whichis marked as PageReadahead; this is a marker to suggest that the applicationhas used up enough of the readahead window that we should start pulling inmore pages.
- structfolio*readahead_folio(structreadahead_control*ractl)¶
Get the next folio to read.
Parameters
structreadahead_control*ractlThe current readahead request.
Context
The folio is locked. The caller should unlock the folio onceall I/O to that folio has completed.
Return
A pointer to the next folio, orNULL if we are done.
- loff_treadahead_pos(structreadahead_control*rac)¶
The byte offset into the file of this readahead request.
Parameters
structreadahead_control*racThe readahead request.
- size_treadahead_length(structreadahead_control*rac)¶
The number of bytes in this readahead request.
Parameters
structreadahead_control*racThe readahead request.
- pgoff_treadahead_index(structreadahead_control*rac)¶
The index of the first page in this readahead request.
Parameters
structreadahead_control*racThe readahead request.
- unsignedintreadahead_count(structreadahead_control*rac)¶
The number of pages in this readahead request.
Parameters
structreadahead_control*racThe readahead request.
- size_treadahead_batch_length(structreadahead_control*rac)¶
The number of bytes in the current batch.
Parameters
structreadahead_control*racThe readahead request.
- ssize_tfolio_mkwrite_check_truncate(structfolio*folio,structinode*inode)¶
check if folio was truncated
Parameters
structfolio*foliothe folio to check
structinode*inodethe inode to check the folio against
Return
the number of bytes in the folio up to EOF,or -EFAULT if the folio was truncated.
- unsignedinti_blocks_per_folio(structinode*inode,structfolio*folio)¶
How many blocks fit in this folio.
Parameters
structinode*inodeThe inode which contains the blocks.
structfolio*folioThe folio.
Description
If the block size is larger than the size of this folio, return zero.
Context
The caller should hold a refcount on the folio to prevent itfrom being split.
Return
The number of filesystem blocks covered by this folio.
Memory pools¶
- voidmempool_exit(mempool_t*pool)¶
exit a mempool initialized with mempool_init()
Parameters
mempool_t*poolpointer to the memory pool which was initialized withmempool_init().
Description
Free all reserved elements inpool andpool itself. This functiononly sleeps if the free_fn() function sleeps.
May be called on a zeroed but uninitialized mempool (i.e. allocated withkzalloc()).
- voidmempool_destroy(mempool_t*pool)¶
deallocate a memory pool
Parameters
mempool_t*poolpointer to the memory pool which was allocated viamempool_create().
Description
Free all reserved elements inpool andpool itself. This functiononly sleeps if the free_fn() function sleeps.
- intmempool_resize(mempool_t*pool,intnew_min_nr)¶
resize an existing memory pool
Parameters
mempool_t*poolpointer to the memory pool which was allocated viamempool_create().
intnew_min_nrthe new minimum number of elements guaranteed to beallocated for this pool.
Description
This function shrinks/grows the pool. In the case of growing,it cannot be guaranteed that the pool will be grown to the newsize immediately, but newmempool_free() calls will refill it.This function may sleep.
Note, the caller must guarantee that no mempool_destroy is calledwhile this function is running. mempool_alloc() &mempool_free()might be called (eg. from IRQ contexts) while this function executes.
Return
0 on success, negative error code otherwise.
- void*mempool_alloc_preallocated(mempool_t*pool)¶
allocate an element from preallocated elements belonging to a specific memory pool
Parameters
mempool_t*poolpointer to the memory pool which was allocated viamempool_create().
Description
This function is similar to mempool_alloc, but it only attempts allocatingan element from the preallocated elements. It does not sleep and immediatelyreturns if no preallocated elements are available.
Return
pointer to the allocated element orNULL if no elements areavailable.
- voidmempool_free(void*element,mempool_t*pool)¶
return an element to the pool.
Parameters
void*elementpool element pointer.
mempool_t*poolpointer to the memory pool which was allocated viamempool_create().
Description
this function only sleeps if the free_fn() function sleeps.
DMA pools¶
- structdma_pool*dma_pool_create_node(constchar*name,structdevice*dev,size_tsize,size_talign,size_tboundary,intnode)¶
Creates a pool of consistent memory blocks, for dma.
Parameters
constchar*namename of pool, for diagnostics
structdevice*devdevice that will be doing the DMA
size_tsizesize of the blocks in this pool.
size_talignalignment requirement for blocks; must be a power of two
size_tboundaryreturned blocks won’t cross this power of two boundary
intnodeoptional NUMA node to allocate structs ‘dma_pool’ and ‘dma_page’ on
Context
not in_interrupt()
Description
Given one of these pools,dma_pool_alloc()may be used to allocate memory. Such memory will all have “consistent”DMA mappings, accessible by the device and its driver without usingcache flushing primitives. The actual size of blocks allocated may belarger than requested because of alignment.
Ifboundary is nonzero, objects returned fromdma_pool_alloc() won’tcross that size boundary. This is useful for devices which haveaddressing restrictions on individual DMA transfers, such as not crossingboundaries of 4KBytes.
Return
a dma allocation pool with the requested characteristics, orNULL if one can’t be created.
- voiddma_pool_destroy(structdma_pool*pool)¶
destroys a pool of dma memory blocks.
Parameters
structdma_pool*pooldma pool that will be destroyed
Context
!in_interrupt()
Description
Caller guarantees that no more memory from the pool is in use,and that nothing will try to use the pool after this call.
- void*dma_pool_alloc(structdma_pool*pool,gfp_tmem_flags,dma_addr_t*handle)¶
get a block of consistent memory
Parameters
structdma_pool*pooldma pool that will produce the block
gfp_tmem_flagsGFP_* bitmask
dma_addr_t*handlepointer to dma address of block
Return
the kernel virtual address of a currently unused block,and reports its dma address through the handle.If such a memory block can’t be allocated,NULL is returned.
- voiddma_pool_free(structdma_pool*pool,void*vaddr,dma_addr_tdma)¶
put block back into dma pool
Parameters
structdma_pool*poolthe dma pool holding the block
void*vaddrvirtual address of block
dma_addr_tdmadma address of block
Description
Caller promises neither device nor driver will again touch this blockunless it is first re-allocated.
- structdma_pool*dmam_pool_create(constchar*name,structdevice*dev,size_tsize,size_talign,size_tallocation)¶
Managed dma_pool_create()
Parameters
constchar*namename of pool, for diagnostics
structdevice*devdevice that will be doing the DMA
size_tsizesize of the blocks in this pool.
size_talignalignment requirement for blocks; must be a power of two
size_tallocationreturned blocks won’t cross this boundary (or zero)
Description
Managed dma_pool_create(). DMA pool created with this function isautomatically destroyed on driver detach.
Return
a managed dma allocation pool with the requestedcharacteristics, orNULL if one can’t be created.
- voiddmam_pool_destroy(structdma_pool*pool)¶
Managed
dma_pool_destroy()
More Memory Management Functions¶
- voidzap_vma_ptes(structvm_area_struct*vma,unsignedlongaddress,unsignedlongsize)¶
remove ptes mapping the vma
Parameters
structvm_area_struct*vmavm_area_struct holding ptes to be zapped
unsignedlongaddressstarting address of pages to zap
unsignedlongsizenumber of bytes to zap
Description
This function only unmaps ptes assigned to VM_PFNMAP vmas.
The entire address range must be fully contained within the vma.
- intvm_insert_pages(structvm_area_struct*vma,unsignedlongaddr,structpage**pages,unsignedlong*num)¶
insert multiple pages into user vma, batching the pmd lock.
Parameters
structvm_area_struct*vmauser vma to map to
unsignedlongaddrtarget start user address of these pages
structpage**pagessource kernel pages
unsignedlong*numin: number of pages to map. out: number of pages that werenotmapped. (0 means all pages were successfully mapped).
Description
Preferred overvm_insert_page() when inserting multiple pages.
In case of error, we may have mapped a subset of the providedpages. It is the caller’s responsibility to account for this case.
The same restrictions apply as invm_insert_page().
- intvm_insert_page(structvm_area_struct*vma,unsignedlongaddr,structpage*page)¶
insert single page into user vma
Parameters
structvm_area_struct*vmauser vma to map to
unsignedlongaddrtarget user address of this page
structpage*pagesource kernel page
Description
This allows drivers to insert individual pages they’ve allocatedinto a user vma. The zeropage is supported in some VMAs,see vm_mixed_zeropage_allowed().
The page has to be a nice clean _individual_ kernel allocation.If you allocate a compound page, you need to have marked it assuch (__GFP_COMP), or manually just split the page up yourself(see split_page()).
NOTE! Traditionally this was done with “remap_pfn_range()” whichtook an arbitrary page protection parameter. This doesn’t allowthat. Your vma protection will have to be set up correctly, whichmeans that if you want a shared writable mapping, you’d betterask for a shared writable mapping!
The page does not need to be reserved.
Usually this function is called from f_op->mmap() handlerunder mm->mmap_lock write-lock, so it can change vma->vm_flags.Caller must set VM_MIXEDMAP on vma if it wants to call thisfunction from other places, for example from page-fault handler.
Return
0 on success, negative error code otherwise.
- intvm_map_pages(structvm_area_struct*vma,structpage**pages,unsignedlongnum)¶
maps range of kernel pages starts with non zero offset
Parameters
structvm_area_struct*vmauser vma to map to
structpage**pagespointer to array of source kernel pages
unsignedlongnumnumber of pages in page array
Description
Maps an object consisting ofnum pages, catering for the user’srequested vm_pgoff
If we fail to insert any page into the vma, the function will returnimmediately leaving any previously inserted pages present. Callersfrom the mmap handler may immediately return the error as their callerwill destroy the vma, removing any successfully inserted pages. Othercallers should make their own arrangements for calling unmap_region().
Context
Process context. Called by mmap handlers.
Return
0 on success and error code otherwise.
- intvm_map_pages_zero(structvm_area_struct*vma,structpage**pages,unsignedlongnum)¶
map range of kernel pages starts with zero offset
Parameters
structvm_area_struct*vmauser vma to map to
structpage**pagespointer to array of source kernel pages
unsignedlongnumnumber of pages in page array
Description
Similar tovm_map_pages(), except that it explicitly sets the offsetto 0. This function is intended for the drivers that did not considervm_pgoff.
Context
Process context. Called by mmap handlers.
Return
0 on success and error code otherwise.
- vm_fault_tvmf_insert_pfn_prot(structvm_area_struct*vma,unsignedlongaddr,unsignedlongpfn,pgprot_tpgprot)¶
insert single pfn into user vma with specified pgprot
Parameters
structvm_area_struct*vmauser vma to map to
unsignedlongaddrtarget user address of this page
unsignedlongpfnsource kernel pfn
pgprot_tpgprotpgprot flags for the inserted page
Description
This is exactly likevmf_insert_pfn(), except that it allows driversto override pgprot on a per-page basis.
This only makes sense for IO mappings, and it makes no sense forCOW mappings. In general, using multiple vmas is preferable;vmf_insert_pfn_prot should only be used if using multiple VMAs isimpractical.
pgprot typically only differs fromvma->vm_page_prot when drivers setcaching- and encryption bits different than those ofvma->vm_page_prot,because the caching- or encryption mode may not be known at mmap() time.
This is ok as long asvma->vm_page_prot is not used by the core vmto set caching and encryption bits for those vmas (except for COW pages).This is ensured by core vm only modifying these page table entries usingfunctions that don’t touch caching- or encryption bits, using pte_modify()if needed. (See for example mprotect()).
Also when new page-table entries are created, this is only done using thefault() callback, and never using the value of vma->vm_page_prot,except for page-table entries that point to anonymous pages as the resultof COW.
Context
Process context. May allocate usingGFP_KERNEL.
Return
vm_fault_t value.
- vm_fault_tvmf_insert_pfn(structvm_area_struct*vma,unsignedlongaddr,unsignedlongpfn)¶
insert single pfn into user vma
Parameters
structvm_area_struct*vmauser vma to map to
unsignedlongaddrtarget user address of this page
unsignedlongpfnsource kernel pfn
Description
Similar to vm_insert_page, this allows drivers to insert individual pagesthey’ve allocated into a user vma. Same comments apply.
This function should only be called from a vm_ops->fault handler, andin that case the handler should return the result of this function.
vma cannot be a COW mapping.
As this is called only for pages that do not currently exist, wedo not need to flush old virtual caches or the TLB.
Context
Process context. May allocate usingGFP_KERNEL.
Return
vm_fault_t value.
- intremap_pfn_range(structvm_area_struct*vma,unsignedlongaddr,unsignedlongpfn,unsignedlongsize,pgprot_tprot)¶
remap kernel memory to userspace
Parameters
structvm_area_struct*vmauser vma to map to
unsignedlongaddrtarget page aligned user address to start at
unsignedlongpfnpage frame number of kernel physical memory address
unsignedlongsizesize of mapping area
pgprot_tprotpage protection flags for this mapping
Note
this is only safe if the mm semaphore is held when called.
Return
0 on success, negative error code otherwise.
- intvm_iomap_memory(structvm_area_struct*vma,phys_addr_tstart,unsignedlonglen)¶
remap memory to userspace
Parameters
structvm_area_struct*vmauser vma to map to
phys_addr_tstartstart of the physical memory to be mapped
unsignedlonglensize of area
Description
This is a simplified io_remap_pfn_range() for common driver use. Thedriver just needs to give us the physical memory range to be mapped,we’ll figure out the rest from the vma information.
NOTE! Some drivers might want to tweak vma->vm_page_prot first to getwhatever write-combining details or similar.
Return
0 on success, negative error code otherwise.
- voidunmap_mapping_pages(structaddress_space*mapping,pgoff_tstart,pgoff_tnr,booleven_cows)¶
Unmap pages from processes.
Parameters
structaddress_space*mappingThe address space containing pages to be unmapped.
pgoff_tstartIndex of first page to be unmapped.
pgoff_tnrNumber of pages to be unmapped. 0 to unmap to end of file.
booleven_cowsWhether to unmap even private COWed pages.
Description
Unmap the pages in this address space from any userspace process whichhas them mmaped. Generally, you want to remove COWed pages as well whena file is being truncated, but not when invalidating pages from the pagecache.
- voidunmap_mapping_range(structaddress_space*mapping,loff_tconstholebegin,loff_tconstholelen,inteven_cows)¶
unmap the portion of all mmaps in the specified address_space corresponding to the specified byte range in the underlying file.
Parameters
structaddress_space*mappingthe address space containing mmaps to be unmapped.
loff_tconstholebeginbyte in first page to unmap, relative to the start ofthe underlying file. This will be rounded down to a PAGE_SIZEboundary. Note that this is different from
truncate_pagecache(), whichmust keep the partial page. In contrast, we must get rid ofpartial pages.loff_tconstholelensize of prospective hole in bytes. This will be roundedup to a PAGE_SIZE boundary. A holelen of zero truncates to theend of the file.
inteven_cows1 when truncating a file, unmap even private COWed pages;but 0 when invalidating pagecache, don’t throw away private data.
- intfollow_pfnmap_start(structfollow_pfnmap_args*args)¶
Look up a pfn mapping at a user virtual address
Parameters
structfollow_pfnmap_args*argsPointer to structfollow_pfnmap_args
Description
The caller needs to setup args->vma and args->address to point to thevirtual address as the target of such lookup. On a successful return,the results will be put into other output fields.
After the caller finished using the fields, the caller must invokeanotherfollow_pfnmap_end() to proper releases the locks and resourcesof such look up request.
During thestart() and end() calls, the results inargs will be validas proper locks will be held. After the end() is called, all the fieldsinfollow_pfnmap_args will be invalid to be further accessed. Furtheruse of such information after end() may require proper synchronizationsby the caller with page table updates, otherwise it can create asecurity bug.
If the PTE maps a refcounted page, callers are responsible to protectagainst invalidation with MMU notifiers; otherwise access to the PFN ata later point in time can trigger use-after-free.
Only IO mappings and raw PFN mappings are allowed. The mmap semaphoreshould be taken for read, and the mmap semaphore cannot be releasedbefore the end() is invoked.
This function must not be used to modify PTE content.
Return
zero on success, negative otherwise.
- voidfollow_pfnmap_end(structfollow_pfnmap_args*args)¶
End a
follow_pfnmap_start()process
Parameters
structfollow_pfnmap_args*argsPointer to structfollow_pfnmap_args
Description
Must be used in pair offollow_pfnmap_start(). See thestart() functionabove for more information.
- intgeneric_access_phys(structvm_area_struct*vma,unsignedlongaddr,void*buf,intlen,intwrite)¶
generic implementation for iomem mmap access
Parameters
structvm_area_struct*vmathe vma to access
unsignedlongaddruserspace address, not relative offset withinvma
void*bufbuffer to read/write
intlenlength of transfer
intwriteset to FOLL_WRITE when writing, otherwise reading
Description
This is a generic implementation forvm_operations_struct.access for aniomem mapping. This callback is used by access_process_vm() when thevma isnot page based.
- intcopy_remote_vm_str(structtask_struct*tsk,unsignedlongaddr,void*buf,intlen,unsignedintgup_flags)¶
copy a string from another process’s address space.
Parameters
structtask_struct*tskthe task of the target address space
unsignedlongaddrstart address to read from
void*bufdestination buffer
intlennumber of bytes to copy
unsignedintgup_flagsflags modifying lookup behaviour
Description
The caller must hold a reference onmm.
Return
number of bytes copied fromaddr (source) tobuf (destination);not including the trailing NUL. Always guaranteed to leave NUL-terminatedbuffer. On any error, return -EFAULT.
- unsignedlongget_pfnblock_flags_mask(conststructpage*page,unsignedlongpfn,unsignedlongmask)¶
Return the requested group of flags for the pageblock_nr_pages block of pages
Parameters
conststructpage*pageThe page within the block of interest
unsignedlongpfnThe target page frame number
unsignedlongmaskmask of bits that the caller is interested in
Return
pageblock_bits flags
- voidset_pfnblock_flags_mask(structpage*page,unsignedlongflags,unsignedlongpfn,unsignedlongmask)¶
Set the requested group of flags for a pageblock_nr_pages block of pages
Parameters
structpage*pageThe page within the block of interest
unsignedlongflagsThe flags to set
unsignedlongpfnThe target page frame number
unsignedlongmaskmask of bits that the caller is interested in
- boolmove_freepages_block_isolate(structzone*zone,structpage*page,intmigratetype)¶
move free pages in block for page isolation
Parameters
structzone*zonethe zone
structpage*pagethe pageblock page
intmigratetypemigratetype to set on the pageblock
Description
This is similar to move_freepages_block(), but handles the specialcase encountered in page isolation, where the block of interestmight be part of a larger buddy spanning multiple pageblocks.
Unlike the regular page allocator path, which moves pages whilestealing buddies off the freelist, page isolation is interested inarbitrary pfn ranges that may have overlapping buddies on both ends.
This function handles that. Straddling buddies are split intoindividual pageblocks. Only the block of interest is moved.
Returnstrue if pages could be moved,false otherwise.
- void__putback_isolated_page(structpage*page,unsignedintorder,intmt)¶
Return a now-isolated page back where we got it
Parameters
structpage*pagePage that was isolated
unsignedintorderOrder of the isolated page
intmtThe page’s pageblock’s migratetype
Description
This function is meant to return a page pulled from the free lists via__isolate_free_page back to the free lists they were pulled from.
- void___free_pages(structpage*page,unsignedintorder,fpi_tfpi_flags)¶
Free pages allocated with
alloc_pages().
Parameters
structpage*pageThe page pointer returned from
alloc_pages().unsignedintorderThe order of the allocation.
fpi_tfpi_flagsFree Page Internal flags.
Description
This function can free multi-page allocations that are not compoundpages. It does not check that theorder passed in matches that ofthe allocation, so it is easy to leak memory. Freeing more memorythan was allocated will probably emit a warning.
If the last reference to this page is speculative, it will be releasedby put_page() which only frees the first page of a non-compoundallocation. To prevent the remaining pages from being leaked, we freethe subsequent pages here. If you want to use the page’s referencecount to decide when to free the allocation, you should allocate acompound page, and use put_page() instead of __free_pages().
Context
May be called in interrupt context or while holding a normalspinlock, but not in NMI context or while holding a raw spinlock.
- void*alloc_pages_exact(size_tsize,gfp_tgfp_mask)¶
allocate an exact number physically-contiguous pages.
Parameters
size_tsizethe number of bytes to allocate
gfp_tgfp_maskGFP flags for the allocation, must not contain __GFP_COMP
Description
This function is similar toalloc_pages(), except that it allocates theminimum number of pages to satisfy the request.alloc_pages() can onlyallocate memory in power-of-two pages.
This function is also limited by MAX_PAGE_ORDER.
Memory allocated by this function must be released byfree_pages_exact().
Return
pointer to the allocated area orNULL in case of error.
- void*alloc_pages_exact_nid(intnid,size_tsize,gfp_tgfp_mask)¶
allocate an exact number of physically-contiguous pages on a node.
Parameters
intnidthe preferred node ID where memory should be allocated
size_tsizethe number of bytes to allocate
gfp_tgfp_maskGFP flags for the allocation, must not contain __GFP_COMP
Description
Likealloc_pages_exact(), but try to allocate on node nid first before fallingback.
Return
pointer to the allocated area orNULL in case of error.
- voidfree_pages_exact(void*virt,size_tsize)¶
release memory allocated via
alloc_pages_exact()
Parameters
void*virtthe value returned by alloc_pages_exact.
size_tsizesize of allocation, same value as passed to
alloc_pages_exact().
Description
Release the memory allocated by a previous call to alloc_pages_exact.
- unsignedlongnr_free_zone_pages(intoffset)¶
count number of pages beyond high watermark
Parameters
intoffsetThe zone index of the highest zone
Description
nr_free_zone_pages() counts the number of pages which are beyond thehigh watermark within all zones at or below a given zone index. For eachzone, the number of pages is calculated as:
nr_free_zone_pages = managed_pages - high_pages
Return
number of pages beyond high watermark.
- unsignedlongnr_free_buffer_pages(void)¶
count number of pages beyond high watermark
Parameters
voidno arguments
Description
nr_free_buffer_pages() counts the number of pages which are beyond the highwatermark within ZONE_DMA and ZONE_NORMAL.
Return
number of pages beyond high watermark within ZONE_DMA andZONE_NORMAL.
- intfind_next_best_node(intnode,nodemask_t*used_node_mask)¶
find the next node that should appear in a given node’s fallback list
Parameters
intnodenode whose fallback list we’re appending
nodemask_t*used_node_masknodemask_t of already used nodes
Description
We use a number of factors to determine which is the next node that shouldappear on a given node’s fallback list. The node should not have appearedalready innode’s fallback list, and it should be the next closest nodeaccording to the distance array (which contains arbitrary distance valuesfrom each node to each node in the system), and should also prefer nodeswith no CPUs, since presumably they’ll have very little allocation pressureon them otherwise.
Return
node id of the found node orNUMA_NO_NODE if no node is found.
- voidsetup_per_zone_wmarks(void)¶
called when min_free_kbytes changes or when memory is hot-{added|removed}
Parameters
voidno arguments
Description
Ensures that the watermark[min,low,high] values for each zone are setcorrectly with respect to min_free_kbytes.
- intalloc_contig_range(unsignedlongstart,unsignedlongend,unsignedmigratetype,gfp_tgfp_mask)¶
tries to allocate given range of pages
Parameters
unsignedlongstartstart PFN to allocate
unsignedlongendone-past-the-last PFN to allocate
unsignedmigratetypemigratetype of the underlying pageblocks (either#MIGRATE_MOVABLE or #MIGRATE_CMA). All pageblocksin range must have the same migratetype and it mustbe either of the two.
gfp_tgfp_maskGFP mask. Node/zone/placement hints are ignored; only someaction and reclaim modifiers are supported. Reclaim modifierscontrol allocation behavior during compaction/migration/reclaim.
Description
The PFN range does not have to be pageblock aligned. The PFN range mustbelong to a single zone.
The first thing this routine does is attempt to MIGRATE_ISOLATE allpageblocks in the range. Once isolated, the pageblocks should notbe modified by others.
Return
zero on success or negative error code. On success allpages which PFN is in [start, end) are allocated for the caller andneed to be freed with free_contig_range().
- structpage*alloc_contig_pages(unsignedlongnr_pages,gfp_tgfp_mask,intnid,nodemask_t*nodemask)¶
tries to find and allocate contiguous range of pages
Parameters
unsignedlongnr_pagesNumber of contiguous pages to allocate
gfp_tgfp_maskGFP mask. Node/zone/placement hints limit the search; only someaction and reclaim modifiers are supported. Reclaim modifierscontrol allocation behavior during compaction/migration/reclaim.
intnidTarget node
nodemask_t*nodemaskMask for other possible nodes
Description
This routine is a wrapper aroundalloc_contig_range(). It scans over zoneson an applicable zonelist to find a contiguous pfn range which can then betried for allocation withalloc_contig_range(). This routine is intendedfor allocation requests which can not be fulfilled with the buddy allocator.
The allocated memory is always aligned to a page boundary. If nr_pages is apower of two, then allocated range is also guaranteed to be aligned to samenr_pages (e.g. 1GB request would be aligned to 1GB).
Allocated pages can be freed with free_contig_range() or by manually calling__free_page() on each allocated page.
Return
pointer to contiguous pages on success, or NULL if not successful.
- structpage*alloc_pages_nolock(intnid,unsignedintorder)¶
opportunistic reentrant allocation from any context
Parameters
intnidnode to allocate from
unsignedintorderallocation order size
Description
Allocates pages of a given order from the given node. This is safe tocall from any context (from atomic, NMI, and also reentrantallocator -> tracepoint -> alloc_pages_nolock_noprof).Allocation is best effort and to be expected to fail easily so nobody shouldrely on the success. Failures are not reported via warn_alloc().See always fail conditions below.
Return
allocated page or NULL on failure. NULL does not mean EBUSY or EAGAIN.It means ENOMEM. There is no reason to call it again and expect !NULL.
- intnuma_nearest_node(intnode,unsignedintstate)¶
Find nearest node by state
Parameters
intnodeNode id to start the search
unsignedintstateState to filter the search
Description
Lookup the closest node by distance ifnid is not in state.
Return
thisnode if it is in state, otherwise the closest node by distance
- intnearest_node_nodemask(intnode,nodemask_t*mask)¶
Find the node inmask at the nearest distance fromnode.
Parameters
intnodea valid node ID to start the search from.
nodemask_t*maska pointer to a nodemask representing the allowed nodes.
Description
This function iterates over all nodes inmask and calculates thedistance from the startingnode, then it returns the node ID that isthe closest tonode, or MAX_NUMNODES if no node is found.
Note thatnode must be a valid node ID usable with node_distance(),providing an invalid node ID (e.g., NUMA_NO_NODE) may result in crashesor unexpected behavior.
- structpage*alloc_pages_mpol(gfp_tgfp,unsignedintorder,structmempolicy*pol,pgoff_tilx,intnid)¶
Allocate pages according to NUMA mempolicy.
Parameters
gfp_tgfpGFP flags.
unsignedintorderOrder of the page allocation.
structmempolicy*polPointer to the NUMA mempolicy.
pgoff_tilxIndex for interleave mempolicy (also distinguishes
alloc_pages()).intnidPreferred node (usually numa_node_id() butmpol may override it).
Return
The page on success or NULL if allocation fails.
- structfolio*vma_alloc_folio(gfp_tgfp,intorder,structvm_area_struct*vma,unsignedlongaddr)¶
Allocate a folio for a VMA.
Parameters
gfp_tgfpGFP flags.
intorderOrder of the folio.
structvm_area_struct*vmaPointer to VMA.
unsignedlongaddrVirtual address of the allocation. Must be insidevma.
Description
Allocate a folio for a specific address invma, using the appropriateNUMA policy. The caller must hold the mmap_lock of the mm_struct of theVMA to prevent it from going away. Should be used for all allocationsfor folios that will be mapped into user space, excepting hugetlbfs, andexcepting where direct use of folio_alloc_mpol() is more appropriate.
Return
The folio on success or NULL if allocation fails.
- structpage*alloc_pages(gfp_tgfp,unsignedintorder)¶
Allocate pages.
Parameters
gfp_tgfpGFP flags.
unsignedintorderPower of two of number of pages to allocate.
Description
Allocate 1 <<order contiguous pages. The physical address of thefirst page is naturally aligned (eg an order-3 allocation will be alignedto a multiple of 8 * PAGE_SIZE bytes). The NUMA policy of the currentprocess is honoured when in process context.
Context
Can be called from any context, providing the appropriate GFPflags are used.
Return
The page on success or NULL if allocation fails.
- intmpol_misplaced(structfolio*folio,structvm_fault*vmf,unsignedlongaddr)¶
check whether current folio node is valid in policy
Parameters
structfolio*foliofolio to be checked
structvm_fault*vmfstructure describing the fault
unsignedlongaddrvirtual address invma for shared policy lookup and interleave policy
Description
Lookup current policy node id for vma,addr and “compare to” folio’snode id. Policy determination “mimics” alloc_page_vma().Called from fault path where we know the vma and faulting address.
Return
NUMA_NO_NODE if the page is in a node that is valid for thispolicy, or a suitable node ID to allocate a replacement folio from.
- voidmpol_shared_policy_init(structshared_policy*sp,structmempolicy*mpol)¶
initialize shared policy for inode
Parameters
structshared_policy*sppointer to inode shared policy
structmempolicy*mpolstruct mempolicy to install
Description
Install non-NULLmpol in inode’s shared policy rb-tree.On entry, the current task has a reference on a non-NULLmpol.This must be released on exit.This is called at get_inode() calls and we can use GFP_KERNEL.
- intmpol_parse_str(char*str,structmempolicy**mpol)¶
parse string to mempolicy, for tmpfs mpol mount option.
Parameters
char*strstring containing mempolicy to parse
structmempolicy**mpolpointer to struct mempolicy pointer, returned on success.
Description
- Format of input:
<mode>[=<flags>][:<nodelist>]
Return
0 on success, else1
- voidmpol_to_str(char*buffer,intmaxlen,structmempolicy*pol)¶
format a mempolicy structure for printing
Parameters
char*bufferto contain formatted mempolicy string
intmaxlenlength ofbuffer
structmempolicy*polpointer to mempolicy to be formatted
Description
Convertpol into a string. Ifbuffer is too short, truncate the string.Recommend amaxlen of at least 51 for the longest mode, “weightedinterleave”, plus the longest flag flags, “relative|balancing”, and todisplay at least a few node ids.
- structfolio¶
Represents a contiguous set of bytes.
Definition:
struct folio { unsigned long flags; union { struct list_head lru; unsigned int mlock_count; struct dev_pagemap *pgmap; }; struct address_space *mapping; union { pgoff_t index; unsigned long share; }; union { void *private; swp_entry_t swap; }; atomic_t _mapcount; atomic_t _refcount;#ifdef CONFIG_MEMCG; unsigned long memcg_data;#elif defined(CONFIG_SLAB_OBJ_EXT); unsigned long _unused_slab_obj_exts;#endif;#if defined(WANT_PAGE_VIRTUAL); void *virtual;#endif;#ifdef LAST_CPUPID_NOT_IN_PAGE_FLAGS; int _last_cpupid;#endif; atomic_t _large_mapcount; atomic_t _nr_pages_mapped;#ifdef CONFIG_64BIT; atomic_t _entire_mapcount; atomic_t _pincount;#endif ; mm_id_mapcount_t _mm_id_mapcount[2]; union { mm_id_t _mm_id[2]; unsigned long _mm_ids; };#ifdef NR_PAGES_IN_LARGE_FOLIO; unsigned int _nr_pages;#endif ; struct list_head _deferred_list;#ifndef CONFIG_64BIT; atomic_t _entire_mapcount; atomic_t _pincount;#endif ; void *_hugetlb_subpool; void *_hugetlb_cgroup; void *_hugetlb_cgroup_rsvd; void *_hugetlb_hwpoison;};Members
flagsIdentical to the page flags.
{unnamed_union}anonymous
lruLeast Recently Used list; tracks how recently this folio was used.
mlock_countNumber of times this folio has been pinned by mlock().
pgmapMetadata for ZONE_DEVICE mappings
mappingThe file this page belongs to, or refers to the anon_vma foranonymous memory.
{unnamed_union}anonymous
indexOffset within the file, in units of pages. For anonymous memory,this is the index from the beginning of the mmap.
sharenumber of DAX mappings that reference this folio. Seedax_associate_entry.
{unnamed_union}anonymous
privateFilesystem per-folio data (see
folio_attach_private()).swapUsed for swp_entry_t if folio_test_swapcache().
_mapcountDo not access this member directly. Use
folio_mapcount()tofind out how many times this folio is mapped by userspace._refcountDo not access this member directly. Use
folio_ref_count()to find how many references there are to this folio.memcg_dataMemory Control Group data.
_unused_slab_obj_extsPlaceholder to match obj_exts in struct slab.
virtualVirtual address in the kernel direct map.
_last_cpupidIDs of last CPU and last process that accessed the folio.
_large_mapcountDo not use directly, call
folio_mapcount()._nr_pages_mappedDo not use outside of rmap and debug code.
_entire_mapcountDo not use directly, call folio_entire_mapcount().
_pincountDo not use directly, call
folio_maybe_dma_pinned()._mm_id_mapcountDo not use outside of rmap code.
{unnamed_union}anonymous
_mm_idDo not use outside of rmap code.
_mm_idsDo not use outside of rmap code.
_nr_pagesDo not use directly, call
folio_nr_pages()._deferred_listFolios to be split under memory pressure.
_entire_mapcountDo not use directly, call folio_entire_mapcount().
_pincountDo not use directly, call
folio_maybe_dma_pinned()._hugetlb_subpoolDo not use directly, use accessor in hugetlb.h.
_hugetlb_cgroupDo not use directly, use accessor in hugetlb_cgroup.h.
_hugetlb_cgroup_rsvdDo not use directly, use accessor in hugetlb_cgroup.h.
_hugetlb_hwpoisonDo not use directly, call raw_hwp_list_head().
Description
A folio is a physically, virtually and logically contiguous setof bytes. It is a power-of-two in size, and it is aligned to thatsame power-of-two. It is at least as large asPAGE_SIZE. If it isin the page cache, it is at a file offset which is a multiple of thatpower-of-two. It may be mapped into userspace at an address which isat an arbitrary page offset, but its kernel virtual address is alignedto its size.
- structptdesc¶
Memory descriptor for page tables.
Definition:
struct ptdesc { unsigned long __page_flags; union { struct rcu_head pt_rcu_head; struct list_head pt_list; struct { unsigned long _pt_pad_1; pgtable_t pmd_huge_pte; }; }; unsigned long __page_mapping; union { pgoff_t pt_index; struct mm_struct *pt_mm; atomic_t pt_frag_refcount;#ifdef CONFIG_HUGETLB_PMD_PAGE_TABLE_SHARING; atomic_t pt_share_count;#endif; }; union { unsigned long _pt_pad_2;#if ALLOC_SPLIT_PTLOCKS; spinlock_t *ptl;#else; spinlock_t ptl;#endif; }; unsigned int __page_type; atomic_t __page_refcount;#ifdef CONFIG_MEMCG; unsigned long pt_memcg_data;#endif;};Members
__page_flagsSame as page flags. Powerpc only.
{unnamed_union}anonymous
pt_rcu_headFor freeing page table pages.
pt_listList of used page tables. Used for s390 gmap shadow pages(which are not linked into the user page tables) and x86pgds.
{unnamed_struct}anonymous
_pt_pad_1Padding that aliases with page’s compound head.
pmd_huge_pteProtected by ptdesc->ptl, used for THPs.
__page_mappingAliases with page->mapping. Unused for page tables.
{unnamed_union}anonymous
pt_indexUsed for s390 gmap.
pt_mmUsed for x86 pgds.
pt_frag_refcountFor fragmented page table tracking. Powerpc only.
pt_share_countUsed for HugeTLB PMD page table share count.
{unnamed_union}anonymous
_pt_pad_2Padding to ensure proper alignment.
ptlLock for the page table.
ptlLock for the page table.
__page_typeSame as page->page_type. Unused for page tables.
__page_refcountSame as page refcount.
pt_memcg_dataMemcg data. Tracked for page tables here.
Description
This struct overlays struct page for now. Do not modify without a goodunderstanding of the issues.
- typevm_fault_t¶
Return type for page fault handlers.
Description
Page fault handlers return a bitmask ofVM_FAULT values.
- enumvm_fault_reason¶
Page fault handlers return a bitmask of these values to tell the core VM what happened when handling the fault. Used to decide whether a process gets delivered SIGBUS or just gets major/minor fault counters bumped up.
Constants
VM_FAULT_OOMOut Of Memory
VM_FAULT_SIGBUSBad access
VM_FAULT_MAJORPage read from storage
VM_FAULT_HWPOISONHit poisoned small page
VM_FAULT_HWPOISON_LARGEHit poisoned large page. Index encodedin upper bits
VM_FAULT_SIGSEGVsegmentation fault
VM_FAULT_NOPAGE->fault installed the pte, not return page
VM_FAULT_LOCKED->fault locked the returned page
VM_FAULT_RETRY->fault blocked, must retry
VM_FAULT_FALLBACKhuge page fault failed, fall back to small
VM_FAULT_DONE_COW->fault has fully handled COW
VM_FAULT_NEEDDSYNC->fault did not modify page tables and needsfsync() to complete (for synchronous page faultsin DAX)
VM_FAULT_COMPLETED->fault completed, meanwhile mmap lock released
VM_FAULT_HINDEX_MASKmask HINDEX value
- enumfault_flag¶
Fault flag definitions.
Constants
FAULT_FLAG_WRITEFault was a write fault.
FAULT_FLAG_MKWRITEFault was mkwrite of existing PTE.
FAULT_FLAG_ALLOW_RETRYAllow to retry the fault if blocked.
FAULT_FLAG_RETRY_NOWAITDon’t drop mmap_lock and wait when retrying.
FAULT_FLAG_KILLABLEThe fault task is in SIGKILL killable region.
FAULT_FLAG_TRIEDThe fault has been tried once.
FAULT_FLAG_USERThe fault originated in userspace.
FAULT_FLAG_REMOTEThe fault is not for current task/mm.
FAULT_FLAG_INSTRUCTIONThe fault was during an instruction fetch.
FAULT_FLAG_INTERRUPTIBLEThe fault can be interrupted by non-fatal signals.
FAULT_FLAG_UNSHAREThe fault is an unsharing request to break COW in aCOW mapping, making sure that an exclusive anon page ismapped after the fault.
FAULT_FLAG_ORIG_PTE_VALIDwhether the fault has vmf->orig_pte cached.We should only access orig_pte if this flag set.
FAULT_FLAG_VMA_LOCKThe fault is handled under VMA lock.
Description
AboutFAULT_FLAG_ALLOW_RETRY andFAULT_FLAG_TRIED: we can specifywhether we would allow page faults to retry by specifying these twofault flags correctly. Currently there can be three legal combinations:
- ALLOW_RETRY and !TRIED: this means the page fault allows retry, and
this is the first try
- ALLOW_RETRY and TRIED: this means the page fault allows retry, and
we’ve already tried at least once
!ALLOW_RETRY and !TRIED: this means the page fault does not allow retry
The unlisted combination (!ALLOW_RETRY && TRIED) is illegal and should neverbe used. Note that page faults can be allowed to retry for multiple times,in which case we’ll have an initial fault with flags (a) then later oncontinuous faults with flags (b). We should always try to detect pendingsignals before a retry to make sure the continuous page faults can still beinterrupted if necessary.
The combination FAULT_FLAG_WRITE|FAULT_FLAG_UNSHARE is illegal.FAULT_FLAG_UNSHARE is ignored and treated like an ordinary read fault whenapplied to mappings that are not COW mappings.
Parameters
structfolio*folioThe folio to test.
Description
We would like to get this info without a page flag, but the stateneeds to survive until the folio is last deleted from the LRU, whichcould be as far down as __page_cache_release.
Return
An integer (not a boolean!) used to sort a folio onto theright LRU list and to account folios correctly.1 iffolio is a regular filesystem backed page cache folioor a lazily freed anonymous folio (e.g. via MADV_FREE).0 iffolio is a normal anonymous folio, a tmpfs folio or otherwiseram or swap backed folio.
Parameters
structfolio*folioThe folio that was on lru and now has a zero reference.
Parameters
structfolio*folioThe folio to test.
Return
The LRU list a folio should be on, as an indexinto the array of LRU lists.
- page_folio¶
page_folio(p)
Converts from page to folio.
Parameters
pThe page.
Description
Every page is part of a folio. This function cannot be called on aNULL pointer.
Context
No reference, nor lock is required onpage. If the callerdoes not hold a reference, this call may race with a folio split, soit should re-check the folio still contains this page after gaininga reference on the folio.
Return
The folio which contains this page.
- folio_page¶
folio_page(folio,n)
Return a page from a folio.
Parameters
folioThe folio.
nThe page number to return.
Description
n is relative to the start of the folio. This function does notcheck that the page number lies withinfolio; the caller is presumedto have a reference to the page.
Parameters
structfolio*folioThe folio.
unsignedlongmaskBits set in this word will be changed.
Description
This must only be used for flags which are changed with the foliolock held. For example, it is unsafe to use for PG_dirty as thatcan be set without the folio lock held. It can also only be usedon flags which are in the range 0-6 as some of the implementationsonly affect those bits.
Return
Whether there are tasks waiting on the folio.
Parameters
conststructfolio*folioThe folio.
Description
The uptodate flag is set on a folio when every byte in the folio isat least as new as the corresponding bytes on storage. Anonymousand CoW folios are always uptodate. If the folio is not uptodate,some of the bytes in it may be; see the is_partially_uptodate()address_space operation.
Parameters
conststructfolio*folioThe folio to test.
Return
True if the folio is larger than one page.
Parameters
conststructpage*pageThe page to test.
Context
Any context.
Return
True for slab pages, false for any other kind of page.
Parameters
conststructpage*pageThe page to test.
Context
Any context.
Return
True for hugetlbfs pages, false for anon pages or pagesbelonging to other filesystems.
Parameters
conststructfolio*folioThe folio to be checked
Description
Determine if a folio has private stuff, indicating that release routinesshould be invoked upon it.
- boolfault_flag_allow_retry_first(enumfault_flagflags)¶
check ALLOW_RETRY the first time
Parameters
enumfault_flagflagsFault flags.
Description
This is mostly used for places where we want to try to avoid takingthe mmap_lock for too long a time when waiting for another conditionto change, in which case we can try to be polite to release themmap_lock in the first round to avoid potential starvation of otherprocesses that would also want the mmap_lock.
Return
true if the page fault allows retry and this is the firstattempt of the fault handling; false otherwise.
Parameters
conststructfolio*folioThe folio.
Description
A folio is composed of 2^order pages. See get_order() for the definitionof order.
Return
The order of the folio.
Parameters
structfolio*folioThe folio.
Description
Reset the order and derived _nr_pages to 0. Must only be used in theprocess of splitting large folios.
Parameters
conststructfolio*folioThe folio.
Description
The folio mapcount corresponds to the number of present user page tableentries that reference any part of a folio. Each such present user pagetable entry must be paired with exactly on folio reference.
For ordindary folios, each user page table entry (PTE/PMD/PUD/...) countsexactly once.
For hugetlb folios, each abstracted “hugetlb” user page table entry thatreferences the entire folio counts exactly once, even when such specialpage table entries are comprised of multiple ordinary page table entries.
Will report 0 for pages which cannot be mapped into userspace, such asslab, page tables and similar.
Return
The number of times this folio is mapped.
Parameters
conststructfolio*folioThe folio.
Return
True if any page in this folio is referenced by user page tables.
Parameters
structpage*pageHead page of a transparent huge page.
Parameters
structpage*pageHead page of a transparent huge page.
Return
Number of bytes in this page.
Parameters
structfolio*folioThe folio.
Context
May be called in any context, as long as you know thatyou have a refcount on the folio. If you do not already have one,folio_try_get() may be the right interface for you to use.
Parameters
structfolio*folioThe folio.
Description
If the folio’s reference count reaches zero, the memory will bereleased back to the page allocator and may be used by anotherallocation immediately. Do not access the memory or thestructfolioafter callingfolio_put() unless you can be sure that it wasn’t thelast reference.
Context
May be called in process or interrupt context, but not in NMIcontext. May be called while holding a spinlock.
Parameters
structfolio*folioThe folio.
intrefsThe amount to subtract from the folio’s reference count.
Description
If the folio’s reference count reaches zero, the memory will bereleased back to the page allocator and may be used by anotherallocation immediately. Do not access the memory or thestructfolioafter callingfolio_put_refs() unless you can be sure that these weren’tthe last references.
Context
May be called in process or interrupt context, but not in NMIcontext. May be called while holding a spinlock.
- voidfolios_put(structfolio_batch*folios)¶
Decrement the reference count on an array of folios.
Parameters
structfolio_batch*foliosThe folios.
Description
Likefolio_put(), but for a batch of folios. This is more efficientthan writing the loop yourself as it will optimise the locks which needto be taken if the folios are freed. The folios batch is returnedempty and ready to be reused for another batch; there is no need toreinitialise it.
Context
May be called in process or interrupt context, but not in NMIcontext. May be called while holding a spinlock.
Parameters
conststructfolio*folioThe folio.
Description
A folio may contain multiple pages. The pages have consecutivePage Frame Numbers.
Return
The Page Frame Number of the first page in the folio.
Parameters
structfolio*folioThe folio to create a PTE for
pgprot_tpgprotThe page protection bits to use
Description
Create a page table entry for the first page of this folio.This is suitable for passing to set_ptes().
Return
A page table entry suitable for mapping this folio.
Parameters
structfolio*folioThe folio to create a PMD for
pgprot_tpgprotThe page protection bits to use
Description
Create a page table entry for the first page of this folio.This is suitable for passing to set_pmd_at().
Return
A page table entry suitable for mapping this folio.
Parameters
structfolio*folioThe folio.
Description
This function checks if a folio has been pinned via a call toa function in the pin_user_pages() family.
For small folios, the return value is partially fuzzy: false is not fuzzy,because it means “definitely not pinned for DMA”, but true means “probablypinned for DMA, but possibly a false positive due to having at leastGUP_PIN_COUNTING_BIAS worth of normal folio references”.
False positives are OK, because: a) it’s unlikely for a folio toget that many refcounts, and b) all the callers of this routine areexpected to be able to deal gracefully with a false positive.
For most large folios, the result will be exactly correct. That’s becausewe have more tracking data available: the _pincount field is usedinstead of the GUP_PIN_COUNTING_BIAS scheme.
For more information, please seepin_user_pages() and related calls.
Return
True, if it is likely that the folio has been “dma-pinned”.False, if the folio is definitely not dma-pinned.
Parameters
conststructpage*pageThe page to query
Description
This returns true ifpage is one of the permanent zero pages.
Parameters
conststructfolio*folioThe folio to query
Description
This returns true iffolio is one of the permanent zero pages.
Parameters
conststructfolio*folioThe folio.
Return
A positive power of two.
Parameters
structfolio*folioThe folio we’re currently operating on.
Description
If you have physically contiguous memory which may span more thanone folio (eg astructbio_vec), use this function to move from onefolio to the next. Do not use it if the memory is only virtuallycontiguous as the folios are almost certainly not adjacent to eachother. This is the folio equivalent to writingpage++.
Context
We assume that the folios are refcounted and/or locked at ahigher level and do not adjust the reference counts.
Return
The nextstructfolio.
Parameters
conststructfolio*folioThe folio.
Description
A folio represents a number of bytes which is a power-of-two in size.This function tells you which power-of-two the folio is. See alsofolio_size() andfolio_order().
Context
The caller should have a reference on the folio to preventit from being split. It is not necessary for the folio to be locked.
Return
The base-2 logarithm of the size of this folio.
Parameters
conststructfolio*folioThe folio.
Context
The caller should have a reference on the folio to preventit from being split. It is not necessary for the folio to be locked.
Return
The number of bytes in this folio.
- boolfolio_maybe_mapped_shared(structfolio*folio)¶
Whether the folio is mapped into the page tables of more than one MM
Parameters
structfolio*folioThe folio.
Description
This function checks if the folio maybe currently mapped into more than oneMM (“maybe mapped shared”), or if the folio is certainly mapped into a singleMM (“mapped exclusively”).
For KSM folios, this function also returns “mapped shared” when a folio ismapped multiple times into the same MM, because the individual page mappingsare independent.
For small anonymous folios and anonymous hugetlb folios, the returnvalue will be exactly correct: non-KSM folios can only be mapped at most onceinto an MM, and they cannot be partially mapped. KSM folios areconsidered shared even if mapped multiple times into the same MM.
- For other folios, the result can be fuzzy:
For partially-mappable large folios (THP), the return value can wronglyindicate “mapped shared” (false positive) if a folio was mapped bymore than two MMs at one point in time.
For pagecache folios (including hugetlb), the return value can wronglyindicate “mapped shared” (false positive) when two VMAs in the same MMcover the same file range.
Further, this function only considers current page table mappings thatare tracked using the folio mapcount(s).
- This function does not consider:
If the folio might get mapped in the (near) future (e.g., swapcache,pagecache, temporary unmapping for migration).
If the folio is mapped differently (VM_PFNMAP).
If hugetlb page table sharing applies. Callers might want to checkhugetlb_pmd_shared().
Return
Whether the folio is estimated to be mapped into more than one MM.
Parameters
conststructfolio*foliothe folio
Description
Calculate the expected folio refcount, taking references from the pagecache,swapcache, PG_private and page table mappings into account. Useful incombination withfolio_ref_count() to detect unexpected references (e.g.,GUP or other temporary references).
Does currently not consider references from the LRU cache. If the foliowas isolated from the LRU (which is the case during migration or split),the LRU cache does not apply.
Calling this function on an unmapped folio -- !folio_mapped() -- that islocked will return a stable result.
Calling this function on a mapped folio will not result in a stable result,because nothing stops additional page table mappings from coming (e.g.,fork()) or going (e.g., munmap()).
Calling this function without the folio lock will also not result in astable result: for example, the folio might get dropped from the swapcacheconcurrently.
However, even when called without the folio lock or on a mapped folio,this function can be used to detect unexpected references early (for example,if it makes sense to even lock the folio and unmap it).
The caller must add any reference (e.g., fromfolio_try_get()) it might beholding itself to the result.
Returns the expected folio refcount.
Parameters
gfp_tgfpGFP flags
unsignedintorderdesired pagetable order
Description
pagetable_alloc allocates memory for page tables as well as a page tabledescriptor to describe that memory.
Return
The ptdesc describing the allocated page tables.
Parameters
structptdesc*ptThe page table descriptor
Description
pagetable_free frees the memory of all page tables described by a pagetable descriptor and the memory for the descriptor itself.
- structvm_area_struct*vma_lookup(structmm_struct*mm,unsignedlongaddr)¶
Find a VMA at a specific address
Parameters
structmm_struct*mmThe process address space.
unsignedlongaddrThe user address.
Return
The vm_area_struct at the given address,NULL otherwise.
- boolvma_is_special_huge(conststructvm_area_struct*vma)¶
Are transhuge page-table entries considered special?
Parameters
conststructvm_area_struct*vmaPointer to the struct vm_area_struct to consider
Description
Whether transhuge page-table entries are considered “special” followingthe definition in vm_normal_page().
Return
true if transhuge page-table entries should be considered special,false otherwise.
Parameters
conststructfolio*folioThe folio.
Description
The refcount is usually incremented by calls tofolio_get() anddecremented by calls tofolio_put(). Some typical users of thefolio refcount:
Each reference from a page table
The page cache
Filesystem private data
The LRU list
Pipes
Direct IO which references this page in the process address space
Return
The number of references to this folio.
Parameters
structfolio*folioThe folio.
Description
If you do not already have a reference to a folio, you can attempt toget one using this function. It may fail if, for example, the foliohas been freed since you found a pointer to it, or it is frozen forthe purposes of splitting or migration.
Return
True if the reference count was successfully incremented.
- intis_highmem(structzone*zone)¶
helper function to quickly check if a struct zone is a highmem zone or not. This is an attempt to keep references to ZONE_{DMA/NORMAL/HIGHMEM/etc} in general code to a minimum.
Parameters
structzone*zonepointer to struct zone variable
Return
1 for a highmem zone, 0 otherwise
- for_each_online_pgdat¶
for_each_online_pgdat(pgdat)
helper macro to iterate over all online nodes
Parameters
pgdatpointer to a pg_data_t variable
- for_each_zone¶
for_each_zone(zone)
helper macro to iterate over all memory zones
Parameters
zonepointer to struct zone variable
Description
The user only needs to declare the zone variable, for_each_zonefills it in.
- structzoneref*next_zones_zonelist(structzoneref*z,enumzone_typehighest_zoneidx,nodemask_t*nodes)¶
Returns the next zone at or below highest_zoneidx within the allowed nodemask using a cursor within a zonelist as a starting point
Parameters
structzoneref*zThe cursor used as a starting point for the search
enumzone_typehighest_zoneidxThe zone index of the highest zone to return
nodemask_t*nodesAn optional nodemask to filter the zonelist with
Description
This function returns the next zone at or below a given zone index that iswithin the allowed nodemask using a cursor as the starting point for thesearch. The zoneref returned is a cursor that represents the current zonebeing examined. It should be advanced by one before callingnext_zones_zonelist again.
Return
the next zone at or below highest_zoneidx within the allowednodemask using a cursor within a zonelist as a starting point
- structzoneref*first_zones_zonelist(structzonelist*zonelist,enumzone_typehighest_zoneidx,nodemask_t*nodes)¶
Returns the first zone at or below highest_zoneidx within the allowed nodemask in a zonelist
Parameters
structzonelist*zonelistThe zonelist to search for a suitable zone
enumzone_typehighest_zoneidxThe zone index of the highest zone to return
nodemask_t*nodesAn optional nodemask to filter the zonelist with
Description
This function returns the first zone at or below a given zone index that iswithin the allowed nodemask. The zoneref returned is a cursor that can beused to iterate the zonelist with next_zones_zonelist by advancing it byone before calling.
When no eligible zone is found, zoneref->zone is NULL (zoneref itself isnever NULL). This may happen either genuinely, or due to concurrent nodemaskupdate due to cpuset modification.
Return
Zoneref pointer for the first suitable zone found
- for_each_zone_zonelist_nodemask¶
for_each_zone_zonelist_nodemask(zone,z,zlist,highidx,nodemask)
helper macro to iterate over valid zones in a zonelist at or below a given zone index and within a nodemask
Parameters
zoneThe current zone in the iterator
zThe current pointer within zonelist->_zonerefs being iterated
zlistThe zonelist being iterated
highidxThe zone index of the highest zone to return
nodemaskNodemask allowed by the allocator
Description
This iterator iterates though all zones at or below a given zone index andwithin a given nodemask
- for_each_zone_zonelist¶
for_each_zone_zonelist(zone,z,zlist,highidx)
helper macro to iterate over valid zones in a zonelist at or below a given zone index
Parameters
zoneThe current zone in the iterator
zThe current pointer within zonelist->zones being iterated
zlistThe zonelist being iterated
highidxThe zone index of the highest zone to return
Description
This iterator iterates though all zones at or below a given zone index.
- intpfn_valid(unsignedlongpfn)¶
check if there is a valid memory map entry for a PFN
Parameters
unsignedlongpfnthe page frame number to check
Description
Check if there is a valid memory map entry aka struct page for thepfn.Note, that availability of the memory map entry does not imply thatthere is actual usable memory at thatpfn. The struct page mayrepresent a hole or an unusable page frame.
Return
1 for PFNs that have memory map entries and 0 otherwise
- structaddress_space*folio_mapping(structfolio*folio)¶
Find the mapping where this folio is stored.
Parameters
structfolio*folioThe folio.
Description
For folios which are in the page cache, return the mapping that thispage belongs to. Folios in the swap cache return the swap mappingthis page is stored in (which is different from the mapping for theswap file or swap device where the data is stored).
You can call this for folios which aren’t in the swap cache or pagecache and it will return NULL.
- int__anon_vma_prepare(structvm_area_struct*vma)¶
attach an anon_vma to a memory region
Parameters
structvm_area_struct*vmathe memory region in question
Description
This makes sure the memory mapping described by ‘vma’ hasan ‘anon_vma’ attached to it, so that we can associate theanonymous pages mapped into it with that anon_vma.
The common case will be that we already have one, whichis handled inline by anon_vma_prepare(). But ifnot we either need to find an adjacent mapping that wecan re-use the anon_vma from (very common when the onlyreason for splitting a vma has been mprotect()), or weallocate a new one.
Anon-vma allocations are very subtle, because we may haveoptimistically looked up an anon_vma in folio_lock_anon_vma_read()and that may actually touch the rwsem even in the newlyallocated vma (it depends on RCU to make sure that theanon_vma isn’t actually destroyed).
As a result, we need to do proper anon_vma locking evenfor the new allocation. At the same time, we do not wantto do any locking for the common case of already havingan anon_vma.
- unsignedlongpage_address_in_vma(conststructfolio*folio,conststructpage*page,conststructvm_area_struct*vma)¶
The virtual address of a page in this VMA.
Parameters
conststructfolio*folioThe folio containing the page.
conststructpage*pageThe page within the folio.
conststructvm_area_struct*vmaThe VMA we need to know the address in.
Description
Calculates the user virtual address of this page in the specified VMA.It is the caller’s responsibility to check the page is actuallywithin the VMA. There may not currently be a PTE pointing at thispage, but if a page fault occurs at this address, this is the pagewhich will be accessed.
Context
Caller should hold a reference to the folio. Caller shouldhold a lock (eg the i_mmap_lock or the mmap_lock) which keeps theVMA from being altered.
Return
The virtual address corresponding to this page in the VMA.
- intfolio_referenced(structfolio*folio,intis_locked,structmem_cgroup*memcg,unsignedlong*vm_flags)¶
Test if the folio was referenced.
Parameters
structfolio*folioThe folio to test.
intis_lockedCaller holds lock on the folio.
structmem_cgroup*memcgtarget memory cgroup
unsignedlong*vm_flagsA combination of all the vma->vm_flags which referenced the folio.
Description
Quick test_and_clear_referenced for all mappings of a folio,
Return
The number of mappings which referenced the folio. Return -1 ifthe function bailed out due to rmap lock contention.
- intmapping_wrprotect_range(structaddress_space*mapping,pgoff_tpgoff,unsignedlongpfn,unsignedlongnr_pages)¶
Write-protect all mappings in a specified range.
Parameters
structaddress_space*mappingThe mapping whose reverse mapping should be traversed.
pgoff_tpgoffThe page offset at whichpfn is mapped withinmapping.
unsignedlongpfnThe PFN of the page mapped inmapping atpgoff.
unsignedlongnr_pagesThe number of physically contiguous base pages spanned.
Description
Traverses the reverse mapping, finding all VMAs which contain a sharedmapping of the pages in the specified range inmapping, and write-protectsthem (that is, updates the page tables to mark the mappings read-only suchthat a write protection fault arises when the mappings are written to).
Thepfn value need not refer to a folio, but rather can reference a kernelallocation which is mapped into userland. We therefore do not require thatthe page maps to a folio with a valid mapping or index field, rather thecaller specifies these inmapping andpgoff.
Return
the number of write-protected PTEs, or an error.
- intpfn_mkclean_range(unsignedlongpfn,unsignedlongnr_pages,pgoff_tpgoff,structvm_area_struct*vma)¶
Cleans the PTEs (including PMDs) mapped with range of [pfn,pfn +nr_pages) at the specific offset (pgoff) within thevma of shared mappings. And since clean PTEs should also be readonly, write protects them too.
Parameters
unsignedlongpfnstart pfn.
unsignedlongnr_pagesnumber of physically contiguous pages srarting withpfn.
pgoff_tpgoffpage offset that thepfn mapped with.
structvm_area_struct*vmavma thatpfn mapped within.
Description
Returns the number of cleaned PTEs (including PMDs).
Parameters
structfolio*folioThe folio to move to our anon_vma
structvm_area_struct*vmaThe vma the folio belongs to
Description
When a folio belongs exclusively to one process after a COW event,that folio can be moved into the anon_vma that belongs to just thatprocess, so the rmap code will not search the parent or sibling processes.
- void__folio_set_anon(structfolio*folio,structvm_area_struct*vma,unsignedlongaddress,boolexclusive)¶
set up a new anonymous rmap for a folio
Parameters
structfolio*folioThe folio to set up the new anonymous rmap for.
structvm_area_struct*vmaVM area to add the folio to.
unsignedlongaddressUser virtual address of the mapping
boolexclusiveWhether the folio is exclusive to the process.
- void__page_check_anon_rmap(conststructfolio*folio,conststructpage*page,structvm_area_struct*vma,unsignedlongaddress)¶
sanity check anonymous rmap addition
Parameters
conststructfolio*folioThe folio containingpage.
conststructpage*pagethe page to check the mapping of
structvm_area_struct*vmathe vm area in which the mapping is added
unsignedlongaddressthe user virtual address mapped
- voidfolio_add_anon_rmap_ptes(structfolio*folio,structpage*page,intnr_pages,structvm_area_struct*vma,unsignedlongaddress,rmap_tflags)¶
add PTE mappings to a page range of an anon folio
Parameters
structfolio*folioThe folio to add the mappings to
structpage*pageThe first page to add
intnr_pagesThe number of pages which will be mapped
structvm_area_struct*vmaThe vm area in which the mappings are added
unsignedlongaddressThe user virtual address of the first page to map
rmap_tflagsThe rmap flags
Description
The page range of folio is defined by [first_page, first_page + nr_pages)
The caller needs to hold the page table lock, and the page must be locked inthe anon_vma case: to serialize mapping,index checking after setting,and to ensure that an anon folio is not being upgraded racily to a KSM folio(but KSM folios are never downgraded).
- voidfolio_add_anon_rmap_pmd(structfolio*folio,structpage*page,structvm_area_struct*vma,unsignedlongaddress,rmap_tflags)¶
add a PMD mapping to a page range of an anon folio
Parameters
structfolio*folioThe folio to add the mapping to
structpage*pageThe first page to add
structvm_area_struct*vmaThe vm area in which the mapping is added
unsignedlongaddressThe user virtual address of the first page to map
rmap_tflagsThe rmap flags
Description
The page range of folio is defined by [first_page, first_page + HPAGE_PMD_NR)
The caller needs to hold the page table lock, and the page must be locked inthe anon_vma case: to serialize mapping,index checking after setting.
- voidfolio_add_new_anon_rmap(structfolio*folio,structvm_area_struct*vma,unsignedlongaddress,rmap_tflags)¶
Add mapping to a new anonymous folio.
Parameters
structfolio*folioThe folio to add the mapping to.
structvm_area_struct*vmathe vm area in which the mapping is added
unsignedlongaddressthe user virtual address mapped
rmap_tflagsThe rmap flags
Description
Like folio_add_anon_rmap_*() but must only be called onnew folios.This means the inc-and-test can be bypassed.The folio doesn’t necessarily need to be locked while it’s exclusiveunless two threads map it concurrently. However, the folio must belocked if it’s shared.
If the folio is pmd-mappable, it is accounted as a THP.
- voidfolio_add_file_rmap_ptes(structfolio*folio,structpage*page,intnr_pages,structvm_area_struct*vma)¶
add PTE mappings to a page range of a folio
Parameters
structfolio*folioThe folio to add the mappings to
structpage*pageThe first page to add
intnr_pagesThe number of pages that will be mapped using PTEs
structvm_area_struct*vmaThe vm area in which the mappings are added
Description
The page range of the folio is defined by [page, page + nr_pages)
The caller needs to hold the page table lock.
- voidfolio_add_file_rmap_pmd(structfolio*folio,structpage*page,structvm_area_struct*vma)¶
add a PMD mapping to a page range of a folio
Parameters
structfolio*folioThe folio to add the mapping to
structpage*pageThe first page to add
structvm_area_struct*vmaThe vm area in which the mapping is added
Description
The page range of the folio is defined by [page, page + HPAGE_PMD_NR)
The caller needs to hold the page table lock.
- voidfolio_add_file_rmap_pud(structfolio*folio,structpage*page,structvm_area_struct*vma)¶
add a PUD mapping to a page range of a folio
Parameters
structfolio*folioThe folio to add the mapping to
structpage*pageThe first page to add
structvm_area_struct*vmaThe vm area in which the mapping is added
Description
The page range of the folio is defined by [page, page + HPAGE_PUD_NR)
The caller needs to hold the page table lock.
- voidfolio_remove_rmap_ptes(structfolio*folio,structpage*page,intnr_pages,structvm_area_struct*vma)¶
remove PTE mappings from a page range of a folio
Parameters
structfolio*folioThe folio to remove the mappings from
structpage*pageThe first page to remove
intnr_pagesThe number of pages that will be removed from the mapping
structvm_area_struct*vmaThe vm area from which the mappings are removed
Description
The page range of the folio is defined by [page, page + nr_pages)
The caller needs to hold the page table lock.
- voidfolio_remove_rmap_pmd(structfolio*folio,structpage*page,structvm_area_struct*vma)¶
remove a PMD mapping from a page range of a folio
Parameters
structfolio*folioThe folio to remove the mapping from
structpage*pageThe first page to remove
structvm_area_struct*vmaThe vm area from which the mapping is removed
Description
The page range of the folio is defined by [page, page + HPAGE_PMD_NR)
The caller needs to hold the page table lock.
- voidfolio_remove_rmap_pud(structfolio*folio,structpage*page,structvm_area_struct*vma)¶
remove a PUD mapping from a page range of a folio
Parameters
structfolio*folioThe folio to remove the mapping from
structpage*pageThe first page to remove
structvm_area_struct*vmaThe vm area from which the mapping is removed
Description
The page range of the folio is defined by [page, page + HPAGE_PUD_NR)
The caller needs to hold the page table lock.
- voidtry_to_unmap(structfolio*folio,enumttu_flagsflags)¶
Try to remove all page table mappings to a folio.
Parameters
structfolio*folioThe folio to unmap.
enumttu_flagsflagsaction and flags
Description
Tries to remove all the page table entries which are mapping thisfolio. It is the caller’s responsibility to check if the folio isstill mapped if needed (use TTU_SYNC to prevent accounting races).
Context
Caller must hold the folio lock.
- voidtry_to_migrate(structfolio*folio,enumttu_flagsflags)¶
try to replace all page table mappings with swap entries
Parameters
structfolio*foliothe folio to replace page table entries for
enumttu_flagsflagsaction and flags
Description
Tries to remove all the page table entries which are mapping this folio andreplace them with special swap entries. Caller must hold the folio lock.
- structpage*make_device_exclusive(structmm_struct*mm,unsignedlongaddr,void*owner,structfolio**foliop)¶
Mark a page for exclusive use by a device
Parameters
structmm_struct*mmmm_struct of associated target process
unsignedlongaddrthe virtual address to mark for exclusive device access
void*ownerpassed to MMU_NOTIFY_EXCLUSIVE range notifier to allow filtering
structfolio**foliopfolio pointer will be stored here on success.
Description
This function looks up the page mapped at the given address, grabs afolio reference, locks the folio and replaces the PTE with specialdevice-exclusive PFN swap entry, preventing access through the processpage tables. The function will return with the folio locked and referenced.
On fault, the device-exclusive entries are replaced with the original PTEunder folio lock, after calling MMU notifiers.
Only anonymous non-hugetlb folios are supported and the VMA must havewrite permissions such that we can fault in the anonymous page writablein order to mark it exclusive. The caller must hold the mmap_lock in readmode.
A driver using this to program access from a device must use a mmu notifiercritical section to hold a device specific lock during programming. Onceprogramming is complete it should drop the folio lock and reference afterwhich point CPU access to the page will revoke the exclusive access.
Notes
This function always operates on individual PTEs mapping individualpages. PMD-sized THPs are first remapped to be mapped by PTEs beforethe conversion happens on a single PTE corresponding toaddr.
While concurrent access through the process page tables is prevented,concurrent access through other page references (e.g., earlier GUPinvocation) is not handled and not supported.
device-exclusive entries are considered “clean” and “old” by core-mm.Device drivers must update the folio state when informed by MMUnotifiers.
Return
pointer to mapped page on success, otherwise a negative error.
- void__rmap_walk_file(structfolio*folio,structaddress_space*mapping,pgoff_tpgoff_start,unsignedlongnr_pages,structrmap_walk_control*rwc,boollocked)¶
Traverse the reverse mapping for a file-backed mapping of a page mapped within a specified page cache object at a specified offset.
Parameters
structfolio*folioEither the folio whose mappings to traverse, or if NULL,the callbacks specified inrwc will be configured suchas to be able to look up mappings correctly.
structaddress_space*mappingThe page cache object whose mapping VMAs we intend totraverse. Iffolio is non-NULL, this should be equal tofolio_mapping(folio).
pgoff_tpgoff_startThe offset withinmapping of the page which we arelooking up. Iffolio is non-NULL, this should be equalto folio_pgoff(folio).
unsignedlongnr_pagesThe number of pages mapped by the mapping. Iffolio isnon-NULL, this should be equal to folio_nr_pages(folio).
structrmap_walk_control*rwcThe reverse mapping walk control object describing howthe traversal should proceed.
boollockedIs themapping already locked? If not, we acquire thelock.
- intmigrate_folio(structaddress_space*mapping,structfolio*dst,structfolio*src,enummigrate_modemode)¶
Simple folio migration.
Parameters
structaddress_space*mappingThe address_space containing the folio.
structfolio*dstThe folio to migrate the data to.
structfolio*srcThe folio containing the current data.
enummigrate_modemodeHow to migrate the page.
Description
Common logic to directly migrate a single LRU folio suitable forfolios that do not have private data.
Folios are locked upon entry and exit.
- intbuffer_migrate_folio(structaddress_space*mapping,structfolio*dst,structfolio*src,enummigrate_modemode)¶
Migration function for folios with buffers.
Parameters
structaddress_space*mappingThe address space containingsrc.
structfolio*dstThe folio to migrate to.
structfolio*srcThe folio to migrate from.
enummigrate_modemodeHow to migrate the folio.
Description
This function can only be used if the underlying filesystem guaranteesthat no other references tosrc exist. For example attached bufferheads are accessed only under the folio lock. If your filesystem cannotprovide this guarantee,buffer_migrate_folio_norefs() may be moreappropriate.
Return
0 on success or a negative errno on failure.
- intbuffer_migrate_folio_norefs(structaddress_space*mapping,structfolio*dst,structfolio*src,enummigrate_modemode)¶
Migration function for folios with buffers.
Parameters
structaddress_space*mappingThe address space containingsrc.
structfolio*dstThe folio to migrate to.
structfolio*srcThe folio to migrate from.
enummigrate_modemodeHow to migrate the folio.
Description
Likebuffer_migrate_folio() except that this variant is more carefuland checks that there are also no buffer head references. This functionis the right one for mappings where buffer heads are directly lookedup and referenced (such as block device mappings).
Return
0 on success or a negative errno on failure.
- unsignedlongdo_mmap(structfile*file,unsignedlongaddr,unsignedlonglen,unsignedlongprot,unsignedlongflags,vm_flags_tvm_flags,unsignedlongpgoff,unsignedlong*populate,structlist_head*uf)¶
Perform a userland memory mapping into the current process address space of lengthlen with protection bitsprot, mmap flagsflags (from which VMA flags will be inferred), and any additional VMA flags to applyvm_flags. If this is a file-backed mapping then the file is specified infile and page offset into the file viapgoff.
Parameters
structfile*fileAn optional
structfilepointer describing the file which is to bemapped, if a file-backed mapping.unsignedlongaddrIf non-zero, hints at (or ifflags has MAP_FIXED set, specifies) theaddress at which to perform this mapping. See mmap (2) for details. Must bepage-aligned.
unsignedlonglenThe length of the mapping. Will be page-aligned and must be at least 1page in size.
unsignedlongprotProtection bits describing access required to the mapping. See mmap(2) for details.
unsignedlongflagsFlags specifying how the mapping should be performed, see mmap (2)for details.
vm_flags_tvm_flagsVMA flags which should be set by default, or 0 otherwise.
unsignedlongpgoffPage offset into thefile if file-backed, should be 0 otherwise.
unsignedlong*populateA pointer to a value which will be set to 0 if no population ofthe range is required, or the number of bytes to populate if it is. Must benon-NULL. See mmap (2) for details as to under what circumstances populationof the range occurs.
structlist_head*ufAn optional pointer to a list head to track userfaultfd unmap eventsshould unmapping events arise. If provided, it is up to the caller to managethis.
Description
This function does not perform security checks on the file and assumes, ifuf is non-NULL, the caller has provided a list head to track unmap eventsfor userfaultfduf.
It also simply indicates whether memory population is required by settingpopulate, which must be non-NULL, expecting the caller to actually performthis task itself if appropriate.
This function will invoke architecture-specific (and if provided andrelevant, file system-specific) logic to determine the most appropriateunmapped area in which to place the mapping if not MAP_FIXED.
Callers which require userland mmap() behaviour should invoke vm_mmap(),which is also exported for module use.
Those which require this behaviour less security checks, userfaultfd andpopulate behaviour, and who handle the mmap write lock themselves, shouldcall this function.
Note that the returned address may reside within a merged VMA if anappropriate merge were to take place, so it doesn’t necessarily specify thestart of a VMA, rather only the start of a valid mapped range of lengthlen bytes, rounded down to the nearest page size.
The caller must write-lock current->mm->mmap_lock.
Return
Either an error, or the address at which the requested mapping hasbeen performed.
- structvm_area_struct*find_vma_intersection(structmm_struct*mm,unsignedlongstart_addr,unsignedlongend_addr)¶
Look up the first VMA which intersects the interval
Parameters
structmm_struct*mmThe process address space.
unsignedlongstart_addrThe inclusive start user address.
unsignedlongend_addrThe exclusive end user address.
Return
The first VMA within the provided range,NULL otherwise. Assumesstart_addr < end_addr.
- structvm_area_struct*find_vma(structmm_struct*mm,unsignedlongaddr)¶
Find the VMA for a given address, or the next VMA.
Parameters
structmm_struct*mmThe mm_struct to check
unsignedlongaddrThe address
Return
The VMA associated with addr, or the next VMA.May returnNULL in the case of no VMA at addr or above.
- structvm_area_struct*find_vma_prev(structmm_struct*mm,unsignedlongaddr,structvm_area_struct**pprev)¶
Find the VMA for a given address, or the next vma and set
pprevto the previous VMA, if any.
Parameters
structmm_struct*mmThe mm_struct to check
unsignedlongaddrThe address
structvm_area_struct**pprevThe pointer to set to the previous VMA
Description
Note that RCU lock is missing here since the external mmap_lock() is usedinstead.
Return
The VMA associated withaddr, or the next vma.May returnNULL in the case of no vma at addr or above.
- void__refkmemleak_alloc(constvoid*ptr,size_tsize,intmin_count,gfp_tgfp)¶
register a newly allocated object
Parameters
constvoid*ptrpointer to beginning of the object
size_tsizesize of the object
intmin_countminimum number of references to this object. If during memoryscanning a number of references less thanmin_count is found,the object is reported as a memory leak. Ifmin_count is 0,the object is never reported as a leak. Ifmin_count is -1,the object is ignored (not scanned and not reported as a leak)
gfp_tgfpkmalloc()flags used for kmemleak internal memory allocations
Description
This function is called from the kernel allocators when a new object(memory block) is allocated (kmem_cache_alloc, kmalloc etc.).
- void__refkmemleak_alloc_percpu(constvoid__percpu*ptr,size_tsize,gfp_tgfp)¶
register a newly allocated __percpu object
Parameters
constvoid__percpu*ptr__percpu pointer to beginning of the object
size_tsizesize of the object
gfp_tgfpflags used for kmemleak internal memory allocations
Description
This function is called from the kernel percpu allocator when a new object(memory block) is allocated (alloc_percpu).
- void__refkmemleak_vmalloc(conststructvm_struct*area,size_tsize,gfp_tgfp)¶
register a newly vmalloc’ed object
Parameters
conststructvm_struct*areapointer to vm_struct
size_tsizesize of the object
gfp_tgfp__vmalloc() flags used for kmemleak internal memory allocations
Description
This function is called from the vmalloc() kernel allocator when a newobject (memory block) is allocated.
- void__refkmemleak_free(constvoid*ptr)¶
unregister a previously registered object
Parameters
constvoid*ptrpointer to beginning of the object
Description
This function is called from the kernel allocators when an object (memoryblock) is freed (kmem_cache_free, kfree, vfree etc.).
- void__refkmemleak_free_part(constvoid*ptr,size_tsize)¶
partially unregister a previously registered object
Parameters
constvoid*ptrpointer to the beginning or inside the object. This alsorepresents the start of the range to be freed
size_tsizesize to be unregistered
Description
This function is called when only a part of a memory block is freed(usually from the bootmem allocator).
- void__refkmemleak_free_percpu(constvoid__percpu*ptr)¶
unregister a previously registered __percpu object
Parameters
constvoid__percpu*ptr__percpu pointer to beginning of the object
Description
This function is called from the kernel percpu allocator when an object(memory block) is freed (free_percpu).
- void__refkmemleak_update_trace(constvoid*ptr)¶
update object allocation stack trace
Parameters
constvoid*ptrpointer to beginning of the object
Description
Override the object allocation stack trace for cases where the actualallocation place is not always useful.
- void__refkmemleak_not_leak(constvoid*ptr)¶
mark an allocated object as false positive
Parameters
constvoid*ptrpointer to beginning of the object
Description
Calling this function on an object will cause the memory block to no longerbe reported as leak and always be scanned.
- void__refkmemleak_transient_leak(constvoid*ptr)¶
mark an allocated object as transient false positive
Parameters
constvoid*ptrpointer to beginning of the object
Description
Calling this function on an object will cause the memory block to not bereported as a leak temporarily. This may happen, for example, if the objectis part of a singly linked list and the ->next reference to it is changed.
- void__refkmemleak_ignore_percpu(constvoid__percpu*ptr)¶
similar to kmemleak_ignore but taking a percpu address argument
Parameters
constvoid__percpu*ptrpercpu address of the object
- void__refkmemleak_ignore(constvoid*ptr)¶
ignore an allocated object
Parameters
constvoid*ptrpointer to beginning of the object
Description
Calling this function on an object will cause the memory block to beignored (not scanned and not reported as a leak). This is usually done whenit is known that the corresponding block is not a leak and does not containany references to other allocated memory blocks.
- void__refkmemleak_scan_area(constvoid*ptr,size_tsize,gfp_tgfp)¶
limit the range to be scanned in an allocated object
Parameters
constvoid*ptrpointer to beginning or inside the object. This alsorepresents the start of the scan area
size_tsizesize of the scan area
gfp_tgfpkmalloc()flags used for kmemleak internal memory allocations
Description
This function is used when it is known that only certain parts of an objectcontain references to other objects. Kmemleak will only scan these areasreducing the number false negatives.
- void__refkmemleak_no_scan(constvoid*ptr)¶
do not scan an allocated object
Parameters
constvoid*ptrpointer to beginning of the object
Description
This function notifies kmemleak not to scan the given memory block. Usefulin situations where it is known that the given object does not contain anyreferences to other objects. Kmemleak will not scan such objects reducingthe number of false negatives.
- void__refkmemleak_alloc_phys(phys_addr_tphys,size_tsize,gfp_tgfp)¶
similar to kmemleak_alloc but taking a physical address argument
Parameters
phys_addr_tphysphysical address of the object
size_tsizesize of the object
gfp_tgfpkmalloc()flags used for kmemleak internal memory allocations
- void__refkmemleak_free_part_phys(phys_addr_tphys,size_tsize)¶
similar to kmemleak_free_part but taking a physical address argument
Parameters
phys_addr_tphysphysical address if the beginning or inside an object. Thisalso represents the start of the range to be freed
size_tsizesize to be unregistered
- void__refkmemleak_ignore_phys(phys_addr_tphys)¶
similar to kmemleak_ignore but taking a physical address argument
Parameters
phys_addr_tphysphysical address of the object
- void*devm_memremap_pages(structdevice*dev,structdev_pagemap*pgmap)¶
remap and provide memmap backing for the given resource
Parameters
structdevice*devhosting device forres
structdev_pagemap*pgmappointer to a struct dev_pagemap
Notes
- 1/ At a minimum the range and type members ofpgmap must be initialized
by the caller before passing it to this function
Description
- 2/ The altmap field may optionally be initialized, in which case
PGMAP_ALTMAP_VALID must be set in pgmap->flags.
- 3/ The ref field may optionally be provided, in which pgmap->ref must be
‘live’ on entry and will be killed and reaped atdevm_memremap_pages_release() time, or if this routine fails.
- 4/ range is expected to be a host memory range that could feasibly be
treated as a “System RAM” range, i.e. not a device mmio range, butthis is not enforced.
- structdev_pagemap*get_dev_pagemap(unsignedlongpfn,structdev_pagemap*pgmap)¶
take a new live reference on the dev_pagemap forpfn
Parameters
unsignedlongpfnpage frame number to lookup page_map
structdev_pagemap*pgmapoptional known pgmap that already has a reference
Description
Ifpgmap is non-NULL and coverspfn it will be returned as-is. Ifpgmapis non-NULL but does not coverpfn the reference to it will be released.
- unsignedlongvma_kernel_pagesize(structvm_area_struct*vma)¶
Page size granularity for this VMA.
Parameters
structvm_area_struct*vmaThe user mapping.
Description
Folios in this VMA will be aligned to, and at least the size of thenumber of bytes returned by this function.
Return
The default size of the folios allocated when backing a VMA.
- boolfolio_isolate_hugetlb(structfolio*folio,structlist_head*list)¶
try to isolate an allocated hugetlb folio
Parameters
structfolio*foliothe folio to isolate
structlist_head*listthe list to add the folio to on success
Description
Isolate an allocated (refcount > 0) hugetlb folio, marking it asisolated/non-migratable, and moving it from the active list to thegiven list.
Isolation will fail iffolio is not an allocated hugetlb folio, or ifit is already isolated/non-migratable.
On success, an additional folio reference is taken that must be droppedusingfolio_putback_hugetlb() to undo the isolation.
Return
True if isolation worked, otherwise False.
Parameters
structfolio*foliothe isolated hugetlb folio
Description
Putback/un-isolate the hugetlb folio that was previous isolated usingfolio_isolate_hugetlb(): marking it non-isolated/migratable and putting itback onto the active list.
Will drop the additional folio reference obtained throughfolio_isolate_hugetlb().
Parameters
structfolio*folioThe folio to mark.
Description
This function will perform one of the following transitions:
inactive,unreferenced -> inactive,referenced
inactive,referenced -> active,unreferenced
active,unreferenced -> active,referenced
When a newly allocated folio is not yet visible, so safe for non-atomic ops,__folio_set_referenced() may be substituted forfolio_mark_accessed().
Parameters
structfolio*folioThe folio to be added to the LRU.
Description
Queue the folio for addition to the LRU. The decision on whetherto add the page to the [in]active [file|anon] list is deferred until thefolio_batch is drained. This gives a chance for the caller offolio_add_lru()have the folio added to the active list usingfolio_mark_accessed().
- voidfolio_add_lru_vma(structfolio*folio,structvm_area_struct*vma)¶
Add a folio to the appropate LRU list for this VMA.
Parameters
structfolio*folioThe folio to be added to the LRU.
structvm_area_struct*vmaVMA in which the folio is mapped.
Description
If the VMA is mlocked,folio is added to the unevictable list.Otherwise, it is treated the same way asfolio_add_lru().
Parameters
structfolio*folioFolio to deactivate.
Description
This function hints to the VM thatfolio is a good reclaim candidate,for example if its invalidation fails due to the folio being dirtyor under writeback.
Context
Caller holds a reference on the folio.
Parameters
structfolio*foliofolio to deactivate
Description
folio_mark_lazyfree() movesfolio to the inactive file list.This is done to accelerate the reclaim offolio.
- voidfolios_put_refs(structfolio_batch*folios,unsignedint*refs)¶
Reduce the reference count on a batch of folios.
Parameters
structfolio_batch*foliosThe folios.
unsignedint*refsThe number of refs to subtract from each folio.
Description
Likefolio_put(), but for a batch of folios. This is more efficientthan writing the loop yourself as it will optimise the locks which needto be taken if the folios are freed. The folios batch is returnedempty and ready to be reused for another batch; there is no needto reinitialise it. Ifrefs is NULL, we subtract one from eachfolio refcount.
Context
May be called in process or interrupt context, but not in NMIcontext. May be called while holding a spinlock.
- voidrelease_pages(release_pages_argarg,intnr)¶
batched put_page()
Parameters
release_pages_argargarray of pages to release
intnrnumber of pages
Description
Decrement the reference count on all the pages inarg. If itfell to zero, remove the page from the LRU and free it.
Note that the argument can be an array of pages, encoded pages,or folio pointers. We ignore any encoded bits, and turn any ofthem into just a folio that gets free’d.
- voidfolio_batch_remove_exceptionals(structfolio_batch*fbatch)¶
Prune non-folios from a batch.
Parameters
structfolio_batch*fbatchThe batch to prune
Description
find_get_entries() fills a batch with both folios and shadow/swap/DAXentries. This function prunes all the non-folio entries fromfbatchwithout leaving holes, so that it can be passed on to folio-only batchoperations.
- voidzpool_register_driver(structzpool_driver*driver)¶
register a zpool implementation.
Parameters
structzpool_driver*driverdriver to register
- intzpool_unregister_driver(structzpool_driver*driver)¶
unregister a zpool implementation.
Parameters
structzpool_driver*driverdriver to unregister.
Description
Module usage counting is used to prevent using a driverwhile/after unloading, so if this is called from moduleexit function, this should never fail; if called fromother than the module exit function, and this returnsfailure, the driver is in use and must remain available.
- boolzpool_has_pool(char*type)¶
Check if the pool driver is available
Parameters
char*typeThe type of the zpool to check (e.g. zsmalloc)
Description
This checks if thetype pool driver is available. This will try to loadthe requested module, if needed, but there is no guarantee the module willstill be loaded and available immediately after calling. If this returnstrue, the caller should assume the pool is available, but must be preparedto handle thezpool_create_pool() returning failure. However if thisreturns false, the caller should assume the requested pool type is notavailable; either the requested pool type module does not exist, or couldnot be loaded, and callingzpool_create_pool() with the pool type willfail.
Thetype string must be null-terminated.
Return
true iftype pool is available, false if not
- structzpool*zpool_create_pool(constchar*type,constchar*name,gfp_tgfp)¶
Create a new zpool
Parameters
constchar*typeThe type of the zpool to create (e.g. zsmalloc)
constchar*nameThe name of the zpool (e.g. zram0, zswap)
gfp_tgfpThe GFP flags to use when allocating the pool.
Description
This creates a new zpool of the specified type. The gfp flags will beused when allocating memory, if the implementation supports it. If theops param is NULL, then the created zpool will not be evictable.
Implementations must guarantee this to be thread-safe.
Thetype andname strings must be null-terminated.
Return
New zpool on success, NULL on failure.
Parameters
structzpool*zpoolThe zpool to destroy.
Description
Implementations must guarantee this to be thread-safe,however only when destroying different pools. The samepool should only be destroyed once, and should not be usedafter it is destroyed.
This destroys an existing zpool. The zpool should not be in use.
Parameters
structzpool*zpoolThe zpool to check
Description
This returns the type of the pool.
Implementations must guarantee this to be thread-safe.
Return
The type of zpool.
- intzpool_malloc(structzpool*zpool,size_tsize,gfp_tgfp,unsignedlong*handle,constintnid)¶
Allocate memory
Parameters
structzpool*zpoolThe zpool to allocate from.
size_tsizeThe amount of memory to allocate.
gfp_tgfpThe GFP flags to use when allocating memory.
unsignedlong*handlePointer to the handle to set
constintnidThe preferred node id.
Description
This allocates the requested amount of memory from the pool.The gfp flags will be used when allocating memory, if theimplementation supports it. The providedhandle will beset to the allocated object handle. The allocation willprefer the NUMA node specified bynid.
Implementations must guarantee this to be thread-safe.
Return
0 on success, negative value on error.
Parameters
structzpool*zpoolThe zpool that allocated the memory.
unsignedlonghandleThe handle to the memory to free.
Description
This frees previously allocated memory. This does not guaranteethat the pool will actually free memory, only that the memoryin the pool will become available for use by the pool.
Implementations must guarantee this to be thread-safe,however only when freeing different handles. The samehandle should only be freed once, and should not be usedafter freeing.
- void*zpool_obj_read_begin(structzpool*zpool,unsignedlonghandle,void*local_copy)¶
Start reading from a previously allocated handle.
Parameters
structzpool*zpoolThe zpool that the handle was allocated from
unsignedlonghandleThe handle to read from
void*local_copyA local buffer to use if needed.
Description
This starts a read operation of a previously allocated handle. The passedlocal_copy buffer may be used if needed by copying the memory into.zpool_obj_read_end() MUST be called after the read is completed to undo anyactions taken (e.g. release locks).
Return
A pointer to the handle memory to be read, iflocal_copy is used,the returned pointer islocal_copy.
- voidzpool_obj_read_end(structzpool*zpool,unsignedlonghandle,void*handle_mem)¶
Finish reading from a previously allocated handle.
Parameters
structzpool*zpoolThe zpool that the handle was allocated from
unsignedlonghandleThe handle to read from
void*handle_memThe pointer returned by
zpool_obj_read_begin()
Description
Finishes a read operation previously started byzpool_obj_read_begin().
- voidzpool_obj_write(structzpool*zpool,unsignedlonghandle,void*handle_mem,size_tmem_len)¶
Write to a previously allocated handle.
Parameters
structzpool*zpoolThe zpool that the handle was allocated from
unsignedlonghandleThe handle to read from
void*handle_memThe memory to copy from into the handle.
size_tmem_lenThe length of memory to be written.
Parameters
structzpool*zpoolThe zpool to check
Description
This returns the total size in pages of the pool.
Return
Total size of the zpool in pages.
- structcgroup_subsys_state*mem_cgroup_css_from_folio(structfolio*folio)¶
css of the memcg associated with a folio
Parameters
structfolio*foliofolio of interest
Description
If memcg is bound to the default hierarchy, css of the memcg associatedwithfolio is returned. The returned css remains associated withfoliountil it is released.
If memcg is bound to a traditional hierarchy, the css of root_mem_cgroupis returned.
Parameters
structpage*pagethe page
Description
Look up the closest online ancestor of the memory cgrouppage is charged toand return its inode number or 0 ifpage is not charged to any cgroup. Itis safe to call this function without holding a reference topage.
Note, this function is inherently racy, because there is nothing to preventthe cgroup inode from getting torn down and potentially reallocated a momentafterpage_cgroup_ino() returns, so it only should be used by callers thatdo not care (such as procfs interfaces).
- voidmod_memcg_state(structmem_cgroup*memcg,enummemcg_stat_itemidx,intval)¶
update cgroup memory statistics
Parameters
structmem_cgroup*memcgthe memory cgroup
enummemcg_stat_itemidxthe stat item - can be enum memcg_stat_item or enum node_stat_item
intvaldelta to add to the counter, can be negative
- void__mod_lruvec_state(structlruvec*lruvec,enumnode_stat_itemidx,intval)¶
update lruvec memory statistics
Parameters
structlruvec*lruvecthe lruvec
enumnode_stat_itemidxthe stat item
intvaldelta to add to the counter, can be negative
Description
The lruvec is the intersection of the NUMA node and a cgroup. Thisfunction updates the all three counters that are affected by achange of state at this level: per-node, per-cgroup, per-lruvec.
- voidcount_memcg_events(structmem_cgroup*memcg,enumvm_event_itemidx,unsignedlongcount)¶
account VM events in a cgroup
Parameters
structmem_cgroup*memcgthe memory cgroup
enumvm_event_itemidxthe event item
unsignedlongcountthe number of events that occurred
- structmem_cgroup*get_mem_cgroup_from_mm(structmm_struct*mm)¶
Obtain a reference on given mm_struct’s memcg.
Parameters
structmm_struct*mmmm from which memcg should be extracted. It can be NULL.
Description
Obtain a reference on mm->memcg and returns it if successful. If mmis NULL, then the memcg is chosen as follows:1) The active memcg, if set.2) current->mm->memcg, if available3) root memcgIf mem_cgroup is disabled, NULL is returned.
- structmem_cgroup*get_mem_cgroup_from_current(void)¶
Obtain a reference on current task’s memcg.
Parameters
voidno arguments
- structmem_cgroup*get_mem_cgroup_from_folio(structfolio*folio)¶
Obtain a reference on a given folio’s memcg.
Parameters
structfolio*foliofolio from which memcg should be extracted.
- structmem_cgroup*mem_cgroup_iter(structmem_cgroup*root,structmem_cgroup*prev,structmem_cgroup_reclaim_cookie*reclaim)¶
iterate over memory cgroup hierarchy
Parameters
structmem_cgroup*roothierarchy root
structmem_cgroup*prevpreviously returned memcg, NULL on first invocation
structmem_cgroup_reclaim_cookie*reclaimcookie for shared reclaim walks, NULL for full walks
Description
Returns references to children of the hierarchy belowroot, orroot itself, orNULL after a full round-trip.
Caller must pass the return value inprev on subsequentinvocations for reference counting, or usemem_cgroup_iter_break()to cancel a hierarchy walk before the round-trip is complete.
Reclaimers can specify a node inreclaim to divide up the memcgsin the hierarchy among all concurrent reclaimers operating on thesame node.
- voidmem_cgroup_iter_break(structmem_cgroup*root,structmem_cgroup*prev)¶
abort a hierarchy walk prematurely
Parameters
structmem_cgroup*roothierarchy root
structmem_cgroup*prevlast visited hierarchy member as returned by
mem_cgroup_iter()
- voidmem_cgroup_scan_tasks(structmem_cgroup*memcg,int(*fn)(structtask_struct*,void*),void*arg)¶
iterate over tasks of a memory cgroup hierarchy
Parameters
structmem_cgroup*memcghierarchy root
int(*fn)(structtask_struct*,void*)function to call for each task
void*argargument passed tofn
Description
This function iterates over tasks attached tomemcg or to any of itsdescendants and callsfn for each task. Iffn returns a non-zerovalue, the function breaks the iteration loop. Otherwise, it will iterateover all tasks and return 0.
This function must not be called for the root memory cgroup.
Parameters
structfolio*folioPointer to the folio.
Description
These functions are safe to use under any of the following conditions:- folio locked- folio_test_lru false- folio frozen (refcount of 0)
Return
The lruvec this folio is on with its lock held.
Parameters
structfolio*folioPointer to the folio.
Description
These functions are safe to use under any of the following conditions:- folio locked- folio_test_lru false- folio frozen (refcount of 0)
Return
The lruvec this folio is on with its lock held and interruptsdisabled.
- structlruvec*folio_lruvec_lock_irqsave(structfolio*folio,unsignedlong*flags)¶
Lock the lruvec for a folio.
Parameters
structfolio*folioPointer to the folio.
unsignedlong*flagsPointer to irqsave flags.
Description
These functions are safe to use under any of the following conditions:- folio locked- folio_test_lru false- folio frozen (refcount of 0)
Return
The lruvec this folio is on with its lock held and interruptsdisabled.
- voidmem_cgroup_update_lru_size(structlruvec*lruvec,enumlru_listlru,intzid,intnr_pages)¶
account for adding or removing an lru page
Parameters
structlruvec*lruvecmem_cgroup per zone lru vector
enumlru_listlruindex of lru list the page is sitting on
intzidzone id of the accounted pages
intnr_pagespositive when adding or negative when removing
Description
This function must be called under lru_lock, just before a page is addedto or just after a page is removed from an lru list.
- unsignedlongmem_cgroup_margin(structmem_cgroup*memcg)¶
calculate chargeable space of a memory cgroup
Parameters
structmem_cgroup*memcgthe memory cgroup
Description
Returns the maximum amount of memorymem can be charged with, inpages.
- voidmem_cgroup_print_oom_context(structmem_cgroup*memcg,structtask_struct*p)¶
Print OOM information relevant to memory controller.
Parameters
structmem_cgroup*memcgThe memory cgroup that went over limit
structtask_struct*pTask that is going to be killed
NOTE
memcg andp’s mem_cgroup can be different when hierarchy isenabled
- voidmem_cgroup_print_oom_meminfo(structmem_cgroup*memcg)¶
Print OOM memory information relevant to memory controller.
Parameters
structmem_cgroup*memcgThe memory cgroup that went over limit
- structmem_cgroup*mem_cgroup_get_oom_group(structtask_struct*victim,structmem_cgroup*oom_domain)¶
get a memory cgroup to clean up after OOM
Parameters
structtask_struct*victimtask to be killed by the OOM killer
structmem_cgroup*oom_domainmemcg in case of memcg OOM, NULL in case of system-wide OOM
Description
Returns a pointer to a memory cgroup, which has to be cleaned upby killing all belonging OOM-killable tasks.
Caller has to call mem_cgroup_put() on the returned non-NULL memcg.
- boolconsume_stock(structmem_cgroup*memcg,unsignedintnr_pages)¶
Try to consume stocked charge on this cpu.
Parameters
structmem_cgroup*memcgmemcg to consume from.
unsignedintnr_pageshow many pages to charge.
Description
Consume the cached charge if enough nr_pages are present otherwise returnfailure. Also return failure for charge request larger thanMEMCG_CHARGE_BATCH or if the local lock is already taken.
returns true if successful, false otherwise.
- int__memcg_kmem_charge_page(structpage*page,gfp_tgfp,intorder)¶
charge a kmem page to the current memory cgroup
Parameters
structpage*pagepage to charge
gfp_tgfpreclaim mode
intorderallocation order
Description
Returns 0 on success, an error code on failure.
Parameters
structpage*pagepage to uncharge
intorderallocation order
- voidmem_cgroup_wb_stats(structbdi_writeback*wb,unsignedlong*pfilepages,unsignedlong*pheadroom,unsignedlong*pdirty,unsignedlong*pwriteback)¶
retrieve writeback related stats from its memcg
Parameters
structbdi_writeback*wbbdi_writeback in question
unsignedlong*pfilepagesout parameter for number of file pages
unsignedlong*pheadroomout parameter for number of allocatable pages according to memcg
unsignedlong*pdirtyout parameter for number of dirty pages
unsignedlong*pwritebackout parameter for number of pages under writeback
Description
Determine the numbers of file, headroom, dirty, and writeback pages inwb’s memcg. File, dirty and writeback are self-explanatory. Headroomis a bit more involved.
A memcg’s headroom is “min(max, high) - used”. In the hierarchy, theheadroom is calculated as the lowest headroom of itself and theancestors. Note that this doesn’t consider the actual amount ofavailable memory in the system. The caller should further cap*pheadroom accordingly.
- structmem_cgroup*mem_cgroup_from_id(unsignedshortid)¶
look up a memcg from a memcg id
- voidmem_cgroup_css_reset(structcgroup_subsys_state*css)¶
reset the states of a mem_cgroup
Parameters
structcgroup_subsys_state*cssthe target css
Description
Reset the states of the mem_cgroup associated withcss. This isinvoked when the userland requests disabling on the default hierarchybut the memcg is pinned through dependency. The memcg should stopapplying policies and should revert to the vanilla state as it may bemade visible again.
The current implementation only resets the essential configurations.This needs to be expanded to cover all the visible parts.
- voidmem_cgroup_calculate_protection(structmem_cgroup*root,structmem_cgroup*memcg)¶
check if memory consumption is in the normal range
Parameters
structmem_cgroup*rootthe top ancestor of the sub-tree being checked
structmem_cgroup*memcgthe memory cgroup to check
Description
- WARNING: This function is not stateless! It can only be used as part
of a top-down tree iteration, not for isolated queries.
Parameters
structfolio*foliofolio being charged
gfp_tgfpreclaim mode
Description
This function is called when allocating a huge page folio, after the page hasalready been obtained and charged to the appropriate hugetlb cgroupcontroller (if it is enabled).
Returns ENOMEM if the memcg is already full.Returns 0 if either the charge was successful, or if we skip the charging.
- intmem_cgroup_swapin_charge_folio(structfolio*folio,structmm_struct*mm,gfp_tgfp,swp_entry_tentry)¶
Charge a newly allocated folio for swapin.
Parameters
structfolio*foliofolio to charge.
structmm_struct*mmmm context of the victim
gfp_tgfpreclaim mode
swp_entry_tentryswap entry for which the folio is allocated
Description
This function charges a folio allocated for swapin. Please call this beforeadding the folio to the swapcache.
Returns 0 on success. Otherwise, an error code is returned.
Parameters
structfolio*oldCurrently circulating folio.
structfolio*newReplacement folio.
Description
Chargenew as a replacement folio forold.old willbe uncharged upon free.
Both folios must be locked,new->mapping must be set up.
- voidmem_cgroup_migrate(structfolio*old,structfolio*new)¶
Transfer the memcg data from the old to the new folio.
Parameters
structfolio*oldCurrently circulating folio.
structfolio*newReplacement folio.
Description
Transfer the memcg data from the old folio to the new folio for migration.The old folio’s data info will be cleared. Note that the memory counterswill remain unchanged throughout the process.
Both folios must be locked,new->mapping must be set up.
- boolmem_cgroup_charge_skmem(structmem_cgroup*memcg,unsignedintnr_pages,gfp_tgfp_mask)¶
charge socket memory
Parameters
structmem_cgroup*memcgmemcg to charge
unsignedintnr_pagesnumber of pages to charge
gfp_tgfp_maskreclaim mode
Description
Chargesnr_pages tomemcg. Returnstrue if the charge fit withinmemcg’s configured limit,false if it doesn’t.
- voidmem_cgroup_uncharge_skmem(structmem_cgroup*memcg,unsignedintnr_pages)¶
uncharge socket memory
Parameters
structmem_cgroup*memcgmemcg to uncharge
unsignedintnr_pagesnumber of pages to uncharge
- int__mem_cgroup_try_charge_swap(structfolio*folio,swp_entry_tentry)¶
try charging swap space for a folio
Parameters
structfolio*foliofolio being added to swap
swp_entry_tentryswap entry to charge
Description
Try to chargefolio’s memcg for the swap space atentry.
Returns 0 on success, -ENOMEM on failure.
- void__mem_cgroup_uncharge_swap(swp_entry_tentry,unsignedintnr_pages)¶
uncharge swap space
Parameters
swp_entry_tentryswap entry to uncharge
unsignedintnr_pagesthe amount of swap space to uncharge
- boolobj_cgroup_may_zswap(structobj_cgroup*objcg)¶
check if this cgroup can zswap
Parameters
structobj_cgroup*objcgthe object cgroup
Description
Check if the hierarchical zswap limit has been reached.
This doesn’t check for specific headroom, and it is not atomiceither. But with zswap, the size of the allocation is only knownonce compression has occurred, and this optimistic pre-check avoidsspending cycles on compression when there is already no room leftor zswap is disabled altogether somewhere in the hierarchy.
- voidobj_cgroup_charge_zswap(structobj_cgroup*objcg,size_tsize)¶
charge compression backend memory
Parameters
structobj_cgroup*objcgthe object cgroup
size_tsizesize of compressed object
Description
This forces the charge afterobj_cgroup_may_zswap() allowedcompression and storage in zwap for this cgroup to go ahead.
- voidobj_cgroup_uncharge_zswap(structobj_cgroup*objcg,size_tsize)¶
uncharge compression backend memory
Parameters
structobj_cgroup*objcgthe object cgroup
size_tsizesize of compressed object
Description
Uncharges zswap memory on page in.
- voidshmem_recalc_inode(structinode*inode,longalloced,longswapped)¶
recalculate the block usage of an inode
Parameters
structinode*inodeinode to recalc
longallocedthe change in number of pages allocated to inode
longswappedthe change in number of pages swapped from inode
Description
We have to calculate the free blocks since the mm can dropundirtied hole pages behind our back.
But normally info->alloced == inode->i_mapping->nrpages + info->swappedSo mm freed is info->alloced - (inode->i_mapping->nrpages + info->swapped)
- unsignedintshmem_mapping_size_orders(structaddress_space*mapping,pgoff_tindex,loff_twrite_end)¶
Get allowable folio orders for the given file size.
Parameters
structaddress_space*mappingTarget address_space.
pgoff_tindexThe page index.
loff_twrite_endend of a write, could extend inode size.
Description
This returns huge orders for folios (when supported) based on the file sizewhich the mapping currently allows at the given index. The index is relevantdue to alignment considerations the mapping might have. The returned ordermay be less than the size passed.
Return
The orders.
Parameters
structfolio*folioThe folio to write
structwriteback_control*wbcHow writeback is to be done
Description
Move the folio from the page cache to the swap cache.
- intshmem_get_folio(structinode*inode,pgoff_tindex,loff_twrite_end,structfolio**foliop,enumsgp_typesgp)¶
find, and lock a shmem folio.
Parameters
structinode*inodeinode to search
pgoff_tindexthe page index.
loff_twrite_endend of a write, could extend inode size
structfolio**folioppointer to the folio if found
enumsgp_typesgpSGP_* flags to control behavior
Description
Looks up the page cache entry atinode &index. If a folio ispresent, it is returned locked with an increased refcount.
If the caller modifies data in the folio, it must callfolio_mark_dirty()before unlocking the folio to ensure that the folio is not reclaimed.There is no need to reserve space before callingfolio_mark_dirty().
- When no folio is found, the behavior depends onsgp:
for SGP_READ,*foliop is
NULLand 0 is returnedfor SGP_NOALLOC,*foliop is
NULLand -ENOENT is returnedfor all other flags a new folio is allocated, inserted into thepage cache and returned locked infoliop.
Context
May sleep.
Return
0 if successful, else a negative error code.
- structfile*shmem_kernel_file_setup(constchar*name,loff_tsize,unsignedlongflags)¶
get an unlinked file living in tmpfs which must be kernel internal. There will be NO LSM permission checks against the underlying inode. So users of this interface must do LSM checks at a higher layer. The users are the big_key and shm implementations. LSM checks are provided at the key or shm level rather than the inode.
Parameters
constchar*namename for dentry (to be seen in /proc/<pid>/maps)
loff_tsizesize to be set for the file
unsignedlongflagsVM_NORESERVE suppresses pre-accounting of the entire object size
- structfile*shmem_file_setup(constchar*name,loff_tsize,unsignedlongflags)¶
get an unlinked file living in tmpfs
Parameters
constchar*namename for dentry (to be seen in /proc/<pid>/maps)
loff_tsizesize to be set for the file
unsignedlongflagsVM_NORESERVE suppresses pre-accounting of the entire object size
- structfile*shmem_file_setup_with_mnt(structvfsmount*mnt,constchar*name,loff_tsize,unsignedlongflags)¶
get an unlinked file living in tmpfs
Parameters
structvfsmount*mntthe tmpfs mount where the file will be created
constchar*namename for dentry (to be seen in /proc/<pid>/maps)
loff_tsizesize to be set for the file
unsignedlongflagsVM_NORESERVE suppresses pre-accounting of the entire object size
- intshmem_zero_setup(structvm_area_struct*vma)¶
setup a shared anonymous mapping
Parameters
structvm_area_struct*vmathe vma to be mmapped is prepared by do_mmap
- structfolio*shmem_read_folio_gfp(structaddress_space*mapping,pgoff_tindex,gfp_tgfp)¶
read into page cache, using specified page allocation flags.
Parameters
structaddress_space*mappingthe folio’s address_space
pgoff_tindexthe folio index
gfp_tgfpthe page allocator flags to use if allocating
Description
This behaves as a tmpfs “read_cache_page_gfp(mapping, index, gfp)”,with any new page allocations done using the specified allocation flags.Butread_cache_page_gfp() uses the ->read_folio() method: which does notsuit tmpfs, since it may have pages in swapcache, and needs to find thosefor itself; although drivers/gpu/drm i915 and ttm rely upon this support.
i915_gem_object_get_pages_gtt() mixes __GFP_NORETRY | __GFP_NOWARN inwith the mapping_gfp_mask(), to avoid OOMing the machine unnecessarily.
- intmigrate_vma_setup(structmigrate_vma*args)¶
prepare to migrate a range of memory
Parameters
structmigrate_vma*argscontains the vma, start, and pfns arrays for the migration
Return
negative errno on failures, 0 when 0 or more pages were migratedwithout an error.
Description
Prepare to migrate a range of memory virtual address range by collecting allthe pages backing each virtual address in the range, saving them inside thesrc array. Then lock those pages and unmap them. Once the pages are lockedand unmapped, check whether each page is pinned or not. Pages that aren’tpinned have the MIGRATE_PFN_MIGRATE flag set (by this function) in thecorresponding src array entry. Then restores any pages that are pinned, byremapping and unlocking those pages.
The caller should then allocate destination memory and copy source memory toit for all those entries (ie with MIGRATE_PFN_VALID and MIGRATE_PFN_MIGRATEflag set). Once these are allocated and copied, the caller must update eachcorresponding entry in the dst array with the pfn value of the destinationpage and with MIGRATE_PFN_VALID. Destination pages must be locked vialock_page().
Note that the caller does not have to migrate all the pages that are markedwith MIGRATE_PFN_MIGRATE flag in src array unless this is a migration fromdevice memory to system memory. If the caller cannot migrate a device pageback to system memory, then it must return VM_FAULT_SIGBUS, which has severeconsequences for the userspace process, so it must be avoided if at allpossible.
For empty entries inside CPU page table (pte_none() or pmd_none() is true) wedo set MIGRATE_PFN_MIGRATE flag inside the corresponding source array thusallowing the caller to allocate device memory for those unbacked virtualaddresses. For this the caller simply has to allocate device memory andproperly set the destination entry like for regular migration. Note thatthis can still fail, and thus inside the device driver you must check if themigration was successful for those entries after callingmigrate_vma_pages(),just like for regular migration.
After that, the callers must callmigrate_vma_pages() to go over each entryin the src array that has the MIGRATE_PFN_VALID and MIGRATE_PFN_MIGRATE flagset. If the corresponding entry in dst array has MIGRATE_PFN_VALID flag set,thenmigrate_vma_pages() to migrate struct page information from the sourcestruct page to the destination struct page. If it fails to migrate thestruct page information, then it clears the MIGRATE_PFN_MIGRATE flag in thesrc array.
At this point all successfully migrated pages have an entry in the srcarray with MIGRATE_PFN_VALID and MIGRATE_PFN_MIGRATE flag set and the dstarray entry with MIGRATE_PFN_VALID flag set.
Oncemigrate_vma_pages() returns the caller may inspect which pages weresuccessfully migrated, and which were not. Successfully migrated pages willhave the MIGRATE_PFN_MIGRATE flag set for their src array entry.
It is safe to update device page table aftermigrate_vma_pages() becauseboth destination and source page are still locked, and the mmap_lock is heldin read mode (hence no one can unmap the range being migrated).
Once the caller is done cleaning up things and updating its page table (if itchose to do so, this is not an obligation) it finally callsmigrate_vma_finalize() to update the CPU page table to point to new pagesfor successfully migrated pages or otherwise restore the CPU page table topoint to the original source pages.
- voidmigrate_device_pages(unsignedlong*src_pfns,unsignedlong*dst_pfns,unsignedlongnpages)¶
migrate meta-data from src page to dst page
Parameters
unsignedlong*src_pfnssrc_pfns returned from
migrate_device_range()unsignedlong*dst_pfnsarray of pfns allocated by the driver to migrate memory to
unsignedlongnpagesnumber of pages in the range
Description
Equivalent tomigrate_vma_pages(). This is called to migrate struct pagemeta-data from source struct page to destination.
- voidmigrate_vma_pages(structmigrate_vma*migrate)¶
migrate meta-data from src page to dst page
Parameters
structmigrate_vma*migratemigrate struct containing all migration information
Description
This migrates struct page meta-data from source struct page to destinationstruct page. This effectively finishes the migration from source page to thedestination page.
- voidmigrate_vma_finalize(structmigrate_vma*migrate)¶
restore CPU page table entry
Parameters
structmigrate_vma*migratemigrate struct containing all migration information
Description
This replaces the special migration pte entry with either a mapping to thenew page if migration was successful for that page, or to the original pageotherwise.
This also unlocks the pages and puts them back on the lru, or drops the extrarefcount, for device pages.
- intmigrate_device_range(unsignedlong*src_pfns,unsignedlongstart,unsignedlongnpages)¶
migrate device private pfns to normal memory.
Parameters
unsignedlong*src_pfnsarray large enough to hold migrating source device private pfns.
unsignedlongstartstarting pfn in the range to migrate.
unsignedlongnpagesnumber of pages to migrate.
Description
migrate_vma_setup() is similar in concept tomigrate_vma_setup() except thatinstead of looking up pages based on virtual address mappings a range ofdevice pfns that should be migrated to system memory is used instead.
This is useful when a driver needs to free device memory but doesn’t know thevirtual mappings of every page that may be in device memory. For example thisis often the case when a driver is being unloaded or unbound from a device.
Likemigrate_vma_setup() this function will take a reference and lock anymigrating pages that aren’t free before unmapping them. Drivers may thenallocate destination pages and start copying data from the device to CPUmemory before callingmigrate_device_pages().
- intmigrate_device_pfns(unsignedlong*src_pfns,unsignedlongnpages)¶
migrate device private pfns to normal memory.
Parameters
unsignedlong*src_pfnspre-popluated array of source device private pfns to migrate.
unsignedlongnpagesnumber of pages to migrate.
Description
Similar tomigrate_device_range() but supports non-contiguous pre-popluatedarray of device pages to migrate.
- structwp_walk¶
Private struct for pagetable walk callbacks
Definition:
struct wp_walk { struct mmu_notifier_range range; unsigned long tlbflush_start; unsigned long tlbflush_end; unsigned long total;};Members
rangeRange for mmu notifiers
tlbflush_startAddress of first modified pte
tlbflush_endAddress of last modified pte + 1
totalTotal number of modified ptes
- intwp_pte(pte_t*pte,unsignedlongaddr,unsignedlongend,structmm_walk*walk)¶
Write-protect a pte
Parameters
pte_t*ptePointer to the pte
unsignedlongaddrThe start of protecting virtual address
unsignedlongendThe end of protecting virtual address
structmm_walk*walkpagetable walk callback argument
Description
The function write-protects a pte and records the range invirtual address space of touched ptes for efficient range TLB flushes.
- structclean_walk¶
Private struct for the clean_record_pte function.
Definition:
struct clean_walk { struct wp_walk base; pgoff_t bitmap_pgoff; unsigned long *bitmap; pgoff_t start; pgoff_t end;};Members
basestructwp_walkwe derive frombitmap_pgoffAddress_space Page offset of the first bit inbitmap
bitmapBitmap with one bit for each page offset in the address_space rangecovered.
startAddress_space page offset of first modified pte relativetobitmap_pgoff
endAddress_space page offset of last modified pte relativetobitmap_pgoff
- intclean_record_pte(pte_t*pte,unsignedlongaddr,unsignedlongend,structmm_walk*walk)¶
Clean a pte and record its address space offset in a bitmap
Parameters
pte_t*ptePointer to the pte
unsignedlongaddrThe start of virtual address to be clean
unsignedlongendThe end of virtual address to be clean
structmm_walk*walkpagetable walk callback argument
Description
The function cleans a pte and records the range invirtual address space of touched ptes for efficient TLB flushes.It also records dirty ptes in a bitmap representing page offsetsin the address_space, as well as the first and last of the bitstouched.
- unsignedlongwp_shared_mapping_range(structaddress_space*mapping,pgoff_tfirst_index,pgoff_tnr)¶
Write-protect all ptes in an address space range
Parameters
structaddress_space*mappingThe address_space we want to write protect
pgoff_tfirst_indexThe first page offset in the range
pgoff_tnrNumber of incremental page offsets to cover
Note
This function currently skips transhuge page-table entries, sinceit’s intended for dirty-tracking on the PTE level. It will warn onencountering transhuge write-enabled entries, though, and can easily beextended to handle them as well.
Return
The number of ptes actually write-protected. Note thatalready write-protected ptes are not counted.
- unsignedlongclean_record_shared_mapping_range(structaddress_space*mapping,pgoff_tfirst_index,pgoff_tnr,pgoff_tbitmap_pgoff,unsignedlong*bitmap,pgoff_t*start,pgoff_t*end)¶
Clean and record all ptes in an address space range
Parameters
structaddress_space*mappingThe address_space we want to clean
pgoff_tfirst_indexThe first page offset in the range
pgoff_tnrNumber of incremental page offsets to cover
pgoff_tbitmap_pgoffThe page offset of the first bit inbitmap
unsignedlong*bitmapPointer to a bitmap of at leastnr bits. The bitmap needs tocover the whole rangefirst_index..**first_index** +nr.
pgoff_t*startPointer to number of the first set bit inbitmap.is modified as new bits are set by the function.
pgoff_t*endPointer to the number of the last set bit inbitmap.none set. The value is modified as new bits are set by the function.
Description
When this function returns there is no guarantee that a CPU hasnot already dirtied new ptes. However it will not clean any ptes notreported in the bitmap. The guarantees are as follows:
All ptes dirty when the function starts executing will end up recordedin the bitmap.
All ptes dirtied after that will either remain dirty, be recorded in thebitmap or both.
If a caller needs to make sure all dirty ptes are picked up and noneadditional are added, it first needs to write-protect the address-spacerange and make sure new writers are blocked in page_mkwrite() orpfn_mkwrite(). And then after a TLB flush following the write-protectionpick up all dirty bits.
This function currently skips transhuge page-table entries, sinceit’s intended for dirty-tracking on the PTE level. It will warn onencountering transhuge dirty entries, though, and can easily be extendedto handle them as well.
Return
The number of dirty ptes actually cleaned.
- boolpcpu_addr_in_chunk(structpcpu_chunk*chunk,void*addr)¶
check if the address is served from this chunk
Parameters
structpcpu_chunk*chunkchunk of interest
void*addrpercpu address
Return
True if the address is served from this chunk.
- boolpcpu_check_block_hint(structpcpu_block_md*block,intbits,size_talign)¶
check against the contig hint
Parameters
structpcpu_block_md*blockblock of interest
intbitssize of allocation
size_talignalignment of area (max PAGE_SIZE)
Description
Check to see if the allocation can fit in the block’s contig hint.Note, a chunk uses the same hints as a block so this can also check againstthe chunk’s contig hint.
- voidpcpu_next_md_free_region(structpcpu_chunk*chunk,int*bit_off,int*bits)¶
finds the next hint free area
Parameters
structpcpu_chunk*chunkchunk of interest
int*bit_offchunk offset
int*bitssize of free area
Description
Helper function for pcpu_for_each_md_free_region. It checksblock->contig_hint and performs aggregation across blocks to find thenext hint. It modifies bit_off and bits in-place to be consumed in theloop.
- voidpcpu_next_fit_region(structpcpu_chunk*chunk,intalloc_bits,intalign,int*bit_off,int*bits)¶
finds fit areas for a given allocation request
Parameters
structpcpu_chunk*chunkchunk of interest
intalloc_bitssize of allocation
intalignalignment of area (max PAGE_SIZE)
int*bit_offchunk offset
int*bitssize of free area
Description
Finds the next free region that is viable for use with a given size andalignment. This only returns if there is a valid area to be used for thisallocation. block->first_free is returned if the allocation request fitswithin the block to see if the request can be fulfilled prior to the contighint.
- void*pcpu_mem_zalloc(size_tsize,gfp_tgfp)¶
allocate memory
Parameters
size_tsizebytes to allocate
gfp_tgfpallocation flags
Description
Allocatesize bytes. Ifsize is smaller than PAGE_SIZE,kzalloc() is used; otherwise, the equivalent of vzalloc() is used.This is to facilitate passing through whitelisted flags. Thereturned memory is always zeroed.
Return
Pointer to the allocated area on success, NULL on failure.
- voidpcpu_mem_free(void*ptr)¶
free memory
Parameters
void*ptrmemory to free
Description
Freeptr.ptr should have been allocated usingpcpu_mem_zalloc().
- voidpcpu_chunk_relocate(structpcpu_chunk*chunk,intoslot)¶
put chunk in the appropriate chunk slot
Parameters
structpcpu_chunk*chunkchunk of interest
intoslotthe previous slot it was on
Description
This function is called after an allocation or free changedchunk.New slot according to the changed state is determined andchunk ismoved to the slot. Note that the reserved chunk is never put onchunk slots.
Context
pcpu_lock.
- voidpcpu_block_update(structpcpu_block_md*block,intstart,intend)¶
updates a block given a free area
Parameters
structpcpu_block_md*blockblock of interest
intstartstart offset in block
intendend offset in block
Description
Updates a block given a known free area. The region [start, end) isexpected to be the entirety of the free area within a block. Choosesthe best starting offset if the contig hints are equal.
- voidpcpu_chunk_refresh_hint(structpcpu_chunk*chunk,boolfull_scan)¶
updates metadata about a chunk
Parameters
structpcpu_chunk*chunkchunk of interest
boolfull_scanif we should scan from the beginning
Description
Iterates over the metadata blocks to find the largest contig area.A full scan can be avoided on the allocation path as this is triggeredif we broke the contig_hint. In doing so, the scan_hint will be beforethe contig_hint or after if the scan_hint == contig_hint. This cannotbe prevented on freeing as we want to find the largest area possiblyspanning blocks.
- voidpcpu_block_refresh_hint(structpcpu_chunk*chunk,intindex)¶
Parameters
structpcpu_chunk*chunkchunk of interest
intindexindex of the metadata block
Description
Scans over the block beginning at first_free and updates the blockmetadata accordingly.
- voidpcpu_block_update_hint_alloc(structpcpu_chunk*chunk,intbit_off,intbits)¶
update hint on allocation path
Parameters
structpcpu_chunk*chunkchunk of interest
intbit_offchunk offset
intbitssize of request
Description
Updates metadata for the allocation path. The metadata only has to berefreshed by a full scan iff the chunk’s contig hint is broken. Block levelscans are required if the block’s contig hint is broken.
- voidpcpu_block_update_hint_free(structpcpu_chunk*chunk,intbit_off,intbits)¶
updates the block hints on the free path
Parameters
structpcpu_chunk*chunkchunk of interest
intbit_offchunk offset
intbitssize of request
Description
Updates metadata for the allocation path. This avoids a blind blockrefresh by making use of the block contig hints. If this fails, it scansforward and backward to determine the extent of the free area. This iscapped at the boundary of blocks.
A chunk update is triggered if a page becomes free, a block becomes free,or the free spans across blocks. This tradeoff is to minimize iteratingover the block metadata to update chunk_md->contig_hint.chunk_md->contig_hint may be off by up to a page, but it will never be morethan the available space. If the contig hint is contained in one block, itwill be accurate.
- boolpcpu_is_populated(structpcpu_chunk*chunk,intbit_off,intbits,int*next_off)¶
determines if the region is populated
Parameters
structpcpu_chunk*chunkchunk of interest
intbit_offchunk offset
intbitssize of area
int*next_offreturn value for the next offset to start searching
Description
For atomic allocations, check if the backing pages are populated.
Return
Bool if the backing pages are populated.next_index is to skip over unpopulated blocks in pcpu_find_block_fit.
- intpcpu_find_block_fit(structpcpu_chunk*chunk,intalloc_bits,size_talign,boolpop_only)¶
finds the block index to start searching
Parameters
structpcpu_chunk*chunkchunk of interest
intalloc_bitssize of request in allocation units
size_talignalignment of area (max PAGE_SIZE bytes)
boolpop_onlyuse populated regions only
Description
Given a chunk and an allocation spec, find the offset to begin searchingfor a free region. This iterates over the bitmap metadata blocks tofind an offset that will be guaranteed to fit the requirements. It isnot quite first fit as if the allocation does not fit in the contig hintof a block or chunk, it is skipped. This errs on the side of cautionto prevent excess iteration. Poor alignment can cause the allocator toskip over blocks and chunks that have valid free areas.
Return
The offset in the bitmap to begin searching.-1 if no offset is found.
- intpcpu_alloc_area(structpcpu_chunk*chunk,intalloc_bits,size_talign,intstart)¶
allocates an area from a pcpu_chunk
Parameters
structpcpu_chunk*chunkchunk of interest
intalloc_bitssize of request in allocation units
size_talignalignment of area (max PAGE_SIZE)
intstartbit_off to start searching
Description
This function takes in astart offset to begin searching to fit anallocation ofalloc_bits with alignmentalign. It needs to scanthe allocation map because if it fits within the block’s contig hint,start will be block->first_free. This is an attempt to fill theallocation prior to breaking the contig hint. The allocation andboundary maps are updated accordingly if it confirms a validfree area.
Return
Allocated addr offset inchunk on success.-1 if no matching area is found.
- intpcpu_free_area(structpcpu_chunk*chunk,intoff)¶
frees the corresponding offset
Parameters
structpcpu_chunk*chunkchunk of interest
intoffaddr offset into chunk
Description
This function determines the size of an allocation to free usingthe boundary bitmap and clears the allocation map.
Return
Number of freed bytes.
- structpcpu_chunk*pcpu_alloc_first_chunk(unsignedlongtmp_addr,intmap_size)¶
creates chunks that serve the first chunk
Parameters
unsignedlongtmp_addrthe start of the region served
intmap_sizesize of the region served
Description
This is responsible for creating the chunks that serve the first chunk. Thebase_addr is page aligned down oftmp_addr while the region end is pagealigned up. Offsets are kept track of to determine the region served. Allthis is done to appease the bitmap allocator in avoiding partial blocks.
Return
Chunk serving the region attmp_addr ofmap_size.
- voidpcpu_chunk_populated(structpcpu_chunk*chunk,intpage_start,intpage_end)¶
post-population bookkeeping
Parameters
structpcpu_chunk*chunkpcpu_chunk which got populated
intpage_startthe start page
intpage_endthe end page
Description
Pages in [page_start,**page_end**) have been populated tochunk. Updatethe bookkeeping information accordingly. Must be called after eachsuccessful population.
- voidpcpu_chunk_depopulated(structpcpu_chunk*chunk,intpage_start,intpage_end)¶
post-depopulation bookkeeping
Parameters
structpcpu_chunk*chunkpcpu_chunk which got depopulated
intpage_startthe start page
intpage_endthe end page
Description
Pages in [page_start,**page_end**) have been depopulated fromchunk.Update the bookkeeping information accordingly. Must be called aftereach successful depopulation.
- structpcpu_chunk*pcpu_chunk_addr_search(void*addr)¶
determine chunk containing specified address
Parameters
void*addraddress for which the chunk needs to be determined.
Description
This is an internal function that handles all but static allocations.Static percpu address values should never be passed into the allocator.
Return
The address of the found chunk.
- void__percpu*pcpu_alloc(size_tsize,size_talign,boolreserved,gfp_tgfp)¶
the percpu allocator
Parameters
size_tsizesize of area to allocate in bytes
size_talignalignment of area (max PAGE_SIZE)
boolreservedallocate from the reserved chunk if available
gfp_tgfpallocation flags
Description
Allocate percpu area ofsize bytes aligned atalign. Ifgfp doesn’tcontainGFP_KERNEL, the allocation is atomic. Ifgfp has __GFP_NOWARNthen no warning will be triggered on invalid or failed allocationrequests.
Return
Percpu pointer to the allocated area on success, NULL on failure.
- voidpcpu_balance_free(boolempty_only)¶
manage the amount of free chunks
Parameters
boolempty_onlyfree chunks only if there are no populated pages
Description
If empty_only isfalse, reclaim all fully free chunks regardless of thenumber of populated pages. Otherwise, only reclaim chunks that have nopopulated pages.
Context
pcpu_lock (can be dropped temporarily)
- voidpcpu_balance_populated(void)¶
manage the amount of populated pages
Parameters
voidno arguments
Description
Maintain a certain amount of populated pages to satisfy atomic allocations.It is possible that this is called when physical memory is scarce causingOOM killer to be triggered. We should avoid doing so until an actualallocation causes the failure as it is possible that requests can beserviced from already backed regions.
Context
pcpu_lock (can be dropped temporarily)
- voidpcpu_reclaim_populated(void)¶
scan over to_depopulate chunks and free empty pages
Parameters
voidno arguments
Description
Scan over chunks in the depopulate list and try to release unused populatedpages back to the system. Depopulated chunks are sidelined to preventrepopulating these pages unless required. Fully free chunks are reintegratedand freed accordingly (1 is kept around). If we drop below the emptypopulated pages threshold, reintegrate the chunk if it has empty free pages.Each chunk is scanned in the reverse order to keep populated pages close tothe beginning of the chunk.
Context
pcpu_lock (can be dropped temporarily)
- voidpcpu_balance_workfn(structwork_struct*work)¶
manage the amount of free chunks and populated pages
Parameters
structwork_struct*workunused
Description
For each chunk type, manage the number of fully free chunks and the number ofpopulated pages. An important thing to consider is when pages are freed andhow they contribute to the global counts.
- voidfree_percpu(void__percpu*ptr)¶
free percpu area
Parameters
void__percpu*ptrpointer to area to free
Description
Free percpu areaptr.
Context
Can be called from atomic context.
- boolis_kernel_percpu_address(unsignedlongaddr)¶
test whether address is from static percpu area
Parameters
unsignedlongaddraddress to test
Description
Test whetheraddr belongs to in-kernel static percpu area. Modulestatic percpu areas are not considered. For those, useis_module_percpu_address().
Return
true ifaddr is from in-kernel static percpu area,false otherwise.
- phys_addr_tper_cpu_ptr_to_phys(void*addr)¶
convert translated percpu address to physical address
Parameters
void*addrthe address to be converted to physical address
Description
Givenaddr which is dereferenceable address obtained via one ofpercpu access macros, this function translates it into its physicaladdress. The caller is responsible for ensuringaddr stays validuntil this function finishes.
percpu allocator has special setup for the first chunk, which currentlysupports either embedding in linear address space or vmalloc mapping,and, from the second one, the backing allocator (currently either vm orkm) provides translation.
The addr can be translated simply without checking if it falls into thefirst chunk. But the current code reflects better how percpu allocatoractually works, and the verification can discover both bugs in percpuallocator itself andper_cpu_ptr_to_phys() callers. So we keep currentcode.
Return
The physical address foraddr.
- structpcpu_alloc_info*pcpu_alloc_alloc_info(intnr_groups,intnr_units)¶
allocate percpu allocation info
Parameters
intnr_groupsthe number of groups
intnr_unitsthe number of units
Description
Allocate ai which is large enough fornr_groups groups containingnr_units units. The returned ai’s groups[0].cpu_map points to thecpu_map array which is long enough fornr_units and filled withNR_CPUS. It’s the caller’s responsibility to initialize cpu_mappointer of other groups.
Return
Pointer to the allocated pcpu_alloc_info on success, NULL onfailure.
- voidpcpu_free_alloc_info(structpcpu_alloc_info*ai)¶
free percpu allocation info
Parameters
structpcpu_alloc_info*aipcpu_alloc_info to free
Description
Freeai which was allocated bypcpu_alloc_alloc_info().
- voidpcpu_dump_alloc_info(constchar*lvl,conststructpcpu_alloc_info*ai)¶
print out information about pcpu_alloc_info
Parameters
constchar*lvlloglevel
conststructpcpu_alloc_info*aiallocation info to dump
Description
Print out information aboutai using loglevellvl.
- voidpcpu_setup_first_chunk(conststructpcpu_alloc_info*ai,void*base_addr)¶
initialize the first percpu chunk
Parameters
conststructpcpu_alloc_info*aipcpu_alloc_info describing how to percpu area is shaped
void*base_addrmapped address
Description
Initialize the first percpu chunk which contains the kernel staticpercpu area. This function is to be called from arch percpu areasetup path.
ai contains all information necessary to initialize the firstchunk and prime the dynamic percpu allocator.
ai->static_size is the size of static percpu area.
ai->reserved_size, if non-zero, specifies the amount of bytes toreserve after the static area in the first chunk. This reservesthe first chunk such that it’s available only through reservedpercpu allocation. This is primarily used to serve module percpustatic areas on architectures where the addressing model haslimited offset range for symbol relocations to guarantee modulepercpu symbols fall inside the relocatable range.
ai->dyn_size determines the number of bytes available for dynamicallocation in the first chunk. The area betweenai->static_size +ai->reserved_size +ai->dyn_size andai->unit_size is unused.
ai->unit_size specifies unit size and must be aligned to PAGE_SIZEand equal to or larger thanai->static_size +ai->reserved_size +ai->dyn_size.
ai->atom_size is the allocation atom size and used as alignmentfor vm areas.
ai->alloc_size is the allocation size and always multiple ofai->atom_size. This is larger thanai->atom_size ifai->unit_size is larger thanai->atom_size.
ai->nr_groups andai->groups describe virtual memory layout ofpercpu areas. Units which should be colocated are put into thesame group. Dynamic VM areas will be allocated according to thesegroupings. Ifai->nr_groups is zero, a single group containingall units is assumed.
The caller should have mapped the first chunk atbase_addr andcopied static data to each unit.
The first chunk will always contain a static and a dynamic region.However, the static region is not managed by any chunk. If the firstchunk also contains a reserved region, it is served by two chunks -one for the reserved region and one for the dynamic region. Theyshare the same vm, but use offset regions in the area allocation map.The chunk serving the dynamic region is circulated in the chunk slotsand available for dynamic allocation like any other chunk.
- structpcpu_alloc_info*pcpu_build_alloc_info(size_treserved_size,size_tdyn_size,size_tatom_size,pcpu_fc_cpu_distance_fn_tcpu_distance_fn)¶
build alloc_info considering distances between CPUs
Parameters
size_treserved_sizethe size of reserved percpu area in bytes
size_tdyn_sizeminimum free size for dynamic allocation in bytes
size_tatom_sizeallocation atom size
pcpu_fc_cpu_distance_fn_tcpu_distance_fncallback to determine distance between cpus, optional
Description
This function determines grouping of units, their mappings to cpusand other parameters considering needed percpu size, allocationatom size and distances between CPUs.
Groups are always multiples of atom size and CPUs which are ofLOCAL_DISTANCE both ways are grouped together and share space forunits in the same group. The returned configuration is guaranteedto have CPUs on different nodes on different groups and >=75% usageof allocated virtual address space.
Return
On success, pointer to the new allocation_info is returned. Onfailure, ERR_PTR value is returned.
- intpcpu_embed_first_chunk(size_treserved_size,size_tdyn_size,size_tatom_size,pcpu_fc_cpu_distance_fn_tcpu_distance_fn,pcpu_fc_cpu_to_node_fn_tcpu_to_nd_fn)¶
embed the first percpu chunk into bootmem
Parameters
size_treserved_sizethe size of reserved percpu area in bytes
size_tdyn_sizeminimum free size for dynamic allocation in bytes
size_tatom_sizeallocation atom size
pcpu_fc_cpu_distance_fn_tcpu_distance_fncallback to determine distance between cpus, optional
pcpu_fc_cpu_to_node_fn_tcpu_to_nd_fncallback to convert cpu to it’s node, optional
Description
This is a helper to ease setting up embedded first percpu chunk andcan be called wherepcpu_setup_first_chunk() is expected.
If this function is used to setup the first chunk, it is allocatedby calling pcpu_fc_alloc and used as-is without being mapped intovmalloc area. Allocations are always whole multiples ofatom_sizealigned toatom_size.
This enables the first chunk to piggy back on the linear physicalmapping which often uses larger page size. Please note that thiscan result in very sparse cpu->unit mapping on NUMA machines thusrequiring large vmalloc address space. Don’t use this allocator ifvmalloc space is not orders of magnitude larger than distancesbetween node memory addresses (ie. 32bit NUMA machines).
dyn_size specifies the minimum dynamic area size.
If the needed size is smaller than the minimum or specified unitsize, the leftover is returned using pcpu_fc_free.
Return
0 on success, -errno on failure.
- intpcpu_page_first_chunk(size_treserved_size,pcpu_fc_cpu_to_node_fn_tcpu_to_nd_fn)¶
map the first chunk using PAGE_SIZE pages
Parameters
size_treserved_sizethe size of reserved percpu area in bytes
pcpu_fc_cpu_to_node_fn_tcpu_to_nd_fncallback to convert cpu to it’s node, optional
Description
This is a helper to ease setting up page-remapped first percpuchunk and can be called wherepcpu_setup_first_chunk() is expected.
This is the basic allocator. Static percpu area is allocatedpage-by-page into vmalloc area.
Return
0 on success, -errno on failure.
- longcopy_from_user_nofault(void*dst,constvoid__user*src,size_tsize)¶
safely attempt to read from a user-space location
Parameters
void*dstpointer to the buffer that shall take the data
constvoid__user*srcaddress to read from. This must be a user address.
size_tsizesize of the data chunk
Description
Safely read from user addresssrc to the buffer atdst. If a kernel faulthappens, handle that and return -EFAULT.
- longcopy_to_user_nofault(void__user*dst,constvoid*src,size_tsize)¶
safely attempt to write to a user-space location
Parameters
void__user*dstaddress to write to
constvoid*srcpointer to the data that shall be written
size_tsizesize of the data chunk
Description
Safely write to addressdst from the buffer atsrc. If a kernel faulthappens, handle that and return -EFAULT.
- longstrncpy_from_user_nofault(char*dst,constvoid__user*unsafe_addr,longcount)¶
Copy a NUL terminated string from unsafe user address.
Parameters
char*dstDestination address, in kernel space. This buffer must be atleastcount bytes long.
constvoid__user*unsafe_addrUnsafe user address.
longcountMaximum number of bytes to copy, including the trailing NUL.
Description
Copies a NUL-terminated string from unsafe user address to kernel buffer.
On success, returns the length of the string INCLUDING the trailing NUL.
If access fails, returns -EFAULT (some data may have been copiedand the trailing NUL added).
Ifcount is smaller than the length of the string, copiescount-1 bytes,sets the last byte ofdst buffer to NUL and returnscount.
- longstrnlen_user_nofault(constvoid__user*unsafe_addr,longcount)¶
Get the size of a user string INCLUDING final NUL.
Parameters
constvoid__user*unsafe_addrThe string to measure.
longcountMaximum count (including NUL)
Description
Get the size of a NUL-terminated string in user space without pagefault.
Returns the size of the string INCLUDING the terminating NUL.
If the string is too long, returns a number larger thancount. Userhas to check the return value against “> count”.On exception (or invalid count), returns 0.
Unlike strnlen_user, this can be used from IRQ handler etc. becauseit disables pagefaults.
- boolwriteback_throttling_sane(structscan_control*sc)¶
is the usual dirty throttling mechanism available?
Parameters
structscan_control*scscan_control in question
Description
The normal page dirty throttling mechanism in balance_dirty_pages() iscompletely broken with the legacy memcg and direct stalling inshrink_folio_list() is used for throttling instead, which lacks all theniceties such as fairness, adaptive pausing, bandwidth proportionalallocation and configurability.
This function tests whether the vmscan currently in progress can assumethat the normal dirty throttling mechanism is operational.
- unsignedlonglruvec_lru_size(structlruvec*lruvec,enumlru_listlru,intzone_idx)¶
Returns the number of pages on the given LRU list.
Parameters
structlruvec*lruveclru vector
enumlru_listlrulru to use
intzone_idxzones to consider (use MAX_NR_ZONES - 1 for the whole LRU list)
- longremove_mapping(structaddress_space*mapping,structfolio*folio)¶
Attempt to remove a folio from its mapping.
Parameters
structaddress_space*mappingThe address space.
structfolio*folioThe folio to remove.
Description
If the folio is dirty, under writeback or if someone else has a refon it, removal will fail.
Return
The number of pages removed from the mapping. 0 if the foliocould not be removed.
Context
The caller should have a single refcount on the folio andhold its lock.
Parameters
structfolio*folioFolio to be returned to an LRU list.
Description
Add previously isolatedfolio to appropriate LRU list.The folio may still be unevictable for other reasons.
Context
lru_lock must not be held, interrupts must be enabled.
Parameters
structfolio*folioFolio to isolate from its LRU list.
Description
Isolate afolio from an LRU list and adjust the vmstat statisticcorresponding to whatever LRU list the folio was on.
The folio will have its LRU flag cleared. If it was found on theactive list, it will have the Active flag set. If it was found on theunevictable list, it will have the Unevictable flag set. These flagsmay need to be cleared by the caller before letting the page go.
Must be called with an elevated refcount on the folio. This is afundamental difference from isolate_lru_folios() (which is calledwithout a stable reference).
The lru_lock must not be held.
Interrupts must be enabled.
Context
Return
true if the folio was removed from an LRU list.false if the folio was not on an LRU list.
- voidcheck_move_unevictable_folios(structfolio_batch*fbatch)¶
Move evictable folios to appropriate zone lru list
Parameters
structfolio_batch*fbatchBatch of lru folios to check.
Description
Checks folios for evictability, if an evictable folio is in the unevictablelru list, moves it to the appropriate evictable lru list. This functionshould be only used for lru folios.
- void__remove_pages(unsignedlongpfn,unsignedlongnr_pages,structvmem_altmap*altmap)¶
remove sections of pages
Parameters
unsignedlongpfnstarting pageframe (must be aligned to start of a section)
unsignedlongnr_pagesnumber of pages to remove (must be multiple of section size)
structvmem_altmap*altmapalternative device page map or
NULLif default memmap is used
Description
Generic helper function to remove section mappings and sysfs entriesfor the section of the memory we are removing. Caller needs to makesure that pages are marked reserved and zones are adjust properly bycalling offline_pages().
- voidtry_offline_node(intnid)¶
Parameters
intnidthe node ID
Description
Offline a node if all memory sections and cpus of the node are removed.
NOTE
The caller must call lock_device_hotplug() to serialize hotplugand online/offline operations before this call.
- void__remove_memory(u64start,u64size)¶
Remove memory if every memory block is offline
Parameters
u64startphysical address of the region to remove
u64sizesize of the region to remove
NOTE
The caller must call lock_device_hotplug() to serialize hotplugand online/offline operations before this call, as required bytry_offline_node().
- unsignedlongmmu_interval_read_begin(structmmu_interval_notifier*interval_sub)¶
Begin a read side critical section against a VA range
Parameters
structmmu_interval_notifier*interval_subThe interval subscription
Description
mmu_iterval_read_begin()/mmu_iterval_read_retry() implement acollision-retry scheme similar to seqcount for the VA range undersubscription. If the mm invokes invalidation during the critical sectionthen mmu_interval_read_retry() will return true.
This is useful to obtain shadow PTEs where teardown or setup of the SPTEsrequire a blocking context. The critical region formed by this can sleep,and the required ‘user_lock’ can also be a sleeping lock.
The caller is required to provide a ‘user_lock’ to serialize both teardownand setup.
The return value should be passed to mmu_interval_read_retry().
- intmmu_notifier_register(structmmu_notifier*subscription,structmm_struct*mm)¶
Register a notifier on a mm
Parameters
structmmu_notifier*subscriptionThe notifier to attach
structmm_struct*mmThe mm to attach the notifier to
Description
Must not hold mmap_lock nor any other VM related lock when callingthis registration function. Must also ensure mm_users can’t go downto zero while this runs to avoid races with mmu_notifier_release,so mm has to be current->mm or the mm should be pinned safely suchas with get_task_mm(). If the mm is not current->mm, the mm_userspin should be released by calling mmput after mmu_notifier_registerreturns.
mmu_notifier_unregister() ormmu_notifier_put() must be always called tounregister the notifier.
While the caller has a mmu_notifier get the subscription->mm pointer will remainvalid, and can be converted to an active mm pointer via mmget_not_zero().
- structmmu_notifier*mmu_notifier_get_locked(conststructmmu_notifier_ops*ops,structmm_struct*mm)¶
Return the single struct mmu_notifier for the mm & ops
Parameters
conststructmmu_notifier_ops*opsThe operations struct being subscribe with
structmm_struct*mmThe mm to attach notifiers too
Description
This function either allocates a new mmu_notifier viaops->alloc_notifier(), or returns an already existing notifier on thelist. The value of the ops pointer is used to determine when two notifiersare the same.
Each call to mmu_notifier_get() must be paired with a call tommu_notifier_put(). The caller must hold the write side of mm->mmap_lock.
While the caller has a mmu_notifier get the mm pointer will remain valid,and can be converted to an active mm pointer via mmget_not_zero().
- voidmmu_notifier_put(structmmu_notifier*subscription)¶
Release the reference on the notifier
Parameters
structmmu_notifier*subscriptionThe notifier to act on
Description
This function must be paired with each mmu_notifier_get(), it releases thereference obtained by the get. If this is the last reference then processto free the notifier will be run asynchronously.
Unlike mmu_notifier_unregister() the get/put flow only calls ops->releasewhen the mm_struct is destroyed. Instead free_notifier is always called torelease any resources held by the user.
As ops->release is not guaranteed to be called, the user must ensure thatall sptes are dropped, and no new sptes can be established beforemmu_notifier_put() is called.
This function can be called from the ops->release callback, however thecaller must still ensure it is called pairwise with mmu_notifier_get().
Modules calling this function must callmmu_notifier_synchronize() intheir __exit functions to ensure the async work is completed.
- intmmu_interval_notifier_insert(structmmu_interval_notifier*interval_sub,structmm_struct*mm,unsignedlongstart,unsignedlonglength,conststructmmu_interval_notifier_ops*ops)¶
Insert an interval notifier
Parameters
structmmu_interval_notifier*interval_subInterval subscription to register
structmm_struct*mmmm_struct to attach to
unsignedlongstartStarting virtual address to monitor
unsignedlonglengthLength of the range to monitor
conststructmmu_interval_notifier_ops*opsInterval notifier operations to be called on matching events
Description
This function subscribes the interval notifier for notifications from themm. Upon return the ops related to mmu_interval_notifier will be calledwhenever an event that intersects with the given range occurs.
Upon return the range_notifier may not be present in the interval tree yet.The caller must use the normal interval notifier read flow viammu_interval_read_begin() to establish SPTEs for this range.
- voidmmu_interval_notifier_remove(structmmu_interval_notifier*interval_sub)¶
Remove a interval notifier
Parameters
structmmu_interval_notifier*interval_subInterval subscription to unregister
Description
This function must be paired withmmu_interval_notifier_insert(). It cannotbe called from any ops callback.
Once this returns ops callbacks are no longer running on other CPUs andwill not be called in future.
- voidmmu_notifier_synchronize(void)¶
Ensure all mmu_notifiers are freed
Parameters
voidno arguments
Description
This function ensures that all outstanding async SRU work frommmu_notifier_put() is completed. After it returns any mmu_notifier_opsassociated with an unused mmu_notifier will no longer be called.
Before using the caller must ensure that all of its mmu_notifiers have beenfully released viammu_notifier_put().
Modules using themmu_notifier_put() API should call this in their __exitfunction to avoid module unloading races.
- size_tballoon_page_list_enqueue(structballoon_dev_info*b_dev_info,structlist_head*pages)¶
inserts a list of pages into the balloon page list.
Parameters
structballoon_dev_info*b_dev_infoballoon device descriptor where we will insert a new page to
structlist_head*pagespages to enqueue - allocated using balloon_page_alloc.
Description
Driver must call this function to properly enqueue balloon pages beforedefinitively removing them from the guest system.
Return
number of pages that were enqueued.
- size_tballoon_page_list_dequeue(structballoon_dev_info*b_dev_info,structlist_head*pages,size_tn_req_pages)¶
removes pages from balloon’s page list and returns a list of the pages.
Parameters
structballoon_dev_info*b_dev_infoballoon device descriptor where we will grab a page from.
structlist_head*pagespointer to the list of pages that would be returned to the caller.
size_tn_req_pagesnumber of requested pages.
Description
Driver must call this function to properly de-allocate a previous enlistedballoon pages before definitively releasing it back to the guest system.This function tries to removen_req_pages from the ballooned pages andreturn them to the caller in thepages list.
Note that this function may fail to dequeue some pages even if the balloonisn’t empty - since the page list can be temporarily empty due to compactionof isolated pages.
Return
number of pages that were added to thepages list.
- vm_fault_tvmf_insert_pfn_pmd(structvm_fault*vmf,pfn_tpfn,boolwrite)¶
insert a pmd size pfn
Parameters
structvm_fault*vmfStructure describing the fault
pfn_tpfnpfn to insert
boolwritewhether it’s a write fault
Description
Insert a pmd size pfn. Seevmf_insert_pfn() for additional info.
Return
vm_fault_t value.
- vm_fault_tvmf_insert_pfn_pud(structvm_fault*vmf,pfn_tpfn,boolwrite)¶
insert a pud size pfn
Parameters
structvm_fault*vmfStructure describing the fault
pfn_tpfnpfn to insert
boolwritewhether it’s a write fault
Description
Insert a pud size pfn. Seevmf_insert_pfn() for additional info.
Return
vm_fault_t value.
- vm_fault_tvmf_insert_folio_pud(structvm_fault*vmf,structfolio*folio,boolwrite)¶
insert a pud size folio mapped by a pud entry
Parameters
structvm_fault*vmfStructure describing the fault
structfolio*foliofolio to insert
boolwritewhether it’s a write fault
Return
vm_fault_t value.
- intio_mapping_map_user(structio_mapping*iomap,structvm_area_struct*vma,unsignedlongaddr,unsignedlongpfn,unsignedlongsize)¶
remap an I/O mapping to userspace
Parameters
structio_mapping*iomapthe source io_mapping
structvm_area_struct*vmauser vma to map to
unsignedlongaddrtarget user address to start at
unsignedlongpfnphysical address of kernel memory
unsignedlongsizesize of map area
Note
this is only safe if the mm semaphore is held when called.