2.Supported File Operations

Below are a discussion of the high level file operations that iomapimplements.

2.1.Buffered I/O

Buffered I/O is the default file I/O path in Linux.File contents are cached in memory (“pagecache”) to satisfy reads andwrites.Dirty cache will be written back to disk at some point that can beforced viafsync and variants.

iomap implements nearly all the folio and pagecache management thatfilesystems have to implement themselves under the legacy I/O model.This means that the filesystem need not know the details of allocating,mapping, managing uptodate and dirty state, or writeback of pagecachefolios.Under the legacy I/O model, this was managed very inefficiently withlinked lists of buffer heads instead of the per-folio bitmaps that iomapuses.Unless the filesystem explicitly opts in to buffer heads, they will notbe used, which makes buffered I/O much more efficient, and the pagecachemaintainer much happier.

2.1.1.structaddress_space_operations

The following iomap functions can be referenced directly from theaddress space operations structure:

  • iomap_dirty_folio

  • iomap_release_folio

  • iomap_invalidate_folio

  • iomap_is_partially_uptodate

The following address space operations can be wrapped easily:

  • read_folio

  • readahead

  • writepages

  • bmap

  • swap_activate

2.1.2.structiomap_write_ops

structiomap_write_ops{structfolio*(*get_folio)(structiomap_iter*iter,loff_tpos,unsignedlen);void(*put_folio)(structinode*inode,loff_tpos,unsignedcopied,structfolio*folio);bool(*iomap_valid)(structinode*inode,conststructiomap*iomap);int(*read_folio_range)(conststructiomap_iter*iter,structfolio*folio,loff_tpos,size_tlen);};

iomap calls these functions:

  • get_folio: Called to allocate and return an active reference toa locked folio prior to starting a write.If this function is not provided, iomap will calliomap_get_folio.This could be used toset up per-folio filesystem statefor a write.

  • put_folio: Called to unlock and put a folio after a pagecacheoperation completes.If this function is not provided, iomap willfolio_unlock andfolio_put on its own.This could be used tocommit per-folio filesystem statethat was set up by->get_folio.

  • iomap_valid: The filesystem may not hold locks between->iomap_begin and->iomap_end because pagecache operationscan take folio locks, fault on userspace pages, initiate writebackfor memory reclamation, or engage in other time-consuming actions.If a file’s space mapping data are mutable, it is possible that themapping for a particular pagecache folio canchange in the time ittakesto allocate, install, and lock that folio.

    For the pagecache, races can happen if writeback doesn’t takei_rwsem orinvalidate_lock and updates mapping information.Races can also happen if the filesystem allows concurrent writes.For such files, the mappingmust be revalidated after the foliolock has been taken so that iomap can manage the folio correctly.

    fsdax does not need this revalidation because there’s no writebackand no support for unwritten extents.

    Filesystems subject to this kind of race must provide a->iomap_valid function to decide if the mapping is still valid.If the mapping is not valid, the mapping will be sampled again.

    To support making the validity decision, the filesystem’s->iomap_begin function may setstructiomap::validity_cookieat the same time that it populates the other iomap fields.A simple validation cookie implementation is a sequence counter.If the filesystem bumps the sequence counter every time it modifiesthe inode’s extent map, it can be placed in thestructiomap::validity_cookie during->iomap_begin.If the value in the cookie is found to be different to the valuethe filesystem holds when the mapping is passed back to->iomap_valid, then the iomap should considered stale and thevalidation failed.

  • read_folio_range: Called to synchronously read in the range that willbe written to. If this function is not provided, iomap will default tosubmitting a bio read request.

Thesestructkiocb flags are significant for buffered I/O with iomap:

  • IOCB_NOWAIT: Turns onIOMAP_NOWAIT.

  • IOCB_DONTCACHE: Turns onIOMAP_DONTCACHE.

2.1.3.structiomap_read_ops

structiomap_read_ops{int(*read_folio_range)(conststructiomap_iter*iter,structiomap_read_folio_ctx*ctx,size_tlen);void(*submit_read)(structiomap_read_folio_ctx*ctx);};

iomap calls these functions:

  • read_folio_range: Called to read in the range. This must be providedby the caller. If this succeeds,iomap_finish_folio_read() must be calledafter the range is read in, regardless of whether the read succeeded orfailed.

  • submit_read: Submit any pending read requests. This function isoptional.

2.1.4.Internal per-Folio State

If the fsblock size matches the size of a pagecache folio, it is assumedthat all disk I/O operations will operate on the entire folio.The uptodate (memory contents are at least as new as what’s on disk) anddirty (memory contents are newer than what’s on disk) status of thefolio are all that’s needed for this case.

If the fsblock size is less than the size of a pagecache folio, iomaptracks the per-fsblock uptodate and dirty state itself.This enables iomap to handle both “bs < ps”filesystemsand large folios in the pagecache.

iomap internally tracks two state bits per fsblock:

  • uptodate: iomap will try to keep folios fully up to date.If there are read(ahead) errors, those fsblocks will not be markeduptodate.The folio itself will be marked uptodate when all fsblocks within thefolio are uptodate.

  • dirty: iomap will set the per-block dirty state when programswrite to the file.The folio itself will be marked dirty when any fsblock within thefolio is dirty.

iomap also tracks the amount of read and write disk IOs that are inflight.This structure is much lighter weight thanstructbuffer_headbecause there is only one per folio, and the per-fsblock overhead is twobits vs. 104 bytes.

Filesystems wishing to turn on large folios in the pagecache should callmapping_set_large_folios when initializing the incore inode.

2.1.5.Buffered Readahead and Reads

Theiomap_readahead function initiates readahead to the pagecache.Theiomap_read_folio function reads one folio’s worth of data intothe pagecache.Theflags argument to->iomap_begin will be set to zero.The pagecache takes whatever locks it needs before calling thefilesystem.

Bothiomap_readahead andiomap_read_folio pass in astructiomap_read_folio_ctx:

structiomap_read_folio_ctx{conststructiomap_read_ops*ops;structfolio*cur_folio;structreadahead_control*rac;void*read_ctx;};
iomap_readahead must set:

  • ops->read_folio_range() andrac

iomap_read_folio must set:

  • ops->read_folio_range() andcur_folio

ops->submit_read() andread_ctx are optional.read_ctx is used topass in any custom data the caller needs accessible in the ops callbacks forfulfilling reads.

2.1.6.Buffered Writes

Theiomap_file_buffered_write function writes aniocb to thepagecache.IOMAP_WRITE orIOMAP_WRITE |IOMAP_NOWAIT will be passed astheflags argument to->iomap_begin.Callers commonly takei_rwsem in either shared or exclusive modebefore calling this function.

2.1.6.1.mmap Write Faults

Theiomap_page_mkwrite function handles a write fault to a folio inthe pagecache.IOMAP_WRITE|IOMAP_FAULT will be passed as theflags argumentto->iomap_begin.Callers commonly take the mmapinvalidate_lock in shared orexclusive mode before calling this function.

2.1.6.2.Buffered Write Failures

After a short write to the pagecache, the areas not written will notbecome marked dirty.The filesystem must arrange tocancelsuchreservationsbecause writeback will not consume the reservation.Theiomap_write_delalloc_release can be called from a->iomap_end function to find all the clean areas of the folioscaching a fresh (IOMAP_F_NEW) delalloc mapping.It takes theinvalidate_lock.

The filesystem must supply a functionpunch to be called foreach file range in this state.This function mustonly remove delayed allocation reservations, incase another thread racing with the current thread writes successfullyto the same region and triggers writeback to flush the dirty data out todisk.

2.1.6.3.Zeroing for File Operations

Filesystems can calliomap_zero_range to perform zeroing of thepagecache for non-truncation file operations that are not aligned tothe fsblock size.IOMAP_ZERO will be passed as theflags argument to->iomap_begin.Callers typically holdi_rwsem andinvalidate_lock in exclusivemode before calling this function.

2.1.6.4.Unsharing Reflinked File Data

Filesystems can calliomap_file_unshare to force a file sharingstorage with another file to preemptively copy the shared data to newlyallocate storage.IOMAP_WRITE|IOMAP_UNSHARE will be passed as theflags argumentto->iomap_begin.Callers typically holdi_rwsem andinvalidate_lock in exclusivemode before calling this function.

2.1.7.Truncation

Filesystems can calliomap_truncate_page to zero the bytes in thepagecache from EOF to the end of the fsblock during a file truncationoperation.truncate_setsize ortruncate_pagecache will take care ofeverything after the EOF block.IOMAP_ZERO will be passed as theflags argument to->iomap_begin.Callers typically holdi_rwsem andinvalidate_lock in exclusivemode before calling this function.

2.1.8.Pagecache Writeback

Filesystems can calliomap_writepages to respond to a request towrite dirty pagecache folios to disk.Themapping andwbc parameters should be passed unchanged.Thewpc pointer should be allocated by the filesystem and mustbe initialized to zero.

The pagecache will lock each folio before trying to schedule it forwriteback.It does not locki_rwsem orinvalidate_lock.

The dirty bit will be cleared for all folios run through the->writeback_range machinery described below even if the writeback fails.This is to prevent dirty folio clots when storage devices fail; an-EIO is recorded for userspace to collect viafsync.

Theops structure must be specified and is as follows:

2.1.8.1.structiomap_writeback_ops

structiomap_writeback_ops{int(*writeback_range)(structiomap_writepage_ctx*wpc,structfolio*folio,u64pos,unsignedintlen,u64end_pos);int(*writeback_submit)(structiomap_writepage_ctx*wpc,interror);};

The fields are as follows:

  • writeback_range: Setswpc->iomap to the space mapping of the filerange (in bytes) given byoffset andlen.iomap calls this function for each dirty fs block in each dirty folio,though it willreuse mappingsfor runs of contiguous dirty fsblocks within a folio.Do not returnIOMAP_INLINE mappings here; the->iomap_endfunction must deal with persisting written data.Do not returnIOMAP_DELALLOC mappings here; iomap currentlyrequires mapping to allocated space.Filesystems can skip a potentially expensive mapping lookup if themappings have not changed.This revalidation must be open-coded by the filesystem; it isunclear ifiomap::validity_cookie can be reused for thispurpose.

    If this methods fails to schedule I/O for any part of a dirty folio, itshould throw away any reservations that may have been made for the write.The folio will be marked clean and an-EIO recorded in thepagecache.Filesystems can use this callback toremovedelalloc reservations to avoid having delalloc reservations forclean pagecache.This function must be supplied by the filesystem.If this succeeds,iomap_finish_folio_write() must be called once writebackcompletes for the range, regardless of whether the writeback succeeded orfailed.

  • writeback_submit: Submit the previous built writeback context.Block based file systems should use the iomap_ioend_writeback_submithelper, other file system can implement their own.File systems can optionally hook into writeback bio submission.This might include pre-write space accounting updates, or installinga custom->bi_end_io function for internal purposes, such asdeferring the ioend completion to a workqueue to run metadata updatetransactions from process context before submitting the bio.This function must be supplied by the filesystem.

2.1.8.2.Pagecache Writeback Completion

To handle the bookkeeping that must happen after disk I/O for writebackcompletes, iomap creates chains ofstructiomap_ioend objects thatwrap thebio that is used to write pagecache data to disk.By default, iomap finishes writeback ioends by clearing the writebackbit on the folios attached to theioend.If the write failed, it will also set the error bits on the folios andthe address space.This can happen in interrupt or process context, depending on thestorage device.Filesystems that need to update internal bookkeeping (e.g. unwrittenextent conversions) should set their own bi_end_io on the biossubmitted by->submit_writebackThis function should calliomap_finish_ioends after finishing itsown work (e.g. unwritten extent conversion).

Some filesystems may wish toamortize the cost of running metadatatransactionsfor post-writeback updates by batching them.They may also require transactions to run from process context, whichimplies punting batches to a workqueue.iomap ioends contain alist_head to enable batching.

Given a batch of ioends, iomap has a few helpers to assist withamortization:

  • iomap_sort_ioends: Sort all the ioends in the list by fileoffset.

  • iomap_ioend_try_merge: Given an ioend that is not in any list anda separate list of sorted ioends, merge as many of the ioends fromthe head of the list into the given ioend.ioends can only be merged if the file range and storage addresses arecontiguous; the unwritten and shared status are the same; and thewrite I/O outcome is the same.The merged ioends become their own list.

  • iomap_finish_ioends: Finish an ioend that possibly has otherioends linked to it.

2.2.Direct I/O

In Linux, direct I/O is defined as file I/O that is issued directly tostorage, bypassing the pagecache.Theiomap_dio_rw function implements O_DIRECT (direct I/O) reads andwrites for files.

ssize_tiomap_dio_rw(structkiocb*iocb,structiov_iter*iter,conststructiomap_ops*ops,conststructiomap_dio_ops*dops,unsignedintdio_flags,void*private,size_tdone_before);

The filesystem can provide thedops parameter if it needs to performextra work before or after the I/O is issued to storage.Thedone_before parameter tells the how much of the request hasalready been transferred.It is used to continue a request asynchronously whenpart of therequesthas already been completed synchronously.

Thedone_before parameter should be set if writes for theiocbhave been initiated prior to the call.The direction of the I/O is determined from theiocb passed in.

Thedio_flags argument can be set to any combination of thefollowing values:

  • IOMAP_DIO_FORCE_WAIT: Wait for the I/O to complete even if thekiocb is not synchronous.

  • IOMAP_DIO_OVERWRITE_ONLY: Perform a pure overwrite for this rangeor fail with-EAGAIN.This can be used by filesystems with complex unaligned I/Owrite paths to provide an optimised fast path for unaligned writes.If a pure overwrite can be performed, then serialisation againstother I/Os to the same filesystem block(s) is unnecessary as there isno risk of stale data exposure or data loss.If a pure overwrite cannot be performed, then the filesystem canperform the serialisation steps needed to provide exclusive accessto the unaligned I/O range so that it can perform allocation andsub-block zeroing safely.Filesystems can use this flag to try to reduce locking contention,but a lot ofdetailed checkingis required to do itcorrectly.

  • IOMAP_DIO_PARTIAL: If a page fault occurs, return whateverprogress has already been made.The caller may deal with the page fault and retry the operation.If the caller decides to retry the operation, it should pass theaccumulated return values of all previous calls as thedone_before parameter to the next call.

Thesestructkiocb flags are significant for direct I/O with iomap:

  • IOCB_NOWAIT: Turns onIOMAP_NOWAIT.

  • IOCB_SYNC: Ensure that the device has persisted data to diskbefore completing the call.In the case of pure overwrites, the I/O may be issued with FUAenabled.

  • IOCB_HIPRI: Poll for I/O completion instead of waiting for aninterrupt.Only meaningful for asynchronous I/O, and only if the entire I/O canbe issued as a singlestructbio.

Filesystems should calliomap_dio_rw from->read_iter and->write_iter, and setFMODE_CAN_ODIRECT in the->openfunction for the file.They should not set->direct_IO, which is deprecated.

If a filesystem wishes to perform its own work before direct I/Ocompletion, it should call__iomap_dio_rw.If its return value is not an error pointer or a NULL pointer, thefilesystem should pass the return value toiomap_dio_complete afterfinishing its internal work.

2.2.1.Return Values

iomap_dio_rw can return one of the following:

  • A non-negative number of bytes transferred.

  • -ENOTBLK: Fall back to buffered I/O.iomap itself will return this value if it cannot invalidate the pagecache before issuing the I/O to storage.The->iomap_begin or->iomap_end functions may also returnthis value.

  • -EIOCBQUEUED: The asynchronous direct I/O request has beenqueued and will be completed separately.

  • Any of the other negative error codes.

2.2.2.Direct Reads

A direct I/O read initiates a read I/O from the storage device to thecaller’s buffer.Dirty parts of the pagecache are flushed to storage before initiatingthe read io.Theflags value for->iomap_begin will beIOMAP_DIRECT withany combination of the following enhancements:

  • IOMAP_NOWAIT, as defined previously.

Callers commonly holdi_rwsem in shared mode before calling thisfunction.

2.2.3.Direct Writes

A direct I/O write initiates a write I/O to the storage device from thecaller’s buffer.Dirty parts of the pagecache are flushed to storage before initiatingthe write io.The pagecache is invalidated both before and after the write io.Theflags value for->iomap_begin will beIOMAP_DIRECT|IOMAP_WRITE with any combination of the following enhancements:

  • IOMAP_NOWAIT, as defined previously.

  • IOMAP_OVERWRITE_ONLY: Allocating blocks and zeroing partialblocks is not allowed.The entire file range must map to a single written or unwrittenextent.The file I/O range must be aligned to the filesystem block sizeif the mapping is unwritten and the filesystem cannot handle zeroingthe unaligned regions without exposing stale contents.

  • IOMAP_ATOMIC: This write is being issued with torn-writeprotection.Torn-write protection may be provided based on HW-offload or by asoftware mechanism provided by the filesystem.

    For HW-offload based support, only a single bio can be created for thewrite, and the write must not be split into multiple I/O requests, i.e.flag REQ_ATOMIC must be set.The file range to write must be aligned to satisfy the requirementsof both the filesystem and the underlying block device’s atomiccommit capabilities.If filesystem metadata updates are required (e.g. unwritten extentconversion or copy-on-write), all updates for the entire file rangemust be committed atomically as well.Untorn-writes may be longer than a single file block. In all cases,the mapping start disk block must have at least the same alignment asthe write offset.The filesystems must set IOMAP_F_ATOMIC_BIO to inform iomap core of anuntorn-write based on HW-offload.

    For untorn-writes based on a software mechanism provided by thefilesystem, all the disk block alignment and single bio restrictionswhich apply for HW-offload based untorn-writes do not apply.The mechanism would typically be used as a fallback for whenHW-offload based untorn-writes may not be issued, e.g. the range of thewrite covers multiple extents, meaning that it is not possible to issuea single bio.All filesystem metadata updates for the entire file range must becommitted atomically as well.

Callers commonly holdi_rwsem in shared or exclusive mode beforecalling this function.

2.2.4.structiomap_dio_ops:

structiomap_dio_ops{void(*submit_io)(conststructiomap_iter*iter,structbio*bio,loff_tfile_offset);int(*end_io)(structkiocb*iocb,ssize_tsize,interror,unsignedflags);structbio_set*bio_set;};

The fields of this structure are as follows:

  • submit_io: iomap calls this function when it has constructed astructbio object for the I/O requested, and wishes to submit itto the block device.If no function is provided,submit_bio will be called directly.Filesystems that would like to perform additional work before (e.g.data replication for btrfs) should implement this function.

  • end_io: This is called after thestructbio completes.This function should perform post-write conversions of unwrittenextent mappings, handle write failures, etc.Theflags argument may be set to a combination of the following:

    • IOMAP_DIO_UNWRITTEN: The mapping was unwritten, so the ioendshould mark the extent as written.

    • IOMAP_DIO_COW: Writing to the space in the mapping required acopy on write operation, so the ioend should switch mappings.

  • bio_set: This allows the filesystem to provide a custom bio_setfor allocating direct I/O bios.This enables filesystems tostash additional per-bio informationfor private use.If this field is NULL, genericstructbio objects will be used.

Filesystems that want to perform extra work after an I/O completionshould set a custom->bi_end_io function via->submit_io.Afterwards, the custom endio function must calliomap_dio_bio_end_io to finish the direct I/O.

2.3.DAX I/O

Some storage devices can be directly mapped as memory.These devices support a new access mode known as “fsdax” that allowsloads and stores through the CPU and memory controller.

2.3.1.fsdax Reads

A fsdax read performs a memcpy from storage device to the caller’sbuffer.Theflags value for->iomap_begin will beIOMAP_DAX with anycombination of the following enhancements:

  • IOMAP_NOWAIT, as defined previously.

Callers commonly holdi_rwsem in shared mode before calling thisfunction.

2.3.2.fsdax Writes

A fsdax write initiates a memcpy to the storage device from the caller’sbuffer.Theflags value for->iomap_begin will beIOMAP_DAX|IOMAP_WRITE with any combination of the following enhancements:

  • IOMAP_NOWAIT, as defined previously.

  • IOMAP_OVERWRITE_ONLY: The caller requires a pure overwrite to beperformed from this mapping.This requires the filesystem extent mapping to already exist as anIOMAP_MAPPED type and span the entire range of the write I/Orequest.If the filesystem cannot map this request in a way that allows theiomap infrastructure to perform a pure overwrite, it must fail themapping operation with-EAGAIN.

Callers commonly holdi_rwsem in exclusive mode before calling thisfunction.

2.3.2.1.fsdax mmap Faults

Thedax_iomap_fault function handles read and write faults to fsdaxstorage.For a read fault,IOMAP_DAX|IOMAP_FAULT will be passed as theflags argument to->iomap_begin.For a write fault,IOMAP_DAX|IOMAP_FAULT|IOMAP_WRITE will bepassed as theflags argument to->iomap_begin.

Callers commonly hold the same locks as they do to call their iomappagecache counterparts.

2.3.3.fsdax Truncation, fallocate, and Unsharing

For fsdax files, the following functions are provided to replace theiriomap pagecache I/O counterparts.Theflags argument to->iomap_begin are the same as thepagecache counterparts, withIOMAP_DAX added.

  • dax_file_unshare

  • dax_zero_range

  • dax_truncate_page

Callers commonly hold the same locks as they do to call their iomappagecache counterparts.

2.3.4.fsdax Deduplication

Filesystems implementing theFIDEDUPERANGE ioctl must call thedax_remap_file_range_prep function with their own iomap read ops.

2.4.Seeking Files

iomap implements the two iterating whence modes of thellseek systemcall.

2.4.1.SEEK_DATA

Theiomap_seek_data function implements the SEEK_DATA “whence” valuefor llseek.IOMAP_REPORT will be passed as theflags argument to->iomap_begin.

For unwritten mappings, the pagecache will be searched.Regions of the pagecache with a folio mapped and uptodate fsblockswithin those folios will be reported as data areas.

Callers commonly holdi_rwsem in shared mode before calling thisfunction.

2.4.2.SEEK_HOLE

Theiomap_seek_hole function implements the SEEK_HOLE “whence” valuefor llseek.IOMAP_REPORT will be passed as theflags argument to->iomap_begin.

For unwritten mappings, the pagecache will be searched.Regions of the pagecache with no folio mapped, or a !uptodate fsblockwithin a folio will be reported as sparse hole areas.

Callers commonly holdi_rwsem in shared mode before calling thisfunction.

2.5.Swap File Activation

Theiomap_swapfile_activate function finds all the base-page alignedregions in a file and sets them up as swap space.The file will befsync()’d before activation.IOMAP_REPORT will be passed as theflags argument to->iomap_begin.All mappings must be mapped or unwritten; cannot be dirty or shared, andcannot span multiple block devices.Callers must holdi_rwsem in exclusive mode; this is alreadyprovided byswapon.

2.6.File Space Mapping Reporting

iomap implements two of the file space mapping system calls.

2.6.1.FS_IOC_FIEMAP

Theiomap_fiemap function exports file extent mappings to userspacein the format specified by theFS_IOC_FIEMAP ioctl.IOMAP_REPORT will be passed as theflags argument to->iomap_begin.Callers commonly holdi_rwsem in shared mode before calling thisfunction.

2.6.2.FIBMAP (deprecated)

iomap_bmap implements FIBMAP.The calling conventions are the same as for FIEMAP.This function is only provided to maintain compatibility for filesystemsthat implemented FIBMAP prior to conversion.This ioctl is deprecated; donot add a FIBMAP implementation tofilesystems that do not have it.Callers should probably holdi_rwsem in shared mode before callingthis function, but this is unclear.