Zoned block devices¶

Zoned storage devices belong to a class of storage devices with an addressspace that is divided into zones. A zone is a group of consecutive LBAs and allzones are contiguous (there are no LBA gaps). Zones may have different types.

• Conventional zones: there are no access constraints to LBAs belonging toconventional zones. Any read or write access can be executed, similarly to aregular block device.

• Sequential zones: these zones accept random reads but must be writtensequentially. Each sequential zone has a write pointer maintained by thedevice that keeps track of the mandatory start LBA position of the next writeto the device. As a result of this write constraint, LBAs in a sequential zonecannot be overwritten. Sequential zones must first be erased using a specialcommand (zone reset) before rewriting.

Zoned storage devices can be implemented using various recording and mediatechnologies. The most common form of zoned storage today uses the SCSI ZonedBlock Commands (ZBC) and Zoned ATA Commands (ZAC) interfaces on ShingledMagnetic Recording (SMR) HDDs.

Solid State Disks (SSD) storage devices can also implement a zoned interfaceto, for instance, reduce internal write amplification due to garbage collection.The NVMe Zoned NameSpace (ZNS) is a technical proposal of the NVMe standardcommittee aiming at adding a zoned storage interface to the NVMe protocol.

On-disk metadata¶

zonefs on-disk metadata is reduced to an immutable super block whichpersistently stores a magic number and optional feature flags and values. Onmount, zonefs usesblkdev_report_zones() to obtain the device zone configurationand populates the mount point with a static file tree solely based on thisinformation. File sizes come from the device zone type and write pointerposition managed by the device itself.

The super block is always written on disk at sector 0. The first zone of thedevice storing the super block is never exposed as a zone file by zonefs. Ifthe zone containing the super block is a sequential zone, the mkzonefs formattool always “finishes” the zone, that is, it transitions the zone to a fullstate to make it read-only, preventing any data write.

Zone type sub-directories¶

Files representing zones of the same type are grouped together under the samesub-directory automatically created on mount.

For conventional zones, the sub-directory “cnv” is used. This directory ishowever created if and only if the device has usable conventional zones. Ifthe device only has a single conventional zone at sector 0, the zone will notbe exposed as a file as it will be used to store the zonefs super block. Forsuch devices, the “cnv” sub-directory will not be created.

For sequential write zones, the sub-directory “seq” is used.

These two directories are the only directories that exist in zonefs. Userscannot create other directories and cannot rename nor delete the “cnv” and“seq” sub-directories.

The size of the directories indicated by the st_size field ofstructstat,obtained with thestat() orfstat() system calls, indicates the number of filesexisting under the directory.

Zone files¶

Zone files are named using the number of the zone they represent within the setof zones of a particular type. That is, both the “cnv” and “seq” directoriescontain files named “0”, “1”, “2”, ... The file numbers also representincreasing zone start sector on the device.

All read and write operations to zone files are not allowed beyond the filemaximum size, that is, beyond the zone capacity. Any access exceeding the zonecapacity is failed with the -EFBIG error.

Creating, deleting, renaming or modifying any attribute of files andsub-directories is not allowed.

The number of blocks of a file as reported bystat() andfstat() indicates thecapacity of the zone file, or in other words, the maximum file size.

Conventional zone files¶

The size of conventional zone files is fixed to the size of the zone theyrepresent. Conventional zone files cannot be truncated.

These files can be randomly read and written using any type of I/O operation:buffered I/Os, direct I/Os, memory mapped I/Os (mmap), etc. There are no I/Oconstraint for these files beyond the file size limit mentioned above.

Sequential zone files¶

The size of sequential zone files grouped in the “seq” sub-directory representsthe file’s zone write pointer position relative to the zone start sector.

Sequential zone files can only be written sequentially, starting from the fileend, that is, write operations can only be append writes. Zonefs makes noattempt at accepting random writes and will fail any write request that has astart offset not corresponding to the end of the file, or to the end of the lastwrite issued and still in-flight (for asynchronous I/O operations).

Since dirty page writeback by the page cache does not guarantee a sequentialwrite pattern, zonefs prevents buffered writes and writeable shared mappingson sequential files. Only direct I/O writes are accepted for these files.zonefs relies on the sequential delivery of write I/O requests to the deviceimplemented by the block layer elevator. An elevator implementing the sequentialwrite feature for zoned block device (ELEVATOR_F_ZBD_SEQ_WRITE elevator feature)must be used. This type of elevator (e.g. mq-deadline) is set by defaultfor zoned block devices on device initialization.

There are no restrictions on the type of I/O used for read operations insequential zone files. Buffered I/Os, direct I/Os and shared read mappings areall accepted.

Truncating sequential zone files is allowed only down to 0, in which case, thezone is reset to rewind the file zone write pointer position to the start ofthe zone, or up to the zone capacity, in which case the file’s zone istransitioned to the FULL state (finish zone operation).

Format options¶

Several optional features of zonefs can be enabled at format time.

• Conventional zone aggregation: ranges of contiguous conventional zones can beaggregated into a single larger file instead of the default one file per zone.

• File ownership: The owner UID and GID of zone files is by default 0 (root)but can be changed to any valid UID/GID.

• File access permissions: the default 640 access permissions can be changed.

IO error handling¶

Zoned block devices may fail I/O requests for reasons similar to regular blockdevices, e.g. due to bad sectors. However, in addition to such known I/Ofailure pattern, the standards governing zoned block devices behavior defineadditional conditions that result in I/O errors.

• A zone may transition to the read-only condition (BLK_ZONE_COND_READONLY):While the data already written in the zone is still readable, the zone canno longer be written. No user action on the zone (zone management command orread/write access) can change the zone condition back to a normal read/writestate. While the reasons for the device to transition a zone to read-onlystate are not defined by the standards, a typical cause for such transitionwould be a defective write head on an HDD (all zones under this head arechanged to read-only).

• A zone may transition to the offline condition (BLK_ZONE_COND_OFFLINE):An offline zone cannot be read nor written. No user action can transition anoffline zone back to an operational good state. Similarly to zone read-onlytransitions, the reasons for a drive to transition a zone to the offlinecondition are undefined. A typical cause would be a defective read-write headon an HDD causing all zones on the platter under the broken head to beinaccessible.

• Unaligned write errors: These errors result from the host issuing writerequests with a start sector that does not correspond to a zone write pointerposition when the write request is executed by the device. Even though zonefsenforces sequential file write for sequential zones, unaligned write errorsmay still happen in the case of a partial failure of a very large direct I/Ooperation split into multiple BIOs/requests or asynchronous I/O operations.If one of the write request within the set of sequential write requestsissued to the device fails, all write requests queued after it willbecome unaligned and fail.

• Delayed write errors: similarly to regular block devices, if the device sidewrite cache is enabled, write errors may occur in ranges of previouslycompleted writes when the device write cache is flushed, e.g. onfsync().Similarly to the previous immediate unaligned write error case, delayed writeerrors can propagate through a stream of cached sequential data for a zonecausing all data to be dropped after the sector that caused the error.

All I/O errors detected by zonefs are notified to the user with an error codereturn for the system call that triggered or detected the error. The recoveryactions taken by zonefs in response to I/O errors depend on the I/O type (readvs write) and on the reason for the error (bad sector, unaligned writes or zonecondition change).

• For read I/O errors, zonefs does not execute any particular recovery action,but only if the file zone is still in a good condition and there is noinconsistency between the file inode size and its zone write pointer position.If a problem is detected, I/O error recovery is executed (see below table).

• For write I/O errors, zonefs I/O error recovery is always executed.

• A zone condition change to read-only or offline also always triggers zonefsI/O error recovery.

Zonefs minimal I/O error recovery may change a file size and file accesspermissions.

• File size changes:Immediate or delayed write errors in a sequential zone file may cause the fileinode size to be inconsistent with the amount of data successfully written inthe file zone. For instance, the partial failure of a multi-BIO large writeoperation will cause the zone write pointer to advance partially, even thoughthe entire write operation will be reported as failed to the user. In suchcase, the file inode size must be advanced to reflect the zone write pointerchange and eventually allow the user to restart writing at the end of thefile.A file size may also be reduced to reflect a delayed write error detected onfsync(): in this case, the amount of data effectively written in the zone maybe less than originally indicated by the file inode size. After such I/Oerror, zonefs always fixes the file inode size to reflect the amount of datapersistently stored in the file zone.

• Access permission changes:A zone condition change to read-only is indicated with a change in the fileaccess permissions to render the file read-only. This disables changes to thefile attributes and data modification. For offline zones, all permissions(read and write) to the file are disabled.

Further action taken by zonefs I/O error recovery can be controlled by the userwith the “errors=xxx” mount option. The table below summarizes the result ofzonefs I/O error processing depending on the mount option and on the zoneconditions:

+--------------+-----------+-----------------------------------------+|              |           |            Post error state             || "errors=xxx" |  device   |                 access permissions      ||    mount     |   zone    | file         file          device zone  ||    option    | condition | size     read    write    read    write |+--------------+-----------+-----------------------------------------+|              | good      | fixed    yes     no       yes     yes   || remount-ro   | read-only | as is    yes     no       yes     no    || (default)    | offline   |   0      no      no       no      no    |+--------------+-----------+-----------------------------------------+|              | good      | fixed    yes     no       yes     yes   || zone-ro      | read-only | as is    yes     no       yes     no    ||              | offline   |   0      no      no       no      no    |+--------------+-----------+-----------------------------------------+|              | good      |   0      no      no       yes     yes   || zone-offline | read-only |   0      no      no       yes     no    ||              | offline   |   0      no      no       no      no    |+--------------+-----------+-----------------------------------------+|              | good      | fixed    yes     yes      yes     yes   || repair       | read-only | as is    yes     no       yes     no    ||              | offline   |   0      no      no       no      no    |+--------------+-----------+-----------------------------------------+

Further notes:

• The “errors=remount-ro” mount option is the default behavior of zonefs I/Oerror processing if no errors mount option is specified.

• With the “errors=remount-ro” mount option, the change of the file accesspermissions to read-only applies to all files. The file system is remountedread-only.

• Access permission and file size changes due to the device transitioning zonesto the offline condition are permanent. Remounting or reformatting the devicewith mkfs.zonefs (mkzonefs) will not change back offline zone files to a goodstate.

• File access permission changes to read-only due to the device transitioningzones to the read-only condition are permanent. Remounting or reformattingthe device will not re-enable file write access.

• File access permission changes implied by the remount-ro, zone-ro andzone-offline mount options are temporary for zones in a good condition.Unmounting and remounting the file system will restore the previous default(format time values) access rights to the files affected.

• The repair mount option triggers only the minimal set of I/O error recoveryactions, that is, file size fixes for zones in a good condition. Zonesindicated as being read-only or offline by the device still imply changes tothe zone file access permissions as noted in the table above.

Mount options¶

zonefs defines several mount options:* errors=<behavior>* explicit-open

Runtime sysfs attributes¶

zonefs defines several sysfs attributes for mounted devices. All attributesare user readable and can be found in the directory /sys/fs/zonefs/<dev>/,where <dev> is the name of the mounted zoned block device.

The attributes defined are as follows.

• max_wro_seq_files: This attribute reports the maximum number ofsequential zone files that can be open for writing. This number correspondsto the maximum number of explicitly or implicitly open zones that the devicesupports. A value of 0 means that the device has no limit and that any zone(any file) can be open for writing and written at any time, regardless of thestate of other zones. When theexplicit-open mount option is used, zonefswill fail any open() system call requesting to open a sequential zone file forwriting when the number of sequential zone files already open for writing hasreached themax_wro_seq_files limit.

• nr_wro_seq_files: This attribute reports the current number of sequentialzone files open for writing. When the “explicit-open” mount option is used,this number can never exceedmax_wro_seq_files. If theexplicit-openmount option is not used, the reported number can be greater thanmax_wro_seq_files. In such case, it is the responsibility of theapplication to not write simultaneously more thanmax_wro_seq_filessequential zone files. Failure to do so can result in write errors.

• max_active_seq_files: This attribute reports the maximum number ofsequential zone files that are in an active state, that is, sequential zonefiles that are partially written (not empty nor full) or that have a zone thatis explicitly open (which happens only if theexplicit-open mount option isused). This number is always equal to the maximum number of active zones thatthe device supports. A value of 0 means that the mounted device has no limiton the number of sequential zone files that can be active.

• nr_active_seq_files: This attributes reports the current number ofsequential zone files that are active. Ifmax_active_seq_files is not 0,then the value ofnr_active_seq_files can never exceed the value ofnr_active_seq_files, regardless of the use of theexplicit-open mountoption.

Examples¶

The following formats a 15TB host-managed SMR HDD with 256 MB zoneswith the conventional zones aggregation feature enabled:

# mkzonefs -o aggr_cnv /dev/sdX# mount -t zonefs /dev/sdX /mnt# ls -l /mnt/total 0dr-xr-xr-x 2 root root     1 Nov 25 13:23 cnvdr-xr-xr-x 2 root root 55356 Nov 25 13:23 seq

The size of the zone files sub-directories indicate the number of filesexisting for each type of zones. In this example, there is only oneconventional zone file (all conventional zones are aggregated under a singlefile):

# ls -l /mnt/cnvtotal 137101312-rw-r----- 1 root root 140391743488 Nov 25 13:23 0

This aggregated conventional zone file can be used as a regular file:

# mkfs.ext4 /mnt/cnv/0# mount -o loop /mnt/cnv/0 /data

The “seq” sub-directory grouping files for sequential write zones has in thisexample 55356 zones:

# ls -lv /mnt/seqtotal 14511243264-rw-r----- 1 root root 0 Nov 25 13:23 0-rw-r----- 1 root root 0 Nov 25 13:23 1-rw-r----- 1 root root 0 Nov 25 13:23 2...-rw-r----- 1 root root 0 Nov 25 13:23 55354-rw-r----- 1 root root 0 Nov 25 13:23 55355

For sequential write zone files, the file size changes as data is appended atthe end of the file, similarly to any regular file system:

# dd if=/dev/zero of=/mnt/seq/0 bs=4096 count=1 conv=notrunc oflag=direct1+0 records in1+0 records out4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00044121 s, 9.3 MB/s# ls -l /mnt/seq/0-rw-r----- 1 root root 4096 Nov 25 13:23 /mnt/seq/0

The written file can be truncated to the zone size, preventing any furtherwrite operation:

# truncate -s 268435456 /mnt/seq/0# ls -l /mnt/seq/0-rw-r----- 1 root root 268435456 Nov 25 13:49 /mnt/seq/0

Truncation to 0 size allows freeing the file zone storage space and restartappend-writes to the file:

# truncate -s 0 /mnt/seq/0# ls -l /mnt/seq/0-rw-r----- 1 root root 0 Nov 25 13:49 /mnt/seq/0

Since files are statically mapped to zones on the disk, the number of blocksof a file as reported bystat() andfstat() indicates the capacity of the filezone:

# stat /mnt/seq/0File: /mnt/seq/0Size: 0             Blocks: 524288     IO Block: 4096   regular empty fileDevice: 870h/2160d  Inode: 50431       Links: 1Access: (0640/-rw-r-----)  Uid: (    0/    root)   Gid: (    0/    root)Access: 2019-11-25 13:23:57.048971997 +0900Modify: 2019-11-25 13:52:25.553805765 +0900Change: 2019-11-25 13:52:25.553805765 +0900Birth: -

The number of blocks of the file (“Blocks”) in units of 512B blocks gives themaximum file size of 524288 * 512 B = 256 MB, corresponding to the device zonecapacity in this example. Of note is that the “IO block” field alwaysindicates the minimum I/O size for writes and corresponds to the devicephysical sector size.