Repository files navigation

The composefs project combines several underlying Linux featuresto provide a very flexible mechanism to support read-onlymountable filesystem trees, stacking on top of an underlying"lower" Linux filesystem.

The key technologies composefs uses are:

• overlayfs as the kernel interface
• EROFS for a mountable metadata tree
• fs-verity (optional) from the lower filesystem

The manner in which these technologies are combined is important.First, to emphasize: composefs does not store any persistent data itself.The underlying metadata and data files must be stored in a valid"lower" Linux filesystem. Usually on most systems, this will be atraditional writable persistent Linux filesystem such asext4,xfs,btrfs etc.

The "tagline" for this project is "The reliability of disk images, the flexibility of files",and is worth explaining a bit more. Disk images have a lot of desirableproperties in contrast to other formats such as tar and zip: they'reefficiently kernel mountable and are very explicit about all detailsof their layout. There are well known tools such asdm-veritywhich can apply to disk images for robust security. However, diskimages have well known drawbacks such as commonly duplicating storagespace on disk, can be difficult to incrementally update, and aregenerally inflexible.

composefs aims to provide a similarly high level of reliability,security, and Linux kernel integration; but with theflexibility of filesfor content - avoiding doubling disk usage, worrying about partitiontables, etc.

A key aspect of the way composefs works is that it's designed tostore "data" (i.e. non-empty regular files) distinct from "metadata"(i.e. everything else).

composefs reads and writes a filesystem image which is reallyjust anEROFSwhich today is loopback mounted.

However, this EROFS filesystem tree is just metadata; the underlyingnon-empty data files can be shared in a distinct "backing store"directory. The EROFS filesystem includestrusted.overlay.redirectextended attributes which tell theoverlayfs mounthow to find the real underlying files.

The key targeted use case for composefs is versioned, immutable executablefilesystem trees (i.e. container images and bootable host systems), wheresome of these filesystems may shareparts of their storage (i.e. somefiles may be different, but not all).

Composefs ships with a mount helper that allows you to easily mountimages by passing the image filename and the base directory forthe content files like this:

mount -t composefs /path/to/image  -o basedir=/path/to/content /mnt

By storing the files content-addressed (e.g. using the hash of the content to namethe file), shared files only need to be stored once, yet can appear inmultiple mounts.

A crucial advantage of composefs in contrast to other approachesis that data files are shared in thepage cache.

This allows launching multiple container images that willreliably share memory.

Composefs also supportsfs-verityvalidation of the content files. When using this, the digest of thecontent files is stored in the image in thetrusted.overlay.metacopyextended attributes which tell overlayfs to validate thatthe content file it uses has a matching enabled fs-verity digest. Thismeans that the backing content cannot be changed in any way (bymistake or by malice) without this being detected when the file isused.

You can also use fs-verity on the image file itself, and pass theexpected fs-verity digest as a mount option, which composefs willvalidate. In this case we have full trust of both data and metadata ofthe mounted file. This solves a weakness that fs-verity has when usedon its own, in that it can only verify file data, not metadata (e.g.inode bits like permissions and ownership, but also directorystructures).

There are multiple container image systems; for those using e.g.OCIa common approach (implemented by both docker and podman for example)is to just untar each layer by itself, and then useoverlayfsto stitch them together at runtime. This is a partial inspirationfor composefs; notably this approach does ensure thatidenticallayers are shared.

However if instead we store the file content in a content-addressedfashion, and then we can generate a composefs file for each layer,continuing to mount them with a chain ofoverlayfsor wecan generate a single composefs for the final merged filesystem tree.

This allows sharing of content files between images, even if themetadata (like the timestamps or file ownership) vary between images.

Together with something likezstd:chunked thiswill speed up pulling container images and make them available forusage, without the need to even create these files if already present!

OSTree already uses a content-addressedobject store. However, normally this has to be checked out into a regular directory (using hardlinksinto the object store for regular files). This directory is thenbind-mounted as the rootfs when the system boots.

OSTree already supports enabling fs-verity on the files in the store,but nothing can protect against changes to the checkout directories. Amalicious user can add, remove or replace files there. We want to usecomposefs to avoid this.

Instead of checking out to a directory, we generate a composefs imagepointing into the object store and mount that as the root fs. We canthen enable fs-verity of the composefs image and embed the digest ofthat in the kernel commandline which specifies the rootfs. Sincecomposefs generation is reproducible, we can even verify that thecomposefs image we generated is correct by comparing its digest to onein the ostree metadata that was generated when the ostree image was built.

For more information on ostree and composefs, seethis tracking issue.

Composefs installs two main tools:

• mkcomposefs: Creates a composefs image given a directory pathname. Can also compute digests and create a content store directory.
• mount.composefs: A mount helper that supports mounting composefs images.

The mount.composefs helper allows you to mount composefs images (of both types).

The basic use is:

mount -t composefs /path/to/image.cfs -o basedir=/path/to/datafiles  /mnt

The default behaviour for fs-verity is that any image files thatspecifies an expected digest needs the backing file to match thatfs-verity digest, at least if this is supported in the kernel. Thiscan be modified with theverity andnoverity options.

Mount options:

• basedir: is the directory to use as a base when resolving relative content paths.
• verity: All image files must specify a fs-verity image.
• noverity: Don't verify fs-verity digests (useful for example if fs-verity is not supported on basedir).
• digest: A fs-verity sha256 digest that the image file must match. If set,verity_check defaults to 2.
• upperdir: Specify an upperdir for the overlayfs filesystem.
• workdir: Specify a workdir for the overlayfs filesystem.
• idmap: Specify a path to a user namespace that is used as an idmap.

There is active work on acomposefs cratewhich has both wrappers for invocations of themkcomposefs andcomposefs-info dump tooling,as well as higher level repository functionality.

The containers/storage Go library hascode wrapping mkcomposefsthat could in theory be extracted to a helper package.

• Live chat:Matrix channel
• Async forums:Github discussions

We have a dedicatedCONTRIBUTING document.