Repository files navigation

---

A configurable bill-of-materials (CBOM) system for declaratively composing and upgrading/downgradingyour hardware-specific embedded Linux operating systems.

Note: this is still an experimental prototype in the sense that Forklift's architectural design andcommand-line interface may undergo significant backwards-incompatible simplifications. WhileForklift is already used in production as a lower-level implementation detail of thePlanktoScope OSand is exposed to advanced users with a command-line interface for customization of PlanktoScope OS,any other use of Forklift is not yet officially documented or supported.

Forklift is a software deployment and configuration system providing a simpler, easier, and safermechanism for provisioning, updating, reconfiguring, recomposing, and extending browser apps,network services, and system files on single-computer systems (such as a Raspberry Pi or a laptop).The design of Forklift makes tradeoffs specific to the ways in which many open-source scientificinstruments need to be deployed and operated (e.g. intermittent internet access, independentadministration by individual people, decentralized management & customization). For a quickthree-minute overview of the motivation for Forklift and how it can be used, please refer tothis demo video:https://www.youtube.com/watch?v=4lHh_NDlFKA

For open-hardware project developers, Forklift enables Linux-based devices andappliances (especially those based on theRaspberry Pi) to have the "interesting" parts of their software programs and configurations to bespecified, composed, deployed, and reversibly upgraded/downgraded as version-controlled changes to aconfigurable bill-of-materials (CBOM) ofsoftware modules. ThePlanktoScope, an open-source microscope forquantitative imaging of plankton, uses Forklift as low-level infrastructure for softwarereleases, deployment, and extensibility in thePlanktoScope OS, ahardware-specific operating system based on the Raspberry Pi OS; and Forklift was designedspecifically to solve the OS/software maintenance, customization, and operations challengesexperienced in the PlanktoScope project.

For end-users operating open-source instruments with application services (e.g. network APIs orbrowser-based interfaces) and/or system services (for e.g. data backups/transfer, hardware support,computer networking, monitoring, etc.), Forklift aims to provide an experience for installing,updating, and uninstalling software and OS customizations similar what is achieved by app stores formobile phones - but with more user control. Forklift also simplifies the process of keeping softwareup-to-date and the process of rolling software back to older versions if needed; this reduces theneed to (for example) re-flash a Raspberry Pi's SD card with a new OS image just to update theapplication software running on the instrument while ensuring the validity of the resulting state ofthe system.

For indie software developers and sysadmins familiar with DevOps and cloud-native patterns, Forkliftis just a GitOps-inspired no-orchestrator system which is small and simple enough to work beyond thecloud - using Docker Compose to avoid the unnecessary architectural complexity and overhead of evenminimal Kubernetes distributions like k0s for systems where a container workload orchestrator isunnecessary; and bundling app deployment with the deployment of system files, executables, andsystemd units from configuration files version-controlled in Git repositories. Thus, Forklift allowshassle-free management of software configurations on one or more machines with only occasionalinternet access (or, in the future, no internet access at all!) and no specialized ops or platformteam.

For people who are Very Into Linux, Forklift is a way to bring a partial subset of the architecturalbenefits of atomic/"immutable" OSes (such asChromeOS,Fedora Atomic Desktops,Universal Blue,Fedora CoreOS/IoT,NixOS,Flatcar Container Linux,GNOME OS,etc.) intomore traditional non-atomic Linux distros (such as Raspberry Pi OS, Debian, etc.), by enablingatomic, composable, and reprovisionable changes for a sufficiently-interesting subset of the OS. Thespecific scope of what Forklift manages depends on how Forklift gets integrated with the OS, butForklift is intended to enable sysadmins to incrementally reduce reliance on system packages, and toincrementally reduce any responsibilities of the base OS beyond acting as a container host withhardware drivers (e.g. for Wi-Fi).

To learn more about the design of Forklift, please refer toForklift's design document.

Currently, design and development of Forklift prioritizes the needs of the PlanktoScope communityand the PlanktoScope project'svalues for its infrastructural software.Thus, for now decisions will be made by the PlanktoScope software's lead maintainer (currently@ethanjli) as a "benevolent dictator"/"mad scientist" in consultationwith the PlanktoScope community in online meetings and discussion channels open to the entirecommunity. This will remain the governance model of Forklift while it's still an experimental tooland still only used for the standard/default version of the PlanktoScope's operating system, inorder to ensure that Forklift develops in a cohesive way consistent with the values mentioned aboveand with intended use of Forklift for the PlanktoScope community. Once Forklift starts being usedfor delivering/maintaining variants of the PlanktoScope's operating system, for integration ofthird-party apps from the PlanktoScope community, or for software configuration by ordinary users,then governance of thegithub.com/PlanktoScope/forkliftrepository will transition from benevolent dictatorship into the PlanktoScope project'sconsensus-based proposals process. In the meantime, weencourage anyone who is interested in using/adapting Forklift to fork this repository forexperimentation and/or tocreate new discussion posts in this repository,though we can't make any guarantees about the stability of any APIs or about our capacity to addressany external code contributions or feature requests.

If other projects beyond the PlanktoScope community decide to use Forklift as part of their softwaredelivery/deployment infrastructure, we can talk about expanding governance of Forklift beyond thePlanktoScope community - feel free to start a discussion in this repository's GitHub Discussionsforum.

For a more guided demo showing off Forklift's capabilities, you can try out Forklift on a RaspberryPi or in a VM:

• Raspberry Pi:github.com/ethanjli/rpi-forklift-demo provides anSD card image of the Raspberry Pi OS with Forklift already integrated into it, so that Forkliftcan be used to provision/update/deprovision Docker Compose apps as well as system config files,scripts, and binaries in the/etc and/usr directories.
• VM:github.com/ethanjli/ublue-forklift-sysext-demoprovides an ISO installer ofBluefin (i.e. a custom image ofFedora Silverblue) with Forklift alreadyintegrated into it, so that Forklift can be used to provision/update/deprovision Docker Composeapps as well assystemd system/configuration extensions.

If you don't want to try out Forklift in either of the above ways, the below instructions will helpyou to set up Forklift on your own system to Docker Compose apps on your system:

First, you will need to download theforklift tool, which is available as a single self-containedexecutable file. You should visit this repository'sreleases page and download an archivefile for your platform and CPU architecture; for example, on a Raspberry Pi 4, you should probablydownload the archive namedforklift_{version number}_linux_arm64.tar.gz (where the version numbershould be substituted). You can extract theforklift binary from the archive using a command like:

tar -xzf forklift_{version number}_{os}_{cpu architecture}.tar.gz forklift

Then you may need to move theforklift binary into a directory in your system path, or you canjust run theforklift binary in your current directory (in which case you should replaceforklift with./forklift in the example commands listed below), or you can just run theforklift binary by its absolute/relative path (in which case you should replaceforklift withthe absolute/relative path of the binary in the example commands listed below).

To deploy a particular version of a published pallet to your computer, you will need to clone apallet and stage it to be applied, and then you will need to apply the staged pallet. Pallets areidentified by the path of their Git repository and a version query (which can be a Git branch name,a Git tag name, or an abbreviated or full Git commit hash). For example, the most recent commit onthemain branch of thegithub.com/forklift-run/pallet-example-minimalcan be identified asgithub.com/forklift-run/pallet-example-minimal@main - this is what we use inthe example commands in this section.

If you are running Docker inrootless mode oryour user is inthedocker group:

• If you want to apply the pallet immediately, you can runforklift pallet switch --apply withyour specified pallet. For example:

  forklift pallet switch --apply github.com/forklift-run/pallet-example-minimal@main

• If you want to apply the pallet later, you can first stage the pallet, and then later apply thestaged pallet using a separateforklift stage apply command. For example:

  # Run now:forklift pallet switch github.com/forklift-run/pallet-example-minimal@main# Run when you want to apply the pallet:forklift stage apply

If you aren't running Docker in rootless mode and your user isn't in adocker group, we recommenda slightly different set of commands:

• If you want to apply the pallet immediately, you can runforklift pallet switch --cache-img=false as a regular user, and then runforklift stage apply asroot; the--cache-img=false flag preventsforklift pallet switch from attempting to make Dockerpre-download all container images required by the pallet, as doing so would require root permissionsfor Forklift to talk to Docker.For example:

  forklift pallet switch --cache-img=false github.com/forklift-run/pallet-example-minimal@mainsudo -E forklift stage apply

• If you want to apply the pallet later, you can first stage the pallet and pre-download allcontainer images required by the pallet, and then later apply the staged pallet using a separateforklift stage apply command (which you can then run even when you don't have internet access).For example:

  # Run now:forklift pallet switch --cache-img=false github.com/forklift-run/pallet-example-minimal@mainsudo -E forklift stage cache-img# Run when you want to apply the pallet:sudo -E forklift stage apply

Note: in the above commands, you can replaceforklift pallet withforklift plt if you want totype fewer characters when running those commands.

First, you will need to make/download a pallet somewhere on your local file system. For example, youcan usegit to clone the latest unstable version (on themain branch) of thegithub.com/forklift-run/pallet-example-minimal pallet using the command:

git clone https://github.com/forklift-run/pallet-example-minimal

Then you will need to download/install theforklift tool (see instructions in the"Download/install forklift" section above). Once you haveforklift,you can run commands using thedev plt subcommand; ifforklift is in your system path, you cansimply run commands within the directory containing your development pallet, or any subdirectory ofit. For example, if your development pallet is at/home/pi/dev/pallet-example-minimal, you can runthe following commands to see some information about your development pallet:

cd /home/pi/dev/pallet-example-minimalforklift dev plt show

You can also run the command from anywhere else on your filesystem by specifying the path of yourdevelopment pallet. For example, if your forklift binary is in/home/pi, you can run any thefollowing sets of commands to see the same information about your development pallet:

cd /home/pi/./forklift dev --cwd ./dev/pallet-example-minimal plt show

cd /etc//home/pi/forklift dev --cwd /home/pi/dev/pallet-example-minimal plt show

You can also use theforklift dev plt require-repo command to require additional Forkliftrepositories for use in your development pallet, and/or to change the versions of Forkliftrepositories already required by your development pallet.

You can also run commands likeforklift dev plt cache-all andforklift dev plt stage --cache-img=false (with appropriate values in the--cwd flag ifnecessary) to download the Forklift repositories specified by your development pallet into yourlocal cache and stage your development pallet to be applied withsudo -E forklift stage apply.This is useful if, for example, you want to make some experimental changes to your developmentpallet and test them on your local machine before committing and pushing those changes onto GitHub.

Finally, you can run theforklift dev plt check command to check the pallet for any problems, suchas violations of resource constraints between package deployments.

You can also override cached repos with repos from your filesystem by specifying one or moredirectories containing one or more repos; then the repos in those directories will be used insteadof the respective repos from the cache, regardless of repo version. For example:

cd /home/pi//home/pi/forklift dev --cwd /home/pi/dev/pallet-example-minimal plt --repos /home/pi/forklift/dev/device-pkgs check

The following projects solve related problems with containers for application software, though theymake different trade-offs compared to Forklift:

• poco enables Git-based management of Docker Compose projectsand collections (catalogs) of projects and repositories and provides some similarfunctionalities to forklift.
• Terraform (an earlyinspiration for this project) has a Docker Provider which enables declarative management of Dockerhosts and Docker Swarms from a Terraform configuration.
• swarm-pack (an early inspiration for this project)uses collections of packages from user-specified Git repositories and enables templatedconfiguration of Docker Compose files, with imperative deployments of packages to a Docker Swarm.
• SwarmManagement uses a single YAML file fordeclarative configuration of an entire Docker Swarm.
• PodmanQuadlets enablemanagement of containers, volumes, and networks using declarative systemd units
• FetchIt enables Git-based management of containers inPodman.
• Projects developingGitOps tools such as ArgoCD, Flux, etc., storecontainer environment configurations as Git repositories but are generally designed forKubernetes.

The following projects solve related problems in composing and updating the base OS, though theymake different trade-offs compared to Forklift (especially because of the PlanktoScope project'slegacy software):

• Rugix is a suite of low-level tools for building and updatingOS images, including support for the Raspberry Pi's official A/B-styletryboot mechanism forOS updates.
• systemd-sysext and systemd-confextprovide a more structured/opinionated way (compared to Forklift) to atomically overlay systemfiles onto the base OS, but they not opinionated about (i.e. don't specify a way for)distributing/provisioning published sysext/confext images, and they are not available on RaspberryPi OS 11 (bullseye), which until recently was a legacy-systems burden for the PlanktoScope OS.Forklift can be used together with systemd-sysext/confext, as a way to download sysext/confextimages, constrain which images can be deployed together/separately, and manage which extensionimages are available to systemd - seethis demo.
• systemd'sPortable Services pattern andportablectltool provide a more structured/constrained/sandboxed way (compared to Forklift) to atomically addsystem services, but they are only designed for self-contained system services. Forklift canprobably used together with systemd Portable Services, as a way to deploy published images onto anOS.
• NixOS enables declarative specification of the entire configuration of thesystem with atomic updates, but it requires learning a bespoke functional programming language andisrather complex. Forklift is inspired bysome of NixOS's design decisions, and it attempts to provide a few of NixOS's benefits but in amuch more approachable, simple, and backwards-compatible system design by aggressively sacrificingalmost all of NixOS's theoretical rigor in favor of practical considerations.
• Thebootc project enables the entire operating system to bedelivered as a bootable OCI container image, but currently it relies on bootupd, whichcurrently only works on RPM-based distros.Forklift can be used together with bootc-based systems, as a way to manage Docker Compose apps(and/or systemd sysexts/confexts) on the system.
• gokrazy enables atomic deployment of Go programs (and also of softwarecontainers!), but it has a very different architecture compared to traditional Linux distros.

The following projects solve related problems in updating (but not in composing) the base OS, makingdifferent trade-offs (especially in their need for OS maintainers or device operators to pay for orself-host a centralized management server - which is a non-starter for the PlanktoScope project)compared to Forklift:

• OSTree enables atomic updates of the base OS, butit is not supported by Raspberry Pi OS, anddelivery of the OS requires operating a server to host OSTree repositories. Forklift can be usedtogether with OSTree-based systems, as a way to provision/replace/deprovision a layer of/etcconfig files over (and independently of, or as a bypass for) OSTree's3-way merge mechanism for/etc,and as a way to manage Docker Compose apps running on the system.
• systemd-sysupdateprovides an image-based mechanism for updating the base OS, container images, and systemd-sysextand systemd-confext images hosted at HTTP/HTTPS URLs. It solves many of the same problems thatForklift attempts to solve, but with different priorities and thus different opinions.
• swupdate andRAUC bothfacilitate local A/B-style OS image updates as well as over-the-air (OTA) updates (when combinedwithhawkbit as a centralized management server, which then mustbe hosted and operated somewhere).
• Mender facilitates A/B-style OTA OS image updates forfleets of embedded Linux devices and can be retrofitted onto non-atomic OSes (such as Debian), butit requires a centralized management server (either as a paid service or self-hosted) fordistributing updates.

Other related OS-level composition+update projects can be found atgithub.com/castrojo/awesome-immutable; otherrelated embedded OS update projects can be found athttps://mkrak.org/wp-content/uploads/2018/04/FOSS-NORTH_2018_Software_Updates.pdf.

We have chosen the following licenses in order to give away our work for free, so that you canfreely use it for whatever purposes you have, with minimal restrictions, while still protecting ourdisclaimer that this work is provided without any warranties at all. If you're using this project,or if you have questions about the licenses, we'd love to hear from you - please start a newdiscussion thread in the "Discussions" tab of this repository on Github or email us atlietk12@gmail.com .

Except where otherwise indicated, source code provided here is covered by the following information:

Copyright Ethan Li and Forklift project contributors

SPDX-License-Identifier:Apache-2.0 OR BlueOak-1.0.0

Software files in this repository are released under theApache 2.0 License and theBlue Oak Model License 1.0.0;you can use the source code provided here either under the Apache License or under theBlue Oak Model License, and you get to decide which license you will agree to.We are making the software available under the Apache license because it'sOSI-approved,but we like the Blue Oak Model License more because it's easier to read and understand.Please read and understand the licenses for the specific language governing permissions andlimitations.

Except where otherwise indicated in this repository, any other files (such as images, media, data,and textual documentation) provided here not already covered by the software license (describedabove) are instead covered by the following information:

Copyright Ethan Li and Forklift project contributors

SPDX-License-Identifier:CC-BY-4.0

Files in this project are released under theCreative Commons Attribution 4.0 International License.Please read and understand the license for the specific language governing permissions andlimitations.