• Home
• Projects
• Contact
• skarnet.org

A new service manager for Linux distributions

The last piece of a full alternative to systemd

Service managers are a fundamental part of Linux distributions — and, more generally, of Unix systems. Unfortunately, for such a critical piece of infrastructure, the offer is lacking. There is no service manager that performs all the duties users can expect from it while being flexible enough to address multiple use cases, including dynamic network management, usability in containers, and usability inmusl-based distributions.

I am working on a service manager that aims to solve the issues in that space. In particular, it aims to be a suitable replacement forOpenRC, which currently powers the Alpine Linux distribution, but which does not provide all the features that Alpine needs. Due to its requirements for small and clean software, and its increasing prevalence in container images, Alpine is a typical case where this new service manager would be useful; but I expect the benefits of the project to extend well beyond Alpine.

This is a long-term endeavour that requires a precise understanding of the specs, a certain level of skill and experience in software design, and copious amounts of coding time; I don't think it would necessarily benefit from a larger team, at least on the coding side, because it's still at a level of complexity that can be fully understood by one person — although, as always, I value external inputs such as technical discussions and user experience returnsimmensely. But it cannot be completed in any reasonable time frame if I am only working on it in my free time. For these reasons, I am looking for funding, in order to be able to work on it full-time; I believe the benefits to corporations, especially ones that use Alpine Linux or similar container-suitable distributions, will far outweigh the expense. This document aims to explain the details of the project.

Table of contents

• Table of contents
• Definitions
• The ideal service manager
• The state of the art
• A comparison table
• The plan
• The developer

Definitions

Not much formalization has been done around terminology such asservice manager andinit system, so these expressions are often used without a clear understanding of their semantics. Let's define these terms first.

Aninit system is the set of software components that is launched by the kernel at boot time, and whose job is to bring the machine to its fully operational state, with all its configuration done and all its services running.

As explained inthis video, aninit system is made of 4 parts:

1. the/sbin/init program
2. thepid 1 program
3. aprocess supervisor system
4. aservice manager.

These parts are conceptually distinct, but should be able to work together. For instance, without going into details, in order to offerstrong supervision, theprocess supervisor system needs to be tied to thepid 1 program.

Most of the time, several of these parts are implemented in the same software suite, which is why we generally talk aboutinit systems.sysvinit, GNU/Linux's traditional init system, is/sbin/init andpid 1 at the same time, and also includes a very rudimentaryprocess supervisor. (But it does not include aservice manager, which is why it is often paired withsysv-rc orOpenRC.)systemd,Upstart and MacOS'slaunchd are "complete" init systems, implementing all 4 parts.

/sbin/init andpid 1 are focused on the early boot of machine and are out of scope of this document.Process supervisor systems are in charge of managing long-lived processess (daemons): making sure they are always started with the exact same environment, restarting them when they die under certain conditions, redirecting their log output to the proper places, providing an interface to send them signals without needing to know their pid, etc. They are also, strictly speaking, out of scope for this document, although we will have to mention them because they need to closely interact with theservice manager.

Theservice manager is the component that brings the machine from the state where nothing is running yet, and nothing is configured, to the state where all the filesystems are mounted, the network interfaces are up, everything that needs to be initialized has been initialized, and all the services configured by the administrator are up and running. Conversely, it must also handlestate changes when required by the admin: stop a service, start another, reconfigure another. In particular, it must be able to reliably stop all the services in an orderly fashion when the administrator requires a shutdown.

The ideal service manager

A good service manager must follow a certain number of constraints that apply to its design, its implementation, or the features it provides.

• The service manager issystem infrastructure software, meaning that it is run at a very low level, and the whole machine operation relies on it. This means that:
  • It must beextremely reliable. This is achieved by having clear and complete semantics, doing the minimum amount of operations, having as few run-time dependencies as possible, and making sure the code remains fully understandable by the dev team at all times and never grows out of control.
  • It must beas lightweight as possible. Resources are meant to be used by applications, not system software; heavyweight system software is unusable on old computers and embedded devices, and wasteful in VMs and containers.
  • It should beeasy to bootstrap. Since it is a critical piece of a distribution's software offer, and one of the first packages to be built when porting a distribution to another architecture or another platform, the service manager should have as few build-time dependencies as possible. Building and installing it should not be an hours-long chore.
• The service manager must alwaysstart and stop services in the same environment, no matter whether it's invoked at boot time or by an administrator command.
• The service manager musttie-in with a process supervision system to make daemon management and monitoring easier.
• The service manager shouldguarantee system bootability. It should provide a mechanism that vets changes to a boot-time set of servicesbefore the reboot; a mechanism that says either "change accepted; if you reboot now, the machine will not get stuck in an unrecoverable state, you will get to the state that you want" or "change invalid, rejected".
• The service manager must beparallel: it must be able to start independent services at the same time. That means it needs to understand and maintain a set of dependencies between all the services it knows about. Traditionally, this has not received much attention, because proper dependency management and parallelization is difficult; and historical service managers have just started services sequentially, following a topologically sorted list. However, there are significant gains to be had with a really parallel service manager, especially in the presence of heavy services that take a long time to start and needlessly monopolize the CPU or needlessly idle.
• The service manager must supportreadiness notification from daemons it starts. It ensures that services are only started once their dependencies not only have been spawned, but are alsoready (e.g. make sure that a daemon has actually created, and is listening to, a socket, before running a client that connects to it).
• The service manager mustinterface with external events, to allow users to define services that dynamically start and stop when certain conditions are met, without needing to encode every possible condition in the service manager configuration. A typical use of this is network daemons listening on a specific interface: they must be started when the interface comes up, and stopped when the interface goes down. Managing the state of network interfaces in a generic way is too complex a problem to be handled in a service manager, and is best delegated to anetwork manager written for this explicit purpose; and when the network manager detects interface state changes, it sends events to the service manager, which reacts accordingly.
• The service manager should supportdeclarative service files, which experience has shown are more intuitive and less error-prone for users. Writing script snippets is unavoidable, but those snippets should be as small as possible, and most common service functionality (setting the user/group a daemon will run as, setting its environment, etc.) should be made available via a declarative syntax that will minimize errors.
• The service manager should beusable in containers as well as in virtual machines and real iron. Increasingly, entire systems with entire service hierarchies are run in containers, and having tools to start container contents in an orderly fashion would be an improvement to the quality of numerous container images. This is especially true for the Alpine distribution, which often serves as a base for container images.

The state of the art

The dichotomy

Over the years, I have done a lot of research on init systems and service managers; they usually fall into one of two categories.

• Traditional service starters. They're morestarters thanmanagers, because they don't have a fully fleshed out dependency model — they're only used to start services at boot time and stop them at shutdown time, with the occasional extra start. All of them are sequential and favor simplicity; when parallelism is added, it's often as an afterthought and it greatly increases code complexity. They are generally not designed to work with a supervision system: if a daemon gets killed, they will not react.
• Integrated init systems. Those were written as attempts to cover the flaws of traditional service starters, and provide all the dependency management, parallelism, integration with supervision, and other features that are necessary to implement a complete init system. Unfortunately, they're tightly integrated (as opposed to modular), complex beasts, and embed a lot of policy within their mechanism, which manifests as being difficult to operate with other software and wanting to overencompass all the administration tasks on a machine rather than follow the "one job, one tool" philosophy.

Typical examples of traditional service starters on Linux includesysv-rc andOpenRC.

The prime example of an integrated init system, on Linux, is obviouslysystemd; there are others, less invasive and less prevalent, likeUpstart (a Ubuntu precursor to systemd). On MacOS,launchd falls into that category.

None of these options are satisfactory. Distributions make do with what they have: some embrace systemd — and in doing so abandon all hope of ever getting rid of it, or of becoming a suitable option for systems with serious requirements of low resource usage or code auditability; others resist the pull, and scrape by, using OpenRC or similar software, and working around the lack of features. Some, likeVoid Linux, have even elected not to use a service manager at all. (They use a process supervision system, which is good, but then have to script their entire init procedure, which is one of their major pain points.)

s6-rc

I have been interested in the init system space for a long time, and have written my ownprocess supervisor,s6, and a companion package,s6-linux-init, that makes it work as a/sbin/init andpid 1 as well. In order for s6 to be a complete init system, aservice manager was needed; this was going to be a much more complex endeavour, so I wrote a small one,s6-rc, as a prototype — to see how far it would go, and to get an idea of the kind of problems that emerge when designing a service manager.

For what it's worth, I consider the attempt a success. s6-rc's current version is 0.5.2.1, and it's a service manager engine that works very well in a lot of cases. It is smaller than OpenRC while having more features, and faster than systemd while remaining strict with its launching order. It was even adopted by some distributions, namelyObarun andArtix.

However, it is only a prototype, and lacks crucial features before it can satisfactorily compete with established players: in particular, it is still lacking dynamic event management, which makes it unsuitable for integration with a network manager, and it lacks a friendly user interface based on declarative service files, which several distributions have clearly signalled they require before considering s6-rc for adoption as their chosen service manager.

So, as of today,there is no service manager that fulfills all the needs of a distribution. It is a "pick your poison" situation.

A comparison table

To visualize that situation, and the path forward, here is a comparison table, showing how OpenRC (as an example of traditional service launcher), systemd (as an example of integrated init system), and s6-rc (the current prototype version) measure up against each other. We list all the requirements of an ideal service manager, which we detailedabove.

The plan

As illustrated, despite being a 3-month 1-person project, the s6-rc prototype fares better than both OpenRC and systemd on the requirements chart. It is the closest we have to our ideal. Unfortunately, it is still not suitable as a flexible, user-friendly service manager that could be used in any distribution; for that, those two red squares on the bottom right need to turn green. And that represents alarge amount of work.

Putting in the work, though, would actually lead to a good piece of software, because it's just a question of implementation. It isstill easier to achieve all the goals of an ideal service manager by using the s6-rc prototype as a basis for the final software than it is to tweak and modify OpenRC or systemd, because the flaws in these aredesign flaws, and cannot be fixed by adding more code or even refactoring.

So, my proposition is to work on a real, production-quality implementation of s6-rc, keeping the sound design but adding the features it needs:

• Aclient-server model where the server, a long-lived process, maintains both thecurrent state and thewanted state of the machine, performing the needed transitions as they become available.
• Dynamic external event management by the server, with a specialized client program that can be invoked by external entities, such as a network manager, to notify the server with an event.
• User-friendly administration commands that hide internal details and follow the model ofopenrc,service orsystemctl. Users will get a command interface they are familiar with.
• Declarative service files. The model ofunit files, where users can just writekey=value lines instead of scripts, has been a resounding success for systemd — and it is also good end-user design (provided the parsercorrectly validates the files). And it can be done as a separate user interface layer, running on top of the more minimalistic engine.

The goal is known, the path is clear, and the milestones are laid out. But it is aserious project, and I would like to see it completed before I die — which is not likely to happen if I can only spend on it the meager amounts of free time I can devote to writing free software while needing to sustain myself writingother software — likely much less pleasant, and much less free.

Which is why I am looking for funding, in order to be able to work on this full-time. My estimate is that it needs about one year of work, and should givereally good returns on investment. Among the probable benefits:

• Faster, more reliable Alpine boots, and faster and more reliable worldwide boots when other distributions start adopting s6-rc
• Clean service ordering in containers
• Easier administration of network equipment with services that depend on dynamic interface states
• Easy integration with configuration managers such as Chef or Puppet
• Ultimately, the emergence of an efficient, reliable and modular software ecosystem as a real alternative to systemd

But the real benefit, of course, is that you will be able to say that all of this happenedthanks to you.

The developer

My name is Laurent Bercot, also known asSka orskarnet in some circles. I have a computer science degree, and have been working with Linux, and participating in the Free Software movement, since 1999.

I have studied supervision systems and low-level Linux userspace for two decades. Init systems are my domain of expertise; I honestly think I am one of the best candidates, if not the best, to write a service manager that can address all the current needs for good and finally provide a credible alternative, feature-wise, to integrated init systems.

I have written C code for about as long as well. My users can vouch that I am in the habit of delivering quality code, with extensive documentation. Free software I have written is availableon the skarnet.org site; the centerpiece is thes6 process supervision system, which manages a significant fraction of all Docker containers in use today.

I am involved in several Free Software communities that have formed around small, compact software, such as themusl libc, theAlpine Linux distribution, or thebusybox project.

I have worked as a freelance C/Linux expert software designer and developer since 2015, and would love nothing more than doing my job on projects I care about most and am at my best with.