Repository files navigation

• Quickstart
• Compatibility with v2
• Goals
• Features
• The Docker Way?
• Init stages
• Installation
• Usage
  • UsingCMD
  • Writing a service script
  • Setting the exit code of the container to the exit code of your main service
  • Fixing ownership and permissions
  • Executing initialization and finalization tasks
  • Writing an optional finish script
  • Logging
  • Dropping privileges
  • Read-only Root Filesystem
  • Container environment
  • Customizing s6-overlay's behaviour
  • syslog
• Performance
• Verifying Downloads
• Notes
• Releases
  • Which architecture to use depending on your TARGETARCH
• Contributing
• Building the overlay yourself
• Upgrade notes

s6-overlay is an easy-to-install (just extract a tarball or two!) set of scripts and utilitiesallowing you to use existing Docker images while usings6as a pid 1 for your container and process supervisor for your services.

Build the following Dockerfile and try it out:

# Use your favorite imageFROM ubuntuARG S6_OVERLAY_VERSION=3.2.1.0RUN apt-get update && apt-get install -y nginx xz-utilsRUN echo "daemon off;" >> /etc/nginx/nginx.confCMD ["/usr/sbin/nginx"]ADD https://github.com/just-containers/s6-overlay/releases/download/v${S6_OVERLAY_VERSION}/s6-overlay-noarch.tar.xz /tmpRUN tar -C / -Jxpf /tmp/s6-overlay-noarch.tar.xzADD https://github.com/just-containers/s6-overlay/releases/download/v${S6_OVERLAY_VERSION}/s6-overlay-x86_64.tar.xz /tmpRUN tar -C / -Jxpf /tmp/s6-overlay-x86_64.tar.xzENTRYPOINT ["/init"]

docker-host $ docker build -t demo .docker-host $ docker run --name s6demo -d -p 80:80 demodocker-host $ docker top s6demo acxfPID                 TTY                 STAT                TIME                COMMAND11735               ?                   Ss                  0:00                \_ s6-svscan11772               ?                   S                   0:00                \_ s6-supervise11773               ?                   Ss                  0:00                | \_ s6-linux-init-s11771               ?                   Ss                  0:00                \_ rc.init11812               ?                   S                   0:00                | \_ nginx11814               ?                   S                   0:00                | \_ nginx11816               ?                   S                   0:00                | \_ nginx11813               ?                   S                   0:00                | \_ nginx11815               ?                   S                   0:00                | \_ nginx11779               ?                   S                   0:00                \_ s6-supervise11785               ?                   Ss                  0:00                | \_ s6-ipcserverd11778               ?                   S                   0:00                \_ s6-supervisedocker-host $ curl --head http://127.0.0.1/HTTP/1.1 200 OKServer: nginx/1.18.0 (Ubuntu)Date: Mon, 17 Jan 2022 13:33:58 GMTContent-Type: text/htmlContent-Length: 612Last-Modified: Mon, 17 Jan 2022 13:32:11 GMTConnection: keep-aliveETag: "61e56fdb-264"Accept-Ranges: bytes

If you're migrating from a previous version of s6-overlay (v2) to thenew version (v3), you may need to make some changes to your servicesor the way you use s6-overlay in order for everything to work smoothly.This document tries to be accurate on how v3 works, but we have aseparate pagelisting the main differences, and things you're likely to notice. Pleaseread it if you're in this situation!

The project has the following goals:

• Be usable on top ofany Docker image
• Make it easy to create new images, that will operate like any other images
• Provide users with a turnkey s6 installation that will give them a stablepid 1, a fast and orderly init sequence and shutdown sequence, and the powerof process supervision and automatically rotated logs.

• A simple init process which allows the end-user to execute tasks like initialization (cont-init.d),finalization (cont-finish.d) and their own services with dependencies between them
• The s6-overlay provides properPID 1 functionality
  • You'll never have zombie processes hanging around in your container, they will be properly cleaned up.
• Multiple processes in a single container
• Able to operate in "The Docker Way"
• Usable with all base images - Ubuntu, CentOS, Fedora, Alpine, Busybox...
• Distributed as a small number of .tar.xz files depending on what exact functionality you need - to keep your image's number of layers small.
• A whole set of utilities included ins6 ands6-portable-utils. They include handy and composable utilities which make our lives much, much easier.
• Log rotating out-of-the-box throughlogutil-service which usess6-log under the hood.
• Some support for Docker'sUSER directive, to run your whole process tree as a specific user. Not compatible with all features, details in thenotes section.

One of the oft-repeated Docker mantras is "one process per container", but we disagree.There's nothing inherentlybad about running multiple processes in a container.The more abstract "onething per container" is our policy - a container should do one thing,such as "run a chat service" or "run gitlab." This may involve multiple processes, which is fine.

The other reason image authors shy away from process supervisors is they believe a process supervisormust restart failed services, meaning the Docker container will never die.

This does effectively break the Docker ecosystem - most images run one process that willexit when there's an error. By exiting on error, you allow the system administrator tohandle failures however they prefer. If your image will never exit, you now need somealternative method of error recovery and failure notification.

Our policy is that if "the thing" fails, then the container should fail, too.We do this by determining which processes can restart, and which should bring downthe container. For example, ifcron orsyslog fails, your container can mostlikely restart it without any ill effects, but ifejabberd fails, the containershould exit so the system administrator can take action.

Our interpretation of "The Docker Way" is thus:

• Containers should do one thing
• Containers should stop when that thing stops

and our init system is designed to do exactly that. Your images will behave likeother Docker images and fit in with the existing ecosystem of images.

See "Writing an optional finish script" under theUsage section for details on stopping "the thing."

Our overlay init is a properly customized one to run appropriately in containerized environments.This section briefly explains how stages work but if you want to know how a complete init systemshould work, you can read this article:How to run s6-svscan as process 1

1. stage 1: Its purpose is to set up the image to execute the supervision tree whichwill handle all the auxiliary services, and to launch stage 2. Stage 1 is where all theblack magic happens, all the container setup details that we handle for you so that you don'thave to care about them.
2. stage 2: This is where most of the end-user provided files are meant to be executed:
   1. Execute legacy oneshot user scripts contained in/etc/cont-init.d.
   2. Run user s6-rc services declared in/etc/s6-overlay/s6-rc.d, following dependencies
   3. Copy legacy longrun user services (/etc/services.d) to a temporary directory and have s6 start (and supervise) them.
3. stage 3: This is the shutdown stage. When the container is supposed to exit, it will:
   1. Send a TERM signal to all legacy longrun services and, if required, wait for them to exit.
   2. Bring down user s6-rc services in an orderly fashion.
   3. Run any finalization scripts contained in/etc/cont-finish.d.
   4. Send all remaining processes aTERM signal. There should not be any remaining processes anyway.
   5. Sleep for a small grace time, to allow stray processes to exit cleanly.
   6. Send all processes aKILL signal. Then the container exits.

s6-overlay comes as a set of tarballs that you can extract onto your image.The tarballs you need are a function of the image you use; most people willneed the first two, and the other ones are extras you can use at yourconvenience.

1. s6-overlay-noarch.tar.xz: this tarball contains the scriptsimplementing the overlay. We call it "noarch" because it is architecture-independent: it only contains scripts and other text files. Everyone whowants to run s6-overlay needs to extract this tarball.
2. s6-overlay-x86_64.tar.xz: replacex86_64 with yoursystem's architecture. This tarball contains all the necessary binariesfrom the s6 ecosystem, all linked statically and out of the way ofyour image's binaries. Unless you know for sure that your image alreadycomes with all the packages providing the binaries used in the overlay,you need to extract this tarball.
3. s6-overlay-symlinks-noarch.tar.xz: this tarball containssymlinks to the s6-overlay scripts so they are accessible via/usr/bin.It is normally not needed, all the scripts are accessible via the PATHenvironment variable, but if you have old user scripts containingshebangs such as#!/usr/bin/with-contenv, installing these symlinkswill make them work.
4. s6-overlay-symlinks-arch.tar.xz: this tarball containssymlinks to the binaries from the s6 ecosystem provided by the secondtarball, to make them accessible via/usr/bin. It is normally notneeded, but if you have old user scripts containing shebangs such as#!/usr/bin/execlineb, installing these symlinks will make them work.
5. syslogd-overlay-noarch.tar.xz: this tarball containsdefinitions for asyslogd service. If you are running daemons thatcannot log to stderr to take advantage of the s6 logging infrastructure,but hardcode the use of the oldsyslog() mechanism, you can extractthis tarball, and your container will run a lightweight emulation of asyslogd daemon, so your syslog logs will be caught and stored to disk.

To install those tarballs, add lines to your Dockerfile that correspondto the functionality you want to install. For instance, most people woulduse the following:

ADD https://github.com/just-containers/s6-overlay/releases/download/v${S6_OVERLAY_VERSION}/s6-overlay-noarch.tar.xz /tmpRUN tar -C / -Jxpf /tmp/s6-overlay-noarch.tar.xzADD https://github.com/just-containers/s6-overlay/releases/download/v${S6_OVERLAY_VERSION}/s6-overlay-x86_64.tar.xz /tmpRUN tar -C / -Jxpf /tmp/s6-overlay-x86_64.tar.xz

Make sure to preserve file permissions when extracting (i.e. to use the-p option totar.)

The project is distributed as a set of standard .tar.xz files, which you extract at the root of your image.(You need the xz-utils package fortar to understand.tar.xz files; it is availablein every distribution, but not always in the default container images, so you may needtoapt install xz-utils orapk add xz, or equivalent, before you canexpand the archives.)

Afterwards, set yourENTRYPOINT to/init.

Right now, we recommend using Docker'sADD directive instead of runningwget orcurlin aRUN directive - Docker is able to handle the https URL when you useADD, whereasyour base image might not be able to use https, or might not even havewget orcurlinstalled at all.

From there, you have a couple of options:

• If you want the container to exit when your program exits: run the program as your image'sCMD.
• If you want the container to run until told to exit, and your program to be supervised by s6:write a service script for your program.

UsingCMD is a convenient way to take advantage of the overlay. YourCMD can be given atbuild time in the Dockerfile, or at run time on the command line, either way is fine. It willbe run as a normal process in the environment set up by s6; when it fails or exits, thecontainer will shut down cleanly and exit. You can run interactive programs in this manner:only the CMD will receive your interactive command, the support processes will be unimpacted.

For example:

FROM busyboxADD https://github.com/just-containers/s6-overlay/releases/download/v${S6_OVERLAY_VERSION}/s6-overlay-noarch.tar.xz /tmpRUN tar -C / -Jxpf /tmp/s6-overlay-noarch.tar.xzADD https://github.com/just-containers/s6-overlay/releases/download/v${S6_OVERLAY_VERSION}/s6-overlay-x86_64.tar.xz /tmpRUN tar -C / -Jxpf /tmp/s6-overlay-x86_64.tar.xzENTRYPOINT ["/init"]

docker-host $ docker build -t s6demo .docker-host $ docker run -ti s6demo /bin/sh/package/admin/s6-overlay/libexec/preinit: notice: /var/run is not a symlink to /run, fixing its6-rc: info: service s6rc-oneshot-runner: startings6-rc: info: service s6rc-oneshot-runner successfully starteds6-rc: info: service fix-attrs: startings6-rc: info: service fix-attrs successfully starteds6-rc: info: service legacy-cont-init: startings6-rc: info: service legacy-cont-init successfully starteds6-rc: info: service legacy-services: startings6-rc: info: service legacy-services successfully started/ # psPID   USER     TIME  COMMAND    1 root      0:00 /package/admin/s6/command/s6-svscan -d4 -- /run/service   17 root      0:00 {rc.init} /bin/sh -e /run/s6/basedir/scripts/rc.init top /bin/sh   18 root      0:00 s6-supervise s6-linux-init-shutdownd   20 root      0:00 /package/admin/s6-linux-init/command/s6-linux-init-shutdownd -c /run/s6/basedir -g 3000 -C -B   24 root      0:00 s6-supervise s6rc-fdholder   25 root      0:00 s6-supervise s6rc-oneshot-runner   31 root      0:00 /package/admin/s6/command/s6-ipcserverd -1 -- /package/admin/s6/command/s6-ipcserver-access -v0 -E -l0 -i data/rules -- /packa   58 root      0:00 /bin/sh   66 root      0:00 ps/ # exits6-rc: info: service legacy-services: stoppings6-rc: info: service legacy-services successfully stoppeds6-rc: info: service legacy-cont-init: stoppings6-rc: info: service legacy-cont-init successfully stoppeds6-rc: info: service fix-attrs: stoppings6-rc: info: service fix-attrs successfully stoppeds6-rc: info: service s6rc-oneshot-runner: stoppings6-rc: info: service s6rc-oneshot-runner successfully stoppeddocker-host $

The other way to use a container with s6-overlay is to make yourservices supervised. You can supervise any number of services;usually they're just support services for the main daemon you run asa CMD, but if that's what you want, nothing prevents you from havingan empty CMD and running your main daemon as a supervised service aswell. In that case, the daemon will be restarted by s6 whenever itexits; the container will only stop when you tell it to do so, eithervia adocker stop command, or from inside the container with the/run/s6/basedir/bin/halt command.

There are two ways of making a supervised service. The old way, whichis still supported, is to make a "pure s6" service directory. Create adirectory with the name of your service in/etc/services.d and put an executablerunfile into it; this is the file in which you'll put your long-lived process execution.For details of supervision of service directories, and how you canconfigure how s6 handles your daemon, you can take a look at theservicedir documentation.A simple example would look like this:

/etc/services.d/myapp/run:

#!/command/execlineb -Pnginx -g "daemon off;"

The new way is to make ans6-rcsource definition directory in the/etc/s6-overlay/s6-rc.d directory,and add the name of that directory to theuser bundle, i.e. create anempty file with the same name in the/etc/s6-overlay/s6-rc.d/user/contents.ddirectory. The format of asource definition directory is described inthis page. Note thatyou can definelongruns, i.e. daemons that will get supervised by s6 justlike with the/etc/services.d method, but alsooneshots, i.e. programs thatwill run once and exit. Your main service is probably alongrun, not aoneshot: you probably need a daemon to stick around.

The advantage of this new format is that it allows you to define dependenciesbetween services: ifB depends onA, thenA will start first, thenB willstart whenA is ready, and when the container is told to exit,B will stopfirst, thenA. If you have a complex architecture where various processesdepends on one another, or simply where you have to mixoneshots andlongrunsin a precise order, this may be for you.

The example above could be rewritten this way:

/etc/s6-overlay/s6-rc.d/myapp/type:

longrun

/etc/s6-overlay/s6-rc.d/myapp/run:

#!/command/execlineb -Pnginx -g "daemon off;"

/etc/s6-overlay/s6-rc.d/user/contents.d/myapp: empty file.(This addsmyapp to the set of services that s6-rc will start atcontainer boot.)

/etc/s6-overlay/s6-rc.d/myapp/dependencies.d/base: empty file.(This tells s6-rc to only startmyapp when all the base servicesare ready: it prevents race conditions.)

We encourage you to switch to the new format, but if you don't need itsbenefits, you can stick with regular service directories in/etc/services.d,it will work just as well.

If you run your main service as a CMD, you have nothing to do: when your CMDexits, or when you rundocker stop, the container will naturally exit with thesame exit code as your service. (Be aware, however, that in thedocker stopcase, your service will get a SIGTERM, in which case the exit code will entirelydepend on how your service handles it - it could trap it and exit 0, trap it andexit something else, or not trap it and let the shell exit its own code for it -normally 130.)

If you run your main service as a supervised service, however, things aredifferent, and you need to tell the container what code to exit with when yousend it adocker stop command. To do that, you need to write afinish script:

• If your service is a legacy service in/etc/services.d, you need anexecutable/etc/services.d/myapp/finish script.
• If your service is an s6-rc one, you need a/etc/s6-overlay/s6-rc.d/myapp/finish file containing your script (thefile may or may not be executable).

Thisfinish script will be run when your service exits, and will taketwo arguments:

• The first argument will be the exit code of your service, or 256 ifyour service was killed by an uncaught signal.
• The second argument is only meaningful if your service was killed byan uncaught signal, and contains the number of said signal.

In thefinish script, you need to write the container exit code youwant to the/run/s6-linux-init-container-results/exitcode file - andthat's it.

For instance, thefinish script for themyapp service above couldbe something like this:

#!/bin/shiftest"$1" -eq 256;then  e=$((128+$2))else  e="$1"fiecho"$e"> /run/s6-linux-init-container-results/exitcode

When you send adocker stop command to your container, themyappservice will be killed and this script will be run; it will writeeithermyapp's exit code (ifmyapp catches the TERM signal) or130 (ifmyapp does not catch the TERM signal) to the special/run/s6-linux-init-container-results/exitcode file, which willbe read by s6-overlay at the end of the container shutdown procedure,and your container will exit with that value.

This section describes a functionality from the versions of s6-overlaythat areanterior to v3. fix-attrs is still supported in v3,but isdeprecated, for several reasons: one of them is that it'sgenerally not good policy to change ownership dynamically when it can bedone statically. Another reason is that it doesn't work with USER containers.Instead of fix-attrs, we now recommend you to take care of ownership andpermissions on host mountsoffline, before running the container. Thisshould be done in your Dockerfile, when you have all the needed information.

That said, here is what we wrote for previous versions and that is stillapplicable today (but please stop depending on it):

Sometimes it's interesting to fix ownership & permissions before proceeding because,for example, you have mounted/mapped a host folder inside your container. Our overlayprovides a way to tackle this issue using files in/etc/fix-attrs.d.This is the pattern format followed by fix-attrs files:

path recurse account fmode dmode

• path: File or dir path.
• recurse: (Set totrue orfalse) If a folder is found, recurse through all containing files & folders in it.
• account: Target account. It's possible to default to fallbackuid:gid if the account isn't found. For example,nobody,32768:32768 would try to use thenobody account first, then fallback touid 32768 instead.If, for instance,daemon account isUID=2 andGID=2, these are the possible values foraccount field:
  • daemon: UID=2 GID=2
  • daemon,3:4: UID=2 GID=2
  • 2:2,3:4: UID=2 GID=2
  • daemon:11111,3:4: UID=2 GID=11111
  • 11111:daemon,3:4: UID=11111 GID=2
  • daemon:daemon,3:4: UID=2 GID=2
  • daemon:unexisting,3:4: UID=2 GID=4
  • unexisting:daemon,3:4: UID=3 GID=2
  • 11111:11111,3:4: UID=11111 GID=11111
• fmode: Target file mode. For example,0644.
• dmode: Target dir/folder mode. For example,0755.

Here you have some working examples:

/etc/fix-attrs.d/01-mysql-data-dir:

/var/lib/mysql true mysql 0600 0700

/etc/fix-attrs.d/02-mysql-log-dirs:

/var/log/mysql-error-logs true nobody,32768:32768 0644 2700/var/log/mysql-general-logs true nobody,32768:32768 0644 2700/var/log/mysql-slow-query-logs true nobody,32768:32768 0644 2700

Here is the old way of doing it:

After fixing attributes (through/etc/fix-attrs.d/) and before startinguser provided services (through s6-rc or/etc/services.d) our overlay willexecute all the scripts found in/etc/cont-init.d, for example:

/etc/cont-init.d/02-confd-onetime:

#!/command/execlineb -Pwith-contenvs6-envuidgid nginxmultisubstitute{  import -u -D0 UID  import -u -D0 GID  import -u CONFD_PREFIX  define CONFD_CHECK_CMD "/usr/sbin/nginx -t -c {{ .src }}"}confd --onetime --prefix="${CONFD_PREFIX}" --tmpl-uid="${UID}" --tmpl-gid="${GID}" --tmpl-src="/etc/nginx/nginx.conf.tmpl" --tmpl-dest="/etc/nginx/nginx.conf" --tmpl-check-cmd="${CONFD_CHECK_CMD}" etcd

This way is still supported. However, there is now a more generic andefficient way to do it: writing your oneshot initialization and finalizationtasks as s6-rc services, by adding service definition directories in/etc/s6-overlay/s6-rc.d, making them part of theuser bundle (so theyare actually started when the container boots), and making them depend onthebase bundle (so they are only started afterbase).

All the information on s6-rc can be foundhere.

When the container is started, the operations are performed in this order:

• (deprecated) Attribute fixing is performed according to files in/etc/fix-attrs.d.
• (legacy) One-shot initialization scripts in/etc/cont-init.d are run sequentially.
• Services in theuser bundle are started by s6-rc, in an order defined bydependencies. Services can be oneshots (initializationtasks) or longruns (daemons that will run throughout the container's lifetime). Ifthe services depend onbase, they are guaranteed to start at this point and notearlier; if they do not, they might have been started earlier, which may causerace conditions - so it's recommended to always make them depend onbase.
• (legacy) Longrun services in/etc/services.d are started.
• Services in theuser2 bundle with the correct dependency are started.(Most people don't need to use this; if you are not sure, stick to theuser bundle.)

When the container is stopped, either because the admin sent a stop command orbecause the CMD exited, the operations are performed in the reverse order:

• Services in theuser2 bundle with the correct dependency are stopped.
• (legacy) Longrun services in/etc/services.d are stopped.
• All s6-rc services are stopped, in an order defined by dependencies. Foroneshots, that means that thedown script in the source definition directoryis executed; that's how s6-rc can perform finalization tasks.
• (legacy) One shot finalization scripts in/etc/cont-finish.d are run sequentially.

The point of theuser2 bundle is to allow user services declared in it tostartafter the/etc/services.d ones; but in order to do so, every serviceinuser2 needs to declare a dependency tolegacy-services. In other words,for a servicefoobar to start late, you need to:

• Define it in/etc/s6-overlay/s6-rc.d/foobar like any other s6-rc service.
• Add an/etc/s6-overlay/s6-rc.d/foobar/dependencies.d/legacy-services file
• Add an/etc/s6-overlay/s6-rc.d/user2/contents.d/foobar file.

That will ensure thatfoobar will startafter everything in/etc/services.d.

By default, services created in/etc/services.d will automatically restart.If a service should bring the container down, you should probably run it asa CMD instead; but if you'd rather run it as a supervised service, then you'llneed to write afinish script, which will be run when the service is down; tomake the container stop, the/run/s6/basedir/bin/halt command must be invoked.Here's an example finish script:

/etc/services.d/myapp/finish:

#!/command/execlineb -S0foreground { redirfd -w 1 /run/s6-linux-init-container-results/exitcode echo 0 }/run/s6/basedir/bin/halt

The first line of the script writes0 to the/run/s6-linux-init-container-results/exitcode file.The second line stops the container. When you stop the container via the/run/s6/basedir/bin/haltcommand run from inside the container,/run/s6-linux-init-container-results/exitcode is read andits contents are used as the exit code for thedocker run command that launched the container.If the file doesn't exist, or if the container is stopped withdocker stop or another reason,that exit code defaults to 0.

It is possible to do more advanced operations in a finish script. For example, here's a scriptfrom that only brings down the service when it exits nonzero:

/etc/services.d/myapp/finish:

#!/command/execlineb -S1if { eltest ${1} -ne 0 -a ${1} -ne 256 }/run/s6/basedir/bin/halt

Note that in general, finish scripts should only be used for local cleanupsafter a daemon dies. If a service is so important that the container needsto stop when it dies, we really recommend running it as the CMD.

Every service can have its dedicated logger. A logger is a s6 service thatautomatically reads from thestdout of your service, and logs the datato an automatically rotated file in the place you want. Note that daemonsusually log to stderr, not stdout, so you should probably start your service'srun script withexec 2>&1 in shell, or withfdmove -c 2 1 in execline, inorder to catchstderr.

s6-overlay provides a utility calledlogutil-service which is a wrapper overthes6-log program.This helper does the following:

• read how s6-log should proceed reading the logging script contained inS6_LOGGING_SCRIPT
• drop privileges to thenobody user (defaulting to65534:65534 if it doesn't exist)
• clean all the environments variables
• execute into s6-log.

s6-log will then run forever, reading data from your service and writing it tothe directory you specified tologutil-service.

Please note:

• Since the privileges are dropped automatically, there is no need to switch users withs6-setuidgid
• You should ensure the log folder either:
  • exists, and is writable by thenobody user
  • does not exist, but the parent folder is writable by thenobody user.

You can create log folders incont-init.d scripts, or as s6-rc oneshots.Here is an example of a logged servicemyapp implemented the old way:

/etc/cont-init.d/myapp-log-prepare:

#!/bin/sh -emkdir -p /var/log/myappchown nobody:nogroup /var/log/myappchmod 02755 /var/log/myapp

/etc/services.d/myapp/run:

#!/bin/shexec2>&1exec mydaemon-in-the-foreground-and-logging-to-stderr

/etc/services.d/myapp/log/run:

#!/bin/shexec logutil-service /var/log/myapp

And here is the same service, myapp, implemented in s6-rc.

/etc/s6-overlay/s6-rc.d/myapp-log-prepare/dependencies.d/base: empty file

/etc/s6-overlay/s6-rc.d/myapp-log-prepare/type:

oneshot

/etc/s6-overlay/s6-rc.d/myapp-log-prepare/up:

if { mkdir -p /var/log/myapp }if { chown nobody:nogroup /var/log/myapp }chmod 02755 /var/log/myapp

/etc/s6-overlay/scripts/myapp-log-prepare

#!/bin/sh -emkdir -p /var/log/myappchown nobody:nogroup /var/log/myappchmod 02755 /var/log/myapp

/etc/s6-overlay/s6-rc.d/myapp/dependencies.d/base: empty file

/etc/s6-overlay/s6-rc.d/myapp-log/dependencies.d/myapp-log-prepare: empty file

/etc/s6-overlay/s6-rc.d/myapp/type:

longrun

/etc/s6-overlay/s6-rc.d/myapp/run:

#!/bin/shexec2>&1exec mydaemon-in-the-foreground-and-logging-to-stderr

/etc/s6-overlay/s6-rc.d/myapp/producer-for:

myapp-log

/etc/s6-overlay/s6-rc.d/myapp-log/type:

longrun

/etc/s6-overlay/s6-rc.d/myapp-log/run:

#!/bin/shexec logutil-service /var/log/myapp

/etc/s6-overlay/s6-rc.d/myapp-log/consumer-for:

myapp

/etc/s6-overlay/s6-rc.d/myapp-log/pipeline-name:

myapp-pipeline

/etc/s6-overlay/s6-rc.d/user/contents.d/myapp-pipeline: empty file

That's a lot of files! A summary of what it all means is:

• myapp-log-prepare is a oneshot, preparing the logging directory.It is a dependency of myapp-log, so it will be startedbefore myapp-log.
• myapp is a producer for myapp-log and myapp-log is a consumer for myapp,so what myapp writes to its stdout will go to myapp-log's stdin. Bothare longruns, i.e. daemons that will be supervised by s6.
• Themyapp | myapp-log pipeline is given a name,myapp-pipeline, andthis name is declared as a part of theuser bundle, so it will be startedwhen the container starts.
• myapp-log-prepare,myapp-log andmyapp all depend on thebasebundle, which means they will only be started when the system is actuallyready to start them.

It really accomplishes the same things as the/etc/cont-init.d plus/etc/services.d method, but it's a lot cleaner underneath, and can handlemuch more complex dependency graphs, so whenever you get the opportunity,we recommend you familiarize yourself with thes6-rcway of declaring your services and your loggers. The full syntax of aservice definition directory, including declaring whether your serviceis a longrun or a oneshot, declaring pipelines, adding service-specifictimeouts if you need them, etc., can be foundhere.

When it comes to executing a service, no matter whether it's a service or a logger,a good practice is to drop privileges before executing it.s6 already includes utilities to do exactly these kind of things:

Inexecline:

#!/command/execlineb -Ps6-setuidgid daemonmyservice

Insh:

#!/bin/shexec s6-setuidgid daemon myservice

If you want to know more about these utilities, please take a look at:s6-setuidgid,s6-envuidgid, ands6-applyuidgid.

If you want your custom script to have container environments available:you can use thewith-contenv helper, which will push all of those into yourexecution environment, for example:

/etc/cont-init.d/01-contenv-example:

#!/command/with-contenv shenv

This script will output the contents of your container environment.

Recent versions of Docker allow running containers with a read-only root filesystem.If your container is in such a case, you should setS6_READ_ONLY_ROOT=1 to informs6-overlay that it should not attempt to write to certain areas - instead, it willperform copies into a tmpfs mounted on/run.

Note that s6-overlay assumes that:

• /run exists and is writable. If it is not, it will attempt to mount a tmpfs there.
• /var/run is a symbolic link to/run, for compatibility with previous versions. If it is not, it will make it so.

In general your default docker settings should already provide a suitable tmpfs in/run.

It is possible somehow to tweak s6-overlay's behaviour by providing an already predefined set of environment variables to the execution context:

• PATH (default =/command:/usr/bin:/bin):this is the default PATH that all the services in the container,including the CMD, will have. Set this variable if you have a lotof services that depend on binaries stored in another directory, e.g./usr/sbin. Note that/command,/usr/bin and/bin will alwaysbe added to that path if they're not already in the one you provide.
• S6_KEEP_ENV (default = 0): if set, then environment is not reset and whole supervision tree sees original set of env vars. It switcheswith-contenv into a nop.
• S6_LOGGING (default = 0):
  • 0: Outputs everything to stdout/stderr.
  • 1: Uses an internalcatch-all logger and persists everything on it, it is located in/var/log/s6-uncaught-logs. Anything run as aCMD is still output to stdout/stderr.
  • 2: Uses an internalcatch-all logger and persists everything on it, including the output ofCMD. Absolutely nothing is written to stdout/stderr.
• S6_CATCHALL_USER (default = root): if set, and ifS6_LOGGING is 1 or 2,then the catch-all logger is run as this user, which must be defined in yourimage's/etc/passwd. Every bit of privilege separation helps a little with security.
• S6_BEHAVIOUR_IF_STAGE2_FAILS (default = 0): determines what the container should doif one of the service scripts fails. This includes:
  • if the early stage2 hook exits nonzero (by default there's no hook)
  • if anything fails infix-attrs
  • if any old-style/etc/cont-init.d or new-styles6-rc oneshot fails
  • if any old-style/etc/services.d or new-styles6-rc longrun is markedas expecting readiness notification, and fails to becomeready in the allotted time (seeS6_CMD_WAIT_FOR_SERVICES_MAXTIME below). The valid values forS6_BEHAVIOUR_IF_STAGE2_FAILSare the following:
  • 0: Continue silently even if a script has failed.
  • 1: Continue but warn with an annoying error message.
  • 2: Stop the container.
• S6_KILL_FINISH_MAXTIME (default = 5000): How long (in milliseconds) the system shouldwait, at shutdown time, for a script in/etc/cont-finish.d to finish naturally. After thisduration, the script will be sent a SIGKILL. Bear in mind that scripts in/etc/cont.finish.dare run sequentially, and the shutdown sequence will potentially wait forS6_KILL_FINISH_MAXTIMEmilliseconds foreach script.
• S6_SERVICES_READYTIME (default = 50): With services declared in/etc/services.d, there isan unavoidable race condition between the moment when services are started and the moment whenthey can be tested for readiness. To avoid that race, we sleep a little time, by default 50milliseconds, before testing for readiness. If your machine is slow or very busy, you mayget errors looking likes6-svwait: fatal: unable to s6_svstatus_read: No such file or directory.In that case, you should increase the sleeping time, by declaring it (in milliseconds) in theS6_SERVICES_READYTIME variable. Note that it only concerns/etc/services.d; s6-rc is immuneto the race condition.
• S6_SERVICES_GRACETIME (default = 3000): How long (in milliseconds)s6 should wait,at shutdown time, for services declared in/etc/services.d to die before proceedingwith the rest of the shutdown.
• S6_KILL_GRACETIME (default = 3000): How long (in milliseconds)s6 should wait, at the end ofthe shutdown procedure when all the processes have received a TERM signal, for them to diebefore sending aKILL signal to makesure they're dead.
• S6_LOGGING_SCRIPT (default = "n20 s1000000 T"): This env decides what to log and how, by default every line will prepend with ISO8601, rotated when the current logging file reaches 1mb and archived, at most, with 20 files.
• S6_CMD_ARG0 (default = not set): Value of this env var will be prepended to anyCMD args passed by docker. Use it if you are migrating an existing image to s6-overlay and want to make it a drop-in replacement: setting this variable to the value of a previously used ENTRYPOINT will help you transition.
• S6_CMD_USE_TERMINAL (default = 0): Set this value to1 if you have a CMD that needs a terminal for its output(typically when you're running your container withdocker run -it), and you have setS6_LOGGING to a nonzero value.This setting will make your CMD actually output to your terminal; the drawback is that its output will not be logged.By default (when this variable is0 or not set), the stdout and stderr of your CMD are logged whenS6_LOGGING is nonzero,which means they go to a pipe even if you're running it in an interactive terminal.
• S6_FIX_ATTRS_HIDDEN (default = 0): Controls howfix-attrs.d scripts process files and directories.
  • 0: Hidden files and directories are excluded.
  • 1: All files and directories are processed.
• S6_CMD_WAIT_FOR_SERVICES (default = 0): By default when the container starts,services in/etc/services.d will be started and execution will proceed tostarting theuser2 bundle and the CMD, if any of these is defined. IfS6_CMD_WAIT_FOR_SERVICES is nonzero, however, the container starting sequencewill wait until the services in/etc/services.d areready before proceedingwith the rest of the sequence. Note that this is only significant if the services in/etc/services.dnotify their readiness to s6.
• S6_CMD_WAIT_FOR_SERVICES_MAXTIME (default = 0, i.e. infinite): The maximum time (in milliseconds) the services could take to bring up before proceding to CMD executing.Set this variable to a positive value if you have services that can potentially block indefinitely and you prefer the container to failif not everything is up after a given time.Note that this value also includes the time setting up legacy container initialization (/etc/cont-init.d) and services (/etc/services.d), sotake that into account when computing a suitable value. In versions of s6-overlay up to 3.1.6.2, the default was 5000 (five seconds),but it caused more unwanted container failures than it solved issues, so now there's no timeout by default: s6-overlay will wait as long asis necessary for all the services to be brought up.
• S6_READ_ONLY_ROOT (default = 0): When running in a container whose root filesystem is read-only, set this env to1 to inform init stage 2 that it should copy user-provided initialization scripts from/etc to/run/s6/etc before it attempts to change permissions, etc. SeeRead-Only Root Filesystem for more information.
• S6_SYNC_DISKS (default = 0): Set this env to1 to inform init stage 3 that it should attempt to sync filesystems before stopping the container. Note: this will likely sync all filesystems on the host.
• S6_STAGE2_HOOK (default = none): If this variable exists, its contentswill be interpreted as a shell excerpt that will be run in the early stage 2,before services are started. This can be used, for instance, to dynamicallypatch the service database at run-time right before it is compiled and run.If the hook program exits nonzero andS6_BEHAVIOUR_IF_STAGE2_FAILS is 2 or more,the container will stop instantly. Please note that running the wrong hook programmay prevent your container from starting properly, or may endanger your security;so only use this if you know exactly what you are doing. When in doubt, leavethis variable undefined.
• S6_VERBOSITY (default = 2): controls the verbosity of s6-rc, and potentiallyother tools, at container start and stop time. The default, 2, is normally verbose:it will list the service start and stop operations. You can make the container quieterby decreasing this number: 1 will only print warnings and errors, and 0 will onlyprint errors. You can also make the containermore verbose, i.e. print tracing anddebug information, by increasing this number up to 5, but the output will quicklybecomevery noisy, and most people shouldn't need this.
• S6_CMD_RECEIVE_SIGNALS (default = 0): decides whether signals sent to thecontainer should be sent to the container's pid 1 or to the CMD. By default, whenyou perform for instance adocker stop, a TERM signal will be sent to thecontainer's pid 1, which will trigger the full container shutdown sequence - butif a CMD is present, it will be among the last processes to be killed, only wheneverything else is down and the container is about to exit. If this variable is1 or more, signals are diverted from pid 1 to the CMD, which means thatdocker stopwill send a SIGTERM to the CMD instead, and the container will only trigger its shutdownprocedure when the CMD is dead. Note that only SIGTERM, SIGQUIT, SIGINT, SIGUSR1,SIGUSR2, SIGPWR and SIGWINCH are diverted; other signals either are ignored orcannot be diverted and are necessarily handled by pid 1. Please be aware that usingthis option may prevent interactive CMDs from working at all - in other words, ifyou're running an interactive CMD in a terminal, don't set this variable; but thatshould be fine since in this case you already have interactive ways of stopping your CMD.
• S6_YES_I_WANT_A_WORLD_WRITABLE_RUN_BECAUSE_KUBERNETES (default = 0): yes, it's amouthful, and yes, we're deadly serious. If you set this variable to 1 (or any non-zerovalue), s6-overlay will accept to boot in a situation where/run belongs to uid 0 andis world-writable (permissions 0777) but the container is run as a non-root user. Thisis the configuration enforced by some Kubernetes environments, and it is completelyinsecure, except that Kubernetes ensures isolation by its own security mechanisms; so,if you're in such a situation, set that variable, and s6-overlay will still print a(literally) bright red warning, but it will boot. Otherwise, by default, s6-overlayrefuses to run when it encounters such a insecure setup that it doesn't have thenecessary privileges to fix.

If software running in your container requires syslog, extract thesyslogd-overlay-noarch.tar.xz tarball:that will give you a small syslogd emulation. Logs will be foundunder various subdirectories of/var/log/syslogd, for instancemessages will be found in the/var/log/syslogd/messages/ directory,the latest logs being available in the/var/log/syslogd/messages/current file.Logging directories are used rather than files so that logs can beautomatically rotated without race conditions (that is a feature ofs6-log).

It is recommended to addsyslog andsysllog users to your image, forprivilege separation; the syslogd emulation processes will run as these usersif they exist. Otherwise they will default to32760:32760 and32761:32761,numeric uids/gids that may already exist on your system.

• The noarch and symlinks tarballs are all tiny. The biggest tarball is theone that contains the binaries; it's around 650 kB.
• Uncompressed on a tmpfs, the overlay scripts use about 120 kB, and thebinaries for x86_64 use about 5.7 MB.
• We haven't yet measured the time it takes for the container to be up and runningonce you rundocker run, but you will notice it's fast. Faster than previousversions of s6-overlay, with fewer delays. And if you convert your/etc/cont-init.dscripts to the s6-rc format, they will be able to run in parallel, so you willgain even more performance. If you have benchmarks, please send them to us!

The s6-overlay releases have a checksum files you can use to verifythe download using SHA256:

ADD https://github.com/just-containers/s6-overlay/releases/download/v${S6_OVERLAY_VERSION}/s6-overlay-noarch.tar.xz /tmpADD https://github.com/just-containers/s6-overlay/releases/download/v${S6_OVERLAY_VERSION}/s6-overlay-noarch.tar.xz.sha256 /tmpADD https://github.com/just-containers/s6-overlay/releases/download/v${S6_OVERLAY_VERSION}/s6-overlay-x86_64.tar.xz /tmpADD https://github.com/just-containers/s6-overlay/releases/download/v${S6_OVERLAY_VERSION}/s6-overlay-x86_64.tar.xz.sha256 /tmpRUNcd /tmp&& sha256sum -c*.sha256

As of version 3.2.1.0, s6-overlay has limited support for running as a user other thanroot:

• Tools likefix-attrs andlogutil-service are unlikely to work (they relyon being able to change UIDs).
• The syslogd emulation will not work.

Generally speaking, if you're running a simple container with a main application andone or two support services, you may benefit from theUSER directive if that isyour preferred way of running containers. However, if you're running more than a fewservices, or daemons that expect a real system with complete Unix infrastructure,then USER is probably not a good idea and you would benefit more from usingprivilege separation between services in your container.

Generally speaking, youshould not run your containers withdocker run -it.It is bad practice to have console access to your containers. That said, if yourCMD is interactive and needs a terminal, s6-overlay will try to support it wheneverpossible, but the nature of terminals makes it difficult to ensure that everythingworks perfectly in all cases.

In particular, if you are stacking virtualization environments and other layersalready have their own kludges for terminals - for instance, if you are runnings6-overlay under qemu - then it is almost guaranteed thatdocker run -it willnot work. However, once the container is running, you should always be able toaccess an interactive shell inside it viadocker exec -it containername /bin/sh.

The same caveats apply to stopping containers with ^C. Normally containers arestopped viadocker stop, or when the CMD exits; ^C is not an officially supportedmethod of stopping them. s6-overlaytries to exit cleanly on ^C, whether thecontainer is running with-it or not, but there will be cases where it isunfortunately impossible.

Over on the releases tab, we have a number of tarballs:

• s6-overlay-noarch.tar.xz: the s6-overlay scripts.
• s6-overlay-${arch}.tar.xz: the binaries for platform${arch}.They are statically compiled and will work with any Linux distribution.
• s6-overlay-symlinks-noarch.tar.xz:/usr/bin symlinks to the s6-overlay scripts. Totally optional.
• s6-overlay-symlinks-arch.tar.xz:/usr/bin symlinks to the skaware binaries. Totally optional.
• syslogd-overlay-noarch.tar.xz: the syslogd emulation. Totally optional.
• s6-overlay-${version}.tar.xz: the s6-overlay source. Download it if you want to build s6-overlay yourself.

We have binaries for at least x86_64, aarch64, arm32, i486, i686, riscv64, and s390x.The full list of supported arches can be found inconf/toolchains.

The${arch} part in thes6-overlay-${arch}.tar.xz tarball usesthe naming conventions of gcc, which are not the ones that Dockeruses. (Everyone does something different in this field depending ontheir needs, and no solution is better than any other, but the Dockerone isworse than others because its naming is inconsistent. The gccconvention is better for us because it simplifies our builds greatly andmakes them more maintainable.)

The following table should help you find the right tarball for youif you're using the TARGETARCH value provided by Docker:

If you need another architecture, ask us and we'll try to make a toolchainfor it. In particular, we know that armv7 is a mess and needs a flurry ofoptions depending on your precise target (and this is one of the reasons whythe Docker naming system isn't good, although arguably the gcc naming systemisn't much better on that aspect).

Any way you want! Open issues, open PRs, we welcome all contributors!

• Download the [s6-overlay source].
• Check theconf/defaults.mkfile for variables you may want to change. Do not modify the file itself.
• Callmake followed by your variable assignments. Example:make ARCH=riscv64-linux-muslto build the overlay for RISCV64.
• The tarballs will be in theoutput subdirectory, unless you changed theOUTPUT variable.

Please seeCHANGELOG.