In this post I'm going to explain several ways to build docker images for multiple architectures. With the ongoing rise of ARM-architectures, for example the Raspberry Pi or Amazon's efficientEC2 A1-Instances, multi-architecture builds will probably gain more focus.

If you're on a single computer, building and running docker images is very easy. Thebuild command analyzes a givenDockerfile and runs the specific instructions. To do so, docker uses the kernel of your OS (or your VM, depending on your setup). This can bound the architecture of the image to the host architecture, especially when you compile a binary inside it.

When using a typical desktop PC, this architecture is probablyx86/amd64. So if you run this image on a different computer with the same architecture, everything is fine. But what to do when a different architecture is the target, for example ARM?

Compile programs for different architectures

Generally speaking, 3 typical techniques are used to compile software for a different architecture, which are briefly explained in this section.

1. Build on the target system

Building your software directly on the target system is obviously the easiest approach, since you probably do not have to change anything in your code.

Let's consider building software for the ARM architecture. For example, you could transfer your code to a Raspberry Pi, install the toolchain and build your code there. Since Docker is supported on this device, you can use the same commands as on your desktop computer.
Typical problems with this approach can be the limited access to ARM-powered hardware and since they are often not that powerful, a slow build performance. E.g., compiling dependencies in C is something which can take a lot of time on a Raspberry Pi (though its get faster with every new board).

2. Emulate the hardware

If you do not have access to the target system, emulation is another viable solution. While in virtualization only certain parts of a computer's hardware are simulated in order to run a guest OS, emulation simulates the complete hardware. This makes emulation slower than virtualization, but it also not limited to the underlying hardware, making it possible to simulate hardware like an ARM-processor on a x86 system.

As you might have guessed, this approach is truly powerful by enabling you to build software for various architectures, but simulating the entire hardware is a huge overhead. Wouldn't it be great to simulate only the hardware components required for building the software?

User-mode emulation

Usually when an executable file is passed to aexec-system call, the kernel expects it to be a native binary for the current system.
If you ever encountered the errorexec user process caused "exec format error" in a docker image, you tried to run a binary which can't be executed on the processor.
Luckily, withbinfmt_misc it is possible to register custom interpreters in the userland to handle foreign binaries.
To do so, you basically register the respective interpreter and a "magic number", which identifies these binaries in the specific format.
The idea is, that in combination with a powerful emulator, you can still run and build binaries for foreign architectures on your system, by emulating only the required parts.

QEMU

One famous emulator which is capable of such a feature, isQEMU. Besides a user-mode emulation, it supports various architectures, full-system emulation and virtualization as well.

To use user-mode emulation with QEMU, we need to register this emulator for some foreign architectures, for example ARM. The Hypriot project has explained this well inthis article. They even build a docker image, which runs inprivileged mode to do this registering of magic numbers with QEMU for you on your host system.
An excerpt of theregister script is put below to give you a rough idea:

# Register new interpreters# - important: using flags 'C' and 'F'echo':qemu-arm:M::\x7fELF\x01\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x28\x00:\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\xff:/qemu-arm:CF'> /proc/sys/fs/binfmt_misc/registerecho':qemu-aarch64:M::\x7fELF\x02\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\xb7\x00:\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\xff:/qemu-aarch64:CF'> /proc/sys/fs/binfmt_misc/registerecho':qemu-ppc64le:M::\x7fELF\x02\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x15\x00:\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\x00:/qemu-ppc64le:CF'> /proc/sys/fs/binfmt_misc/register# Show resultsecho"---"echo"Installed interpreter binaries:"ls-al /qemu-*echo"---"cd /proc/sys/fs/binfmt_miscforfilein*;do    case"${file}"instatus|register);;*)echo"Registered interpreter=${file}"cat${file}echo"---";;esacdone

3. Use a cross-compiler

The last option is using a cross-compiler to build software for different architectures.
While a standard compiler builds for the system it's running on, a cross-compiler can generate binaries for other architectures as well.
Since it does not rely on any emulation but runs natively on your system, the build performance is as good as option 1.
Modern languages put in a lot of effort to support the feature well, especially withGolang this is a breeze. For example, you specify the OS (GOOS) and the target architecture (GOARCH) to cross compile:

# build for macGOOS=darwinGOARCH=386 go build main.go# build for Raspberry PiGOOS=linuxGOARCH=arm go build main.go

If multi-arch builds are an ongoing task for you, definitely check out such programming languages.

Docker images and multi-arch

Let's take a look at how docker support multi-arch images. Afaik, two approaches are mainly used:

Separate image

One option is to make a different image for each architecture. This can be done by creating a different repository or setting a designated tag for each supported platform.
This usually requires a separate Dockerfile too, as seen in thecoreos/flannel repository for example:

Dockerfile

# AMD 64FROM alpineENVFLANNEL_ARCH=amd64ADD dist/qemu-$FLANNEL_ARCH-static /usr/bin/qemu-$FLANNEL_ARCH-static...

Dockerfile.arm

FROM arm32v6/alpineENV FLANNEL_ARCH=armADD dist/qemu-$FLANNEL_ARCH-static /usr/bin/qemu-$FLANNEL_ARCH-static...

By passing the architecture as build arguments or environment variables to the Dockerfile, you can run instructions specific to these platforms. In this example, an env variableFLANNEL_ARCH is used for this purpose.

Manifests

A Docker manifest is a very simple concept. Basically, it's an object containing a list of image references for each supported architecture. A docker client can then pull an image by inspecting this manifest file returned by the registry, search the list for a matchingplatform and then load the image by the identifyingdigest.

An example docker manifest file, containing images forlinux/arm,linux/amd64 andlinux/ppc64le, is shown below:

{"schemaVersion":2,"mediaType":"application/vnd.docker.distribution.manifest.list.v2+json","manifests":[{"mediaType":"application/vnd.docker.distribution.manifest.v2+json","size":424,"digest":"sha256:f67dcc5fc786f04f0743abfe0ee5dae9bd8caf8efa6c8144f7f2a43889dc513b","platform":{"architecture":"arm","os":"linux"}},{"mediaType":"application/vnd.docker.distribution.manifest.v2+json","size":424,"digest":"sha256:b64ca0b60356a30971f098c92200b1271257f100a55b351e6bbe985638352f3a","platform":{"architecture":"amd64","os":"linux"}},{"mediaType":"application/vnd.docker.distribution.manifest.v2+json","size":425,"digest":"sha256:df436846483aff62bad830b730a0d3b77731bcf98ba5e470a8bbb8e9e346e4e8","platform":{"architecture":"ppc64le","os":"linux"}}]}

While this feature is currentlyexperimental on the docker client, it's already integrated in the docker registry and containerd. TheOCI Image specification has included manifests too.

Generating manifests

As stated above, you currently need to enable experimental features on the docker client to work with manifests. Once you've done that,docker manifest should be a valid command. Generating a manifest is then very easy via the CLI.

Let's say we developed asuperapp and built an image foramd64 andarm manually.
Now we want to publish these two images,app/superapp-amd64 andapp/superapp-arm, as "one" imageapp/superapp by using a manifest.
This can be achieved with the following two commands:

docker manifest create app/superapp app/superapp-amd64 app/superapp-arm# Created manifest list docker.io/app/superapp# Push the manifestdocker manifest push app/superapp

Using Docker'sbuildx for ARM builds

In June 2019, Docker announced tooling support for building docker images and the ARM architecture as an experimental feature.
On Docker Desktop seehttps://engineering.docker.com/2019/04/multi-arch-images/ and on Linux seehttps://engineering.docker.com/2019/06/getting-started-with-docker-for-arm-on-linux/ for a setup guide.

This feature combines the approaches above, namely QEMU, binfmt_misc, and manifests and bundles them as a single toolbuildx. It allows you to write a single Dockerfile, which can be used to build images for various platforms without changing it. And with the QEMU user-mode emulation, you can build and run images which have a different architecture than your current system.

A simple example

Let's package a simple Go-application as a docker image forarm64 andamd64. We write a simple program that reports the OS and the architecture of the system.

main.go

packagemainimport"fmt"import"runtime"funcmain(){fmt.Printf("OS: %s\nArchitecture: %s\n",runtime.GOOS,runtime.GOARCH)}

If we take a look at the Dockerfile, we see that it is not aware of the architectures which should be supported.

FROM golang:alpine AS builderRUNmkdir /appADD. /app/WORKDIR /appRUN go build-o report.FROM busyboxRUNmkdir /appWORKDIR /appCOPY--from=builder /app/report.CMD["./report"]

Now comes the interesting part, the cross-architecture build withbuildx. Let's build an image forlinux/amd64 andlinux/arm64:

docker buildx build--platform linux/amd64,linux/arm64-t foo/bar--push.

• with the--platform flag you specify the platforms the image should be built for
• just likedocker build, the-t flag lets you define the image tag
• --push pushes the image directly to the registry after an successful build

A simple architecture-aware example

Sometimes, you do not want to build everything natively in the Dockerfile, for example by downloading prebuilt third-party binaries for the target architecture. Docker'sbuildx has you covered here as well, as it exposes for each value in--platformarguments likeTARGETARCH orTARGETOS.

For instance, this is aDockerfile for a multi-architecture image for ipfs-cluster. In order to skip the compilation of the ipfs-cluster binary from source, we download the right version for our architecture viawget.

FROMgolang:1.12-stretchASbuilder# This dockerfile builds and runs ipfs-cluster-service.ENV SUEXEC_VERSION v0.2ENV TINI_VERSION v0.16.1ENV IPFS_CLUSTER_VERSION v0.11.0ARG TARGETARCHRUNset-x\&&cd /tmp\&& git clone https://github.com/ncopa/su-exec.git\&&cdsu-exec\&& make\### native build ###  && git checkout -q $SUEXEC_VERSION \...RUNwget https://dist.ipfs.io/ipfs-cluster-ctl/${IPFS_CLUSTER_VERSION}/ipfs-cluster-ctl_${IPFS_CLUSTER_VERSION}_linux-${TARGETARCH}.tar.gz ...

When built with the--platform linux/amd64,linux/arm64,linux/arm flag, theTARGETARCH is set toamd64,arm,arm64 once in order to download the correct binary. Still, native builds are possible, since we're emulating the hardware. For instance,su-exec is simply compiled withmake. Nice!

Caveats

As an experimental tool,buildx still has some rough edges for the developer. The main issues I encountered where:

• You can only push the images directly to a registry. A file export of the docker image is only possible in theOCI format.Issue 166,Issue 186
• When an error is thrown while building your image, it's often very hard to decipher the error message in a multi-arch error log. If you can, first try to build with the usualdocker build command for your system and if it works, switch tobuildx.
• Due to the emulation, the build is still magnitudes slower, especially if you need to compile dependencies of your software. Check for prebuilt dependencies/binaries online and load the correct version, usingTARGETARCH for example.

Wrapping it up

I hope I could give you a brief overview of the various techniques for multi-architecture docker images or raise your interest in it.
It's really exciting to see that Docker invests in simplifying this process and whilebuildx is still young, it's already usable.
Besides building on your local machine, installing the user-mode emulation is also possible on CI-servers, so a multi-architecture build pipeline can be set up.
Thinking of containers as a way to package software, it's a logical next step to make it available for various architectures as easy as possible.

Further reading

• containers with binfmt_misc
• docker buildx setup
• Demo with buildx
• Multiarch images with Go and without buildx