Building Secure Images with GitHub Actions
When I initially started building Docker images onlyamd64 was relevant, and having a build ⇒ Scan ⇒ Test ⇒ Push cycle was as easy as using docker save and docker load. But witharm64 becoming more and more popular, including with my own home cluster, I needed to add images forarm64. For a while this meant I was pushing images with-amd64 and-arm64 suffixes before combining them into one “image”. All because I want to ensure no images with CVE’s or bugs are pushed. Skipping those is not acceptable for me as a Software Engineer.
The initial problem is that most tooling for building you will find with a quick search will build and push multi arch images in one move. While this is cool if you want to get a multi arch image out there, it does not meet my requirements. So instead I kept building mainly the same way, but now I had another dimension on my CI matrix that is the arch to build for. To be specificlinux/amd64 andlinux/arm64. (The OS + arch combination is referred to as the image platform.) This will let me test each variant of an image across all supported platforms using QEMU to simulate the arch, short for architecture, they are build for and will run on. At the end of the pipeline where each image is pushed, I had two steps:
- One to push all different image variants
- One to create a manifest combining each platform for each variant and push that
Today, as I’m writing this which is two days ago when you first might be reading this, I finally figured out how to do this without pushing to an external registry first. Not going to bore and confuse you with the changes between my previous iteration, instead I’ll walk you through the, be it a somewhat stripped, workflow that makes this possible.
This set up solves a few issues:
- BuildX is cool, but it doesn’t let you save and load multiplatform images as docker build does
- No images with suffixes on the registry anymore that soil it with images anyone will unlikely use directly
- Only one step to push instead of two
Note: This post is based on thisPR onwyrihaximusnet/docker-redirect if you just want to skip to the workflow. It’s a project I started to learn a ton of languages just a bit, but it got knocked into the background due to the impact the covid pandemic had on me.
Set up
The workflow needs some basic set up which includes the image name, which registries to push to, a job that does some JSON magic, and a job that will make the supported platforms available. (The jobs could use some polishing preferably into a way that doesn’t require any jobs. But that is an improvement for another time.)
name: Continuous Integrationenv: DOCKER_IMAGE: wyrihaximusnet/redirect DOCKER_IMAGE_REGISTRIES_SECRET_MAPPING: '{"ghcr.io":"GHCR_TOKEN","docker.io":"HUB_PASSCODE"}'on: push: schedule: - cron: '0 0 * * 0'jobs: registry-matrix: name: Extract registries from registry secret mapping if: (github.event_name == 'push' || github.event_name == 'schedule') && github.ref == 'refs/heads/master' runs-on: ubuntu-latest needs: - tests outputs: registry: ${{ steps.registry-matrix.outputs.registry }} steps: - uses: actions/checkout@v4 - id: registry-matrix name: Extract registries from registry secret mapping run: | echo "registry=$(printenv DOCKER_IMAGE_REGISTRIES_SECRET_MAPPING | jq -c 'keys')" >> $GITHUB_OUTPUT supported-arch-matrix: name: Supported processor architectures runs-on: ubuntu-latest outputs: arch: ${{ steps.supported-arch-matrix.outputs.arch }} steps: - uses: actions/checkout@v4 - id: supported-arch-matrix name: Generate Arch run: | echo "arch=[\"linux/amd64\",\"linux/arm64\"]" >> $GITHUB_OUTPUTBuilding
To build the image for multiple platforms we need one thing: QEMU to emulate the arch we’re building for if it’s not the runners native arch. So we have to make sure it’s installed before we can build:
- name: Set up QEMU uses: docker/setup-qemu-action@v3Once it’s set up we can build the image using the normal docker build command. We pass in the platform using the--platform flag and use the environment variablePLATFORM_PAIR we created at the start of the job as the suffix on the image tag:
docker image build --platform=${{ matrix.platform }} -t "${DOCKER_IMAGE}:reactphp-${{ env.PLATFORM_PAIR }}" --no-cache .Once the image has been built we use good olddocker save to save the image to a tarball, for later use we make sure we include the platform in the file name:
docker save "${DOCKER_IMAGE}:reactphp-${{ env.PLATFORM_PAIR }}" -o ./docker-image/docker_image-${{ env.PLATFORM_PAIR }}.tarThen, we upload the directory the tarball is in as an artifact, and make sure we use the platform in the name, this will come in handy later:
- uses: actions/upload-artifact@v4 with: name: docker-image-reactphp-${{ env.PLATFORM_PAIR }} path: ./docker-imageThe full job:
build-docker-image: name: Build reactphp Docker (${{ matrix.platform }}) strategy: fail-fast: false matrix: platform: ${{ fromJson(needs.supported-arch-matrix.outputs.arch) }} needs: - supported-arch-matrix runs-on: ubuntu-latest steps: - name: Prepare run: | platform=${{ matrix.platform }} echo "PLATFORM_PAIR=${platform//\//-}" >> $GITHUB_ENV - name: Set up QEMU uses: docker/setup-qemu-action@v3 - uses: actions/checkout@v4 - run: mkdir ./docker-image - run: docker image build --platform=${{ matrix.platform }} --build-arg BUILD_DATE=`date -u +"%Y-%m-%dT%H:%M:%SZ"` --build-arg VCS_REF=`git rev-parse --short HEAD` -t "${DOCKER_IMAGE}:reactphp-${{ env.PLATFORM_PAIR }}" --no-cache . - run: docker save "${DOCKER_IMAGE}:reactphp-${{ env.PLATFORM_PAIR }}" -o ./docker-image/docker_image-${{ env.PLATFORM_PAIR }}.tar - uses: actions/upload-artifact@v4 with: name: docker-image-reactphp-${{ env.PLATFORM_PAIR }} path: ./docker-imageCVE Scanning/Testing
For this post’s sake it doesn’t matter if I cover testing or scanning, so only the set up is covered and the rest is left for your imagination. (You can always check the PR mentioned earlier in this post of course.)
Depending on what you are going to do, you will need to install QEMU again if you are going to run the image. In the project this is taken from the image will be started andk6 is used to test it functionally.
Next we’ll get the image artifact:
- uses: actions/download-artifact@v4 with: name: docker-image-reactphp-${{ env.PLATFORM_PAIR }} path: /tmp/docker-imageNext we load the image into Docker, this works fine because it’s only built for a single platform and no multi platform manifest is at play:
docker load --input /tmp/docker-image/docker_image-${{ env.PLATFORM_PAIR }}.tarAfter that, go wild and do what you have to do to make sure the image is up to spec, in my case I scan for CVE’s to make sure they don’t make it onto the registry:
echo -e "${{ env.DOCKER_IMAGE }}:reactphp-${{ env.PLATFORM_PAIR }}" | xargs -I % sh -c 'docker run -v /tmp/trivy:/var/lib/trivy -v /var/run/docker.sock:/var/run/docker.sock -t aquasec/trivy:latest --cache-dir /var/lib/trivy image --exit-code 1 --no-progress --format table %'The full job:
go-wild: name: Scan reactphp for vulnerabilities (${{ matrix.platform }}) strategy: fail-fast: false matrix: platform: ${{ fromJson(needs.supported-arch-matrix.outputs.arch) }} needs: - supported-arch-matrix - build-docker-image runs-on: ubuntu-latest steps: - name: Prepare run: | platform=${{ matrix.platform }} echo "PLATFORM_PAIR=${platform//\//-}" >> $GITHUB_ENV - name: Set up QEMU uses: docker/setup-qemu-action@v3 - uses: actions/checkout@v4 - uses: actions/download-artifact@v4 with: name: docker-image-reactphp-${{ env.PLATFORM_PAIR }} path: /tmp/docker-image - run: docker load --input /tmp/docker-image/docker_image-${{ env.PLATFORM_PAIR }}.tar - run: rm -Rf /tmp/docker-image/ - run: # Go wildPushing
The reason we don’t need a public registry is because for the pushing we’ll run one locally as a service on the job. We’ll use it in pretty much the same way as the public registry, but this way we don’t clutter it with temporary tags:
services: registry: image: registry:2 ports: - 5000:5000Before we can push we need QEMU again, and Buildx running on the host network:
- name: Set up QEMU uses: docker/setup-qemu-action@v3- name: Set up Docker Buildx uses: docker/setup-buildx-action@v3 with: driver-opts: network=hostNext, we download all artifacts for this specific image using thepattern option on thedownload-artifact action:
- uses: actions/download-artifact@v4 with: pattern: docker-image-reactphp-* path: /tmp/docker-image merge-multiple: trueOnce they are all downloaded, and in the same directory we can load them into docker one by one:
- run: | for f in /tmp/docker-image/docker_image-*.tar; do docker load --input $f doneBefore we can use the images to combine them into one, we have to retag and push them to the local registry. Which comes down to prefixing the existing tag withlocalhost:5000/.
- run: | archs=${{ join(fromJson(needs.supported-arch-matrix.outputs.arch), ',') }} for arch in ${archs//,/ } do docker tag "${{ env.DOCKER_IMAGE }}:reactphp-${arch//\//-}" "localhost:5000/${{ env.DOCKER_IMAGE }}:reactphp-${arch//\//-}" docker push "localhost:5000/${{ env.DOCKER_IMAGE }}:reactphp-${arch//\//-}" doneThe easiest way possible to combine multiple images for different platforms into one is by using some DockerFROM magic. There are few build inARGs you can use in theFROM instruction of a Dockerfile. In this case we useTARGETOS andTARGETARCH because those match withlinux andarm64 inlinux/arm64.
echo "FROM localhost:5000/${{ env.DOCKER_IMAGE }}:reactphp-\${TARGETOS}-\${TARGETARCH}" >> docker-file-${{ matrix.registry }}-wyrihaximusnet-redirect-reactphpThis way we only have to tell Buildx which Dockerfile to use, which platforms to build, what the image tag will be, and to push it when done:
docker buildx build -f docker-file-${{ matrix.registry }}-wyrihaximusnet-redirect-reactphp --platform=${{ join(fromJson(needs.supported-arch-matrix.outputs.arch), ',') }} -t ${{ matrix.registry }}/${{ env.DOCKER_IMAGE }}:reactphp . --pushThe full job:
push-image: if: (github.event_name == 'push' || github.event_name == 'schedule') && github.ref == 'refs/heads/master' name: Push reactphp to ${{ matrix.registry }} strategy: fail-fast: false matrix: registry: ${{ fromJson(needs.registry-matrix.outputs.registry) }} needs: - supported-arch-matrix - go-wild - registry-matrix runs-on: ubuntu-latest services: registry: image: registry:2 ports: - 5000:5000 steps: - name: Set up QEMU uses: docker/setup-qemu-action@v3 - name: Set up Docker Buildx uses: docker/setup-buildx-action@v3 with: driver-opts: network=host - uses: actions/download-artifact@v4 with: pattern: docker-image-reactphp-* path: /tmp/docker-image merge-multiple: true - run: | for f in /tmp/docker-image/docker_image-*.tar; do docker load --input $f done - run: rm -Rf /tmp/docker-image/ - run: | archs=${{ join(fromJson(needs.supported-arch-matrix.outputs.arch), ',') }} for arch in ${archs//,/ } do docker tag "${{ env.DOCKER_IMAGE }}:reactphp-${arch//\//-}" "localhost:5000/${{ env.DOCKER_IMAGE }}:reactphp-${arch//\//-}" docker push "localhost:5000/${{ env.DOCKER_IMAGE }}:reactphp-${arch//\//-}" done - name: Login to ${{ matrix.registry }} run: | echo "${{ env.DOCKER_PASSWORD }}" | \ docker login ${{ matrix.registry }} \ --username "${{ env.DOCKER_USER }}" \ --password-stdin env: DOCKER_USER: ${{ secrets.HUB_USERNAME }} DOCKER_PASSWORD: ${{ secrets[fromJson(env.DOCKER_IMAGE_REGISTRIES_SECRET_MAPPING)[matrix.registry]] }} - name: Create merge Dockerfile run: echo "FROM localhost:5000/${{ env.DOCKER_IMAGE }}:reactphp-\${TARGETOS}-\${TARGETARCH}" >> docker-file-${{ matrix.registry }}-wyrihaximusnet-redirect-reactphp - run: cat docker-file-${{ matrix.registry }}-wyrihaximusnet-redirect-reactphp - name: Merged different arch imags into one run: docker buildx build -f docker-file-${{ matrix.registry }}-wyrihaximusnet-redirect-reactphp --platform=${{ join(fromJson(needs.supported-arch-matrix.outputs.arch), ',') }} -t ${{ matrix.registry }}/${{ env.DOCKER_IMAGE }}:reactphp . --pushConclusion
This has been something I’ve been wanting to do for a few years, ever since I started building multi platform images. Will polish it before putting it with the rest of my centralizedGitHub Action Workflows. But for now, I’m happy 😎.