Posted onMar 6, 2021

Managing Secrets During Docker Build

#docker #secrets #credentials #security

Original post from:https://hugoprudente.github.io

What would be the best way to manage my secrets during a docker build?

Checking official and unofficial projects available inhub.docker.com,
I have collected the 4 (four) most common cases on how users are storing and managing their secrets.

There are cases that during the build you would use a token or secret file for fetch information from a
repo or other application to setup a configuration that will not be possible during runtime.

Some of those cases also doesn't fit the multistage building as fetching a package from pip.

Scenarios

I need to install apython using a private pip that I created for this lab.

To achieve that you only need to addpip.conf file as below to/root/.pip/pip.conf.

[global]index-url=https://hugo.prudente:My$3cr3tP4$$@private.pip/playlisttimeout=60extra-index-url=https://pypi.python.org/simple

Looks simple, let's see how we manage it.

Method 1

Here we copy thepip.conf to the container and don't remove it on the end.

FROM python:latestCOPY pip.conf /root/.pip/pip.confRUNpipinstallplaylist

on ⛵ k3s(nerdweek) ~/post via 🐍 v3.9.1(osx)➜ docker build-t secret:v1.on ⛵ k3s(nerdweek) ~/post via 🐍 v3.9.1(osx)➜ dockerhistorysecret:v1IMAGE          CREATED              CREATED BY                                      SIZE      COMMENT0d6589d4b95f   About a minute ago   RUN /bin/sh-c pipinstallplaylist# bui…   14.1MB    buildkit.dockerfile.v0<missing>      4 minutes ago        COPY pip.conf /root/.pip/pip.conf# buildkit    200B      buildkit.dockerfile.v0<missing>      10 days ago          /bin/sh-c#(nop)  CMD ["python3"]              0B

Let's check if the file on the end of the build is present and it was leaked.

on ⛵ k3s (nerdweek) ~/post via 🐍 v3.9.1 (osx)❯ docker run -it secret:v1 cat /root/.pip/pip.conf[global]index-url = https://hugo.prudente:My$3cr3tP4$$@private.pip/playlisttimeout=60extra-index-url = https://pypi.python.org/simple

Method 2

Here we copy thepip.conf to the container and remove it with aRUN statement on the end.

FROM python:latestCOPY pip.conf /root/.pip/pip.confRUNpipinstallplaylistRUNrm /root/.pip/pip.conf

on ⛵ k3s(nerdweek) ~/post via 🐍 v3.9.1(osx)➜ docker build-t secret:v2.on ⛵ k3s(nerdweek) ~/post via 🐍 v3.9.1(osx)➜ dockerhistorysecret:v2IMAGE          CREATED         CREATED BY                                      SIZE      COMMENT42f04cdc6577   6 seconds ago   RUN /bin/sh-crm /root/.pip/pip.conf# buil…   0B        buildkit.dockerfile.v0<missing>      4 minutes ago   RUN /bin/sh-c pipinstallplaylist# bui…   14.1MB    buildkit.dockerfile.v0<missing>      7 minutes ago   COPY pip.conf /root/.pip/pip.conf# buildkit    200B      buildkit.dockerfile.v0<missing>      10 days ago     /bin/sh-c#(nop)  CMD ["python3"]              0B

Let's check again if the file was present.

on ⛵ k3s(nerdweek) ~/post via 🐍 v3.9.1(osx)➜ docker run-it secret:v2cat /root/.pip/pip.confcat: /root/.pip/pip.conf: No such file or directory

Method 3

Here we copy thepip.conf to the container and remove it in the sameRUN statement as thepip install

FROM python:latestCOPY pip.conf /root/.pip/pip.confRUNpipinstallplaylist&&rm /root/.pip/pip.conf

on ⛵ k3s(nerdweek) ~/post via 🐍 v3.9.1(osx)➜ docker build-t secret:v3.on ⛵ k3s(nerdweek) ~/post via 🐍 v3.9.1(osx)➜ dockerhistorysecret:v3IMAGE          CREATED              CREATED BY                                      SIZE      COMMENTa2bf2672abaf   About a minute ago   RUN /bin/sh-c pipinstallplaylist&&rm…   14.1MB    buildkit.dockerfile.v0<missing>      17 minutes ago       COPY pip.conf /root/.pip/pip.conf# buildkit    200B      buildkit.dockerfile.v0<missing>      10 days ago          /bin/sh-c#(nop)  CMD ["python3"]              0B

Let's check once more if the file was present.

on ⛵ k3s(nerdweek) ~/post via 🐍 v3.9.1(osx)➜ docker run-it secret:v3cat /root/.pip/pip.confcat: /root/.pip/pip.conf: No such file or directory

Method 4

Here we create thepip.conf using thegenerate.sh script that receive the
SECRET asARG with the--build-arg options and we remove it on the sameRUN statement.

#!/bin/shSECRET=$1mkdir-p /root/.pipcat> /root/.pip/pip.conf<<EOF[global]index-url = https://hugo.prudente:${SECRET}@private.pip/playlisttimeout=60extra-index-url = https://pypi.python.org/simpleEOF

FROM python:latestARG SECRETCOPY generate.sh /generate.shRUN/generate.sh${SECRET}&& pipinstallplaylist&&rm /root/.pip/pip.conf

on ⛵ k3s(nerdweek) ~/post via 🐍 v3.9.1(osx)➜ on ⛵ k3s(nerdweek) ~/post via 🐍 v3.9.1(osx)➜ docker build-t secret:v4--progress plain--build-argSECRET=My$3cr3tP4$$.➜ on ⛵ k3s(nerdweek) ~/post via 🐍 v3.9.1(osx)➜ dockerhistory-H secret:v4IMAGE          CREATED         CREATED BY                                      SIZE      COMMENTd2ca3623139f   2 minutes ago   RUN |1SECRET=My$3cr3tP4$$ /b…   14.1MB    buildkit.dockerfile.v0<missing>      2 minutes ago   COPY generate.sh /generate.sh# buildkit        261B      buildkit.dockerfile.v0<missing>      2 minutes ago   ARG SECRET                                      0B        buildkit.dockerfile.v0<missing>      10 days ago     /bin/sh-c#(nop)  CMD ["python3"]              0B

Last but not least, let's check the presece of the file.

on ⛵ k3s(nerdweek) ~/post via 🐍 v3.9.1(osx)❯ docker run-it secret:v4cat /root/.pip/pip.confcat: /root/.pip/pip.conf: No such file or directory

The credential is not on thepip.conf file but it's visible during thedocker history.

Preliminary Results

Here is a matrix on where our secrets have been leaked.

Method	Runtime	Inspection
secret:v1	Yes	No
secret:v2	No	No
secret:v3	No	No
secret:v4	No	Yes

With the preliminary results we can already exclude methods 1 and 4 as we can
consider them insecure due to the credentials being visible at some point.

The method 4, I also used--build-args SECRET=${SECRET} and the secrets
leaks on the same way.

Dive Deep

OverlayFS

Is the kernel implementation for a union-filesystem, an overlay-filesystem tries
to present a filesystem which is the result over overlaying one filesystem on
top of the other.

In short taking the example of the image above imagine you have 2 directories
thelower andupper where thelower is a read-only directory for the
consumer, but they are still read-write from the Linux Operational system.

When a file is modified on theupper the change happens normally but if a
change is made on a file of thelower directory a copy of it is created on the
upper to become accessible, once the modification is complete another
process is responsible to fetch the modification and write on thelower
directory.

So this union of directories merged together as one unique block limited by the
cgroups is what docker and it's storage driver uses on our example.

The AUFS that's also a union-filesystem also presents the same behaviour,
although some of locations may be slightly different.

Checking the Filesystem

Now that I know how OverlayFS works let's isolate the directories (layers) used
by thepython:latest so we can filter out only the ones that we are interested
on, the ones that havepip.conf file.

Inspect the containers I can find the directories used on the OverlayFS.

on ⛵ k3s(nerdweek) ~/post via 🐍 v3.9.1(osx)➜ docker inspect python:latest |grepDir |grep-Eo"([a-z0-9]{25,64})"9e7c768dda91c4fa7ed6a57c7cb784834033bff92bd11ff6d062d4de11c0f89817bf53c98685ae36487eb55f0d2256d168f210a688ef51deef760de1a699cbdf4cb1dbbf58a2b1ca8df6d9d977a66fe918aee21434fcd656f1a68f1f412d75ff358dd0944f115e2a273c5259dd1432b44e36908cf223f8ce0d9f74550430f577c034592b1a26552525742ed81e7fbce2139817b634d48db8349dbebf15a4591419d471e0407c0f1ca14eb1cb8c46aaef9357037cad5dc170cb6a4af3c1feab40e005796f193e62e9db78de1df20999daca1a96a0bebed19c1dd906b1b4da8542badc6aa65b2d3f10b0cdff3fc04bf3a64b551af1dd9e01b6ecd38ed71abdc3da8d6ff96b718838005288a94cdc9fd408d1f70d7e9cbab678ebeb4521d11b366don ⛵ k3s(nerdweek) ~/post via 🐍 v3.9.1(osx)➜ docker inspect python:latest |grepDir |grep-Eo"([a-z0-9]{25,64})"> layers.python

With the layers saved in the filelayers.python I can use a similar command to
exclude the know python layers and get only the ones added by our build.

on ⛵ k3s(nerdweek) ~/post via 🐍 v3.9.1(osx)➜ docker inspect secret:v1 |grepDir |grep-Eo"([a-z0-9]{25,64})" |grep-v-f layers.python  |uniqe1kq2j71b7clcwtn0lbmqa1g9v6zy2xgzrow2mgpyq9d0vch6l➜ docker inspect secret:v2 |grepDir |grep-Eo"([a-z0-9]{25,64})" |grep-v-f layers.python  |uniqv6zy2xgzrow2mgpyq9d0vch6le1kq2j71b7clcwtn0lbmqa1g9oudofb9c0iaqog9sff81f8053on ⛵ k3s(nerdweek) ~/post via 🐍 v3.9.1(osx)➜ docker inspect secret:v3 |grepDir |grep-Eo"([a-z0-9]{25,64})" |grep-v-f layers.python  |uniqe1kq2j71b7clcwtn0lbmqa1g9hqsrze873a2uz7tjsgbqdo3sdon ⛵ k3s(nerdweek) ~/post via 🐍 v3.9.1(osx)➜ docker inspect secret:v4 |grepDir |grep-Eo"([a-z0-9]{25,64})" |grep-v-f layers.python  |uniq78gz619okusgrq4jo4ek863ugoq22x88p26hzqmr1w1qpf8mm4

Now that we have the layers I have created a list just to make it simpler to
find it when we check the directory.

Accessing the OverlayFS directories

Using one of the commands below as root you will find the entry point where
Docker Storage driver creates the file-system hierarchy used by the whole
eco-system.

Linux

cd /var/lib/docker/

MacOS

docker run-it--rm--privileged--pid=host justincormack/nsenter1cd /var/lib/docker/

Once in the/var/lib/docker directory using a simplels and filtering the
layers previous stored in the temporary file and expanding it I could find the
specific layers that have thepip.conf

/var/lib/docker/overlay2# ls | grep -f /tmp/layers | xargs find | grep pip.confe1kq2j71b7clcwtn0lbmqa1g9/diff/root/.pip/pip.confhqsrze873a2uz7tjsgbqdo3sd/diff/root/.pip/pip.confoudofb9c0iaqog9sff81f8053/diff/root/.pip/pip.conf

So I accessed each of of those to confirm if the file was present or if was just
its shadow left by the directory union.

/var/lib/docker/overlay2# cat e1kq2j71b7clcwtn0lbmqa1g9/diff/root/.pip/pip.conf[global]index-url= https://hugo.prudente:My$3cr3tP4$$@private.pip/playlisttimeout=60extra-index-url= https://pypi.python.org/simpl/var/lib/docker/overlay2# cat hqsrze873a2uz7tjsgbqdo3sd/diff/root/.pip/pip.confcat: can\'t open'hqsrze873a2uz7tjsgbqdo3sd/diff/root/.pip/pip.conf': No such device or address/var/lib/docker/overlay2# cat oudofb9c0iaqog9sff81f8053/diff/root/.pip/pip.confcat: can\'t open'oudofb9c0iaqog9sff81f8053/diff/root/.pip/pip.conf': No such device or address

So 1 of 3 layers have the file present so I have checked from which container
that layer belong to and here's the surprise.

That 1 layer is shared with 3 of 4 builds that we have created, meaning that
during adocker pull three diferente containers could leak mypip.conf
secret.

Results

The updated matrix consolidating the results on where our secrets have leaked.

Method	Runtime	Inspection	OverlayFS
secret:v1	Yes	No	Yes
secret:v2	No	No	Yes
secret:v3	No	No	Yes
secret:v4	NO	Yes	No

So even knowing that the file is not acessible from the container directly if
you have access to pull the container on a full read-write system you would be
able to retreive the secrets.

But now what's the best way to build the container and do not have such issue?

Solution

From 18.09 or newer Docker have introduced the Docker BuildKit that brings some
extra funcionality to the Docker builds.

The builds using BuildKit different from the legacy allows the usage of the
--secret that allows the capacity of binding a file during build runtime
similar to the tradicional runtime that we achieve with-v option.

It's usage is quite simple let's build a container and run our tests again.

FROM python:latestRUN--mount=type=secret,id=pip.conf,dst=/root/.pip/pip.conf\      pipinstallplaylist

on ⛵ k3s(nerdweek) ~/post via 🐍 v3.9.1(osx)➜ docker build--file Dockerfile--secretid=pip.conf,src=pip.conf-t secret:v5.

Now that we have thesecret:v5 build lets confirm.

on ⛵ k3s(nerdweek) ~/post via 🐍 v3.9.1(osx)➜ dockerhistorysecret:v5IMAGE          CREATED          CREATED BY                                      SIZE      COMMENT266a21bb36ae   36 seconds ago   RUN /bin/sh-c pipinstallplaylist# bui…   14.1MB    buildkit.dockerfile.v0<missing>      11 days ago      /bin/sh-c#(nop)  CMD ["python3"]              0Bi

The history in this case is clean, not even mention the mount forpip.conf

on ⛵ k3s(nerdweek) ~/post via 🐍 v3.9.1(osx)❯ docker run-it secret:v5cat /root/.pip/pip.confcat: /root/.pip/pip.conf: No such file or directory

The file is also not present on the system.

on ⛵ k3s(nerdweek) ~/post via 🐍 v3.9.1(osx)❯ docker inspect secret:v5 |grepDir |grep-Eo"([a-z0-9]{25,64})" |grep-v-f layers.python  |uniqcnpw0dw9o05lmdz3j9j62jzpton ⛵ k3s(nerdweek) ~/post via 🐍 v3.9.1(osx)/var/lib/docker/overlay2#lscnpw0dw9o05lmdz3j9j62jzpt | xargs find |greppip.conffind: committed: No such file or directoryfind: diff: No such file or directoryfind:link: No such file or directoryfind: lower: No such file or directoryfind: work: No such file or directory

And the most important one the file doesn't exist on the layer/directory that we just
created meaning that if we use in a base image our scretes are safe.