Repository files navigation

Envoy proxy for Cilium withminimal Envoy extensions and Cilium policy enforcement filters. Ciliumuses this as its host proxy for enforcing HTTP and other L7 policiesas specified innetworkpoliciesfor the cluster. Cilium proxy is distributed within the Cilium images.

The following table shows the Cilium proxy version compatibility with supported upstream Cilium versions.Other combinations may work but are not tested.

Cilium proxy is best built with the provided build containers. For alocal host build consultthe builderDockerfilefor the required dependencies.

Container builds require Docker Buildkit and optionally Buildx formulti-arch builds. Builds are currently only supported for amd64 andarm64 targets. For arm64 both native and cross compile on amd64 aresupported. Container builds produce container images bydefault. These images can not be run by themselves as they do notcontain the required runtime dependencies. To run the Cilium proxy thebinary/usr/bin/cilium-envoy needs to be copied from the image to acompatible runtime environment, such as Ubuntu 20.04, or 22.04.

The provided container build tools work on both Linux and macOS.

To build the Cilium proxy in a docker container for the hostarchitecture only:

make docker-image-envoy

This will write the image to the local Docker registry.

Depending on hour host CPU and memory resources a fresh build can takean hour or more. Docker caching will speed up subsequent builds.

If your build fails due to a compiler failure the most likely reasonis the compiler running out of memory. You can mitigate this bylimiting the number of concurrent build jobs by passing environmentvariableBAZEL_BUILD_OPTS=--jobs=2 tomake. By default thenumber of jobs is the number of CPUs available for the build, andfor some complex C++ sources this may be too much. Note thatchanging the value ofBAZEL_BUILD_OPTS invalidates Docker cachesfor the build stages.

Build target architecture can be specified by passingARCHenvironment variable tomake. Supported values areamd64 (only onamd64 hosts),arm64 (on arm64 or amd64 hosts), andmulti (on amd64hosts).multi builds for all the supported architectures, currrentlyamd64 and arm64:

ARCH=multi make docker-image-envoy

This will try to push the images to the container registry. Appropriateauthentication is required. (Pushing to the local Docker registry isn'tsupported for multi-arch builds. SeeDocker documentation)

Builds will be performed concurrently when building for multiplearchitectures on a single machine. You most likely need to limit thenumber of jobs allowed for each builder, see the note above fordetails.

Docker builds are done using Docker Buildx by default whenARCH isexplicitly passed tomake. You can also force Docker Buildx to beused when building for the host platform only (by not definingARCH)by definingDOCKER_BUILDX=1. A new buildx builder instance will becreated for amd64 and arm64 cross builds if the current builder is settodefault.

Buildx builds will push the build result toquay.io/cilium/cilium-envoy:<GIT_SHA> by default. You can changethe first two parts of this by definingDOCKER_DEV_ACCOUNT=docker.io/me for your own docker hub account.You can also request the build results to be output to your localdirectory instead by definingDOCKER_BUILD_OPTS=--output=out,whereout is a local directory name or useDOCKER_BUILD_OPTS="--output=type=docker" to load it into thelocal Docker daemon.

By default Raspberry Pi OS and other OSes using theRaspberry Pi kernel willnot be able to use Envoy as their defaultCONFIG_ARM64_VA_BITS_39configurationis not compatible with tcmalloc.

A workaround is to compile the Envoy proxy withgperftools:

ARCH=arm64 BAZEL_BUILD_OPTS="--define tcmalloc=gperftools" make docker-image-envoy

This image can then be used in theEnvoy DaemonSet mode.

Docker build uses cached Bazel artifacts fromquay.io/cilium/cilium-envoy-builder:main-archive-latest bydefault. You can override this by definingARCHIVE_IMAGE=<ref>:

ARCH=multi ARCHIVE_IMAGE=docker.io/me/cilium-envoy-archive make docker-image-envoy

Bazel build artifacts contain toolchain specific data and binariesthat are not compatible between native and cross-compiledbuilds. For now the image ref shown above is for builds on amd64only (native amd64, cross-compiled arm64).

DefineNO_CACHE=1 to clear the local build cache before the build, andNO_ARCHIVE=1 to buildfrom scratch, but be warned that this can take a long time.

By default the build also tries to pull Docker build caches fromdocker.io/cilium/cilium-dev:cilium-envoy-cache. You can overridethis with our own build cache, which you can also update with theCACHE_PUSH=1 definition:

ARCH=multi CACHE_REF=docker.io/me/cilium-proxy:cache CACHE_PUSH=1 make docker-image-envoy

NO_CACHE=1 can be used to disable docker cache pulling.

In a CI environment it might be a good idea to push a new cache imageafter each main branch commit.

Build and push a new version of the pre-compiled Envoy dependencies by:

ARCH=multi make docker-builder-archive

By default the pre-compiled dependencies image is tagged asquay.io/cilium/cilium-envoy-builder:main-archive-latest. Youcan override the first two parts of this by definingDOCKER_DEV_ACCOUNT=docker.io/me,BUILDER_ARCHIVE_TAG=my-builder-archive, or completely by definingARCHIVE_IMAGE=<ref>.

Pre-compiled Envoy dependencies need to be updated only when Envoyversion is updated or patched enough to increase compilation timesignificantly. To do this you should update Envoy version inENVOY_VERSION and supplyNO_CACHE=1 andNO_ARCHIVE=1 on the make line, e.g.:

ARCH=multi NO_CACHE=1 NO_ARCHIVE=1 BUILDER_ARCHIVE_TAG=main-archive-latest make docker-builder-archive

The required Bazel version typically changes from one Envoy release toanother. To create a new builder image first update the required Bazelversion at.bazelversion and then run:

ARCH=multi NO_CACHE=1 NO_ARCHIVE=1 make docker-image-builder

The builder can not be cross-compiled as native build tools are neededfor native arm64 builds. This means that for non-native builds QEMUCPU emulation is used instead of cross-compilation. If you have anarm64 machine you can create a Docker buildx builder to use it fornative builds.

The builder image is tagged as"quay.io/cilium/cilium-envoy-builder:bazel-". Change theBUILDER_BASE ARG inDockerfile to use the new builder and commit theresult.

For testing purposes you can defineDOCKER_DEV_ACCOUNT as explainedabove to push the builder into a different registry or account.

To run Cilium Envoy integration tests in a docker container:

make docker-tests

This runs the integration tests after loading Bazel build cache forEnvoy dependencies fromquay.io/cilium/cilium-envoy-builder:test-main-archive-latest. DefineNO_ARCHIVE=1 andNO_CACHE=1 to compile tests from scratch.

This command fails if any of the integration tests fail, printing thefailing test logs on console.

Note that cross-compiling is not supported for running tests, sospecifyingARCH is only supported for the native platform.ARCH=multi will fail.

Build and push a new version of the pre-compiled test dependencies by:

make docker-tests-archive

By default the pre-compiled test dependencies image is tagged asquay.io/cilium/cilium-envoy-builder:test-main-archive-latest. Youcan override the first two parts of this by definingDOCKER_DEV_ACCOUNT=docker.io/me,TESTS_ARCHIVE_TAG=my-test-archive, or completely by definingARCHIVE_IMAGE=<ref>.

Pre-compiled Envoy test dependencies need to be updated only whenEnvoy version is updated or patched enough to increase compilationtime significantly. To do this you should update Envoy versioninENVOY_VERSION and supplyNO_ARCHIVE=1 andNO_CACHE=1 onthe make line, e.g.:

ARCH=amd64 NO_ARCHIVE=1 NO_CACHE=1 make docker-tests-archive

Cilium project vendors the EnvoyxDS API, including Cilium extensions, from this repository. To updatethe generated API files, run:

rm -r go/envoy/*make api

rm is needed to clean up API files that are no longer generated forEnvoy.Do not remove files atgo/cilium/ as some of them are notautomatically generated!

Commit the results and updateCilium to vendor this new commit.