Use the Kubernetes executor to use Kubernetes clusters for your builds. The executor calls the Kubernetescluster API and creates a pod for each GitLab CI job.

The Kubernetes executor divides the build into multiple steps:

1. Prepare: Create the Pod against the Kubernetes Cluster.This creates the containers required for the build and services to run.
2. Pre-build: Clone, restore cache, and download artifacts from previousstages. This step runs on a special container as part of the pod.
3. Build: User build.
4. Post-build: Create cache, upload artifacts to GitLab. This step also usesthe special container as part of the pod.

How the runner creates Kubernetes pods

The following diagram shows the interaction between a GitLab instance and a runner hosted on a Kubernetes cluster. The runner calls the Kubernetes API to create pods on the cluster.

The pod consists of the following containers for eachservice defined in the.gitlab-ci.yml orconfig.toml files:

• A build container defined asbuild.
• A helper container defined ashelper.
• A services containers defined assvc-X, whereX is[0-9]+.

Services and containers run in the same Kubernetespod and share the same localhost address. The following restrictions apply:

• The services are accessible through their DNS names. If youuse an older version, you must uselocalhost.
• You cannot use several services that use the same port. For example, you cannot have twomysql services at the same time.

sequenceDiagram    participant G as GitLab instance    participant R as Runner on Kubernetes cluster    participant Kube as Kubernetes API    participant P as POD    R->>+G: Get a CI job.        loop        G-->R: ;    end    Note over R,G: POST /api/v4/jobs/request    G->>+R: CI job data.    R-->>-Kube: Create a POD to run the CI job.    Note over R,Kube: POST to Kube API    P->>+P: Execute job.    Note over P: CI build job = Prepare + Pre-build + Build + Post-build    P->>+G: Job logs

The interaction in the diagram is valid for any Kubernetes cluster. For example, turnkeysolutions hosted on the major public cloud providers, or self-managed Kubernetes installations.

Connect to the Kubernetes API

Use the following options to connect to the Kubernetes API. The user account provided must havepermission to create, list, and attach to Pods in the specified namespace.

If you’re running GitLab Runner in the Kubernetes cluster, omitthese fields so that the GitLab Runner auto-discovers the Kubernetes API.

If you’re running GitLab Runner externally to the Cluster, these settings ensure that GitLab Runnerhas access to the Kubernetes API on the cluster.

Set the bearer token for Kubernetes API calls

To set the bearer token for API calls to create pods, use theKUBERNETES_BEARER_TOKENvariable. This allows project owners to use project secret variables to specify a bearer token.

When specifying the bearer token, you mustset theHost configuration setting.

variables:KUBERNETES_BEARER_TOKEN:thebearertokenfromanothernamespace

Configure runner API permissions

To configure permissions for the core API group, update thevalues.yml file for GitLab Runner Helm charts.

You can either:

• Setrbac.create totrue.
• Specify a service accountserviceAccount.name: <service_account_name> with the followingpermissions in thevalues.yml file.

You can use the following YAML role definition to create a role with the required permissions.

apiVersion:rbac.authorization.k8s.io/v1kind:Rolemetadata:name:gitlab-runnernamespace:defaultrules:-apiGroups:[""]resources:["events"]verbs:-"list"# Required when `print_pod_warning_events=true`-"watch"# Required when `FF_PRINT_POD_EVENTS=true`-apiGroups:[""]resources:["namespaces"]verbs:-"create"# Required when `kubernetes.NamespacePerJob=true`-"delete"# Required when `kubernetes.NamespacePerJob=true`-apiGroups:[""]resources:["pods"]verbs:-"create"-"delete"-"get"-"list"# Required when using Informers (https://docs.gitlab.com/runner/executors/kubernetes/#informers)-"watch"# Required when `FF_KUBERNETES_HONOR_ENTRYPOINT=true`, `FF_USE_LEGACY_KUBERNETES_EXECUTION_STRATEGY=false`, using Informers (https://docs.gitlab.com/runner/executors/kubernetes/#informers)-apiGroups:[""]resources:["pods/attach"]verbs:-"create"# Required when `FF_USE_LEGACY_KUBERNETES_EXECUTION_STRATEGY=false`-"delete"# Required when `FF_USE_LEGACY_KUBERNETES_EXECUTION_STRATEGY=false`-"get"# Required when `FF_USE_LEGACY_KUBERNETES_EXECUTION_STRATEGY=false`-"patch"# Required when `FF_USE_LEGACY_KUBERNETES_EXECUTION_STRATEGY=false`-apiGroups:[""]resources:["pods/exec"]verbs:-"create"-"delete"-"get"-"patch"-apiGroups:[""]resources:["pods/log"]verbs:-"get"# Required when `FF_KUBERNETES_HONOR_ENTRYPOINT=true`, `FF_USE_LEGACY_KUBERNETES_EXECUTION_STRATEGY=false`, `FF_WAIT_FOR_POD_TO_BE_REACHABLE=true`-"list"# Required when `FF_KUBERNETES_HONOR_ENTRYPOINT=true`, `FF_USE_LEGACY_KUBERNETES_EXECUTION_STRATEGY=false`-apiGroups:[""]resources:["secrets"]verbs:-"create"-"delete"-"get"-"update"-apiGroups:[""]resources:["serviceaccounts"]verbs:-"get"-apiGroups:[""]resources:["services"]verbs:-"create"-"get"

• Theevent permission is needed only:

  • For GitLab 16.2.1 and later.

• Thenamespace permission is needed only:

  • When enabling namespace isolation by usingnamespace_per_job.

• Thepods/log permission is only needed when one of the following scenarios are true:

  • TheFF_KUBERNETES_HONOR_ENTRYPOINT feature flag is enabled.

  • TheFF_USE_LEGACY_KUBERNETES_EXECUTION_STRATEGY feature flag is disabled when theCI_DEBUG_SERVICES variable is set totrue.

  • TheFF_WAIT_FOR_POD_TO_BE_REACHABLE feature flag is enabled.

Informers

In GitLab Runner 17.9.0 and later, a Kubernetes informer tracks build podchanges. This helps the executor detect the changes more quickly.

The informer requireslist andwatch permissions forpods. When the executorstarts the build, it checks the Kubernetes API for the permissions.If all permissions are granted, the executor uses an informer.If any permission is missing, GitLab Runner logs a warning. The build continuesand uses the previous mechanism to track the build pod’s status and changes.

Configuration settings

Use the following settings in theconfig.toml file to configure the Kubernetes executor.

CPU requests and limits

Memory requests and limits

Helper container memory sizing recommendations

For optimal performance, set helper container memory limits based on your workload requirements:

• Workloads with caching and artifact generation: Minimum 250 MiB
• Basic workloads without cache/artifacts: Might work with lower limits (128-200 MiB)

Basic configuration example:

[[runners]]  executor ="kubernetes"  [runners.kubernetes]    helper_memory_limit ="250Mi"    helper_memory_request ="250Mi"    helper_memory_limit_overwrite_max_allowed ="1Gi"

Job-specific memory overrides:

Use theKUBERNETES_HELPER_MEMORY_LIMIT variable to adjust memory for specific jobs without requiring administrator changes:

job_with_higher_helper_memory_limit:variables:KUBERNETES_HELPER_MEMORY_LIMIT:"512Mi"script:

This approach allows developers to optimize resource usage per job while maintaining cluster-wide limits throughhelper_memory_limit_overwrite_max_allowed.

Storage requests and limits

Otherconfig.toml settings

Configuration example

The following sample shows an example configuration of theconfig.toml filefor the Kubernetes executor.

concurrent =4[[runners]]  name ="myRunner"  url ="https://gitlab.com/ci"  token ="......"  executor ="kubernetes"  [runners.kubernetes]    host ="https://45.67.34.123:4892"    cert_file ="/etc/ssl/kubernetes/api.crt"    key_file ="/etc/ssl/kubernetes/api.key"    ca_file ="/etc/ssl/kubernetes/ca.crt"    namespace ="gitlab"    namespace_overwrite_allowed ="ci-.*"    bearer_token_overwrite_allowed =true    privileged =true    cpu_limit ="1"    memory_limit ="1Gi"    service_cpu_limit ="1"    service_memory_limit ="1Gi"    helper_cpu_limit ="500m"    helper_memory_limit ="100Mi"    poll_interval =5    poll_timeout =3600    dns_policy ="cluster-first"    priority_class_name ="priority-1"    logs_base_dir ="/tmp"    scripts_base_dir ="/tmp"    [runners.kubernetes.node_selector]      gitlab ="true"    [runners.kubernetes.node_tolerations]"node-role.kubernetes.io/master" ="NoSchedule""custom.toleration=value" ="NoSchedule""empty.value=" ="PreferNoSchedule""onlyKey" =""

Configure the executor service account

To configure the executor service account, you can set theKUBERNETES_SERVICE_ACCOUNT environment variable or use the--kubernetes-service-account flag.

Pods and containers

You can configure pods and containers to control how jobs are executed.

Default labels for job pods

Default annotations for job pods

The following annotations are added by default on the Pod running the jobs:

To overwrite default annotations, use thepod_annotations in the GitLab Runner configuration.You can also overwrite annotations for each CI/CD job in the.gitlab-ci.yml file.

Pod lifecycle

Apod’s lifecyclecan be affected by:

• Setting thepod_termination_grace_period_seconds property in theTOML configuration file.The process running on the pod can run for the given duration after theTERM signal is sent.A kill signal is sent if the Pod is not successfully terminated after this period of time.
• Enabling theFF_USE_POD_ACTIVE_DEADLINE_SECONDS feature flag.When enabled and the job times out, the pod running the CI/CD job is marked asfailed and all associated containers are killed. To have the job time out on GitLab first,activeDeadlineSeconds is set toconfigured timeout + 1 second.

Overwrite pod tolerations

To overwrite Kubernetes pod tolerations:

1. In theconfig.toml or Helmvalues.yaml file, to enable the overwrite of CI job pod tolerations, define a regular expression fornode_tolerations_overwrite_allowed.This regular expression validates the values of CI variable names that start withKUBERNETES_NODE_TOLERATIONS_.

   runners: ... config:|   [[runners]]     [runners.kubernetes]       node_tolerations_overwrite_allowed =".*"

2. In the.gitlab-ci.yml file, define one or more CI variables to overwrite CI job pod tolerations.

   variables:KUBERNETES_NODE_TOLERATIONS_1:'node-role.kubernetes.io/master:NoSchedule'KUBERNETES_NODE_TOLERATIONS_2:'custom.toleration=value:NoSchedule'KUBERNETES_NODE_TOLERATIONS_3:'empty.value=:PreferNoSchedule'KUBERNETES_NODE_TOLERATIONS_4:'onlyKey'KUBERNETES_NODE_TOLERATIONS_5:''# tolerate all taints

Overwrite pod labels

To overwrite Kubernetes pod labels for each CI/CD job:

1. In the.config.yaml file, define a regular expression forpod_labels_overwrite_allowed.

2. In the.gitlab-ci.yml file, set theKUBERNETES_POD_LABELS_* variables with values ofkey=value. The pod labels are overwritten to thekey=value. You can apply multiple values:

   variables:KUBERNETES_POD_LABELS_1:"Key1=Val1"KUBERNETES_POD_LABELS_2:"Key2=Val2"KUBERNETES_POD_LABELS_3:"Key3=Val3"

Overwrite pod annotations

To overwrite Kubernetes pod annotations for each CI/CD job:

1. In the.config.yaml file, define a regular expression forpod_annotations_overwrite_allowed.

2. In the.gitlab-ci.yml file, set theKUBERNETES_POD_ANNOTATIONS_* variables and usekey=value for the value.Pod annotations are overwritten to thekey=value. You can specify multiple annotations:

   variables:KUBERNETES_POD_ANNOTATIONS_1:"Key1=Val1"KUBERNETES_POD_ANNOTATIONS_2:"Key2=Val2"KUBERNETES_POD_ANNOTATIONS_3:"Key3=Val3"

In the example below, thepod_annotations and thepod_annotations_overwrite_allowed are set.This configuration allows overwrite of any of thepod_annotations configured in theconfig.toml.

[[runners]]# usual configuration  executor ="kubernetes"  [runners.kubernetes]    image ="alpine"    pod_annotations_overwrite_allowed =".*"    [runners.kubernetes.pod_annotations]"Key1" ="Val1""Key2" ="Val2""Key3" ="Val3""Key4" ="Val4"

Overwrite generated pod specifications

This feature is inbeta. We strongly recommend that you usethis feature on a test Kubernetes cluster before you use it on a production cluster. To use this feature, you mustenable theFF_USE_ADVANCED_POD_SPEC_CONFIGURATIONfeature flag.

To add feedback for improvements before the feature is made generally available, usethis issue.

To modify thePodSpec generated by the runner manager, use thepod_spec setting in theconfig.toml file.

Thepod_spec setting:

• Overwrites and completes fields for the generated pod specification.
• Overwrites configuration values that might have been set in yourconfig.toml under[runners.kubernetes].

You can configure multiplepod_spec settings.

You cannot set thepatch_path andpatch in the samepod_spec configuration, otherwise an error occurs.

Example of multiplepod_spec configurations in theconfig.toml:

[[runners]]  [runners.kubernetes]    [[runners.kubernetes.pod_spec]]      name ="hostname"      patch ='''        hostname: "custom-pod-hostname"      '''      patch_type ="merge"    [[runners.kubernetes.pod_spec]]      name ="subdomain"      patch ='''        subdomain: "subdomain"      '''      patch_type ="strategic"    [[runners.kubernetes.pod_spec]]      name ="terminationGracePeriodSeconds"      patch ='''        [{"op": "replace", "path": "/terminationGracePeriodSeconds", "value": 60}]      '''      patch_type ="json"

Merge patch strategy

Themerge patch strategy appliesa key-value replacement on the existingPodSpec.If you use this strategy, thepod_spec configuration in theconfig.tomloverwrites the values in the finalPodSpecobject before it is generated. Because the values are completely overwritten, you should use this patch strategy with caution.

Example of apod_spec configuration with themerge patch strategy:

concurrent =1check_interval =1log_level ="debug"shutdown_timeout =0[session_server]  session_timeout =1800[[runners]]  name =""  url ="https://gitlab.example.com"  id =0  token ="__REDACTED__"  token_obtained_at = 0001-01-01T00:00:00Z  token_expires_at = 0001-01-01T00:00:00Z  executor ="kubernetes"  shell ="bash"  environment = ["FF_USE_ADVANCED_POD_SPEC_CONFIGURATION=true","CUSTOM_VAR=value"]  [runners.kubernetes]    image ="alpine"    ...    [[runners.kubernetes.pod_spec]]      name ="build envvars"      patch ='''        containers:        - env:          - name: env1            value: "value1"          - name: env2            value: "value2"          name: build      '''      patch_type ="merge"

With this configuration, the finalPodSpec has only one container calledbuild with two environment variablesenv1 andenv2. The example above make the related CI Job failed as:

• Thehelper container specification is removed.
• Thebuild container specification lost all necessary configuration set by GitLab Runner.

To prevent the job from failing, in this example, thepod_spec must contain the untouched properties generated by GitLab Runner.

JSON patch strategy

Thejson patch strategy uses theJSON Patch specificationto give control over thePodSpec objects and arrays to update. You cannot use this strategy onarray properties.

Example of apod_spec configuration with thejson patch strategy. In this configuration,a newkey: value pair is added to the existingnodeSelector. The existing values are not overwritten.

concurrent =1check_interval =1log_level ="debug"shutdown_timeout =0[session_server]  session_timeout =1800[[runners]]  name =""  url ="https://gitlab.example.com"  id =0  token ="__REDACTED__"  token_obtained_at = 0001-01-01T00:00:00Z  token_expires_at = 0001-01-01T00:00:00Z  executor ="kubernetes"  shell ="bash"  environment = ["FF_USE_ADVANCED_POD_SPEC_CONFIGURATION=true","CUSTOM_VAR=value"]  [runners.kubernetes]    image ="alpine"    ...    [[runners.kubernetes.pod_spec]]      name ="val1 node"      patch ='''        [{ "op": "add", "path": "/nodeSelector", "value": { key1: "val1" } }]      '''      patch_type ="json"

Strategic patch strategy

Thisstrategic patch strategy uses the existingpatchStrategy applied to each field of thePodSpec object.

Example of apod_spec configuration with thestrategic patch strategy. In this configuration,aresource request is set to on the build container.

concurrent =1check_interval =1log_level ="debug"shutdown_timeout =0[session_server]  session_timeout =1800[[runners]]  name =""  url ="https://gitlab.example.com"  id =0  token ="__REDACTED__"  token_obtained_at = 0001-01-01T00:00:00Z  token_expires_at = 0001-01-01T00:00:00Z  executor ="kubernetes"  shell ="bash"  environment = ["FF_USE_ADVANCED_POD_SPEC_CONFIGURATION=true","CUSTOM_VAR=value"]  [runners.kubernetes]    image ="alpine"    ...    [[runners.kubernetes.pod_spec]]      name ="cpu request 500m"      patch ='''        containers:        - name: build          resources:            requests:              cpu: "500m"      '''      patch_type ="strategic"

With this configuration, aresource request is set to on the build container.

Best practices

• Test the addedpod_spec in a test environment before deployment in a production environment.
• Make sure that thepod_spec configuration does not negatively impact the GitLab Runner generated specification.
• Do not use themerge patch strategy for complex pod specification updates.
• Where possible, use theconfig.toml when the configuration is available. For example, the following configuration replaces the first environment variables set by GitLab Runner by the one set in the custompod_spec instead of adding the environment variable set to the existing list.

concurrent =1check_interval =1log_level ="debug"shutdown_timeout =0[session_server]  session_timeout =1800[[runners]]  name =""  url ="https://gitlab.example.com"  id =0  token ="__REDACTED__"  token_obtained_at = 0001-01-01T00:00:00Z  token_expires_at = 0001-01-01T00:00:00Z  executor ="kubernetes"  shell ="bash"  environment = ["FF_USE_ADVANCED_POD_SPEC_CONFIGURATION=true","CUSTOM_VAR=value"]  [runners.kubernetes]    image ="alpine"    ...    [[runners.kubernetes.pod_spec]]      name ="build envvars"      patch ='''        containers:        - env:          - name: env1            value: "value1"          name: build      '''      patch_type ="strategic"

Create aPVC for each build job by modifying the Pod Spec

To create aPersistentVolumeClaim for each build job make sure to check out how to enablethePod Spec functionality.

Kubernetes allows you to create an ephemeralPersistentVolumeClaim attached to a pod’s lifecycle.This approach works ifdynamic provisioning is enabled on your Kubernetes cluster.EachPVC can request a newVolume. The volume is also tied to the pod’s lifecycle.

Afterdynamic provisioning is enabled, theconfig.toml can be modified as follows to create anephemeralPVC:

[[runners.kubernetes.pod_spec]]  name ="ephemeral-pvc"  patch ='''    containers:    - name: build      volumeMounts:      - name: builds        mountPath: /builds    - name: helper      volumeMounts:      - name: builds        mountPath: /builds    volumes:    - name: builds      ephemeral:        volumeClaimTemplate:          spec:            storageClassName: <The Storage Class that will dynamically provision a Volume>            accessModes: [ ReadWriteOnce ]            resources:              requests:                storage: 1Gi  '''

Set a security policy for the pod

Configure thesecurity contextin theconfig.toml to set a security policy for the build pod.

Use the following options:

Example of a pod security context in theconfig.toml:

concurrent =%(concurrent)scheck_interval =30  [[runners]]    name ="myRunner"    url ="gitlab.example.com"    executor ="kubernetes"    [runners.kubernetes]      helper_image ="gitlab-registry.example.com/helper:latest"      [runners.kubernetes.pod_security_context]        run_as_non_root =true        run_as_user =59417        run_as_group =59417        fs_group =59417

Remove old runner pods

Sometimes old runner pods are not cleared. This can happen when the runner manager is incorrectly shut down.

To handle this situation, you can use the GitLab Runner Pod Cleanup application to schedule cleanup of old pods. For more information, see:

• The GitLab Runner Pod Cleanup projectREADME.
• GitLab Runner Pod Cleanupdocumentation.

Set a security policy for the container

Configure thecontainer security contextin theconfig.toml executor to set a container security policy for the build, helper, or service pods.

Use the following options:

In the following example in theconfig.toml, the security context configuration:

• Sets a pod security context.
• Overridesrun_as_user andrun_as_group for the build and helper containers.
• Specifies that all service containers inheritrun_as_user andrun_as_group from the pod security context.

concurrent =4check_interval =30  [[runners]]    name ="myRunner"    url ="gitlab.example.com"    executor ="kubernetes"    [runners.kubernetes]      helper_image ="gitlab-registry.example.com/helper:latest"      [runners.kubernetes.pod_security_context]        run_as_non_root =true        run_as_user =59417        run_as_group =59417        fs_group =59417      [runners.kubernetes.init_permissions_container_security_context]        run_as_user =1000        run_as_group =1000      [runners.kubernetes.build_container_security_context]        run_as_user =65534        run_as_group =65534        [runners.kubernetes.build_container_security_context.capabilities]          add = ["NET_ADMIN"]      [runners.kubernetes.helper_container_security_context]        run_as_user =1000        run_as_group =1000      [runners.kubernetes.service_container_security_context]        run_as_user =1000        run_as_group =1000

Set a pull policy

Use thepull_policy parameter in theconfig.toml file to specify a single or multiple pull policies.The policy controls how an image is fetched and updated, and applies to the build image, helper image, and any services.

To determine which policy to use, seethe Kubernetes documentation about pull policies.

For a single pull policy:

[runners.kubernetes]  pull_policy ="never"

For multiple pull policies:

[runners.kubernetes]# use multiple pull policies  pull_policy = ["always","if-not-present"]

When you define multiple policies, each policy is attempted until the image is obtained successfully.For example, when you use[ always, if-not-present ], the policyif-not-present is used if thealways policy fails due to a temporary registry problem.

To retry a failed pull:

[runners.kubernetes]  pull_policy = ["always","always"]

The GitLab naming convention is different to the Kubernetes one.

Specify container capabilities

You can specify theKubernetes capabilitiesto use in the container.

To specify the container capabilities, use thecap_add andcap_drop options in theconfig.toml. Container runtimes can alsodefine a default list of capabilities, like those inDockeror thecontainer.

There is alist of capabilities that the runner drops by default.Capabilities that you list incap_add option are excluded from being dropped.

Example configuration in theconfig.toml file:

concurrent =1check_interval =30[[runners]]  name ="myRunner"  url ="gitlab.example.com"  executor ="kubernetes"  [runners.kubernetes]# ...    cap_add = ["SYS_TIME","IPC_LOCK"]    cap_drop = ["SYS_ADMIN"]# ...

When you specify the capabilities:

• User-definedcap_drop has priority over user-definedcap_add. If you define the same capability in both settings,only the capability fromcap_drop is passed to the container.
• Remove theCAP_ prefix from capability identifiers passed to the container configuration.For example, if you want to add or drop theCAP_SYS_TIME capability,in the configuration file, enter the string,SYS_TIME.
• The owner of the Kubernetes clustercan define a PodSecurityPolicy,where specific capabilities are allowed, restricted, or added by default. These rules take precedence over any user-defined configuration.

Overwrite container resources

You can overwrite Kubernetes CPU and memory allocations for each CI/CDjob. You can apply settings for requests and limits for the build, helper, and service containers.

To overwrite container resources, use the following variables in the.gitlab-ci.yml file.

The values for the variables are restricted to themaximum overwritesetting for that resource. If the maximum overwrite has not been set for a resource, the variable is not used.

variables:KUBERNETES_CPU_REQUEST:"3"KUBERNETES_CPU_LIMIT:"5"KUBERNETES_MEMORY_REQUEST:"2Gi"KUBERNETES_MEMORY_LIMIT:"4Gi"KUBERNETES_EPHEMERAL_STORAGE_REQUEST:"512Mi"KUBERNETES_EPHEMERAL_STORAGE_LIMIT:"1Gi"KUBERNETES_HELPER_CPU_REQUEST:"3"KUBERNETES_HELPER_CPU_LIMIT:"5"KUBERNETES_HELPER_MEMORY_REQUEST:"2Gi"KUBERNETES_HELPER_MEMORY_LIMIT:"4Gi"KUBERNETES_HELPER_EPHEMERAL_STORAGE_REQUEST:"512Mi"KUBERNETES_HELPER_EPHEMERAL_STORAGE_LIMIT:"1Gi"KUBERNETES_SERVICE_CPU_REQUEST:"3"KUBERNETES_SERVICE_CPU_LIMIT:"5"KUBERNETES_SERVICE_MEMORY_REQUEST:"2Gi"KUBERNETES_SERVICE_MEMORY_LIMIT:"4Gi"KUBERNETES_SERVICE_EPHEMERAL_STORAGE_REQUEST:"512Mi"KUBERNETES_SERVICE_EPHEMERAL_STORAGE_LIMIT:"1Gi"

Define a list of services

Define a list ofservices in theconfig.toml.

concurrent =1check_interval =30  [[runners]]    name ="myRunner"    url ="gitlab.example.com"    executor ="kubernetes"    [runners.kubernetes]      helper_image ="gitlab-registy.example.com/helper:latest"      [[runners.kubernetes.services]]        name ="postgres:12-alpine"        alias ="db1"      [[runners.kubernetes.services]]        name ="registry.example.com/svc1"        alias ="svc1"        entrypoint = ["entrypoint.sh"]        command = ["executable","param1","param2"]        environment = ["ENV=value1","ENV2=value2"]

If the service environment includesHEALTHCHECK_TCP_PORT, GitLab Runner waits until the serviceresponds on that port before starting user CI scripts. You can also configure theHEALTHCHECK_TCP_PORTenvironment variable in aservices section of.gitlab-ci.yml.

Overwrite service containers resources

If a job has multiple service containers, you can set explicitresource requests and limits to each service container.Use the variables attribute in each serviceto overwrite container resources specified in.gitlab-ci.yml.

services:-name:redis:5alias:redis5variables:KUBERNETES_SERVICE_CPU_REQUEST:"3"KUBERNETES_SERVICE_CPU_LIMIT:"6"KUBERNETES_SERVICE_MEMORY_REQUEST:"3Gi"KUBERNETES_SERVICE_MEMORY_LIMIT:"6Gi"KUBERNETES_EPHEMERAL_STORAGE_REQUEST:"2Gi"KUBERNETES_EPHEMERAL_STORAGE_LIMIT:"3Gi"-name:postgres:12alias:MY_relational-database.12variables:KUBERNETES_CPU_REQUEST:"2"KUBERNETES_CPU_LIMIT:"4"KUBERNETES_MEMORY_REQUEST:"1Gi"KUBERNETES_MEMORY_LIMIT:"2Gi"KUBERNETES_EPHEMERAL_STORAGE_REQUEST:"1Gi"KUBERNETES_EPHEMERAL_STORAGE_LIMIT:"2Gi"

These specific settings take precedence over the general settings for the job.The values are still restricted to themaximum overwrite settingfor that resource.

Overwrite the Kubernetes default service account

To overwrite the Kubernetes service account for each CI/CD job in the.gitlab-ci.yml file,set the variableKUBERNETES_SERVICE_ACCOUNT_OVERWRITE.

You can use this variable to specify a service account attached to the namespace, which you may needfor complex RBAC configurations.

variables:KUBERNETES_SERVICE_ACCOUNT_OVERWRITE:ci-service-account

To ensure only designated service accounts are used during CI runs, define a regular expressionfor either:

• Theservice_account_overwrite_allowed setting.
• TheKUBERNETES_SERVICE_ACCOUNT_OVERWRITE_ALLOWED environment variable.

If you don’t set either, the overwrite is disabled.

Set the RuntimeClass

Useruntime_class_name to set theRuntimeClass for each job container.

If you specify a RuntimeClass name but did not configure it in the cluster, or the feature is not supported,the executor fails to create jobs.

concurrent =1check_interval =30  [[runners]]    name ="myRunner"    url ="gitlab.example.com"    executor ="kubernetes"    [runners.kubernetes]      runtime_class_name ="myclass"

Change the base directory for build logs and scripts

You can change the directory whereemptyDir volumes are mounted to the pod for build logs and scripts.You can use the directory to:

• Run job pods with a modified image.
• Run as an unprivileged user.
• CustomizeSecurityContext settings.

To change the directory:

• For build logs, setlogs_base_dir.
• For build scripts, setscripts_base_dir.

The expected value is a string that represents a base directory without the trailing slash(for example,/tmp or/mydir/example).The directory must already exist.

This value is prepended to the generated path for build logs and scripts.For example:

  [[runners]]    name ="myRunner"    url ="gitlab.example.com"    executor ="kubernetes"    [runners.kubernetes]      logs_base_dir ="/tmp"      scripts_base_dir ="/tmp"

This configuration would result in anemptyDir volume mounted in:

• /tmp/logs-${CI_PROJECT_ID}-${CI_JOB_ID} for build logsinstead of the default/logs-${CI_PROJECT_ID}-${CI_JOB_ID}.
• /tmp/scripts-${CI_PROJECT_ID}-${CI_JOB_ID} for build scripts.

User namespaces

In Kubernetes 1.30 and later, you can isolate the user running in the container from the one onthe host withuser namespaces.A process running as root in the container can run as a different unprivileged user on the host.

With user namespaces, you can have more control over which images are used to run your CI/CD jobs.Operations that require additional settings (such as running as root) can also functionwithout opening up additional attack surface on the host.

To use this feature, ensure your cluster has beenproperly configured.The following example addspod_spec for thehostUsers keyand disables both privileged pods and privilege escalation:

  [[runners]]    environment = ["FF_USE_ADVANCED_POD_SPEC_CONFIGURATION=true"]    builds_dir ="/tmp/builds"  [runners.kubernetes]    logs_base_dir ="/tmp"    scripts_base_dir ="/tmp"    privileged =false    allowPrivilegeEscalation =false  [[runners.kubernetes.pod_spec]]    name ="hostUsers"    patch ='''      [{"op": "add", "path": "/hostUsers", "value": false}]    '''    patch_type ="json"

With user namespaces, you cannot use the default path for the build directory (builds_dir),build logs (logs_base_dir), or build scripts (scripts_base_dir).Even the container’s root user does not have the permission to mount volumes.They also cannot create directories in the root of the container’s file system.

Instead, you canchange the base directory for build logs and scripts.You can also change the build directory by setting[[runners]].builds_dir.

Operating system, architecture, and Windows kernel version

GitLab Runner with the Kubernetes executor can run builds on differentoperating systems if the configured cluster has nodes running those operating systems.

The system determines the helper image’s operating system, architecture, and Windows kernel version(if applicable). It then uses those parameters for other aspects of the build, for examplethe containers or images to use.

The following diagram explains how the system detects these details:

%%|fig-align: centerflowchart TB  init[<b>Initial defaults</b>:<br/>OS: linux</br>Arch: amd64]  hasAutoset{Configuration<br/><tt><a href="https://docs.gitlab.com/runner/configuration/advanced-configuration/">helper_image_autoset_arch_and_os</a> == true</tt>?}  setArch[<b>Update</b>:<br/>Arch: <i>same as runner</i>]  isWin{GitLab Runner runs on Windows?}  setWin[<b>Update</b>:<br/>OS: windows<br/>KernelVersion: <i>same as runner</i>]  hasNodeSel{<a href="https://docs.gitlab.com/runner/configuration/advanced-configuration/"><tt>node_selector</tt></a> configured<br/>in <tt>runners.kubernetes</tt> section?}  hasNodeSelOverride{<tt>node_selector</tt> configured<br/><a href="https://docs.gitlab.com/runner/executors/kubernetes/#overwrite-the-node-selector">as overwrite</a>?}  updateNodeSel[<b>Update from <tt>node_selector</tt></b> if set:<br/>OS: from <tt>kubernetes.io/os</tt><br/>Arch: from <tt>kubernetes.io/arch</tt><br/>KernelVersion: from <tt>node.kubernetes.io/windows-build</tt>]  updateNodeSelOverride[<b>Update from <tt>node_selector</tt> overwrites</b> if set:</br>OS: from <tt>kubernetes.io/os</tt><br/>Arch: from <tt>kubernetes.io/arch</tt><br/>KernelVersion: from <tt>node.kubernetes.io/windows-build</tt>]  result[final <b>OS</b>, <b>Arch</b>, <b>kernelVersion</b>]  init --> hasAutoset  hasAutoset -->|false | hasNodeSel  hasAutoset -->|true | setArch  setArch --> isWin  isWin -->|false | hasNodeSel  isWin -->|true | setWin  setWin --> hasNodeSel  hasNodeSel -->|false | hasNodeSelOverride  hasNodeSel -->|true | updateNodeSel  updateNodeSel --> hasNodeSelOverride  hasNodeSelOverride -->|false | result  hasNodeSelOverride -->|true | updateNodeSelOverride  updateNodeSelOverride --> result

The following are the only parameters that influence the operating system, architecture, and Windows kernel version selection of the build.

• Thehelper_image_autoset_arch_and_os configuration
• Thekubernetes.io/os,kubernetes.io/arch, andnode.kubernetes.io/windows-build label selectors from:
  • node_selector configuration
  • node_selector overwrites

Other parameters don’t influence the selection process described above.However, you can use parameters likeaffinity to further limit the nodes on which builds are scheduled.

Nodes

Specify the node to execute builds

Use thenode_selector option to specify which node in a Kubernetes cluster can be used to execute the builds.It is akey=value pair instring=string format (string:string in the case of environment variables).

Runner uses the information provided to determine the operating system and architecture for the build. This ensures thatthe correcthelper image is used. The default operating system and architecture islinux/amd64.

You can use specific labels to schedule nodes with different operating systems and architectures.

Example forlinux/arm64

  [[runners]]    name ="myRunner"    url ="gitlab.example.com"    executor ="kubernetes"    [runners.kubernetes.node_selector]"kubernetes.io/arch" ="arm64""kubernetes.io/os" ="linux"

Example forwindows/amd64

Kubernetes for Windows has certainlimitations.If you are using process isolation, you must also provide the specific Windows build version with thenode.kubernetes.io/windows-build label.

  [[runners]]    name ="myRunner"    url ="gitlab.example.com"    executor ="kubernetes"# The FF_USE_POWERSHELL_PATH_RESOLVER feature flag has to be enabled for PowerShell# to resolve paths for Windows correctly when Runner is operating in a Linux environment# but targeting Windows nodes.    environment = ["FF_USE_POWERSHELL_PATH_RESOLVER=true"]    [runners.kubernetes.node_selector]"kubernetes.io/arch" ="amd64""kubernetes.io/os" ="windows""node.kubernetes.io/windows-build" ="10.0.20348"

Overwrite the node selector

To overwrite the node selector:

1. In theconfig.toml or Helmvalues.yaml file, enable overwriting of the node selector:

   runners: ... config:|   [[runners]]     [runners.kubernetes]       node_selector_overwrite_allowed =".*"

2. In the.gitlab-ci.yml file, define the variable to overwrite the node selector:

   variables:KUBERNETES_NODE_SELECTOR_* = ''

In the following example, to overwrite the Kubernetes node architecture,the settings are configured in theconfig.toml and.gitlab-ci.yml file:

concurrent =1check_interval =1log_level ="debug"shutdown_timeout =0listen_address =':9252'[session_server]  session_timeout =1800[[runners]]  name =""  url ="https://gitlab.com/"  id =0  token ="__REDACTED__"  token_obtained_at ="0001-01-01T00:00:00Z"  token_expires_at ="0001-01-01T00:00:00Z"  executor ="kubernetes"  shell ="bash"  [runners.kubernetes]    host =""    bearer_token_overwrite_allowed =false    image ="alpine"    namespace =""    namespace_overwrite_allowed =""    pod_labels_overwrite_allowed =""    service_account_overwrite_allowed =""    pod_annotations_overwrite_allowed =""    node_selector_overwrite_allowed ="kubernetes.io/arch=.*"# <--- allows overwrite of the architecture

job:image:IMAGE_NAMEvariables:KUBERNETES_NODE_SELECTOR_ARCH:'kubernetes.io/arch=amd64'# <--- select the architecture

Define a list of node affinities

Define a list ofnode affinitiesto add to a pod specification at build time.

concurrent =1[[runners]]  name ="myRunner"  url ="gitlab.example.com"  executor ="kubernetes"  [runners.kubernetes]    [runners.kubernetes.affinity]      [runners.kubernetes.affinity.node_affinity]        [[runners.kubernetes.affinity.node_affinity.preferred_during_scheduling_ignored_during_execution]]          weight =100          [runners.kubernetes.affinity.node_affinity.preferred_during_scheduling_ignored_during_execution.preference]            [[runners.kubernetes.affinity.node_affinity.preferred_during_scheduling_ignored_during_execution.preference.match_expressions]]              key ="cpu_speed"              operator ="In"              values = ["fast"]            [[runners.kubernetes.affinity.node_affinity.preferred_during_scheduling_ignored_during_execution.preference.match_expressions]]              key ="mem_speed"              operator ="In"              values = ["fast"]        [[runners.kubernetes.affinity.node_affinity.preferred_during_scheduling_ignored_during_execution]]          weight =50          [runners.kubernetes.affinity.node_affinity.preferred_during_scheduling_ignored_during_execution.preference]            [[runners.kubernetes.affinity.node_affinity.preferred_during_scheduling_ignored_during_execution.preference.match_expressions]]              key ="core_count"              operator ="In"              values = ["high","32"]            [[runners.kubernetes.affinity.node_affinity.preferred_during_scheduling_ignored_during_execution.preference.match_fields]]              key ="cpu_type"              operator ="In"              values = ["arm64"]      [runners.kubernetes.affinity.node_affinity.required_during_scheduling_ignored_during_execution]        [[runners.kubernetes.affinity.node_affinity.required_during_scheduling_ignored_during_execution.node_selector_terms]]          [[runners.kubernetes.affinity.node_affinity.required_during_scheduling_ignored_during_execution.node_selector_terms.match_expressions]]            key ="kubernetes.io/e2e-az-name"            operator ="In"            values = ["e2e-az1","e2e-az2"            ]

Define nodes where pods are scheduled

Use pod affinity and anti-affinity to constrain the nodesyour pod is eligibleto be scheduled on, based on labels on other pods.

Example configuration in theconfig.toml:

concurrent =1[[runners]]  name ="myRunner"  url ="gitlab.example.com"  executor ="kubernetes"  [runners.kubernetes]    [runners.kubernetes.affinity]      [runners.kubernetes.affinity.pod_affinity]        [[runners.kubernetes.affinity.pod_affinity.required_during_scheduling_ignored_during_execution]]          topology_key ="failure-domain.beta.kubernetes.io/zone"          namespaces = ["namespace_1","namespace_2"]          [runners.kubernetes.affinity.pod_affinity.required_during_scheduling_ignored_during_execution.label_selector]            [[runners.kubernetes.affinity.pod_affinity.required_during_scheduling_ignored_during_execution.label_selector.match_expressions]]              key ="security"              operator ="In"              values = ["S1"]        [[runners.kubernetes.affinity.pod_affinity.preferred_during_scheduling_ignored_during_execution]]        weight =100        [runners.kubernetes.affinity.pod_affinity.preferred_during_scheduling_ignored_during_execution.pod_affinity_term]          topology_key ="failure-domain.beta.kubernetes.io/zone"          [runners.kubernetes.affinity.pod_affinity.preferred_during_scheduling_ignored_during_execution.pod_affinity_term.label_selector]            [[runners.kubernetes.affinity.pod_affinity.preferred_during_scheduling_ignored_during_execution.pod_affinity_term.label_selector.match_expressions]]              key ="security_2"              operator ="In"              values = ["S2"]      [runners.kubernetes.affinity.pod_anti_affinity]        [[runners.kubernetes.affinity.pod_anti_affinity.required_during_scheduling_ignored_during_execution]]          topology_key ="failure-domain.beta.kubernetes.io/zone"          namespaces = ["namespace_1","namespace_2"]          [runners.kubernetes.affinity.pod_anti_affinity.required_during_scheduling_ignored_during_execution.label_selector]            [[runners.kubernetes.affinity.pod_anti_affinity.required_during_scheduling_ignored_during_execution.label_selector.match_expressions]]              key ="security"              operator ="In"              values = ["S1"]          [runners.kubernetes.affinity.pod_anti_affinity.required_during_scheduling_ignored_during_execution.namespace_selector]            [[runners.kubernetes.affinity.pod_anti_affinity.required_during_scheduling_ignored_during_execution.namespace_selector.match_expressions]]              key ="security"              operator ="In"              values = ["S1"]        [[runners.kubernetes.affinity.pod_anti_affinity.preferred_during_scheduling_ignored_during_execution]]        weight =100        [runners.kubernetes.affinity.pod_anti_affinity.preferred_during_scheduling_ignored_during_execution.pod_affinity_term]          topology_key ="failure-domain.beta.kubernetes.io/zone"          [runners.kubernetes.affinity.pod_anti_affinity.preferred_during_scheduling_ignored_during_execution.pod_affinity_term.label_selector]            [[runners.kubernetes.affinity.pod_anti_affinity.preferred_during_scheduling_ignored_during_execution.pod_affinity_term.label_selector.match_expressions]]              key ="security_2"              operator ="In"              values = ["S2"]          [runners.kubernetes.affinity.pod_anti_affinity.preferred_during_scheduling_ignored_during_execution.pod_affinity_term.namespace_selector]            [[runners.kubernetes.affinity.pod_anti_affinity.preferred_during_scheduling_ignored_during_execution.pod_affinity_term.namespace_selector.match_expressions]]              key ="security_2"              operator ="In"              values = ["S2"]

Networking

Configure a container lifecycle hook

Usecontainer lifecycle hooks to runcode configured for a handler when the corresponding lifecycle hook is executed.

You can configure two types of hooks:PreStop andPostStart. Each of them allows only one type of handler to be set.

Example configuration in theconfig.toml file:

[[runners]]  name ="kubernetes"  url ="https://gitlab.example.com/"  executor ="kubernetes"  token ="yrnZW46BrtBFqM7xDzE7dddd"  [runners.kubernetes]    image ="alpine:3.11"    privileged =true    namespace ="default"    [runners.kubernetes.container_lifecycle.post_start.exec]      command = ["touch","/builds/postStart.txt"]    [runners.kubernetes.container_lifecycle.pre_stop.http_get]      port =8080      host ="localhost"      path ="/test"      [[runners.kubernetes.container_lifecycle.pre_stop.http_get.http_headers]]        name ="header_name_1"        value ="header_value_1"      [[runners.kubernetes.container_lifecycle.pre_stop.http_get.http_headers]]        name ="header_name_2"        value ="header_value_2"

Use the following settings to configure each lifecycle hook:

KubernetesLifecycleExecAction

KubernetesLifecycleHTTPGet

KubernetesLifecycleHTTPGetHeader

KubernetesLifecycleTcpSocket

Configure pod DNS settings

Use the following options to configure theDNS settingsof the pods.

Example configuration in theconfig.toml file:

concurrent =1check_interval =30[[runners]]  name ="myRunner"  url ="https://gitlab.example.com"  token ="__REDACTED__"  executor ="kubernetes"  [runners.kubernetes]    image ="alpine:latest"    [runners.kubernetes.dns_config]      nameservers = ["1.2.3.4",      ]      searches = ["ns1.svc.cluster-domain.example","my.dns.search.suffix",      ]      [[runners.kubernetes.dns_config.options]]        name ="ndots"        value ="2"      [[runners.kubernetes.dns_config.options]]        name ="edns0"

KubernetesDNSConfigOption

Default list of dropped capabilities

GitLab Runner drops the following capabilities by default.

User-definedcap_add has priority over the default list of dropped capabilities.If you want to add the capability that is dropped by default, add it tocap_add.

• NET_RAW

Add extra host aliases

This feature is available in Kubernetes 1.7 and higher.

Configure ahost aliases toinstruct Kubernetes to add entries to/etc/hosts file in the container.

Use the following options:

Example configuration in theconfig.toml file:

concurrent =4[[runners]]# usual configuration  executor ="kubernetes"  [runners.kubernetes]    [[runners.kubernetes.host_aliases]]      ip ="127.0.0.1"      hostnames = ["web1","web2"]    [[runners.kubernetes.host_aliases]]      ip ="192.168.1.1"      hostnames = ["web14","web15"]

You can also configure host aliases by using the command-line parameter--kubernetes-host_aliases with JSON input.For example:

gitlab-runner register --kubernetes-host_aliases'[{"ip":"192.168.1.100","hostnames":["myservice.local"]},{"ip":"192.168.1.101","hostnames":["otherservice.local"]}]'

Volumes

Using the cache with the Kubernetes executor

When the cache is used with the Kubernetes executor, a volume called/cache is mounted on the pod. During jobexecution, if cached data is needed, the runner checks if cached data is available. Cached data is available ifa compressed file is available on the cache volume.

To set the cache volume, use thecache_dir setting in theconfig.toml file.

• If available, the compressed file is extracted into the build folder and can then be used in the job.
• If not available, the cached data is downloaded from the configured storage and saved into thecache dir as a compressed file.The compressed file is then extracted into thebuild folder.

Configure volume types

You can mount the following volume types:

• hostPath
• persistentVolumeClaim
• configMap
• secret
• emptyDir
• csi

Example of a configuration with multiple volume types:

concurrent =4[[runners]]# usual configuration  executor ="kubernetes"  [runners.kubernetes]    [[runners.kubernetes.volumes.host_path]]      name ="hostpath-1"      mount_path ="/path/to/mount/point"      read_only =true      host_path ="/path/on/host"    [[runners.kubernetes.volumes.host_path]]      name ="hostpath-2"      mount_path ="/path/to/mount/point_2"      read_only =true    [[runners.kubernetes.volumes.pvc]]      name ="pvc-1"      mount_path ="/path/to/mount/point1"    [[runners.kubernetes.volumes.config_map]]      name ="config-map-1"      mount_path ="/path/to/directory"      [runners.kubernetes.volumes.config_map.items]"key_1" ="relative/path/to/key_1_file""key_2" ="key_2"    [[runners.kubernetes.volumes.secret]]      name ="secrets"      mount_path ="/path/to/directory1"      read_only =true      [runners.kubernetes.volumes.secret.items]"secret_1" ="relative/path/to/secret_1_file"    [[runners.kubernetes.volumes.empty_dir]]      name ="empty-dir"      mount_path ="/path/to/empty_dir"      medium ="Memory"    [[runners.kubernetes.volumes.csi]]      name ="csi-volume"      mount_path ="/path/to/csi/volume"      driver ="my-csi-driver"      [runners.kubernetes.volumes.csi.volume_attributes]        size ="2Gi"

hostPath volume

Configure thehostPath volume to instruct Kubernetes to mounta specified host path in the container.

Use the following options in theconfig.toml file:

persistentVolumeClaim volume

Configure thepersistentVolumeClaim volume toinstruct Kubernetes to use apersistentVolumeClaim defined in a Kubernetes cluster and mount it in the container.

Use the following options in theconfig.toml file:

configMap volume

Configure theconfigMap volume to instruct Kubernetes to use aconfigMapdefined in a Kubernetes cluster and mount it in the container.

Use the following options in theconfig.toml:

Each key from theconfigMap is changed into a file and stored in the mount path. By default:

• All keys are included.
• TheconfigMap key is used as the filename.
• The value is stored in the file contents.

To change the default key and value storage, use theitems option. If you use theitems option,only specified keysare added to the volumes and all other keys are skipped.

secret volume

Configure asecret volume to instruct Kubernetes to useasecret defined in a Kubernetes cluster and mount it in the container.

Use the following options in theconfig.toml file:

Each key from selectedsecret is changed into a file stored in the selected mount path. By default:

• All keys are included.
• TheconfigMap key is used as the filename.
• The value is stored in the file contents.

To change default key and value storage, use theitems option. If you use theitems option,only specified keysare added to the volumes and all other keys are skipped.

emptyDir volume

Configure anemptyDir volumeto instruct Kubernetes to mount an empty directory in the container.

Use the following options in theconfig.toml file:

csi volume

Configure aContainer Storage Interface (csi) volume to instructKubernetes to use a customcsi driver to mount an arbitrary storage system in the container.

Use the following options in theconfig.toml:

Mount volumes on service containers

Volumes defined for the build container are also automatically mounted for all services containers. You can use this functionality as an alternative toservices_tmpfs (available only to Docker executor), to mount database storage in RAM to speed up tests.

Example configuration in theconfig.toml file:

[[runners]]# usual configuration  executor ="kubernetes"  [runners.kubernetes]    [[runners.kubernetes.volumes.empty_dir]]      name ="mysql-tmpfs"      mount_path ="/var/lib/mysql"      medium ="Memory"

Custom volume mount

To store the builds directory for the job, define custom volume mounts to theconfiguredbuilds_dir (/builds by default).If you usepvc volumes,based on theaccess mode,you might be limited to running jobs on one node.

Example configuration in theconfig.toml file:

concurrent =4[[runners]]# usual configuration  executor ="kubernetes"  builds_dir ="/builds"  [runners.kubernetes]    [[runners.kubernetes.volumes.empty_dir]]      name ="repo"      mount_path ="/builds"      medium ="Memory"

Persistent per-concurrency build volumes

The build directories in Kubernetes CI jobs are ephemeral by default.If you want to persist your Git clone across jobs (to makeGIT_STRATEGY=fetch work),you must mount a persistent volume claim for your build folder.Because multiple jobs can run concurrently, you must eitheruse aReadWriteMany volume, or have one volume for each potentialconcurrent job on the same runner. The latter is likely to be more performant.Here is an example of such a configuration:

concurrent =4[[runners]]  executor ="kubernetes"  builds_dir ="/mnt/builds"  [runners.kubernetes]    [[runners.kubernetes.volumes.pvc]]# CI_CONCURRENT_ID identifies parallel jobs of the same runner.      name ="build-pvc-$CI_CONCURRENT_ID"      mount_path ="/mnt/builds"

In this example, create the persistent volume claims namedbuild-pvc-0 tobuild-pvc-3 yourself.Create as many as the runner’sconcurrent setting dictates.

Use a helper image

After you set the security policy, thehelper image must conform to the policy.The image does not receive privileges from the root group, so you must ensure that the user ID is part of the root group.

The following example creates a user and group callednonroot and sets the helper image to run as that user.

ARG tagFROM registry.gitlab.com/gitlab-org/ci-cd/gitlab-runner-ubi-images/gitlab-runner-helper-ocp:${tag}USER rootRUN groupadd -g59417 nonroot&&\    useradd -u59417 nonroot -g nonrootWORKDIR /home/nonrootUSER 59417:59417

Using Docker in builds

When you use Docker in your builds, there are several considerationsyou should be aware of.

Exposed/var/run/docker.sock

There is risk involved if you use therunners.kubernetes.volumes.host_path optionto expose/var/run/docker.sock of your host into your build container.Be careful when you run builds in the same cluster as your productioncontainers. The node’s containers are accessible from the build container.

Usingdocker:dind

If you run thedocker:dind, also called thedocker-in-docker image,containers must run in privileged mode. This may have potential risksand cause additional issues.

The Docker daemon runs as a separate container in the pod because it is started as aservice,typically in the.gitlab-ci.yml. Containers in pods only share volumes assigned to them andan IP address, that they use to communicate with each other withlocalhost. Thedocker:dindcontainer does not share/var/run/docker.sock and thedocker binary tries to use it by default.

To configure the client use TCP to contact the Docker daemon,in the other container, include the environment variables ofthe build container:

• DOCKER_HOST=tcp://docker:2375 for no TLS connection.
• DOCKER_HOST=tcp://docker:2376 for TLS connection.

In Docker 19.03 and later, TLS is enabled bydefault but you must map certificates to your client.You can enable non-TLS connection for Docker-in-Docker ormount certificates. For more information, seeUse Docker In Docker Workflow with Docker executor.

Prevent host kernel exposure

If you usedocker:dind or/var/run/docker.sock, the Docker daemonhas access to the underlying kernel of the host machine. This means that anylimits set in the pod do not work when Docker images are built.The Docker daemon reports the full capacity of the node, regardless oflimits imposed on the Docker build containers spawned by Kubernetes.

If you run build containers in privileged mode, or if/var/run/docker.sock is exposed,the host kernel may become exposed to build containers. To minimize exposure, specify a labelin thenode_selector option. This ensures that the node matches the labels before any containerscan be deployed to the node. For example, if you specify the labelrole=ci, the build containersonly run on nodes labeledrole=ci, and all other production services run on other nodes.

To further separate build containers, you can use nodetaints.Taints prevent other pods from scheduling on the same nodes as thebuild pods, without extra configuration for the other pods.

Restrict Docker images and services

You can restrict the Docker images that are used to run your jobs.To do this, you specify wildcard patterns. For example, to allow imagesfrom your private Docker registry only:

[[runners]](...)  executor ="kubernetes"  [runners.kubernetes](...)    allowed_images = ["my.registry.tld:5000/*:*"]    allowed_services = ["my.registry.tld:5000/*:*"]

Or, to restrict to a specific list of images from this registry:

[[runners]](...)  executor ="kubernetes"  [runners.kubernetes](...)    allowed_images = ["my.registry.tld:5000/ruby:*","my.registry.tld:5000/node:*"]    allowed_services = ["postgres:9.4","postgres:latest"]

Restrict Docker pull policies

In the.gitlab-ci.yml file, you can specify a pull policy. This policy determines howa CI/CD job should fetch images.

To restrict which pull policies can be used from those specified in the.gitlab-ci.yml file, useallowed_pull_policies.

For example, to allow only thealways andif-not-present pull policies:

[[runners]](...)  executor ="kubernetes"  [runners.kubernetes](...)    allowed_pull_policies = ["always","if-not-present"]

• If you don’t specifyallowed_pull_policies, the default is the value in thepull_policy keyword.
• If you don’t specifypull_policy, the cluster’s imagedefault pull policy is used.
• The job uses only the pull policies that are listed in bothpull_policy andallowed_pull_policies.The effective pull policy is determined by comparing the policies inpull_policy keywordandallowed_pull_policies. GitLab uses theintersectionof these two policy lists.For example, ifpull_policy is["always", "if-not-present"] andallowed_pull_policiesis["if-not-present"], then the job uses onlyif-not-present because it’s the only pull policy defined in both lists.
• The existingpull_policy keyword must include at least one pull policy specified inallowed_pull_policies.The job fails if none of thepull_policy values matchallowed_pull_policies.

Job execution

GitLab Runner useskube attach instead ofkube exec by default. This should avoid problems like when ajob is marked successful midwayin environments with an unstable network.

Followissue #27976 for progress on legacy execution strategy removal.

Configure the number of request attempts to the Kubernetes API

By default, the Kubernetes executor retries specific requests to the Kubernetes API after five failed attempts. The delay is controlled bya backoff algorithm with a 500 millisecond floor and a customizable ceiling with default value of two seconds.To configure the number of retries, use theretry_limit option in theconfig.toml file.Similarly, for backoff ceiling, use theretry_backoff_max option.The following failures are automatically retried:

• error dialing backend
• TLS handshake timeout
• read: connection timed out
• connect: connection timed out
• Timeout occurred
• http2: client connection lost
• connection refused
• tls: internal error
• io.unexpected EOF
• syscall.ECONNRESET
• syscall.ECONNREFUSED
• syscall.ECONNABORTED
• syscall.EPIPE

To control the amount of retries for each error, use theretry_limits option.Therety_limits specifies the amount of retries for each error separately,and is a map of error messages to the amount of retries.The error message can be a substring of the error message returned by the Kubernetes API.Theretry_limits option has precedence over theretry_limit option.

For example, configure theretry_limits option to retry the TLS related errors in yourenvironment 10 times instead of the default five times:

[[runners]]  name ="myRunner"  url ="https://gitlab.example.com/"  executor ="kubernetes"  [runners.kubernetes]    retry_limit =5    [runners.kubernetes.retry_limits]"TLS handshake timeout" =10"tls: internal error" =10

To retry an entirely different error, such asexceeded quota 20 times:

[[runners]]  name ="myRunner"  url ="https://gitlab.example.com/"  executor ="kubernetes"  [runners.kubernetes]    retry_limit =5    [runners.kubernetes.retry_limits]"exceeded quota" =20

Container entrypoint known issues

The container entry point has the following known issues:

• If an entrypoint is defined in the Dockerfile for an image, it must open a valid shell. Otherwise, the job hangs.

  • To open a shell, the system passes the command asargsfor the build container.

• File type CI/CD variablesare not written to disk when the entrypoint is executed. The file is only accessiblein the job during script execution.

• The following CI/CD variables are not accessible in the entrypoint. You can usebefore_script to makeany setup changes before running script commands:

  • CI/CD variables defined in the settings.
  • Masked CI/CD variables.

Before GitLab Runner 17.4:

• The entrypoint logs were not forwarded to the build’s log.
• With the Kubernetes executor withkube exec, GitLab Runner did not wait for the entrypoint to open a shell (seeabove).

Starting with GitLab Runner 17.4, the entrypoint logs are now forwarded. The system waitsfor the entrypoint to run and spawn the shell. This has the followingimplications:

• IfFF_KUBERNETES_HONOR_ENTRYPOINT is set, and the image’s entrypoint takeslonger thanpoll_timeout (default: 180 s), the build fails. Thepoll_timeout value (and potentiallypoll_interval)must be adapted if the entrypoint is expected to run longer.
• WhenFF_KUBERNETES_HONOR_ENTRYPOINTandFF_USE_LEGACY_KUBERNETES_EXECUTION_STRATEGY are set, the system adds astartup probeto the build container, so that it knows when the entrypoint is spawningthe shell. If a custom entrypoint uses the providedargsto spawn the expected shell, then the startup probe is resolvedautomatically. However, if the container image is spawning the shell withoutusing the command passed in throughargs, the entrypoint must resolve thestartup probe itself by creating a file named.gitlab-startup-marker insidethe root of the build directory.The startup probe checks everypoll_interval for the.gitlab-startup-markerfile. If the file is not present inpoll_timeout, the pod is consideredunhealthy, and the system abort the build.

Restrict access to job variables

When using Kubernetes executor, users with access to the Kubernetes cluster can read variables used in the job. By default, job variables are stored in:

• Pod’s environment section

To restrict access to job variable data, you should use role-based access control (RBAC). When you use RBAC, only GitLab administrators have access to the namespace used by the GitLab Runner.

If you need other users to access the GitLab Runner namespace, set the followingverbs to restrict the user access in the GitLab Runner namespace:

• Forpods andconfigmaps:
  • get
  • watch
  • list
• Forpods/exec andpods/attach, usecreate.

Example RBAC definition for authorized users:

kind:RoleapiVersion:rbac.authorization.k8s.io/v1metadata:name:gitlab-runner-authorized-usersrules:-apiGroups:[""]resources:["configmaps","pods"]verbs:["get","watch","list"]-apiGroups:[""]resources:["pods/exec","pods/attach"]verbs:["create"]

Resources check during prepare step

Prerequisites:

• image_pull_secrets orservice_account is set.
• resource_availability_check_max_attempts is set to a number greater than zero.
• KubernetesserviceAccount used with theget andlist permissions.

GitLab Runner checks if the new service accounts or secrets are available with a 5-second interval between each try.

• This feature is disabled by default. To enable this feature, setresource_availability_check_max_attempts to any value other than0.The value you set defines the amount of times the runner checks for service accounts or secrets.

Overwrite the Kubernetes namespace

Prerequisites:

• In thevalues.yml file for GitLab Runner Helm charts,rbac.clusterWideAccess is set totrue.
• The runner haspermissions configured in the core API group.

You can overwrite Kubernetes namespaces to designate a namespace for CI purposes, and deploy a customset of pods to it. The pods spawned by the runner are in the overwritten namespace toenable access between containers during the CI stages.

To overwrite the Kubernetes namespace for each CI/CD job, set theKUBERNETES_NAMESPACE_OVERWRITEvariable in the.gitlab-ci.yml file.

variables:KUBERNETES_NAMESPACE_OVERWRITE:ci-${CI_COMMIT_REF_SLUG}

To use only designated namespaces during CI runs, in theconfig.toml file, define a regular expression fornamespace_overwrite_allowed:

[runners.kubernetes]    ...    namespace_overwrite_allowed ="ci-.*"