This page describes how to useGKE Sandbox to protect thehost kernel on your nodes when containers in the Pod execute unknown oruntrusted code, or need extra isolation from the node. This page explainshow you can enable GKE Sandbox and monitor your clusters when GKE Sandbox isrunning.

This page is for Security specialists who must isolate their workloads foradditional protection from unknown or untrusted code.To learn more aboutcommon roles and example tasks that we reference in Google Cloud content, seeCommon GKE user roles and tasks.

Before reading this page, ensure that you're familiar withthegeneral overview of GKE Sandbox.

Enable GKE Sandbox

GKE Sandbox is ready to use in Autopilot clusters runningGKE version 1.27.4-gke.800 and later. To start deployingAutopilot workloads in a sandbox, skip toWorking with GKE Sandbox.

To use GKE Sandbox in new or existing GKE Standardclusters, you must manually enable GKE Sandbox on the cluster.

For more information about GPU version releases, seeGPU Model Support for details on GPU version releases.

Before you begin

Before you start, make sure that you have performed the following tasks:

• Enable the Google Kubernetes Engine API.
Enable Google Kubernetes Engine API
• If you want to use the Google Cloud CLI for this task,install and theninitialize the gcloud CLI. If you previously installed the gcloud CLI, get the latest version by running thegcloud components update command. Earlier gcloud CLI versions might not support running the commands in this document.Note: For existing gcloud CLI installations, make sure to set thecompute/regionproperty. If you use primarily zonal clusters, set thecompute/zone instead. By setting a default location, you can avoid errors in the gcloud CLI like the following:One of [--zone, --region] must be supplied: Please specify location. You might need to specify the location in certain commands if the location of your cluster differs from the default that you set.

Enable GKE Sandbox on a new Standard cluster

The default node pool, which is created when you create a new cluster, can't useGKE Sandbox if it's the only node pool in the cluster, becauseGKE-managed system workloads must run separately from untrustedsandboxed workloads. To enable GKE Sandbox during cluster creation, you mustadd at least one extra node pool to the cluster.

gcloudcontainernode-poolscreateNODE_POOL_NAME\--cluster=CLUSTER_NAME\--node-version=NODE_VERSION\--machine-type=MACHINE_TYPE\--image-type=cos_containerd\--sandboxtype=gvisor

gcloudcontainernode-poolscreateNODE_POOL_NAME\--cluster=CLUSTER_NAME\--node-version=NODE_VERSION\--machine-type=MACHINE_TYPE\--accelerator=type=GPU_TYPE,gpu-driver-version=DRIVER_VERSION\--image-type=cos_containerd\--sandboxtype=gvisor

gcloudcontainernode-poolscreateNODE_POOL_NAME\--cluster=CLUSTER_NAME\--node-version=NODE_VERSION\--num-nodes=NUM_NODES\--tpu-topology=TPU_TOPOLOGY\--machine-type=MACHINE_TYPE\--image-type=cos_containerd\--sandboxtype=gvisor

Enable GKE Sandbox on an existing Standard cluster

You can enable GKE Sandbox on an existing Standard cluster byadding a new node pool and enabling the feature for that node pool.

gcloudcontainernode-poolscreateNODE_POOL_NAME\--cluster=CLUSTER_NAME\--machine-type=MACHINE_TYPE\--image-type=cos_containerd\--sandboxtype=gvisor

gcloudcontainernode-poolscreateNODE_POOL_NAME\--cluster=CLUSTER_NAME\--node-version=NODE_VERSION\--machine-type=MACHINE_TYPE\--accelerator=type=GPU_TYPE,gpu-driver-version=DRIVER_VERSION\--image-type=cos_containerd\--sandboxtype=gvisor

gcloudcontainernode-poolscreateNODE_POOL_NAME\--cluster=CLUSTER_NAME\--node-version=NODE_VERSION\--num-nodes=NUM_NODES\--tpu-topology=TPU_TOPOLOGY\--machine-type=MACHINE_TYPE\--image-type=cos_containerd\--sandboxtype=gvisor

Optional: Enable monitoring and logging

It is optional but recommended that you enable Cloud Logging andCloud Monitoring on the cluster, so that gVisor messages are logged.These services are enabled by default for new clusters.

You can use the Google Cloud console to enable these features on an existing cluster.

1. Go to theGoogle Kubernetes Engine page in the Google Cloud console.

   Go to Google Kubernetes Engine

2. Click the name of the cluster you want to modify.

3. UnderFeatures, in theCloud Loggingfield, clickeditEdit Cloud Logging.

4. Select theEnable Cloud Logging checkbox.

5. ClickSave Changes.

6. Repeat the same steps for theCloud Monitoring andManaged Service for Prometheus fields to enable those features.

Use GKE Sandbox in Autopilot and Standard

In Autopilot clusters and in Standard clusters withGKE Sandbox enabled, you request a sandboxed environment for a Pod byspecifying thegvisor RuntimeClass in the Pod specification.

For Autopilot clusters, ensure that you're running GKEversion 1.27.4-gke.800 or later.

Run an application in a sandbox

To make a Deployment run on a node with GKE Sandbox enabled, set itsspec.template.spec.runtimeClassName togvisor, as shown in the followingexample:

# httpd.yamlapiVersion:apps/v1kind:Deploymentmetadata:name:httpdlabels:app:httpdspec:replicas:1selector:matchLabels:app:httpdtemplate:metadata:labels:app:httpdspec:runtimeClassName:gvisorcontainers:-name:httpdimage:httpd

Create the Deployment:

kubectlapply-fhttpd.yaml

The Pod is deployed to a node with GKE Sandbox enabled. To verify thedeployment, find the node where the Pod is deployed:

kubectlgetpods

The output is similar to the following:

NAME                    READY   STATUS    RESTARTS   AGEhttpd-db5899bc9-dk7lk   1/1     Running   0          24s

From the output, find the name of the Pod in the output, and then check thevalue for RuntimeClass:

kubectlgetpodsPOD_NAME-ojsonpath='{.spec.runtimeClassName}'

The output isgvisor.

Alternatively, you can list the RuntimeClass of each Pod, and look for the Podswhere it is set togvisor:

kubectlgetpods-ojsonpath=$'{range .items[*]}{.metadata.name}: {.spec.runtimeClassName}\n{end}'

The output is the following:

POD_NAME: gvisor

This method of verifying that the Pod is running in a sandbox is trustworthybecause it does not rely on any data within the sandbox itself. Anythingreported from within the sandbox is untrustworthy, because it could bedefective or malicious.

Run a Pod with accelerators on GKE Sandbox

To run a GPU or TPU workload on GKE Sandbox, add theruntimeClassName: gvisor field to your manifest like the followingexamples:

• Example manifest for Standard mode GPU Pods:

  apiVersion:v1kind:Podmetadata:name:my-gpu-podspec:runtimeClassName:gvisorcontainers:-name:my-gpu-containerimage:nvidia/samples:vectoradd-cuda10.2resources:limits:nvidia.com/gpu:1

• Example manifest for Autopilot mode GPU Pods:

  apiVersion:v1kind:Podmetadata:name:my-gpu-podspec:runtimeClassName:gvisornodeSelector:cloud.google.com/gke-gpu-driver-version:"latest"cloud.google.com/gke-accelerator:nvidia-tesla-t4containers:-name:my-gpu-containerimage:nvidia/samples:vectoradd-cuda10.2resources:limits:nvidia.com/gpu:1

• Example manifest for Standard or Autopilot mode TPU Pods:

  apiVersion:v1kind:Podmetadata:name:my-tpu-podspec:runtimeClassName:gvisornodeSelector:cloud.google.com/gke-tpu-accelerator:tpu-v5-lite-podslicecloud.google.com/gke-tpu-topology:1x1containers:-name:my-tpu-containerimage:us-docker.pkg.dev/cloud-tpu-images/jax-ai-image/tpu:latestcommand:-bash--c-|python -c 'import jax; print("TPU cores:", jax.device_count())'resources:limits:google.com/tpu:1requests:google.com/tpu:1

You can run any Autopilot or Standard mode accelerator Podsthat meet the version and accelerator type requirements on GKE Sandbox byadding theruntimeClassName: gvisor field to the manifest. To run GPU Pods inGKE, see the following:

• Deploy GPU workloads in Autopilot
• Run GPUs in GKE Standard node pools

To run TPU Pods in GKE, see the following:

• Deploy TPU workloads in Autopilot
• Run TPUs in GKE Standard node pools

Supported GPU types for Autopilot are the following:

• nvidia-gb200: NVIDIA GB200 NVL72 (Preview)
• nvidia-b200: NVIDIA B200 (180GB) (Preview)
• nvidia-h200-141gb: NVIDIA H200 (141GB) (Preview)
• nvidia-h100-mega-80gb: NVIDIA H100 Mega (80GB)
• nvidia-h100-80gb: NVIDIA H100 (80GB)
• nvidia-a100-80gb: NVIDIA A100 (80GB)
• nvidia-tesla-a100: NVIDIA A100 (40GB)
• nvidia-rtx-pro-6000: NVIDIA RTX PRO 6000 (Preview)
• nvidia-l4: NVIDIA L4
• nvidia-tesla-t4: NVIDIA T4

Run a regular Pod along with sandboxed Pods

The steps in this section apply to Standard mode workloads. You don'tneed to run regular Pods alongside sandbox Pods in Autopilot mode,because theAutopilot pricing modeleliminates the need to manually optimize the number of Pods scheduled on nodes.

After enabling GKE Sandbox on a node pool, you can run trusted applicationson those nodes without using a sandbox by using node taints and tolerations.These Pods are referred to as "regular Pods" to distinguish them from sandboxedPods.

Regular Pods, just like sandboxed Pods, are prevented from accessing otherGoogle Cloud services or cluster metadata. This prevention is part of thenode's configuration. If your regular Pods or sandboxed Pods require access toGoogle Cloud services, useWorkload Identity Federation for GKE.

labels:sandbox.gke.io/runtime:gvisor

taints:-effect:NoSchedulekey:sandbox.gke.io/runtimevalue:gvisor

affinity:nodeAffinity:requiredDuringSchedulingIgnoredDuringExecution:nodeSelectorTerms:-matchExpressions:-key:sandbox.gke.io/runtimeoperator:Invalues:-gvisor

tolerations:-effect:NoSchedulekey:sandbox.gke.io/runtimeoperator:Equalvalue:gvisor

For example, the following manifest modifies the manifest used inRunning an application in a sandbox so that it runs asa regular Pod on a node with sandboxed Pods, by removing the runtimeClass andadding both the taint and toleration described earlier.

# httpd-no-sandbox.yamlapiVersion:apps/v1kind:Deploymentmetadata:name:httpd-no-sandboxlabels:app:httpdspec:replicas:1selector:matchLabels:app:httpdtemplate:metadata:labels:app:httpdspec:containers:-name:httpdimage:httpdaffinity:nodeAffinity:requiredDuringSchedulingIgnoredDuringExecution:nodeSelectorTerms:-matchExpressions:-key:sandbox.gke.io/runtimeoperator:Invalues:-gvisortolerations:-effect:NoSchedulekey:sandbox.gke.io/runtimeoperator:Equalvalue:gvisor

First, verify that the Deployment is not running in a sandbox:

kubectlgetpods-ojsonpath=$'{range .items[*]}{.metadata.name}: {.spec.runtimeClassName}\n{end}'

The output is similar to:

httpd-db5899bc9-dk7lk: gvisorhttpd-no-sandbox-5bf87996c6-cfmmd:

Thehttpd Deployment created earlier is running in a sandbox, because itsruntimeClass isgvisor. Thehttpd-no-sandbox Deployment has no value forruntimeClass, so it is not running in a sandbox.

Next, verify that the non-sandboxed Deployment is running on a node withGKE Sandbox by running the following command:

kubectlgetpod-ojsonpath=$'{range .items[*]}{.metadata.name}: {.spec.nodeName}\n{end}'

The name of the node pool is embedded in the value ofnodeName. Verify thatthe Pod is running on a node in a node pool with GKE Sandbox enabled.

You can't disable GKE Sandbox in GKE Autopilotclusters or in GKE Standard node pools. If you want tostop using GKE Sandbox,delete the node pool.

What's next

• Learn more aboutmanaging node pools.
• Read thesecurity overview.