This page shows you how to enforce encryption of data in-usein your nodes and workloads by using Confidential Google Kubernetes Engine Nodes. Enforcingencryption can help increase the security of your workloads.

This page is for Security specialists who implement security measures onGKE. To learn more aboutcommon roles and example tasks that we reference in Google Cloud content, seeCommon GKE user roles and tasks.

Before reading this page, ensure that you're familiar with the concept ofdata-in-use.

What is Confidential GKE Nodes?

You can encrypt your workloads with Confidential GKE Nodes orConfidential mode for Hyperdisk Balanced.

Confidential GKE Nodes

Confidential GKE Nodes uses Compute Engine Confidential VMto protect data in use with hardware-based memory encryption.Confidential GKE Nodes supports the following Confidential Computingtechnologies:

• AMD Secure Encrypted Virtualization (SEV)
• AMD Secure Encrypted Virtualization-Secure Nested Paging (SEV-SNP)
• Intel Trust Domain Extensions (TDX)

For more information about these technologies and for help choosing the optimaltechnology for your requirements, see theConfidential VM overview.

Confidential GKE Nodes doesn't change the security measures thatGKE applies to cluster control planes. To learn about thesemeasures, seeControl plane security.For visibility over who accesses control planes in your Google Cloudprojects, useAccess Transparency.

You can enable Confidential GKE Nodes for entire clusters, forspecific node pools, or for specific workloads. The following table describesthese enablement methods:

Confidential mode for Hyperdisk Balanced

You can also enableConfidential mode for Hyperdisk Balancedon your boot disk storage, which encrypts your data on additionalhardware-backed enclaves.

You can enable Confidential mode for Hyperdisk Balanced when doing one of the following:

• Create a new cluster
• Create a new node pool

You cannot update an existing cluster or a node pool to change theConfidential mode for Hyperdisk Balanced setting.

The following table shows you the GKE behavior that applies whenyou enable Confidential mode for Hyperdisk Balanced setting at the cluster level or at the nodepool level:

Pricing

The following pricing applies, depending on how you configureConfidential GKE Nodes:

Additionally, Confidential GKE Nodes might generate slightly more logdata on startup than nodes that aren't Confidential GKE Nodes. Formore information about logs pricing, seePricing for Google Cloud Observability.

Before you begin

Before you start, make sure that you have performed the following tasks:

• Enable the Google Kubernetes Engine API.
Enable Google Kubernetes Engine API
• If you want to use the Google Cloud CLI for this task,install and theninitialize the gcloud CLI. If you previously installed the gcloud CLI, get the latest version by running thegcloud components update command. Earlier gcloud CLI versions might not support running the commands in this document.Note: For existing gcloud CLI installations, make sure to set thecompute/regionproperty. If you use primarily zonal clusters, set thecompute/zone instead. By setting a default location, you can avoid errors in the gcloud CLI like the following:One of [--zone, --region] must be supplied: Please specify location. You might need to specify the location in certain commands if the location of your cluster differs from the default that you set.

Requirements

Confidential GKE Nodes has the following requirements:

• Autopilot clusters must run GKE version1.30.2 or later.

• Standard clusters must run one of the following GKEversions, depending on the Confidential Computing technology and node operatingsystem that you choose:

  • Container-Optimized OS:

    • AMD SEV: any GKE version.
    • AMD SEV-SNP: 1.32.2-gke.1297000 or later.
    • Intel TDX: 1.32.2-gke.1297000 or later.

  • Ubuntu:

    • AMD SEV: 1.33.5-gke.1350000 or later and 1.34.1-gke.2037000 orlater.
    • AMD SEV-SNP: 1.33.5-gke.1350000 or later and 1.34.1-gke.2037000 orlater.
    • Intel TDX: 1.33.5-gke.1697000 or later and 1.34.1-gke.2909000 orlater.

• To enable Confidential GKE Nodes at the workload level,Autopilot and Standard clusters must run GKEversion 1.33.3-gke.1392000 or later.

• Nodes must use a machine type that supports the Confidential Computingtechnology that you choose. For more information, seeMachine types, CPUs, and zones.

• Cluster control planes and nodes must be in a location that supports theConfidential Computing technology that you choose. For more information, seeView supported zones.

Configure Confidential GKE Nodes at the workload level

To enable Confidential GKE Nodes in individual workloads, you use aComputeClass.ComputeClasses are Kubernetes custom resources that let you define theattributes of new nodes that GKE creates and let you controlfallback behavior if hardware isn't available. ComputeClasses support all ofthe available Confidential Computing technologies for GKE, even ifa specific technology isn't supported at the cluster or node pool level.

To configure Confidential GKE Nodes at the workload level, followthese steps:

1. Optional: For GKE Standard clusters,enable node-autoprovisioning for the cluster.For Autopilot clusters, skip this step.

   If you choose to use ComputeClasses with only manually created nodepools, you must alsomanually configure node taints and node labelson those node pools. This additional manual configuration might result inyou noticing fewer benefits from using a ComputeClass to createConfidential GKE Nodes.

2. Save the following ComputeClass manifest as a YAML file. For moreinformation about the supported fields and values for ComputeClasses, seetheComputeClass CustomResourceDefinition.

   apiVersion:cloud.google.com/v1kind:ComputeClassmetadata:name:COMPUTE_CLASSspec:nodePoolConfig:confidentialNodeType:CONFIDENTIAL_COMPUTE_TECHNOLOGYnodePoolAutoCreation:enabled:trueactiveMigration:optimizeRulePriority:truepriorityDefaults:location:zones:['ZONE1','ZONE2']priorities:-machineFamily:MACHINE_SERIES-machineType:MACHINE_TYPEwhenUnsatisfiable:DoNotScaleUp

   Replace the following:

   • COMPUTE_CLASS: a name for your new computeclass.

   • CONFIDENTIAL_COMPUTE_TECHNOLOGY: theConfidential Computing technology to use. The following values aresupported:

     • SEV for AMD SEV
     • SEV_SNP for AMD SEV-SNP
     • TDX for Intel TDX

   • ZONE1,ZONE2,...: the zones to create nodes in,such as['us-central1-a','us-central1-b']. Specify zones that supportthe Confidential Computing technology that you specified. For moreinformation, seeView supported zones.

   • MACHINE_SERIES: the machine series for thenodes, such asn2d. Specify a machine series that supports theConfidential Computing technology that you specified. For more information,seeMachine types, CPUs, and zones.

   • MACHINE_TYPE the machine type for the nodes,such asn2d-standard-4. Specify a machine type that supports theConfidential Computing technology that you specified. For more information,seeMachine types, CPUs, and zones.

   This manifest includes thespec.nodePoolAutoCreation field with a value ofenabled: true, which lets GKE create new node pools byusing node auto-provisioning.

3. Create the ComputeClass in your cluster:

   kubectlapply-fPATH_TO_COMPUTE_CLASS_MANIFEST

   ReplacePATH_TO_COMPUTE_CLASS_MANIFEST with the pathto the ComputeClass manifest file.

4. Save the following Deployment manifest as a YAML file:

   apiVersion:apps/v1kind:Deploymentmetadata:name:helloweblabels:app:hellospec:selector:matchLabels:app:hellotemplate:metadata:labels:app:hellospec:nodeSelector:# Replace with the name of a compute classcloud.google.com/compute-class:COMPUTE_CLASScontainers:-name:hello-appimage:us-docker.pkg.dev/google-samples/containers/gke/hello-app:1.0ports:-containerPort:8080resources:requests:cpu:"250m"memory:"1Gi"

5. Create the Deployment:

   kubectlapply-fPATH_TO_DEPLOYMENT_MANIFEST

   ReplacePATH_TO_DEPLOYMENT_MANIFEST with the path tothe Deployment manifest file.

New nodes that GKE creates for this workload use theConfidential Computing technology that you specified. These nodes run in oneof the zones that you selected, on a machine type that you defined in theComputeClass. GKE adds node taints and node labels so thatonly Pods that select the same ComputeClass can run on the nodes.

Configure Confidential GKE Nodes at the cluster level

• Run the following command:

  gcloudcontainerclusterscreate-autoCLUSTER_NAME\--location=CONTROL_PLANE_LOCATION\--enable-confidential-nodes

  Replace the following:

  • CLUSTER_NAME: the name of the Autopilotcluster.
  • CONTROL_PLANE_LOCATION: the Compute Engineregion for the cluster control plane, such asus-central1.

  The cluster must run version 1.30.2 or later.For more information, seeCreate an Autopilot cluster with a specific release channel and version.

Create a new Standard cluster

You can specify a Confidential Computing technology for your nodes when youcreate a Standard cluster. Specifying the technology when you create acluster has the following effects:

• You can create only node pools or workloads that useConfidential GKE Nodes in that cluster.
• You can't update the cluster to disable Confidential GKE Nodes.
• You can't override the cluster-level Confidential Computing technology inindividual node pools or workloads.

Configuring a Confidential Computing setting at the cluster level is permanent.As a result, consider the following use cases before you create your cluster:

• To use node auto-provisioning in your cluster, you must configure specificoptions in your cluster creation command. For more information, see theUse Confidential GKE Nodes with node auto-provisioningsection.
• To use different Confidential Computing technologies to encrypt specificnode pools in the cluster, skip this section andconfigure Confidential GKE Nodes at the node pool level.

To create a Standard mode cluster that usesConfidential GKE Nodes, select one of the following options:

gcloudcontainerclusterscreateCLUSTER_NAME\--location=LOCATION\--machine-type=MACHINE_TYPE\--node-locations=ZONE1,ZONE2,...\--confidential-node-type=CONFIDENTIAL_COMPUTE_TECHNOLOGY

For more information about creating clusters, seeCreating a regional cluster.

For any node pool created with the Confidential mode for Hyperdisk Balanced setting, only the nodes in thenode pool are restricted to the setup configuration. For any new node poolscreated in the cluster, you must set up confidential mode at creation.

Configure node auto-provisioning

You can configurenode auto-provisioningto use Confidential GKE Nodes in auto-provisioned node pools. Nodeauto-provisioning supports the following Confidential Computing technologies:

• AMD SEV
• AMD SEV-SNP

To use Confidential GKE Nodes with node auto-provisioning, specify the--enable-confidential-nodes gcloud CLI flag when you create acluster. The following additional considerations apply:

• When you create the cluster, choose a Confidential Computing technology thatnode auto-provisioning supports. This choice isirreversible at thecluster level.
• When you enable node auto-provisioning on an existing cluster, the clustermust already use a Confidential GKE Nodes technology that nodeauto-provisioning supports.
• When you deploy workloads, verify that the workloads select theConfidential Computing technology of the cluster and a compatible machineseries for that technology.

Configure Confidential GKE Nodes at the node pool level

You can enable Confidential GKE Nodes on specific node pools ifConfidential GKE Nodes is disabled at the cluster level.

Confidential mode for Hyperdisk Balanced setting must be specified during node pool creation request.

Create a new node pool

To create a new node pool with Confidential GKE Nodes enabled, selectone of the following options:

gcloudcontainernode-poolscreateNODE_POOL_NAME\--location=LOCATION\--cluster=CLUSTER_NAME\--machine-type=MACHINE_TYPE\--node-locations=ZONE1,ZONE2,...\--confidential-node-type=CONFIDENTIAL_COMPUTE_TECHNOLOGY

Update an existing node pool

This change requires recreating the nodes, which can cause disruption to yourrunning workloads. For details about this specific change, find thecorresponding row in themanual changes that recreate the nodes using a nodeupgrade strategy without respecting maintenancepoliciestable. To learn more about node updates, seePlanning for node updatedisruptions.

You can update existing node pools to use Confidential GKE Nodes or toswitch the Confidential Computing technology that the nodes use. The node poolmust meet all of the following requirements:

• The nodes use a machine type that supports the chosen Confidential Computingtechnology.
• The nodes run in zones that support the chosen Confidential Computingtechnology.

Because the machine type and location availability might vary betweenConfidential Computing technologies, you might need to update these attributesfor your node pool before you enable a specific Confidential Computingtechnology.

To update an existing node pool to use Confidential GKE Nodes, selectone of the following options:

Place workloads on only confidential node pools

If you enable Confidential GKE Nodes at the cluster level, all of yourworkloads run on confidential nodes. You don't need to make changes to yourmanifests. However, if you enable Confidential GKE Nodes only forspecific Standard mode node pools at the node pool level, you shoulddeclaratively express that your workloads must run only on node pools withConfidential GKE Nodes.

• To require that a workload runs on a specific Confidential Computing technology,use anode selectorwith thecloud.google.com/gke-confidential-nodes-instance-type label, likein the following example:

  apiVersion:v1kind:Podspec:# For readability, lines are omitted from this example manifestnodeSelector:cloud.google.com/gke-confidential-nodes-instance-type:"CONFIDENTIAL_COMPUTE_SELECTOR"

  ReplaceCONFIDENTIAL_COMPUTE_SELECTOR with the name ofthe technology that the node pool uses. This field supports the followingvalues, which you must specify in uppercase:

  • SEV: AMD SEV
  • SEV_SNP: AMD SEV-SNP
  • TDX: Intel TDX

• To let a workload run on any confidential nodes, regardless of theConfidential Computing technology, use anode affinity rule,like in the following example:

  apiVersion:v1kind:Podspec:containers:-name:confidential-appimage:us-docker.pkg.dev/google-samples/containers/gke/hello-app:1.0affinity:nodeAffinity:requiredDuringSchedulingIgnoredDuringExecution:nodeSelectorTerms:-matchExpressions:-key:cloud.google.com/gke-confidential-nodes-instance-typeoperator:Exists

• To let a workload run on nodes that use only a subset of the availableConfidential Computing technologies, use a node affinity rule that's similarto the following example:

  apiVersion:v1kind:Podspec:containers:-name:confidential-appimage:us-docker.pkg.dev/google-samples/containers/gke/hello-app:1.0affinity:nodeAffinity:requiredDuringSchedulingIgnoredDuringExecution:nodeSelectorTerms:-matchExpressions:-key:cloud.google.com/gke-confidential-nodes-instance-typeoperator:Invalues:-SEV-SEV_SNP-TDX

  In thevalues field, specify only the Confidential Computing technologiesthat you want to run the workload on.

Verify that Confidential GKE Nodes is enabled

You can check whether your clusters or nodes useConfidential GKE Nodes by inspecting the clusters, node pools, nodes,or Compute Engine instances.

Verify for clusters

Select one of the following options:

gcloudcontainerclustersdescribeCLUSTER_NAME\--location=CONTROL_PLANE_LOCATION\--format='yaml(confidentialNodes.confidentialInstanceType)'

confidentialNodes:  confidentialInstanceType:CONFIDENTIAL_COMPUTE_SELECTOR

Verify for node pools

Select one of the following options:

gcloudcontainernode-poolsdescribeNODE_POOL_NAME\--location=CONTROL_PLANE_LOCATION\--format='yaml(confidentialNodes.confidentialInstanceType,enableConfidentialStorage)'

confidentialNodes:  confidentialInstanceType:CONFIDENTIAL_COMPUTE_SELECTOR

confidentialNodes:  confidentialInstanceType:CONFIDENTIAL_COMPUTE_SELECTORenableConfidentialStorage: true

Verify for a specific node

1. Find the node name:

   kubectlgetnodes

2. Describe the node:

   kubectldescribeNODE_NAME

   ReplaceNODE_NAME with the name of a node toinspect.

   The output is similar to the following:

   Name:               gke-cluster-1-default-pool-affsf335r-asdfRoles:              <none>Labels:             cloud.google.com/gke-boot-disk=pd-balanced                    cloud.google.com/gke-container-runtime=containerdcloud.google.com/gke-confidential-nodes-instance-type=CONFIDENTIAL_COMPUTE_SELECTOR                    cloud.google.com/gke-nodepool=default-pool                    cloud.google.com/gke-os-distribution=cos                    cloud.google.com/machine-family=e2# lines omitted for clarity

   In this output, thecloud.google.com/gke-confidential-nodes-instance-typenode label indicates that the node is a confidential node.

Set organization policy constraints

You can define an organization policy constraint to ensure that all VM resourcescreated across your organization are Confidential VM instances.For GKE, you can customize theRestrict Non-ConfidentialComputing constraint to require that all new clusters are created withone of the available Confidential Computing technologies enabled. Add thecontainer.googleapis.com API Service name to the deny list whenenforcing organization policy constraints,like in the following example:

gcloudresource-managerorg-policiesdeny\constraints/compute.restrictNonConfidentialComputingcompute.googleapis.comcontainer.googleapis.com\--project=PROJECT_ID

ReplacePROJECT_ID with your project ID.

Create a PersistentVolume for Confidential mode for Hyperdisk Balanced

For guidance on allowable values for throughput or IOPS, seePlan the performance level for your Hyperdisk volume.

The following examples show how you can create a Confidential mode for Hyperdisk Balanced StorageClass for each Hyperdisk type:

To find the name of the StorageClasses available in your cluster, run thefollowing command:

kubectlgetsc

Limitations

Confidential GKE Nodes has the following limitations:

• All of thelimitations of Compute Engine Confidential VM instancesalso apply to Confidential GKE Nodes.
• Confidential GKE Nodes isn't compatible withsole tenant nodes.
• Confidential GKE Nodes only supportsusing ephemeral storage on local SSDs,but doesn't support using local SSDs in general.
• Confidential GKE Nodes supports Container-Optimized OS andUbuntu node images. Windows nodes aren't supported.
• Autopilot clusters support only AMD SEV at the cluster level. To usea different Confidential Computing technology,configure Confidential GKE Nodes at the workload level.
• Node auto-provisioning supports only AMD SEV and AMD SEV-SNP at the node poollevel. To use a different Confidential Computing technology,configure Confidential GKE Nodes at the workload level.

Confidential mode for Hyperdisk Balanced has the following limitations:

• All of thelimitations of using customer-managed encryption keys (CMEK) to encrypt disksapply to Confidential mode for Hyperdisk Balanced.
• Confidential mode for Hyperdisk Balanced is supported only on Confidential GKE Nodes that useAMD SEV as the Confidential Computing technology.

Live migration limitations

Compute Engine Confidential VM that use the N2Dmachine type and use AMD SEV as the Confidential Computing technologysupportlive migration,which minimizes the potential workload disruption from a host maintenanceevent. Live migration occurs in the following GKE versions:

• 1.27.10-gke.1218000 and later
• 1.28.6-gke.1393000 and later
• 1.29.1-gke.1621000 and later

If your node pools were already running a supported version when live migrationwas added,manually upgrade the node pools to the same or a different supported version. Upgrading the nodes triggers noderecreation, and the new nodes have live migration enabled.

For details about which Compute Engine machine types support livemigration, seeSupported configurations.

If ahost maintenance eventoccurs on a node that doesn't support live migration, the node enters aNotReady state. Running Pods will experience disruptions until the nodebecomes ready again. If the maintenance takes more than five minutes,GKE might try to recreate the Pods on other nodes.

Disable Confidential GKE Nodes

This change requires recreating the nodes, which can cause disruption to yourrunning workloads. For details about this specific change, find thecorresponding row in themanual changes that recreate the nodes using a nodeupgrade strategy without respecting maintenancepoliciestable. To learn more about node updates, seePlanning for node updatedisruptions.

You can only disable Confidential GKE Nodes in Standard modenode pools. If the node pool is in a cluster that uses Confidential GKE Nodesat the cluster level, you can't disable the feature at the node pool level.To disable Confidential GKE Nodes, select one of the following options:

gcloudcontainernode-poolsupdateNODE_POOL_NAME\--cluster=CLUSTER_NAME\--location=CLUSTER_LOCATION\--no-enable-confidential-nodes

What's next

• Learn more about Confidential VM
• Learn more about Google Cloud encryption at rest
• Learn more about Google Cloud encryption in transit
• Learn more about customer-managed encryption keys (CMEK)
• Learn how to remotely attest that workloads are running on Confidential VM
• Learn how to run GPUs on Confidential GKE Nodes