This page shows you how to encrypt data that's stored in yourGoogle Kubernetes Engine (GKE) control plane using keys that you manage inCloud Key Management Service (Cloud KMS). You should already be familiar withconcepts likeetcd, theGKE cluster architecture,andCloud KMS.

This page describes one part of a set of optional control plane features in GKE that lets you perform tasks like verifying your control plane security posture or configuring encryption and credential signing in the control plane using keys that you manage. For details, seeAbout GKE control plane authority.

By default, Google Cloud applies various security measures to the managed control plane. This page describesoptional capabilities that give you more visibility or control over the GKE control plane.

About control plane boot disk and etcd encryption

By default, GKE encrypts the boot disk of a control plane node,the disk that stores data in etcd, and the Google Cloud internaloperational backup of etcd by using encryption keys that Google Cloudmanages. For details about this default encryption, seeDefault encryption at rest. You canoptionally use your own encryptionkeys that you manage using Cloud KMS to encrypt these resources. Tolearn more, seeControl plane boot disk and etcd encryption.

You create keys in Cloud KMS that GKE uses to encryptyour control plane resources. Consider the following when you create theseresources:

• You can use one key ring for all of the keys in a cluster, regardless ofthe purpose of each key. If you have an existing key ring that you used fora different purpose, likesetting up your own certificate authorities,you can use that key ring for this guide.
• You should create the keys in the same Google Cloud location as yourcluster for better latency.
• For most use cases, you can use thesoftware Cloud KMS keyprotection level. You can also use hardware keys with Cloud HSM.
• You must specify the--purpose flag with theencryption value becausethese keys are used for symmetric encryption.
• You shouldn't modify thedefault duration for key destruction.

Usage with other GKE control plane authority features

GKE control plane authority provides the following features related toself-managed keys that you must enable at the same time when you create acluster:

• Encrypt control plane components (this page)
• Run your own certificate authorities (CAs) and keys

You can only enable these features when you create a new GKEcluster. You can't update existing clusters to use these features. To use bothof these features in the same cluster, perform all of the key and CAconfiguration procedures inboth guides, and then run the cluster creationcommand that enables both sets of features as described in theCreate a cluster section.

Before you begin

Before you start, make sure that you have performed the following tasks:

• Enable the Google Kubernetes Engine API.
Enable Google Kubernetes Engine API
• If you want to use the Google Cloud CLI for this task,install and theninitialize the gcloud CLI. If you previously installed the gcloud CLI, get the latest version by running thegcloud components update command. Earlier gcloud CLI versions might not support running the commands in this document.Note: For existing gcloud CLI installations, make sure to set thecompute/regionproperty. If you use primarily zonal clusters, set thecompute/zone instead. By setting a default location, you can avoid errors in the gcloud CLI like the following:One of [--zone, --region] must be supplied: Please specify location. You might need to specify the location in certain commands if the location of your cluster differs from the default that you set.

• Ensure that your key project has a Cloud KMS key ring for yourcluster. You can use any existing key ring in your cluster location. Tocreate a new key ring, seeCreate a key ring.

• Enable the Cloud Key Management Service API.

  Roles required to enable APIs

  To enable APIs, you need the Service Usage Admin IAM role (roles/serviceusage.serviceUsageAdmin), which contains theserviceusage.services.enable permission.Learn how to grant roles.

  Enable the API

Identify projects

We recommend that you use separate Google Cloud projects as follows:

• Key project: contains all keys.
• Cluster project: contains your GKE clusters.

You can optionally use the same project for your keys andGKE clusters, but we recommend that you use separate projects sothat the teams that manage your keys and cryptographic operations are separatefrom the teams that manage your clusters.

Required roles and permissions

To get the permissions that you need to run your own encryption keys, ask your administrator to grant you the following IAM roles:

• Create Cloud KMS keys:Cloud KMS Admin (roles/cloudkms.admin) on your key project
• Create GKE clusters:Kubernetes Engine Cluster Admin (roles/container.clusterAdmin) on your cluster project

For more information about granting roles, seeManage access to projects, folders, and organizations.

You might also be able to get the required permissions throughcustom roles or otherpredefined roles.

Requirements

Your cluster must run GKE version 1.31.1-gke.1846000 or later.

Limitations

• You can only configure boot disk and etcd encryption keys during clustercreation.

• For regional Standard mode clusters and for Autopilotclusters, the region in which you create a cluster must have capacity forConfidential mode for Hyperdisk Balanced in at least three zones in that region.

  For zonal Standard mode clusters, the cluster zonemust have Hyperdisk Balanced capacity. For help withcapacity, contact Cloud Customer Care.

• GKE only supports keys from Cloud KMS. You can't useanother Kubernetes KMS provider or another encryption provider.

• Cloud External Key Manager (Cloud EKM) keys aren't supported.

• You can't access or interact with the Google Cloud internaloperational backups of etcd, which are only for disaster recovery.

• Multi-regional keyrings aren't supported. You must use a regional key ring.

• The regions and zones in which you can use GKE control plane authority depend on whether you also want to use specific features, as follows:

  • To encrypt your control plane boot disks with a customer-managed encryption key, your cluster must be in one of the following regions:
    • asia-east1
    • asia-northeast1
    • asia-southeast1
    • europe-west1
    • europe-west4
    • us-central1
    • us-east1
    • us-east4
    • us-east5
    • us-south1
    • us-west1
    • us-west3
    • us-west4
  • To use Confidential GKE Nodes with GKE control plane authority, your cluster must be in aregion that supports Confidential mode for Hyperdisk Balanced.

  If you don't use these features, you can use GKE control plane authority in any Google Cloud location.

Create keys

In this section, you create an encryption key for the boot disks and etcd disksin your control plane and a separate encryption key for the Google Cloudinternal operational backup of etcd. You can use one key ring to hold all ofthese keys and any other keys for the cluster.

1. Create the encryption key for your control plane boot disks and etcd disks:

   gcloudkmskeyscreateKCP_DISK_KEY_NAME\--keyring=KEYRING_NAME\--location=LOCATION\--purpose="encryption"\--protection-level=PROTECTION_LEVEL\--project=KEY_PROJECT_ID

   Replace the following:

   • KCP_DISK_KEY_NAME: the name for the encryptionkey for your control plane boot disks and etcd disks.
   • KEYRING_NAME: the name of the key ring to holdyour encryption keys for the cluster.
   • LOCATION: the Google Cloud location forthe key ring. This must be the same as your cluster location. For a listof regions, filter for "Region" in theCloud KMS locations table.
   • PROTECTION_LEVEL: the protection level for thekey, likesoftware orhsm.
   • KEY_PROJECT_ID: the project ID of yourkey project.

2. Create the etcd internal backup encryption key:

   gcloudkmskeyscreateETCD_BACKUP_KEY_NAME\--keyring=KEYRING_NAME\--location=LOCATION\--purpose="encryption"\--protection-level=PROTECTION_LEVEL\--project=KEY_PROJECT_ID

   ReplaceETCD_BACKUP_KEY_NAME with a name for theetcd internal backup encryption key.

Grant IAM roles to the GKE service agent

In this section, you grant IAM roles on the keys thatyou created to theGKE service agentin the cluster project. The GKE service agent requires theseroles to use these keys to encrypt the corresponding control plane resources.

1. Find your cluster project number:

   gcloudprojectsdescribeCLUSTER_PROJECT_ID\--format='value(projectNumber)'

   ReplaceCLUSTER_PROJECT_ID with the project ID ofyour GKE cluster project.

   The output is similar to the following:

   1234567890

2. Grant theCloud KMS CryptoKey Encrypter/Decrypter(roles/cloudkms.cryptoKeyEncrypterDecrypter) role on the encryption keyfor boot disks and etcd disks to the GKE service agent in thecluster project:

   gcloudkmskeysadd-iam-policy-bindingKCP_DISK_KEY_NAME\--location=LOCATION\--keyring=KEYRING_NAME\--member="serviceAccount:service-CLUSTER_PROJECT_NUMBER@container-engine-robot.iam.gserviceaccount.com"\--role=roles/cloudkms.cryptoKeyEncrypterDecrypter\--project=KEY_PROJECT_ID

   Replace the following:

   • KCP_DISK_KEY_NAME: the name of the diskencryption key.
   • LOCATION: the Google Cloud location forthe key.
   • KEYRING_NAME: the name of the key ring thatcontains the encryption key.
   • CLUSTER_PROJECT_NUMBER: the numerical projectnumber of the cluster project, which you found in the previous step.
   • KEY_PROJECT_ID: the project ID of yourkey project.

3. Grant theCloud KMS CryptoKey Encrypter/Decrypter Via Delegation(roles/cloudkms.cryptoKeyEncrypterDecrypterViaDelegation) role on the encryption keyfor boot disks and etcd disks to the GKE service agent in thecluster project:

   gcloudkmskeysadd-iam-policy-bindingKCP_DISK_KEY_NAME\--location=LOCATION\--keyring=KEYRING_NAME\--member="serviceAccount:service-CLUSTER_PROJECT_NUMBER@container-engine-robot.iam.gserviceaccount.com"\--role=roles/cloudkms.cryptoKeyEncrypterDecrypterViaDelegation\--project=KEY_PROJECT_ID

4. Grant theCloud KMS Key User role on the encryption keys for boot disks and etcd disks to the GKE service agent in the cluster project for the key rotation:

   gcloudkmskeysadd-iam-policy-bindingKCP_DISK_KEY_NAME\--location=LOCATION\--keyring=KEYRING_NAME\--member="serviceAccount:service-CLUSTER_PROJECT_NUMBER@container-engine-robot.iam.gserviceaccount.com"\--role=roles/container.cloudKmsKeyUser\--project=KEY_PROJECT_ID

5. Grant theCloud KMS CryptoKey Encrypter(roles/cloudkms.cryptoKeyEncrypter) role on the etcd internalbackup encryption key to the GKE service agent in the clusterproject:

   gcloudkmskeysadd-iam-policy-bindingETCD_BACKUP_KEY_NAME\--location=LOCATION\--keyring=KEYRING_NAME\--member="serviceAccount:service-CLUSTER_PROJECT_NUMBER@container-engine-robot.iam.gserviceaccount.com"\--role=roles/cloudkms.cryptoKeyEncrypter\--project=KEY_PROJECT_ID

   ReplaceETCD_BACKUP_KEY_NAME with the name of theetcd operational backup encryption key.

   Granting theroles/cloudkms.cryptoKeyEncrypter role preventsGKE from performing database restorations on your behalf andsignificantly increases the amount of time to restore functionality when adatabase problem occurs. To allow GKE to perform restorationsfor you, grant theroles/cloudkms.cryptoKeyEncrypterDecrypter roleinstead.

Use encryption keys in a cluster

This section shows you how to identify the paths to your encryption keys.

1. Identify the path to your disk encryption key:

   gcloudkmskeysdescribeKCP_DISK_KEY_NAME\--keyring=KEYRING_NAME\--location=LOCATION\--project=KEY_PROJECT_ID\--format="value(name)"

   Replace the following:

   • KCP_DISK_KEY_NAME: the name of the encryptionkey for control plane boot disks and etcd disks.
   • KEYRING_NAME: the name of the keyring that contains the key.
   • LOCATION: the Google Cloud location of thekey.
   • KEY_PROJECT_ID: the project ID of yourkey project.

   The output is similar to the following:

   projects/KEY_PROJECT_ID/locations/LOCATION/keyRings/KEYRING_NAME/cryptoKeys/disk-encryption-key

2. Identify the path to your etcd internal backup encryption key:

   gcloudkmskeysdescribeETCD_BACKUP_KEY_NAME\--keyring=KEYRING_NAME\--location=LOCATION\--project=KEY_PROJECT_ID\--format="value(name)"

   ReplaceETCD_BACKUP_KEY_NAME with the name of theetcd operational backup encryption key.

   The output is similar to the following:

   projects/KEY_PROJECT_ID/locations/LOCATION/keyRings/KEYRING_NAME/cryptoKeys/etcd-backup-encryption-key

Create a cluster

In this section, you create a cluster with different options specifieddepending on which GKE control plane authority features you want toconfigure. You can only configure these features on a cluster during clustercreation. The following commands create Standard mode clusters. Tocreate Autopilot mode clusters instead, use the same flags with thegcloud container clusters create-auto command.

• To create a cluster that configures disk encryption and runs your own CAs andservice account signing keys, do the following:

  1. Perform all of the key and CA configuration steps inRun your own certificate authorities and keys.
  2. Find the paths to each of the service account keys and CAs by using theinstructions inSet up CAs and keys on a new cluster.

  3. Create a cluster:

     gcloudcontainerclusterscreateCLUSTER_NAME\--location=LOCATION\--project=CLUSTER_PROJECT_ID\--control-plane-disk-encryption-key=PATH_TO_DISK_KEY\--gkeops-etcd-backup-encryption-key=PATH_TO_ETCD_BACKUP_KEY\--service-account-signing-keys=PATH_TO_SIGNING_KEY_VERSION\--service-account-verification-keys=PATH_TO_VERIFICATION_KEY_VERSION\--cluster-ca=PATH_TO_CLUSTER_CA\--etcd-peer-ca=PATH_TO_ETCD_PEER_CA\--etcd-api-ca=PATH_TO_ETCD_API_CA\--aggregation-ca=PATH_TO_AGGREGATION_CA

     Replace the following:

     • CLUSTER_NAME: the name of your new cluster.
     • LOCATION: the location of your new cluster.
     • CLUSTER_PROJECT_ID: the project ID of yourcluster project.
     • PATH_TO_DISK_KEY: the path to your diskencryption key from the previous steps on this page.
     • PATH_TO_ETCD_BACKUP_KEY: the path to your etcdinternal backup encryption key from the previous steps on this page.
     • PATH_TO_SIGNING_KEY_VERSION: the path to theKubernetes ServiceAccount signing key version in Cloud KMS.
     • PATH_TO_VERIFICATION_KEY_VERSION: the path to theKubernetes ServiceAccount verification key version in Cloud KMS.
     • PATH_TO_CLUSTER_CA: the path to the clusterCA pool.
     • PATH_TO_ETCD_PEER_CA: the path to the etcdpeer CA pool.
     • PATH_TO_ETCD_API_CA: the path to the etcdAPI CA pool.
     • PATH_TO_AGGREGATION_CA: the path to theaggregation CA pool.

• To create a cluster that only configures disk encryption using the keys thatyou created in this guide, run the following command:

  gcloudcontainerclusterscreateCLUSTER_NAME\--location=LOCATION\--project=CLUSTER_PROJECT_ID\--control-plane-disk-encryption-key=PATH_TO_DISK_KEY\--gkeops-etcd-backup-encryption-key=PATH_TO_ETCD_BACKUP_KEY

  Replace the following:

  • CLUSTER_NAME: the name of your new cluster.
  • LOCATION: the location of your new cluster.
  • CLUSTER_PROJECT_ID: the project ID of yourcluster project.
  • PATH_TO_DISK_KEY: the path to your diskencryption key from the previous steps.
  • PATH_TO_ETCD_BACKUP_KEY: the path to your etcdinternal backup encryption key from the previoussteps.

You can also specify all of these flags when you create a new Standardmode cluster.

Verify the encryption key status

This section shows you how to verify the encryption key that was used duringcluster creation. You can perform this verification using Cloud Logging orby using the Google Cloud CLI.

Use Logging to verify keys

To verify the keys using Logging, do the following:

1. In the Google Cloud console, go to theLogs Explorer page:

   Go to Logs Explorer

2. Get the cluster creation log by specifying the following query:

   resource.type="gke_cluster"resource.labels.cluster_name="CLUSTER_NAME"resource.labels.location="CLUSTER_LOCATION"protoPayload.serviceName="container.googleapis.com"protoPayload.methodName=~"google.container.v(1|1alpha1|1beta1).ClusterManager.CreateCluster"protoPayload.request.cluster.userManagedKeysConfig:*

3. ClickRun query.

In the output, check that the cluster creation parameters included a key paththat corresponds to the key that you set up in Cloud KMS, like in thefollowing example:

# lines omitted for clarityuserManagedKeysConfig: {  controlPlaneDiskEncryptionKey: "projects/KEY_PROJECT_ID/locations/LOCATION/keyRings/KEY_RING_NAME/cryptoKeys/KCP_DISK_KEY_NAME"  gkeopsEtcdBackupEncryptionKey: "projects/KEY_PROJECT_ID/locations/LOCATION/keyRings/KEY_RING_NAME/cryptoKeys/ETCD_BACKUP_KEY_NAME"}

Use the gcloud CLI to verify keys

To use the gcloud CLI to verify the encryption key, do the following:

1. For the disk encryption key, run the following command:

   gcloudcontainerclustersdescribeCLUSTER_NAME\--location=LOCATION\--format="value(userManagedKeysConfig.controlPlaneDiskEncryptionKey)"

2. For the etcd internal backup encryption key, run the following command:

   gcloudcontainerclustersdescribeCLUSTER_NAME\--location=LOCATION\--format="value(userManagedKeysConfig.gkeopsEtcdBackupEncryptionKey)"

Rotate etcd and control plane disk encryption keys

This feature is subject to the "Pre-GA Offerings Terms" in the General Service Terms section of theService Specific Terms. Pre-GA features are available "as is" and might have limited support. For more information, see thelaunch stage descriptions.

The encryption keys that you create don't expire. To improve your securityposture, regularly rotate these keys and re-encrypt your resources with newkey versions. For more information, seeRotate etcd and control plane boot disk encryption keys.

What's next

• Learn about what else you can view in the control plane
• Run your own certificate authorities and keys in GKE
• Verify control plane VM integrity