This tutorial shows you how to encrypt KubernetesSecretsat the application layer by using a key that you manage inCloud Key Management Service(Cloud KMS). The process of encrypting Secrets providesan additional layer of security for sensitive workloads.

This page is for Security specialists who want to encrypt Secrets. To learn more aboutcommon roles and example tasks that we reference in Google Cloud content, seeCommon GKE user roles and tasks.

Before reading this page, ensure that you're familiar with the following concepts:

• Key rotation
• Envelope encryption

To follow step-by-step guidance for this task directly in the Google Cloud console, clickGuide me:

Guide me

Overview

By default, Google Kubernetes Engine (GKE)encrypts customer content stored at rest,including Secrets. GKE handles and manages this defaultencryption for you without any additional action on your part.Application-layer secrets encryption provides an additional layer of security for sensitive data, suchas Secrets. You use a key managed withCloud KMS toencrypt data at rest at the application layer.

The state of Kubernetes API objects in your cluster, like Secrets, is storedin a database. Thiscluster state database uses one of the followingtechnologies:

• etcd: etcd database instances run on everycontrol plane VM. Application-layer secrets encryption helps to protect against attackers who getaccess to an offline copy of etcd.
• Spanner: GKE runs aSpanner database on Google's infrastructure. The database isseparate from the cluster's control plane VMs and can't be downloaded as anoffline copy.

To use application-layer secrets encryption, you must first create a Cloud KMS key and givethe GKE service account access to the key. You can use a key thathas any of theprotection levels supported by Cloud KMS.

Ensure that the key is in the same location as the cluster to decrease latency and toprevent cases where resources depend on services spread across multiple failuredomains. After creating a key, you can enable the feature on a new or existingcluster by specifying the key you want to use. When you enable the feature,GKE encrypts all new and existing Secrets using your encryptionkey.

Envelope encryption

Kubernetes offers envelope encryption ofSecrets with aKMS provider, meaning that a local key, commonlycalled adata encryption key (DEK), is used to encrypt the Secrets. The DEKitself is encrypted with another key called thekey encryption key (KEK).Kubernetes does not store the KEK.

Envelope encryption has the following benefits:

• Improved performance compared to public key encryption:GKE only uses the Cloud KMS API to encrypt new DEKs withthe KEK or to decrypt a DEK when the local cache is empty.
• Better management of keys at scale: a single KEK can encrypt multipleDEKs. The number of keys that you need to store in the Cloud KMSservice is far fewer than the number of keys that encrypt your data.
• Ability to use a central root of trust: Secrets that are stored inKubernetes can rely on an external root of trust. This means that you canuse a central root of trust, for example ahardware securitymodule, for all yourSecrets. An adversary accessing your containers offline can't obtain yourSecrets.

With application-layer secrets encryption in GKE, GKE encrypts yourSecrets using local DEKs and theAES-CBC provider.GKE encrypts the DEKs with a KEK that you manage inCloud KMS.

To learn more about envelope encryption, seeEnvelope encryption.

What happens when you create a Secret

When you create a new Secret, here's what happens:

1. The Kubernetes API server generates a unique DEK for the Secret usinga random number generator.

2. The Kubernetes API server uses the DEK locally to encrypt the Secret.

3. TheKMS pluginsends the DEK to Cloud KMS for encryption. The KMSplugin uses your project's GKE service account toauthenticate to Cloud KMS.

4. Cloud KMS encrypts the DEK using the KEK, and sends itback to the KMS plugin.

5. The Kubernetes API server saves the encrypted Secret and the encrypted DEK.The plaintext DEK is not saved to a disk and is only stored in the API servermemory.

6. The Kubernetes API server creates a cache entry mapping the encrypted DEK tothe in-memory plaintext DEK. This lets the API server decrypt the Secretwithout using Cloud KMS.

When a client requests a Secret from the Kubernetes API server, here's whathappens:

1. The Kubernetes API server retrieves the encrypted Secret and the encrypted DEK.

2. The Kubernetes API server checks the cache for an existing mapping entry anddecrypts the Secret without using Cloud KMS.

3. If a cache entry is not found, the KMS plugin sends the DEK toCloud KMS for decryption using the KEK. The decrypted DEK is thenused to decrypt the Secret.

4. The Kubernetes API server returns the decrypted Secret to the client.

What happens when you destroy a key

If you plan to destroy an old KEK version after akey rotation, use the new KEK version tore-encrypt the Secret first. You can optionally retain theprevious KEK version to avoid re-encrypting Secrets, but you continue to bebilled for all active KEKs in Cloud KMS. For pricing details, seeCloud KMS pricing.

Unless you use aService Account Token Volume Projection, serviceaccounts used by your workloads on GKE also use Secrets, andif a key is destroyed these become unavailable. The inability to access thesemeans that the workloads will fail.

The following exceptions apply:

• Pods with existing access to Secrets as mounted volumes or environmentvariables retain access.

• The Kubernetes API server can still use cached DEK mapping entries to decrypta Secret after you destroy the KEK. This lets restarted or rescheduled Podsaccess the Secret unless one of the following situations occur:

  • The cluster control plane is restarted.
  • The Kubernetes API server Pod is restarted.
  • The DEK mapping entry for the Secret is not in the Kubernetes API servercache.

Before you destroy a KEK,check if it is being used by your cluster.You can alsocreate an alerting policyfor key destruction in Cloud KMS. Only destroy previous KEK versions when you're sure thatno data in your cluster is encrypted using the previous version. Before youdestroy a KEK, check whether the key is in use. For details, seeDestroy and restore key versions.

You can schedule a key version for destruction after a configurable period.During this time, if you reconsider, you canrestore the key versionto prevent deletion. However, after the scheduled destruction time arrivesand the key version is destroyed, this action is irreversible.Any data encrypted with this key version will become permanently undecryptable.

Before you begin

• To do the exercises in this topic, you need two Google Cloud projects:

  • Key project: This is where you create a KEK.

  • Cluster project: This is where you create a cluster that enablesapplication-layer secrets encryption.

  You can use the same project for your key project and cluster project.But the recommended practice is to useseparate projects.

• In your key project, ensure that you have enabled the Cloud KMS API.

  Enable Cloud KMS API

• In your key project, the user who creates the key ring and key needs thefollowing IAM permissions:

  • cloudkms.keyRings.getIamPolicy
  • cloudkms.keyRings.setIamPolicy

  These permissions (and many more) are granted to the pre-definedroles/cloudkms.adminIdentity and Access Management role. You can learnmore aboutgranting permissions to manage keysin the Cloud KMS documentation.

• In your cluster project, ensure that you have enabled the Google Kubernetes Engine API.

  Enable Google Kubernetes Engine API

• Ensure that you have installed theGoogle Cloud CLI.

• Updategcloud to the latest version:

  gcloud components update

Create a Cloud KMS key

To create a Cloud KMS key, you must first create akey ring. Keys andkey rings are regional resources. When you create a key ring, specify a locationthat matches the location of your GKE cluster:

• Azonal clustershould use a key ring from a superset region. For example, a cluster in theus-central1-a zone can only use a key in theus-central1 region.

• Aregional cluster shoulduse a key ring from the same location. For example, a cluster in theasia-northeast1 region should be protected with a key ring from theasia-northeast1 region.

The Cloud KMSglobal region is not supported for use withGKE.

You can use the gcloud CLI or the Google Cloud console.

gcloudkmskeyringscreateRING_NAME\--location=LOCATION\--project=KEY_PROJECT_ID

gcloudkmskeyscreateKEY_NAME\--location=LOCATION\--keyring=RING_NAME\--purpose=encryption\--project=KEY_PROJECT_ID

Grant permission to use the key

The GKE service account in your cluster project has the followingname:

service-CLUSTER_PROJECT_NUMBER@container-engine-robot.iam.gserviceaccount.com

ReplaceCLUSTER_PROJECT_NUMBER with your clusterproject number.To find your project number using the gcloud CLI, run the followingcommand:

gcloudprojectsdescribeCLUSTER_PROJECT_ID\--format="value(projectNumber)"

To grant access to the service account, you can use the Google Cloud console or thegcloud CLI.

gcloudkmskeysadd-iam-policy-bindingKEY_NAME\--location=LOCATION\--keyring=RING_NAME\--member=serviceAccount:SERVICE_ACCOUNT_NAME\--role=roles/cloudkms.cryptoKeyEncrypterDecrypter\--project=KEY_PROJECT_ID

Ensure that the key has enough quota if it is a Cloud HSM key

If you use a Cloud HSM key, the Google Cloud project that contains thekey is limited by yourkey quota. Ensure thatyou have enough quota to use your Cloud HSM keys with application-layer secrets encryption. Ifyour quota is exhausted, your nodes might lose connectivity to your clustercontrol plane.

Enable application-layer secrets encryption

You can enable application-layer secrets encryption on new or existing GKE Standardand GKE Autopilot clusters using the gcloud CLIor the Google Cloud console.

Enable on a new cluster

You can create a new cluster with application-layer secrets encryption enabled by using theGoogle Cloud console or the gcloud CLI.

gcloudcontainerclusterscreate-autoCLUSTER_NAME\--cluster-version=latest\--location=CONTROL_PLANE_LOCATION\--database-encryption-key=projects/KEY_PROJECT_ID/locations/LOCATION/keyRings/RING_NAME/cryptoKeys/KEY_NAME\--project=CLUSTER_PROJECT_ID

Enable on an existing cluster

You can use the gcloud CLI or the Google Cloud console to update anexisting cluster to use application-layer secrets encryption. GKE encrypts all of yourexisting and new Secrets using your specified encryption key.

Updating an existing cluster to use application-layer secrets encryption restarts the clustercontrol plane. During this process, GKEre-encrypts all existing Secrets with the new key, so expect a long-running operation time. For zonal clusters, the control plane becomesunavailable.

gcloudcontainerclustersupdateCLUSTER_NAME\--location=CONTROL_PLANE_LOCATION\--database-encryption-key=projects/KEY_PROJECT_ID/locations/LOCATION/keyRings/RING_NAME/cryptoKeys/KEY_NAME\--project=CLUSTER_PROJECT_ID

Update a Cloud KMS key

You can use the gcloud CLI or the Google Cloud console to update anexisting cluster to use a new Cloud KMS key.

Updating an existing cluster to use a new Cloud KMS keyrestarts the cluster control plane. During this process, GKEre-encrypts all existing Secrets with the new key, so expect a long-runningoperation time. For zonal clusters, the control plane becomes unavailable duringthe update.

gcloudcontainerclustersupdateCLUSTER_NAME\--location=CONTROL_PLANE_LOCATION\--database-encryption-key=projects/KEY_PROJECT_ID/locations/LOCATION/keyRings/RING_NAME/cryptoKeys/KEY_NAME\--project=CLUSTER_PROJECT_ID

Disable application-layer secrets encryption

To disable application-layer secrets encryption, you can use the gcloud CLI or theGoogle Cloud console.

gcloudcontainerclustersupdateCLUSTER_NAME\--location=CONTROL_PLANE_LOCATION\--disable-database-encryption\--project=CLUSTER_PROJECT_ID

Verify that application-layer secrets encryption is enabled

You can check to see whether a cluster is using application-layer secrets encryption using theGoogle Cloud console or the gcloud CLI.

gcloudcontainerclustersdescribeCLUSTER_NAME\--location=CONTROL_PLANE_LOCATION\--format='value(databaseEncryption)'\--project=CLUSTER_PROJECT_ID

keyName=projects/KEY_PROJECT_ID/locations/LOCATION/keyRings/RING_NAME/cryptoKeys/KEY_NAME;state=ENCRYPTED

Rotate your keys

When you perform a key rotation, your existing secrets remain encrypted with theprevious key encryption key (KEK) version. To ensure a newer KEK version wraps aSecret,re-encrypt the Secret after Key rotation.

For example, you create and store a Secret,Secret1.It is encrypted withDEK1, which itself is wrapped withKEKv1.

After the KEK rotates, you re-encryptSecret1 so that it is wrapped byDEK2,which in turn is wrapped withKEKv2, the rotated KEK.

The key version rotation is an eventually consistent operation and therecould be a delay before the new key version takes effect. SeeConsistency of key versions for moreinformation.

Re-encrypt your Secrets

After performing a key rotation, you should re-encrypt your Secrets to wrap themwith the new version of the KEK. You can re-encrypt your Secrets in one of thefollowing ways:

• Manually, by running akubectl command in a terminal.
• Automatically, by using a recurring workload, such as aCronJob, to run akubectlcommand at regular intervals.

After a key rotation, waitat least three hours for the new version to becomeconsistent. Then, trigger re-encryption by running a command like the following:

kubectlgetsecrets--namespace=NAMESPACE-ojson\|kubectlannotate--overwrite-f-encryption-key-rotation-time="TIME"

Replace the following:

• NAMESPACE: the namespace to update Secrets in. InStandard clusters, you can optionally use the--all-namespacesflag instead to update every Secret in the cluster with the same command.In Autopilot clusters, you can update only the namespaces that youown.
• TIME: a string that indicates when the rotation happens(for example,20200909-090909).

After you rotate your keys, the previous key version continues to exist, andmight incur costs. Key version destruction is permanent, so make sure thatthe previous key version is no longer in use before you destroy it. For moreinformation, seeView key usage.

Limitations

• GKE supports up to 30,000 secrets per cluster forapplication-layer secrets encryption. If you store more than 30,000 secrets, your cluster mightbecome unstable at upgrade time, causing a potential outage for yourworkloads.
• Make sure the average size of metadata of a secret in every namespace islower than 5KiB. If the average size of metadata is above 5KiB, your clustermight enter a bad state where some secrets are encrypted while others aredecrypted after enabling the feature or disabling the feature.

• You must select a key in the same region as the cluster. For example, azonal cluster inus-central1-a can only use a key in the regionus-central1. For regional clusters, keys must be in the same location todecrease latency and to prevent cases where resources depend on servicesspread across multiple failure domains.

  The key does not need to be in the same project as the cluster. Formore information about the supported locations for Cloud KMS, seeGoogle Cloud locations.

• GKE only supports keys from Cloud KMS. You cannotuse another KubernetesKMS provideror anotherencryption provider.

Troubleshooting

For information about troubleshooting application-layer secrets encryption, including resolving issueswith failed Secret encryption updates, seeTroubleshoot application-layer secrets encryption.

Cloud KMS key is disabled

GKE's default service account cannot use a disabledCloud KMS key forapplication-layer secrets encryption.

To re-enable a disabled key, seeEnable a disabled key version.

Cloud KMS key version is destroyed

When the cluster status contains the message:KEY_VERSION_URI is not enabled, current state is: DESTROYED, it means that the key version used forapplication-layer secrets encryption wasdestroyed.

ReplaceKEY_VERSION_URI with the URI of the key version.

What's next

• Learn more about Secrets in Kubernetes.
• Learn more about Secret management using Cloud KMS.
• Learn how to harden your cluster.