Run your own certificate authorities and keys in GKE

Note: This document uses ageneral availability feature namedGKE control plane authority that's only available for eligible Google Cloud customers. To request access to this feature, contact your Google Cloud account team.

This page shows you how to configure your Google Kubernetes Engine (GKE) clustercontrol plane with certificate authorities (CAs) and keys that you manage. Thisguidance is intended for security administrators who have specificorganizational compliance or policy requirements for control over credentialissuance and signing.

This page describes one part of a set of optional control plane features in GKE that lets you perform tasks like verifying your control plane security posture or configuring encryption and credential signing in the control plane using keys that you manage. For details, seeAbout GKE control plane authority.

By default, Google Cloud applies various security measures to the managed control plane. This page describesoptional capabilities that give you more visibility or control over the GKE control plane.

You should already be familiar with the following concepts:

Control plane credential components

GKE clusters use specific CAs and keys to issue credentials inthe cluster, like X.509 certificates or ServiceAccount tokens. You can createkeys inCloud Key Management Service (Cloud KMS)and CAs inCertificate Authority Service(CA Service) and configure your clusters to use these resourcesinstead of Google Cloud-managed CAs and keys.

To learn more about the specific components that you create, seeSelf-managed CAs and keys.

Usage with other GKE control plane authority features

GKE control plane authority provides the following features related toself-managed keys:

Note: You can only enable these features when you create a new GKEcluster. You can't update existing clusters to use these features. To use bothof these features in the same cluster, perform all of the key and CAconfiguration procedures inboth guides, and then run the cluster creationcommand that enables both sets of features as described in theCreate a cluster section.

Objectives

  • Create keys in Cloud KMS
  • Create CAs in CA Service
  • Grant Identity and Access Management (IAM) roles to the GKE service agent
  • Create a GKE cluster that uses your CAs and keys
  • Verify that the cluster uses your CAs and keys

Costs

In this document, you use the following billable components of Google Cloud:

To generate a cost estimate based on your projected usage, use thepricing calculator.

New Google Cloud users might be eligible for afree trial.

When you finish the tasks that are described in this document, you can avoid continued billing by deleting the resources that you created. For more information, seeClean up.

Before you begin

  1. Sign in to your Google Cloud account. If you're new to Google Cloud, create an account to evaluate how our products perform in real-world scenarios. New customers also get $300 in free credits to run, test, and deploy workloads.

  2. Install the Google Cloud CLI.

    Note: If you installed the gcloud CLI previously, make sure you have the latest version by runninggcloud components update.

  3. If you're using an external identity provider (IdP), you must first sign in to the gcloud CLI with your federated identity.

  4. Toinitialize the gcloud CLI, run the following command:

    gcloudinit

  5. Verify that billing is enabled for your Google Cloud project.

  6. Enable the Kubernetes Engine, Certificate Authority Service, and Cloud Key Management Service APIs:

    Roles required to enable APIs

    To enable APIs, you need the Service Usage Admin IAM role (roles/serviceusage.serviceUsageAdmin), which contains theserviceusage.services.enable permission.Learn how to grant roles.

    gcloudservicesenablecontainer.googleapis.com privateca.googleapis.com cloudkms.googleapis.com

  7. Install the Google Cloud CLI.

    Note: If you installed the gcloud CLI previously, make sure you have the latest version by runninggcloud components update.

  8. If you're using an external identity provider (IdP), you must first sign in to the gcloud CLI with your federated identity.

  9. Toinitialize the gcloud CLI, run the following command:

    gcloudinit

  10. Verify that billing is enabled for your Google Cloud project.

  11. Enable the Kubernetes Engine, Certificate Authority Service, and Cloud Key Management Service APIs:

    Roles required to enable APIs

    To enable APIs, you need the Service Usage Admin IAM role (roles/serviceusage.serviceUsageAdmin), which contains theserviceusage.services.enable permission.Learn how to grant roles.

    gcloudservicesenablecontainer.googleapis.com privateca.googleapis.com cloudkms.googleapis.com
  12. Ensure that your environment is eligible to use GKE control plane authority features. To opt-in to these features, contact your Google Cloud sales team.
  13. To reliably track credential issuance and usage, ensure that the following Data Access audit logs are enabled:
    • Cloud Key Management Service (KMS) API:DATA_READ
    • Certificate Authority Service:ADMIN_READ

    To enable these log types, seeEnable Data Access audit logs.

Required roles and permissions

To get the permissions that you need to run your own CAs and keys, ask your administrator to grant you the following IAM roles:

For more information about granting roles, seeManage access to projects, folders, and organizations.

You might also be able to get the required permissions throughcustom roles or otherpredefined roles.

Requirements

You must use GKE version 1.31.1-gke.1846000 or later.

Limitations

The following limitations apply:

  • You can only use keys from Cloud KMS. You can't use another KMSprovider or another encryption provider.
  • Cloud External Key Manager (Cloud EKM) keys aren't supported.
  • You can only use CAs from CA Service.

  • The regions and zones in which you can use GKE control plane authority depend on whether you also want to use specific features, as follows:

    • To encrypt your control plane boot disks with a customer-managed encryption key, your cluster must be in one of the following regions:
      • asia-east1
      • asia-northeast1
      • asia-southeast1
      • europe-west1
      • europe-west4
      • us-central1
      • us-east1
      • us-east4
      • us-east5
      • us-south1
      • us-west1
      • us-west3
      • us-west4
    • To use Confidential GKE Nodes with GKE control plane authority, your cluster must be in aregion that supports Confidential mode for Hyperdisk Balanced.

    If you don't use these features, you can use GKE control plane authority in any Google Cloud location.

Prepare the environment

In this section, you identify Google Cloud projects that you'll use inthis tutorial, and you create a key ring in Cloud KMS to hold yourkeys.

Identify projects

We recommend that you use separate Google Cloud projects as follows:

  • Key project: contains all keys and CAs.
  • Cluster project: contains your GKE clusters.

You can optionally use the same project for your keys, CAs, andGKE clusters, but we recommend that you use separate projects sothat the teams that manage cryptographic operations in your organization areseparated from the teams that manage cluster operations.

Create a key ring

Create a key ring in the key project to hold all of the keys for a specificcluster. You must create the key ring in the same location as yourGKE cluster.

Note: You can't delete a key ring after creation. Key rings don't incuradditional costs. However, having too many key rings in a project might makemanaging your resources in Cloud KMS more tedious.

Run the following command:

gcloudkmskeyringscreateKEY_RING_NAME\--location=LOCATION\--project=KEY_PROJECT_ID

Replace the following:

  • KEY_RING_NAME: a name for your key ring.
  • KEY_PROJECT_ID: the project ID of your key project.
  • LOCATION: the Google Cloud region where youwant to create the key ring. This must be the same region where yourGKE cluster is located.

Create keys

For each of the credential authorities like service account keys and CAs, youcreate a key using Cloud KMS. This section shows you how to create thekeys that GKE uses to sign and verify credentials in the cluster.You can specify your own properties for these keys depending on yourorganization's needs. For details, see theCreate a keypage and theprojects.locations.keyRings.cryptoKeysAPI reference.

Caution: Don't enable automatic key rotation with Cloud KMS. Rotatingyour encryption keys requires additional steps to ensure that your control planeresources are re-encrypted with new key versions before the original keyversions are destroyed.

Consider the following when you create these resources in Cloud KMS:

To create your keys, run the following commands:

  1. Create the Kubernetes ServiceAccount signing key, which you also specify asthe service account verification key during cluster creation:

    gcloudkmskeyscreatesa-signing-key\--keyring=KEY_RING_NAME\--location=LOCATION\--purpose="asymmetric-signing"\--protection-level=hsm\--default-algorithm=rsa-sign-pkcs1-4096-sha256\--project=KEY_PROJECT_ID

    ReplaceKEY_PROJECT_ID with the project ID for yourdedicated key project.

  2. Create the cluster root CA key:

    gcloudkmskeyscreatecluster-ca-key\--keyring=KEY_RING_NAME\--location=LOCATION\--purpose="asymmetric-signing"\--protection-level=hsm\--default-algorithm=ec-sign-p256-sha256\--project=KEY_PROJECT_ID

  3. Create the etcd peer root CA key:

    gcloudkmskeyscreateetcd-peer-ca-key\--keyring=KEY_RING_NAME\--location=LOCATION\--purpose="asymmetric-signing"\--protection-level=hsm\--default-algorithm=ec-sign-p256-sha256\--project=KEY_PROJECT_ID

  4. Create the etcd API root CA key:

    gcloudkmskeyscreateetcd-api-ca-key\--keyring=KEY_RING_NAME\--location=LOCATION\--purpose="asymmetric-signing"\--protection-level=hsm\--default-algorithm=ec-sign-p256-sha256\--project=KEY_PROJECT_ID

  5. Create the aggregation root CA key:

    gcloudkmskeyscreateaggregation-ca-key\--keyring=KEY_RING_NAME\--location=LOCATION\--purpose="asymmetric-signing"\--protection-level=hsm\--default-algorithm=ec-sign-p256-sha256\--project=KEY_PROJECT_ID

Create the CAs

After creating the keys for each of the control plane functions, use each keyto create the CA pools and corresponding root CAs usingCA Service:

  1. Create the cluster CA pool:

    gcloudprivatecapoolscreatecluster-ca-pool\--location=LOCATION\--tier=enterprise\--project=KEY_PROJECT_ID\--no-publish-crl--no-publish-ca-cert

    The--no-publish-crl flag and the--no-publish-ca-cert flag are optional.Omitting these flags publishes certificates to a Cloud Storagebucket. For details, seeEnable CA certificate and CRL publication for CAs in a CA pool.

  2. Create the cluster root CA:

    gcloudprivatecarootscreatecluster-root-ca\--pool=cluster-ca-pool\--location=LOCATION\--kms-key-version=projects/KEY_PROJECT_ID/locations/LOCATION/keyRings/KEY_RING_NAME/cryptoKeys/cluster-ca-key/cryptoKeyVersions/1\--subject="CN=cluster-ca, O=ORGANIZATION"\--project=KEY_PROJECT_ID\--auto-enable

    ReplaceORGANIZATION with your organization name.

  3. Create the etcd peer CA pool:

    gcloudprivatecapoolscreateetcd-peer-ca-pool\--location=LOCATION\--tier=enterprise\--project=KEY_PROJECT_ID\--no-publish-crl--no-publish-ca-cert

  4. Create the etcd peer root CA:

    gcloudprivatecarootscreateetcd-peer-root-ca\--pool=etcd-peer-ca-pool\--location=LOCATION\--kms-key-version=projects/KEY_PROJECT_ID/locations/LOCATION/keyRings/KEY_RING_NAME/cryptoKeys/etcd-peer-ca-key/cryptoKeyVersions/1\--subject="CN=etcd-peer-ca, O=ORGANIZATION"\--project=KEY_PROJECT_ID\--auto-enable

  5. Create the etcd API CA pool:

    gcloudprivatecapoolscreateetcd-api-ca-pool\--location=LOCATION\--tier=enterprise\--project=KEY_PROJECT_ID\--no-publish-crl--no-publish-ca-cert

  6. Create the etcd API root CA:

    gcloudprivatecarootscreateetcd-api-root-ca\--pool=etcd-api-ca-pool\--location=LOCATION\--kms-key-version=projects/KEY_PROJECT_ID/locations/LOCATION/keyRings/KEY_RING_NAME/cryptoKeys/etcd-api-ca-key/cryptoKeyVersions/1\--subject="CN=etcd-api-ca, O=ORGANIZATION"\--project=KEY_PROJECT_ID\--auto-enable

  7. Create the aggregation CA pool:

    gcloudprivatecapoolscreateaggregation-ca-pool\--location=LOCATION\--tier=enterprise\--project=KEY_PROJECT_ID\--no-publish-crl--no-publish-ca-cert

  8. Create the aggregation root CA:

    gcloudprivatecarootscreateaggregation-root-ca\--pool=aggregation-ca-pool\--location=LOCATION\--kms-key-version=projects/KEY_PROJECT_ID/locations/LOCATION/keyRings/KEY_RING_NAME/cryptoKeys/aggregation-ca-key/cryptoKeyVersions/1\--subject="CN=aggregation-ca, O=ORGANIZATION"\--project=KEY_PROJECT_ID\--auto-enable

Grant IAM roles to the GKE service agent

TheGKE service agentrequires access to the resources that you created in Cloud KMS and inCA Service. The service agent uses these resources to sign,verify, and issue credentials in the cluster. You can use the followingpredefined IAM roles:

To grant these roles to the GKE service agent, do the following:

  1. Find the project number of your cluster project:

    gcloudprojectsdescribeCLUSTER_PROJECT_ID\--format='value(projectNumber)'

    ReplaceCLUSTER_PROJECT_ID with the project ID ofyour cluster project.

  2. Grant the Kubernetes Engine KMS Crypto Key User role on the service accountsigning key that you created inCreate keys:

    gcloudkmskeysadd-iam-policy-bindingsa-signing-key\--location=LOCATION\--keyring=KEY_RING_NAME\--member="serviceAccount:service-CLUSTER_PROJECT_NUMBER@container-engine-robot.iam.gserviceaccount.com"\--role=roles/container.cloudKmsKeyUser\--project=KEY_PROJECT_ID

    ReplaceCLUSTER_PROJECT_NUMBER with the projectnumber of the cluster project.

  3. Grant the CA Service Certificate Manager role on the CA pools that youcreated inCreate the CAs:

    gcloudprivatecapoolsadd-iam-policy-bindingcluster-ca-pool\--location=LOCATION\--member="serviceAccount:service-CLUSTER_PROJECT_NUMBER@container-engine-robot.iam.gserviceaccount.com"\--role=roles/privateca.certificateManager\--project=KEY_PROJECT_IDgcloudprivatecapoolsadd-iam-policy-bindingetcd-peer-ca-pool\--location=LOCATION\--member="serviceAccount:service-CLUSTER_PROJECT_NUMBER@container-engine-robot.iam.gserviceaccount.com"\--role=roles/privateca.certificateManager\--project=KEY_PROJECT_IDgcloudprivatecapoolsadd-iam-policy-bindingetcd-api-ca-pool\--location=LOCATION\--member="serviceAccount:service-CLUSTER_PROJECT_NUMBER@container-engine-robot.iam.gserviceaccount.com"\--role=roles/privateca.certificateManager\--project=KEY_PROJECT_IDgcloudprivatecapoolsadd-iam-policy-bindingaggregation-ca-pool\--location=LOCATION\--member="serviceAccount:service-CLUSTER_PROJECT_NUMBER@container-engine-robot.iam.gserviceaccount.com"\--role=roles/privateca.certificateManager\--project=KEY_PROJECT_ID

Grant additional roles when not using the gcloud CLI

This section describes additional configuration steps that you must perform ifyou plan to configure your CAs and keys by using a client like Terraform orthe Google Cloud console instead of using the gcloud CLI. If youuse the gcloud CLI, skip this section and go to theSet up CAs and keys on a new cluster section.

When you use the gcloud CLI to set up your CAs and keys, asdescribed in this page, the gcloud CLI automatically creates andconfigures a service agent for CA Service and grantsIAM roles to the service agent. However, if you use a client likeTerraform or the Google Cloud console to configure your Google Cloudenvironment, you must perform these configuration steps manually as follows foryour key project:

  1. Trigger creation of the CA Service service agent.

    Preview — triggering service agent creation

    This feature is subject to the "Pre-GA Offerings Terms" in the General Service Terms section of theService Specific Terms. Pre-GA features are available "as is" and might have limited support. For more information, see thelaunch stage descriptions.

    gcloudbetaservicesidentitycreate--service=privateca.googleapis.com\--project=KEY_PROJECT_ID

  2. Find the project number of your key project:

    gcloudprojectsdescribeKEY_PROJECT_ID\--format='value(projectNumber)'

  3. Grant theViewer (roles/viewer)role and theCloud KMS CryptoKey Signer/Verifier(roles/cloudkms.signerVerifier) role on all of the root CA keys that youcreated in theCreate keys section:

    forkeyincluster-ca-keyetcd-peer-ca-keyetcd-api-ca-keyaggregation-ca-keydogcloudkmskeysadd-iam-policy-binding$key\--keyring=KEY_RING_NAME\--location=LOCATION\--role=roles/viewer\--member="serviceAccount:service-KEY_PROJECT_NUMBER@gcp-sa-privateca.iam.gserviceaccount.com"\--project=KEY_PROJECT_IDgcloudkmskeysadd-iam-policy-binding$key\--keyring=KEY_RING_NAME\--location=LOCATION\--role=roles/cloudkms.signerVerifier\--member="serviceAccount:service-KEY_PROJECT_NUMBER@gcp-sa-privateca.iam.gserviceaccount.com"\--project=KEY_PROJECT_IDdone

    ReplaceKEY_PROJECT_NUMBER with the key projectnumber from the output of the previous step.

    This command is afor loop that iterates through the root CA keys,granting each role on that key to the CA Service serviceagent. If you used different names for the root CA keys, run these commandsmanually for each key.

Set up CAs and keys on a new cluster

Warning: Deleting keys or CAs that are in use by the cluster might lead toirrecoverable clusters. Before you delete keys or CAs, ensure that your clusterno longer uses the resources.

After you create keys, CA pools, root CAs, and grant IAM roles tothe GKE service agent, create a new cluster that uses theseresources.

The flags that you specify in the cluster creation command require thefollowing resource paths as values:

Note: Before you perform these steps, decide on whether you also want to encryptyour control plane boot disks and etcd disks with keys that you manage. If youdo, perform all of the key configuration steps inEncrypt etcd and control plane boot disksand specify the optional flags described in the cluster creation command in thissection.

To configure a new cluster to use your keys and CAs, do the following steps:

  1. Find the path to the latest enabled service account signing key version:

    gcloudkmskeysversionslist\--key=sa-signing-key\--keyring=KEY_RING_NAME\--location=LOCATION\--project=KEY_PROJECT_ID\--filter="STATE=ENABLED"--sort-by=~--format="value(name)"|sed1q

    ReplaceKEY_PROJECT_ID with the project ID of thekey project.

    The output is similar to the following:

    projects/KEY_PROJECT_ID/locations/LOCATION/keyRings/KEY_RING_NAME/cryptoKeys/sa-signing-key/cryptoKeyVersions/1

  2. Find the paths to each of the CA pools that you created:

    gcloudprivatecapoolslist--format="get(name)"\--project=KEY_PROJECT_ID

    The output is similar to the following:

    projects/KEY_PROJECT_ID/locations/LOCATION/caPools/cluster-ca-poolprojects/KEY_PROJECT_ID/locations/LOCATION/caPools/etcd-peer-ca-poolprojects/KEY_PROJECT_ID/locations/LOCATION/caPools/etcd-api-ca-poolprojects/KEY_PROJECT_ID/locations/LOCATION/caPools/aggregation-ca-pool

    Ensure that the output contains all of the CA pools that you created forGKE.

Create a cluster

In this section, you create a cluster with different options specifieddepending on which GKE control plane authority features you want toconfigure. You can only configure these features on a cluster during clustercreation. The following commands create Standard mode clusters. Tocreate Autopilot mode clusters instead, use the same flags with thegcloud container clusters create-auto command.

  • To only configure the CAs and keys that you created in this tutorial, run thefollowing command:

    gcloudcontainerclusterscreateexample-cluster\--location=LOCATION\--project=CLUSTER_PROJECT_ID\--cluster-version=VERSION\--service-account-signing-keys=projects/KEY_PROJECT_ID/locations/LOCATION/keyRings/KEY_RING_NAME/cryptoKeys/sa-signing-key/cryptoKeyVersions/1\--service-account-verification-keys=projects/KEY_PROJECT_ID/locations/LOCATION/keyRings/KEY_RING_NAME/cryptoKeys/sa-signing-key/cryptoKeyVersions/1\--cluster-ca=projects/KEY_PROJECT_ID/locations/LOCATION/caPools/cluster-ca-pool\--etcd-peer-ca=projects/KEY_PROJECT_ID/locations/LOCATION/caPools/etcd-peer-ca-pool\--etcd-api-ca=projects/KEY_PROJECT_ID/locations/LOCATION/caPools/etcd-api-ca-pool\--aggregation-ca=projects/KEY_PROJECT_ID/locations/LOCATION/caPools/aggregation-ca-pool

    Replace the following:

    • CLUSTER_PROJECT_ID: the project ID of the clusterproject.
    • VERSION: the GKE version of thecluster. Must be 1.31.1-gke.1846000 or later.

  • To configure the CAs and keys as well as control plane boot disk encryptionand etcd encryption, do the following:

    1. Perform all of the key configuration steps inEncrypt etcd and control plane boot disks.
    2. Find the paths to each of the keys by using the instructions inUse encryption keys in a cluster.

    3. Create a cluster:

      gcloudcontainerclusterscreateexample-cluster\--location=LOCATION\--project=CLUSTER_PROJECT_ID\--cluster-version=VERSION\--service-account-signing-keys=projects/KEY_PROJECT_ID/locations/LOCATION/keyRings/KEY_RING_NAME/cryptoKeys/sa-signing-key/cryptoKeyVersions/1\--service-account-verification-keys=projects/KEY_PROJECT_ID/locations/LOCATION/keyRings/KEY_RING_NAME/cryptoKeys/sa-signing-key/cryptoKeyVersions/1\--cluster-ca=projects/KEY_PROJECT_ID/locations/LOCATION/caPools/cluster-ca-pool\--etcd-peer-ca=projects/KEY_PROJECT_ID/locations/LOCATION/caPools/etcd-peer-ca-pool\--etcd-api-ca=projects/KEY_PROJECT_ID/locations/LOCATION/caPools/etcd-api-ca-pool\--aggregation-ca=projects/KEY_PROJECT_ID/locations/LOCATION/caPools/aggregation-ca-pool\--control-plane-disk-encryption-key=PATH_TO_DISK_KEY\--gkeops-etcd-backup-encryption-key=PATH_TO_ETCD_BACKUP_KEY

      Replace the following:

      • CLUSTER_PROJECT_ID: the project ID of thecluster project.
      • VERSION: the GKE version of thecluster. Must be 1.31.1-gke.1846000 or later.
      • PATH_TO_DISK_KEY: the path to your diskencryption key.
      • PATH_TO_ETCD_BACKUP_KEY: the path to your etcdinternal backup encryption key.

    You can also use these flags when you create a new Standard modecluster.

Verify that the cluster uses your specified keys and CAs

This section shows you how to verify the keys and CAs that were used duringcluster creation. You can perform this verification using Cloud Logging orby using the Google Cloud CLI.

Use Logging to verify keys and CAs

To verify the keys and CAs using Logging, do the following:

  1. In the Google Cloud console, go to theLogs Explorer page:

    Go to Logs Explorer

  2. Specify the following query:

    resource.type="gke_cluster"resource.labels.cluster_name="CLUSTER_NAME"resource.labels.location="CLUSTER_LOCATION"protoPayload.serviceName="container.googleapis.com"protoPayload.methodName=~"google.container.v(1|1alpha1|1beta1).ClusterManager.CreateCluster"protoPayload.request.cluster.userManagedKeysConfig:*

    protoPayload.request.cluster.userManagedKeysConfig:* filters the resultsfor cluster creation logs that include keys and CAs that you manage.

  3. ClickRun query.

In the results, expand your cluster creation log. Verify that the paths to keys and CAs are the same as those that you created for that cluster, like in the following example:

# lines omitted for clarityuserManagedKeysConfig: {  aggregationCa: "projects/KEY_PROJECT_ID/locations/LOCATION/caPools/aggregation-ca-pool"  clusterCa: "projects/KEY_PROJECT_ID/locations/LOCATION/caPools/cluster-ca-pool"  etcdApiCa: "projects/KEY_PROJECT_ID/locations/LOCATION/caPools/etcd-api-ca-pool"  etcdPeerCa: "projects/KEY_PROJECT_ID/locations/LOCATION/caPools/etcd-peer-ca-pool"  serviceAccountSigningKeys: [    0: "projects/KEY_PROJECT_ID/locations/LOCATION/keyRings/KEY_RING_NAME/cryptoKeys/sa-signing-key/cryptoKeyVersions/1"  ]  serviceAccountVerificationKeys: [    0: "projects/KEY_PROJECT_ID/locations/LOCATION/keyRings/KEY_RING_NAME/cryptoKeys/sa-signing-key/cryptoKeyVersions/1"  ]}

Use the gcloud CLI to verify keys and CAs

To verify that the cluster uses the CAs and keys that you created, run thefollowing command:

gcloudcontainerclustersdescribeexample-cluster\--location=LOCATION\--project=CLUSTER_PROJECT_ID

The output should include theuserManagedKeysConfig field like in thefollowing example:

# lines omitted for clarityuserManagedKeysConfig:  sa-signing-key: projects/KEY_PROJECT_ID/locations/LOCATION/keyRings/KEY_RING_NAME/cryptoKeys/sa-signing-key/cryptoKeyVersions/1  sa-verification-key: projects/KEY_PROJECT_ID/locations/LOCATION/keyRings/KEY_RING_NAME/cryptoKeys/sa-signing-key/cryptoKeyVersions/1  cluster-ca: projects/KEY_PROJECT_ID/locations/LOCATION/caPools/cluster-ca-pool  etcd-peer-ca: projects/KEY_PROJECT_ID/locations/LOCATION/caPools/etcd-peer-ca-pool  etcd-api-ca: projects/KEY_PROJECT_ID/locations/LOCATION/caPools/etcd-api-ca-pool  aggregation-ca: projects/KEY_PROJECT_ID/locations/LOCATION/caPools/aggregation-ca-pool

Clean up

To avoid incurring charges to your Google Cloud account for the resources used in this tutorial, either delete the project that contains the resources, or keep the project and delete the individual resources.

Delete the projects

    Caution: Deleting a project has the following effects:
    • Everything in the project is deleted. If you used an existing project for the tasks in this document, when you delete it, you also delete any other work you've done in the project.
    • Custom project IDs are lost. When you created this project, you might have created a custom project ID that you want to use in the future. To preserve the URLs that use the project ID, such as anappspot.com URL, delete selected resources inside the project instead of deleting the whole project.

    If you plan to explore multiple architectures, tutorials, or quickstarts, reusing projects can help you avoid exceeding project quota limits.

    Delete a Google Cloud project:

    gcloud projects deletePROJECT_ID

Delete individual resources

  1. Delete the cluster:

    gcloudcontainerclustersdeleteexample-cluster\--location=LOCATION\--project=CLUSTER_PROJECT_ID

  2. Disable the root CAs:

    gcloudprivatecarootsdisablecluster-root-ca\--location=LOCATION\--pool=projects/KEY_PROJECT_ID/locations/LOCATION/caPools/cluster-ca-pool\--project=KEY_PROJECT_IDgcloudprivatecarootsdisableetcd-peer-root-ca\--location=LOCATION\--pool=projects/KEY_PROJECT_ID/locations/LOCATION/caPools/etcd-peer-ca-pool\--project=KEY_PROJECT_IDgcloudprivatecarootsdisableetcd-api-root-ca\--location=LOCATION\--pool=projects/KEY_PROJECT_ID/locations/LOCATION/caPools/etcd-api-ca-pool\--project=KEY_PROJECT_IDgcloudprivatecarootsdisableaggregation-root-ca\--location=LOCATION\--pool=projects/KEY_PROJECT_ID/locations/LOCATION/caPools/aggregation-ca-pool\--project=KEY_PROJECT_ID

  3. Delete the root CAs:

    gcloudprivatecarootsdeletecluster-root-ca\--location=LOCATION\--pool=projects/KEY_PROJECT_ID/locations/LOCATION/caPools/cluster-ca-pool\--project=KEY_PROJECT_IDgcloudprivatecarootsdeleteetcd-peer-root-ca\--location=LOCATION\--pool=projects/KEY_PROJECT_ID/locations/LOCATION/caPools/etcd-peer-ca-pool\--project=KEY_PROJECT_IDgcloudprivatecarootsdeleteetcd-api-root-ca\--location=LOCATION\--pool=projects/KEY_PROJECT_ID/locations/LOCATION/caPools/etcd-api-ca-pool\--project=KEY_PROJECT_IDgcloudprivatecarootsdeleteaggregation-root-ca\--location=LOCATION\--pool=projects/KEY_PROJECT_ID/locations/LOCATION/caPools/aggregation-ca-pool\--project=KEY_PROJECT_ID

  4. Delete the CA pools:

    gcloudprivatecapoolsdeletecluster-ca-pool--location=LOCATION\--project=KEY_PROJECT_IDgcloudprivatecapoolsdeleteetcd-peer-ca-pool--location=LOCATION\--project=KEY_PROJECT_IDgcloudprivatecapoolsdeleteetcd-api-ca-pool--location=LOCATION\--project=KEY_PROJECT_IDgcloudprivatecapoolsdeleteaggregation-ca-pool--location=LOCATION\--project=KEY_PROJECT_ID

  5. Delete the keys:

    gcloudkmskeysversionsdestroy1\--location=LOCATION\--keyring=KEY_RING_NAME\--key=sa-signing-key\--project=KEY_PROJECT_IDgcloudkmskeysversionsdestroy1\--location=LOCATION\--keyring=KEY_RING_NAME\--key=cluster-ca-key\--project=KEY_PROJECT_IDgcloudkmskeysversionsdestroy1\--location=LOCATION\--keyring=KEY_RING_NAME\--key=etcd-peer-ca-key\--project=KEY_PROJECT_IDgcloudkmskeysversionsdestroy1\--location=LOCATION\--keyring=KEY_RING_NAME\--key=etcd-api-ca-key\--project=KEY_PROJECT_IDgcloudkmskeysversionsdestroy1\--location=LOCATION\--keyring=KEY_RING_NAME\--key=aggregation-ca-key\--project=KEY_PROJECT_ID

You can't delete key rings from Cloud KMS. However, key rings don'tincur additional costs.

What's next

Except as otherwise noted, the content of this page is licensed under theCreative Commons Attribution 4.0 License, and code samples are licensed under theApache 2.0 License. For details, see theGoogle Developers Site Policies. Java is a registered trademark of Oracle and/or its affiliates.

Last updated 2025-12-15 UTC.