Rotate customer-managed control plane CAs and keys

Note: This document uses ageneral availability feature namedGKE control plane authority that's only available for eligible Google Cloud customers. To request access to this feature, contact your Google Cloud account team.

Preview — customer-managed credential rotation

This feature is subject to the "Pre-GA Offerings Terms" in the General Service Terms section of theService Specific Terms. Pre-GA features are available "as is" and might have limited support. For more information, see thelaunch stage descriptions.

This page shows cluster administrators and Security engineers how to rotatethe certificate authorities (CAs) and service account signing keys that youconfigured for GKE control plane authority.

You should be familiar with the following concepts:

Plan credential rotations

This page shows you how to rotate the following credential components in yourcontrol plane:

  • The cluster root CA, the aggregation root CA, the etcd API root CA, and theetcd peer root CA.
  • The Kubernetes ServiceAccount signing and verification keys.

You can also rotate the customer-managed encryption keys that you used toencrypt your control plane boot disks, etcd disks, and the etcd internalbackup that Google Cloud uses for disaster recovery. For more information,seeRotate etcd and control plane boot disk encryption keys.

Best practice:

Validate your key and credential rotation proceduresbefore an actualincident occurs.

Rotate your credentials to avoid CA expiry, to mitigate key version compromises,and as part of your organization's security practices. To plan how often yourotate specific GKE control plane authority resources, consider thefollowing:

  • By default, GKE certificates signed by the root CAs inCertificate Authority Service (CA Service) expire one year after thecreation date.
  • Keys in Cloud Key Management Service (Cloud KMS) don't expire. Perform a manual keyrotation only if your organization has requirements for key rotation. Tominimize disruptions to running workloads, don't configure automatic keyrotation for these keys.

Before you begin

Before you start, make sure that you have performed the following tasks:

  • Have an existing cluster that usesself-managed CAs and service account keys

  • Identify the project IDs of the following Google Cloud projects:

    • Key project: the project that contains your Cloud KMS andCA Service resources.
    • Cluster project: the project that contains your GKEcluster.

  • To do the validation tasks on this page, verify that the following Data Accessaudit logs are enabled:

    • Cloud Key Management Service (KMS) API:DATA_READ
    • Certificate Authority Service:ADMIN_READ

    To enable these log types, seeEnable Data Access audit logs.

Required roles and permissions

To get the permissions that you need to rotate your customer-managed CAs and keys, ask your administrator to grant you the following IAM roles:

For more information about granting roles, seeManage access to projects, folders, and organizations.

You might also be able to get the required permissions throughcustom roles or otherpredefined roles.

Limitations

The asymmetric Cloud KMS keys that you use for service account signingand verification don't support automatic key rotation.

Rotate the service account signing and verification keys

When you setup GKE control plane authority, you add a serviceaccount signing key and service account verification keys to your cluster.GKE uses these keys to sign and validate bearer tokens forKubernetes ServiceAccounts. During a rotation, you add your new key to the listof service account verification keys, wait for changes to propagate, and thenreplace the service account signing key with the new key.

To rotate the service account keys, follow these steps:

  1. Get the full resource name of the original key version of the serviceaccount signing key:

    gcloudcontainerclustersdescribeCLUSTER_NAME\--project=CLUSTER_PROJECT_ID\--location=CONTROL_PLANE_LOCATION\--format="value(userManagedKeysConfig.serviceAccountSigningKeys)"

    Replace the following:

    • CLUSTER_NAME: the name of your cluster.
    • CONTROL_PLANE_LOCATION: the location of the cluster.
    • CLUSTER_PROJECT_ID: the project ID of thecluster project.

    The output is similar to the following:

    projects/KEY_PROJECT_ID/locations/CONTROL_PLANE_LOCATION/keyRings/KEYRING_NAME/cryptoKeys/SIGNING_KEY_NAME/cryptoKeyVersions/ORIGINAL_SIGNING_KEY_VERSION

    In this output,SIGNING_KEY_NAME is the name ofthe key andORIGINAL_SIGNING_KEY_VERSION is thenumber of the original signing key version.

  2. Create a new key version for the service account signing key:

    gcloudkmskeysversionscreate\--key=SIGNING_KEY_NAME\--keyring=KEYRING_NAME\--location=CONTROL_PLANE_LOCATION\--project=KEY_PROJECT_ID

    Replace the following:

    • SIGNING_KEY_NAME: the name of the serviceaccount signing key.
    • KEYRING_NAME: the name of the key ring thatholds the key.
    • CONTROL_PLANE_LOCATION: the Google Cloud location ofthe key ring.
    • KEY_PROJECT_ID: the project ID of yourkey project.

  3. Get the full resource name of the new key version:

    gcloudkmskeysversionslist\--key=SIGNING_KEY_NAME\--keyring=KEYRING_NAME\--location=CONTROL_PLANE_LOCATION\--project=KEY_PROJECT_ID\--filter="STATE=ENABLED"--sort-by=~\--format="value(name)"|sed1q

    The output is similar to the following:

    projects/KEY_PROJECT_ID/locations/CONTROL_PLANE_LOCATION/keyRings/KEYRING_NAME/cryptoKeys/SIGNING_KEY_NAME/cryptoKeyVersions/NEW_SIGNING_KEY_VERSION

    In this output,SIGNING_KEY_NAME is the name ofthe key andNEW_SIGNING_KEY_VERSION is thenumber of the new signing key version.

  4. Add the new key version to the set of service account verification keys forthe cluster:

    gcloudcontainerclustersupdateCLUSTER_NAME\--location=CONTROL_PLANE_LOCATION\--project=CLUSTER_PROJECT_ID\--service-account-verification-keys=ORIGINAL_KEY_VERSION_PATH,NEW_KEY_VERSION_PATH

    Replace the following:

    • ORIGINAL_KEY_VERSION_PATH: the full resourcename of the original signing key version from the output of the firststep in this section. For example,projects/example-key-project/locations/us-central1/keyRings/example-keyring/cryptokeys/example-signing-key/cryptoKeyVersions/1.
    • NEW_KEY_VERSION_PATH: the full resource name ofthe new signing key version from the output of the previous step. Forexample,projects/example-key-project/locations/us-central1/keyRings/example-keyring/cryptokeys/example-signing-key/cryptoKeyVersions/2.

    After the cluster update operation completes, it might take up to 10minutes for the new key version path to propagate to the KubernetesAPI server and every GKE API endpoint.

  5. To confirm that the new key version path is fully propagated, do thefollowing:

    1. Get the public component of the cluster signing keys from theGKE API as a JSON Web Key Set (JWKS):

      curlhttps://container.googleapis.com/v1/projects/CLUSTER_PROJECT_ID/locations/CONTROL_PLANE_LOCATION/clusters/CLUSTER_NAME/jwks

      The output is similar to the following:

      {  "keys": [    {      "kty": "RSA",      "alg": "RS256",      "use": "sig",      "kid": "KEY1_ID",      "n": "KEY1_MODULUS",      "e": "KEY1_EXPONENT"    },    {      "kty": "RSA",      "alg": "RS256",      "use": "sig",      "kid": "KEY2_ID",      "n": "KEY2_MODULUS",      "e": "KEY2_EXPONENT"    }  ]}

      This output must have two key entries, which indicates that both of thekey versions are available in the GKE API. If you onlysee one key entry, wait for 10 minutes and retry the command.

    2. Connect to the cluster so that you can runkubectl commands:

      gcloudcontainerclustersget-credentialsCLUSTER_NAME\--location=CONTROL_PLANE_LOCATION

    3. Get the public component of the cluster signing keys from theKubernetes API server as a JWKS:

      kubectlget--raw/openid/v1/jwks|jq

      The output is similar to the following:

      {  "keys": [    {      "kty": "RSA",      "alg": "RS256",      "use": "sig",      "kid": "KEY1_ID",      "n": "KEY1_MODULUS",      "e": "KEY1_EXPONENT"    },    {      "kty": "RSA",      "alg": "RS256",      "use": "sig",      "kid": "KEY2_ID",      "n": "KEY2_MODULUS",      "e": "KEY2_EXPONENT"    }  ]}

      This output must have two key entries, which indicates that both of thekey versions are available in the Kubernetes API server. If you onlysee one key entry, wait for 10 minutes and retry the command.

  6. Update the cluster to use the new key version as the service account signingkey:

    gcloudcontainerclustersupdateCLUSTER_NAME\--location=CONTROL_PLANE_LOCATION\--project=CLUSTER_PROJECT_ID\--service-account-signing-keys=NEW_KEY_VERSION_PATH

  7. Verify that new service account tokens use the new key version:

    1. Create a ServiceAccount:

      kubectlcreateserviceaccounttest-sa-1

    2. Create a token for the ServiceAccount:

      kubectlcreatetokentest-sa-1

    3. Extract a recently signed digest from Logging:

      exportSIGNED_DIGEST=$(gcloudloggingread\'resource.type="gke_cluster" '\'AND resource.labels.cluster_name="CLUSTER_NAME"'\'AND protoPayload.methodName="google.cloud.gkeauth.v1.Auth.SignServiceAccountJWT" '\'AND protoPayload.metadata.subject="system:serviceaccount:default:test-sa-1"'\--freshness=1h\--bucket=_Required\--location=global\--view=_AllLogs\--order=DESC\--limit=1\--format="value(protoPayload.metadata.toBeSignedDigest)")
      1. Validate that the new signing key version is in use:
      gcloudloggingread\'resource.type="cloudkms_cryptokeyversion" '\'AND protoPayload.methodName="AsymmetricSign" '\'AND protoPayload.request.digest.sha256="'${SIGNED_DIGEST}'"'\--freshness=1h\--bucket=_Default\--location=global\--view=_AllLogs\--order=DESC\--limit=1\--format="value(protoPayload.resourceName)"```Theoutputistheresourcepathofthenewsigningkeyversion.

  8. Wait for every token in the cluster that uses the original service accountsigning key version to expire. By default, the token lifetime is one hour,with a configurable maximum of 24 hours. To check the configured tokenlifetime in your cluster, run the following command:

    kubectlgetpods-A-ojson|jq-r'.items[]?.spec?.volumes[]?.projected?.sources[]?.serviceAccountToken?.expirationSeconds | select(. != null)'|sort-nr|head-n1
    Note: if you store ServiceAccount tokens in Secrets, recreate the Secretsso that Kubernetes generates new tokens that are signed by the new keyversion.

  9. Wait for the configured token lifetime from the output of the previous stepto pass. After this period of time passes, all of the bound tokens in yourcluster use the new service account signing key version.

  10. Remove the original key version from the list of verification keys for thecluster:

    gcloudcontainerclustersupdateCLUSTER_NAME\--location=CONTROL_PLANE_LOCATION\--project=CLUSTER_PROJECT_ID\--service-account-verification-keys=NEW_KEY_VERSION_PATH

  11. Optional: Disable the original key version. After you verify that theoriginal key version isn't being used and that the cluster is healthy,destroy the key version.

    Unless you're rotating the keys in response to a critical event, werecommend that you wait for a few days before you destroy the original keyversion. For more information, seeDestroy and restore key versions.

After you complete these steps, every new and existing service account token inyour cluster is signed by the new key version. The API server rejects anyrequests that use a bearer token with the original key version, because theoriginal key version doesn't exist in the cluster configuration.

Rotate the GKE control plane authority CAs

The following sections show you how to rotate the root CAs that the cluster usesfor GKE control plane authority.

Create new key versions for the CAs

  1. Create a new key version for the cluster root CA key:

    gcloudkmskeysversionscreate\--key=CLUSTER_CA_KEY_NAME\--keyring=KEYRING_NAME\--location=CONTROL_PLANE_LOCATION\--project=KEY_PROJECT_ID

    Replace the following:

    • CLUSTER_CA_KEY_NAME: the name of the clusterroot CA key for the cluster.
    • KEYRING_NAME: the name of the key ring thatholds the key.
    • CONTROL_PLANE_LOCATION: the Google Cloud location ofthe key ring. This is the same as your cluster location.
    • KEY_PROJECT_ID: the project ID of yourkey project.

  2. Create a new key version for the aggregation root CA key:

    gcloudkmskeysversionscreate\--key=AGGREGATION_CA_KEY_NAME\--keyring=KEYRING_NAME\--location=CONTROL_PLANE_LOCATION\--project=KEY_PROJECT_ID

    ReplaceAGGREGATION_CA_KEY_NAME with the name of theaggregation root CA key for the cluster.

  3. Create a new key version for the etcd API root CA key:

    gcloudkmskeysversionscreate\--key=ETCD_API_CA_KEY_NAME\--keyring=KEYRING_NAME\--location=CONTROL_PLANE_LOCATION\--project=KEY_PROJECT_ID

    ReplaceETCD_API_CA_KEY_NAME with the name of theetcd API root CA key for the cluster.

  4. Create a new key version for the etcd peer root CA key:

    gcloudkmskeysversionscreate\--key=ETCD_PEER_CA_KEY_NAME\--keyring=KEYRING_NAME\--location=CONTROL_PLANE_LOCATION\--project=KEY_PROJECT_ID

    ReplaceETCD_PEER_CA_KEY_NAME with the name of theetcd peer root CA key for the cluster.

Create new root CAs

  1. Get the full resource name of the new cluster root CA key version:

    gcloudkmskeysversionslist\--key=CLUSTER_CA_KEY_NAME\--keyring=KEYRING_NAME\--location=CONTROL_PLANE_LOCATION\--project=KEY_PROJECT_ID\--filter="STATE=ENABLED"--sort-by=~\--format="value(name)"|sed1q

    The output is similar to the following:

    projects/KEY_PROJECT_ID/locations/CONTROL_PLANE_LOCATION/keyRings/KEYRING_NAME/cryptoKeys/CLUSTER_CA_KEY_NAME/cryptoKeyVersions/VERSION

    In this output,VERSION is the number of the newkey version.

  2. Create a new cluster root CA in the cluster CA pool:

    gcloudprivatecarootscreateCLUSTER_ROOT_CA_NAME\--pool=CLUSTER_CA_POOL_NAME\--location=CONTROL_PLANE_LOCATION\--kms-key-version=CLUSTER_CA_KEY_PATH\--subject="CN=cluster-ca, O=ORGANIZATION"\--project=KEY_PROJECT_ID

    Replace the following:

    • CLUSTER_ROOT_CA_NAME: a name for your new root CA.
    • CLUSTER_CA_POOL_NAME: the name of the cluster CApool.
    • CLUSTER_CA_KEY_PATH: the full resource name of thenew key version from the output of the preceding step.
    • ORGANIZATION: your organization name.

  3. Get the full resource name of the new aggregation root CA key version:

    gcloudkmskeysversionslist\--key=AGGREGATION_CA_KEY_NAME\--keyring=KEYRING_NAME\--location=CONTROL_PLANE_LOCATION\--project=KEY_PROJECT_ID\--filter="STATE=ENABLED"--sort-by=~\--format="value(name)"|sed1q

    The output is similar to the following:

    projects/KEY_PROJECT_ID/locations/CONTROL_PLANE_LOCATION/keyRings/KEYRING_NAME/cryptoKeys/AGGREGATION_CA_KEY_NAME/cryptoKeyVersions/VERSION

  4. Create a new aggregation root CA in the aggregation CA pool:

    gcloudprivatecarootscreateAGGREGATION_ROOT_CA_NAME\--pool=AGGREGATION_CA_POOL_NAME\--location=CONTROL_PLANE_LOCATION\--kms-key-version=AGGREGATION_CA_KEY_PATH\--subject="CN=aggregation-ca, O=ORGANIZATION"\--project=KEY_PROJECT_ID

    Replace the following:

    • AGGREGATION_ROOT_CA_NAME: a name for your newaggregation root CA.
    • AGGREGATION_CA_POOL_NAME: the name of theaggregation CA pool.
    • AGGREGATION_CA_KEY_PATH: the full resource name ofthe new key version from the output of the preceding step.

  5. Get the full resource name of the new etcd API root CA key version:

    gcloudkmskeysversionslist\--key=ETCD_API_CA_KEY_NAME\--keyring=KEYRING_NAME\--location=CONTROL_PLANE_LOCATION\--project=KEY_PROJECT_ID\--filter="STATE=ENABLED"--sort-by=~\--format="value(name)"|sed1q

    The output is similar to the following:

    projects/KEY_PROJECT_ID/locations/CONTROL_PLANE_LOCATION/keyRings/KEYRING_NAME/cryptoKeys/ETCD_API_CA_KEY_NAME/cryptoKeyVersions/VERSION

  6. Create a new etcd API root CA in the etcd API CA pool:

    gcloudprivatecarootscreateETCD_API_ROOT_CA_NAME\--pool=ETCD_API_CA_POOL_NAME\--location=CONTROL_PLANE_LOCATION\--kms-key-version=ETCD_API_CA_KEY_PATH\--subject="CN=etcd-api-ca, O=ORGANIZATION"\--project=KEY_PROJECT_ID

    Replace the following:

    • ETCD_API_ROOT_CA_NAME: a name for your newetcd API root CA.
    • ETCD_API_CA_POOL_NAME: the name of theetcd API CA pool.
    • ETCD_API_CA_KEY_PATH: the full resource name ofthe new key version from the output of the preceding step.

  7. Get the full resource name of the new etcd peer root CA key version:

    gcloudkmskeysversionslist\--key=ETCD_PEER_CA_KEY_NAME\--keyring=KEYRING_NAME\--location=CONTROL_PLANE_LOCATION\--project=KEY_PROJECT_ID\--filter="STATE=ENABLED"--sort-by=~\--format="value(name)"|sed1q

    The output is similar to the following:

    projects/KEY_PROJECT_ID/locations/CONTROL_PLANE_LOCATION/keyRings/KEYRING_NAME/cryptoKeys/ETCD_PEER_CA_KEY_NAME/cryptoKeyVersions/VERSION

  8. Create a new etcd peer root CA in the etcd peer CA pool:

    gcloudprivatecarootscreateETCD_PEER_ROOT_CA_NAME\--pool=ETCD_PEER_CA_POOL_NAME\--location=CONTROL_PLANE_LOCATION\--kms-key-version=ETCD_PEER_CA_KEY_PATH\--subject="CN=etcd-peer-ca, O=ORGANIZATION"\--project=KEY_PROJECT_ID

    Replace the following:

    • ETCD_PEER_ROOT_CA_NAME: a name for your newetcd peer root CA.
    • ETCD_PEER_CA_POOL_NAME: the name of theetcd peer CA pool.
    • ETCD_PEER_CA_KEY_PATH: the full resource name ofthe new key version from the output of the preceding step.

  9. Before you continue, propagate the root CA changes to the cluster trustbundle by following the instructions in theRestart the control plane and nodes section.

Replace the original root CAs with the new root CAs

After you restart the control plane and nodes, follow these steps:

  1. Enable the new cluster root CA:

    gcloudprivatecarootsenableCLUSTER_ROOT_CA_NAME\--pool=CLUSTER_CA_POOL_NAME\--location=CONTROL_PLANE_LOCATION\--project=KEY_PROJECT_ID

  2. Enable the new aggregation root CA:

    gcloudprivatecarootsenableAGGREGATION_ROOT_CA_NAME\--pool=AGGREGATION_CA_POOL_NAME\--location=CONTROL_PLANE_LOCATION\--project=KEY_PROJECT_ID

  3. Enable the new etcd API root CA:

    gcloudprivatecarootsenableETCD_API_ROOT_CA_NAME\--pool=ETCD_API_CA_POOL_NAME\--location=CONTROL_PLANE_LOCATION\--project=KEY_PROJECT_ID

  4. Enable the new etcd peer root CA:

    gcloudprivatecarootsenableETCD_PEER_ROOT_CA_NAME\--pool=ETCD_PEER_CA_POOL_NAME\--location=CONTROL_PLANE_LOCATION\--project=KEY_PROJECT_ID

  5. Disable the original cluster root CA:

    gcloudprivatecarootsdisableORIGINAL_CLUSTER_ROOT_CA\--pool=CLUSTER_CA_POOL_NAME\--location=CONTROL_PLANE_LOCATION\--project=KEY_PROJECT_ID

    ReplaceORIGINAL_CLUSTER_ROOT_CA with the name ofthe original cluster root CA that you're rotating.

  6. Disable the original aggregation root CA:

    gcloudprivatecarootsdisableORIGINAL_AGGREGATION_ROOT_CA\--pool=AGGREGATION_CA_POOL_NAME\--location=CONTROL_PLANE_LOCATION\--project=KEY_PROJECT_ID

    ReplaceORIGINAL_AGGREGATION_ROOT_CA with the nameof the original aggregation root CA that you're rotating.

  7. Disable the original etcd API root CA:

    gcloudprivatecarootsdisableORIGINAL_ETCD_API_ROOT_CA\--pool=ETCD_API_CA_POOL_NAME\--location=CONTROL_PLANE_LOCATION\--project=KEY_PROJECT_ID

    ReplaceORIGINAL_ETCD_API_ROOT_CA with the name ofthe original etcd API root CA that you're rotating.

  8. Disable the original etcd peer root CA:

    gcloudprivatecarootsdisableORIGINAL_ETCD_PEER_ROOT_CA\--pool=ETCD_PEER_CA_POOL_NAME\--location=CONTROL_PLANE_LOCATION\--project=KEY_PROJECT_ID

    ReplaceORIGINAL_ETCD_PEER_ROOT_CA with the name ofthe original etcd peer root CA that you're rotating.

  9. At this point, the new root CAs issue all new certificates in the cluster.To issue new certificates for thekubelet on each node,Restart the control plane and nodes. This step isrequired becausekubelet certificates have long lifetimes.

After multiple days where the cluster remains in a healthy state, you candelete the original root CAs, as described in the next section.

Delete the original root CAs

This section shows you how to delete your original root CAs. Before you followthese steps, verify the following:

  • The cluster remained in a healthy state for multiple days after you replacedthe original root CAs with the new root CAs.
  • You replaced all of the certificates issued by the original root CAs with newcertificates.

To delete the original root CAs, use thegcloud privateca roots deletecommand, as described in the following steps. In these commands, the--ignore-active-certificates flag deletes the CA after a grace period, even ifthe CA has un-revoked or un-expired certificates.

  1. Delete the original cluster root CA:

    gcloudprivatecarootsdeleteORIGINAL_CLUSTER_ROOT_CA\--pool=CLUSTER_CA_POOL_NAME\--location=CONTROL_PLANE_LOCATION\--project=KEY_PROJECT_ID\--ignore-active-certificates

    ReplaceORIGINAL_CLUSTER_ROOT_CA with the name ofthe original cluster root CA that you're rotating.

  2. Delete the original aggregation root CA:

    gcloudprivatecarootsdeleteORIGINAL_AGGREGATION_ROOT_CA\--pool=AGGREGATION_CA_POOL_NAME\--location=CONTROL_PLANE_LOCATION\--project=KEY_PROJECT_ID\--ignore-active-certificates

    ReplaceORIGINAL_AGGREGATION_ROOT_CA with the nameof the original aggregation root CA that you're rotating.

  3. Delete the original etcd API root CA:

    gcloudprivatecarootsdeleteORIGINAL_ETCD_API_ROOT_CA\--pool=ETCD_API_CA_POOL_NAME\--location=CONTROL_PLANE_LOCATION\--project=KEY_PROJECT_ID\--ignore-active-certificates

    ReplaceORIGINAL_ETCD_API_ROOT_CA with the name ofthe original etcd API root CA that you're rotating.

  4. Delete the original etcd peer root CA:

    gcloudprivatecarootsdeleteORIGINAL_ETCD_PEER_ROOT_CA\--pool=ETCD_PEER_CA_POOL_NAME\--location=CONTROL_PLANE_LOCATION\--project=KEY_PROJECT_ID\--ignore-active-certificates

    ReplaceORIGINAL_ETCD_PEER_ROOT_CA with the name ofthe original etcd peer root CA that you're rotating.

  5. Optional: Propagate the changes to root CAs to the cluster trust bundle.For instructions, see theRestart the control plane and nodes section. If youskip this step, the original root CAs are removed during the next controlplane and node version upgrade.

Success: You rotated all of the root CAs in your control plane. All of the CAcertificates in your cluster are issued by your new root CAs. The original rootCAs no longer exist in the cluster trust bundle.

Restart the control plane and nodes

When you make changes to the root CA configuration of your cluster, such asenabling root CAs, disabling root CAs, or revoking certificates, you mustpropagate those changes to the cluster's trust bundle. To propagate changes totrust bundle for the cluster, you restart the control plane and (in somescenarios) the nodes.

  1. Upgrade the cluster control plane to the same version that it alreadyuses.

    Caution: In zonal clusters, the control plane becomes unavailable until theupgrade operation completes. In Autopilot clusters and regionalStandard clusters, the control plane remains available.

    1. Find the GKE version that control plane already uses:

      gcloudcontainerclustersdescribeCLUSTER_NAME\--location=CONTROL_PLANE_LOCATION\--project=CLUSTER_PROJECT_ID\--format="value(currentMasterVersion)"

      Replace the following:

      • CLUSTER_NAME: the name of your GKEcluster.
      • CLUSTER_VERSION: the GKE versionthat the cluster already runs.
      • CLUSTER_PROJECT_ID: the project ID of yourcluster project.

    2. Upgrade the control plane:

      gcloudcontainerclustersupgradeCLUSTER_NAME\--master\--location=CONTROL_PLANE_LOCATION\--cluster-version=CLUSTER_VERSION\--project=CLUSTER_PROJECT_ID

  2. If you manually generate certificates by using KubernetesCertificateSigningRequests, re-issue all of those certificates and providethe new certificates to API clients.

  3. For cluster root CA rotation only, trigger node recreation by upgradingeach of your node pools to the same version that they already use.

    Caution: GKE immediately begins recreating the nodes for thischange using the node upgrade strategy, regardless of active maintenancepolicies. GKE depends onresource availabilityfor the change. Disabling node auto-upgradesdoesn't prevent this change.Prepare the workloads that run on your nodes for disruption before youinitiate this change.

    1. Find the GKE version that the node pool uses:

      gcloudcontainernode-poolsdescribeNODE_POOL_NAME\--cluster=CLUSTER_NAME\--location=CONTROL_PLANE_LOCATION\--project=CLUSTER_PROJECT_ID\--format="value(version)"

      ReplaceNODE_POOL_NAME with the name of the nodepool to upgrade.

    2. Upgrade the node pool:

      gcloudcontainerclustersupgradeCLUSTER_NAME\--node-pool=NODE_POOL_NAME\--location=CONTROL_PLANE_LOCATION\--cluster-version=CLUSTER_VERSION\--project=CLUSTER_PROJECT_ID

If you performed the steps in this section during a CA rotation, proceed to thenext phase of the rotation, which is one of the following sections on this page:

What's next

Except as otherwise noted, the content of this page is licensed under theCreative Commons Attribution 4.0 License, and code samples are licensed under theApache 2.0 License. For details, see theGoogle Developers Site Policies. Java is a registered trademark of Oracle and/or its affiliates.

Last updated 2026-02-19 UTC.