Rotate your cluster credentials

This page explains how to rotate your GKE cluster credentials.Planning for and regularly rotating your cluster credentials is crucial forkeeping your clusters in a healthy state. On this page, you learn how to performcredential rotations. You also learn about best practices for planning regularrotations.

This page is for Security specialists who are responsible for the lifecycle ofcredentials on GKE clusters. To learn more aboutcommon roles and example tasks that we reference in Google Cloud content, seeCommon GKE user roles and tasks.

About credential rotations in GKE

The cluster root Certificate Authority (CA) has a limited lifetime. When the CAexpires, any credentials that were signed by the CA are no longer valid,including the cluster client certificate (from theMasterAuth API field), thekey and certificate for the API server, and the kubelet client certificates.Your cluster credential lifetime depends on when you created the cluster or whenyou last rotated your credentials. For details,check the credentiallifetime.

You can perform acredential rotation to revoke and issue new credentialsfor your cluster. This operation rotates the cluster CA private key and requiresre-creation of nodes to use new credentials. You muststart andfinish a credential rotationfor your clusterbefore your current credentials expire. In addition torotating credentials, credential rotation also performs anIP rotation.

When to perform a credential rotation

Key Point: Plan for and perform credential rotations well in advance of yourcluster root CA expiring. Your cluster enters theDEGRADED state seven daysbefore the current credentials expire. This is your last opportunity to rotateyour credentials before your cluster and workloads become unrecoverable.

You should perform credential rotations regularly and in advance of your currentcredential expiry date. Credential rotations require node re-creation to use thenew credentials, which might be disruptive to running workloads. Planmaintenance periods and perform the rotations duringmaintenancewindowsto avoid unexpected workload downtime or unresponsive API clients outside thecluster.

To learn more about how maintenance availability affects cluster credentialrotation, and what type of disruption your cluster experiences during the stepsof a rotation, see the row for credential rotation in the table ofmanualchanges that recreate the nodes using a node upgrade strategy and respectingmaintenancepolicies.GKE depends onresourceavailability for updatingthe nodes. To learn more about node updates, seePlanning for node updatedisruptions.

Cluster credential lifetime

Cluster credential lifetime typically depends on when the cluster wascreated or when credentials were most recently rotated:

  • Clusters created prior to approximately October 2021 have a 5 year CAlifetime.
  • Clusters created after approximately October 2021 have a 30 year CAlifetime.
  • Clusters rotated after approximately January 2022 have a 30 year CAlifetime.

Find clusters with expiring or expired credentials

If your cluster's credentials will expire in the next 180 days, or yourcluster's credentials have already expired, GKE delivers guidancewith an insight and recommendation to explain that you must perform a credentialrotation for this cluster. This guidance includes the date of the expiration ofthe credentials. You can view this guidance in the Google Cloud console. Or,you canview thisguidancewith the gcloud CLI, or the Recommender API, specifying theCLUSTER_CA_EXPIRATION subtype.

If you receive an insight and recommendation for a cluster, you mustperform acredential rotation, or GKEautomatically starts a credential rotation within 30 days of the current CAexpiry date, as explained in the next section. Once thecredential rotation has completed it can take up to36 hours for the insight and recommendation to resolve.

GKE automation policy to prevent cluster outages

To prevent your cluster from entering an unrecoverable state if your currentcredentials expire, GKE automatically starts a credentialrotation within 30 days of your current CA expiry date. For example, yourcluster CA expires on January 6, 2024 and you don't rotate your credentials byDecember 5, 2023. GKE starts an automatic rotation on or afterDecember 7, 2023, and attempts to complete this rotation seven days after theoperation starts. This automatic rotation is a last-resort attempt to prevent acluster outage, and has the following considerations:

  • Automatic rotations generally respect maintenance windows or maintenanceexclusions, however GKE reserves the right to perform stepswithin 30 days of expiration to rotate the credentials, regardless ofmaintenance availability. Within 30 days, GKE ignoresmaintenance availability for the first step, which isstarting therotation.
  • If maintenance availability prevents GKE from completing therotation initially, GKE continues to attempt to complete therotation until the date that the credentials expire, after which the clusterbecomes unrecoverable.
  • When the credential rotation completes, the expiring credentials arerevoked. Kubernetes API clients outside the cluster, like kubectl in localenvironments, won't work until you configure the clients to use the newcredentials.
  • Node pool re-creations during the rotation might cause disruptions torunning workloads.
Warning: GKE-initiated automatic rotations are alast-resortoutage prevention measure. Don't rely only on these automatic rotations;they're a preventative emergency measure to avoid complete outages and can beblocked by maintenance availability and related constraints. To learn more aboutthese constraints, seeMaintenance window preventing node updatecompletion.

Before you begin

Before you start, make sure that you have performed the following tasks:

  • Enable the Google Kubernetes Engine API.
    • Enable Google Kubernetes Engine API
  • If you want to use the Google Cloud CLI for this task,install and theninitialize the gcloud CLI. If you previously installed the gcloud CLI, get the latest version by running thegcloud components update command. Earlier gcloud CLI versions might not support running the commands in this document.Note: For existing gcloud CLI installations, make sure to set thecompute/regionproperty. If you use primarily zonal clusters, set thecompute/zone instead. By setting a default location, you can avoid errors in the gcloud CLI like the following:One of [--zone, --region] must be supplied: Please specify location. You might need to specify the location in certain commands if the location of your cluster differs from the default that you set.

Check credential lifetime

We recommend that you check your credential lifetime before and after you performa credential rotation so that you know the validity of your cluster root CA.

To check the credential lifetime for a single cluster, run the following command:

gcloudcontainerclustersdescribeCLUSTER_NAME\--locationLOCATION\--format"value(masterAuth.clusterCaCertificate)"\|base64--decode\|opensslx509-noout-dates

The output is similar to the following:

notBefore=Mar 17 16:45:34 2023 GMTnotAfter=Mar  9 17:45:34 2053 GMT

If you run this command after starting a credential rotation, the outputis the lifetime of your original certificate. This certificate remains validuntil youcomplete the rotation. After you completethe rotation, the output is the lifetime of your new certificate.

To check the credential lifetime for all clusters in a project, run the following command:

gcloudcontainerclusterslist--projectPROJECT_ID\--format="value(name,masterAuth.clusterCaCertificate)"|\whileread-rclusterca;do\expiry_date=$(echo-e"$ca"|base64--decode|opensslx509-noout-enddate|awk-F'=''{print $2}');\printf"%-40s  | %s\n""$cluster""$expiry_date";\done|\column-t|\awk-F',''BEGIN{print "Cluster Name    |  Certificate Expiry Date"} {print}'

Perform a credential rotation

Any credential rotation involves the following steps:

  1. Start the rotation: the control plane starts serving on a new IP addressin addition to the original IP address. New credentials are issued toworkloads and the control plane.
  2. Recreate nodes: GKE recreates cluster nodes so that thenodes use the new IP address and credentials, respecting availability frommaintenance windows and exclusions. You can also manually recreate yournodes by performing a node version upgrade to the same GKEversion that the nodes already run.
  3. Update API clients: after starting the rotation, update any cluster APIclients, such as development machines usingkubectl, to communicate withthe control plane using the new IP address.
  4. Complete the rotation: the control plane stops serving traffic over theoriginal IP address. Old credentials are revoked, including anyexisting static credentials for Kubernetes ServiceAccounts.

When you start a credential rotation, or when GKEautomaticallystarts a rotation, GKE performs these stepsautomatically, including attempting to complete the rotation. At each step, ifcluster expiration is more than 30 days from now, GKE respectsmaintenance availability. During automatic rotations before cluster expiration,GKE reserves the right to ignore maintenance availability toprevent your cluster from becoming unrecoverable. Within 30 days,GKE ignores maintenance availability for the first step, which isstarting the rotation.

If you don't complete a credential rotation within seven days of starting it,GKE attempts to complete the rotation for you. If any nodes inyour cluster still use the previous credentials, the automatic completionoperation fails, but GKE continues to attempt completion untilthe credentials expire and the cluster becomes unrecoverable. You should plan tomanually track and complete any credential rotations that you start. To overridemaintenance availability blockers, run the commands in each of the sections thatfollow to manually trigger those phases of the rotation process. Don't rely onautomatic completion, which is a best-effort measure.

Start the rotation

To start a credential rotation, run the following command:

gcloudcontainerclustersupdateCLUSTER_NAME\--locationLOCATION\--start-credential-rotation

This command creates new credentials, issues thesecredentials to the control plane, and configures the control plane to serve ontwo IP addresses: the original IP address and a new IP address.

Caution: In zonal Standard clusters, which have a single-node controlplane, this command causes brief downtime for the cluster API server.Autopilot clusters and regional Standard clusters havereplicated control plane nodes and don't experience downtime.

Restart in-cluster components with long-lived connections

After you start an IP address or credential rotation, the cluster's CertificateAuthority (CA) is also rotated. Some in-cluster components that maintainlong-lived TLS connections, for examplemetrics-server andkonnectivity-agent, might not automatically trust the new cluster CA.

This situation can cause those connections to fail when communicating with nodes or othercluster endpoints, and you might see TLS handshake errors in the component logs,such asx509: certificate signed by unknown authority.

If you observe these errors after starting a rotation, you might need tomanually restart your Pods. A restart forces the Pod to re-initialize itsconnection and load the new cluster CA certificates. For example, to restart themetrics-server andkonnectivity-agent, run the following commands:

kubectlrolloutrestartdeploymentmetrics-server-nkube-systemkubectlrolloutrestartdeploymentkonnectivity-agent-nkube-system

Recreate nodes

Warning: Node recreation errors might occur if a rotation is attempted afterthe cluster's CA has already expired. This error is indicated by logs thatcontain the following message:x509: certificate has expired or is not yet valid. To try and recover fromthis error,complete the rotation, create new nodepools, migrate your workloads, and then delete the old node pools. However,because clusters with expired CAs are usually unrecoverable, this effort mightfail and you might have to create a new cluster.

After reconfiguring the API server to serve on a new IP address,GKE automatically updates your nodes to use the new IP addressand credentials if there is maintenance availability. GKEupgrades all of your nodes to the same GKE version that the nodesalready run, which recreates the nodes. For more information, refer toNodepoolupgrades.

By default, GKE automatically completes credential rotationsseven days after you start the operation. If an active maintenance window orexclusion in your cluster prevents GKE from recreating some nodesduring this seven day period, the credential rotation initially fails tocomplete. However, GKE continues to try to recreate the nodes andcomplete the rotation until maintenance availability lets GKEproceed. During major events like Google Cloud Next, GKEmight also pause automatic node recreations so that you don't experiencedisruptions.

If you use maintenance exclusions or maintenance windows that could result ina failed rotation, force GKE to re-create your nodes by doing oneof the following steps:

  • In Autopilot clusters, manually upgrade your control plane:

    gcloudcontainerclustersupgradeCLUSTER_NAME\--location=LOCATION\--cluster-version=VERSION

    ReplaceVERSION with thesame GKEversion that the cluster already uses.

  • In Standard clusters,manually upgrade each node pool.

For more information, seemanual changes that respect GKE maintenance policies.

Note: When GKE recreates nodes during a credential rotation,those nodes have certificates from the previous certificate authority (CA)and the new CA. This is intended behavior to reduce the risk of disruptionsduring the rotation. After the credential rotation is completed, all new andexisting nodes only have certificates from the new CA.

Check the progress of node pool recreation

  1. To monitor the rotation operation, run the following command:

    gcloudcontaineroperationslist\--filter="operationType=UPGRADE_NODES AND status=RUNNING"\--format="value(name)"

    This command returns theoperation ID of the node upgrade operation.

  2. To poll the operation, pass the operation ID to the following command:

    gcloudcontaineroperationswaitOPERATION_ID

Node pools are recreated one-by-one, and each has its own operation. If you havemultiple node pools, use these instructions to poll each operation.

Update API clients

After starting the credential rotation, you must update all API clients outsidethe cluster (such askubectl on developer machines) to use the new credentialsand point to the new IP address of the control plane.

To update your API clients, run the following command for each client:

gcloudcontainerclustersget-credentialsCLUSTER_NAME\--locationLOCATION
Warning: If you are using the Kubernetes Certificates API, you also mustissue new certificates.

Update Kubernetes ServiceAccount credentials

If you use static credentials for ServiceAccounts in your cluster,switch toshort-lived credentials.Completing the rotation invalidates existing ServiceAccountcredentials. If you don't want to use short-lived credentials,ensure that you recreate your static credentials for all ServiceAccountsin the cluster before you complete the rotation.

To find static ServiceAccount credentials that exist in your cluster, run the following command:

kubectlgetsecrets--all-namespaces--field-selectortype=kubernetes.io/service-account-token

If the output of this command isNo resources found, your cluster has nostatic ServiceAccount credentials.

Note: If you store static credentials in secret management stores external toGKE, you must also check for credentials in those tools.

Update hardcoded IP addresses and firewall rules

If you hardcoded the IP address of the control plane in your environment, or ifyou have firewall rules that target the IP address of the control plane, updatethe addresses to the new IP address. If you complete the rotation withoutupdating IP addresses in applications and in firewall rules, those resourcesmight experience disruptions when GKE stops serving on theprevious control plane IP address.

Complete the rotation

After updating API clients outside the cluster, complete the rotation toconfigure the control plane to serve only with the new credentials and the newIP address:

gcloudcontainerclustersupdateCLUSTER_NAME\--location=LOCATION\--complete-credential-rotation
Caution: In zonal Standard clusters, which have a single-node controlplane, this command causes brief downtime for the cluster API server.Autopilot clusters and regional Standard clusters havereplicated control plane nodes and don't experience downtime.

If the credential rotation fails to complete and returns an error messagesimilar to the following, refer toError 400: Node pool requires recreation:

ERROR: (gcloud.container.clusters.update) ResponseError: code=400, message=Node pool "test-pool-1" requires recreation.

GKE respects maintenance availability when automaticallycompleting the rotation, however GKE might ignore thisavailability within 30 days of expiration to prevent the cluster from becomingunrecoverable. If rotation completion initially fails, and the rotation startedat least seven days ago, GKE attempts to complete the rotationuntil the date that the credentials expire, after which the cluster becomesunrecoverable.

What's next

Except as otherwise noted, the content of this page is licensed under theCreative Commons Attribution 4.0 License, and code samples are licensed under theApache 2.0 License. For details, see theGoogle Developers Site Policies. Java is a registered trademark of Oracle and/or its affiliates.

Last updated 2025-12-15 UTC.