Rotating user cluster certificate authorities

Google Distributed Cloud uses certificates and private keys toauthenticate and encrypt connections between system components in user clusters.The admin cluster creates a new set of certificate authorities (CAs) for eachuser cluster, and uses CA certificates to issue additional leafcertificates for system components. The admin cluster manages distribution ofthe public CA certificates and leaf certificate key pairs to system componentsto establish their secure communication.

The user cluster CA rotation feature lets you trigger a rotation of thecore system certificates in a user cluster. During arotation, the admin cluster replaces the core system CAs for the user clusterwith newly generated CAs, and distributes the new public CA certificates andleaf certificate key pairs to user cluster system components. The rotationhappens incrementally, so that system components can continue to communicateduring the rotation. Note, however, that workloads and nodes are restartedduring the rotation.

There are three system CAs managed by the admin cluster for each user cluster:

  • The etcd CA secures communication from the API server to the etcd replicas andalso traffic between etcd replicas. This CA is self-signed.
  • The cluster CA secures communication between the API server and all internalKubernetes API clients (kubelets, controllers, schedulers).This CA is self-signed.
  • The front-proxy CA secures communication withaggregated APIs.This CA is self-signed.

Also, you might be using anorg CA tosign the certificate configured by theauthentication.sni option.This CA and the SNI certificate are used to serve the Kubernetes API to clientsoutside the cluster. You manage this CA and manually generate the SNIcertificate. Neither this CA nor the SNI certificate is affected by the usercluster CA rotation feature.

Limitations

Warning: A CA rotation revokes the old CAs at the end of the operation. Thisinvalidates kubeconfig files and authentication configuration files that wereissued and distributed based on the old CA. These credentials stop workingafter a CA rotation. The following instructions include details on how todownload a new user cluster kubeconfig from the admin cluster and regenerateauthentication configuration files after the CA rotation completes.

  • Note the following limitation with advanced clusters:

  • CA certificate rotation is limited to the etcd, cluster, andfront-proxy CAs mentioned previously.

  • CA certificate rotation is limited to certificates issued automatically byGoogle Distributed Cloud. It doesn't update certificates issued manually by anadministrator, even if those certificates are signed by the system CAs.

  • A CA rotation restarts the API server, other control-plane processes, andeach node in the cluster multiple times. Each stage of a CA rotationprogresses similarly to a cluster upgrade. While the user cluster does remainoperational during a CA rotation, you should expect that workloads to berestarted and rescheduled. You should also expect brief periods ofcontrol-plane downtime if your user cluster does not have ahigh-availability control plane.

  • You must update the user cluster kubeconfig file and authenticationconfiguration files after a CA rotation. This is because the old clustercertificate is revoked, and the credentials in the kubeconfig file nolonger work.

  • After a CA rotation is started, it cannot be paused or rolled-back.

  • A CA rotation might take considerable time to complete, depending on the size ofthe user cluster.

Perform a CA rotation

  1. Start the rotation:

    gkectl update credentials certificate-authorities rotate \    --configUSER_CLUSTER_CONFIG \    --kubeconfigADMIN_CLUSTER_KUBECONFIG

    Replace the following:

    • USER_CLUSTER_CONFIG: the path of the user clusterconfiguration file

    • ADMIN_CLUSTER_KUBECONFIG: the path of the admin clusterkubeconfig file

The behavior of the command differs depending on whether advanced clusteris enabled:

Not enabled

If advanced cluster isn't enabled on the cluster, the command is asynchronousand starts the CA rotation and then exits. You don't need to watch theoutput of the command for the entire duration of the CA rotation. Instead, youcan periodically check on the progress by running thegkectl update credentials certificate-authorities status command.

If the CA rotation starts successfully, you see a message similar tofollowing:

successfully started the CA rotation with CAVersion 2, use gkectl update credentials certificate-authorities status command to view the current state of CA rotation

If a CA rotation is already in progress, you see an error message similarto the following:

Exit with error:admission webhook "vonpremusercluster.onprem.cluster.gke.io" denied the request: requests must not modify CAVersion when cluster is not ready: ready condition is not true: ClusterCreateOrUpdate: Creating or updating user cluster control plane workloads

To view the status of the rotation:

gkectl update credentials certificate-authorities status \    --configUSER_CLUSTER_CONFIG \    --kubeconfigADMIN_CLUSTER_KUBECONFIG

The preceding command reports theCAVersion, which is an integer thesystem automatically increments to differentiate the CAs used before andafter a rotation. The command also reports a status (True orFalse)that indicates whether the CA rotation is complete, and a message describingwhichCAVersion is currently in use by each component of the system.

If the CA rotation has already completed, you see a message similar tothis:

State of CARotation with CAVersion 2 is -status: True,reason: CARotationCompleted,message: Control plane has CA bundle [2], certs from CA 2, CA 2 is CSR signer. Data plane has CA bundle [2], CA 2 was CSR signer at last restart.

If the CA rotation is still in progress, you see a message similar to this:

State of CARotation with CAVersion 2 is -status: False,reason: CARotationProgressed,message: Control plane has CA bundle [1 2], certs from CA 2, CA 1 is CSR signer. Data plane has CA bundle [1 2], CA 1 was CSR signer at last restart.

Update user cluster credentials

On clusters that don't have advanced cluster enabled, you must get a new usercluster kubeconfig file from the admin cluster. This is because the CArotation revokes the CA that the old kubeconfig file was based on.

Get a new kubeconfig file:

kubectl --kubeconfigADMIN_CLUSTER_KUBECONFIG get secret admin \  -nUSER_CLUSTER_NAME -o jsonpath='{.data.admin\.conf}' \  | base64 --decode >USER_CLUSTER_NAME-kubeconfig

Enabled

If advanced cluster is enabled, thegkectl update credentialscertificate-authorities rotate command is synchronous. The command outputsstatus messages to the admin workstation as the CA rotation progresses.

After the CA is rotated successfully, the command exits and a new kubeconfigfile is automatically generated. The command output provides the name of thenew kubeconfig file, and is similar to the following:

Beginning CA rotation with generated CA...Successfully rotated CA for user cluster "USER_CLUSTER_NAME". Thekubeconfig file "/home/ubuntu"/USER_CLUSTER_NAME-kubeconfig" hasbeen updated.

The new kubeconfig is located in the same directory as the admin clusterkubeconfig that you specified in the command. The name of the new kubeconfigisUSER_CLUSTER_NAME-kubeconfig.

Distribute the new kubeconfig file

Distribute the new kubeconfig file to everyone who uses a kubeconfig file tointeract with the cluster.

Update authentication configuration files

After the CA rotation completes, authentication configuration files must beupdated and redistributed. For more information seeManage user identity.

Control plane certificates rotation

Without rotation, both the user cluster CAs and control-plane certificatesexpire five years from the date the cluster was created. The user cluster'scontrol-plane certificates are automatically rotated within ten hours of eachuser cluster upgrade, but the CAs are not automatically rotated. This means aCA rotation must be performed at least once every five years in addition toregular version upgrades.

To prevent a user cluster from becoming unavailable, control-plane certificatesare rotated within ten hours following a user cluster upgrade. When thishappens, a message appears in the user cluster's CA rotation status.

To view the last version a user cluster has been upgraded to when control-planecertificates were rotated:

gkectl update credentials certificate-authorities status \--configUSER_CLUSTER_CONFIG \--kubeconfigADMIN_CLUSTER_KUBECONFIG

The information appears at the end of themessage field within ten hours of anupgrade. For example:

Last Leaf Certificates Rotation Version: 1.16.0-gke.0.

Troubleshooting a CA rotation

Thegkectl diagnose command supports checking the expected status of acompleted CA rotation against a user-cluster. For instructions on how to rungkectl diagnose on a user cluster, seeDiagnosing cluster issues.If you experience issues with a CA rotation, contact Google support andprovide thegkectl diagnose output.

Note: Because this command checks for the expected end state of a CA rotation,it is expected for it to report that some certificates have not yet been rotatedwhen run during a CA rotation.

Except as otherwise noted, the content of this page is licensed under theCreative Commons Attribution 4.0 License, and code samples are licensed under theApache 2.0 License. For details, see theGoogle Developers Site Policies. Java is a registered trademark of Oracle and/or its affiliates.

Last updated 2025-12-15 UTC.