This page shows you how to configure node auto-upgrades in Google Kubernetes Engine (GKE).

Node auto-upgrades help you keep thenodes in yourcluster up-to-date withthe cluster control plane version when your control plane isupdated on your behalf.When you create a new cluster or node pool with the Google Cloud console orthegcloud command, node auto-upgrade is enabled by default.

You can learn more aboutcluster and node upgrades.

Node auto-upgrades provide several benefits:

• Lower management overhead: You don't have to manually track and updateyour nodes when the control plane is upgraded on your behalf.
• Better security: Sometimes new binaries are released to fix a securityissue. With auto-upgrades, GKE automatically ensures thatsecurity updates are applied and kept up to date.
• Ease of use: Provides a simple way to keep your nodes up to date with thelatest Kubernetes features.

Node pools with auto-upgrades enabled are scheduled for upgradeswhen they meet the selection criteria (announced in therelease notes).Rollouts are phased across multiple weeks to ensure cluster and fleet stability.When the upgrade is performed, nodes are drained and re-created to match thecurrent control plane version. Modifications on the boot disk of a node VM donot persist across node re-creations. To preserve modifications across nodere-creation, use aDaemonSet.

Node auto-upgrade is not available forAlpha clusters. If you are usinga cluster with Windows Server node pools, reviewUpgrading Windows Server node poolsbefore enabling node auto-upgrade.

Before you begin

Before you start, make sure that you have performed the following tasks:

• Enable the Google Kubernetes Engine API.
Enable Google Kubernetes Engine API
• If you want to use the Google Cloud CLI for this task,install and theninitialize the gcloud CLI. If you previously installed the gcloud CLI, get the latest version by running thegcloud components update command. Earlier gcloud CLI versions might not support running the commands in this document.Note: For existing gcloud CLI installations, make sure to set thecompute/regionproperty. If you use primarily zonal clusters, set thecompute/zone instead. By setting a default location, you can avoid errors in the gcloud CLI like the following:One of [--zone, --region] must be supplied: Please specify location. You might need to specify the location in certain commands if the location of your cluster differs from the default that you set.

• Ensure that you have an existing Standardcluster. When you create a new cluster with the Google Cloud console or the Google Cloud CLI command, node auto-upgrade is enabled by default. To create a Standard cluster, seeCreate a regional cluster.

Check the state of auto-upgrade for an existing node pool

You can check whether auto-upgrade is enabled or disabled for a node pool usingthe Google Cloud console or thegcloud command.

gcloudcontainernode-poolsdescribeNODE_POOL_NAME\--clusterCLUSTER_NAME\--locationCONTROL_PLANE_LOCATION

management:  autoUpgrade: true

Enable node auto-upgrades for an existing node pool

When you create a new cluster with the Google Cloud console or thegcloud command,node auto-upgrade is enabled by default.

You can enable node auto-upgrade if it is currently disabled.

gcloudcontainernode-poolsupdateNODE_POOL_NAME\--clusterCLUSTER_NAME\--locationCONTROL_PLANE_LOCATION\--enable-autoupgrade

For more control over when nodes can be auto-upgraded, consider configuringmaintenance windows and exclusions.

Check the status of node upgrades

You can check the status of an upgrade usinggcloud container operations.

View a list of every running and completed operation in the cluster fromthe last 12 days if there's fewer than 5,000 operations, or the last 5,000operations:

gcloudcontaineroperationslist\--location=CONTROL_PLANE_LOCATION

Each operation is assigned anoperation ID and an operation type as well asstart and end times, target cluster, and status. The list appears similar tothe following example:

NAME                              TYPE                ZONE           TARGET              STATUS_MESSAGE  STATUS  START_TIME                      END_TIMEoperation-1505407677851-8039e369  CREATE_CLUSTER      us-west1-a     my-cluster                          DONE    20xx-xx-xxT16:47:57.851933021Z  20xx-xx-xxT16:50:52.898305883Zoperation-1505500805136-e7c64af4  UPGRADE_CLUSTER     us-west1-a     my-cluster                          DONE    20xx-xx-xxT18:40:05.136739989Z  20xx-xx-xxT18:41:09.321483832Zoperation-1505500913918-5802c989  DELETE_CLUSTER      us-west1-a     my-cluster                          DONE    20xx-xx-xxT18:41:53.918825764Z  20xx-xx-xxT18:43:48.639506814Z

To get more information about a specific operation, specify the operation ID asshown in the following command:

gcloudcontaineroperationsdescribeOPERATION_ID\--location=CONTROL_PLANE_LOCATION

For example:

gcloud container operations describe operation-1507325726639-981f0ed6endTime: '20xx-xx-xxT21:40:05.324124385Z'name: operation-1507325726639-981f0ed6operationType: UPGRADE_CLUSTERselfLink: https://container.googleapis.com/v1/projects/.../kubernetes-engine/docs/zones/us-central1-a/operations/operation-1507325726639-981f0ed6startTime: '20xx-xx-xxT21:35:26.639453776Z'status: DONEtargetLink: https://container.googleapis.com/v1/projects/.../kubernetes-engine/docs/zones/us-central1-a/clusters/...zone: us-central1-a

If the upgrade was cancelled or failed and is partially completed,you canresume orroll back the upgrade.

Check node pool upgrade settings

You can see details on the node upgrade strategy being used for your node poolsusing thegcloud container node-poolsdescribe command. Forblue-green upgrades, the command also returns thecurrentphaseof the upgrade.

Run the following command:

gcloudcontainernode-poolsdescribeNODE_POOL_NAME\--cluster=CLUSTER_NAME\--location=CONTROL_PLANE_LOCATION

Replace the following:

• NODE_POOL_NAME: the name of the node pool to describe.
• CLUSTER_NAME: the name of the cluster of the node poolto describe.
• CONTROL_PLANE_LOCATION: the location (region or zone)for the control plane, such asus-central1 orus-central1-a.

This command will output the current upgrade settings. The following exampleshows the output if you are using the blue-green upgrade strategy.

upgradeSettings:  blueGreenSettings:    nodePoolSoakDuration: 1800s    standardRolloutPolicy:      batchNodeCount: 1      batchSoakDuration: 10s  strategy: BLUE_GREEN

If you are using the blue-green upgrade strategy, the output also includesdetails about the blue-green upgrade settings and its current intermediate phase.The following example shows what this might look like:

updateInfo:  blueGreenInfo:    blueInstanceGroupUrls:    - https://www.googleapis.com/compute/v1/projects/{PROJECT_ID}/zones/{LOCATION}/instanceGroupManagers/{BLUE_INSTANCE_GROUP_NAME}    bluePoolDeletionStartTime: {BLUE_POOL_DELETION_TIME}    greenInstanceGroupUrls:    - https://www.googleapis.com/compute/v1/projects/{PROJECT_ID}/zones/{LOCATION}/instanceGroupManagers/{GREEN_INSTANCE_GROUP_NAME}     greenPoolVersion: {GREEN_POOL_VERSION}    phase: DRAINING_BLUE_POOL

Disable node auto-upgrades

Although not recommended, you can disable node auto-upgrade for an existing nodepool if the underlying cluster isn't enrolled in areleasechannel.

Considerations before disabling node auto-upgrades

If you disable node auto-upgrades for a node pool, GKE does notupdate the version of the nodes. Opting out of node auto-upgrades does not blockGKE from upgrading your cluster's control plane.

Disabling prevents version updates, but not all maintenance tasks

Disabling node auto-upgrades only prevents GKE from updating theversion of the nodes, but does not prevent GKE from initiatingother maintenance tasks. For example, even with node auto-upgrades disabled,triggeringIP address rotation,Cloud DNS,enabling network policy, orPSC migration on acluster recreates all nodes at the same version as the control plane, regardlessof the version selected for the node pool. To control the timing of maintenance,useMaintenance windows andexclusions.

Disabling means responsibility of control plane-nodes compatibility

If you disable node auto-upgrade for a node pool, you are responsible forensuring that the cluster's nodes run a version compatible with the cluster'scontrol plane version, adhering to theGKE version skewpolicy. Disabling nodeauto-upgrades only prevents node pools from being upgraded untilthe end ofstandard support.GKE upgrades nodes that are running an unsupported version afterthe version has reached the end of support to ensure cluster health.

For details regarding the timing of the end of support, see theEstimatedschedule for releasechannels.Nodes running unsupported versions might not be upgraded immediately uponversion end of support, and actual timing can vary at Google's discretion.

Disabling does not stop ongoing operations

Disabling node auto-upgrades does not stop or cancel any ongoingupgrades to nodes in node pools. To cancel or stop ongoing upgrades, followCancelling a node upgrade.Canceling a node upgrade can be helpful in situations where you find workloadsare failing on upgraded nodes and you want to prevent further disruption.

If the upgrade is fully completed for the entire node pool, then the upgradecannot be rolled back or cancelled. To downgrade the node pool, seeDowngradingnodepools.

Disable node auto-upgrades for an existing node pool

gcloudcontainernode-poolsupdateNODE_POOL_NAME\--clusterCLUSTER_NAME\--locationCONTROL_PLANE_LOCATION\--no-enable-autoupgrade

Create a node pool with node auto-upgrades enabled

gcloudcontainernode-poolscreateNODE_POOL_NAME\--clusterCLUSTER_NAME\--locationCONTROL_PLANE_LOCATION\--enable-autoupgrade

Receive upgrade notifications

GKE publishes upgrade notifications toPub/Sub, providing you with a channel toreceive information from GKE about your clusters.

For more information, seeReceiving cluster upgrade notifications.

Change surge upgrade parameters

To learn more about changing surge upgrade parameters, seeConfigure surge upgrades.

Exercise control during a node pool upgrade

During automatic upgrades and manually-initiated node pool upgrades, you cantake the following actions.

Cancel a node pool upgrade

You can cancel an upgrade at any time. To learn more about what happens when youcancel a surge upgrade, seeCancel a surge upgrade. To learn more about what happens when you cancel a blue-greenupgrade, seeCancel a blue-green upgrade.

1. Get the upgrade's operation ID:

   gcloudcontaineroperationslist\--location=CONTROL_PLANE_LOCATION

2. Cancel the upgrade:

   gcloudcontaineroperationscancelOPERATION_ID\--location=CONTROL_PLANE_LOCATION

Refer to thegcloud container operations canceldocumentation.

Resume a node pool upgrade

You can resume an upgrade bymanually initiating the upgradeagain, specifying the target version from the original upgrade.

If, for example, an upgrade failed, or if you paused an ongoing upgrade, youcould resume the canceled upgrade by starting the same upgrade again on the nodepool, specifying the target version from the initial upgrade operation.

To learn more about what happens when you resume an upgrade, seeResume a surge upgradeandblue-green upgrade.

To resume an upgrade, use the following command:

gcloudcontainerclustersupgradeCLUSTER_NAME\--node-pool=NODE_POOL_NAME\--location=CONTROL_PLANE_LOCATION\--cluster-versionVERSION

Replace the following:

• NODE_POOL_NAME: the name of the node pool for whichyou want to resume the node pool upgrade.
• CLUSTER_NAME: the name of the cluster of the node poolfor which you want to resume the upgrade.
• CONTROL_PLANE_LOCATION: the location (region or zone)for the control plane, such asus-central1 orus-central1-a.
• VERSION: the target version of the canceled nodepool upgrade.

For more information, refer to thegcloud container clusters upgradedocumentation.

Roll back a node pool upgrade

You can roll back a node pool to downgrade the upgraded nodes to their originalstate from before the node pool upgrade started.

Use therollback command if an in-progress upgrade wascancelled,the upgrade failed, or the upgrade is incomplete due to amaintenance windowtiming out. Alternatively, if you want to specify the version, follow theinstructions todowngradethe node pool.

To learn more about what happens when you roll back a node pool upgrade, seeRoll back a surge upgradeorRoll back a blue-green upgrade.

To roll back an upgrade, run the following command:

gcloudcontainernode-poolsrollbackNODE_POOL_NAME\--clusterCLUSTER_NAME\--location=CONTROL_PLANE_LOCATION

Replace the following:

• NODE_POOL_NAME: the name of the node pool for which toto roll back the node pool upgrade.
• CLUSTER_NAME: the name of the cluster of the node poolfor which to roll back the upgrade.
• CONTROL_PLANE_LOCATION: the location (region or zone)for the control plane, such asus-central1 orus-central1-a.

Refer to thegcloud container node-pools rollbackdocumentation.

Complete a node pool upgrade

If you are using the blue-green upgrade strategy, you can complete a nodepool upgrade during theSoak phase,skipping the rest of the soak time.

To learn how completing a node pool upgrade works, seeComplete a node pool upgrade.

To complete an upgrade when using the blue-green upgradestrategy, run the following command:

gcloudcontainernode-poolscomplete-upgradeNODE_POOL_NAME\--clusterCLUSTER_NAME\--location=CONTROL_PLANE_LOCATION

Replace the following:

• NODE_POOL_NAME: the name of the node pool for whichyou want to complete the upgrade.
• CLUSTER_NAME: the name of the cluster of the nodepool for which you want to complete the upgrade.
• CONTROL_PLANE_LOCATION: the location (region or zone)for the control plane, such asus-central1 orus-central1-a.

Refer to thegcloud container node-pools complete-upgradedocumentation.

What's next

• Learn more aboutnode pools.
• Learn more aboutcluster and node pool upgrades.
• Manually upgrade a cluster or node pool.
• Learn more aboutrelease channels.
• Troubleshoot cluster upgrades.