This page describes how Google Kubernetes Engine (GKE) publishes clusternotifications to Cloud Logging by default, and, optionally, toPub/Sub. These notifications have informationabout events relevant to your cluster configuration, such as available orongoing upgrades, security bulletins, and end of support dates.

Overview

When certain events occur that are relevant to your GKE clusters,such as important available upgrades or security bulletins, GKEsends these notifications toCloud Logging. To findthese logs in Cloud Logging, seeViewing cluster notifications inCloud Logging.

GKE also publishes notifications about those events as messagesto Pub/Sub topics that you configure. You can receive thesenotifications on a Pub/Subsubscription,integrate with third-partyservices,and filter for the notification types you want to receive. For more informationabout how to set up cluster notifications with Pub/Sub, seeReceivecluster notifications throughPub/Sub.

Benefits

Cluster notifications provide the following benefits:

• You are notified whensecurity bulletinsthat are specific to your clusters are issued, which provides you withaccurate risk and impact information.
• You are notified when there is a new GKE version available foryour cluster, allowing you to better plan for testing and qualifications, andto help ensure a smooth and predictable upgrade process. Previously, you hadto check theGKE releasenotes or theGKE APIto discover when a new GKE version was released.
• You are notified when either GKE or a user initiatesclusterupgrades, and when theupgrade operation finishes, providing you with more visibility into thebackground operations of your cluster.
• You are notified when your cluster is running a GKE minorversion that is at or near the end of support.

• You can choose whether to use Cloud Logging or Pub/Sub:

  • Cluster notifications are sent to Cloud Logging by default. You canuse all the capabilities of Cloud Logging, includingquerying andviewing logs, andconfiguring log-basedalerting policies.
  • Pub/Sub is highly extensible, giving youflexibility in how you process incoming notifications.For example, you could integrate withSlack to forwardnotifications to a Slack channel, or initiateCloud Run functions to run custom processes.When custom processes are required (for example,orchestrating a staging to production workflow to test and certify anupgrade), you can use the notification to auto-trigger these workflows.

Types of upgrade notifications

GKE sends the following cluster notification types:

• SecurityBulletinEvent
• UpgradeAvailableEvent
• UpgradeEvent

• UpgradeInfoEvent, which includes the followingtypes of events:

  • Upgrade operation is complete
  • Minor version at or near the end of support
  • New patch versions change to new Container-Optimized OS milestone during extended support

If youview cluster notifications in Cloud Logging, youcan use the capabilities of Cloud Loggingto filter the logs. If you use Pub/Sub, you canfilter thenotifications you receive so that you're notified only for relevantevents.

SecurityBulletinEvent

When GKE issues a security bulletin that directly correlates toyour cluster configuration or version, GKE sends aSecurityBulletinEventnotification, providing you with information about the vulnerability, the impact,and, if applicable, actions you can take.

When a new version becomes available on arelease channel,GKE sends anUpgradeAvailableEventnotification to clusters on that release channel to inform the clusters that anew version is now available. This notification provides one week of advancenotice for patch versions and at least 2-4 weeks for minor versions (dependingon the channel). For more information, refer toWhat versions are available ina channel.

If you use Windows Server node pools, Windows version information is sent aspart of theUpgradeAvailableEvent notification.

UpgradeEvent

When you or GKE initiates an upgrade, GKEsends anUpgradeEventnotification, telling you that an upgrade has begun. Ideally, you should use theUpgradeAvailableEvent notification type to be aware of the upcoming upgrade sothat you can either upgrade in advance or take necessary measures to prepare,such as setting upmaintenance windows.

TheUpgradeEvent notification is sent at the start of the upgrade operation.The operation ID is passed in the message.

UpgradeInfoEvent

GKE sends anUpgradeInfoEventnotification for different types of events, which are outlined in the nextsections.

For more information about filtering for a specific type ofUpgradeInfoEvent,seeFilterUpgradeInfoEvent cluster notifications.

Upgrade operation is complete

When GKE finishes the operation to automatically or manuallyupgrade a cluster's control plane or nodes, GKE sends anotification to inform you that the operation is complete. The operationcompletes with one of the following states:

• SUCCEEDED: GKE successfully upgraded the control plane ornodes.
• FAILED: GKE failed to upgrade the control plane or nodes.
• CANCELED: GKE canceled the upgrade operation for technical orbusiness reasons, or youcanceled the upgradeoperation.

Use the notification to confirm the success of an upgrade operation.

Minor version at or near the end of support

When your cluster runs a GKE minor version that is near theendof standard support orend of extendedsupport, or has reachedeither of those milestones, GKE sends notifications that you mustupgrade your cluster control plane or nodes to the next supported minor version.Running a supported minor version ensures that you continue receiving securitypatches, bug fixes, and support. GKE sends one notification 30days before the end of support, and one notification at the end of support, ifyour cluster still runs the minor version.

GKE sends cluster-level notifications, although multiplecomponents of your cluster might be affected, and your cluster can run differentminor versions at the same time. If the minor version is reaching the end ofstandard support and you need time to prepare for an upgrade to a supportedversion, you canswitch to the Extended release channel to get long-termsupport. Otherwise,GKE schedulesautomatic upgrades at the end ofsupport. Thesenotifications help ensure that you're prepared for the enforcement of these endof support policies.

A notification includes the following details:

• The cluster that is affected.
• The current version that is at or near the end of support.
• The end of support date.

For more details about the timeline of support for GKE minorversions, see theGKE minor versionlifecycle.

New patch versions change to new Container-Optimized OS milestone during extended support

When your cluster is enrolled in the Extended channel during the extendedsupport period, and the Container-Optimized OS milestone that's used bythe GKE minor version reaches the end of support before the minorversion, GKE sends out a cluster notification. GKEsends this notification when the first patch version to use the new milestonebecomes available in the Extended channel.

This notification includes the following details:

• The cluster that is affected.
• The patch version which uses the new milestone.
• The existing and new milestones.
• How GKE pauses automatic patch upgrades for the nodes.

The patch version which uses the new milestone eventually becomes the patchauto-upgrade target for the cluster, and node auto-upgrades are paused. Clusteradministrators must decide which of the following next steps to take:

• Manually upgrade the cluster nodes to the next patch version which uses thenext Container-Optimized OS milestone.
• Manually upgrade the cluster to the next minor version.
• Forgo patch upgrades until GKE upgrades the cluster to the nextminor version towards the end of extended support.

To learn more, seeThe Container-Optimized OS milestone reaches the endof support before the minorversion end of extended support.

Viewing cluster notifications in Cloud Logging

To view logs for GKE clusters, seeAccessing yourlogs.

To opt out of storing these logs, you can configure anexclusionfilter.

View logs in Cloud Logging with the following filter to see all types ofcluster notifications:

logName=projects/PROJECT_ID/logs/container.googleapis.com%2Fnotifications

ReplacePROJECT_ID with your Google Cloud project ID.

View logs with the following filter to see a specific cluster notification type,such asUpgradeEvent:

jsonPayload.@type=type.googleapis.com/google.container.v1beta1.NOTIFICATION_TYPE

ReplaceNOTIFICATION_TYPE with theclusternotification type for whichever logs you want to see.

FilterUpgradeInfoEvent cluster notifications

View logs with the following filter to see a specificUpgradeInfoEvent, suchas the notification for where an upgrade operation is complete:

jsonPayload.@type=type.googleapis.com/google.container.v1beta1.UpgradeInfoEventjsonPayload.eventType=EVENT_TYPE

ReplaceEVENT_TYPE with one of the following:

• Upgrade operation iscomplete:UPGRADE_LIFECYCLE
• Minor version at or near the end ofsupport:END_OF_SUPPORT
• New patch versions change to new Container-Optimized OS milestoneduring extended support:COS_MILESTONE_VERSION_UPDATE

Filtering notifications to Pub/Sub

You can filter cluster notifications to ensure that you receive only thenotifications that you want in Pub/Sub. You can apply filtering fornotifications to Pub/Sub in one of the following ways:

• Filtering notifications in GKE.
• Filtering notifications in Pub/Sub.

To view and filter notifications in Cloud Logging, seeViewing clusternotifications in Cloud Logging.

For instructions on filtering notifications in GKE, seeReceive cluster notifications throughPub/Sub.

Filtering notifications in GKE doesn't affect which logs appearin Cloud Logging.

Filtering notifications in Pub/Sub

Pub/Sub supports filtering messages in your subscription using afiltering syntax. When you use this method, GKE delivers allnotification types to your Pub/Sub topic. Pub/Sub filtersmessages based on your subscription configuration and delivers the messages youwant to receive.

For example, you could filter forUpgradeEvent andUpgradeAvailableEventnotifications using the following syntax in your subscription:

attributes.type_url = "type.googleapis.com/google.container.v1beta1.UpgradeEvent" OR "type.googleapis.com/google.container.v1beta1.UpgradeAvailableEvent"

You are stillcharged for undelivered messagesfiltered by your subscription. You alsocannot modify the filters afteryou've configured the subscription. However, the filtering syntax is moreextensible than filtering in GKE.

To learn more about filtering your Pub/Sub subscription, seeFiltering messages.

Consuming Pub/Sub messages

Pub/Sub messages contain two fields:data (string) andattributes(string-to-string map).

For GKE notifications, thedata field contains human-readableinformation. Theattributes field has generic notification information likethe notification type, your project ID, cluster name, and cluster location.Theattributes.payload field is a JSON-parsable string that contains specificnotification information, such as the details of a security bulletin.

Notifications always contain the following attributes:

{"resourceType":"MASTER","operation":"operation-1595889094437-87b7254a","operationStartTime":"2020-07-27T22:31:34.437652293Z","currentVersion":"1.15.12-gke.2","targetVersion":"1.15.12-gke.9"}

GKE will always sendbeta notification types. However, you canparse the payload to display the corresponding GA notification type if it isavailable.

Sample cluster notification messages

In addition to the text in thedata field, each message thatGKE sends to Cloud Logging or Pub/Sub has specificvalues in theattributes.type_url andattributes.payload fields. Thefollowing tables show you examples of the information you might receive for eachnotification type:

SecurityBulletinEvent

The output is similar to the following for aSecurityBulletinEventmessage:

{"resourceTypeAffected":"RESOURCE_TYPE_CONTROLPLANE","bulletinId":"GCP-2021-001","cveIds":["CVE-2021-3156"],"severity":"Medium","briefDescription":"A vulnerability was recently discovered in the Linux utility sudo, described in CVE-2021-3156, that may allow an attacker with unprivileged local shell access on a system with sudo installed to escalate their privileges to root on the system.","affectedSupportedMinors":["1.18","1.19"],"patchedVersions":["1.18.9-gke.1900","1.19.9-gke.1900"],"suggestedUpgradeTarget":"1.19.9-gke.1900","bulletinUri":"https://cloud.google.com/anthos/clusters/docs/security-bulletins#gcp-2021-001"}

UpgradeAvailableEvent

The output is similar to the following for anUpgradeAvailableEventmessage:

{"version":"1.17.15-gke.800","resourceType":"MASTER","releaseChannel":{"channel":"RAPID"},"windowsVersions":[{"imageType":"WINDOWS_SAC","osVersion":"10.0.18363.1198","supportEndDate":{"day":10,"month":5,"year":2022}},{"imageType":"WINDOWS_LTSC","osVersion":"10.0.17763.1577","supportEndDate":{"day":9,"month":1,"year":2024}}]}

UpgradeEvent

The output is similar to the following for anUpgradeEventmessage:

{"resourceType":"MASTER","operation":"operation-1595889094437-87b7254a","operationStartTime":"2020-07-27T22:31:34.437652293Z","currentVersion":"1.15.12-gke.2","targetVersion":"1.15.12-gke.9"}

UpgradeInfoEvent

The output is similar to the following for anUpgradeInfoEventmessage for when anupgrade operationcompletes, such as this example for anode pool upgrade:

{"currentVersion":"1.31.1-gke.1846000","endTime":"2024-11-06T17:12:54.111640650Z","operation":"operation-1730912205658-de2f88a8-6290-4718-b2c1-fb19611060b8","resource":"projects/PROJECT_ID/locations/CLUSTER_LOCATION/clusters/CLUSTER_NAME/nodePools/NODE_POOL_NAME","resourceType":"NODE_POOL""startTime":"2024-11-06T16:56:45.658321844Z","state":"SUCCEEDED","targetVersion":"1.31.1-gke.2105000"}

This output is distinct from when the messages are for aminor version at ornear the end of support, or whennew patchversions change to new Container-Optimized OS milestone during extendedsupport.

What's next

• Learn how toreceive cluster notifications throughPub/Sub.
• Learn how toconfigure cluster notifications for third-party services.