This document describes how Google manages security vulnerabilities and patchesfor Google Kubernetes Engine (GKE).

This page is for Security specialists who support the resolution of securityissues or vulnerabilities which need strategic assistance, such as incidents andissues escalated from support. To learn more about common roles and exampletasks that we reference in Google Cloud content, seeCommon GKE user roles and tasks.

Patching shared responsibility

Patching is a shared responsibility between Google and the customer. Differentplatforms have different shared responsibilities. See the following topics oneach platform for more details:

• GKE
• Google Distributed Cloud (on-premises)
• GKE on AWS
• Google Distributed Cloud for bare metal
• GKE on Azure

How we discover vulnerabilities

Google has made large investments in proactive security design and hardening,but even the best designed software systems can have vulnerabilities. To findthose vulnerabilities and patch them before they can be exploited, Google hasmade significant investments on multiple fronts.

For patching purposes, GKE is an Operating System (OS) layer withcontainers running on top. The operating systems, Container-Optimized OSor Ubuntu, are hardened and contain the minimum amount of softwarerequired to run containers. GKE features run as containers on topof the base images. GKE consistently works to reduce the numberof dependencies that system components have, which helps to reduce the attacksurface and improve the efficiency of vulnerability management. For example,GKE system components might use minimal base images whenpossible.

Google identifies and fixes vulnerabilities and missing patches in the followingways:

• Container-Optimized OS: Google scans images to identify potentialvulnerabilities and missing patches. The Container-Optimized OS teamreviews and resolves issues.

• Ubuntu: Canonical provides Google with OS builds that have all availablesecurity patches applied.

Google scans containers usingContainer Registry Artifact Analysisto discover vulnerabilities and missing patches in Kubernetes and Google-managedcontainers. If fixes are available, the scanner automatically begins thepatching and release process.

In addition to automated scanning, Google discovers and patches vulnerabilitiesunknown to scanners in the following ways.

Google performs its own audits, penetration testing, and vulnerability discoveryacross all platforms. For a list of platforms, see the previous sectionPatching shared responsibility.

Specialized teams inside Google andtrusted third-party security vendors conduct their own attack research. Google hasalso worked with theCNCF to provide much of the organization and technical consulting expertise for theKubernetes security audit.

Google actively engages with the security research community throughmultiple vulnerability reward programs.A dedicated Google Cloud vulnerability rewards program providessignificant bounties,including $133,337 for the best cloud vulnerability found each year. For GKE, there is a program that rewardssecurity researchers if they are able tobreak our security controls.The program covers all GKE software dependencies.

Google collaborates with other industry and open source software partners whoshare vulnerabilities, security research, and patches before the public releaseof the vulnerability. The goal of this collaboration is to patch large pieces ofInternet infrastructure before the vulnerability is announced to the public. Insome cases, Google contributes vulnerabilities found to this community. Forexample, Google's Project Zero discovered and publicized theSpectre and Meltdown vulnerabilities. The Google Cloud Security team also regularly finds andfixesvulnerabilities in the Kernel-based Virtual Machine (KVM).

Google's security collaboration happens on many levels. Sometimes it occursformally through programs where organizations sign up to receive pre-releasenotifications about software vulnerabilities for products such asKubernetes andEnvoy.Collaboration also happens informally due to our engagementwith many open source projects such as the Linux kernel, container runtimes,virtualization technology, and others.

For Kubernetes, Google is an active and founding member of theSecurity Response Committee (SRC) and wrote much of theSecurity Release Process.Google is a member of theKubernetes Distributors Listthat receives prior notification of vulnerabilities and has been involved inthe triage, patching, mitigation development, and communication of nearlyevery serious Kubernetes security vulnerability. Google also discovered multipleKubernetesvulnerabilitiesourselves.

While less severe vulnerabilities are discovered and patched outside of theseprocesses, most critical security vulnerabilities are reported privately throughone of the channels previously listed. Early reporting gives Google time beforethe vulnerability becomes public to research how it affects GKE,develop patches or mitigations, and prepare advice and communications forcustomers. When possible, Google patches all clusters before the public releaseof the vulnerability.

How vulnerabilities are classified

GKE makes large investments in security hardening the entire stack,including the OS, container, Kubernetes, and network layers, in addition to settinggood defaults, security-hardened configurations, and managed components. Combined,these efforts help to reduce the impact and likelihood of vulnerabilities.

The GKE security team classifies vulnerabilities according to theKubernetes vulnerability scoring system. Classifications consider many factors includingGKE configuration and security hardening.Because of these factors and the investments GKE makes in security,GKE vulnerability classifications mightdiffer from other classification sources.

The following table describes vulnerability severity categories:

See thesecurity bulletins forexample vulnerabilities, fixes and mitigations, and their ratings.

How vulnerabilities are patched for GKE clusters

Patching a vulnerability involves upgrading to a newGKE version number. GKEversions include versioned components for the operating system,Kubernetes components, and other containers that make up the GKEplatform. Fixing some vulnerabilities requires only a control plane upgrade,performed automatically by Google on GKE, while others requireboth control plane and node upgrades.

To keep clusters patched and hardened against vulnerabilities of all severities,werecommendusingnode auto-upgrade on GKE(on by default). For clusters enrolled in release channels, patch releases arepromoted as they meet each channel's qualification requirements. If you need aGKE patch release before it reaches your cluster's channel, youcanmanually upgrade to the patch version if the patch release is on the sameminor version as one available in your cluster's release channel.

On other platforms where clusters run, Google recommends upgradingat least monthly. For a list of platforms, see the previous sectionPatching shared responsibility.

Some security scanners or manual version checks might incorrectly assume thata component like runc or containerd is missing a specific upstream securitypatch. GKE regularly patches components and only performs packageversion upgrades when required, which means that GKE componentsare functionally similar to their upstream counterparts even if the versionnumber of the GKE component doesn't match the upstream versionnumber. For details about a specific CVE, seeGKE security bulletins.

Patching timelines

Google's goal is to mitigate detected vulnerabilities within a time periodappropriate for the risks they represent. GKE is included withinGoogle Cloud's FedRAMP provisional ATOwhich requires that known vulnerabilities to be remediated within specifictimeframes according to their severity level as specified in control RA-5(d) ofthe "Summary of Continuous Monitoring Activities & Deliverables" table in theFedRAMP Continuous Monitoring Strategy Guide.

For every known vulnerability, GKE's goal is to release patchversions that remediate the vulnerability within the corresponding timeframe.The Google Cloud FedRAMP provisional ATO doesn't includeGoogle Distributed Cloud, GKE on AWS, GKE on Azure, orGKE attached clusters, butwe aim for the same remediation timeframes on those products. AfterGKE makes patch versions available to remediate a knownvulnerability, upgrade your clusters to those versions to meet patchingtimeframes for your organization.

How vulnerabilities and patches are communicated

The best source for current information about vulnerabilities and securitypatches is in thesecurity bulletins feedfor the following products:

• GKE
• Google Distributed Cloud
• GKE on AWS
• GKE on Azure
• Google Distributed Cloud

These bulletins follow a common Google Cloud vulnerability numbering scheme and arelinked from themain Google Cloud bulletins pageand theGKE Release Notes.Each security bulletins page has an RSS feed where users can subscribe to updates.

Vulnerabilities are sometimes kept private under embargoes for a limited time.Embargoes help prevent early publication of vulnerabilities that might lead towidespread exploitation attempts before steps can be taken to address them. Inembargo situations, the release notes refer to "security updates" until theembargo has been lifted. After the embargo is lifted, Google updates releasenotes to include the specific vulnerabilities.

The GKE security teampublishes security bulletinsfor High and Critical severity vulnerabilities. When customer action is requiredto address these High and Critical vulnerabilities, Google contacts customers byemail. In addition, Google might also contact customers with support contractsthrough support channels.