This document describes how to mitigate the impact of an attacker compromisingtheOAuth bearer tokens that are used by thegcloud CLI.

An attacker can compromise these OAuth tokens if they gain access to an endpointwhere a legitimate user account or service account has already authenticatedwith the gcloud CLI. The attacker can then copy these tokens to another endpointthat they control to make requests that impersonate the legitimate identity.Even after you remove the attacker's access to the compromised endpoint, theattacker cancontinue to make authenticated API requests using the copied tokens. To helpmitigate this risk, you can control access to your systems by usingcredentials that are short-lived and context-aware.

This document is intended for security teams or cloud architects who areresponsiblefor securing their cloud resources from illegitimate access. This documentintroduces the available controls that you can use to proactively reduce theimpact of compromised gcloud CLI OAuth tokens and remediate yourenvironment after an endpoint has been compromised.

Overview

To understand how this threat works, you must understand how thegcloud CLI stores OAuth 2.0 credentials and how those credentialscan be abused if compromised by an attacker.

Types of credentials stored by the gcloud CLI

The gcloud CLI usesOAuth 2.0 access tokens to authenticate requests for Google Cloud APIs. The OAuth flow varies by thecredential types used, but generally the access token and other credentials areaccessible locally. In each case, the access token expires after 60 minutes, butother credential types might be persistent.

When youauthorize the gcloud CLI with a user account,the gcloud CLI initiates a three-legged OAuth consent flow to accessGoogle Cloud APIs on the user's behalf. After the user completes theconsent flow, the gcloud CLI receives an access token and a refreshtoken that allows it to request new access tokens. The long-lived refresh token persists until itsexpiration conditions are met.

When youauthorize the gcloud CLI with a service account,the gcloud CLI initiates a two-legged OAuth flow to accessGoogle Cloud APIs as the service account identity. After you activate aservice account from a private key file, this key is used to periodically requestan access token. The long-lived private key is stored in the gcloud CLIconfiguration and remains valid until youdelete the service account key.

When you run the gcloud CLI inside a Google Cloudenvironment, like Compute Engine or Cloud Shell, the application canautomaticallyfind credentials andauthenticate as a service account.For example, in Compute Engine, an application like the gcloud CLIcan query the metadata server for an access token. Google manages androtates the private signing key that is used to create the access token, and thelong-lived credentials aren't exposed to the application.

When youauthenticate with workload identity federation,applications authenticate based on a credential from an external identityprovider and receive a federated short-lived access token. For more informationon how to store and managethe long-lived credentials used by the external identity provider, seebest practices for using workload identity federation.

How an attacker can use compromised OAuth tokens

If an attacker manages to compromise an endpoint, credentials such as OAuthtokens are valuable targets because they let attackers persist or escalatetheir access.

A developer might have a legitimate need to view their own credentials whenwriting and debugging code. For example, a developer might need toauthenticate for using REST requests to Google Cloud services when working with an unsupported clientlibrary. The developer can view the credentials through various methods,including the following:

• Viewing thegcloud CLI configuration files on the local filesystem
• Querying theCompute Engine metadata server
• Using commands likegcloud auth print-access-token orgcloudauth describeIDENTITY

However, an attacker might use these same techniques after they havecompromised an endpoint.

If an attacker compromises an endpoint, such as a developer workstation, theprimary threat is that the attacker can run gcloud CLI commands orother code with the legitimate credentials of the authenticated identity. Inaddition, the attacker might copy the OAuth tokens to another endpoint that theycontrol to persist their access. When this credential theft happens, there is asecondary threat that the attacker can still use the long-lived OAuth tokens tohave persistent access even after you remove access to the compromisedendpoint.

If the attacker manages to compromise OAuth tokens, they cancomplete the following actions:

• An attacker can impersonate the compromised user or service account.API traffic that uses the compromised tokens is logged as if it came fromthe compromised user or service account, making it difficult to distinguishnormal and malicious activity in logs.
• An attacker can refresh the access token indefinitely using a persistentrefresh token associated with a user or a private key associated with aservice account.
• An attacker can bypass authentication with the user's password or 2-stepverification because the tokens are granted after the sign-in flow.

Best practices for mitigating risks

Implement the controls described in the following sections to help mitigatethe risk of compromised gcloud CLI tokens. If you're following thesecurity best practices that are described in theenterprise foundations blueprint orlanding zone design in Google Cloud,you might already have these controls in place.

Set session length for Google Cloud services

To reduce how long an attacker can exploit a compromised token,set the session length for Google Cloud services.For new customers, adefault session length of 16 hoursis automatically enforced. Customers who created their Google Cloud organizationprior to 2023 might have a default setting to never require reauthentication. Reviewthis setting to ensure that you have a reauthentication policy with a session lengththat is between 1 and 24 hours. The reauthentication policy invalidates the refreshtoken and forces the user to regularly reauthenticate the gcloud CLI withtheir password or security key.

The session length for Google Cloud services is a distinctsetting fromsession length for Google services,which controls web sessions for sign-in across Google Workspace services butdoesn't control reauthentication for the Google Cloud. If you useGoogle Workspace services, set the session length for both.

Configure VPC Service Controls

ConfigureVPC Service Controls across your environment to help ensure that only Google Cloud API traffic thatoriginates within your defined perimeter can accesssupported resources.The service perimeter limits the usefulness of compromised credentials becausethe perimeter blocks requests to restricted services that originate fromattacker-controlled endpoints that are outside of your environment.

Configure Chrome Enterprise Premium

Configure Chrome Enterprise Premium policies to helpsecure the Google Cloud console and Google Cloud APIs. Configure a Chrome Enterprise Premium access leveland binding to selectively allow attributes that are evaluated on every APIrequest, including IP-based access orcertificate-based access for mutual TLS.Requests that use compromised authorization credentials but don't meet theconditions that are defined in your Chrome Enterprise Premium policy are rejected.

Chrome Enterprise Premium is auser-centric control that rejects user API traffic that doesn't meet definedconditions. VPC Service Controls is a resource-centric control that definesthe perimeters within which resources can communicate. VPC Service Controlsapplies to all user identities and service account identities, butChrome Enterprise Premium applies only to user identities within your organization.When used together, Chrome Enterprise Premium and VPC Service Controls reduce theeffectiveness of compromised credentials on an attacker-controlled machine thatis outside of your environment.

Enforce 2-step verification for remote server access

If you let developers access Compute Engine resources using SSH,configureOS Login with 2-step verification.This enforces an additional checkpoint where a user must reauthenticate withtheir password or security key. An attacker with compromised OAuth tokens but nopassword or security key is blocked by this feature.

Remote Desktop Protocol (RDP) access to Windows instances on Compute Engine doesn't support theOS Login service, so 2-step verification can't be granularly enforced for RDPsessions. When usingIAP Desktop or Google Chrome-based RDP plugins, set coarse-grained controls likesession length for Google services and2-step verification settings for the user's web sessions and disable theAllow user to trust thedevice setting under 2-step verification.

Restrict the use of service account keys

When you use a service account key to authenticate, the key value is stored inthe gcloud CLI configuration files, separately from the downloadedkey file. An attacker with access to your environment could copy the key fromthe gcloud CLI configuration or copy the key file from your localfilesystem or internal code repository. Therefore, in addition to your plan tomitigate compromised access tokens, consider how you manage downloaded serviceaccount key files.

Reviewmore secure alternatives for authentication to reduce or eliminate your use cases that depend on a service account key, andenforce the organization policy constraintiam.disableServiceAccountKeyCreation to disable service account key creation.

Consider the principle of least privilege

When designing IAM policies, considerleast privilege.Grant users the roles that they require to accomplish a task at the smallestscope. Don't grant them roles that they don't require.Review and apply role recommendations to avoid IAM policies with unused and excessive roles in yourenvironment.

Protect your endpoints

Consider how an attacker might gain physical access or remote access to yourendpoints, like developer workstations or Compute Engine instances.While a plan to address the threat of compromised OAuth tokens isimportant, also consider how to respond to the threat of how an attackercan compromise your trusted endpoints in the first place. If an attacker hasaccess to your trusted endpoints, they can run gcloud CLIcommands or other code directly on the endpoint itself.

Although comprehensive protection for developer workstations is beyond the scopeof this document, evaluate how your security tools and operations can helpprotect and monitor your endpoints for compromise. Consider the followingquestions:

• How is the physical security of developer workstations protected?
• How do you identify and respond to network breaches?
• How do users get remote access to SSH or RDP sessions?
• How might other persistent credentials like SSH keys or service accountkeys be compromised?
• Are there workflows that use persistent credentials that could bereplaced with short-lived credentials?
• Are there shared devices where someone could read another user's cachedgcloud CLI credentials?
• Can a user authenticate with gcloud CLI from an untrusteddevice?
• How does approved traffic connect to resources inside your VPCService Control perimeter?

Ensure that your security operations address each of these questions.

Align your response teams

Ensure in advance that security teams who are responsible for incident responsehave appropriate access across the Google Cloud console and the Admin Console.If separate teams manage the Google Cloud console and the Admin Console, youmight have a delayed response during an incident.

To remove access from a compromised user account, your incident response teamneeds an Admin Console role, such asUser Management Admin.To assess whether suspicious activity has occurred in your Google Cloudresources, this team might also needIAM roles,such as Security Reviewer across all projects or Logs Viewer on a centralizedlog sink. The necessary roles for your security team will vary based on thedesign and operation of your environment.

Best practices for remediation after a security incident

After an endpoint is compromised, as part ofyour incident management plan, determine how to respond to the primary threat of a compromised endpoint andhow to mitigate potential ongoing damage from the secondary threat ofcompromised tokens. If an attacker has persistent access to thedeveloper workstation, they might copy tokens again after the legitimate userreauthenticates. If you suspect that gcloud CLI tokens might becompromised, open a ticket withCloud Customer Care and complete the recommendations in the following sections. These actions canhelp limit the impact of an event like this in your Google Cloudorganization.

The recommendations in this section overlap with the general guidance onhandling compromised Google Cloud credentials,but focus specifically on the threat of gcloud CLI tokens that arecopied from a compromised endpoint.

Expire active tokens for all user accounts with Google Cloud session control

If you haven't already enforcedGoogle Cloud session control,immediately enable this with a short reauthentication frequency. This controlhelps ensure that all refresh tokens expire at the end of the duration that youdefine, which limits the duration that an attacker can use the compromisedtokens.

Manually invalidate tokens for compromised user accounts

Review the guidance forhandling compromised credentials for any user identities who could have been compromised. In particular,removing gcloud CLI credentials is the most effective method for a security team to address compromised OAuthtokens for user identities. To immediately invalidate refresh tokens and accesstokens for the gcloud CLI and force the user to reauthenticate withtheir password or security key, remove the gcloud CLI from a user's listof connected applications.

An individual user can alsoremove gcloud CLI credentials for their individual account.

Other methods, like suspending the user, resetting the user's password, orresetting sign-in cookies don't specifically address the threat of compromisedOAuth tokens. These methods are generally useful for incident response but don'tinvalidate the access tokens the attacker already controls. For example, if youchose to suspend a user during an investigation but don't revokegcloud CLI tokens, the access tokens might still be valid if thesuspended user is restored before the access tokens expire.

Programmatically invalidate tokens for many user accounts

If you suspect a breach but can't identify which users were affected, considerrevoking active sessions for all users in yourorganization more quickly than the reauthentication policy allows.

This approach can be disruptive to legitimate users and terminate long-runningprocesses that depend on user credentials. If you choose to adopt this approach,prepare a scripted solution for your security operations center (SOC) to run inadvance and test it with a few users.

The following sample code uses the Workspace Admin SDK to identify all useridentities in your Google Workspace or Cloud Identity account who haveaccess to the gcloud CLI. If a user has authorized thegcloud CLI, the script revokes the refresh token and access tokenand forces them to reauthenticate with their password or security key. Forinstructions on how to enable the Admin SDK API and run this code, see theGoogle Apps Script Quickstart.

/** * Remove access to the Google Cloud CLI for all users in an organization * @see https://developers.google.com/admin-sdk/directory/reference/rest/v1/tokens * @see https://developers.google.com/admin-sdk/directory/reference/rest/v1/users * @see https://developers.google.com/apps-script/guides/services/advanced#enabling_advanced_services */functionlistUsersAndInvalidate(){constusers=AdminDirectory.Users.list({customer:'my_customer'// alias to represent your account's customerId}).users;if(!users||users.length===0){Logger.log('Nousersfound.');return;}for(constuserofusers){lettokens=AdminDirectory.Tokens.list(user.primaryEmail).itemsif(!tokens||tokens.length===0){continue;}for(consttokenoftokens){if(token.clientId==="32555940559.apps.googleusercontent.com"){AdminDirectory.Tokens.remove(user.primaryEmail,token.clientId)Logger.log('Invalidatedthetokensgrantedtogcloudforuser%s',user.primaryEmail)}}}}

Invalidate and rotate credentials for service accounts

Unlike access tokens that are granted to user identities, access tokens thatare granted to service accounts can't be invalidated through the Admin Consoleor commands likegcloud auth revoke.Additionally, the session duration that you specify inGoogle Cloud session control applies to user accounts in your Cloud Identity orGoogle Workspace directory, but not to service accounts. Therefore, yourincident response for compromised service accounts needs to address both thepersistent key files and the short-lived access tokens.

If you suspect credentials for a service account were compromised,disable the service account,delete service account keys if any exist, and then, after 60 minutes,enable the service account.Deleting a service account key can invalidate the long-lived credential so thatan attacker can't request a new access token, but it doesn't invalidate theaccess tokens already granted. To ensure access tokens aren't abused withintheir 60 minute lifetime, you must disable the service account for 60 minutes.

Alternatively, you can delete and replace the service account to immediatelyrevoke all short-lived and long-lived credentials, but this might require moredisruptive work to replace the service account in applications.

What's next

• Handle compromised Google Cloud credentials
• Authorize the gcloud CLI
• Authenticate as a service account
• Explore reference architectures, diagrams, and best practices about Google Cloud.Take a look at ourCloud Architecture Center.