This document describes best practices for protecting SSH credentials.

By default, Compute Engine uses public key-based SSH authentication: Usersare authenticatedby something they have, which is an SSH private key. If users'private keys aren't properly secured, they might fall into the hands of bad actorswho might use these keys to access your VM instances.

The following sections contains best practices that can help you avoid key leakageand reduce the potential impact of leaked private keys:

• Treat SSH private keys similar to service account keys
• Use ephemeral SSH keys for machine users
• Use IAP to complement SSH public key authentication
• Use multi-factor authentication
• Use non-exportable or passphrase-protected private keys
• Use host keys to authenticate the host
• Don't leave personal credentials on VMs
• Don't submit SSH private keys to source code repositories

The document focuses on practices that are either specific to Google Cloud orof particular relevance when using SSH on Google Cloud. The document doesn'tcover best practices for specific SSH client or server implementations.

Treat SSH private keys similar to service account keys

Some of your VM instances may have anattachedservice account. Attaching a service account to a VM lets workloads running onthese VMsrequest short-lived access tokens from the metadata serverso that they can access Google Cloud APIs and resources.

When you connect to a VM with an attached service account by using SSH, you canalsorequest short-lived access tokens from the metadataserver.Granting a user SSH access to a VM is therefore similar to granting the userpermission toact as the attached service account.Because of that similarity, treat SSH private keys, especiallywhen they're not passphrase-protected,like service account keys:Both kinds of keys, if leaked, might grant a bad actor access to Google Cloudresources.

Use ephemeral SSH keys for machine users

Deployment pipelines or automation processes might require SSH access to VM instancesto perform deployments or apply configuration changes. Instead of letting theseworkloads use a long-lived SSH key pair, let them use a new, ephemeral SSH key every time they run.

To use ephemeral SSH keys, let your deployment pipelines or automation processes perform the following steps:

1. Authenticate as a service account in a way that doesn't involve a key or secret,for example by using anattached service accountorworkload identity federation.
2. Generate a temporary SSH key pair by using a tool such asssh-keygen.

3. Publish the public key to Google Cloud, specifying a near-futureexpiry date (such as 1h in the future).

   OS Login lets you specify a key expiration date whenyou publish a key.Similarly, you canspecify an expiry datewhen you publish an SSH public key to project or VM metadata.

4. Use the private key to establish SSH connections to VM instances.

5. Optionally, unpublish the public key and delete the private key.

For example:

#GenerateRSAkeypairwithoutpassphrasessh-keygen-trsa-fephemeral_key-q-N""-V30m#Publishkeytotheserviceaccount's OS Login profile, with 30 min expirygcloud compute os-login ssh-keys add --key-file ephemeral_key.pub --ttl 30m# Look up the service account'sUNIXusernameUSERNAME=$(gcloudcomputeos-logindescribe-profile--format "value(posixAccounts[0].username)")#Authenticateusingtheserviceaccount'sUNIXuserandpublickeyssh$USERNAME@VMwhoami#Removekeygcloudcomputeos-loginssh-keysremove--key-file ephemeral_key.pub

While an ephemeral SSH private key might still be leaked, it can only be usedfor a short time. Using ephemeral SSH keys can therefore reduce your risk ofcredential leakage, and lets you use Cloud IAM as the primarymeans of authentication and authorization.

Use IAP to complement SSH public key authentication

By default, SSH private keys can be used independently of Google credentials:If a user's private SSH key is leaked, a bad actor can use the key to connectand authenticate to any VM instances that the key is authorized to access. It'snot necessary for the bad actor to know the user's username or password, or toeven possess any Google credential.

Security controls such astwo-step verificationandlimiting the session length for Google Cloud servicescan be effective ways to reduce the risk of credential theft, but these controlsonly apply to resources that require Google credentials.

To ensure that SSH keys can't be used without valid Google credentials, useIAP to govern SSH access anduse firewall policiesto enforce that all SSH access is performed through IAP.

IAP acts as a reverse proxy and only permits users to establish SSH connections toVM instances if they successfully authenticated using their Google credentials.Additionally, IAP lets yourestrict which VMs users can connect to,and enforce context-aware access.

Use multi-factor authentication

Using IAP to govern SSH access makes it more difficult for abad actor to access VM instances using leaked credentials, but doesn't make itimpossible: For example, a bad actor might compromise a workstation and findboth, a private SSH key and cached gcloud CLI credentials – enoughto pass IAP's authentication and authorization checks, and connectto the user's VM instances.

You can reduce the possible impact of such credential theft attacks by configuringCloud Identity or Google Workspace to require multi-factorauthentication (MFA).

If youCloud Identity or Google Workspace is your primary identity provider,do the following enforce MFA:

1. Configure Cloud Identity or Google Workspace toenforce 2-step verification.
2. Limit the session length for Google Cloud servicesso that cached credentials are automatically invalidated and users have toperiodically re-authenticate and perform MFA.

If youuse single sign-on with an external IdP,do the following instead:

1. Configure Cloud Identity or Google Workspace tolimit the session length for Google Cloud servicesso that cached credentials are automatically invalidated and users have toperiodically re-authenticate using the external IdP.
2. Configure your external IdP to require MFA, and limit its session length sothat users have to perform MFA every time their Google Cloud session expires.

To ensure that MFA also applies to SSH access, you must also do at least one of the following:

1. Use IAP to control network accessso that users have to periodically perform MFA to refresh their Google credentials.
2. Enable OS Login 2FA for individualVM instances or entire projects so that users have to perform MFA every timethey establish an SSH connection.

Users that have theCompute Instance Admin or an equivalent role for a VM instanceor project can disable OS Login 2FA by modifying instance metadata. Theeffectiveness of OS Login 2FA is therefore limited if you don't also enforceMFA in Cloud Identity or your external IdP.

Use non-exportable or passphrase-protected private keys

Many SSH clients default to storing SSH private keys as files on disk. For example,gcloud compute ssh generates an SSH key pair on first use, and stores it in yourhome directory. Your operating system might protect your files from being accessedby other users, but if a bad actor can overcome file system permissions (forexample, by copying and mounting the disk on another machine), they can copy thekey elsewhere, and use it without your knowledge.

Some SSH clients let you avoid using file-based keys and offer alternative options to manage SSH private keys, such as:

• Using a hardware-backed key: Modern versions of OpenSSH let you use FIDO2security keys for authentication, and you canconfigure OS Loginso that it only permits security keys that are enrolled in Cloud Identityor Google Workspace. Using hardware-backed keys helps you avoid storingany private key material on your computer's file system.
• Using your operating system's key storage facilities: For example,IAP Desktop avoids using file-based keys and instead usesWindows CNG to protect your SSH keys.

If using hardware-backed or operating system-managed keys isn't an option, youcan use a passphrase to protect your SSH private key: To use a passphrase-protectedSSH key, a bad actor not only needs a copy of the private key, but also needs toknow the key's passphrase.

Use host keys to authenticate the host

When you create an SSH connection to a VM instance, you identify the VM instanceby its name or IP address. Names and IP addresses can be reassigned and reused,and the name that referred to a certain VM instance yesterday might not refer tothe same VM instance today. Bad actors might deliberately reassign or reuse namesor IP addresses tospoof VM instances and lure users into connecting to acompromised VM.

SSH clients can detect situations where a previously-trusted VM instance wasreplaced with a different VM instance by using SSH host keys: A VM's SSH hostkey is generated on first boot and is used to identify the instance. SSH clientstypically request and store a VM's host key on the first connection and verifythat the VM's host key hasn't changed on subsequent connections.

SSH host keys work based on thetrust on first usescheme. The effectiveness of SSH host keys can be undermined if a bad actor usesa man in the middle (MITM) attack to let a client connect to and trust the wrongVM on first use. A better way to obtain a host key is to obtain it over a trustedside channel before connecting to a VM for the first time.

You can let the gcloud CLIobtain host keys over a back channel by enabling guest attributesin your project. The gcloud CLI then reads a VM's host key before youfirst connect to it, and saves them on your local computer.

Don't leave personal credentials on VMs

When youauthorize the gcloud CLI, the toolobtains anOAuth refresh tokenand stores it in your local home directory. When you subsequently run a gcloud CLIcommand, the gcloud CLI uses the refresh token to authenticate you automatically.

Your local computer might be inaccessible to other users, but on a VM instance,your home directory can also be accessed by other users that have sudoprivileges on the VM.

If a bad actor manages to obtain sudo privileges on a VM, they might scan forrefresh tokens and other credentials in other users' home directories, anduse these credentials to escalate their privileges or extend their access to other resources(lateral movement).

When you're connected to a VM instance over SSH, avoid authorizing the gcloud CLIor Application Default Credentials (ADC) with your personal credentials, and letthe gcloud CLI use the VM's attached service account instead.Similarly, avoid running other tools that might store personal credentials inyour home directory.

You can further reduce risks bylimiting the session length for Google Cloud servicesso that stored OAuth refresh tokens automatically expire after a certain amount of time.

Don't submit SSH private keys to source code repositories

Some automation tools like Ansible use SSH to access and manage VM instances.Because such tools might have access to many VM instances (and their attachedservice accounts), the SSH private keys used by such tools can be particularlysensitive.

If you submit an SSH private key to a source code repository, there is anincreased risk that the key becomes accessible to unauthorized users and badactors:

• Bad actors might scan the source code of public source repositories for leaked keys.
• In the future, you might decide to turn a private source repository into a publicrepository, without checking it for keys first.
• Other team members might store copies of the source code on their workstation.

To mitigate these risks, store the SSH private key in a secure location that'sseparate from the source code and use ephemeral SSH keys when possible.

• Continue reading aboutbest practices for auditing SSH access