This document describes best practices for securely connecting to Compute Enginevirtual machine (VM) instances, includingstoring host keys by enabling guest attributesandpreventing VMs from being reached from the public internet.

If you usegcloud compute ssh to connectto your Linux VMs, you can add a layer of security by storingyour host keys as guest attributes.

Storing SSH host keys as guest attributes improves the security of yourconnections by helping to protect against vulnerabilities such asman-in-the-middle (MITM) attacks. On the initial boot of a VM, ifguest attributes are enabled, Compute Engine stores your generatedhost keys as guest attributes. After that, Compute Engine uses thesestored host keys to verify all subsequent connections to the VM.

Host keys can be stored as guest attributes on the following public operatingsystem images:

• Debian
• Ubuntu
• Red Hat Enterprise Linux (RHEL)
• CentOS
• SUSE Linux Enterprise Server (SLES)

To write host keys to guest attributes, you must enable guest attributes beforeyou boot the VM for the first time. You can enable guest attributeseither on select VMs during VM creation or on your entireproject.

After you enable guest attributes for a project or VM, the Guest OS agentautomatically publishes the host key as a guest attribute. If you usegcloud compute ssh instead of a plain SSH client, then thegcloud CLI automatically reads the attributes and updates theknown_hosts file the next time you connect.

To store host keys as guest attributes, complete the following steps:

1. Before you boot your VM for the first time,enable guest attributeseither on select VMs during VM creation or on yourentire project.

2. Connect to your VM by usinggcloud compute ssh.

   1. Ensure that you have the latest version of the Google Cloud CLI:

      gcloud components update

   2. Connect to the VM:

      gcloud compute ssh --project=PROJECT_ID \ --zone=ZONE \VM_NAME

      Replace the following:

      • PROJECT_ID: the ID of the project thatcontains the VM
      • ZONE: the name of the zone in which theVM is located
      • VM_NAME: the name of the VM

      If you haveset default propertiesfor the Google Cloud CLI, you can omit the--projectand--zone flags from this command. For example:

      gcloud compute sshVM_NAME

   3. Review the startup message. For example, a Debian operating system mightdisplay the following message:

      Writing 3 keys toYOUR_HOME_DIRECTORY/.ssh/google_compute_known_hostsLinux host-key-2 4.9.0-9-amd64 #1 SMP Debian 4.9.168-1+deb9u3 (2019-06-16) x86_64

To confirm that host keys are stored as guest attributes for this VM,either review the host key values to verify that SSH keys are written to guestattributes for the VM (Option 1), or review the serial port for thepresence of host keys (Option 2):

Option 1: Review the host key values

You can use the Google Cloud CLI to verify that SSH keys are written to guestattributes:

gcloud compute instances get-guest-attributesVM_NAME \  --query-path="hostkeys/" \  --zone=ZONE

Replace the following:

• VM_NAME: the name of the VM
• ZONE: the name of the zone in which the VM islocated

The output is similar to the following:

NAMESPACE  KEY                  VALUEhostkeys   ecdsa-sha2-nistp256  AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBJAGpTm                                V3mFxBTHK1NIu9a7kVQWaHsZVaFUsqF8cLxQRQ+N96/Djiiuz1tucHQ8vBTJI=hostkeys   ssh-ed25519          AAAAC3NzaC1lZDI1NTE5AAAAIM/WYBn3jIEW5t3BZumx0X/Htm61J6S9FcU8Lhostkeys   ssh-rsa              AAAAB3NzaC1yc2EAAAADAQABAAABAQDU3jReR/MoSttlWYfauW6qEqS2dhe5                                Zdd3guYk2H7ZyxblNuP56nOl/IMuniVmsFa9v8W6MExViu6G5Cy4iIesot09                                1hsgkG0U7sbWrXM10PQ8pnpI3B5arplCiEMhRtXy64rlW3Nx156bLdcxv5l+                                7Unu4IviKlY43uqqwSyTv+V8q4ThpQ9dNbk1Gg838+KzazljzHahtbIaE1rm                                I0L1lUqKiKLSLKuBgrI2Y/WSuqvqGEz+bMH7Ri4ht+7sAwykph6FbOgKqoBI                                hVWBo38/Na/gEuvtmgULUwK+xy9zWg9k8k/Qtihc6El9GD9y

Option 2: Review the serial port

1. View theserial port output.
2. Selectserial port 1.

3. Search for the following message:

   INFO Wrote ssh-rsa host key to guest attributes

   If your image uses a supported operating system but the guest attributessetting wasn't enabled before the first VM boot, you might see the followingmessage:

   Unable to write ssh-rsa host key to guest attributes

   This means that host keys aren't stored as guest attributes for thisVM. If you want to store host keys for additional VMs that youplan to create, enable guest attributes before the first boot of theVM.

Preventing VMs from being reached from the public internet

When developing projects on Compute Engine, there are a variety ofscenarios in which you want to keep the VMs from being reached from thepublic internet:

• Web services are still under development and not ready to be exposedto external users because they are feature incomplete or have not yetbeen configured with HTTPS.
• The VM might be providing services designed to be consumed only by otherVMs in the project.
• VMs should only be reached through dedicated interconnect options fromcompany offices or data centers.

Even when a service is intentionally internet-facing, it is important thatcommunication with the service be restricted to the target user groups, andoccur over secure channels, such as SSH or HTTPS, to protect sensitiveinformation.

This article demonstrates several methods for securing communications withVMs with external IP addressesandVMs without external IP addresses. Whether or not you securecommunications with these methods, Google Cloud always allows communicationbetween a VM instance and itscorresponding metadata server. For more information, seealways allowed traffic.

Protecting services on machines with external IP addresses

When VMs have a public IP address, it is important that only theservices and traffic you intend to be exposed are reachable, and for those thatare exposed, any sensitive information is secured in transit. There are severalmethods for protecting services on VMs with external IP addresses explained inthis document, includingfirewalls,HTTPS and SSL,port forwarding over SSH, andSOCKS proxy over SSH.

Firewalls

Your first line of defense is to restrict who can reach the VM usingfirewalls.By creating firewall rules, you can restrict all traffic to a network ortarget machines on a given set of ports to specific source IP addresses.

Firewalls are not a standalone solution. Restricting traffic tospecific source IPs does not protect sensitive information, such aslogin credentials, commands that create or destroy resources or files, orlogs. When running a web service on a publicly-accessible machine, suchas a Compute Engine VM with an external IP, you mustencrypt all communication between your host and the deployed VMto ensure proper security.

In addition, firewalls aren't always the appropriate solution. For example,firewalls are not ideal for development environments that do not have staticIP addresses, such as roaming laptops.

HTTPS and SSL

For production web systems, you should configure HTTPS/SSL. HTTPS/SSL can beset up either by setting up a VM to terminate HTTPS or by configuringHTTPS load balancing. HTTPS/SSL does involve some initial complexity,requiring you to perform the following tasks:

• Register a domain name.
• Acquire an SSL certificate from a certificate authority.
• Register the certificate with your HTTPS load balancerand its connected VMs, or configure an SSL-terminated web server orproxy on one or more Compute Engine VMs.

Port forwarding over SSH

You can use theGoogle Cloud CLI to start a server on a given local port that forwards all traffic to aremote host over an SSH connection.

First, take note of the VM and port that are providing the serviceto which you would like to establish a secure connection. Next, run thefollowing command:

gcloud compute sshVM_NAME \    --projectPROJECT_ID \    --zoneZONE \    -- -NLLOCAL_PORT:localhost:REMOTE_PORT

• VM_NAME is the name of the VM to which you'd like to connect.
• PROJECT_ID is yourGoogle Cloud project ID.
• ZONE: The zone in which your VM is running, forexample,us-central1-a.
• LOCAL_PORT: The local port you're listening on, for example,2222.
• REMOTE_PORT: The remote port you're connecting to, for example8888.

For example, if you specify a local port of '2222' and a remote port of '8888',and you openhttp://localhost:2222/ in your browser,the HTTP connection uses the SSH tunnel that you created to your remote host toconnect to the specified VM using SSH. The HTTP connection will then use the SSHtunnel to connect to port8888 on the same machine, but over an encrypted,secure SSH connection.

Thegcloud command creates and maintains an SSH connection while the SSHsession is active. As soon as you exit the SSH session, port forwarding usinghttp://VM_NAME:LOCAL_PORT stops working.

To create more than one port forwarding rule, you can specifymultiple rules on a single command line by repeating the flags:

gcloud compute sshVM_NAME \    --projectPROJECT_ID \    --zoneZONE \    -- -NLLOCAL_PORT:localhost:REMOTE_PORT \    -- -NLLOCAL_PORT:localhost:REMOTE_PORT

If you want to connect to a number of different hosts in your clouddeployment, the easiest way to do so is to change your browser to do thelookups directly from yournetwork. This approachlets you use the short name of the hosts instead of looking up eachhost's IP address, opening up ports for each service, or creating an SSH tunnelfor each host/port pair.

The approach that you use here is as follows:

1. Set up a single SSH tunnel to one of the hosts on the network, and create aSOCKS proxy on that host.
2. Change the browser configuration to do all the lookups using that SOCKS proxyhost.

Note that because you are tunnelingall traffic using that host, avoid usingthat browser or that specific profile to browse the web becauseyou need to dedicate that bandwidth to your cloud service. In general, youmight want to use a separate browser profile and switch to it when necessary.

Start the SOCKS proxy

To start your SOCKS proxy, run the following command:

gcloud compute sshVM_NAME \    --projectPROJECT_ID \    --zoneZONE    --ssh-flag="-D" \    --ssh-flag="LOCAL_PORT" \    --ssh-flag="-N"

Replace the following:

• VM_NAME: The name of the VM to whichyou would like to connect.
• PROJECT_ID: YourGoogle Cloud project ID.
• ZONE: The zone in which your VM is running, forexample,us-central1-a.
• LOCAL_PORT: The local port you're listening on, for example,1080.

Note that, in this case, you don't need to specify a remote port. Because aSOCKS proxy does not bind to any specific remote port, any connection youmake using the SOCKS proxy will be resolved relative to the host you connectto.

By using a SOCKS proxy, you can connect to any VM that shares aCompute Engine network with your proxy VM by using the VM'sshort name.In addition, you can connect to any port on a given VM.

This approach is much more flexible than the simple port-forwarding method, butwill also require you to change the settings in your web browser to utilize theproxy.

Next, configure either Chrome or Firefox to use the proxy.

/usr/bin/google-chrome \    --user-data-dir="$HOME/chrome-proxy-profile" \    --proxy-server="socks5://localhost:1080"

"/Applications/Google Chrome.app/Contents/MacOS/Google Chrome" \    --user-data-dir="$HOME/chrome-proxy-profile" \    --proxy-server="socks5://localhost:1080"

"C:\Program Files (x86)\Google\Chrome\Application\chrome.exe" ^    --user-data-dir="%USERPROFILE%\chrome-proxy-profile" ^    --proxy-server="socks5://localhost:1080"

Connecting to VMs without external IP addresses

When VMs do not have external IP addresses (including VMs that are backends forHTTPS and SSL proxy load balancers) they can only be reached by the following:

• Other VMs on the network
• Identity-Aware Proxy's TCP forwarding feature
• Themetadata server
• Google Cloud SDK
• ManagedVPN gateway

You can provision VMs in your network to actas trusted relays for inbound connections, also known asbastionhosts. Additionally, you can configure aCloud NATfor outbound network traffic, or set up theinteractive serial console to maintain ortroubleshoot VMs without external IP addresses.

Bastion hosts

Bastion hosts provide an external facing point of entry into a network containing privatenetwork instances, as illustrated in the following diagram.

This host can provide a single point of fortification oraudit and can be started and stopped to enable or disable inbound SSH. By usinga bastion host, you can connect to a VM that does not have an externalIP address. This approach allows you to connect to a development environment ormanage the database instance for your external application, for example, withoutconfiguring additional firewall rules.

A complete hardening ofa bastion host is outside the scope of this article, but some initial stepstaken can include:

• Limit the CIDR range of source IPs that can communicate with the bastion.
• Configure firewall rules to allow SSH traffic to private VMs from onlythe bastion host.

By default, SSH on VMs is configured to use private keys forauthentication. When using a bastion host, you log into the bastion host first,and then into your target private VM. Because of this two-step login,which is why bastion hosts are sometimes called "jump servers," you should usessh forwarding instead of storing the target machine's private key onthe bastion host as a way of reaching the target machine. You need to do thiseven if using the same key pair for both bastion and target VMs becausethe bastion has direct access to only the public half of the key pair.

To learn how to use a bastion host instance to connect to other VMs onyour Google Cloud network, seeConnect to Linux VMs using a bastion host.

To learn how to usessh forwarding and other methods to connect to VMsthat do not have an external IP address, seeseeConnecting to VMs that do not have external IP addresses.

IAP for TCP forwarding

Using SSH with IAP's TCP forwarding feature wraps an SSHconnection inside HTTPS. IAP's TCP forwarding feature then sendsit to the remote VM.

To learn how to connect to a remote VM with IAP, seeConnect to Linux VMs using Identity-Aware Proxy.

VPN

Cloud VPN lets you connect your existing network to yourGoogle Cloud network by using an IPsec connection to a VPN gateway device.This allows direct routing of traffic from your premises to the private IPinterfaces of Compute Engine VMs. Traffic is encrypted as ittransits over public links to Google.

For details on setting up, configuring, and using VPN withCompute Engine, see theCloud VPN documentation.

To learn how to connect to VMs on your Google Cloud network throughan existing VPN rather than through external IP addresses of VMs, readConnect to Linux VMs using Cloud VPN or Cloud Interconnect.

Outbound traffic using Cloud NAT

When a VM does not have an external IP address assigned, it cannot makedirect connections to external services, including other Google Cloudservices. To allow these VMs to reach services on the public internet, youcan set up and configureCloud NAT, which can routetraffic on behalf of any VM on the network. Do not consider a singleVM to be highly available or able to support high traffic throughput formultiple VMs.

Interactive serial console access

When a VM doesn't have an external IP address, you might stillneed to interact with the VM for troubleshooting or maintenance purposes.Setting up aBastion host, as discussed earlier, is one option butmight require more setupthan worthwhile for your needs. If you want to troubleshoot a VM withoutan external IP address, considerenabling interactive access on the serial console,which allows you to interact with a VM's serial console using SSH andrun commands against the serial console.

To learn more, readInteracting with the Serial Console.

HTTPS and SSL proxy load balancers

VMs that are backends for HTTPS and SSL proxy load balancers do nothave to have external IP addresses to be accessed through the load balancer. Toaccess these resources directly requires the use of methods listed in the sectionConnecting to VMs without external IP addresses.

To learn more, read theload balancing documentation forthose load balancers.