Google Cloud offers configurable health checks forGoogle Cloud loadbalancer backends, Cloud Service Mesh backends, andapplication-based autohealing for managed instancegroups. This document covers keyhealth checking concepts.

Unless otherwise noted, Google Cloud health checks are implemented bydedicated software tasks that connect to backends according to parametersspecified in a health check resource. Each connection attempt is called aprobe. Google Cloud records the success or failure of each probe.

Based on a configurable number of sequential successful or failed probes,an overall health state is computed for each backend. Backendsthat respond successfully for the configured number of times are consideredhealthy. Backends that fail to respond successfully for a separatelyconfigurable number of times areunhealthy.

The overall health state of each backend determines eligibility to receive newrequests or connections. You can configure the criteria that define a successfulprobe. This is discussed in detail in the sectionHow health checkswork.

Health checks implemented by dedicated software tasks use special routes thataren't defined in your Virtual Private Cloud (VPC) network. For more information,seePaths for health checks.

Health check categories, protocols, and ports

Health checks have acategory and aprotocol. The two categories arehealthchecks andlegacy health checks and their supported protocols are as follows:

• Health checks

  • Regional (gRPC, TCP, SSL, HTTP, HTTPS, or HTTP/2)
  • Regional gRPC (with TLS) (Preview)
  • Global (gRPC, TCP, SSL, HTTP, HTTPS, or HTTP/2)
  • Global gRPC (with TLS) (Preview)

• Legacy health checks:

  • Legacy global (HTTP)
  • Legacy global (HTTPS)

The protocol and port determine how health check probes are done. For example, ahealth check can use the HTTP protocol on TCP port 80, or it can use the TCPprotocol for a named port in an instance group.

You cannot convert a legacy health check to a health check, and you cannotconvert a health check to a legacy health check.

• Category: health check or legacy health check. Onlytarget pool-basedexternal passthrough Network Load Balancersrequire legacy health checks. For all other products, you'll use regularhealth checks.
• Protocol: protocol that Google Cloud uses to probe the backends.It'sbest to use a health check (or legacy health check) whose protocolmatches the protocol used by the load balancer's backend service or target pool.However, the health check protocols and load balancer protocols donotneed to be the same.
• Port specification: ports that Google Cloud uses with the protocol.You must specify a port for your health check. Health checks have two portspecification methods:--port and--use-serving-port. For legacy healthchecks, there is one method:--port. For more information about health checkport requirements per load balancer, seePort specificationflags.

The next section describes valid health check selections for each type of loadbalancer and backend.

Load balancer guide

This table shows the supported health check category and scope for each loadbalancer type.

Additional usage notes

• For backend instance groups, zonal NEGs withGCE_VM_IP endpoints, and zonalNEGs withGCE_VM_IP_PORT endpoints, probers only attempt to connect to VMinstances (or VM instances that contain endpoints) if the VMs are running.Probers don't attempt to connect to instances (or VM instances that containendpoints) if they are stopped.

• A target pool-based external passthrough Network Load Balancer must use a legacy HTTP health check. Itcannot use a legacy HTTPS health check or any non-legacy health check. If youuse a target pool-based external passthrough Network Load Balancer to balance TCP traffic, you need to runan HTTP service on the VMs being load balanced so that they can respond tohealth check probes.
  
  Foralmost all other load balancer types, youmust use regular, non-legacyhealth checks where the protocol matches the load balancer's backend serviceprotocol.

• For backend services that use the gRPC protocol, use only gRPC or TCP healthchecks. Don't use HTTP(S) or HTTP/2 health checks.

• Certain Envoy-based load balancers that use hybrid NEG backendsdon't support gRPC health checks. For more information, see theHybrid NEGsoverview.

Health checking with Cloud Service Mesh

Note the following differences in behavior when you're using health checks withCloud Service Mesh.

• With Cloud Service Mesh, health checking behavior for network endpoints of thetypeINTERNET_FQDN_PORT andNON_GCP_PRIVATE_IP_PORT differs from healthchecking behavior for other types of network endpoints. Instead of using thededicated software tasks, Cloud Service Mesh programs Envoy proxies to performhealth checks for internet NEGs (INTERNET_FQDN_PORT endpoints) and hybridNEGs (NON_GCP_PRIVATE_IP_PORT endpoints).

  Envoy supports the following protocols for health checking:

  • HTTP
  • HTTPS
  • HTTP/2
  • TCP

• When Cloud Service Mesh is integrated with Service Directory andyou bind a Service Directory service to a Cloud Service Meshbackend service, you cannot set a health check on the backend service.

How health checks work

The following sections describe how health checks work.

Probes

When youcreate a health check ora legacy health check, you specify thefollowing flags or accept their default values. Each health check orlegacy health check that you create is implemented bymultipleprobes. These flags control how frequentlyeachprobe evaluates instances in instance groups or endpoints in zonal NEGs.

A health check's settings cannot be configured on a per-backend basis. Healthchecks are associated with an entire backend service. For a target pool-basedexternal passthrough Network Load Balancer, a legacy HTTP health check is associated with theentire target pool. Thus, the parameters for the probe are the same for allbackends referenced by a given backend service or target pool.

Probe IP ranges and firewall rules

For health checks to work, you must create ingressallow firewall rules sothat traffic from Google Cloud probers can connect to your backends. Forinstructions, seeCreate required firewallrules.

The following table shows the source IP ranges to allow for each load balancer:

Security considerations for probe IP ranges

Consider the following information when planning health checks and the necessaryfirewall rules:

• The probe IP ranges belong to Google. Google Cloud usesspecialroutes outside of your VPCnetwork, but within Google's production network, for communicating betweenhealth check probers and backend VMs.

• The probe IP ranges are used exclusively within Google's production networkfor health check and load balancing. Google Cloud and Google'sproduction network prevent the probe IP ranges from being used for any otherpurpose by enforcing the following:

  • Google edge routers drop packets from the internet if the packets spoofsource IP addresses from a probe IP range.

  • You can't use the probe IP ranges for subnets in your VPCnetworks. For more information, seeProhibited IPv4 subnetranges andIPv6specifications.

Importance of firewall rules

Google Cloud requires that you create the necessary ingressallowfirewall rules to permit traffic from probers to your backends:

• The probe IP ranges are a complete set ofpossible IP addresses used byGoogle Cloud probers. If you usetcpdump or a similar tool, youmight not observe traffic from all IP addresses in all probe IP ranges.As a best practice, create ingress firewall rules that allowall of the probeIP ranges as sources. Google Cloud can implement new probersautomatically without notification.

• As a best practice, limit these rules to just the protocols and ports thatmatch those used by your health checks.

If you don't have ingressallow firewall rules that permit the health check,theimplieddeny ingressrule blocksinbound traffic. When probers can't contact your backends, the load balancerconsiders your backends to be unhealthy.

Multiple probes and frequency

Google Cloud sends health check probes from multiple redundant systemscalled probers. Probers use specificsource IP ranges.Google Cloud does not rely on just one prober to implement a healthcheck—multiple probers simultaneously evaluate the instances in instancegroup backends or the endpoints in zonal NEG backends. If one prober fails,Google Cloud continues to track backend health states.

Theinterval and timeout settings that you configure for a healthcheck are applied to each prober. For a given backend, software access logs andtcpdump show more frequent probes than your configured settings.

This is expected behavior, and you cannot configure the number of probersthat Google Cloud uses for health checks. However, you can estimate theeffect of multiple simultaneous probes by considering the following factors.

• To estimate the probe frequency per backend service, consider the following:

  • Base frequency per backend service. Each health check has anassociated check frequency, inversely proportional to the configuredcheck interval:

    ¹⁄_{(check interval)}

    When you associate a health check with a backend service, you establisha base frequency used by each prober for backends on that backendservice.

  • Probe scale factor. The backend service's base frequency is multipliedby the number of simultaneous probers that Google Cloud uses. Thisnumber can vary, but is generally between 5 and 10.

• Multiple forwarding rules for internal passthrough Network Load Balancers. If youhave configured multiple internal forwarding rules (each having a different IPaddress) pointing to the same regional internal backend service,Google Cloud uses multiple probers to check each IP address. The probefrequency per backend service is multiplied by the number of configuredforwarding rules.

• Multiple forwarding rules for external passthrough Network Load Balancers. If you haveconfigured multiple forwarding rules that point to the same backend service ortarget pool, Google Cloud uses multiple probers to check each IPaddress. The probe frequency per backend VM, is multiplied by thenumber of configured forwarding rules.

• Multiple target proxies for external Application Load Balancers.If you have multiple target proxies that direct traffic to the sameURL map, Google Cloud usesmultiple probers to check the IP address associated with each target proxy.The probe frequency per backend service is multiplied by the number ofconfigured target proxies.

• Multiple target proxies for external proxy Network Load Balancers andregional internal proxy Network Load Balancers.If you have configured multiple target proxies that direct traffic to the samebackend service,Google Cloud uses multiple probers to check the IP address associatedwith each target proxy. The probe frequency per backend service is multipliedby the number of configured target proxies.

• Sum over backend services. If a backend isused by multiple backend services, the backend instances are contacted asfrequently as the sum of frequencies for each backend service's health check.

  With zonal NEG backends, it's more difficult to determine the exact number ofhealth check probes. For example, the same endpoint can be in multiple zonalNEGs. Those zonal NEGs don't necessarily have the same set of endpoints,and different endpoints can point to the same backend.

Destination for probe packets

The following table shows the network interface and destination IP addresses towhich health check probers send packets, depending on the type of load balancer.

For external passthrough Network Load Balancers and internal passthrough Network Load Balancers, the application must bind tothe load balancer's IP address (or any IP address0.0.0.0).

Success criteria for HTTP, HTTPS, and HTTP/2

HTTP, HTTPS, and HTTP/2 health checks always require an HTTP200 (OK) responsecode to be received before the health check timeout. All other HTTP responsecodes, including redirect response codes like301 and302, are consideredunhealthy.

In addition to requiring an HTTP200 (OK) response code, you can:

• Configure each health check prober to send HTTP requests to a specific requestpath instead of the default request path,/.

• Configure each health check prober to check for the presence of an expectedresponse string in the HTTP response body. The expected response string mustconsist only of single-byte, printable ASCII characters, located within thefirst 1,024 bytes of the HTTP response body.

The following table lists valid combinations of request path and response flagsthat are available for HTTP, HTTPS, and HTTP/2 health checks.

Success criteria for SSL and TCP

TCP and SSL health checks have the following base success criteria:

• For TCP health checks, a health check prober must successfully open aTCP connection to the backend before the health check timeout.

• For SSL health checks, a health check prober must successfully open a TCPconnection to the backendand complete the TLS/SSL handshake before thehealth check timeout.

• For TCP health checks, the TCP connection must be closed in one of thefollowing ways:

  • By the health check prober sending either a FINor RST (reset) packet, or
  • By the backend sending a FIN packet. If a backend sends a TCP RST packet,the probemight be considered unsuccessful if the health check prober hasalready sent a FIN packet.

The following table lists valid combinations of request and response flags thatare available for TCP and SSL health checks. Both request and response flagsmust consist only of single-byte, printable ASCII characters, each string beingno more than 1,024 characters long.

Success criteria for gRPC

gRPC health checks are used only with gRPC applications, Google Cloud loadbalancers, and Cloud Service Mesh. Google Cloud supports two types of gRPChealth checks:

• GRPC_WITH_TLS health checksare used for health checking gRPC backends with TLS enabled. They supportunauthenticated TLS encryption, which means that the health checks don'tverify the server's identity.
• GRPC health checks are used for health checking insecure gRPC backends. Theydon't support authentication and encryption, so they can't be used for gRPCbackends with TLS enabled.

If you are using gRPC health checks (with or without TLS), make sure that thegRPC service sends the RPC response with the statusOK and the status fieldset toSERVING orNOT_SERVING accordingly.

For more information, see the following:

• Additional flag for gRPC healthchecks
• gRPC documentation for healthchecks

Success criteria for legacy health checks

If the response received by the legacy health check probe isHTTP 200 OK,the probe is considered successful. All other HTTP response codes, including aredirect (301,302), are considered unhealthy.

Health state

Google Cloud uses the following configuration flags to determine theoverall health state of each backend to which traffic is load balanced.

Google Cloud considers backends to behealthy after this healthythreshold has been met. Healthy backends are eligible to receive new connections.

Google Cloud considers backends to beunhealthy when the unhealthythreshold has been met. Unhealthy backends are not eligible to receive newconnections; however, existing connections arenot immediately terminated.Instead, the connection remains open until a timeout occurs or until traffic isdropped.

Existing connections might fail to return responses, depending on the cause forfailing the probe. An unhealthy backend can become healthy if it is able to meetthe healthy threshold again.

The specific behavior when all backends are unhealthy differs depending on thetype of load balancer that you're using:

Additional notes

The following sections include some more notes about using health checkson Google Cloud.

Certificates and health checks

Google Cloud health check probers don't perform certificate validation,even for protocols that require that your backends use certificates (SSL, HTTPS,and HTTP/2)—for example:

• You can use self-signed certificates or certificates signed by any certificateauthority (CA).
• Certificates that have expired or that are not yet valid are acceptable.
• Neither theCN nor thesubjectAlternativeName attributes need to match aHost header or DNS PTR record.

Headers

Health checks that use any protocol, but not legacy health checks, allow you toset a proxy header by using the--proxy-header flag.

Health checks that use HTTP, HTTPS, or HTTP/2 protocols and legacy healthchecks allow you to specify an HTTPHost header by using the--host flag.

If you're using any custom request headers, note that the load balancer addsthese headers only to the client requests, not to the health check probes. Ifyour backend requires a specific header for authorization that is missing fromthe health check packet, the health check might fail.

Example health check

Suppose you set up a health check with the following settings:

• Interval: 30 seconds
• Timeout: 5 seconds
• Protocol: HTTP
• Unhealthy threshold: 2 (default)
• Healthy threshold: 2 (default)

With these settings, the health check behaves as follows:

1. Multiple redundant systems are simultaneously configured with thehealth check parameters. Interval and timeout settings are appliedto eachsystem. For more information, seeMultiple probes andfrequency.

2. Each health check prober does the following:

   1. Initiates an HTTP connection from one of thesource IPaddresses to the backend instance every 30 seconds.
   2. Waits up to five seconds for an HTTP200 (OK) status code (thesuccesscriteria for HTTP, HTTPS, and HTTP/2protocols).

3. A backend is considered unhealthy when at least one health check probesystem does the following:

   1. Does not receive anHTTP 200 (OK) response code for twoconsecutive probes. For example, the connection might be refused, orthere might be a connection or socket timeout.
   2. Receives two consecutive responses that don't match theprotocol-specific success criteria.

4. A backend is considered healthy when at least one health check probesystem receives two consecutive responses that match the protocol-specificsuccess criteria.

In this example, each prober initiates a connection every 30 seconds. Thirtyseconds elapses between a prober's connection attempts regardless of theduration of the timeout (whether or not the connection timed out). In otherwords, the timeout must always be less than or equal to the interval, andthe timeout never increases the interval.

In this example, each prober's timing looks like the following, in seconds:

1. t=0: Start probe A.
2. t=5: Stop probe A.
3. t=30: Start probe B.
4. t=35: Stop probe B.
5. t=60: Start probe C.
6. t=65: Stop probe C.

What's next

• To create, modify, and use health checks, seeUse health checks.
• To troubleshoot health checks, enablehealth checklogging.