This document introduces the concepts that you need to understandhow to configure an external Application Load Balancer.

An external Application Load Balancer is a proxy-based Layer 7 load balancer that enables you to runand scale your services behind a single external IP address. The external Application Load Balancerdistributes HTTP and HTTPS traffic to backends hosted on a variety ofGoogle Cloud platforms (such as Compute Engine,Google Kubernetes Engine (GKE), and Cloud Storage), as well as externalbackends connected over the internet or through hybrid connectivity.For details, seeApplication Load Balancer overview: Usecases.

Modes of operation

You can configure an external Application Load Balancer in the following modes:

• Global external Application Load Balancer. This is a global load balancer that isimplemented as a managed service onGoogle Front Ends(GFEs). Ituses the open-sourceEnvoy proxyto supportadvanced trafficmanagementcapabilities such as traffic mirroring, weight-based traffic splitting, andrequest-based or response-based header transformations.
• Classic Application Load Balancer. This is the classic external Application Load Balancer that isglobal in Premium Tier but can be configured to be regional in StandardTier. This load balancer is implemented onGoogle Front Ends(GFEs). GFEsare distributed globally and operate together using Google's global networkand control plane.
• Regional external Application Load Balancer. This is a regional load balancer that isimplemented as a managed service on the open-sourceEnvoyproxy. It includesadvanced trafficmanagementcapabilities such as traffic mirroring, weight-based traffic splitting, andrequest-based or response-based header transformations.

Identify the mode

   gcloud compute forwarding-rules describeFORWARDING_RULE_NAME

• For regional external Application Load Balancers only, aproxy-onlysubnet is used to send connections from the loadbalancer to the backends.

• Anexternal forwarding rule specifies an external IPaddress, port, and target HTTP(S) proxy. Clients use the IP address and portto connect to the load balancer.

• Atarget HTTP(S) proxy receives a request from theclient. The HTTP(S) proxy evaluates the request by using the URL map to maketraffic routing decisions. The proxy can also authenticate communications byusing SSL certificates.

  • For HTTPS load balancing, the target HTTPS proxy usesSSLcertificates to prove its identity to clients. Atarget HTTPS proxy supports up to thedocumentednumber ofSSL certificates.

• The HTTP(S) proxy uses aURL map to make a routingdetermination based on HTTP attributes (such as the request path, cookies,or headers). Based on the routing decision, the proxy forwards clientrequests to specific backend services or backend buckets. The URL map canspecify additional actions, such as sending redirects to clients.

• Abackend service distributes requests to healthybackends. Theglobal external Application Load Balancers also supportbackend buckets. One or morebackends must be connected to the backend service or backend bucket.

• Ahealth check periodically monitors the readiness ofyour backends. This reduces the risk that requests might be sent to backendsthat can't service the request.

• Firewall rules for your backends to accept healthcheck probes. Regional external Application Load Balancers require an additional firewallrule to allow traffic from the proxy-only subnet to reach the backends.

Global

This diagram shows the components of a global external Application Load Balancer deployment.This architecture applies to both, the global external Application Load Balancer, and theclassic Application Load Balancer in Premium Tier.

Regional

This diagram shows the components of a regional external Application Load Balancer deployment.

Proxy-only subnet

Proxy-only subnets are only required for regional external Application Load Balancers.

The proxy-only subnet provides a set of IP addresses that Google uses to runEnvoy proxies on your behalf. You must createone proxy-only subnet in eachregion of a VPC network where you use regional external Application Load Balancers.The--purpose flag for this proxy-only subnet is set toREGIONAL_MANAGED_PROXY. Allregional Envoy-based loadbalancers in the same regionand VPC network share a pool of Envoy proxies from the sameproxy-only subnet. Further:

• Proxy-only subnets areonly used for Envoy proxies, not your backends.
• Backend VMs or endpoints of all regional external Application Load Balancers in a region andVPC network receive connections from the proxy-only subnet.
• The IP address of the regional external Application Load Balancer isnot located in theproxy-only subnet. The load balancer's IP address is defined by its externalmanaged forwarding rule, which is described below.

If you previously created a proxy-only subnet with--purpose=INTERNAL_HTTPS_LOAD_BALANCER,migrate the subnet'spurpose toREGIONAL_MANAGED_PROXY before you can create other Envoy-based load balancersin the same region of the VPCnetwork.

Forwarding rules and IP addresses

Forwarding rules route trafficby IP address, port, and protocol to a load balancing configuration consistingof a target proxy, URL map, and one or more backend services.

IP address specification. Each forwarding rule provides a single IP addressthat can be used in DNS records for your application. No DNS-based loadbalancing is required. You can either specify the IP address to be used or letCloud Load Balancing assign one for you.

Port specification. Each forwarding rule for an Application Load Balancer canreference asingle port from1-65535. Tosupport multiple ports, you must configure multiple forwarding rules. You canconfigure multiple forwarding rules to use the same external IP address (VIP)and to reference the same target HTTP(S) proxy as long as the overallcombination of IP address, port, and protocol is unique for each forwardingrule. This way, you can use a single load balancer with a shared URL map as aproxy for multiple applications.

The type of forwarding rule, IP address, and load balancing scheme used byexternal Application Load Balancers depends on the mode of the load balancer and whichNetworkService Tier the load balancer is in.

For the complete list of protocols supported by external Application Load Balancerforwarding rules in each mode, seeLoad balancerfeatures.

Forwarding rules and VPC networks

This section describes how forwarding rules used by external Application Load Balancers areassociated with VPC networks.

Target proxies

Target proxies terminate HTTP(S)connections from clients. One or more forwarding rules direct traffic to thetarget proxy, and the target proxy consults the URL map to determine how toroute traffic to backends.

Do not rely on the proxy to preserve the case of request or response headernames. For example, aServer: Apache/1.0 response header might appear at theclient asserver: Apache/1.0.

The following table specifies the type of target proxy required byexternal Application Load Balancers.

In addition to headers added by the target proxy, the load balancer adjustsother HTTP headers in the following ways:

• For the global external Application Load Balancer, both request and response headers might beconverted to lowercase.

  The only exception to this is when you use global internet NEG backends withHTTP/1.1. For details about how HTTP/1.1 headers are processed with globalinternet NEGs, see theInternet NEGsoverview.

• For the classic Application Load Balancer, request and response headers areconverted to lowercase except when you use HTTP/1.1. With HTTP/1.1, headersare proper-cased instead. The first letter of the header's key and everyletter following a hyphen (-) is capitalized to preserve compatibilitywith HTTP/1.1 clients. For example,user-agent is changed toUser-Agent,andcontent-encoding is changed toContent-Encoding.

• Some headers are coalesced. When there are multiple instances of the sameheader key (for example,Via), the load balancer combines their valuesinto a single comma-separated list for a single header key. Only the headerswhose values can be represented as a comma-separated list are coalesced.Other headers, such asSet-Cookie, are never coalesced.

Host header

When the load balancer makes the HTTP request, the load balancer preserves theHost header of the original request.

X-Forwarded-For header

The load balancer appends two IP addresses to theX-Forwarded-For header, separated by a single comma, in the following order:

1. The IP address of the client that connects to the load balancer
2. The IP address of the load balancer's forwarding rule

If the incoming request doesnot include anX-Forwarded-For header,the resulting header is as follows:

X-Forwarded-For: <client-ip>,<load-balancer-ip>

If the incoming request already includes anX-Forwarded-For header,the load balancer appends its values to the existing header:

X-Forwarded-For: <existing-value>,<client-ip>,<load-balancer-ip>

It is possible to remove existing header values by usingcustom requestheaders on thebackend service. The following example uses the--custom-request-header flagto recreate theX-Forwarded-For header by using the variablesclient_ip_address andserver_ip_address. This configuration replaces theincomingX-Forwarded-For header with only the client and the load balancer IPaddress.

--custom-request-header=x-forwarded-for:{client_ip_address},{server_ip_address}

<existing-value>,<client-ip>,<load-balancer-ip>,<GFE-ip>,<backend-ip>

URL maps define matching patterns forURL-based routing of requests to the appropriate backend services. The URL mapallows you to divide your traffic by examining the URL components tosend requests to different sets of backends. A default service is defined tohandle any requests that don't match a specified host rule or path matchingrule.

In some situations, such as themulti-region loadbalancing example, you might notdefine any URL rules and rely only on the default service.

URL maps support several advanced traffic management features such asheader-based traffic steering, weight-based traffic splitting, and requestmirroring. For more information, see the following:

• Traffic management overview forglobal external Application Load Balancer

• Traffic management overview forregional external Application Load Balancer

The following table specifies the type of URL map required by external Application Load Balancers ineach mode.

SSL certificates

External Application Load Balancers using target HTTPS proxies require private keys andSSL certificates as part of the load balancer configuration.

• Google Cloud provides twoconfiguration methods for assigning privatekeys and SSL certificates to target HTTPS proxies: Compute Engine SSLcertificates and Certificate Manager. For a description of eachconfiguration, seeCertificate configurationmethods in the SSLcertificates overview.

• Google Cloud provides twocertificate types: Self-managed andGoogle-managed. For a description of each type, seeCertificatetypes in the SSLcertificates overview.

The type of external Application Load Balancer (global, regional, or classic) determines whichconfiguration methods and certificate types are supported. For details, seeCertificates and Google Cloud loadbalancers in theSSL certificates overview.

SSL policies

SSL policies specify the set ofSSL features that Google Cloud load balancers use when negotiating SSLwith clients.

By default, HTTPS Load Balancing uses a set of SSL features that provides goodsecurity and wide compatibility. Some applications require more control overwhich SSL versions and ciphers are used for their HTTPS or SSL connections. Youcan define an SSL policy to specify the set of SSL features that your loadbalancer uses when negotiating SSL with clients. In addition, you can apply thatSSL policy to your target HTTPS proxy.

The following table specifies the SSL policy support for load balancers in eachmode.

Backend services

A backend service provides configuration information to the load balancer sothat it can direct requests to its backends—for example,Compute Engine instance groups or network endpoint groups (NEGs). Formore information about backend services, seeBackend servicesoverview.

For an example showing how to set up a load balancer with a backend service anda Compute Engine backend, seeSetting up an external Application Load Balancer with aCompute Engine backend.

Backend service scope

The following table indicates which backend service resource and scope is usedby external Application Load Balancers:

Protocol to the backends

Backend services for Application Load Balancers must use one of the followingprotocols to send requests to backends:

• HTTP, which uses HTTP/1.1 and no TLS
• HTTPS, which uses HTTP/1.1 and TLS
• HTTP/2, which uses HTTP/2 and TLS (HTTP/2 without encryption isn'tsupported.)
• H2C, which uses HTTP/2 over TCP. TLS isn't required. H2C isn't supportedfor classic Application Load Balancers.

The load balancer only uses the backend service protocol that you specify tocommunicate with its backends. The load balancer doesn't fall back to adifferent protocol if it is unable to communicate with backends using thespecified backend service protocol.

The backend service protocol doesn't need to match the protocol used by clientsto communicate with the load balancer. For example, clients can send requeststo the load balancer using HTTP/2, but the load balancer can communicate withbackends using HTTP/1.1 (HTTP or HTTPS).

Backend buckets

Backend buckets direct incoming traffic to Cloud Storage buckets. For anexample showing how to add a bucket to a external Application Load Balancer, seeSetting up aload balancer with backendbuckets. Forgeneral information about Cloud Storage, seeWhat isCloud Storage?

Backends

The following table specifies the backends and related features supported byexternal Application Load Balancers in each mode.

Backends and VPC networks

The restrictions on where backends can be located depend on the type ofload balancer.

For global external Application Load Balancer backends, the following applies:

• Backend instances (for instance groupbackends) and backend endpoints (for NEG backends) can be located in anyVPC networkwithin the same organization. The differentVPC networks don't need to be connected usingVPC Network Peering because GFEs communicate directlywith backends in their respective VPC networks.

• Cloud Storage buckets aren't associated with a VPCnetwork. They can be located in any project within the sameorganization.

  Global external Application Load Balancers also supportShared VPCenvironments where you can share VPC networks andtheir associated resources across projects. However, forglobal external Application Load Balancers, you don't need to configure Shared VPC to beable to reference backend services or backend buckets from other projects inyour organization.

For classic Application Load Balancer backends, the following applies:

• All backend instances from instance group backends and all backend endpointsfrom NEG backends must be locatedin thesame project. However, an instance group backend or a NEG can use a differentVPC network in that project. The different VPCnetworks don't need to be connected using VPC Network Peering because GFEscommunicate directly with backends in their respective VPCnetworks.

• Cloud Storage buckets aren't associated with a VPCnetwork. However, they must be located in the same project as the loadbalancer.

For regional external Application Load Balancer backends, the following applies:

• For instance groups, zonal NEGs, and hybrid connectivity NEGs, all backendsmust be located in the same project and region as the backend service.However, a load balancer can reference a backend that uses a differentVPC network in the same project as the backend service.Connectivity between the load balancer'sVPC network and the backend VPC networkcan be configured using either VPC Network Peering, Cloud VPNtunnels, Cloud Interconnect VLAN attachments, or a Network Connectivity Centerframework.

  Backend network definition

  • For zonal NEGs and hybrid NEGs, you explicitly specify theVPC network when you create the NEG.
  • For managed instance groups, the VPC network is defined inthe instance template.
  • For unmanaged instance groups, the instance group'sVPC network is set to match the VPC networkof thenic0 interface for the first VM added to the instance group.

  Backend network requirements

  Your backend's network must satisfyone of the following networkrequirements:

  • The backend's VPC network must exactly match theforwarding rule's VPC network.

  • The backend's VPC network must be connected to theforwarding rule's VPC network usingVPC Network Peering. You must configure subnet route exchanges toallow communication between the proxy-only subnet in the forwarding rule'sVPC network and the subnets used by the backend instancesor endpoints.

• Both the backend's VPC network and the forwarding rule'sVPC network must beVPCspokesattached to the sameNetwork Connectivity Centerhub.Import and export filters must allow communication between the proxy-onlysubnet in the forwarding rule's VPC network and the subnetsused by backend instances or endpoints.

• For all other backend types, all backends must be located in the sameVPC network and region.

  Regional external Application Load Balancers also support Shared VPCenvironments where you can share VPC networks and theirassociated resources across projects. If you want theregional external Application Load Balancer's backend service and backends to be in a differentproject from the forwarding rule, you need to configure the load balancer in aShared VPC environment with cross-project servicereferencing.

Backends and network interfaces

If you use instance group backends, packets are always delivered tonic0. Ifyou want to send packets to non-nic0 interfaces (eithervNICs orDynamic Network Interfaces), useNEG backends instead.

If you use zonal NEG backends, packets are sent to whatever network interface isrepresented by the endpoint in the NEG. The NEG endpoints must be in the sameVPC network as the NEG's explicitly defined VPCnetwork.

Health checks

Each backend service specifies a health check that periodically monitors thebackends' readiness to receive a connection from the load balancer. This reducesthe risk that requests might be sent to backends that can't service the request.Health checks don't check if the application itself is working.

For the health check probes, you must create an ingress allowfirewallrule that allows health check probes to reach your backendinstances. Typically, health check probes originate from Google's centralizedhealth checking mechanism.

Regional external Application Load Balancers that use hybrid NEG backends are an exception tothis rule because their health checks originate from the proxy-only subnetinstead. For details, see theHybrid NEGsoverview.

Health check protocol

Although it is not required and not always possible, it is a best practice touse a health check whose protocol matches theprotocol of the backendservice.For example, an HTTP/2 health check most accurately tests HTTP/2 connectivity tobackends. In contrast, regional external Application Load Balancers that use hybrid NEG backendsdon't support gRPC health checks. For the list of supported health checkprotocols, seeLoad balancingfeatures.

The following table specifies the scope of health checks supported byexternal Application Load Balancers in each mode.

For more information about health checks, see the following:

• Health checks overview
• Creating health checks

Firewall rules

The load balancer requires the following firewall rules:

• For the global external Application Load Balancers, an ingress allowrule to permit traffic fromGoogle Front Ends (GFEs) to reach your backends. For theregional external Application Load Balancer, an ingress allow rule to permit trafficfrom theproxy-only subnet to reach your backends.
• An ingress allow rule to permit traffic from the health check probes ranges.For more information about health check probes and why it's necessary toallow traffic from them, seeProbe IP ranges and firewallrules.

Firewall rules are implemented at the VM instance level, not on GFE proxies. Youcannot use Google Cloud firewall rules to prevent traffic from reachingthe load balancer. For the global external Application Load Balancer and the classic Application Load Balancer, you can useGoogle Cloud Armor to achieve this.

The ports for these firewall rules must be configured as follows:

• Allow traffic to the destination port for each backend service's health check.

• For instance group backends: Determine the ports to be configured by themapping between the backend service'snamedport andthe port numbers associated with that named port on each instance group. Theport numbers can vary among instance groups assigned to the same backendservice.

• ForGCE_VM_IP_PORT NEG backends: Allow traffic to the port numbers of theendpoints.

The following table summarizes the required source IP address ranges for thefirewall rules:

• External Gateways created using theGKE Gatewaycontroller can use any mode ofan external Application Load Balancer. You control the load balancer's mode by choosing aGatewayClass. TheGKE Gateway controller always usesGCE_VM_IP_PORT zonal NEGbackends.

• External Ingresses created using theGKE Ingresscontroller are alwaysclassic Application Load Balancers. The GKE Ingress controllerprefers to useGCE_VM_IP_PORT zonal NEG backends, though instance groupbackends are also supported.

• You can useGCE_VM_IP_PORT zonal NEG created and managed byGKE Services as backends for any Application Load Balancer orProxy Network Load Balancer. For more information, seeContainer-native loadbalancing through standalone zonalNEGs.

Shared VPC architecture

External Application Load Balancers support networks that use Shared VPC.Shared VPC lets organizations connect resources from multiple projectsto a common VPC network so that they can communicate with eachother securely and efficiently by using internal IP addresses from that network.If you're not already familiar with Shared VPC, read theShared VPC overview.

There are many ways to configure an external Application Load Balancer within a Shared VPCnetwork. Regardless of type of deployment, all the components of the loadbalancer must be in the same organization.

While you can create all the load balancing components and backends in theShared VPC host project, this type of deployment doesn't separatenetwork administration and service development responsibilities.

All load balancer components and backends in a service project

The following architecture diagram shows a standard Shared VPCdeployment where all load balancer components and backends are in a serviceproject. This deployment type is supported by all Application Load Balancers.

The load balancer components and backends must use the same VPCnetwork.

Serverless backends in a Shared VPC environment

For a load balancer that is using a serverless NEG backend, the backendCloud Run or Cloud Run functions service must be in thesame project as the serverless NEG.

Additionally, for the regional external Application Load Balancer that supports cross-projectservice referencing, the backend service, serverless NEG, and theCloud Run service must always be in the same service project.

Cross-project service referencing

Cross-project service referencing is a deployment model where the loadbalancer's frontend and URL map are in one project and the load balancer'sbackend service and backends are in a different project.

Cross-project service referencing lets organizations configure one centralload balancer and route traffic to hundreds of services distributed acrossmultiple different projects. You can centrally manage all traffic routing rulesand policies in one URL map. You can also associate the load balancer with asingle set of hostnames and SSL certificates. You can therefore optimize thenumber of load balancers needed to deploy your application, and lowermanageability, operational costs, and quota requirements.

By having different projects for each of your functional teams, you can alsoachieve separation of roles within your organization. Service owners can focuson building services in service projects, while network teams can provision andmaintain load balancers in another project, and both can be connected by usingcross-project service referencing.

Service owners can maintain autonomy over the exposure of their services andcontrol which users can access their services by using the load balancer. This isachieved by a special IAM role called theCompute Load Balancer Services User role(roles/compute.loadBalancerServiceUser).

Cross-project service referencing support differs based on the type ofload balancer:

• For global external Application Load Balancers: your load balancer's frontend and URL map canreference backend services or backend buckets from any project within the sameorganization. No VPC network restrictions apply. While you canuse a Shared VPC environment to configure a cross-project deploymentas shown in thisexample, thisisn't a requirement.

• For regional external Application Load Balancers: you must create the load balancer in aShared VPC environment. The load balancer's frontend and URL map mustbe in a host or service project, and the load balancer's backend services andbackends can be distributed across host or service projects in the sameShared VPC environment.

To learn how to configure Shared VPC for aregional external Application Load Balancer—with and without cross-project servicereferencing—seeSet up a regional external Application Load Balancer withShared VPC.

Usage notes for cross-project service referencing

• Cross-project service referencing can be used with instance groups, serverless NEGs, and most other supported backend types. However, the following limitations apply:

  • With global external Application Load Balancers, you can't reference a cross-project backend service if the backend service has serverless NEG backends with App Engine.

  • With regional external Application Load Balancers, you can't reference a cross-project backend service if the backend service has regional internet NEG backends.
• Cross-project service referencing isnot supported for the classic Application Load Balancer.
• Google Cloud doesn't differentiate between resources (for example, backend services) using the same name across multiple projects. Therefore, when you are using cross-project service referencing, we recommend that you use unique backend service names across projects within your organization.
• If you see an error such as "Cross-project references for this resource are not allowed", make sure that you have the permission to use the resource. An administrator of the project that owns the resource must grant you theCompute Load Balancer Services User role (roles/compute.loadBalancerServiceUser). This role can be granted at the project level or at the resource level. For an example, seeGrant permissions to the Compute Load Balancer Admin to use the backend service or backend bucket.

Example 1: Load balancer frontend and backend in different service projects

Here is an example of a Shared VPC deployment where the load balancer'sfrontend and URL map are created in service project A and the URL map referencesa backend service in service project B.

In this case, Network Admins or Load Balancer Admins in service project Arequire access to backend services in service project B. Service project Badmins grant the Compute Load Balancer Services User role(roles/compute.loadBalancerServiceUser) to Load Balancer Admins inservice project A who want to reference the backendservice in service project B.

Example 2: Load balancer frontend in the host project and backends in service projects

Here is an example of a Shared VPC deployment where the load balancer'sfrontend and URL map are created in the host project and the backend services(and backends) are created in service projects.

In this case, Network Admins or Load Balancer Admins in the host projectrequire access to backend services in the service project. Service projectadmins grant the Compute Load Balancer Services User role(roles/compute.loadBalancerServiceUser) toto Load Balancer Admins in the host project A who want to reference the backendservice in the service project.

Example 3: Load balancer frontend and backends in different projects

Here is an example of a deployment where the global external Application Load Balancer's frontendand URL map are created in a different project from the load balancer's backendservice and backends. This type of deployment doesn't use Shared VPCand is supported only for global external Application Load Balancers.

To learn more about this setup, seeSet up cross-project service referencing with a backend service and a backend bucket.

High availability and failover

High availability and failover in external Application Load Balancers can be configured at the loadbalancer level. This is handled by creating backup regional external Application Load Balancers inany region where you require backup.

The following table describes the failover behavior.

HTTP/2 support

HTTP/2 is a major revision of the HTTP/1 protocol. There are 2 modes of HTTP/2support:

• HTTP/2 over TLS
• Cleartext HTTP/2 over TCP

HTTP/2 over TLS

HTTP/2 over TLS is supported for connections between clients and theexternal Application Load Balancer, and for connections between the load balancer and its backends.

The load balancer automatically negotiates HTTP/2 with clients as part of theTLS handshake by using the ALPN TLS extension. Even if a load balancer isconfigured to use HTTPS, modern clients default to HTTP/2. This is controlledon the client, not on the load balancer.

If a client doesn't support HTTP/2 and the load balancer is configured to useHTTP/2 between the load balancer and the backend instances, the load balancermight still negotiate an HTTPS connection or accept unsecured HTTP requests.Those HTTPS or HTTP requests are then transformed by the load balancer to proxythe requests over HTTP/2 to the backend instances.

To use HTTP/2 over TLS, you must enable TLS on your backends and set thebackend service protocol toHTTP2. Formore information, seeEncryption from the load balancer to thebackends.

HTTP/2 max concurrent streams

The HTTP/2SETTINGS_MAX_CONCURRENT_STREAMSsetting describes the maximum number of streams that an endpoint accepts,initiated by the peer. The value advertised by an HTTP/2 client to aGoogle Cloud load balancer is effectively meaningless because the loadbalancer doesn't initiate streams to the client.

In cases where the load balancer uses HTTP/2 to communicate with a server thatis running on a VM, the load balancer respects theSETTINGS_MAX_CONCURRENT_STREAMS value advertised by the server, up to amaximum value of100. In the request direction (Google Cloud loadbalancer → gRPC server), the load balancer uses the initialSETTINGS framefrom the gRPC server to determine how many streams per connection can be in usesimultaneously. If the server advertises a value higher than100, the loadbalancer uses 100 as the maximum number of concurrent streams. If a value ofzero is advertised, the load balancer can't forward requests to the server, andthis might result in errors.

HTTP/2 dynamic header table size

HTTP/2 significantly improves upon HTTP/1.1 with features like multiplexingand HPACK header compression. HPACK uses a dynamic table that enhances headercompression, making everything faster. To understand the impact of dynamicheader table size changes in HTTP/2, how this feature can improve performanceand how a specific bug in a various HTTP client libraries could cause issuesin HPACK header compression, refer to thecommunityarticle.

HTTP/2 limitations

• HTTP/2 between the load balancer and the instance can require significantlymore TCP connections to the instance than HTTP or HTTPS. Connection pooling,an optimization that reduces the number of these connections with HTTP orHTTPS, isn't available with HTTP/2. As a result, you might see high backendlatencies because backend connections are made more frequently.
• HTTP/2 between the load balancer and the backend doesn't support runningthe WebSocket Protocol over a single stream of an HTTP/2 connection (RFC8441).
• HTTP/2 between the load balancer and the backend doesn't support serverpush.
• The gRPC error rate and request volume aren't visible in theGoogle Cloud API or the Google Cloud console. If the gRPC endpointreturns an error, the load balancer logs and the monitoring data report the200 OK HTTP status code.

HTTP/2 over cleartext TCP

HTTP/2 over cleartext TCP, represented by the string "h2c" perRFC 7540,lets you use HTTP/2 without TLS encryption. It is supported for the followingconnections:

• Client to load balancer: Supported automatically; no special configurationis required.

  Important: gRPC over H2C is not supported for this connection.

• Load balancer to its backends: Supported by setting thebackend service protocol toH2C.

H2C support is also available for load balancers created using theGKE Gateway controller and Cloud Service Mesh,but isn't supported for classic Application Load Balancers.

HTTP/3 support

HTTP/3 is anext-generation internet protocol. It is built on top ofIETFQUIC, a protocoldeveloped from the originalGoogleQUIC protocol. HTTP/3 issupported between the external Application Load Balancer, Cloud CDN, and clients.

• IETF QUIC is a transport layer protocol that provides congestion control andreliability similar to TCP, uses TLS 1.3 for security, and improvedperformance.
• HTTP/3 is an application layer built on top of IETF QUIC, and it relies onQUIC to handle multiplexing, congestion control, loss detection, andretransmission.
• HTTP/3 allows faster client connection initiation, eliminates head-of-lineblocking in multiplexed streams, and supports connection migration when aclient's IP address changes.
• HTTP/3 is supported for connections between clients and the load balancer,not connections between the load balancer and its backends.
• HTTP/3 connections use theBBRcongestion control algorithm.

HTTP/3 on your load balancer can improve web page load times, reduce videorebuffering, and improve throughput on higher latency connections.

The following table specifies the HTTP/3 support for external Application Load Balancers in eachmode.

How HTTP/3 is negotiated

When HTTP/3 is enabled, the load balancer advertises this support to clients,allowing clients that support HTTP/3 to attempt to establish HTTP/3 connectionswith the HTTPS load balancer.

• Properly implemented clients always fall back to HTTPS or HTTP/2 when theycan't establish an HTTP/3 connection.
• Clients that support HTTP/3 use their cached prior knowledge of HTTP/3support to save unnecessary round trips in the future.
• Because of this fallback, enabling or disabling HTTP/3 in the load balancerdoesn't disrupt the load balancer's ability to connect to clients.

Support is advertised in theAlt-SvcHTTP response header. When HTTP/3 is enabled, responses from the load balancerinclude the followingalt-svc header value:

alt-svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000"

gcloud compute target-https-proxies createHTTPS_PROXY_NAME \    --global \    --quic-override=QUIC_SETTING

POST https://www.googleapis.com/v1/compute/projects/PROJECT_ID/global/targetHttpsProxies/TARGET_PROXY_NAME/setQuicOverride{  "quicOverride":QUIC_SETTING}

WebSocket support

Google Cloud HTTP(S)-based load balancers support the websocket protocolwhen you use HTTP or HTTPS as the protocol to the backend.The load balancer doesn't require any configuration to proxy websocketconnections.

The websocket protocol provides a full-duplex communication channel betweenclients and the load balancer. Formore information, seeRFC 6455.

The websocket protocol works as follows:

1. The load balancer recognizes a websocketUpgrade request froman HTTP or HTTPS client. The request contains theConnection: Upgrade andUpgrade: websocket headers, followed by other relevant websocket relatedrequest headers.
2. Backend sends a websocketUpgrade response. The backend instance sends a101 switching protocol response withConnection: Upgrade andUpgrade: websocket headers and other other websocket relatedresponse headers.
3. The load balancer proxies bidirectional traffic for the duration of thecurrent connection.

If the backend instance returns a status code426 or502,the load balancer closes the connection.

Session affinity for websockets works the same as for any other request.For more information, seeSessionaffinity.

gRPC support

gRPC is an open-source frameworkfor remote procedure calls. It is based on the HTTP/2 standard. Use cases forgRPC include the following:

• Low-latency, highly scalable, distributed systems
• Developing mobile clients that communicate with a cloud server
• Designing new protocols that must be accurate, efficient, andlanguage-independent
• Layered design to enable extension, authentication, and logging

To use gRPC with your Google Cloud applications, you must proxy requestsend-to-end over HTTP/2. To do this, you create an Application Load Balancer withone of the following configurations:

• HTTP/2 over TLS between the client and the load balancerand H2C between the load balancer and the backend: you create an HTTPS loadbalancer (configured with a target HTTPS proxy and SSL certificate).Additionally, you configurethe load balancer to use HTTP/2 for unencrypted connections between the loadbalancer and its backends by setting the backend service protocol toH2C.

• End-to-end encrypted traffic using HTTP/2 over TLS: you create an HTTPS loadbalancer (configured with a target HTTPS proxy and SSL certificate). The loadbalancer negotiates HTTP/2 with clients as part of the SSL handshake by usingthe ALPN TLS extension.

  Additionally, you must make sure that thebackends can handle TLStraffic andconfigure the load balancer to use HTTP/2 for encrypted connections betweenthe load balancer and its backends by setting thebackend serviceprotocol toHTTP2.

  The load balancer might still negotiate HTTPS with some clients oraccept unsecured HTTP requests on a load balancer that is configured to useHTTP/2 between the load balancer and the backend instances. Those HTTP orHTTPS requests are transformed by the load balancer to proxy the requests overHTTP/2 to the backend instances.

If you want to configure an Application Load Balancer by using HTTP/2 withGoogle Kubernetes Engine Ingress or by using gRPC and HTTP/2 with Ingress, seeHTTP/2for load balancing with Ingress.

If you want to configure an external Application Load Balancer by using HTTP/2 withCloud Run,seeUse HTTP/2 behind a load balancer.

For information about troubleshooting problems with HTTP/2, seeTroubleshootingissues with HTTP/2 to the backends.

For information about HTTP/2 limitations, seeHTTP/2limitations.

TLS support

By default, an HTTPS target proxy accepts only TLS 1.0, 1.1, 1.2, and 1.3 whenterminating client SSL requests.

When the classic Application Load Balancer uses HTTPS as the backend service protocol, itcan negotiate TLS 1.0, 1.1, 1.2, or 1.3 to the backend.

Mutual TLS support

Mutual TLS, or mTLS, is an industry standard protocol for mutual authenticationbetween a client and a server. mTLS helps ensure that both the client and serverauthenticate each other by verifying that each holds a valid certificate issuedby a trusted certificate authority (CA). Unlike standard TLS, where only theserver is authenticated, mTLS requires both the client and server to presentcertificates, confirming the identities of both parties before communication isestablished.

All of the Application Load Balancers support mTLS. With mTLS, the load balancerrequests that the client send a certificate to authenticate itself during theTLS handshake with the load balancer. You can configure aCertificate Manager truststore that the load balancer then uses to validate the client certificate'schain of trust.

For more information about mTLS, seeMutual TLSauthentication.

TLS 1.3 early data support

TLS 1.3 was defined inRFC8446 and introduces the conceptofearly data, also known aszero-round-trip time (0-RTT) data, which can improve application performancefor resumed connections by 30 to 50%.

With TLS 1.2, two round trips are required before data can be securelytransmitted. TLS 1.3 reduces this to one round trip (1-RTT) for new connections,allowing clients to send application data immediately after the first serverresponse. Additionally, TLS 1.3 introduces the concept of early data for resumedsessions, enabling clients to send application data with the initialClientHello, thereby reducing the effective round-tip time tozero (0-RTT).TLS 1.3 early data allows the backend server to begin processing client databefore the handshake process with the client is complete, thereby reducinglatency; however, care must be taken to mitigate replay risks.

Because early data is sent before the handshake is complete, an attacker canattempt to capture and replay requests. To mitigate this, the backend servermust carefully control early data usage, limiting its use to idempotentrequests. HTTP methods that are intended to be idempotent but which mighttrigger nonidempotent changes—such as a GET request modifying adatabase—must not accept early data. In such cases, the backend servermust reject requests with the HTTPEarly-Data: 1 header by returning an HTTP425 Too Early status code.

Requests with early data have theHTTP Early-Data header set to a value of1, which indicates to the backend server that the request has been conveyed inTLS early data. It also indicates that the client understands theHTTP425 TooEarly status code.

TLS early data (0-RTT) modes

You can configure TLS early data using one of the following modes on the targetHTTPS proxy resource of the load balancer.

• STRICT. This enables TLS 1.3 early data for requests with safe HTTPmethods (GET, HEAD, OPTIONS, TRACE), and HTTP requests that don't have queryparameters. Requests that send early data containing nonidempotent HTTPmethods (such as POST or PUT) or with query parameters are rejected with anHTTP425 status code.

• PERMISSIVE. This enables TLS 1.3 early data for requests with safe HTTPmethods (GET, HEAD, OPTIONS, TRACE). This mode doesn't deny requeststhat include query parameters. The application owner must ensure thatearly data is safe to use for each request path, particularly forendpoints where request replay might cause unintended side effects, suchas logging or database updates triggered by GET requests.

• DISABLED. TLS 1.3 early data isn't advertised, and any (invalid)attempts to send early data are rejected. If your applications aren'tequipped to handle early data requests safely, you must disable earlydata. TLS 1.3 early data is disabled by default.

• UNRESTRICTED (not recommended for most workloads). This enables TLS 1.3early data for requests with any HTTP method including nonidempotentmethods, such as POST. This mode doesn't enforce any other limitations.This mode can be valuable for gRPC use cases. However, we don't recommendthis method unless you have evaluated your security posture and mitigatedthe risk of replay attacks using other mechanisms.

Configure TLS early data

To explicitly enable or disable TLS early data, do the following:

PATCH https://compute.googleapis.com/compute/v1/projects/{project}/global/targetHttpsProxies/TARGET_HTTPS_PROXY{    "tlsEarlyData":"TLS_EARLY_DATA_MODE",    "fingerprint": "FINGERPRINT"}

After you have configured TLS early data, you can issue requests from an HTTPclient that supports TLS early data. You can observe lower latency forresumed requests.

If a non-RFC-compliant client sends a request with a nonidempotent method orwith query parameters, the request is denied. You see an HTTP425 Early statuscode in the load balancer logs and the following HTTP response:

  HTTP/1.1 425 Too Early  Content-Type: text/html; charset=UTF-8  Referrer-Policy: no-referrer  Content-Length: 1558  Date: Thu, 03 Aug 2024 02:45:14 GMT  Connection: close  <!DOCTYPE html>  <html lang=en>  <title>Error 425 (Too Early)</title>  <p>The request was sent to the server too early, please retry. That's  all we know.</p>  </html>

Limitations

• HTTPS load balancers don't send aclose_notify closurealert when terminatingSSL connections. That is, the load balancer closes the TCP connectioninstead of performing an SSL shutdown.

• HTTPS load balancers support only lowercase characters in domains in acommon name (CN) attribute or a subject alternative name (SAN) attributeof the certificate. Certificates with uppercase characters in domains arereturned only when set as theprimarycertificate in thetarget proxy.

• HTTPS load balancers don't use the Server Name Indication (SNI) extensionwhen connecting to the backend, except for load balancers withInternet NEGbackends.For more information, seeEncryption from the load balancer to thebackends.

• Google Cloud doesn't guarantee that an underlying TCP connection canremain open for the entirety of the value of the backend service timeout.Client systems must implement retry logic instead of relying on a TCPconnection to be open for long periods of time.

• When you create a regional external Application Load Balancer in Premium Tier using theGoogle Cloud console, onlyregionssupporting Standard Tier are available in the Google Cloud console.For access to other regions, use either the gcloud CLI or theAPI.

• The GFE proxies used by global and classic Application Load Balancers don't supportearly200 OK responses that are sent before the request's POST payloadhas been fully proxied to the backend. Sending an early200 OK responsecauses the GFE to close the connection to the backend.

  Your backend must respond with100 Continue responses after the requestheaders are received, and then wait until the request's POST payload has beenfully proxied before responding with the final200 OK response code.

• When using regional external Application Load Balancers with Cloud Run ina Shared VPC environment, standalone VPC networks inservice projects can send traffic to any otherCloud Run services deployed in any other serviceprojects within the same Shared VPC environment. This is a knownissue.

• Cloud CDN lets you force an object or set of objects to beignored by the cache by requesting acacheinvalidation. When you're using aglobal external Application Load Balancer withShared VPC cross-project servicereferencing, by default, service project admins won't have therequired permissions to request cache invalidations. This is because cacheinvalidation is configured in the frontend project (that is, the projectthat has the forwarding rule, target proxy, and URL map of the loadbalancer). Thus, cache invalidations can only be issued by principals whohave the IAM roles for configuring load balancer related resources in thefrontend projects (for example, the Compute Network Admin role). Serviceadmins, who control provisioning of the backend services in a separateproject, need to work with the load balancer administrator of the frontendproject to issue cache invalidation for their cross-project services.

What's next

• To learn how external Application Load Balancers handle connections, route traffic,and maintain session affinity, seeRequest distribution forexternal Application Load Balancers.

• To learn how to deploy a global external Application Load Balancer, seeSetting up anexternal Application Load Balancer with a Compute Enginebackend.

• To learn how to deploy a regional external Application Load Balancer, seeSetting up aregional external Application Load Balancer with a Compute Enginebackend.

• If you are an existing user of the classic Application Load Balancer, make sure that youreviewMigrationoverview when you plan a new deployment with the global external Application Load Balancer.

• To learn how to automate your external Application Load Balancer setup with Terraform, seeTerraform module examples forexternal Application Load Balancers.

• To learn how to configure advanced traffic management capabilities availablewith the global external Application Load Balancer, seeTraffic management overview forglobal external Application Load Balancers.

• To learn how to configure advanced traffic management capabilitiesavailable with the regional external Application Load Balancer, seeTraffic managementoverview for regional external Application Load Balancers.

• To learn about serving websites, seeServingwebsites.

• To find the locations for Google PoPs, seeGFE locations.

• To learn about capacity management, seeCapacity management with loadbalancingtutorialandApplication capacity optimizations with global loadbalancing.

• To learn how to use Certificate Manager to provision andmanage SSL certificates, see theCertificate Manageroverview.

• To insert custom logic into the load balancing data path, configureService Extensions plugins or callouts.

• For regional external Application Load Balancers only, you can try usingApigee Shadow APIDiscovery to findshadow APIs (also known as undocumented APIs) in your existingGoogle Cloud infrastructure. Make sure that you read the associatedlimitationsbefore you enable Shadow API Discovery.