Cloud Load Balancing supports proxying traffic to external backendsoutside Google Cloud. To define an external backend for a load balancer, youuse a resource called an internet network endpoint group (NEG).

You can use this type of deployment when you want to serve content from anexternal backend, but you want your Google Cloud load balancer to bethe frontend. This lets you do the following:

• Use Google Edge infrastructure for terminating your user connections.
• Direct the connections to your external backend.
• Deliver traffic to your public endpoint across Google's private backbone,which improves reliability and can decrease latency between client and server.
• With global load balancers, you can useCloud CDN (Content Delivery Network) to cache content for your externalbackend.

Figure 1 shows an external Application Load Balancer with multiple backendtypes, one of which is an external backend configured with an internet NEG.

Internet NEG backends are supported with various global and regional loadbalancers. Depending on your load balancer (global or regional), internet NEGsupport differs with respect to DNS, health checking, available number ofendpoints, and traffic routing behaviors.

The sections that follow explain how external backends are used withCloud Load Balancing. If you want to use an external backend withCloud Service Mesh, seeCloud Service Mesh with internet network endpointgroups.

Note: Cloud Load Balancing also supports connecting to backends on-premises orin other cloud environments outside Google Cloud by usinghybridNEGs. However, connections tohybrid NEG endpoints make use ofhybrid connectivity. Incontrast, connections to internet NEG endpoints use the internet while keepingyour traffic on Google's high performance backbone for the longest possibledistance, for reduced latency and improved throughput.

Terminology

The following terms are sometimes used interchangeably because they havethe same or similar meanings:

• external backend: A backend that resides outside of Google Cloud and isreachable across the internet. The endpoint in an internet NEG.
• custom origin: Same as anexternal backend. In CDN,origin is theindustry-standard term for a backend instance that serves web content.
• internet network endpoint group (NEG): The Google Cloud API resource that youuse to specify an external backend.
• external endpoint: Same as anexternal backend.

This document uses the termexternal backend except when referring to theinternet NEG API resource.

Load balancer components

This section describes the load balancing architecture and resources required toconfigure a load balancer with an external backend. The load balancer requiresspecial configuration only for the backend service. Thefrontend configuration is the same as any other load balancer.

The following figures show the Google Cloud resources required to set upa load balancer with an external backend.

You can only use internet NEGs on thePremium network servicetier.

Frontend configuration

No special frontend configuration is required for creating a load balancer withan internet NEG backend.Forwardingrules are used to route trafficby IP address, port, and protocol to a target proxy. Thetargetproxy then terminates connections fromclients. Additionally,Envoy-based loadbalancers require a proxy-only subnet.

Application Load Balancers also useURL mapsto set up URL-based routing of requests to theappropriate backend services.

For more details about each of these components, refer to the architecture sectionsof the respective load balancer:

• External Application Load Balancer overview
• Internal Application Load Balancer overview
• External proxy Network Load Balancer overview
• Internal proxy Network Load Balancer overview

Internet NEG

An internet NEG is a resource used to define an external backend for theload balancer. There are two types of internet NEGs: global internet NEG andregional internet NEG. They differ in the scope (global versus regional) andbehavior. The external backend referenced by a global internet NEG must bereachable exclusively over the internet; it can't be reachable overCloud VPN or Cloud Interconnect. If the external backendreferences a Google API or service, the service must be reachable throughTCP port80 or443 by using theHTTP,HTTPS, orHTTP/2 protocol.

There are two ways to configure the external endpoint referenced by the NEG:INTERNET_FQDN_PORT orINTERNET_IP_PORT. If theINTERNET_IP_PORT format ischosen, only a public internet routable IP address can be used; if theINTERNET_FQDN_PORT format is chosen the FQDN can be resolved to either apublic internet routable IP address or to a private IP address depending on thescope of the endpoint: regional or global.

Global internet NEGs are powered by Google Front End (GFE) technologies.They use afixed set of fixed IP addresses for egress trafficto all clients. They support only one endpoint per NEG and one internet NEG perbackend service.

The following table describes how different load balancers support globalinternet NEGs.

Load balancers	Endpoint type	Endpoint definition	Scope	Health checks
Global external Application Load Balancer Classic Application Load Balancer	INTERNET_FQDN_PORT	A publicly resolvable fully qualified domain name and an optional port. For example,backend.example.com:443†. The domain name must be resolvable byGoogle's public DNS infrastructure. Onlyone endpoint per NEG is allowed.	Global	Not supported
	INTERNET_IP_PORT	A publicly routable IP address and an optional port. For example,8.8.8.8:4431. The IP address can't be an RFC 1918 address. Onlyone endpoint per NEG is allowed.

Regional internet NEGs are powered by managed Envoy proxies. Each NEGcan have multiple endpoints, and a backend service can includemultiple internet NEGs.

For egress traffic, you can configure Cloud NAT gateways to set thesource IP addresses. Alternatively, you can route traffic using learned routeswithin your VPC network. While Cloud NAT isn't requiredfor this routing method, it is supported.

The following table describes how different load balancers support regionalinternet NEGs.

Load balancers	Endpoint type	Endpoint definition	Scope	Health checks
Regional external Application Load Balancer Regional internal Application Load Balancer Regional external proxy Network Load Balancer Regional internal proxy Network Load Balancer	INTERNET_FQDN_PORT	Either a publicly or privately resolvable fully qualified domain name and an optional port. For example,backend.example.com:443*. The domain name resolution process follows theCloud DNS name resolution order process. A maximum of 256 endpoints per NEG are allowed.	Regional	Envoy distributed health checks
	INTERNET_IP_PORT	Only a publicly routable IP address and an optional port. For example,8.8.8.8:4432. The IP address can't be an RFC 1918 address. A maximum of 256 endpoints per NEG are allowed.

Note: If you're planning a cross-cloud deployment with a regional internet NEG, youcan use Cloud Location Finder to identify the optimal region or zone for yourdeployment based on factors like distance, network latency, carbon footprint(Google CFE%), or the territory code (in case you have regulatory requirementsfor your network traffic). For details, see theCloud Location Finderdocumentation (Preview).

DNS resolution for regionalINTERNET_FQDN_PORT endpoints

If your domain is resolvable over the internet, no other configuration is neededto set up DNS. However, if you're resolving private FQDNs, you'll need toconfigure Cloud DNS to facilitate DNS resolution. The name must behosted onCloud DNS or be resolvable using DNS forwarding from Cloud DNSto an on-premises DNS or DNS peering if referencing a Private DNS zone inanother VPC network.

Start by creating aCloud DNSzone to host the DNS records in yourproject. Then add the DNS records to it. For specific configuration steps, seetheCloud DNS documentation. TheCloud DNS resolution order is detailed atName resolutionorder.

If you're using Shared VPC, note the specificnetworkrequirements. You can also use other featuresof Cloud DNS, such asforwardingzones, to fetch records from anon-premises DNS server.

IP address resolution for globalINTERNET_FQDN_PORT endpoints

When anINTERNET_FQDN_PORT endpoint points to a DNS record that returnsmultiple IP addresses, the IP address is resolved in the following way:

• The load balancer uses a DNS resolver in a Google Cloud region that isclosest to its client on the internet. If the DNS record for yourINTERNET_FQDN_PORT endpoint returns different IP addresses based on thelocation of the client, make sure that each of those IP addresses can bereached by the load balancer.

• The load balancer attempts to connect to the first IPaddress in the DNS response. If that IP address isn't reachable, the loadbalancer returns an HTTP502 (Bad Gateway) response. This is true even ifother IP addresses from the DNS response are available.

For more information about the IP ranges and locations used by Google's DNSresolver infrastructure, see theGoogle public DNSdocumentation.Names that can't be resolved by the public DNS system aren't usable asexternal backends.

IP address resolution for regionalINTERNET_FQDN_PORT endpoints

Regional internet NEGs support domain name resolution using bothCloud DNS and Google's public DNS.

• For Public DNS resolution, Cloud DNS forwards traffic to Google'spublic DNS servers.
• For Cloud DNS, the domain name must be one of the following:
  • Hosted on Cloud DNS
  • Be resolvable using DNS forwarding from Cloud DNS to anon-premises DNS server
  • Be resolvable using DNS peering if you are referencing a private DNS zonein another VPC network.

If the DNS server returns multiple IP addresses, Envoy load balances trafficamong the returned IP addresses based on the configured load balancing algorithm(round robin, least request, and so on). The list of endpoints is periodicallyrefreshed based on DNS TTL. You can configureretrypolicies to force Envoy toattempt to connect to another IP address if one fails.

Backend service

Backend services provide configurationinformation to the load balancer. Load balancers use the information in abackend service to direct incoming traffic to one or more attached backends.

To set up a load balancer with an external backend, you configure the backendservice with an internet NEG backend. When you add an internet NEG to a backendservice, the following considerations apply, depending on the scope of theNEG:

• The backend service can't also use other backend types (such as zonal NEGs orinstance groups) as backends.

• Number of NEGs per backend service

  • Global NEGs. You can add only one internet NEG backend to abackend service.
  • Regional NEGs. You can add up to50 internetNEGs to the same backendservice.

• Number of endpoints per NEG

  • Global NEGs. You can add only one endpoint to an internet NEG.

    Because only one endpoint is allowed in each global internet NEG, loadbalancing isn't actually performed. The load balancer serves as the frontendonly, and it proxies traffic to the specified external backend. This meansthat you can't use any of the load balancing modes, such as rate,connection, or utilization.

  • Regional NEGs. You can add up to256 endpoints perNEG to the same backendservice.

  Regional NEGs don't support load balancing modes, such as rate, connection,or utilization. All the endpoints of all the NEGs attached to a backendservice are pooled into a single group. Load balancing traffic among thispool of endpoints is handled using Envoy load balancing algorithms. For theload balancing policy algorithms supported, seelocalityLbPolicy in theregionalbackend service API documentation.

• Health checks

  • Global NEGs. The backend service can't reference ahealthcheck.
  • Regional NEGs. The load balancer usesdistributed Envoy healthchecks.

• The backend service's load balancing scheme must match the scheme required bythe load balancer you are deploying. For the complete list, seeBackendservices.

• The backend service protocol must be one ofHTTP,HTTPS, orHTTP2.

  We strongly recommend you use either HTTPS or HTTP/2 as the protocolwhen configuring a backend service with an internet NEG so that communicationbetween the load balancer and the backend is encrypted and authenticated whentransiting the public internet.

  Additionally, when using HTTPS or HTTP/2 as the backend protocol, make surethat you use anINTERNET_FQDN_PORT endpoint to create your external backend.This has two advantages:

  • It ensures that the load balancer validates the SSL server certificatepresented by the external backend and verifies that the followinginformation is true:

    • The certificate is signed by well-known certificate authorities (CAs).
    • The certificate isn't expired.
    • The certificate signature is valid.
    • The configured FQDN matches one of the Subject Alternative Names (SANs)in the certificate.

    If you create the external backend by using anINTERNET_IP_PORT endpoint,SSL server certificate validation isn't performed.

  • The SSL Server Name Indication (SNI) extension is only supported withINTERNET_FQDN_PORT endpoints. The configured FQDN issent an SNI in the client hello during the SSL handshake between the loadbalancer and the external endpoint. The SNI isn't sent when you use anINTERNET_IP_PORT endpoint because IP address literals aren't allowed intheHostName field of an SNI payload.

Health checks

Your health check configuration varies depending on the type of load balancer:

• Global external Application Load Balancer and classic Application Load Balancer. A backend servicewith a global internet NEG doesn't supporthealth checks.

  If your external backend becomes unreachable or if the configured hostname(FQDN) cannot be resolved, the load balancer returns an HTTP502 (BadGateway) response to its clients.

• Regional external Application Load Balancer, regional internal Application Load Balancer,regional external proxy Network Load Balancer, and regional internal proxy Network Load Balancer. Healthchecks areoptional. Health check probes for these load balancers originatefrom theproxy-only subnet and arethen NAT-translated (by using Cloud NAT) to either pre-reserved IPaddresses or auto-allocated NAT IP addresses. For details, seeRegional NEGs:Use a Cloud NAT gateway.

  Distributed Envoy health checks are created by using thesameGoogle Cloud console, gcloud CLI, and APIprocesses as centralized health checks.No other configuration is required.

  Points to note:

  • gRPC health checks are not supported.
  • Health checks with PROXY protocol v1 enabled are not supported.

  • Because the Envoy data plane handles health checks, you cannot use theGoogle Cloud console, the API, or the gcloud CLI to check thehealth status of these external endpoints. For hybrid NEGs with Envoy-basedload balancers, the Google Cloud console shows the health check status asN/A. This is expected.

  • Every Envoy proxy assigned to the proxy-only subnet in the region in theVPC network initiates health checks independently. Therefore,you might see an increase in network traffic because of health checking. Theincrease depends on the number of Envoy proxies assigned to yourVPC network in a region, the amount of traffic received bythese proxies, and the number of endpoints that each Envoy proxy needs tohealth check. In the worst case scenario, network traffic because of healthchecks increases at a quadratic(O(n^2)) rate.

  • Health check logs for distributed Envoy health checks don't includedetailed health states. For details about what is logged, seeHealth checklogging. To furthertroubleshoot connectivity from Envoy proxies to your NEG endpoints, youshould also check the respective load balancer logs.

    • Regional external Application Load Balancer logging
    • Internal Application Load Balancer logging
    • Proxy Network Load Balancer logging

Enable the external backend to receive requests

Configure your external backend to allow traffic from Google Cloud.

This procedure depends on the scope of the NEG: global or regional.

Global NEGs: Allowlist default Google egress IP addresses

If you're using a global internet NEG, youmust add the IP address ranges that Google uses to send requests toexternal backends to an allowlist. To look up the IP addresses to be added to anallowlist, query the_cloud-eoips.googleusercontent.com DNS TXT record byusing a tool likedig ornslookup.

For an example, seeAllow the external backend to receive traffic fromGoogle Cloud.

Regional NEGs: Use a Cloud NAT gateway

If you're using a regional internet NEG, you'll firstset up aCloud NAT gatewayto allocate a set of IP address ranges from where Google Cloud trafficshould originate.

The gateway endpoint should be of typeENDPOINT_TYPE_MANAGED_PROXY_LB.

The Cloud NAT gateway can be configured to either automaticallyallocate external IP addresses based on demand or use a manually pre-reservedset of external IP addresses.

• Automatically allocated IP addresses

  Use automatically allocated IP addresses if your external backendenvironment doesn't require you to add specific Google CloudIP addresses that can send traffic to the external backend to an allowlist.

• Manually allocated IP addresses

  Use manually allocated IP addresses only if your external backend environmentrequires you to add specific Google Cloud IP addresses to an allowlist.Because each Envoy assigned to your proxy subnet consumes an entireIP address, make sure that the pool of reserved IP addresses is big enoughto accommodate all Envoys.

  Warning: If you provision fewer NAT IP addresses than thenumber of assignedEnvoy proxies,requests sent to the internet NEG might result in HTTP 5xx errors.To ensure that you are informed when such an event occurs, set up an alert forthenat_allocation_failed metric.Contact support if you need help calculating the number of IP addresses thatmust be allocated for your load balancer in a specific region.

  If you experience connectivity issues at scale, check whether you've reachedtheCloud NAT limits. Bydefault, you are limited to 50 manually allocated NAT IP addresses pergateway.

Dynamic port allocation is supported for regional internet NEGs. IPaddresses can be shared by proxies, thus fully utilized.

This Cloud NAT configuration applies to the entire proxy-only subnet.Internet traffic associated with all theregional Envoy-based loadbalancers in the region sharethe same NAT gateway.

Cloud NAT usage incurs charges for both user traffic and health checktraffic. For details about how regional internet NEG pricing works, seeRegional internet NEG pricing.

NAT gateways configured on proxy-only subnets don't support logging andmonitoring. That is, the--enable-logging and--log-filter flags aren'tsupported.

Authenticate requests to the external backend

This section applies only to Application Load Balancers.

To authenticate requests sent to your external backend, youcan do one of the following:

• Set a custom header to indicate that the request camefrom a Google Cloud load balancer by using acustom requestheader.For example, you can use 16 or more cryptographically random bytes as ashared key.

  Implementing custom header transformations depends on the type of loadbalancer you're using:

  • Global external Application Load Balancer and classic Application Load Balancer. Custom headertransformations can be configured on either thebackendservice or theURLmap.

    For example, you can configure the external backend to expect a particularvalue for the HTTP request'sHost header, and you can configure the loadbalancer to set theHost header to that expected value. If you don'tconfigure a custom request header, the load balancer preserves the headersthat the client used to connect to the load balancer and includes the sameheader in its response. However, note that modifying theHost header isnot supported in the URL map.

    There are additional limitations associated with configuring theHost header. For details, seeCreate custom headers in backendservices. For a specificexample, seeSet up a global external Application Load Balancer with an externalbackend.

  • Regional external Application Load Balancer and regional internal Application Load Balancer. Custom headertransformations can only be configured on theURLmap.

    For these Envoy-based load balancers,Host andauthority are specialkeywords reserved by Google Cloud. You can't modify these headers forthese load balancers. Instead, we recommend that you create other customheaders (for example,MyHost) so that you don't interfere with thereserved header names.

• Enable IAP andverify that the signed JWTin the request header is signed by Google, and that theaud (audience) claimcontains the project number where your load balancer isdefined.

  Note the following:

  • IAP isn't compatible with Cloud CDN.
  • IAP isn't supported with proxy Network Load Balancers(internal and external).

• Enable private originauthentication, whichgives an external Application Load Balancer long-term access to a private Amazon Simple StorageService (Amazon S3) bucket or other compatible object stores.Cloud CDN (and therefore, private origin authentication) isn'tsupported with regional external Application Load Balancers and regional internal Application Load Balancers.

Logs

Requests proxied to an external backend are logged to Cloud Logging in thesame way that requests for other backends are logged.

For more information, see the following:

• Global external Application Load Balancer and classic Application Load Balancer logging andmonitoring

• Regional external Application Load Balancer logging
• Regional internal Application Load Balancer logging
• Proxy Network Load Balancer logging

If you enable Cloud CDN for an external Application Load Balancer using an externalbackend, cache hits are also logged.

Header processing with external backends

Different load balancers might handle header processing in different ways whenthey proxy requests to an external backend. Review the following information tounderstand how your load balancer type might process HTTP headers.

Global external Application Load Balancers and classic Application Load Balancers

When a global external Application Load Balancer or a classic Application Load Balancer proxies requests to anexternal backend, it adjusts HTTP headers in the following ways:

• Some headers are coalesced. When there are multiple instances of the sameheader key (for example,Via), the load balancer combines their valuesinto a single comma-separated list for a single header key. Only the headerswhose values can be represented as a comma-separated list are coalesced. Otherheaders, such asSet-Cookie, are never coalesced.

• Headers are proper-cased when the backend service's protocol isHTTP orHTTPS:

  • The first letter of the header's key and every letter following a hyphen(-) is capitalized to preserve compatibility with HTTP/1.1 clients. Forexample,user-agent is changed toUser-Agent, andcontent-encodingis changed toContent-Encoding.

  • Certain headers, such asAccept-CH (clienthints),are converted to match standard mixed letter representation.

• Some headers are added, or values are appended to them. The external Application Load Balancersalways add or modifycertain headerssuch asVia andX-Forwarded-For.

Regional external Application Load Balancers and regional internal Application Load Balancers

There is no special header processing for Envoy-based load balancers usinginternet NEGs. To learn how Envoy typically processes headers, seeHTTP headermanipulation.

Limitations

• Review thebackend service section forlimitations associated with configuring internet NEGs as backends.
• When you modify a load balancer to change the backend from an internet NEG toany other backend type, or change the backend from any other backend type toan internet NEG, your application experiences a temporary downtime of about30-90 seconds. For example, during this downtime, clients sending requests toglobal external Application Load Balancers see502 errors with the error codefailed_to_connect_to_backend. This is expected behavior.

• The following advanced traffic management features aren't supported withglobal internet NEG backends:
  • Request mirroring
  • Retry policies
  • Trafficpolicies(including load balancing locality policy, session affinity, andoutlier detection)

• Review theCloud NAT gateway section for limitationsassociated with NAT gateways configured on proxy-only subnets.

Quotas and limits

For information about quotas and limits, see theNEG backendsquota table and theEndpoints per NEGquota table.

Pricing

Egress traffic to external internet NEG endpoints is charged at internet egressrates forPremium Tier networking. Thesource is based on the client location, and the destination is based on thelocation of your public endpoint.

If you configured a Cloud NAT gateway to map your regional Envoy-basedload balancer's proxy-only subnet, you'll incur Cloud NAT charges.Cloud NAT gateways allocated for load balancers incur hourly chargesequivalent to a network with more than 32 VM instances. For details, seeCloud NAT pricing.

For more information, seeCloud Load Balancingpricing.

What's next

• Set up a global external Application Load Balancer with an internetNEG
• Set up a classic Application Load Balancer with an internetNEG
• Set up Cloud CDN for an external Application Load Balancer with an externalbackend
• Cloud Service Mesh with internet network endpointgroups overview

• Set up a regional external Application Load Balancer with an internetNEG
• Set up a regional external proxy Network Load Balancer with an internetNEG