This page describes how failover works for external Application Load Balancers. The failoverconfiguration involves two load balancers: a primary load balancer and a backupload balancer. For the purpose of this discussion, theprimary load balanceris the load balancer for which you want to configure failover. Thebackup loadbalancer is the load balancer that receives connections when the primary loadbalancer starts failing health checks.

Failover and failback are the automatic processes that route traffic to and froma load balancer. When Cloud DNS detects an outage and routes traffic from theprimary load balancer to the backup load balancer, the process is calledfailover. When Cloud DNS reverses this and redirects traffic to theprimary load balancer, the process is calledfailback.

How failover works

Global to regional failover for external Application Load Balancers is handled by creating two ormore regional external Application Load Balancers in the regions where you want the traffic tofailover to. Only regional external Application Load Balancers can be used as backup load balancers.Regional external Application Load Balancers are not only self-contained within individualGoogle Cloud regions, they are also isolated from anyglobal external Application Load Balancer or classic Application Load Balancer infrastructure running in thesame region.

Regional external Application Load Balancers work best as failover load balancers forglobal external Application Load Balancers because they are both based on Envoy proxies and processtraffic in very similar ways. This is in contrast to a classic Application Load Balancerthat hasnotable differences in how traffic ishandled.

In summary, the following failover scenarios are supported:

• From a global external Application Load Balancer to a regional external Application Load Balancer
• From a regional external Application Load Balancer to a regional external Application Load Balancer
• From a classic Application Load Balancer to a regional external Application Load Balancer

Important: The failover configuration described on this page is anactive-passivefailover configuration in which traffic fails over to a backup load balancer onlywhen the primary load balancer is unavailable. This is achieved by using aCloud DNSfailover routing policy.Regional external Application Load Balancers also support anactive-active configuration,where you can deploy multiple load balancers in different regions that are allserving traffic simultaneously. This can be configured by using aCloud DNSgeolocation routingpolicy. For details, seeHigh availability forregional external Application Load Balancers.

Failover and failback workflow

The following setup demonstrates failover from a global external Application Load Balancer to tworegional external Application Load Balancers, with one in each region where the global load balancerhas deployed backends.

The following sections describe a typical workflow with the different componentsinvolved in a failover configuration.

1. Detect failures in the primary load balancer

   Google Cloud useshealthchecks to detect whether yourprimary external Application Load Balancer is healthy. You configure these health checks to sendprobes fromthree source regions. These three source regions must berepresentative of the regions from where your clients will access the loadbalancer. For example, if you have a global external Application Load Balancer and most of yourclient traffic originates from North America and Europe, you can configureprobes originating from two regions in North America and one region in Europe.

   If health checks originating from two or more of these regions fail, thistriggers failover to the backup regional external Application Load Balancer.

   Additional notes:

   • You must specify exactly three source regions when youcreate the healthcheck. Only global health checks can specify sourceregions.
   • HTTP, HTTPS, and TCP health checks are supported.
   • The health check probes actually originate from a Point of Presence (PoP) onthe internet within some small distance of the configured Google Cloudsource region.

2. Route traffic to backup load balancers

   If the primary load balancer starts failing health checks, Google Clouduses Cloud DNS failover routing policies to determine how to routetraffic to the backup load balancers.

   The duration of the outage, or the time it takes for traffic to failover fromthe primary to backup load balancers, is determined by the DNS TTL value, thehealth check interval, and thehealth check's unhealthythreshold. Forrecommended settings, seeBest practices.

3. Failback to the primary load balancer

   After the health checks start passing again, failback to the primary loadbalancer is automatic. There is no downtime expected during failback becauseboth the backup and primary load balancers are serving traffic.

4. Test failover periodically

   We recommend that you periodically test the failover workflow as part of yourbusiness continuity plan. Make sure to test both gradual and instantaneousshifts in traffic from primary to backup load balancers. After verifying thatfailover works, trigger a failback to verify that traffic is rerouted back tothe primary load balancer as expected.

Configure failover

To configure failover, perform the following steps:

1. Review your existing primary load balancer configuration andcheck that the features (such as security features, traffic management androuting features, and CDN) used by the primary load balancer areavailable with the backup regional external Application Load Balancer. If similar features arenot available, then this load balancer might not be a good candidate forfailover.
2. Create the backup regional external Application Load Balancer with a configurationthat mirrors the primary load balancer as much as possible.
3. Create the health check and the DNS routingpolicy to detect outages and route traffic from theprimary to the backup load balancer during failover.

Review primary load balancer configuration

Before you begin, verify that the backup regional external Application Load Balancer supports allfeatures currently used with your primary load balancer.

To avoid traffic disruption, review the following differences:

• GKE deployments. GKE users shouldnote that load balancers deployed using GKE Gateway aremore compatible with this failover mechanism than load balancers deployedusing the GKE Ingress controller. This is becauseGKE Gateway supports configuration of both the global andregional external Application Load Balancers. However, the GKE Ingresscontroller supports only the classic Application Load Balancer.

  For best results, use GKE Gateway to deploy both theprimary and backup load balancers.

• Cloud CDN. Regional external Application Load Balancers don't supportCloud CDN. Therefore, in the event of afailure, any operations relying on Cloud CDN are also affected.For better redundancy, we recommend configuring a third-party CDN solutionthat can act as a fallback to Cloud CDN.

• Cloud Armor. If you use Cloud Armor for yourprimary load balancer, make sure that you also configure the sameCloud Armor features when configuring the backupregional external Application Load Balancer. Cloud Armor has different features availablein the regional versus the global scope. For more information, see the followingsections in the Cloud Armor documentation:

  • Regional Cloud Armor security policies
  • Global Cloud Armor security policies

• SSL certificates. If you want to use a common SSL certificate for both theprimary and backup load balancers, confirm that the type of SSLcertificate being used with the primary load balancer is compatible with thebackup regional external Application Load Balancer. Review the differences between the SSLcertificates available with global, regional, and classic load balancers.For details, see the following sections:

  • Compute Engine SSL certificates
  • Certificate Manager SSL certificates

• Backend buckets. Regional external Application Load Balancers don't supportCloud Storage buckets as backends. You can't set up failover forload balancers using backend buckets.

Configure the backup load balancer

The backup load balancer is a regional external Application Load Balancer that you configure in theregion where you want traffic to be redirected in the event of a failure.

Note the following considerations as you configure your backup load balancer:

• You must configure the features of the backup regional external Application Load Balancer to be assimilar as possible to the primary load balancer so that traffic is processedsimilarly across both deployments.

  • Global external Application Load Balancer. The regional external Application Load Balancers support mostof the same features as the global external Application Load Balancers, witha fewexceptions. The regional load balancer also supports the sameadvanced traffic management capabilities as the global load balancer, whichmakes it easier to achieve equivalence between the primary and backup loadbalancers.

  • Classic Application Load Balancer. With the classic Application Load Balancer, featureparity between the primary and backup load balancer is harder to achievebecause the regional external Application Load Balancer is an Envoy-based load balancer thatprocesses traffic differently. Make sure you test the failover and failbackthoroughly before deploying to production.

  To view the specific capabilities of the regional, global, and classicApplication Load Balancers, see theLoad balancer feature comparisonpage.

  We recommend that you use an automation framework such as Terraform to helpachieve and maintain consistency in load balancer configurations across bothprimary and backup deployments.

• We recommend that you set up backup regional external Application Load Balancers in every regionwhere you have backends. For example, if you fail over from a global deploymentwith instance groups in five regions to backup regional load balancers inthree regions only, you risk overloading your backend services in these threeregions while backend services in the remaining two regions are idle.

  Additionally, we recommend that you configureCloud DNS to useweighted round robinpolicies when rerouting failovertraffic from a primary global load balancer to these backup regional loadbalancers. Assign weights to each backup load balancer by taking into accountthe maximum sizes of the backend instance groups in different regions.

• Regional external Application Load Balancers support both Premium and StandardNetwork Service Tiers. If latency is not your primary concern during failover,we recommend that you set up the backup regional external Application Load Balancers usingStandard Tier. Using Standard Tier infrastructure offers additional isolationfrom the Premium Tier infrastructure used by global external Application Load Balancers.

• If you want to use the same backends for both primary and backup loadbalancers, you create the backup regional external Application Load Balancer in the regionwhere the backends are located. If you've enabled autoscaling for the backendinstance groups, you must fulfil therequirements for sharingbackends across deployments.

• If needed, reserve additional Envoy proxies for regional external Application Load Balancers tohelp ensure that, in case of a failover event, the additional traffic doesn'tdisrupt any other load balancer deployments in the same region. For details,seeReserve additional proxy-only subnet capacity.

To learn how to configure a regional external Application Load Balancer, seeSet up aregional external Application Load Balancer with VM instance groupbackends.

Reserve additional proxy-only subnet capacity

All regional Envoy-based load balancers in a region and VPCnetwork share the same pool of Envoy proxies. In a failover event, the backupregional external Application Load Balancers see an increase in proxy usage to handle failovertraffic from the primary load balancer. To help ensure that capacity is alwaysavailable for the backup load balancers, we recommend that you review the sizeof yourproxy-only subnet. Werecommend that you calculate the estimated number of proxies needed to handletraffic in a given region and increase capacity if needed. This also helpsensure that failover events don't disrupt other regional Envoy-based loadbalancers in the same region and network.

Generally, a regional external Application Load Balancer proxy can manage up to:

• 600 (HTTP) or 150 (HTTPS) new connections per second
• 3,000 active connections
• 1,400 requests per second

If you're using DNS policies to split traffic across multiple backup loadbalancers in different regions, you must take that into account when estimatingproxy requirements per region and network. A larger proxy-only subnet letsGoogle Cloud assign a larger number of Envoy proxies to your load balancerwhen necessary.

You can't expand a proxy-only subnet in the same way that you would for aprimary address range (with theexpand-ip-rangecommand). Instead, you mustcreate a backup proxy-only subnet that meets your needs and then promote it tothe active role.

To learn how to change the size of your proxy-only subnet,seeChange the size or address range of a proxy-onlysubnet.

Sharing backends between primary and backup load balancers

To achieve complete infrastructural redundancy, you must introduce redundancy atboth the load balancer level and at the backend level. That means that you mustconfigure your backup regional external Application Load Balancers with backends (instance groups ornetwork endpoint groups) that have no overlap with the primary load balancers.

However, if youdo want to share a backend instance group between the primaryand secondary load balancers,and autoscaling is enabled for the instancegroups, you must fulfill the following requirements to help ensure that properfailover occurs:

• The autoscaler must be set up with CPU-based scaling only. The loadbalancer's utilization-based scaling method is not supported.
• Both the global and regional backend services must use only theUTILIZATIONbalancing mode. Using theRATE balancing mode is not recommended becauseyour instances could receive 2x traffic from both global and regional loadbalancers during the failover process.
• Scale-in controls must be configured to prevent the autoscaler fromprematurely scaling down the group during downtime when traffic is switchingover from the global load balancer to the regional load balancer. Thisdowntime can be as high as the sum of DNS TTL plus the health check intervalconfigured.

Failure to set up autoscaling correctly can result in a secondary outageduring failover because the loss of traffic from the global load balancercauses the instance group to rapidly shrink.

Configure Cloud DNS and health checks

This section describes how to useCloud DNS andGoogle Cloud health checksto configure your Cloud Load Balancing environment to detect outages and routetraffic to the backup load balancers.

Use the following steps to configure the required health check and routingpolicies:

1. Create a health check for the primary load balancer's forwarding rule IPaddress.

   gcloud compute health-checks create httpHEALTH_CHECK_NAME \    --global \    --source-regions=SOURCE_REGION_1,SOURCE_REGION_2,SOURCE_REGION_3 \    --use-serving-port \    --check-interval=HEALTH_CHECK_INTERVAL \    --healthy-threshold=HEALTHY_THRESHOLD \    --unhealthy-threshold=UNHEALTHY_THRESHOLD \    --request-path=REQUEST_PATH

   Replace the following:

   • HEALTH_CHECK_NAME: the name of the health check
   • SOURCE_REGION: the three Google Cloudregions from which health checks are performed. You must specifyexactly three source regions.
   • HEALTH_CHECK_INTERVAL: the amount of time in secondsfrom the start of one probe issued by one prober to the start of the nextprobe issued by the same prober. The minimum supported value is 30 seconds.For recommended values, seeBest practices.
   • HEALTHY_THRESHOLD andUNHEALTHY_THRESHOLD: the number of sequentialprobes that must succeed or fail for the VM instance to be consideredhealthy or unhealthy. If either is omitted, Google Cloud uses adefault threshold of 2.
   • REQUEST_PATH: the URL path to whichGoogle Cloud sends health check probe requests.If omitted, Google Cloud sends probe requests to the root path, /.If the endpoints being health-checked are private, which is not typical forexternal forwarding rule IP addresses, you can set this path to/afhealthz.

2. Create a Cloud DNS record set in Cloud DNS and apply arouting policy to it. The routing policy must be configured to resolve yourdomain name to either the primary load balancer's forwarding rule IP address,or, in the event of a health check failure, to one of the backup loadbalancers' forwarding rule IP addresses.

   gcloud dns record-sets createDNS_RECORD_SET_NAME \    --ttl=TIME_TO_LIVE \    --type=RECORD_TYPE \    --zone="MANAGED_ZONE_NAME" \    --routing-policy-type=FAILOVER \    --routing-policy-primary-data=PRIMARY_LOAD_BALANCER_FORWARDING_RULE \    --routing-policy-backup-data_type=GEO \    --routing-policy-backup-data="BACKUP_REGION_1=BACKUP_LOAD_BALANCER_1_IP_ADDRESS[;BACKUP_REGION_2=BACKUP_LOAD_BALANCER_2_IP_ADDRESS;BACKUP_REGION_3=BACKUP_LOAD_BALANCER_3_IP_ADDRESS]" \    --health-check=HEALTH_CHECK_NAME \    --backup-data-trickle-ratio=BACKUP_DATA_TRICKLE_RATIO

   Replace the following:

   • DNS_RECORD_SET_NAME: the DNS or domain name of therecord set to add—for example,test.example.com
   • TIME_TO_LIVE: the time to live (TTL) for the recordset in number of seconds. For recommended values, seeBestpractices.
   • RECORD_TYPE: therecordtype—for example,A
   • MANAGED_ZONE_NAME: the name of the managed zone whoserecord sets you want to manage—for example,my-zone-name
   • PRIMARY_LOAD_BALANCER_FORWARDING_RULE: the forwarding rulename of the primary load balancer
   • BACKUP_REGION: the regionswhere the backup load balancers are deployed
   • BACKUP_LOAD_BALANCER_IP_ADDRESS: the forwarding ruleIP addresses of the backup load balancers corresponding to each region
   • BACKUP_DATA_TRICKLE_RATIO: the ratio of traffic tosend to the backup load balancers, even when the primary load balancer ishealthy. The ratio must be between 0 and 1, such as 0.1. The default isset to 0.

Best practices

Here are some best practices to keep in mind when you configure theCloud DNS record and health checks:

• The time it takes for traffic to fail over from primary to backup loadbalancers (that is, the duration of the outage) depends on the DNS TTL value,the health check interval, and thehealth check's unhealthythreshold parameter.

  With Google's Cloud DNS, the upper bound for this period can becalculated using the following formula:

  Duration of outage = DNS TTL + Health Check Interval * Unhealthy Threshold

  For failover configuration, we recommend setting the DNS TTL to 30-60seconds. Higher TTLs lead to longer downtimes because clients on the internetcontinue to access the primary external Application Load Balancers even after DNS has failed overto the backup regional external Application Load Balancer.

• Configure thehealthy andunhealthy threshold parameters in the healthchecks such that youavoid unnecessary failovers caused by transient errors. Higher thresholdsincrease the time it takes for traffic to fail over to backup load balancers.

• To help ensure that your failover setup works as expected, you can set up yourDNS routing policy to always send a small percentage of traffic to the backupload balancers even when the primary load balancers are healthy. This can bedone by using the--backup-data-trickle-ratio parameter when you create theDNS record set.

  You can configure the percentage of the traffic sent to the backup as afraction from 0 to 1. The typical value is 0.1, although Cloud DNSlets you send 100 percent of the traffic to the backup VIP addresses, tomanually trigger a failover.