Set up a cross-region internal Application Load Balancer with hybrid connectivity

This page shows how to deploy a cross-region internal Application Load Balancer to load balancetraffic to network endpoints that are on-premises or in other public clouds andthat are reachable by usinghybrid connectivity.

If you haven't already done so, review theHybrid connectivity NEGsoverview to understand thenetwork requirements to set up hybrid load balancing.

Setup overview

The example sets up a cross-region internal Application Load Balancer for mixed zonal and hybridconnectivity NEG backends, as shown in the following figure:

Cross-region internal Application Load Balancer example for mixed zonal and hybrid connectivity NEG backends.
Cross-region internal Application Load Balancer example for mixed zonal and hybrid connectivity NEG backends (click to enlarge).

You must configure hybrid connectivity before setting up a hybridload balancing deployment. Depending on your choice of hybrid connectivityproduct, use either Cloud VPN or Cloud Interconnect (Dedicatedor Partner).

Set up an SSL certificate resource

Create a Certificate Manager SSL certificate resource as described inthe following:

We recommend using a Google-managed certificate.

Permissions

To set up hybrid load balancing, you must have the following permissions:

  • On Google Cloud

    • Permissions to establish hybrid connectivity between Google Cloud andyour on-premises environment or other cloud environments. For the listof permissions needed, see the relevantNetwork Connectivity productdocumentation.
    • Permissions to create a hybrid connectivity NEG and the load balancer.TheCompute Load Balancer Adminrole(roles/compute.loadBalancerAdmin) contains the permissions required toperform the tasks described in this guide.

  • On your on-premises environment or other non-Google Cloud cloudenvironment

    • Permissions to configure network endpoints that allow services on youron-premises environment or other cloud environments to be reachable fromGoogle Cloud by using anIP:Port combination. For more information,contact your environment's network administrator.
    • Permissions to create firewall rules on your on-premises environment orother cloud environments to allow Google's health check probes to reach theendpoints.

Additionally, to complete the instructions on this page, you need to create ahybrid connectivity NEG, a load balancer, and zonal NEGs (and their endpoints)to serve as Google Cloud-based backends for the load balancer.

You should be either a projectOwneror Editor, or you should have thefollowingCompute Engine IAMroles.

TaskRequired role
Create networks, subnets, and load balancer componentsCompute Network Admin (roles/compute.networkAdmin)
Add and remove firewall rulesCompute Security Admin (roles/compute.securityAdmin)
Create instancesCompute Instance Admin (roles/compute.instanceAdmin)

Establish hybrid connectivity

Your Google Cloud and on-premises environment or other cloud environmentsmust be connected throughhybrid connectivity by usingeither Cloud Interconnect VLAN attachments or Cloud VPNtunnels with Cloud Router or Router appliance VMs. We recommend that youuse a high availability connection.

A Cloud Router enabled withglobal dynamicroutinglearns about the specific endpoint through Border Gateway Protocol (BGP) andprograms it into your Google Cloud VPC network. Regionaldynamic routing is not supported. Static routes are also not supported.

You can use either the same network or a different VPC networkwithin the same project to configure both hybrid networking(Cloud Interconnect or Cloud VPN or a Router appliance VM) and the load balancer. Notethe following:

  • If you use different VPC networks, the two networks must beconnected using either VPC Network Peering or they must beVPCspokeson the sameNetwork Connectivity Centerhub.

  • If you use the same VPC network, ensure that yourVPC network's subnet CIDR ranges don't conflict with yourremote CIDR ranges. When IP addresses overlap, subnet routes are prioritizedover remote connectivity.

For instructions, see the following documentation:

Important: Don't proceed with the instructions on this page until you set uphybrid connectivity between your environments.

Set up your environment that is outside Google Cloud

Perform the following steps to set up your on-premises environment or other cloudenvironment for hybrid load balancing:

  • Configure network endpoints to expose on-premises services toGoogle Cloud (IP:Port).
  • Configure firewall rules on your on-premises environment or other cloud environment.
  • Configure Cloud Router to advertise certain required routes to yourprivate environment.

Set up network endpoints

After you set up hybrid connectivity, you configure one or more networkendpoints within your on-premises environment or other cloud environments thatare reachable through Cloud Interconnect or Cloud VPN orRouter appliance by using anIP:port combination. ThisIP:portcombination is configured as one or more endpoints for the hybrid connectivityNEG that is created in Google Cloud later on in this process.

If there are multiple paths to the IP endpoint, routingfollows the behavior described in theCloud Routeroverview.

Set up firewall rules

The following firewall rules must be created on your on-premises environmentor other cloud environment:

  • Create an ingress allow firewall rule in on-premises or other cloud environments to allow traffic from the region'sproxy-only subnet to reach the endpoints.

  • Allowing traffic from Google's health check probe ranges isn't required for hybridNEGs. However, if you're using a combination of hybrid and zonal NEGs ina single backend service, you need to allow traffic from theGooglehealth check probe ranges for the zonal NEGs.

Advertise routes

Configure Cloud Router toadvertise the following custom IPranges to youron-premises environment or other cloud environment:

  • The range of the region's proxy-only subnet.

Set up the Google Cloud environment

For the following steps, make sure you use the same VPC network(calledNETWORK in this procedure) thatwas used to configure hybrid connectivity between the environments.

Additionally, make sure the regions used(calledREGION_A andREGION_B in this procedure)are the same as those used to create the Cloud VPN tunnel orCloud Interconnect VLAN attachments.

Optionally, you canconfigure DNS routing policiesof typeGEO to route client traffic to the load balancer VIP in the regionclosest to the client during regional outages.

Configure the backend subnets

Use this subnet to create the load balancer's zonal NEG backends:

Console

  1. In the Google Cloud console, go to theVPC networks page.

    Go to VPC networks

  2. Go to the network that was used to configure hybrid connectivity betweenthe environments.

  3. In theSubnets section:

    • Set theSubnet creation mode toCustom.
    • In theNew subnet section, enter the following information:
      • Provide aName for the subnet.
      • Select aRegion:REGION_A
      • Enter anIP address range.
    • ClickDone.

  4. ClickCreate.

  5. To add more subnets in different regions, clickAdd subnet and repeatthe previous steps forREGION_B

gcloud

  1. Create subnets in the network that was used to configure hybridconnectivity between the environments.

    gcloud compute networks subnets createSUBNET_A \    --network=NETWORK \    --range=LB_SUBNET_RANGE1 \    --region=REGION_A
    gcloud compute networks subnets createSUBNET_B \    --network=NETWORK \    --range=LB_SUBNET_RANGE2 \    --region=REGION_B

API

Make aPOST request to thesubnetworks.insert method.ReplacePROJECT_ID with your project ID.

POST https://compute.googleapis.com/compute/v1/projects/PROJECT_ID/regions/REGION_A/subnetworks{ "name": "SUBNET_A", "network": "projects/PROJECT_ID/global/networks/NETWORK", "ipCidrRange": "LB_SUBNET_RANGE1", "region": "projects/PROJECT_ID/regions/REGION_A",}

Make aPOST request to thesubnetworks.insert method.ReplacePROJECT_ID with your project ID.

POST https://compute.googleapis.com/compute/v1/projects/PROJECT_ID/regions/REGION_B/subnetworks{ "name": "SUBNET_B", "network": "projects/PROJECT_ID/global/networks/NETWORK", "ipCidrRange": "LB_SUBNET_RANGE2", "region": "projects/PROJECT_ID/regions/REGION_B",}

Replace the following:

  • SUBNET_A andSUBNET_B: the name of the subnets
  • LB_SUBNET_RANGE1 andLB_SUBNET_RANGE2: the IP address range forthe subnets
  • REGION_A andREGION_B:the regions where you have configured the load balancer

Configure the proxy-only subnet

Aproxy-only subnet provides aset of IP addresses that Google uses to run Envoy proxies on your behalf. Theproxies terminate connections from the client and create new connections to thebackends.

This proxy-only subnet is used by allEnvoy-based regional loadbalancers in thesame region of the VPC network. There can only beone active proxy-only subnet for a given purpose, per region, per network.

Important: Don't try to assign addresses from the proxy-only subnet to your loadbalancer's forwarding rule or backends. You assign the forwarding rule's IPaddress and the backend instance IP addresses froma different subnet range (or ranges), not this one.Google Cloud reserves this subnet range for Google Cloud-managedproxies.

Console

If you're using the Google Cloud console, you can wait and create the proxy-only subnet later on theLoad balancing page.

If you want to create the proxy-only subnet now, use the following steps:

  1. In the Google Cloud console, go to theVPC networks page.

    Go to VPC networks

  2. Click the name of the VPC network.
  3. On theSubnets tab, clickAdd subnet.
  4. Provide aName for the proxy-only subnet.
  5. In theRegion list, selectREGION_A.
  6. In thePurpose list, selectCross-region Managed Proxy.
  7. In theIP address range field, enter10.129.0.0/23.
  8. ClickAdd.

Create the proxy-only subnet inREGION_B

  1. ClickAdd subnet.
  2. Provide aName for the proxy-only subnet.
  3. In theRegion list, selectREGION_B.
  4. In thePurpose list, selectCross-region Managed Proxy.
  5. In theIP address range field, enter10.130.0.0/23.
  6. ClickAdd.

gcloud

Create the proxy-only subnets with thegcloud compute networks subnets create command.

    gcloud compute networks subnets createPROXY_SN_A \        --purpose=GLOBAL_MANAGED_PROXY \        --role=ACTIVE \        --region=REGION_A \        --network=NETWORK \        --range=PROXY_ONLY_SUBNET_RANGE1
    gcloud compute networks subnets createPROXY_SN_B \        --purpose=GLOBAL_MANAGED_PROXY \        --role=ACTIVE \        --region=REGION_B \        --network=NETWORK \        --range=PROXY_ONLY_SUBNET_RANGE2

Replace the following:

  • PROXY_SN_A andPROXY_SN_B: the name of the proxy-only subnets
  • PROXY_ONLY_SUBNET_RANGE1 andPROXY_ONLY_SUBNET_RANGE2: the IP address range for the proxy-only subnets
  • REGION_A andREGION_B: the regions where you have configured the load balancer

API

Create the proxy-only subnets with thesubnetworks.insert method, replacingPROJECT_ID with your project ID.

    POST https://compute.googleapis.com/compute/v1/projects/PROJECT_ID/regions/REGION_A/subnetworks    {      "name": "PROXY_SN_A",      "ipCidrRange": "PROXY_ONLY_SUBNET_RANGE1",      "network": "projects/PROJECT_ID/global/networks/NETWORK",      "region": "projects/PROJECT_ID/regions/REGION_A",      "purpose": "GLOBAL_MANAGED_PROXY",      "role": "ACTIVE"    }
    POST https://compute.googleapis.com/compute/v1/projects/PROJECT_ID/regions/REGION_B/subnetworks    {      "name": "PROXY_SN_B",      "ipCidrRange": "PROXY_ONLY_SUBNET_RANGE2",      "network": "projects/PROJECT_ID/global/networks/NETWORK",      "region": "projects/PROJECT_ID/regions/REGION_B",      "purpose": "GLOBAL_MANAGED_PROXY",      "role": "ACTIVE"    }

Create firewall rules

In this example, you create the following firewall rules for the zonal NEGbackends on Google Cloud:

  • fw-allow-health-check: An ingress rule, applicable to theinstances being load balanced, that allows traffic fromGoogle Cloud health checking systems (130.211.0.0/22and35.191.0.0/16). This example uses the target tagallow-health-check toidentify the zonal NEGs to which it should apply.
  • fw-allow-ssh: An ingress rule that allows incoming SSH connectivity on TCPport 22 from any address. You can choose a more restrictive source IP rangefor this rule; for example, you can specify just the IP ranges of the systemsfrom which you will initiate SSH sessions. This example uses the target tagallow-ssh to identify the virtual machine (VM) instances to which it should apply.

  • fw-allow-proxy-only-subnet: An ingress rule that allows connections from theproxy-only subnet to reach the zonal NEG backends.

Console

  1. In the Google Cloud console, go to theFirewall policies page.

    Go to Firewall policies

  2. ClickCreate firewall rule to create the rule to allow traffic fromhealth check probes:

    1. Enter aName offw-allow-health-check.
    2. ForNetwork, selectNETWORK.
    3. ForTargets, selectSpecified target tags.
    4. Populate theTarget tags field withallow-health-check.
    5. SetSource filter toIPv4 ranges.
    6. SetSource IPv4 ranges to130.211.0.0/22 and35.191.0.0/16.
    7. ForProtocols and ports, selectSpecified protocols andports.
    8. SelectTCP and then enter80 for the port number.
    9. ClickCreate.

  3. ClickCreate firewall rule again to create the rule to allow incomingSSH connections:

    1. Name:fw-allow-ssh
    2. Network:NETWORK
    3. Priority:1000
    4. Direction of traffic: ingress
    5. Action on match: allow
    6. Targets: Specified target tags
    7. Target tags:allow-ssh
    8. Source filter:IPv4 ranges
    9. Source IPv4 ranges:0.0.0.0/0
    10. Protocols and ports: ChooseSpecified protocols and ports.
    11. SelectTCP and then enter22 for the port number.
    12. ClickCreate.

  4. ClickCreate firewall rule again to create the rule to allow incomingconnections from the proxy-only subnet:

    1. Name:fw-allow-proxy-only-subnet
    2. Network:NETWORK
    3. Priority:1000
    4. Direction of traffic: ingress
    5. Action on match: allow
    6. Targets: Specified target tags
    7. Target tags:allow-proxy-only-subnet
    8. Source filter:IPv4 ranges
    9. Source IPv4 ranges:PROXY_ONLY_SUBNET_RANGE1andPROXY_ONLY_SUBNET_RANGE2
    10. Protocols and ports: ChooseSpecified protocols and ports
    11. SelectTCP and then enter80 for the port number.
    12. ClickCreate.

gcloud

  1. Create thefw-allow-health-check-and-proxy rule to allowthe Google Cloud health checks to reach thebackend instances on TCP port80:

    gcloud compute firewall-rules create fw-allow-health-check \    --network=NETWORK \    --action=allow \    --direction=ingress \    --target-tags=allow-health-check \    --source-ranges=130.211.0.0/22,35.191.0.0/16 \    --rules=tcp:80

  2. Create thefw-allow-ssh firewall rule to allow SSH connectivity toVMs with the network tagallow-ssh. When you omitsource-ranges,Google Cloudinterprets the rule to mean anysource.

    gcloud compute firewall-rules create fw-allow-ssh \    --network=NETWORK \    --action=allow \    --direction=ingress \    --target-tags=allow-ssh \    --rules=tcp:22

  3. Create an ingress allow firewall rule for the proxy-only subnet to allowthe load balancer to communicate with backend instances on TCP port80:

    gcloud compute firewall-rules create fw-allow-proxy-only-subnet \    --network=NETWORK \    --action=allow \    --direction=ingress \    --target-tags=allow-proxy-only-subnet \    --source-ranges=PROXY_ONLY_SUBNET_RANGE1,PROXY_ONLY_SUBNET_RANGE2 \    --rules=tcp:80

Set up the zonal NEG

For Google Cloud-based backends, we recommend you configure multiple zonalNEGs in the same region where you configuredhybridconnectivity.

For this example, set up a zonal NEG (withGCE_VM_IP_PORT type endpoints)in theREGION_A region. First create the VMs intheZONE_A zone. Thencreate a zonal NEG in theZONE_A zone, and thenadd the VMs' network endpoints to the NEG.To support high availability, set up a similar zonal NEG in theREGION_B region. If backends in one region happen to bedown, traffic fails over to the other region.

Create VMs

Console

  1. In the Google Cloud console, go to theVM instances page.

    Go to VM instances

  2. Repeat steps 3 to 8 for each VM, using the following nameand zone combinations.

    • Name: ofvm-a1
      • Zone:ZONE_A inthe regionREGION_A
      • Subnet:SUBNET_A
    • Name: ofvm-b1
      • Zone:ZONE_B inthe regionREGION_B
      • Subnet:SUBNET_B

  3. ClickCreate instance.

  4. Set the name as indicated in the preceding step.

  5. For theRegion, choose as indicated in the earlier step.

  6. For theZone, choose as indicated in the earlier step.

  7. In theBoot disk section, ensure thatDebian GNU/Linux 12 (bookworm) is selected for the boot diskoptions. ClickChoose to change the image if necessary.

  8. In theAdvanced options section, expandNetworking, and then dothe following:

    • Add the followingNetwork tags:allow-ssh,allow-health-check, andallow-proxy-only-subnet.
    • In theNetwork interfaces section, clickAdd a network interfacemake the following changes, and then clickDone:
      • Network:NETWORK
      • Subnetwork: as indicated in the earlier step.
      • Primary internal IP: Ephemeral (automatic)
      • External IP: Ephemeral

    • ExpandManagement. In theAutomation field, copy and pastethe following script contents. The script contents are identical forall VMs:

      #! /bin/bashapt-get updateapt-get install apache2 -ya2ensite default-ssla2enmod sslvm_hostname="$(curl -H "Metadata-Flavor:Google" \http://metadata.google.internal/computeMetadata/v1/instance/name)"echo "Page served from: $vm_hostname" | \tee /var/www/html/index.htmlsystemctl restart apache2

  9. ClickCreate.

gcloud

Create the VMs by running the following command, using these combinations for the name of the VM and its zone. The script contents are identical for both VMs.

gcloud compute instances createVM_NAME \    --zone=GCP_NEG_ZONE \    --image-family=debian-12 \    --image-project=debian-cloud \    --tags=allow-ssh,allow-health-check,allow-proxy-only-subnet \    --subnet=LB_SUBNET_NAME \    --metadata=startup-script='#! /bin/bash      apt-get update      apt-get install apache2 -y      a2ensite default-ssl      a2enmod ssl      vm_hostname="$(curl -H "Metadata-Flavor:Google" \      http://metadata.google.internal/computeMetadata/v1/instance/name)"      echo "Page served from: $vm_hostname" | \      tee /var/www/html/index.html      systemctl restart apache2'
  • VM_NAME ofvm-a1
    • The zoneGCP_NEG_ZONE asZONE_Ain the regionREGION_A
    • The subnetLB_SUBNET_NAME asSUBNET_A
  • VM_NAME ofvm-b1
    • ZoneGCP_NEG_ZONE asZONE_Bin the regionREGION_B
    • SubnetLB_SUBNET_NAME asSUBNET_B

Create the zonal NEG

Console

To create a zonal network endpoint group:

  1. In the Google Cloud console, go to theNetwork Endpoint Groups page.

    Go to Network Endpoint Groups

  2. Repeat steps 3 to 8 for each zonal NEG, using the following nameand zone combinations:

    • Name:neg-1
      • Zone:ZONE_A in theregionREGION_A
      • Subnet:SUBNET_A
    • Name:neg-2
      • Zone:ZONE_B in theregionREGION_B
      • Subnet:SUBNET_B

  3. ClickCreate network endpoint group.

  4. Set the name as indicated in the preceding step.

  5. Select theNetwork endpoint group type:Network endpoint group(Zonal).

  6. Select theNetwork:NETWORK

  7. Select theSubnetwork as indicated in earlier step.

  8. Select theZone as indicated in earlier step.

  9. Enter theDefault port:80.

  10. ClickCreate.

Add endpoints to the zonal NEG:

  1. In the Google Cloud console, go to theNetwork Endpoint Groups page.

    Go to the Network endpoint groups

  2. Click theName of the network endpoint group created in the previousstep. Yousee theNetwork endpoint group details page.

  3. In theNetwork endpoints in this group section, clickAdd networkendpoint. You see theAdd network endpoint page.

  4. Select aVM instance to add its internal IP addresses as networkendpoints. In theNetwork interface section, the name, zone,and subnet of the VM is displayed.

  5. Enter theIP address of the new network endpoint.

  6. Select thePort type.

    1. If you selectDefault, the endpoint uses the default port80for all endpoints in the network endpoint group. This is sufficientfor our example because the Apache server is serving requests atport80.
    2. If you selectCustom, enter thePort number for the endpointto use.

  7. To add more endpoints, clickAdd network endpoint and repeat theprevious steps.

  8. After you add all the endpoints, clickCreate.

gcloud

  1. Create zonal NEGs (withGCE_VM_IP_PORT endpoints)using the name, zone, and subnet combinations.Use thegcloud compute network-endpoint-groupscreate command.

    gcloud compute network-endpoint-groups createGCP_NEG_NAME \    --network-endpoint-type=GCE_VM_IP_PORT \    --zone=GCP_NEG_ZONE \    --network=NETWORK \    --subnet=LB_SUBNET_NAME
    • Name:neg-1
      • ZoneGCP_NEG_ZONE:ZONE_A in theregionREGION_A
      • SubnetLB_SUBNET_NAME:SUBNET_A
    • Name:neg-2
      • ZoneGCP_NEG_ZONE:ZONE_B in theregionREGION_B
      • SubnetLB_SUBNET_NAME:SUBNET_B

    You can either specify a port using the--default-port option whilecreating the NEG, orspecify a port number for eachendpointas shown in the next step.

  2. Add endpoints toneg1 andneg2.

    gcloud compute network-endpoint-groups update neg1 \    --zone=ZONE_A \    --add-endpoint='instance=vm-a1,port=80'
    gcloud compute network-endpoint-groups update neg2 \    --zone=ZONE_B \    --add-endpoint='instance=vm-b1,port=80'

Set up the hybrid connectivity NEG

Note: If you're usingdistributed Envoy healthchecks with hybrid connectivity NEG backends (supported only forEnvoy-basedload balancers), make sure that you configure unique networkendpoints for all the NEGs attached to the same backend service. Addingthe same network endpoint to multiple NEGs results in undefined behavior.

When creating the NEG, use a zone that minimizes the geographicdistance between Google Cloud and your on-premises or other cloudenvironment. For example, if you are hosting a service in an on-premisesenvironment in Frankfurt, Germany, you can specify theeurope-west3-aGoogle Cloud zone when you create the NEG.

And, if you're using Cloud Interconnect, the zone usedto create the NEG is in the same region where theCloud Interconnect attachment was configured.

Hybrid NEGs support only thedistributed Envoy healthchecks.

Console

To create a hybrid connectivity network endpoint group:

  1. In the Google Cloud console, go to theNetwork Endpoint Groups page.

    Go to Network endpoint groups

  2. ClickCreate network endpoint group.

  3. Repeat steps 4 to 9 for each hybrid NEG, using the following name and zone combinations.

    • NameON_PREM_NEG_NAME:hybrid-1
      • Zone:ON_PREM_NEG_ZONE1
      • Subnet:SUBNET_A
    • NameON_PREM_NEG_NAME:hybrid-2
      • Zone:ON_PREM_NEG_ZONE2
      • Subnet:SUBNET_B

  4. Set the name as indicated in the previous step.

  5. Select theNetwork endpoint group type:Hybrid connectivity networkendpoint group (Zonal).

  6. Select theNetwork:NETWORK

  7. For theSubnet, choose as indicated in the previous step.

  8. For theZone, choose as indicated in the previous step.

  9. Enter theDefault port.

  10. ClickCreate

Add endpoints to the hybrid connectivity NEG:

  1. In the Google Cloud console, go to theNetwork Endpoint Groups page.

    Go to Network endpoint groups

  2. Click theName of the network endpoint group created in the previousstep. Yousee theNetwork endpoint group detail page.

  3. In theNetwork endpoints in this group section, clickAdd networkendpoint. You see theAdd network endpoint page.

  4. Enter theIP address of the new network endpoint.

  5. Select thePort type.

    1. If you selectDefault, the endpoint uses the default portfor all endpoints in the network endpoint group.
    2. If you selectCustom, you can enter a differentPort numberfor the endpoint to use.

  6. To add more endpoints, clickAdd network endpoint and repeat theprevious steps.

  7. After you add all the non-Google Cloud endpoints,clickCreate.

gcloud

  1. Create a hybrid connectivity NEG that uses the following name combinations.Use thegcloud compute network-endpoint-groupscreate command.

    gcloud compute network-endpoint-groups createON_PREM_NEG_NAME \    --network-endpoint-type=NON_GCP_PRIVATE_IP_PORT \    --zone=ON_PREM_NEG_ZONE \    --network=NETWORK
    • NameON_PREM_NEG_NAME:hybrid-1
      • ZoneON_PREM_NEG_ZONE:ON_PREM_NEG_ZONE1
    • NameON_PREM_NEG_NAME:hybrid-2
      • ZoneGCP_NEG_ZONE:ON_PREM_NEG_ZONE2

  2. Add the on-premises backend VM endpoint toON_PREM_NEG_NAME:

    gcloud compute network-endpoint-groups updateON_PREM_NEG_NAME \    --zone=ON_PREM_NEG_ZONE \    --add-endpoint="ip=ON_PREM_IP_ADDRESS_1,port=PORT_1" \    --add-endpoint="ip=ON_PREM_IP_ADDRESS_2,port=PORT_2"

You can use this command to add the network endpoints youpreviouslyconfigured on premises or in your cloud environment.Repeat--add-endpoint as many times as needed.

Configure the load balancer

Console

Note: You cannot use the Google Cloud console to create a load balancerthat has mixed zonal and hybrid connectivity NEG backends in a singlebackend service. Use either gcloud or the REST API instead.

gcloud

  1. Define the HTTP health check with thegcloud compute health-checkscreate http command.

    gcloud compute health-checks create http gil7-basic-check \   --use-serving-port \   --global

  2. Create the backend service and enable logging with thegcloud compute backend-servicescreate command.

    gcloud compute backend-services createBACKEND_SERVICE \  --load-balancing-scheme=INTERNAL_MANAGED \  --protocol=HTTP \  --enable-logging \  --logging-sample-rate=1.0 \  --health-checks=gil7-basic-check \  --global-health-checks \  --global

  3. Add backends to the backend service with thegcloud compute backend-servicesadd-backend command.

    gcloud compute backend-services add-backendBACKEND_SERVICE \  --global \  --balancing-mode=RATE \  --max-rate-per-endpoint=MAX_REQUEST_RATE_PER_ENDPOINT \  --network-endpoint-group=neg1 \  --network-endpoint-group-zone=ZONE_A \  --network-endpoint-group=neg2 \  --network-endpoint-group-zone=ZONE_B

    For details aboutconfiguring the balancing mode, see the gcloud CLI documentationfor the--max-rate-per-endpointflag.

  4. Add the hybrid NEGs as a backend to the backend service.

    gcloud compute backend-services add-backendBACKEND_SERVICE \  --global \  --balancing-mode=RATE \  --max-rate-per-endpoint=MAX_REQUEST_RATE_PER_ENDPOINT \  --network-endpoint-group=hybrid1 \  --network-endpoint-group-zone=ON_PREM_NEG_ZONE1 \  --network-endpoint-group=hybrid2 \  --network-endpoint-group-zone=ON_PREM_NEG_ZONE2 \

    For detailsabout configuring the balancing mode, see the gcloud CLIdocumentation for the--max-rate-per-endpointparameter.

  5. Create the URL map with thegcloud compute url-mapscreate command.

    gcloud compute url-maps create gil7-map \  --default-service=BACKEND_SERVICE \  --global

  6. Create the target proxy.

    For HTTP:

    Create the target proxy with thegcloud compute target-http-proxiescreate command.

    gcloud compute target-http-proxies create gil7-http-proxy \  --url-map=gil7-map \  --global

    For HTTPS:

    To create a Google-managed certificate, see the following documentation:

    After you create the Google-managed certificate,attach the certificate to the target proxy.Certificate maps are not supported by cross-region internal Application Load Balancers.

    To create a self-managed certificate, see the following documentation:

    Assign your file paths to variable names.

    export LB_CERT=PATH_TO_PEM_FORMATTED_FILE
    export LB_PRIVATE_KEY=PATH_TO_PEM_LB_PRIVATE_FILE

    Create an all region SSL certificate using thegcloud certificate-managercertificates create command.

    gcloud certificate-manager certificates create gilb-certificate \  --private-key-file=$LB_CERT \  --certificate-file=$LB_PRIVATE_KEY \  --scope=all-regions

    Use the SSL certificate to create a target proxy with thegcloudcompute target-https-proxies create command

    gcloud compute target-https-proxies create gil7-https-proxy \  --url-map=gil7-map \  --certificate-manager-certificates=gilb-certificate \  --global

  7. Create two forwarding rules, one with a VIPIP_ADDRESS1 in theREGION_A region andanother one with a VIPIP_ADDRESS2in theREGION_B region.For the forwarding rule's IP address, use theLB_SUBNET_RANGE1orLB_SUBNET_RANGE2 IPaddress range. If you try to use theproxy-only subnet, forwarding rule creation fails.

    For custom networks, you must reference the subnet in the forwardingrule. Note that this is the VM subnet, not the proxy subnet.

    For HTTP:

    Use thegcloud compute forwarding-rulescreate commandwith the correct flags.

    gcloud compute forwarding-rules createFWRULE_A \  --load-balancing-scheme=INTERNAL_MANAGED \  --network=NETWORK \  --subnet=SUBNET_A \  --subnet-region=REGION_A \  --address=IP_ADDRESS1 \  --ports=80 \  --target-http-proxy=gil7-http-proxy \  --global
    gcloud compute forwarding-rules createFWRULE_B \  --load-balancing-scheme=INTERNAL_MANAGED \  --network=NETWORK \  --subnet=SUBNET_B \  --subnet-region=REGION_B \  --address=IP_ADDRESS2 \  --ports=80 \  --target-http-proxy=gil7-http-proxy \  --global

    For HTTPS:

    Create the forwarding rule with thegcloud compute forwarding-rulescreate commandwith the correct flags.

    gcloud compute forwarding-rules createFWRULE_A \  --load-balancing-scheme=INTERNAL_MANAGED \  --network=NETWORK \  --subnet=SUBNET_A \  --subnet-region=REGION_A \  --address=IP_ADDRESS1 \  --ports=443 \  --target-https-proxy=gil7-https-proxy \  --global
    gcloud compute forwarding-rules createFWRULE_B \  --load-balancing-scheme=INTERNAL_MANAGED \  --network=NETWORK \  --subnet=SUBNET_B \  --subnet-region=REGION_B \  --address=IP_ADDRESS2 \  --ports=443 \  --target-https-proxy=gil7-https-proxy \  --global

Connect your domain to your load balancer

After you create the load balancer, note the IP address that is associated withthe load balancer—for example,IP_ADDRESS1 andIP_ADDRESS2.To point your domain to your load balancer, create anA record by usingCloud DNS or domain registration service. Ifyou added multiple domains to your SSL certificate, you must add anA recordfor each one, all pointing to the load balancer's IP address.

Test the load balancer

Create a VM instance to test connectivity

  1. Create a client VM:

    gcloud compute instances create l7-ilb-client-a \    --image-family=debian-12 \    --image-project=debian-cloud \    --network=NETWORK \    --subnet=SUBNET_A \    --zone=ZONE_A \    --tags=allow-ssh
    gcloud compute instances create l7-ilb-client-b \    --image-family=debian-12 \    --image-project=debian-cloud \    --network=NETWORK \    --subnet=SUBNET_B \    --zone=ZONE_B \    --tags=allow-ssh

  2. Use SSH to connect to each client instance.

    gcloud compute ssh l7-ilb-client-a \   --zone=ZONE_A
    gcloud compute ssh l7-ilb-client-b \   --zone=ZONE_B

  3. Verify that the IP address is serving its hostname.

    • Verify that the client VM can reach both IP addresses. The command shouldsucceed and return the name of the backend VM which served the request:

      For HTTP testing:

      curlIP_ADDRESS1
      curlIP_ADDRESS2

      For HTTPS testing:

      curl -k -s 'https://DOMAIN_NAME:443' --connect-toDOMAIN_NAME:443:IP_ADDRESS1:443
      curl -k -s 'https://DOMAIN_NAME:443' --connect-toDOMAIN_NAME:443:IP_ADDRESS2:443

      ReplaceDOMAIN_NAME with your application domain name, forexample,test.example.com.

      The-k flag causes curl to skip certificate validation.

    • Optional: Use the configuredDNS record to resolvethe IP address closest to the client VM. For example,DNS_ENTRYcan beservice.example.com.

      curlDNS_ENTRY

Run 100 requests

Run 100 curl requests and confirm from the responses that they are loadbalanced.

For HTTP:

  • Verify that the client VM can reach both IP addresses.The command should succeed and return the name of the backend VM whichserved the request:

    {  RESULTS=  for i in {1..100}  do    RESULTS="$RESULTS:$(curl --silentIP_ADDRESS1)"  done  echo "***"  echo "*** Results of load-balancing toIP_ADDRESS1: "  echo "***"  echo "$RESULTS" | tr ':' '\n' | grep -Ev "^$" | sort | uniq -c  echo}
    {  RESULTS=  for i in {1..100}  do    RESULTS="$RESULTS:$(curl --silentIP_ADDRESS2)"  done  echo "***"  echo "*** Results of load-balancing toIP_ADDRESS2: "  echo "***"  echo "$RESULTS" | tr ':' '\n' | grep -Ev "^$" | sort | uniq -c  echo}

For HTTPS:

  • Verify that the client VM can reach both IP addresses.The command should succeed and return the name of the backend VM whichserved the request.

    ReplaceDOMAIN_NAME with your application domain name, forexample,test.example.com.

    {  RESULTS=  for i in {1..100}  do    RESULTS="$RESULTS:$(curl -k -s 'https://DOMAIN_NAME:443' --connect-toDOMAIN_NAME:443:IP_ADDRESS1:443)"  done  echo "***"  echo "*** Results of load-balancing toIP_ADDRESS1: "  echo "***"  echo "$RESULTS" | tr ':' '\n' | grep -Ev "^$" | sort | uniq -c  echo}
    {  RESULTS=  for i in {1..100}  do    RESULTS="$RESULTS:$(curl -k -s 'https://DOMAIN_NAME:443' --connect-toDOMAIN_NAME:443:IP_ADDRESS2:443)"  done  echo "***"  echo "*** Results of load-balancing toIP_ADDRESS2: "  echo "***"  echo "$RESULTS" | tr ':' '\n' | grep -Ev "^$" | sort | uniq -c  echo}

Test failover

  1. Verify failover to backends in theREGION_Aregion when backends in theREGION_Bare unhealthy or unreachable.We simulate this by removing all thebackends fromREGION_B:

    gcloud compute backend-services remove-backendBACKEND_SERVICE \   --balancing-mode=RATE \   --network-endpoint-group=neg2 \   --network-endpoint-group-zone=ZONE_B

  2. Use SSH to connect to the client VM inREGION_B.

    gcloud compute ssh l7-ilb-client-b \   --zone=ZONE_B

  3. Send requests to the load balanced IP address in theREGION_B region.The output is similar to the following:

    {RESULTS=for i in {1..100}do  RESULTS="$RESULTS:$(curl -k -s 'https://DOMAIN_NAME:443' --connect-toDOMAIN_NAME:443:IP_ADDRESS2:443)"doneecho "***"echo "*** Results of load-balancing toIP_ADDRESS2: "echo "***"echo "$RESULTS" | tr ':' '\n' | grep -Ev "^$" | sort | uniq -cecho}

Additional configuration options

This section expands on the configuration example to provide alternative andadditional configuration options. All of the tasks are optional. You canperform them in any order.

Configure DNS routing policies

If your clients are in multiple regions, you might want to make yourcross-region internal Application Load Balancer accessible by using VIPs in these regions. You can useDNS routing policies of typeGEO to route clienttraffic to the load balancer VIP in the region closest to the client. Thismulti-region setup minimizes latency and network transit costs. In addition, itlets you set up a DNS-based, global, load balancing solution that providesresilience against regional outages.

Cloud DNS supports health checking and enables automatic failover whenthe endpoints fail their health checks. During a failover, Cloud DNSautomatically adjusts the traffic split among the remaining healthy endpoints.For more information, seeManage DNS routing policies and healthchecks.

gcloud

To create a DNS entry with a 30 second TTL, use thegcloud dns record-sets create command.

gcloud dns record-sets createDNS_ENTRY --ttl="30" \  --type="A" --zone="service-zone" \  --routing-policy-type="GEO" \  --routing-policy-data="REGION_A=gil7-forwarding-rule-a@global;REGION_B=gil7-forwarding-rule-b@global" \  --enable-health-checking

Replace the following:

  • DNS_ENTRY: DNS or domain name of the record-set

    For example,service.example.com

  • REGION_A andREGION_B:the regions where you have configured the load balancer

API

Create the DNS record by making aPOST request to theResourceRecordSets.create method.ReplacePROJECT_ID with your project ID.

POST https://www.googleapis.com/dns/v1/projects/PROJECT_ID/managedZones/SERVICE_ZONE/rrsets{  "name": "DNS_ENTRY",  "type": "A",  "ttl": 30,  "routingPolicy": {    "geo": {      "items": [        {          "location": "REGION_A",          "healthCheckedTargets": {            "internalLoadBalancers": [              {                "loadBalancerType": "globalL7ilb",                "ipAddress": "IP_ADDRESS",                "port": "80",                "ipProtocol": "tcp",                "networkUrl": "https://www.googleapis.com/compute/v1/projects/PROJECT_ID/global/networks/lb-network",                "project": "PROJECT_ID"              }            ]          }        },        {          "location": "REGION_B",          "healthCheckedTargets": {            "internalLoadBalancers": [              {                "loadBalancerType": "globalL7ilb",                "ipAddress": "IP_ADDRESS_B",                "port": "80",                "ipProtocol": "tcp",                "networkUrl": "https://www.googleapis.com/compute/v1/projects/PROJECT_ID/global/networks/lb-network",                "project": "PROJECT_ID"              }            ]          }        }      ]    }  }}

Update client HTTP keepalive timeout

The load balancer created in the previous steps has been configured witha default value for theclient HTTP keepalivetimeout.

To update the client HTTP keepalive timeout, use the following instructions.

Console

  1. In the Google Cloud console, go to theLoad balancing page.

    Go to Load balancing.

  2. Click the name of the load balancer that you want to modify.
  3. ClickEdit.
  4. ClickFrontend configuration.
  5. ExpandAdvanced features. ForHTTP keepalive timeout, enter a timeout value.
  6. ClickUpdate.
  7. To review your changes, clickReview and finalize, and then clickUpdate.

gcloud

For an HTTP load balancer, update the target HTTP proxy by using thegcloud compute target-http-proxies update command:

      gcloud compute target-http-proxies updateTARGET_HTTP_PROXY_NAME \          --http-keep-alive-timeout-sec=HTTP_KEEP_ALIVE_TIMEOUT_SEC \          --global

For an HTTPS load balancer, update the target HTTPS proxy by using thegcloud compute target-https-proxies update command:

      gcloud compute target-https-proxies updateTARGET_HTTPS_PROXY_NAME \          --http-keep-alive-timeout-sec=HTTP_KEEP_ALIVE_TIMEOUT_SEC \          --global

Replace the following:

  • TARGET_HTTP_PROXY_NAME: the name of the target HTTP proxy.
  • TARGET_HTTPS_PROXY_NAME: the name of the target HTTPS proxy.
  • HTTP_KEEP_ALIVE_TIMEOUT_SEC: the HTTP keepalive timeout value from 5 to 600 seconds.

Enable outlier detection

You can enableoutlierdetectionon global backend services to identify unhealthy serverless NEGs and reduce thenumber the requests sent to the unhealthy serverless NEGs.

Outlier detection is enabled on the backend service by using one of the following methods:

  • TheconsecutiveErrors method (outlierDetection.consecutiveErrors), inwhich a5xx series HTTP status code qualifies as an error.
  • TheconsecutiveGatewayFailure method(outlierDetection.consecutiveGatewayFailure), in which only the502,503, and504 HTTP status codes qualify as an error.

Use the following steps to enable outlier detection for an existing backendservice. Note that even after enabling outlier detection, some requests can besent to the unhealthy service and return a5xx status code tothe clients. To further reduce the error rate, you can configure more aggressivevalues for the outlier detection parameters. For more information, see theoutlierDetection field.

Console

  1. In the Google Cloud console, go to theLoad balancing page.

    Go to Load balancing

  2. Click the name of the load balancer whose backend service you want toedit.

  3. On theLoad balancer details page, clickEdit.

  4. On theEdit cross-region internal Application Load Balancer page, clickBackend configuration.

  5. On theBackend configuration page, clickEdit for the backend service thatyou want to modify.

  6. Scroll down and expand theAdvanced configurations section.

  7. In theOutlier detection section, select theEnable checkbox.

  8. ClickEdit to configureoutlier detection.

    Verify that the following options are configured with these values:

    PropertyValue
    Consecutive errors5
    Interval1000
    Base ejection time30000
    Max ejection percent50
    Enforcing consecutive errors100

    In this example, the outlier detection analysis runs every one second. Ifthe number of consecutive HTTP5xx status codesreceived by anEnvoy proxy is five or more, the backend endpoint is ejected from theload-balancing pool of that Envoy proxy for 30 seconds. When theenforcing percentage is set to 100%, the backend service enforces theejection of unhealthy endpoints from the load-balancing pools of thosespecific Envoy proxies every time the outlier detection analysis runs. Ifthe ejection conditions are met, up to 50% of the backend endpoints fromthe load-balancing pool can be ejected.

  9. ClickSave.

  10. To update the backend service, clickUpdate.

  11. To update the load balancer, on theEdit cross-region internal Application Load Balancer page, clickUpdate.

gcloud

  1. Export the backend service into a YAML file.

    gcloud compute backend-services exportBACKEND_SERVICE_NAME \  --destination=BACKEND_SERVICE_NAME.yaml --global

    ReplaceBACKEND_SERVICE_NAME with the name of thebackend service.

  2. Edit the YAML configuration of the backend service to add the fields foroutlier detection as highlighted in the following YAML configuration,in theoutlierDetection section:

    In this example, the outlier detection analysis runs every one second. Ifthe number of consecutive HTTP5xx status codesreceived by anEnvoy proxy is five or more, the backend endpoint is ejected from theload-balancing pool of that Envoy proxy for 30 seconds. When theenforcing percentage is set to 100%, the backend service enforces theejection of unhealthy endpoints from the load-balancing pools of thosespecific Envoy proxies every time the outlier detection analysis runs. Ifthe ejection conditions are met, up to 50% of the backend endpoints fromthe load-balancing pool can be ejected.

    name:BACKEND_SERVICE_NAMEbackends:- balancingMode: UTILIZATION  capacityScaler: 1.0  group: https://www.googleapis.com/compute/v1/projects/PROJECT_ID/regions/REGION_A/networkEndpointGroups/SERVERLESS_NEG_NAME- balancingMode: UTILIZATION  capacityScaler: 1.0  group: https://www.googleapis.com/compute/v1/projects/PROJECT_ID/regions/REGION_B/networkEndpointGroups/SERVERLESS_NEG_NAME_2outlierDetection:  baseEjectionTime:    nanos: 0    seconds: 30  consecutiveErrors: 5  enforcingConsecutiveErrors: 100  interval:    nanos: 0    seconds: 1  maxEjectionPercent: 50port: 80selfLink: https://www.googleapis.com/compute/v1/projects/PROJECT_ID/global/backendServices/BACKEND_SERVICE_NAMEsessionAffinity: NONEtimeoutSec: 30...

    Replace the following:

    • BACKEND_SERVICE_NAME: the name of the backendservice
    • PROJECT_ID: the ID of your project
    • REGION_A andREGION_B:the regions where the load balancer has been configured.
    • SERVERLESS_NEG_NAME: the name of the firstserverless NEG
    • SERVERLESS_NEG_NAME_2: the name of the secondserverless NEG

  3. Update the backend service by importing the latest configuration.

    gcloud compute backend-services importBACKEND_SERVICE_NAME \  --source=BACKEND_SERVICE_NAME.yaml --global

    Outlier detection is now enabled on the backend service.

What's next

Except as otherwise noted, the content of this page is licensed under theCreative Commons Attribution 4.0 License, and code samples are licensed under theApache 2.0 License. For details, see theGoogle Developers Site Policies. Java is a registered trademark of Oracle and/or its affiliates.

Last updated 2025-10-24 UTC.