Set up a cross-region internal Application Load Balancer with Cloud Run

This document shows you how to deploy a cross-region internal Application Load Balancer withCloud Run. To set this up, you use aserverless NEG backend for the load balancer.

Although this document describes a Cloud Run configuration, aserverless NEG for Cloud Run can point to either aCloud Run resource or a Cloud Run functions (2nd gen) resource.

Serverless NEGs let you useCloud Run services with your load balancer. After you configurea load balancer with the serverless NEG backend, requests to the load balancerare routed to the Cloud Run backend.

Cross-regional load balancing provides redundancy, so that if a region isunreachable, traffic is automatically diverted to another region.Based on the location of the Envoy, proxy traffic is distributed toCloud Run services as follows:

  • If multi-region Cloud Run services are configured in the sameregion as the Envoy, the NEG that is located in the same region as the Envoyis preferred. Traffic is sent to the failover region only if outlierdetection is enabled and the local NEG is unhealthy.
  • If multi-region Cloud Run services are not configured in thesame region as the Envoy, traffic is distributed evenly acrossall NEGs. The NEGs located closer are not preferred.
  • IfIdentity-Aware Proxy is enabled, only a singleserverless NEG is supported. You can, however, configure additionalCloud Run services but the load balancer does not send anytraffic to them.

Before you begin

Before following this guide, familiarize yourself with the following:

Deploy a Cloud Run service

The instructions on this page assume you already have aCloud Run service running.

For the example on this page, you can useany of theCloud Run quickstarts to deploy aCloud Run service.

To prevent access to the Cloud Run service from theinternet,restrict ingress tointernal. Trafficfrom the internal Application Load Balancer is consideredinternal traffic.

Placing the Cloud Run service in multiple regions helpsprevent failures in a single region. To deploy theCloud Run service inREGION_A andREGION_B regions, run the following commands:

gcloud

gcloud run deployCLOUD_RUN_SERVICE_NAMEA \   --platform=managed \   --allow-unauthenticated \   --ingress=internal \   --region=REGION_A \   --image=IMAGE_URLA
gcloud run deployCLOUD_RUN_SERVICE_NAMEB \   --platform=managed \   --allow-unauthenticated \   --ingress=internal \   --region=REGION_B \   --image=IMAGE_URLB

Note the name of the service that you create. The rest of this page shows youhow to set up a load balancer that routes requests to this service.

Set up an SSL certificate resource

Create a Certificate Manager SSL certificate resource as follows:

We recommend using a Google-managed certificate.

Permissions

To follow this guide, you must be able to create instances and modify anetwork in a project. You must be either a projectowner or editor, or you must have all ofthe followingCompute Engine IAM roles.

TaskRequired role
Create networks, subnets, and load balancer componentsCompute Network Admin
Add and remove firewall rulesCompute Security Admin
Create instancesCompute Instance Admin

For more information, see the following guides:

Setup overview

You can configure the cross-region internal Application Load Balancer as described in the following diagram:

Cross-region internal Application Load Balancer with Cloud Run deployment.
Cross-region internal Application Load Balancer with Cloud Run deployment (click to enlarge).

As shown in the diagram, this example creates a cross-region internal Application Load Balancer in aVPC network, with one backendservice and two Cloud Run deployments inREGION_A andREGION_B regions.

The cross-region internal Application Load Balancer setup is described as follows:

  1. A VPC network with the following subnets:

    • SubnetSUBNET_A and a proxy-only subnet inREGION_A.
    • SubnetSUBNET_B and a proxy-only subnet inREGION_B.

    You must createproxy-only subnetsin each region of a VPC network where you usecross-region internal Application Load Balancers. The region'sproxy-only subnet is shared among all cross-region internal Application Load Balancers inthe region. Source addresses of packets sent from the load balancerto your service's backends are allocated from theproxy-only subnet.In this example, the proxy-only subnet for the regionREGION_A has a primary IP addressrange of10.129.0.0/23 and forREGION_Bhas a primary IP address range of10.130.0.0/23 which is therecommended subnet size.

  2. A firewall rule that permits proxy-only subnet traffic flows in yournetwork. This means adding one rule that allows TCP port80,443, and8080 traffic from10.129.0.0/23 and10.130.0.0/23 (the range of theproxy-only subnets in this example).

  3. Another firewall rule for thehealthcheck probes.

  4. A high availability setup that has serverless backends forCloud Run deployments inREGION_AandREGION_Bregions. If the backends in one region happen to be down, traffic fails overto the other region.

  5. A global backend service that monitors the usage and health ofbackends. Ensure that youenable outlier detection onthe backend service.

  6. A global URL map that parses the URL of a request and forwardsrequests to specific backend services based on the host and path of therequest URL.

  7. A global target HTTP or HTTPS proxy, which receives a request from theuser and forwards it to the URL map. For HTTPS, configure a regional SSLcertificate resource. The target proxy uses the SSL certificate to decrypt SSLtraffic if you configure HTTPS load balancing. The target proxy can forwardtraffic to your instances by using HTTP or HTTPS.

  8. Global forwarding rules, which have the internal IP address of your loadbalancer, to forward each incoming request to the target proxy.

    The internal IP address associated with the forwarding rule can come fromany subnet in the same network and region. Note the following conditions:

    • The IP address can (but does not need to) come from the same subnet asthe backend instance groups.
    • The IP address must not come from a reserved proxy-only subnet that hasits--purpose flag set toGLOBAL_MANAGED_PROXY.
    • If you want touse the same internal IP address with multiple forwardingrules, set the IP address--purpose flag toSHARED_LOADBALANCER_VIP.

  9. Optional:Configure DNS routing policiesof typeGEO to route client traffic to the load balancer VIP inthe region closest to the client.

Configure the network and subnets

Within the VPC network, configure a subnet in each regionwhere your backends are configured. In addition, configure aproxy-only-subnetin each region in which you want to configure the load balancer.

This example uses the following VPC network, region, andsubnets:

  • Network. The network is acustom mode VPCnetwork namedNETWORK.

  • Subnets for backends. A subnet namedSUBNET_A in theREGION_A region uses10.1.2.0/24 for its primaryIP range. Subnet namedSUBNET_B in theREGION_B region uses10.1.3.0/24 for its primaryIP range.

  • Subnet for proxies. A subnet namedPROXY_SN_A in theREGION_A region uses10.129.0.0/23 for its primaryIP range. A subnet namedPROXY_SN_B in theREGION_B region uses10.130.0.0/23 for its primaryIP range.

Cross-region internal Application Load Balancers can be accessed from any region within the VPC.So clients from any region can globally access your load balancer backends.

Note: Subsequent steps in this guide use the network, region,and subnet parameters as outlined here.

Configure the backend subnets

Console

  1. In the Google Cloud console, go to theVPC networks page.

    Go to VPC networks

  2. ClickCreate VPC network.

  3. Provide aName for the network.

  4. In theSubnets section, set theSubnet creation mode toCustom.

  5. Create a subnet for the load balancer's backends. In theNew subnetsection, enter the following information:

    • Provide aName for the subnet.
    • Select aRegion:REGION_A
    • Enter anIP address range:10.1.2.0/24

  6. ClickDone.

  7. ClickAdd subnet.

  8. Create a subnet for the load balancer's backends. In theNewsubnet section, enter the following information:

    • Provide aName for the subnet.
    • Select aRegion:REGION_B
    • Enter anIP address range:10.1.3.0/24

  9. ClickDone.

  10. ClickCreate.

gcloud

  1. Create the custom VPC network with thegcloud computenetworks create command:

    gcloud compute networks createNETWORK --subnet-mode=custom

  2. Create a subnet in theNETWORKnetwork in theREGION_A region withthegcloud compute networks subnets create command:

    gcloud compute networks subnets createSUBNET_A \    --network=NETWORK \    --range=10.1.2.0/24 \    --region=REGION_A

  3. Create a subnet in theNETWORKnetwork in theREGION_B region withthegcloud compute networks subnets create command:

    gcloud compute networks subnets createSUBNET_B \    --network=NETWORK \    --range=10.1.3.0/24 \    --region=REGION_B

API

Make aPOST request to thenetworks.insert method.ReplacePROJECT_ID with your project ID.

POST https://compute.googleapis.com/compute/v1/projects/PROJECT_ID/global/networks{ "routingConfig": {   "routingMode": "regional" }, "name": "NETWORK", "autoCreateSubnetworks": false}

Make aPOST request to thesubnetworks.insert method.ReplacePROJECT_ID with your project ID.

POST https://compute.googleapis.com/compute/v1/projects/PROJECT_ID/regions/REGION_A/subnetworks{ "name": "SUBNET_A", "network": "projects/PROJECT_ID/global/networks/NETWORK", "ipCidrRange": "10.1.2.0/24", "region": "projects/PROJECT_ID/regions/REGION_A",}

Make aPOST request to thesubnetworks.insert method.ReplacePROJECT_ID with your project ID.

POST https://compute.googleapis.com/compute/v1/projects/PROJECT_ID/regions/REGION_B/subnetworks{ "name": "SUBNET_B", "network": "projects/PROJECT_ID/global/networks/NETWORK", "ipCidrRange": "10.1.3.0/24", "region": "projects/PROJECT_ID/regions/REGION_B",}

Configure the proxy-only subnet

Aproxy-only subnet provides aset of IP addresses that Google Cloud uses to run Envoy proxies on your behalf.The proxies end connections from the client and create connections to thebackends.

This proxy-only subnet is used by all Envoy-based regional loadbalancers in the same region as the VPC network. There can only beone active proxy-only subnet for a given purpose, per region, per network.

Important: Don't try to assign addresses from the proxy-only subnet to your loadbalancer's forwarding rule or backends. You assign the forwarding rule's IPaddress and the backend instance IP addresses froma different subnet range (or ranges), not this one.Google Cloud reserves this subnet range for Google Cloud-managedproxies.

Console

If you're using the Google Cloud console, you can wait and create the proxy-only subnet later on theLoad balancing page.

If you want to create the proxy-only subnet now, use the following steps:

  1. In the Google Cloud console, go to theVPC networks page.

    Go to VPC networks

  2. Click the name of the VPC network.
  3. On theSubnets tab, clickAdd subnet.
  4. Provide aName for the proxy-only subnet.
  5. In theRegion list, selectREGION_A.
  6. In thePurpose list, selectCross-region Managed Proxy.
  7. In theIP address range field, enter10.129.0.0/23.
  8. ClickAdd.

Create the proxy-only subnet inREGION_B

  1. ClickAdd subnet.
  2. Provide aName for the proxy-only subnet.
  3. In theRegion list, selectREGION_B.
  4. In thePurpose list, selectCross-region Managed Proxy.
  5. In theIP address range field, enter10.130.0.0/23.
  6. ClickAdd.

gcloud

Create the proxy-only subnets with thegcloud compute networks subnets create command.

    gcloud compute networks subnets createPROXY_SN_A \        --purpose=GLOBAL_MANAGED_PROXY \        --role=ACTIVE \        --region=REGION_A \        --network=NETWORK \        --range=10.129.0.0/23
    gcloud compute networks subnets createPROXY_SN_B \        --purpose=GLOBAL_MANAGED_PROXY \        --role=ACTIVE \        --region=REGION_B \        --network=NETWORK \        --range=10.130.0.0/23

API

Create the proxy-only subnets with thesubnetworks.insert method, replacingPROJECT_ID with your project ID.

    POST https://compute.googleapis.com/compute/v1/projects/PROJECT_ID/regions/REGION_A/subnetworks    {      "name": "PROXY_SN_A",      "ipCidrRange": "10.129.0.0/23",      "network": "projects/PROJECT_ID/global/networks/NETWORK",      "region": "projects/PROJECT_ID/regions/REGION_A",      "purpose": "GLOBAL_MANAGED_PROXY",      "role": "ACTIVE"    }
    POST https://compute.googleapis.com/compute/v1/projects/PROJECT_ID/regions/REGION_B/subnetworks    {      "name": "PROXY_SN_B",      "ipCidrRange": "10.130.0.0/23",      "network": "projects/PROJECT_ID/global/networks/NETWORK",      "region": "projects/PROJECT_ID/regions/REGION_B",      "purpose": "GLOBAL_MANAGED_PROXY",      "role": "ACTIVE"    }

Create the serverless NEGs

  1. Create a serverless NEG for your Cloud Runservice:

    gcloud compute network-endpoint-groups create gl7ilb-serverless-neg-a \   --region=REGION_A \   --network-endpoint-type=serverless  \   --cloud-run-service=CLOUD_RUN_SERVICE_NAMEA
    gcloud compute network-endpoint-groups create gl7ilb-serverless-neg-b \   --region=REGION_B \   --network-endpoint-type=serverless  \   --cloud-run-service=CLOUD_RUN_SERVICE_NAMEB

Configure the load balancer

Traffic going from the load balancer to the serverless NEG backends uses specialroutes defined outside your VPC that are not subject to firewallrules. Therefore, if your load balancer only has serverless NEG backends, youdon't need to create firewall rules to allow traffic from the proxy-only subnetto the serverless backend.

Console

Select the load balancer type

  1. In the Google Cloud console, go to theLoad balancing page.

    Go to Load balancing

  2. ClickCreate load balancer.
  3. ForType of load balancer, selectApplication Load Balancer (HTTP/HTTPS) and clickNext.
  4. ForPublic facing or internal, selectInternal and clickNext.
  5. ForCross-region or single region deployment, selectBest for cross-region workloads and clickNext.
  6. ClickConfigure.

Basic configuration

  1. Provide aName for the load balancer.
  2. ForNetwork, selectNETWORK.

Configure the frontend with two forwarding rules

For HTTP:

  1. ClickFrontend configuration.
    1. Provide aName for the forwarding rule.
    2. In theSubnetwork region list, selectREGION_A.

      Reserve a proxy-only subnet

      3. If you already configured theproxy-only subnet, theReserve subnet button isn't displayed.You can continue with the next steps.
    3. In theSubnetwork list, selectSUBNET_A.
    4. In theIP address list, clickCreate IP address. TheReserve a static internal IP address page opens.
      • Provide aName for the static IP address.
      • In theStatic IP address list, selectLet me choose.
      • In theCustom IP address field, enter10.1.2.99.
      • SelectReserve.
  2. ClickDone.
  3. To add the second forwarding rule, clickAdd frontend IP and port.
    1. Provide aName for the forwarding rule.
    2. In theSubnetwork region list, selectREGION_B.

      Reserve a proxy-only subnet

      3. Because you have already configured theproxy-only subnet, theReserve subnet button isn't displayed.You can continue with the next steps.
    3. In theSubnetwork list, selectSUBNET_B.
    4. In theIP address list, clickCreate IP address. TheReserve a static internal IP address page opens.
      • Provide aName for the static IP address.
      • In theStatic IP address list, selectLet me choose.
      • In theCustom IP address field, enter10.1.3.99.
      • SelectReserve.
  4. ClickDone.

For HTTPS:

To assign an SSL certificate to the target HTTPS proxy of the load balancer, you need to use a Certificate Manager certificate.

  1. ClickFrontend configuration.
    1. Provide aName for the forwarding rule.
    2. In theProtocol field, selectHTTPS (includes HTTP/2).
    3. Ensure that thePort is set to443.
    4. In theSubnetwork region list, selectREGION_A.

      Reserve a proxy-only subnet

      5. Because you have already configured theproxy-only subnet,theReserve subnet button isn't displayed.You can continue with the next steps.
    5. In theSubnetwork list, selectSUBNET_A.
    6. In theIP address list, clickCreate IP address. TheReserve a static internal IP address page opens.
      • Provide aName for the static IP address.
      • In theStatic IP address list, selectLet me choose.
      • In theCustom IP address field, enter10.1.3.99.
      • SelectReserve.
    7. ClickAdd certificate to select an existing certificate or create a new certificate.

      If you already have an existing Certificate Manager certificate to select, do the following:

      1. ClickAdd Certificate.
      2. ClickSelect an existing certificate and select the certificate from the list of certificates.
      3. ClickSelect.

      After you select the new Certificate Manager certificate, it appears in the list of certificates.

      To create a new Certificate Manager certificate, do the following:

      1. ClickAdd Certificate.
      2. ClickCreate a new certificate.
      3. To create a new certificate, follow the steps starting fromstep 3 as outlined in any one of the following configuration methods in the Certificate Manager documentation:
    8. Select an SSL policy from theSSL policy list. If you have not created any SSL policies, adefault Google Cloud SSL policy is applied.
    9. ClickDone.

    Add the second frontend configuration:

    1. Provide aName for the frontend configuration.
    2. In theProtocol field, selectHTTPS (includes HTTP/2).
    3. Ensure that thePort is set to443.
    4. In theSubnetwork region list, selectREGION_B.

      Reserve a proxy-only subnet

      5. Because you have already configured theproxy-only subnet,theReserve subnet button isn't displayed.You can continue with the next steps.
    5. In theSubnetwork list, selectSUBNET_B.
    6. In theIP address list, clickCreate IP address. TheReserve a static internal IP address page opens.
      • Provide aName for the static IP address.
      • In theStatic IP address list, selectLet me choose.
      • In theCustom IP address field, enter10.1.3.99.
      • SelectReserve.
    7. ClickAdd certificate and then select an existing certificateor create a new certificate.
    8. Select an SSL policy from theSSL policy list. If you have not created any SSL policies, adefault Google Cloud SSL policy is applied.
    9. ClickDone.
    Configure the backend service
    1. ClickBackend configuration.
    2. In theCreate or select backend services list, clickCreate a backend service.
    3. Provide aName for the backend service.
    4. ForProtocol, selectHTTP.
    5. ForNamed Port, enterhttp.
    6. In theBackend type list, selectServerless network endpoint group.
    7. In theNew backend section:
      1. In theServerless network endpoint group list, selectgl7ilb-serverless-neg-a.
      2. ClickDone.
      3. To add another backend, clickAdd backend.
      4. In theServerless network endpoint group list, selectgl7ilb-serverless-neg-b.
      5. ClickDone.

    Configure the routing rules

    1. ClickRouting rules.
    2. ForMode, selectSimple host and path rule.
    3. Ensure that there is only one backend service for any unmatched host and any unmatched path.

    Review the configuration

    1. ClickReview and finalize.
    2. Review your load balancer configuration settings.
    3. ClickCreate.

gcloud

  1. Define the backend service with thegcloud compute backend-servicescreate command.

    gcloud compute backend-services create gil7-backend-service \  --load-balancing-scheme=INTERNAL_MANAGED \  --protocol=HTTP \  --global

  2. Add backends to the backend service with thegcloud compute backend-servicesadd-backend command.

    gcloud compute backend-services add-backend gil7-backend-service \  --network-endpoint-group=gl7ilb-serverless-neg-a \  --network-endpoint-group-region=REGION_A \  --global
    gcloud compute backend-services add-backend gil7-backend-service \  --network-endpoint-group=gl7ilb-serverless-neg-b \  --network-endpoint-group-region=REGION_B \  --global

  3. Create the URL map with thegcloud compute url-mapscreate command.

    gcloud compute url-maps create gil7-map \  --default-service=gil7-backend-service \  --global

  4. Create the target proxy.

    For HTTP:

    Create the target proxy with thegcloud compute target-http-proxiescreate command.

    gcloud compute target-http-proxies create gil7-http-proxy \  --url-map=gil7-map \  --global

    For HTTPS:

    To create a Google-managed certificate, see the following documentation:

    After you create the Google-managed certificate,attach the certificate directly to the target proxy.Certificate maps are not supported by cross-region internal Application Load Balancers.

    To create a self-managed certificate, see the following documentation:

    Assign your file paths to variable names.

    export LB_CERT=PATH_TO_PEM_FORMATTED_FILE
    export LB_PRIVATE_KEY=PATH_TO_LB_PRIVATE_KEY_FILE

    Create an all region SSL certificate using thegcloud certificate-managercertificates create command.

    gcloud certificate-manager certificates create gilb-certificate \  --private-key-file=$LB_PRIVATE_KEY \  --certificate-file=$LB_CERT \  –-scope=all-regions

    Use the SSL certificate to create a target proxy with thegcloudcompute target-https-proxies create command

    gcloud compute target-https-proxies create gil7-https-proxy \  --url-map=gil7-map \  --certificate-manager-certificates=gilb-certificate

  5. Create two forwarding rules: one with a VIP (10.1.2.99) in theREGION_B region andanother one with a VIP (10.1.3.99) in theREGION_A region.

    For custom networks, you must reference the subnet in the forwardingrule. Note that this is the virtual machine (VM) instance subnet, not theproxy subnet.

    For HTTP:

    Use thegcloud compute forwarding-rulescreate commandwith the correct flags.

    gcloud compute forwarding-rules create gil7-forwarding-rule-a \  --load-balancing-scheme=INTERNAL_MANAGED \  --network=NETWORK \  --subnet=SUBNET_B \  --subnet-region=REGION_B \  --address=10.1.3.99 \  --ports=80 \  --target-http-proxy=gil7-http-proxy \  --global
    gcloud compute forwarding-rules create gil7-forwarding-rule-b \  --load-balancing-scheme=INTERNAL_MANAGED \  --network=NETWORK \  --subnet=SUBNET_A \  --subnet-region=REGION_A \  --address=10.1.2.99 \  --ports=80 \  --target-http-proxy=gil7-http-proxy \  --global

    For HTTPS:

    Create the forwarding rule with thegcloud compute forwarding-rulescreate commandwith the correct flags.

    gcloud compute forwarding-rules create gil7-forwarding-rule-a \  --load-balancing-scheme=INTERNAL_MANAGED \  --network=NETWORK \  --subnet=SUBNET_B \  --address=10.1.3.99 \  --ports=443 \  --target-https-proxy=gil7-https-proxy \  --global
    gcloud compute forwarding-rules create gil7-forwarding-rule-b \  --load-balancing-scheme=INTERNAL_MANAGED \  --network=NETWORK \  --subnet=SUBNET_A \  --address=10.1.2.99 \  --ports=443 \  --target-https-proxy=gil7-https-proxy \  --global

API

Create the global backend service by making aPOST request to thebackendServices.insert method,replacingPROJECT_ID with your project ID.

POST https://compute.googleapis.com/compute/v1/projects/PROJECT_ID/global/backendServices{"name": "gil7-backend-service","backends": [  {    "group": "projects/PROJECT_ID/zones/ZONE_A/instanceGroups/gl7ilb_serverless_negwest",    "balancingMode": "UTILIZATION"  },  {    "group": "projects/PROJECT_ID/zones/ZONE_B/instanceGroups/gl7ilb_serverless_negeast",  }],"loadBalancingScheme": "INTERNAL_MANAGED"}

Create the URL map by making aPOST request to theurlMaps.insert method,replacingPROJECT_ID with your project ID.

POST https://compute.googleapis.com/compute/v1/projects/PROJECT_ID/global/urlMaps{"name": "l7-ilb-map","defaultService": "projects/PROJECT_ID/global/backendServices/gil7-backend-service"}

For HTTP:

Create the target HTTP proxy by making aPOST request to thetargetHttpProxies.insert method,replacingPROJECT_ID with your project ID.

POST https://compute.googleapis.com/compute/v1/projects/PROJECT_ID/global/targetHttpProxy{"name": "l7-ilb-proxy","urlMap": "projects/PROJECT_ID/global/urlMaps/l7-ilb-map"}

Create the forwarding rule by making aPOST request to theglobalforwardingRules.insert method,replacingPROJECT_ID with your project ID.

POST https://compute.googleapis.com/compute/v1/projects/PROJECT_ID/global/forwardingRules{"name": "gil7-forwarding-rule-a","IPAddress": "10.1.2.99","IPProtocol": "TCP","portRange": "80-80","target": "projects/PROJECT_ID/global/targetHttpProxies/l7-ilb-proxy","loadBalancingScheme": "INTERNAL_MANAGED","subnetwork": "projects/PROJECT_ID/regions/REGION_A/subnetworks/SUBNET_A","network": "projects/PROJECT_ID/global/networks/NETWORK","networkTier": "PREMIUM"}
POST https://compute.googleapis.com/compute/v1/projects/PROJECT_ID/global/forwardingRules{"name": "gil7-forwarding-rule-b","IPAddress": "10.1.3.99","IPProtocol": "TCP","portRange": "80-80","target": "projects/PROJECT_ID/global/targetHttpProxies/l7-ilb-proxy","loadBalancingScheme": "INTERNAL_MANAGED","subnetwork": "projects/PROJECT_ID/regions/REGION_B/subnetworks/SUBNET_B","network": "projects/PROJECT_ID/global/networks/NETWORK","networkTier": "PREMIUM"}

For HTTPS:

Read the certificate and private key files, and then create the SSLcertificate. The following example showshow to do this with Python.

Create the target HTTPS proxy by making aPOST request to thetargetHttpsProxies.insert method,replacingPROJECT_ID with your project ID.

POST https://compute.googleapis.com/compute/v1/projects/PROJECT_ID/global/targetHttpsProxy{"name": "l7-ilb-proxy","urlMap": "projects/PROJECT_ID/global/urlMaps/l7-ilb-map","sslCertificates": /projects/PROJECT_ID/global/sslCertificates/SSL_CERT_NAME}

Create the forwarding rule by making aPOST request to theglobalForwardingRules.insert method,replacingPROJECT_ID with your project ID.

POST https://compute.googleapis.com/compute/v1/projects/PROJECT_ID/global/forwardingRules{"name": "gil7-forwarding-rule-a","IPAddress": "10.1.2.99","IPProtocol": "TCP","portRange": "80-80","target": "projects/PROJECT_ID/global/targetHttpsProxies/l7-ilb-proxy","loadBalancingScheme": "INTERNAL_MANAGED","subnetwork": "projects/PROJECT_ID/regions/REGION_A/subnetworks/SUBNET_A","network": "projects/PROJECT_ID/global/networks/NETWORK","networkTier": "PREMIUM"}
POST https://compute.googleapis.com/compute/v1/projects/PROJECT_ID/global/forwardingRules{"name": "gil7-forwarding-rule-b","IPAddress": "10.1.3.99","IPProtocol": "TCP","portRange": "80-80","target": "projects/PROJECT_ID/global/targetHttpsProxies/l7-ilb-proxy","loadBalancingScheme": "INTERNAL_MANAGED","subnetwork": "projects/PROJECT_ID/regions/REGION_B/subnetworks/SUBNET_B","network": "projects/PROJECT_ID/global/networks/NETWORK","networkTier": "PREMIUM"}

Test the load balancer

Now that the load balancing service is running, you can send traffic to theforwarding rule and observe the traffic being dispersed to different instances.

Configure the firewall rule

This example requires thefw-allow-ssh firewall rule for the test client VM.fw-allow-ssh is an ingress rule that is applicable to the test client VM andthat allows incoming SSH connectivity on TCP port22 from anyaddress. You can choose a more restrictive source IP address range for thisrule; for example, you can specify just the IP address ranges of the system fromwhich you initiate SSH sessions. This example uses the target tagallow-ssh.

gcloud

  1. Create thefw-allow-ssh firewall rule to allow SSH connectivity toVMs with the network tagallow-ssh. When you omitsource-ranges,Google Cloudinterprets the rule to mean anysource.

    gcloud compute firewall-rules create fw-allow-ssh \    --network=NETWORK \    --action=allow \    --direction=ingress \    --target-tags=allow-ssh \    --rules=tcp:22

Create a VM instance to test connectivity

  1. Create a client VM:

    gcloud compute instances create l7-ilb-client-a \    --image-family=debian-12 \    --image-project=debian-cloud \    --network=NETWORK \    --subnet=SUBNET_A \    --zone=ZONE_A \    --tags=allow-ssh
    gcloud compute instances create l7-ilb-client-b \    --image-family=debian-12 \    --image-project=debian-cloud \    --network=NETWORK \    --subnet=SUBNET_B \    --zone=ZONE_B \    --tags=allow-ssh

  2. Connect, using SSH, to each client instance.

    gcloud compute ssh l7-ilb-client-a \   --zone=ZONE_A
    gcloud compute ssh l7-ilb-client-b \   --zone=ZONE_B

  3. Verify that the IP address is serving its hostname.

    • Verify that the client VM can reach both IP addresses. The command succeedsand returns the name of the backend VM that served the request:

      For HTTP testing:

      curl 10.1.2.99
      curl 10.1.3.99

      For HTTPS testing:

      curl -k -s 'https://DOMAIN_NAME:443' --connect-toDOMAIN_NAME:443:10.1.2.99:443
      curl -k -s 'https://DOMAIN_NAME:443' --connect-toDOMAIN_NAME:443:10.1.3.99:443

      ReplaceDOMAIN_NAME with your application domain name, forexample,test.example.com.

      The-k flag causes curl to skip certificate validation.

    • Optional: Use the configuredDNS record to resolvethe IP address.

      curl service.example.com

Run 100 requests and confirm that they are load balanced

For HTTP:

  {    RESULTS=    for i in {1..100}    do      RESULTS="$RESULTS:$(curl --silent 10.1.2.99)"    done    echo ""    echo " Results of load-balancing to 10.1.2.99: "    echo "***"    echo "$RESULTS" | tr ':' '\n' | grep -Ev "^$" | sort | uniq -c    echo  }

  {    RESULTS=    for i in {1..100}    do      RESULTS="$RESULTS:$(curl --silent 10.1.3.99)"    done    echo ""    echo " Results of load-balancing to 10.1.3.99: "    echo "***"    echo "$RESULTS" | tr ':' '\n' | grep -Ev "^$" | sort | uniq -c    echo  }

For HTTPS:

ReplaceDOMAIN_NAME with your application domain name, for example,test.example.com.

  {    RESULTS=    for i in {1..100}    do      RESULTS="$RESULTS:$(curl -k -s 'https://DOMAIN_NAME:443' --connect-toDOMAIN_NAME:443:10.1.2.99:443)"    done    echo ""    echo " Results of load-balancing to 10.1.2.99: "    echo "***"    echo "$RESULTS" | tr ':' '\n' | grep -Ev "^$" | sort | uniq -c    echo  }

  {    RESULTS=    for i in {1..100}    do        RESULTS="$RESULTS:$(curl -k -s 'https://DOMAIN_NAME:443' --connect-toDOMAIN_NAME:443:10.1.3.99:443)"    done    echo ""    echo " Results of load-balancing to 10.1.3.99: "    echo "***"    echo "$RESULTS" | tr ':' '\n' | grep -Ev "^$" | sort | uniq -c    echo  }

Test failover

  1. Verify failover to backends in theREGION_Aregion when backends in theREGION_B regions are unhealthy orunreachable. We simulate this by removing all of thebackends fromREGION_B:

    gcloud compute backend-services remove-backend gil7-backend-service \   --network-endpoint-group=gl7ilb-serverless-neg-b \   --network-endpoint-group-zone=ZONE_B

  2. Connect, using SSH, to a client VM inREGION_B.

    gcloud compute ssh l7-ilb-client-b \   --zone=ZONE_B

  3. Send requests to the load balanced IP address in theREGION_B region.The command output displays responses from backend VMs inREGION_A.

    ReplaceDOMAIN_NAME with your application domain name, forexample,test.example.com.

    {RESULTS=for i in {1..100}do  RESULTS="$RESULTS:$(curl -k -s 'https://DOMAIN_NAME:443' --connect-toDOMAIN_NAME:443:10.1.3.99:443)"doneecho "***"echo "*** Results of load-balancing to 10.1.3.99: "echo "***"echo "$RESULTS" | tr ':' '\n' | grep -Ev "^$" | sort | uniq -cecho}

Additional configuration options

This section expands on the configuration example to provide alternative andadditional configuration options. All of the tasks are optional. You can performthem in any order.

Using a URL mask

When creating a serverless NEG, instead of selecting a specificCloud Run service, you can use a URL mask to point tomultiple services serving at the same domain. A URL mask is a template of yourURL schema. The serverless NEG uses this template to extract the servicename from the incoming request's URL and map the request to the appropriateservice.

URL masks are particularly useful if your service ismapped to a customdomain rather than to the default addressthat Google Cloud provides for the deployed service. A URL mask lets youtarget multiple services and versions with a single rule even when yourapplication is using a custom URL pattern.

If you haven't already done so, make sure you read theServerless NEGSoverview: URLMasks.

Construct a URL mask

To construct a URL mask for your load balancer, start with the URL of yourservice. This example uses a sample serverless app running athttps://example.com/login. This is the URL where the app'slogin serviceis served.

  1. Remove thehttp orhttps from the URL. You are left withexample.com/login.
  2. Replace the service name with a placeholder for the URL mask.
    • Cloud Run: Replace theCloud Run service name with theplaceholder<service>. If the Cloud Runservice has a tag associated with it,replace the tag name with the placeholder<tag>.In this example, the URL mask you are left with isexample.com/<service>.

  3. Optional: If the service name can beextracted from the path portion of the URL, the domain can be omitted. Thepath part of the URL mask is distinguishedby the first slash (/) character. If a slash (/) is not present in theURL mask, the mask is understood to represent the host only. Therefore, forthis example, the URL mask can be reduced to/<service>.

    Similarly, if<service> can be extracted from the host part of theURL, you can omit the path altogether from the URL mask.

    You can also omitany host or subdomain components that come before the first placeholder aswell as any path components that come after the last placeholder. In suchcases, the placeholder captures the required information for the component.

Here are a few more examples that demonstrate these rules:

This table assumes that you have a custom domain calledexample.com andall your Cloud Run services are beingmappedto this domain.

Service, Tag nameCloud Run custom domain URLURL mask
service: loginhttps://login-home.example.com/web<service>-home.example.com
service: loginhttps://example.com/login/webexample.com/<service> or /<service>
service: login, tag: testhttps://test.login.example.com/web<tag>.<service>.example.com
service: login, tag: testhttps://example.com/home/login/testexample.com/home/<service>/<tag> or /home/<service>/<tag>
service: login, tag: testhttps://test.example.com/home/login/web<tag>.example.com/home/<service>

Creating a serverless NEG with a URL mask

Console

For a new load balancer, you can use the same end-to-end process asdescribed previously in this document. When configuring the backend service,instead of selecting a specific service, enter a URL mask.

If you have an existing load balancer, you can edit the backend configurationand have the serverless NEG point to a URL mask instead of a specific service.

To add a URL mask-based serverless NEG to an existing backend service,do the following:

  1. In the Google Cloud console, go to theLoad balancing page.
    Go to Load balancing
  2. Click the name of the load balancer that has the backend service you want to edit.
  3. On theLoad balancer details page, clickEdit.
  4. On theEdit global external Application Load Balancer page, clickBackend configuration.
  5. On theBackend configuration page, clickEdit for the backend service you want to modify.
  6. ClickAdd backend.
  7. SelectCreate Serverless network endpoint group.
    1. For theName, enterhelloworld-serverless-neg.
    2. UnderRegion, the region of the load balancer is displayed.
    3. UnderServerless network endpoint group type,Cloud Run is the only supported network endpoint group type.
      1. SelectUse URL Mask.
      2. Enter a URL mask. For information about how to create a URL mask, seeConstructing a URL mask.
      3. ClickCreate.

  8. In theNew backend, clickDone.
  9. ClickUpdate.

gcloud

To create a serverless NEG with a sample URL mask ofexample.com/<service>:

gcloud compute network-endpoint-groups createSERVERLESS_NEG_MASK_NAME \    --region=REGION \    --network-endpoint-type=serverless \    --cloud-run-url-mask="example.com/<service>"

Use the same IP address between multiple internal forwarding rules

For multiple internal forwarding rules to share the same internal IP address,you must reserve the IP address and set its--purpose flag toSHARED_LOADBALANCER_VIP.

gcloud

gcloud compute addresses createSHARED_IP_ADDRESS_NAME \    --region=REGION \    --subnet=SUBNET_NAME \    --purpose=SHARED_LOADBALANCER_VIP
If you need to redirect HTTP traffic to HTTPS, you can create two forwardingrules that use a common IP address. For more information, seeSet upHTTP-to-HTTPS redirect forinternal Application Load Balancers.

Configure DNS routing policies

If your clients are in multiple regions, you might want to make yourcross-region internal Application Load Balancer accessible by using VIPs in these regions. You can useDNS routing policies of typeGEO to route clienttraffic to the load balancer VIP in the region closest to the client. Thismulti-region setup minimizes latency and network transit costs. In addition, itlets you set up a DNS-based, global, load balancing solution that providesresilience against regional outages.

Cloud DNS supports health checking and enables automatic failover whenthe endpoints fail their health checks. During a failover, Cloud DNSautomatically adjusts the traffic split among the remaining healthy endpoints.For more information, seeManage DNS routing policies and healthchecks.

gcloud

To create a DNS entry with a 30 second TTL, use thegcloud dns record-sets create command.

gcloud dns record-sets createDNS_ENTRY --ttl="30" \  --type="A" --zone="service-zone" \  --routing-policy-type="GEO" \  --routing-policy-data="REGION_A=gil7-forwarding-rule-a@global;REGION_B=gil7-forwarding-rule-b@global" \  --enable-health-checking

Replace the following:

  • DNS_ENTRY: DNS or domain name of the record-set

    For example,service.example.com

  • REGION_A andREGION_B:the regions where you have configured the load balancer

API

Create the DNS record by making aPOST request to theResourceRecordSets.create method.ReplacePROJECT_ID with your project ID.

POST https://www.googleapis.com/dns/v1/projects/PROJECT_ID/managedZones/SERVICE_ZONE/rrsets{  "name": "DNS_ENTRY",  "type": "A",  "ttl": 30,  "routingPolicy": {    "geo": {      "items": [        {          "location": "REGION_A",          "healthCheckedTargets": {            "internalLoadBalancers": [              {                "loadBalancerType": "globalL7ilb",                "ipAddress": "IP_ADDRESS",                "port": "80",                "ipProtocol": "tcp",                "networkUrl": "https://www.googleapis.com/compute/v1/projects/PROJECT_ID/global/networks/lb-network",                "project": "PROJECT_ID"              }            ]          }        },        {          "location": "REGION_B",          "healthCheckedTargets": {            "internalLoadBalancers": [              {                "loadBalancerType": "globalL7ilb",                "ipAddress": "IP_ADDRESS_B",                "port": "80",                "ipProtocol": "tcp",                "networkUrl": "https://www.googleapis.com/compute/v1/projects/PROJECT_ID/global/networks/lb-network",                "project": "PROJECT_ID"              }            ]          }        }      ]    }  }}

Enable outlier detection

You can enableoutlierdetectionon global backend services to identify unhealthy serverless NEGs and reduce thenumber the requests sent to the unhealthy serverless NEGs.

Outlier detection is enabled on the backend service by using one of the following methods:

  • TheconsecutiveErrors method (outlierDetection.consecutiveErrors), inwhich a5xx series HTTP status code qualifies as an error.
  • TheconsecutiveGatewayFailure method(outlierDetection.consecutiveGatewayFailure), in which only the502,503, and504 HTTP status codes qualify as an error.

Use the following steps to enable outlier detection for an existing backendservice. Note that even after enabling outlier detection, some requests can besent to the unhealthy service and return a5xx status code tothe clients. To further reduce the error rate, you can configure more aggressivevalues for the outlier detection parameters. For more information, see theoutlierDetection field.

Console

  1. In the Google Cloud console, go to theLoad balancing page.

    Go to Load balancing

  2. Click the name of the load balancer whose backend service you want toedit.

  3. On theLoad balancer details page, clickEdit.

  4. On theEdit cross-region internal Application Load Balancer page, clickBackend configuration.

  5. On theBackend configuration page, clickEdit for the backend service thatyou want to modify.

  6. Scroll down and expand theAdvanced configurations section.

  7. In theOutlier detection section, select theEnable checkbox.

  8. ClickEdit to configureoutlier detection.

    Verify that the following options are configured with these values:

    PropertyValue
    Consecutive errors5
    Interval1000
    Base ejection time30000
    Max ejection percent50
    Enforcing consecutive errors100

    In this example, the outlier detection analysis runs every one second. Ifthe number of consecutive HTTP5xx status codesreceived by anEnvoy proxy is five or more, the backend endpoint is ejected from theload-balancing pool of that Envoy proxy for 30 seconds. When theenforcing percentage is set to 100%, the backend service enforces theejection of unhealthy endpoints from the load-balancing pools of thosespecific Envoy proxies every time the outlier detection analysis runs. Ifthe ejection conditions are met, up to 50% of the backend endpoints fromthe load-balancing pool can be ejected.

  9. ClickSave.

  10. To update the backend service, clickUpdate.

  11. To update the load balancer, on theEdit cross-region internal Application Load Balancer page, clickUpdate.

gcloud

  1. Export the backend service into a YAML file.

    gcloud compute backend-services exportBACKEND_SERVICE_NAME \  --destination=BACKEND_SERVICE_NAME.yaml --global

    ReplaceBACKEND_SERVICE_NAME with the name of thebackend service.

  2. Edit the YAML configuration of the backend service to add the fields foroutlier detection as highlighted in the following YAML configuration,in theoutlierDetection section:

    In this example, the outlier detection analysis runs every one second. Ifthe number of consecutive HTTP5xx status codesreceived by anEnvoy proxy is five or more, the backend endpoint is ejected from theload-balancing pool of that Envoy proxy for 30 seconds. When theenforcing percentage is set to 100%, the backend service enforces theejection of unhealthy endpoints from the load-balancing pools of thosespecific Envoy proxies every time the outlier detection analysis runs. Ifthe ejection conditions are met, up to 50% of the backend endpoints fromthe load-balancing pool can be ejected.

    name:BACKEND_SERVICE_NAMEbackends:- balancingMode: UTILIZATION  capacityScaler: 1.0  group: https://www.googleapis.com/compute/v1/projects/PROJECT_ID/regions/REGION_A/networkEndpointGroups/SERVERLESS_NEG_NAME- balancingMode: UTILIZATION  capacityScaler: 1.0  group: https://www.googleapis.com/compute/v1/projects/PROJECT_ID/regions/REGION_B/networkEndpointGroups/SERVERLESS_NEG_NAME_2outlierDetection:  baseEjectionTime:    nanos: 0    seconds: 30  consecutiveErrors: 5  enforcingConsecutiveErrors: 100  interval:    nanos: 0    seconds: 1  maxEjectionPercent: 50port: 80selfLink: https://www.googleapis.com/compute/v1/projects/PROJECT_ID/global/backendServices/BACKEND_SERVICE_NAMEsessionAffinity: NONEtimeoutSec: 30...

    Replace the following:

    • BACKEND_SERVICE_NAME: the name of the backendservice
    • PROJECT_ID: the ID of your project
    • REGION_A andREGION_B:the regions where the load balancer has been configured.
    • SERVERLESS_NEG_NAME: the name of the firstserverless NEG
    • SERVERLESS_NEG_NAME_2: the name of the secondserverless NEG

  3. Update the backend service by importing the latest configuration.

    gcloud compute backend-services importBACKEND_SERVICE_NAME \  --source=BACKEND_SERVICE_NAME.yaml --global

    Outlier detection is now enabled on the backend service.

Deleting a serverless NEG

A network endpoint group cannot be deleted if it is attached to a backendservice. Before you delete a NEG, ensure that it is detached from thebackend service.

Console

  1. To make sure the serverless NEG you want to delete is not in use by any backend service, go to theBackend services tab on theLoad balancing components page.
    Go to Backend services
  2. If the serverless NEG is in use, do the following:
    1. Click the name of the backend service that is using the serverless NEG.
    2. ClickEdit.
    3. From the list ofBackends, click to remove the serverless NEG backend from the backend service.
    4. ClickSave.

  3. Go to theNetwork endpoint group page in the Google Cloud console.
    Go to Network endpoint group
  4. Select the checkbox for the serverless NEG you want to delete.
  5. ClickDelete.
  6. ClickDelete again to confirm.

gcloud

To remove a serverless NEG from a backend service, you must specifythe region where the NEG was created.

gcloud compute backend-services remove-backendBACKEND_SERVICE_NAME \    --network-endpoint-group=SERVERLESS_NEG_NAME \    --network-endpoint-group-region=REGION \    --region=REGION

To delete the serverless NEG:

gcloud compute network-endpoint-groups deleteSERVERLESS_NEG_NAME \    --region=REGION

What's next

Except as otherwise noted, the content of this page is licensed under theCreative Commons Attribution 4.0 License, and code samples are licensed under theApache 2.0 License. For details, see theGoogle Developers Site Policies. Java is a registered trademark of Oracle and/or its affiliates.

Last updated 2025-12-15 UTC.