An internal passthrough Network Load Balancer is a regional load balancer that enables you to run andscale your services behind an internal IP address. You can use aninternal passthrough Network Load Balancer as the next hop to which packets are forwarded along thepath to their final destination. To do this, you set the load balancer as thenext hop in astatic route.

Before reviewing the information on this page, you should already be familiarwith concepts from the following:

• Routes overview
• Internal passthrough Network Load Balancer overview

An internal passthrough Network Load Balancer next hop is useful in the following cases:

• To load balance traffic across multiple VMs that are functioningas gateway or router VMs.

• To use gatewayvirtualappliancesas your next hop for a default route. With this configuration, virtual machine(VM) instances in your Virtual Private Cloud (VPC) network send traffic to theinternet through a set of load-balanced virtual gateway VMs.

• To send traffic through multiple load balancers in two or moredirections by using the same set of multi-NIC gateway or router VMs asbackends. To do this, you create a load balancer and use it as the next hopfor a static route ineach VPC network. Eachinternal passthrough Network Load Balancer operates within a single VPC network,distributing traffic to the network interfaces of backend VMs in that network.

Architecture

In the following diagram, a VM instance group of router VMs serves as thebackend for two different load balancers.The first internal passthrough Network Load Balancer sends packets tonic0 of the backendVMs, and the second internal passthrough Network Load Balancer sends packets tonic1 on thesame backends.

Benefits of using your internal passthrough Network Load Balancer as a next hop

When the load balancer is a next hop for a static route, no specialconfiguration is needed within the guest operating systems of the client VMs inthe VPC network where the route is defined.Client VMs send packets to the load balancer's backends throughVPC network routing, in abump-in-the-wirefashion.

Using an internal passthrough Network Load Balancer as a next hop for a static routeprovides the same benefits as a standalone internal passthrough Network Load Balancer. The loadbalancer's health check ensures that new connections are routed to healthybackend VMs. By using a managed instance group as a backend, you canconfigure autoscaling to grow or shrink the set of VMs based on servicedemand.

Specifications

The following are specifications for using internal passthrough Network Load Balancers as next hops.

Routes

You can create a static route to passTCP, UDP, and other protocoltraffic to an internal passthrough Network Load Balancer where the load balancer is thenext hop for the static route. The route can be an external (publicly routable)CIDR prefix or an internal CIDR prefix, if the prefix doesn't conflict with asubnet route. Forexample, you can replace your default route (0.0.0.0/0) with a route thatdirects traffic to third-party backend VMs for packet processing.

Caution: When sending traffic to Google APIs and services, don't route thetraffic through a next-hop VM or through a next-hop internal passthrough Network Load Balancer.Instead, route such traffic through a next-hop default internetgateway. This includes traffic to Google APIs and services sent by means ofPrivate Google Access.

Options for specifying the next hop

You can specify an internal passthrough Network Load Balancer next hop in one of two ways:

• By using the forwarding rule's name and region
• By using the forwarding rule's IP address

For details about the project and VPC network in which aninternal passthrough Network Load Balancer next hop can reside, seeNext hops andfeatures.

You can exchange static routes with internal passthrough Network Load Balancer next hops usingVPC Network Peering. For details, seeOptions for exchanging staticroutes.

Note: A load balancer whose forwarding rule uses theL3_DEFAULT protocol cannot be the next hop for a static route. If this static route is created, traffic is silently dropped.

Client IP session affinity

Internal passthrough Network Load Balancers offer two similar "client IP address"sessionaffinity options:

• Client IP (CLIENT_IP): A two-tuple hash of a packet's source IP addressand destination IP address. When an internal passthrough Network Load Balancer isnot a route's nexthop, packets sent to the load balancer's forwarding rule IP address share acommon destination IP address—the forwarding rule IP address. In thissituation, one of the addresses used by the two-tuple hash remains constant.Thus, if the number of configured and healthy backends doesn't change andpackets have identical source IP addresses, this two-tuple session affinityoption selects the same backend.
• Client IP, no destination (CLIENT_IP_NO_DESTINATION): A one-tuple hashof a packet's source IP address. When you are usingan internal passthrough Network Load Balancer as a next hop, the destination IP address often variesbecause destination IP addresses are those specified by the route'sdestination attribute. In this situation, the two-tuple hashClient IP (CLIENT_IP) session affinity cannot select the same backendeven when the number of configured and healthy backends doesn't change andpackets have identical source IP addresses. (An exception to this rule is whenonly one backend is configured.) If you require packets with identical sourceIP addresses to be routed to the same backend, you must use theClient IP,no destination (CLIENT_IP_NO_DESTINATION) session affinity option.

Destination range

The destination of astatic routecannot be equal to or more specific than asubnetroute. Note thatmore specific means that thesubnet mask is longer. This rule applies to all static routes, includingwhen the next hop is an internal passthrough Network Load Balancer. For example, suppose that yoursubnet route is10.140.0.0/20. The destination of a static route can'tbe the same (10.140.0.0/20), and it can't be more specific, as in10.140.0.0/22.

Same VPC network and region

Static routes that use internal passthrough Network Load Balancers as next hops are limited tothe following:

• A single VPC network. The load balancer and thestatic route must be in the same VPC network.

• A single region or all regions. Unless you configureglobalaccess, the static routeis only available to resources in the same region as the load balancer. Thisregional restriction is enforced even though the route itself is part of therouting table for the entire VPC network. If you enableglobal access, the static route is available to resources in any region.

Advertising the static route

To advertise the prefix (destination) for the static route, you can useCloud Routercustom advertisementmode. The scope of the routeadvertisement depends on the load balancer's global access setting, as follows:

• When global access is disabled, the internal passthrough Network Load Balancer is only available toVMs, Cloud VPN tunnels, and Cloud Interconnect attachments(VLANs) that are in the same region as the load balancer. Consequently, acustom route advertisement for a static route's prefix only makes senseif the Cloud Router and load balancer are in the same region.

• When global access is enabled, the internal passthrough Network Load Balancer is available to VMs,Cloud VPN tunnels, and Cloud Interconnect attachments(VLANs) that are in any region. With global dynamic routing, on-premisessystems can use the static route from any connected region.

The following table summarizes the accessibility of the load balancer.

Global access	VPC network dynamic routing mode	Load balancer access
Disabled	Regional	Accessible by routers in thesame region
Disabled	Global	Accessible by routers in thesame region
Enabled	Regional	Accessible by all routers inany region
Enabled	Global	Accessible by all routers inany region

For moreinformation, seeInternal passthrough Network Load Balancers and connectednetworks.

Order of operations

You must create an internal passthrough Network Load Balancer before you can create astatic route that uses it as a next hop. The load balancer must existbefore you can create the route. If you try to create a route that refers to anonexistent load balancer, Google Cloud returns an error.

You specify an internal passthrough Network Load Balancer next hop by using the forwardingrule's name and the load balancer's region, or by using the internal IP addressassociated with the forwarding rule.

After you've created a route with a next hop that refers to an internal passthrough Network Load Balancer,you cannot delete the load balancer unless you first delete theroute. Specifically, you can't delete an internal forwarding rule until nostatic route uses that load balancer as a next hop.

Backend requirements

• You must configure all of the internal passthrough Network Load Balancer's backend VMsto allow IP forwarding (--can-ip-forward = True). For more information,seeConsiderations common to instance and internal passthrough Network Load Balancer nexthops.

• You cannot use an internal passthrough Network Load Balancer whose backends areGoogle Kubernetes Engine (GKE) nodes as a next hop for a static route.Software on the nodes can only route traffic to Pods if the destination matchesan IP address managed by the cluster, not an arbitrary destination.

Processing of TCP, UDP, and other protocol traffic

Important: The information in this section applies only to custom routescreated before May 15, 2021.

If an internal passthrough Network Load Balancer as next hop route was created before May 15, 2021,the load balancer forwards only TCP and UDP traffic on all ports to the backendVMs, regardless of the forwarding rule or backend service configuration. Allother traffic, such as ICMP pings, will be handled by the next most specificroute in your VPC network.

If a route created before May 15, 2021 was still in operation on August 16,2021, it was automatically migrated to forward all protocol trafficstarting August 16, 2021.

When an internal passthrough Network Load Balancer is deployed as a next hop, Google Cloudforwardsall traffic on all ports to the backend VMs, regardless of thefollowing:

• The forwarding rule's protocol and port configuration
• The backend service's protocol configuration

The internal passthrough Network Load Balancer, which is the next hop of the route, seamlessly supportsforwarding all the traffic for the protocols supported by Google CloudVPC networks (such as TCP, UDP, and ICMP).

Additional considerations

• Supported forwarding rules. Google Cloud supports only next hopinternal passthrough Network Load Balancer forwarding rules. Google Cloud doesnot support nexthop forwarding rules used by other load balancers, protocol forwarding, oras Private Service Connect endpoints.

• Specification methods and forwarding rule network and project. You canspecify a next hop forwarding rule by using one of following three methods.The specification method that you use determines whether the forwardingrule's network must match the route's network and in what project theforwarding rule can be located.

  Choose one of the following methods and ensure that the IP version of your forwarding rule matches the IP version of the static route that you create:

  • By forwarding rule name (--next-hop-ilb) and region(--next-hop-ilb-region): when you specify a next hop forwardingrule by name and region, the forwarding rule's network must match theroute's VPC network. The forwarding rule must be locatedin the same project that contains the forwarding rule'snetwork (a standalone project or a Shared VPC host project).

  • By forwarding rule resource link: a forwarding rule's resourcelink uses the format/projects/PROJECT_ID/regions/REGION/forwardingRules/FORWARDING_RULE_NAME, wherePROJECT_ID is theproject ID of the project that contains the forwarding rule,REGION is the forwarding rule's region, andFORWARDING_RULE_NAME is the forwarding rule'sname. Whenyou specify a next hop forwarding rule by its resource link, theforwarding rule's network must match the route's VPCnetwork. The forwarding rule can be located ineither the project thatcontains the forwarding rule's network (a standalone project or aShared VPC host project)or a Shared VPC serviceproject.

  • By a forwarding rule IP address: when you specify a next hopforwarding rule by its IPv4 or IPv6 address, the forwardingrule's network can beeither the route's VPC networkor a VPC network that's connected to the route'sVPC network by using either VPC Network Peering orNetwork Connectivity Center. Network Connectivity Center supports a next hop internal passthrough Network Load Balancerin a VPC spoke subject toNetwork Connectivity Centerrequirements for connectivity.The forwarding rule can be located ineither the project thatcontains the forwarding rule's network (a standalone project or aShared VPC host project)or a Shared VPC serviceproject.

• Effect of global access. Custom static routes using internal passthrough Network Load Balancernext hops are programmed in all regions. Whether the next hop is usabledepends on the load balancer'sglobalaccess setting. With globalaccess enabled, the load balancer next hop is accessible in all regions ofthe VPC network. With global access disabled, the loadbalancer next hop is only accessible in the same region as the loadbalancer. With global access disabled, packets sent from another region to aroute using an internal passthrough Network Load Balancer next hop are dropped.

• When all backends are unhealthy. When all backends of aninternal passthrough Network Load Balancer fail health checks, the routes using that load balancernext hop are still in effect. Packets processed by the route are sent to oneof the next hop load balancer's backends according totrafficdistribution.

• Forwarding rules that use a common internal IP address(--purpose=SHARED_LOADBALANCER_VIP) are not supported. Next hopinternal passthrough Network Load Balancers andinternal passthrough Network Load Balancer forwarding rules with a common IPaddressare mutually exclusive features. A next hop internal passthrough Network Load Balancer must use anIP address that is unique to the load balancer's forwarding rule so thatonly one backend service (one load balancer) is unambiguously referenced.It's possible for forwarding rules that use a common internal IP address toreference different backend services (different internal passthrough Network Load Balancers).

• Multiple routes with the same destinations and priorities, but differentnext hop internal passthrough Network Load Balancers. Google Cloudnever distributes trafficamong two or more next hop internal passthrough Network Load Balancers using ECMP. Instead,Google Cloud selects just one next hop internal passthrough Network Load Balancer using adeterministic, internal algorithm. To avoid this ambiguity, you can useunique network tags for each route.

  

• Multiple routes with the same destinations, priorities, and next hopinternal passthrough Network Load Balancers. Without a network tag, Google Cloud does notallow you to create multiple static routes that have the same combination ofdestination, priority, and internal passthrough Network Load Balancer next hop. With network tags, youcan create multiple static routes having the same combination of destination,priority, and internal passthrough Network Load Balancer next hop.

Use cases

You can use an internal passthrough Network Load Balancer as a next hop in multiple deployments andtopologies.

For each example, note the following guidelines:

• Each VM interfacemust be in a separate VPC network.

• You cannot use backend VMs or load balancers to route traffic between subnetsin the same VPC network because subnet routes cannot beoverridden.

• The internal passthrough Network Load Balancer is asoftware-defined pass-through loadbalancer. Packets are deliveredto backend VMs without alterations to source or destination information(addresses or addresses and ports).

  Routing, packet filtering, proxying, and address translation arethe responsibility of the virtual appliance VMs that serve as backends forthe internal passthrough Network Load Balancer.

Using an internal passthrough Network Load Balancer as the next hop to a NAT gateway

This use case load balances traffic from internal VMs to multiple NAT gatewayinstances that route traffic to the internet.

Hub and spoke: Exchanging next-hop routes by using VPC Network Peering

In addition to exchanging subnet routes, you can configureVPC Network Peering to export and import customstatic and dynamic routes. Static routes thathave a next hop of the default internet gateway are excluded. Customstatic routes that use next-hop internal passthrough Network Load Balancers are included.

You can configure a hub-and-spoke topology with your next-hop firewall virtualappliances located in thehub VPC network by doing thefollowing:

• In thehub VPC network, create an internal passthrough Network Load Balancerwith firewall virtual appliances as the backends.
• In thehub VPC network, create a static route, and setthe next hop to be the internal passthrough Network Load Balancer.
• Connect thehub VPC network to each of thespokeVPC networks by using VPC Network Peering.
• For each peering, configure thehub network to export its custom routes, andconfigure the correspondingspoke network to import custom routes. The routewith the load balancer next hop is one of the routes that thehub networkexports.

Subject to therouting order, the next hopfirewall appliance load balancer in thehub VPC network isavailable in the spoke networks:

• to clients in the same region as the load balancer, if global access isdisabled
• to clients in all regions, if global access is enabled, according to therouting order.

Load balancing to multiple NICs

In the following use case, the backend VMs are virtual appliance instances (forexample, packet inspection, routing, or gateway VMs) with NICs in multipleVPC networks. These virtual appliance instances can be commercialsolutions from third parties or solutions that you build yourself. The virtualappliances are Compute Engine VMs withmultiple NICs.

This example shows a single set of backend virtual appliances in a managed VMinstance group.

In the VPC network calledtesting, the internal passthrough Network Load Balancer hasa forwarding rule calledfr-ilb1. In the example, this load balancerdistributes traffic to thenic0 interface.

In the VPC network calledproduction, a differentinternal passthrough Network Load Balancer has a forwarding rule calledfr-ilb2. This load balancerdistributes traffic to a different interface,nic1 in this example.

For a detailed configuration setup, seeLoad balancing to multiple backendNICs.

Symmetric hashing

The preceding example doesn't use source network address translation(SNAT). SNAT isn't required because Google Cloud usessymmetric hashing. This means that when packets belong to the same flow,Google Cloud calculates the same hash. In other words, the hash doesn'tchange when the source IP address:port is swapped with the destination IPaddress:port.

Notes:

• Symmetric hashing is enabled automatically when you create theinternal passthrough Network Load Balancer forwarding rule on or after June 22, 2021.

• To enable symmetric hashing on existing internal passthrough Network Load Balancers, you mustre-create the forwarding rule and the next-hop route, as described inEnabling symmetrichashing.

• Symmetric hashing is only supported with internal passthrough Network Load Balancers.

• Symmetric hashing is supported with the following session affinity types forprotocols TCP and UDP:

  • Client IP (CLIENT_IP)
  • Client IP and protocol (CLIENT_IP_PROTO)
  • Client IP, protocol, and port (CLIENT_IP_PORT_PROTO)

  For more information about these settings, seeSession affinityoptions.

• You can optionally use SNAT if your use case requires it for some reason.

What's next

• To configure an internal passthrough Network Load Balancer to be a next hop, seeSet up an internal passthrough Network Load Balancer for third-party appliancesorDeploy a hub-and-spoke network by using a load balanceras the next hop.
• To configure and test an internal passthrough Network Load Balancer, seeSet up an internal passthrough Network Load Balancer with VM instance group backends.
• To troubleshoot next hop issues with your internal passthrough Network Load Balancer, seeTroubleshoot internal passthrough Network Load Balancers.