The Google Cloud internal proxy Network Load Balancer is a proxy-based load balancer powered byopen source Envoy proxy software andthe Andromeda network virtualizationstack.

The internal proxy Network Load Balancer is a Layer 4 load balancer that lets you run and scaleyour TCP service traffic behind a regional internal IP address that is accessibleonly to clients in the same VPC network or clients connected toyour VPC network. The load balancer first terminates the TCPconnection between the client and the load balancer at an Envoy proxy. The proxyopens a second TCP connection to backends hosted in Google Cloud, on-premises,or other cloud environments. For more use cases,seeProxy Network Load Balanceroverview.

Modes of operation

You can configure an internal proxy Network Load Balancer in the following modes:

• Regional internal proxy Network Load Balancer. This is a regional load balancer thatis implemented as a managed service based on the open sourceEnvoyproxy. With regional mode, all clients andbackends are from a specified region, which helps when you need regionalcompliance.

• Cross-region internal proxy Network Load Balancer. This is a multi-region load balancerthat is implemented as a managed service based on the open sourceEnvoyproxy. The cross-region modelets you load balance traffic to backend services that are globallydistributed with backends in multiple regions, including traffic managementthat ensures traffic is directed tothe closest backend. This load balancer also enables high availability.Placing backends in multiple regions helps avoid failures in asingle region. If one region's backends are down, traffic can fail over toanother region. The load balancer's forwarding rule IP addresses are alwaysaccessible in all regions.

  The following table describes the important differences between regionaland cross-region modes:

  Feature	Regional internal proxy Network Load Balancer	Cross-region internal proxy Network Load Balancer
  Virtual IP address (VIP) of the load balancer.	Allocated from a subnet in a specific Google Cloud region.	Allocated from a subnet in a specific Google Cloud region. VIP addresses from multiple regions can share the same global backend service.
  Client access	Not globally accessible by default. You can optionally enable global access.	Always globally accessible. Clients from any Google Cloud region can send traffic to the load balancer.
  Load balanced backends	Regional backends. Load balancer can only send traffic to backends that are in the same region as the proxy of the load balancer.	Global backends. Load balancer can send traffic to backends in any region.
  High availability and failover	Automatic failover to healthy backends in the same region.	Automatic failover to healthy backends in the same or different regions.

Identify the mode

gcloud compute forwarding-rules describeFORWARDING_RULE_NAME

Regional

This diagram shows the components of a regional internal proxy Network Load Balancerdeployment in Premium Tier.

Cross-region

This diagram shows the components of a cross-region internal proxy Network Load Balancerdeployment in Premium Tier within the same VPC network. Eachglobal forwarding rule uses a regional IP address that the clients use toconnect.

Proxy-only subnet

In the previous diagram, theproxy-onlysubnet provides a set of IP addressesthat Google uses to run Envoy proxies on your behalf. You must create aproxy-only subnet in each region of a VPC network where you usean Envoy-based internal proxy Network Load Balancer.

The following table describes proxy-only subnet requirements forinternal proxy Network Load Balancers:

Further:

• Proxy-only subnets areonly used for Envoy proxies, not your backends.
• Backend virtual machine (VM) instances or endpoints of all internal proxy Network Load Balancersin a region andVPC network receive connections from the proxy-only subnet.
• The IP address of an internal proxy Network Load Balancer isnot located in the proxy-onlysubnet. The load balancer's IP address is defined by its internal managedforwarding rule.

Forwarding rules and IP addresses

Forwarding rulesroute traffic by IP address, port, and protocol to a load balancingconfiguration consisting of a target proxy and a backend service.

IP address specification. Each forwarding rule references a single regionalIP address that you can use in DNS records for your application. You can eitherreserve a static IP address that you can use or let Cloud Load Balancing assignone for you. We recommend that you reserve a static IP address; otherwise, youmust update your DNS record with the newly assigned ephemeral IP addresswhenever you delete a forwarding rule and create a new one.

Clients use the IP address and port to connect to the load balancer's Envoyproxies—the forwarding rule's IP address is the IP address of the loadbalancer (sometimes called a virtual IP address or VIP). Clients connecting toa load balancer must use TCP. For the complete list of supportedprotocols, seeLoad balancer feature comparison.

The internal IP address associated with the forwarding rule can come from asubnet in the same network and region as your backends.

Port specification. Each forwarding rule that you use in aninternal proxy Network Load Balancer can reference asingle port from1-65535. Tosupport multiple ports, you must configure multiple forwarding rules.

The following table shows the forwarding rule requirements forinternal proxy Network Load Balancers:

Forwarding rules and VPC networks

This section describes how forwarding rules used by external Application Load Balancers areassociated with VPC networks.

Target proxies

The internal proxy Network Load Balancer terminates TCP connections from the client and createsnew connections to the backends. By default, the original client IP address andport information isn't preserved. You can preserve this information by usingthePROXY protocol.Thetarget proxy routes incomingrequests directly to the load balancer's backend service.

The following table shows the target proxy APIs required byinternal proxy Network Load Balancers:

Backend service

Abackend service directs incomingtraffic to one or more attached backends. A backend is either an instance groupor a network endpoint group. The backend contains balancing mode information todefine fullness based on connections (or, for instance group backends only,utilization).

Each internal proxy Network Load Balancer has a single backend service resource.

The following table specifies the backend service requirements forinternal proxy Network Load Balancers:

Supported backends

The backend service supports the following types of backends:

All of the backends must be of the same type (instance groups or NEGs). You cansimultaneously use different types of instance group backends, or you cansimultaneously use different types of NEG backends, but you cannot use instancegroup and NEG backends together on the same backend service.

You can mix zonal NEGs and hybrid NEGs within the same backend service.

To minimize service interruptions to your users, enable connectiondraining on backend services. Such interruptions can happen when a backendis terminated, removed manually, or removed by an autoscaler. To learn moreabout using connection draining to minimize service interruptions, seeEnable connection draining.

Backends and VPC networks

• For instance groups, zonal NEGs, and hybrid connectivity NEGs, all backendsmust be located in the same project and region as the backend service.However, a load balancer can reference a backend that uses a differentVPC network in the same project as the backend service.Connectivity between the load balancer'sVPC network and the backend VPC networkcan be configured using either VPC Network Peering, Cloud VPNtunnels, Cloud Interconnect VLAN attachments, or a Network Connectivity Centerframework.

  Backend network definition

  • For zonal NEGs and hybrid NEGs, you explicitly specify theVPC network when you create the NEG.
  • For managed instance groups, the VPC network is defined inthe instance template.
  • For unmanaged instance groups, the instance group'sVPC network is set to match the VPC networkof thenic0 interface for the first VM added to the instance group.

  Backend network requirements

  Your backend's network must satisfyone of the following networkrequirements:

  • The backend's VPC network must exactly match theforwarding rule's VPC network.

  • The backend's VPC network must be connected to theforwarding rule's VPC network usingVPC Network Peering. You must configure subnet route exchanges toallow communication between the proxy-only subnet in the forwarding rule'sVPC network and the subnets used by the backend instancesor endpoints.

• Both the backend's VPC network and the forwarding rule'sVPC network must beVPCspokesattached to the sameNetwork Connectivity Centerhub.Import and export filters must allow communication between the proxy-onlysubnet in the forwarding rule's VPC network and the subnetsused by backend instances or endpoints.

• For all other backend types, all backends must be located in the sameVPC network and region.

Backends and network interfaces

If you use instance group backends, packets are always delivered tonic0. Ifyou want to send packets to non-nic0 interfaces (eithervNICs orDynamic Network Interfaces), useNEG backends instead.

If you use zonal NEG backends, packets are sent to whatever network interface isrepresented by the endpoint in the NEG. The NEG endpoints must be in the sameVPC network as the NEG's explicitly defined VPCnetwork.

Protocol for communicating with the backends

When you configure a backend service for an internal proxy Network Load Balancer, you set the protocolthat the backend service uses to communicate with the backends. The loadbalancer uses only the protocol that you specify, and doesn't attempt tonegotiate a connection with the other protocol. Theinternal proxy Network Load Balancers only support TCP for communicating with the backends.

Health check

Each backend service specifies a health check that periodically monitors thebackends' readiness to receive a connection from the load balancer. This reducesthe risk that requests might be sent to backends that can't service the request.Health checks don't check if the application itself is working.

Health check protocol

Although it is not required and not always possible, it is a best practice touse a health check whose protocol matches theprotocol of the backendservice.For example, a TCP health check most accurately tests TCP connectivity tobackends. For the list of supported health check protocols, see theHealth checks section of the Load balancer feature comparison page.

The following table specifies the scope of health checks supported byinternal proxy Network Load Balancers in each mode:

For more information about health checks, see the following:

• Health checks overview
• Use health checks

Firewall rules

Internal proxy Network Load Balancers require the following firewall rules:

• An ingress allow rule that permits traffic from the Google health checkprobes. For more information about the specific health check probe IP addressranges and why it's necessary to allow traffic from them, seeProbe IP rangesand firewall rules.
• An ingress allow rule that permits traffic from theproxy-onlysubnet.

The ports for these firewall rules must be configured as follows:

• Allow traffic to the destination port for each backend service's health check.

• For instance group backends: Determine the ports to be configured by themapping between the backend service'snamedport andthe port numbers associated with that named port on each instance group. Theport numbers can vary among instance groups assigned to the same backendservice.

• ForGCE_VM_IP_PORT NEG backends, allow traffic to the port numbers of theendpoints.

There are certain exceptions to the firewall rule requirements for these loadbalancers:

• Allowing traffic from Google's health check probe ranges isn't required for hybridNEGs. However, if you're using a combination of hybrid and zonal NEGs ina single backend service, you need to allow traffic from theGooglehealth check probe ranges for the zonal NEGs.
• For regional internet NEGs, health checks are optional. Traffic from loadbalancers usingregional internet NEGs originates from theproxy-only subnet and is thenNAT-translated (by using Cloud NAT) to either manually or automatically allocatedNAT IP addresses. This traffic includes both health check probes and userrequests from the load balancer to the backends. For details, seeRegional NEGs:Use a Cloud NAT gateway.

Client access

Clients can be in the same network or in a VPC networkconnected by usingVPC Network Peering.

For regional internal proxy Network Load Balancers, clients must be in the same region as the loadbalancer by default.You canenable global accessto allow clients from any region to access your load balancer.

For cross-region internal proxy Network Load Balancers, global access is enabled by default.Clients from any region can access your load balancer.

The following table summarizes client access for regional internal proxy Network Load Balancers:

Shared VPC architecture

The internal proxy Network Load Balancer supports networks that use Shared VPC.Shared VPC lets you maintain a clear separation of responsibilitiesbetween network administrators and service developers. Your development teamscan focus on building services in service projects, and the networkinfrastructure teams can provision and administer load balancing. If you're notalready familiar with Shared VPC, read theShared VPC overview documentation.

Traffic distribution

An internal proxy Network Load Balancer distributes traffic to its backends as follows:

1. Connections originating from a single client are sent to the same zoneas long as healthy backends (instance groups or NEGs) within that zone areavailable and have capacity, as determined bythebalancing mode.For regional internal proxy Network Load Balancers, the balancing mode can beCONNECTION (instancegroup or NEG backends) orUTILIZATION (instance group backends only).
2. Connections from a client are sent to the same backendif you have configuredsession affinity.
3. After a backend is selected, traffic is then distributed among instances(in an instance group) or endpoints (in a NEG) according to aload balancing policy. For the load balancing policyalgorithms supported, see thelocalityLbPolicy setting in theregionalbackend service APIdocumentation.

Session affinity

Session affinity lets you configure the load balancer's backend service to sendall requests from the same client to the same backend, as long as the backend ishealthy and has capacity.

• None

  A session affinity setting ofNONE doesnot mean that there is nosession affinity. It means that no session affinity option is explicitly configured.

  Hashing is always performed to select a backend. And a session affinity setting ofNONE means that the load balancer uses a 5-tuple hash to select a backend. The 5-tuplehash consists of the source IP address, the source port, the protocol, the destination IP address,and the destination port.

  A session affinity ofNONE is the default value.

• Client IP affinity

  Client IP session affinity (CLIENT_IP) is a 2-tuple hash created from thesource and destination IP addresses of the packet. Client IP affinity forwardsall requests from the same client IP address to the same backend, as long asthat backend has capacity and remains healthy.

  When you use client IP affinity, keep the following in mind:

  • The packet destination IP address is only the same as the load balancer forwarding rule's IP address if the packet is sent directly to the load balancer.
  • The packet source IP address might not match an IP address associated with the original client if the packet is processed by an intermediate NAT or proxy system before being delivered to a Google Cloud load balancer. In situations where many clients share the same effective source IP address, some backend VMs might receive more connections or requests than others.

Keep the following in mind when configuring session affinity:

• Don't rely on session affinity for authentication or security purposes.Session affinity can break whenever thenumber of serving and healthy backends changes. For more details, seeLosingsession affinity.

• The default values of the--session-affinity and--subsetting-policyflags are bothNONE, and only one of them at a time can be set to adifferent value.

Losing session affinity

All session affinity options require the following:

• The selected backend instance or endpoint must remain configured as a backend. Session affinity can break when one of the following events occurs:
  • You remove the selected instance from its instance group.
  • Managed instance group autoscaling or autohealing removes the selected instance from its managed instance group.
  • You remove the selected endpoint from its NEG.
  • You remove the instance group or NEG that contains the selected instance or endpoint from the backend service.
• The selected backend instance or endpoint must remain healthy. Session affinity can break when the selected instance or endpoint fails health checks.

• The instance group or NEG that contains the selected instance or endpointmust not be full as defined by itstarget capacity. (Forregional managed instance groups, the zonal component of the instance groupthat contains the selected instance must not be full.) Session affinity canbreak when the instance group or NEG is full and other instance groups orNEGs are not. Because fullness can change in unpredictable ways when usingtheUTILIZATION balancing mode, you should use theRATE orCONNECTIONbalancing mode to minimize situations when session affinity can break.

• Thetotal number of configured backend instances or endpoints must remainconstant. When at least one of the following events occurs, the number ofconfigured backend instances or endpoints changes, and session affinity canbreak:

  • Adding new instances or endpoints:

    • You add instances to an existing instance group on the backend service.
    • Managed instance group autoscaling adds instances to a managed instancegroup on the backend service.
    • You add endpoints to an existing NEG on the backend service.
    • You add non-empty instance groups or NEGs to the backend service.

  • Removing any instance or endpoint,not just the selected instance orendpoint:

    • You remove any instance from an instance group backend.
    • Managed instance group autoscaling or autohealing removes any instancefrom a managed instance group backend.
    • You remove any endpoint from a NEG backend.
    • You remove any existing, non-empty backend instance group or NEG fromthe backend service.

• Thetotal number of healthy backend instances or endpoints must remainconstant. When at least one of the following events occurs, the number ofhealthy backend instances or endpoints changes, and session affinity canbreak:

  • Any instance or endpoint passes its health check, transitioning fromunhealthy to healthy.
  • Any instance or endpoint fails its health check, transitioning fromhealthy to unhealthy or timeout.

Failover

If a backend becomes unhealthy, traffic is automatically redirected to healthybackends.

The following table describes the failover behavior for internal proxy Network Load Balancers:

Load balancing for GKE applications

If you are building applications in Google Kubernetes Engine (GKE), you can usestandalone zonal NEGs to load balance traffic directly to containers. Withstandalone NEGs you are responsible for creating theService object that creates thezonal NEG, and then associating the NEG with the backend service so that theload balancer can connect to the Pods.

Related GKE documentation:

• Container-native load balancing through standaloneNEGs

Quotas and limits

For information about quotas and limits, seeQuotas andlimits.

Limitations

• The internal proxy Network Load Balancer doesn't support IPv6 traffic.
• The internal proxy Network Load Balancer doesn't support Shared VPC deploymentswhere the load balancer's frontend is in one host or service project andthe backend service and backends are in another service project (alsoknown as cross-project service referencing).
• Google Cloud does not make any guarantees on the lifetime of TCPconnections when you use internal proxy Network Load Balancers. Clients should beresilient to dropped connections, both due to broader internet issuesand due to regularly scheduled restarts of the Envoy proxies managing theconnections.

What's next

• Set up a cross-region internal Application Load Balancer with VM instance group backends

• Set up a cross-region internal Application Load Balancer with hybrid connectivity

• Set up a regional internal proxy Network Load Balancer with an instance groupbackend

• Set up a regional internal proxy Network Load Balancer with a zonal NEGbackend

• Set up a regional internal proxy Network Load Balancer with a hybrid NEGbackend

• Set up a regional internal proxy Network Load Balancer with an internet NEGbackend

• View metrics withCloud Monitoring