This guide describes how to troubleshoot configuration issuesfor a Google Cloud internal Application Load Balancer. Before following thisguide, familiarize yourself with the following:

• Internal Application Load Balancer overview
• Proxy-only subnets
• Internal Application Load Balancer logging andmonitoring

Troubleshoot common issues with Network Analyzer

Network Analyzerautomatically monitors your VPC network configuration and detectsboth suboptimal configurations and misconfigurations. It identifies networkfailures, provides root cause information, and suggests possible resolutions. Tolearn about the different misconfiguration scenarios that are automaticallydetected by Network Analyzer, seeLoad balancer insightsin the Network Analyzer documentation.

Network Analyzer is available in the Google Cloud console as a part ofNetwork Intelligence Center.

Backends have incompatible balancing modes

When creating a load balancer, you might see the error:

Validation failed for instance groupINSTANCE_GROUP:backend services1 and2 point to the same instance groupbut the backends have incompatible balancing_mode. Values should be the same.

This happens when you try to use the same backend in two different loadbalancers, and the backends don't have compatible balancing modes.

For more information, see the following:

• Restrictions and guidance for instancegroups
• Change the balancing mode of a loadbalancer

Load balanced traffic does not have the source address of the original client

This is expected behavior. An internal Application Load Balancer operates as an HTTP(S)reverse proxy (gateway). When a client program opens a connection to the IPaddress of an INTERNAL_MANAGED forwarding rule, the connection terminates at aproxy. The proxy processes the requests that arrive over that connection. Foreach request, the proxy selects a backend to receive the request based on theURL map and other factors. The proxy then sends the request to the selectedbackend. As a result, from the point of view of the backend, the source of anincoming packet is an IP address from the region'sproxy-onlysubnet.

Requests are rejected by the load balancer

For each request, the proxy selects a backend to receive the request based on apath matcher in the load balancer's URL map. If the URL map doesn't have a pathmatcher defined for a request, it cannot select a backend service, so it returnsan HTTP404 (Not Found) response code.

Load balancer doesn't connect to backends

The firewalls protecting your backend servers need to be configured to allowingress traffic from the proxies in theproxy-only subnetrange that you allocated to your internal HTTP(S) loadbalancer's region.

The proxies connect to backends using the connection settings specified by theconfiguration of your backend service. If these values don't match theconfiguration of the server(s) running on your backends, the proxy cannot forwardrequests to the backends.

Health check probes can't reach the backends

To verify that health check traffic reaches your backend VMs,enablehealth check logging and searchfor successful log entries.

Clients cannot connect to the load balancer

The proxies listen for connections to the load balancer's IP address and portconfigured in the forwarding rule (for example,10.1.2.3:80), and with theprotocol specified in the forwarding rule (HTTP or HTTPS). If your clients can'tconnect, ensure that they are using the correct address, port, and protocol.

Ensure that a firewall isn't blocking traffic between your client instancesand the load balanced IP address.

Ensure that the clients are in the same region as the load balancer. InternalHTTP(S) Load Balancing is a regional product, so all clients (and backends) mustbe in the same region as the load balancer resource.

Organizational policy restriction for Shared VPC

If you are using Shared VPC and you cannot create a new internal Application Load Balancerin a particular subnet, an organization policy might be thecause. In the organization policy, add the subnet to the list of allowedsubnets or contact your organization administrator. For more information, seeconstraints/compute.restrictSharedVpcSubnetworks.

Load balancer doesn't distribute traffic evenly across zones

You might observe an imbalance in your internal Application Load Balancer traffic across zones.This can happen especially when there is low utilization (< 10%) of yourbackend capacity.

Such behavior can affect overall latency due to traffic being sent to only a fewservers in one zone.

To even out the traffic distribution across zones, you can make the followingconfiguration changes:

• Use theRATEbalancingmode with a lowmax-rate-per-instance target capacity.
• Use theLocalityLbPolicy backendtrafficpolicywith a load balancing algorithm ofLEAST_REQUEST.

Unexplained5xx errors

For error conditions caused by a communications issue between the load balancerproxy and its backends, the load balancer generates an HTTP status code(5xx) and returns that status code to the client. Not all HTTP5xxerrors are generated by the load balancer—for example, if a backend sendsan HTTP5xx response to the load balancer, the load balancer relays thatresponse to its client. To determine whether an HTTP5xx response was relayedfrom a backend or if it was generated by the load balancer proxy, see theproxyStatus field of theload balancerlogs.

Configuration changes to the internal Application Load Balancer, such as addition orremoval of a backend service, can result in a brief period of time where userssee the HTTP status code503. While these configuration changespropagate toEnvoys globally,you see log entries where theproxyStatus field matches theconnection_refused log string.

If HTTP5xx status codes persist longer than a few minutes after you completethe load balancer configuration, take the following steps to troubleshoot HTTP5xx responses:

1. Verify that there is afirewall rule configured to allow healthchecks.In theabsence of one, load balancer logs typically have aproxyStatus matchingdestination_unavailable, which indicates that the load balancer considersthe backend to be unavailable.

2. Verify that health check traffic reaches your backend VMs. To do this,enablehealth check logging and searchfor successful log entries.

   For new load balancers, the lack of successful health check log entriesdoesn't mean that health check traffic is not reaching your backends. Itmight mean that the backend's initial health state has not yet changed fromUNHEALTHY to a different state. You see successful health check log entriesonly after the health check prober receives an HTTP200 OK response from thebackend.

3. Verify that the keepalive configuration parameter for the HTTP serversoftware running on the backend instance is not less than the keepalivetimeout of the load balancer, whose value is fixed at 10minutes (600 seconds) and is not configurable.

   The load balancer generates an HTTP5xx status code when the connection tothe backend has unexpectedly closed while sending the HTTP request or beforethe complete HTTP response has been received. This can happen because thekeepalive configuration parameter for the web server software running on thebackend instance is less than the fixed keepalive timeout of theload balancer. Ensure that the keepalive timeout configuration for HTTPserver software on each backend is set to slightly greater than 10 minutes(the recommended value is 620 seconds).

Limitations

If you are having trouble using an internal Application Load Balancer with otherGoogle Cloud networking features, note the current compatibilitylimitations.