Global external Application Load Balancer logging and monitoring

This document shows you how to configure and useCloud LoggingandCloud Monitoring with classic Application Load Balancers,global external Application Load Balancers, and Cloud CDN.

Logging

You can enable, disable, and view logs for an external Application Load Balancerbackend service.For external Application Load Balancers withbackendbuckets, loggingis automatically enabled and cannot be disabled.

You enable or disable logging for each backend service. You can configurewhether to log all requests or a randomly sampled fraction.

You must ensure that you don't have a logs exclusion that applies toexternal Application Load Balancers. For information about how to verify thatCloud HTTP LoadBalancer logs are allowed, seeExclusion filters.

Note: If the backend service is createdafter the GeneralAvailability release of Logging forexternal Application Load Balancers, logging is disabled for a backend service.If you enabled logging during the alpha release orbetarelease, logging continues to be enabled for the backend service.

Logs sampling and collection

The requests (and corresponding responses) handled by load balancer backendvirtual machine (VM) instances are sampled. These sampled requests are thenprocessed to generate logs. You control the fraction of the requests that areemitted as log entries according to thelogConfig.sampleRate parameter.WhenlogConfig.sampleRate is1.0 (100%), this means that logs aregenerated forall of the requests and written to Cloud Logging.

Optional fields

Log records contain required fields and optional fields. TheWhat islogged section lists which fields are optional and which arerequired. All required fields are always included. You can customize whichoptional fields you keep.

  • If you selectinclude all optional, all optional fields in the logrecord format are included in the logs. When new optional fields are addedto the record format, the logs automatically include the new fields.

  • If you selectexclude all optional, all optional fields are omitted.

  • If you selectcustom, you can specify the optional fields that you want toinclude, such astls.protocol,tls.cipher,orca_load_report.cpu_utilization,orca_load_report.mem_utilization.

For information about customizing optional fields, seeEnable logging on a new backend service.

Enabling logging on a new backend service

Console

  1. In the Google Cloud console, go to theLoad Balancing page.

    Go to Load balancing

  2. Click the name of your load balancer.

  3. ClickEdit.

  4. ClickBackend Configuration.

  5. SelectCreate a backend service.

  6. Complete the required backend service fields.

  7. In theLogging section, select theEnable logging checkbox.

  8. Set aSample rate fraction. You can set a number from0.0 through1.0, where0.0 means that no requests are logged and1.0 means that100% of the requests are logged. The default value is1.0.

  9. Optional: To include all the optional fields in the logs, in theOptional fields section, clickInclude all optional fields.

  10. To finish editing the backend service, clickUpdate.

  11. To finish editing the load balancer, clickUpdate.

gcloud

Create a backend service and enable logging by using thegcloud compute backend-services createcommand.

gcloud compute backend-services createBACKEND_SERVICE \    --global \    --enable-logging \    --logging-sample-rate=VALUE \    --load-balancing-scheme=EXTERNAL_MANAGED \    --logging-optional=LOGGING_OPTIONAL_MODE \    --logging-optional-fields=OPTIONAL_FIELDS

Thegcloud compute backend-services create command supports thefollowing fields:

  • --global indicates that the backend service is global. Use this fieldfor backend services used with global external Application Load Balancers.
  • --enable-logging enables logging for that backend service.
  • --logging-sample-rate lets you specify a value from0.0 through1.0, where0.0 means that no requests are logged and1.0 means that 100% of the requests are logged. This field is only meaningful with the--enable-logging parameter. Enabling logging but setting the sampling rate to0.0 is equivalent to disabling logging. The default value is1.0.

  • --logging-optional lets you specify the optional fields thatyou want to include in the logs. These fields are supported only withglobal external Application Load Balancers.

    • INCLUDE_ALL_OPTIONAL to include all optional fields.

    • EXCLUDE_ALL_OPTIONAL (default) to exclude all optional fields.

    • CUSTOM to include a custom list of optional fields thatyou specify inOPTIONAL_FIELDS.

  • --logging-optional-fields lets you specify a comma-separated list of optional fields that you want to include in the logs.

    For example,tls.protocol,tls.cipher can only be set ifLOGGING_OPTIONAL_MODE is set toCUSTOM. If you usecustommetrics and want tolog elements of the ORCA load report, you setLOGGING_OPTIONAL_MODE toCUSTOM andspecify which elements must be logged in theOPTIONAL_FIELDS field. For example,orca_load_report.cpu_utilization,orca_load_report.mem_utilization.

Enabling logging on an existing backend service

Console

  1. In the Google Cloud console, go to theLoad Balancing page.

    Go to Load balancing

  2. Click the name of your load balancer.

  3. ClickEdit.

  4. ClickBackend Configuration.

  5. ClickEdit next to yourbackend service.

  6. In theLogging section, select theEnable logging checkbox.

  7. In theSample rate field, set the sampling probability. You can set anumber from0.0 through1.0, where0.0 means that no requests arelogged and1.0 means that 100% of the requests are logged. The defaultvalue is1.0.

  8. Optional: To include all the optional fields in the logs, in theOptional fields section, clickInclude all optional fields.

  9. To finish editing the backend service, clickUpdate.

  10. To finish editing the load balancer, clickUpdate.

gcloud

Enable logging on an existing backend service with thegcloud compute backend-services update command.

gcloud compute backend-services updateBACKEND_SERVICE \    --global \    --enable-logging \    --logging-sample-rate=VALUE \    --logging-optional=LOGGING_OPTIONAL_MODE \    --logging-optional-fields=OPTIONAL_FIELDS

where

  • --global indicates that the backend service is global. Use this fieldfor backend services used with global external Application Load Balancers.
  • --enable-logging enables logging for that backend service.
  • --logging-sample-rate lets you specify a value from0.0 through1.0, where0.0 means that no requests are logged and1.0 means that 100% of the requests are logged. Only meaningful with the--enable-logging parameter. Enabling logging but setting thesampling rate to0.0 is equivalent to disabling logging. The defaultvalue is1.0.

  • --logging-optional lets you specify the optional fields thatyou want to include in the logs. These fields are supported only withglobal external Application Load Balancers.

    • INCLUDE_ALL_OPTIONAL to include all optional fields.

    • EXCLUDE_ALL_OPTIONAL (default) to exclude all optionalfields.

    • CUSTOM to include a custom list of optional fields thatyou specify inOPTIONAL_FIELDS.

  • --logging-optional-fields lets you specify a comma-separated list ofoptional fields that you want to include in the logs.

    For example,tls.protocol,tls.cipher. Can only be set ifLOGGING_OPTIONAL_MODE is set toCUSTOM.If you usecustommetrics and want tolog elements of the ORCA load report, you setLOGGING_OPTIONAL_MODE toCUSTOM andspecify which elements must be logged in theOPTIONAL_FIELDS field. For example,orca_load_report.cpu_utilization,orca_load_report.mem_utilization.

Disabling or modifying logging on an existing backend service

Console

  1. In the Google Cloud console, go to theLoad Balancing page.

    Go to Load balancing

  2. Click the name of your load balancer.

  3. ClickEdit.

  4. ClickBackend Configuration.

  5. ClickEdit next to yourbackend service.

  6. To disable logging entirely, in theLogging section, clear theEnable logging checkbox.

  7. If you leave logging enabled, you can set a differentSample ratefraction. You can set a number from0.0 through1.0, where0.0means that no requests are logged and1.0 means that 100% of therequests are logged. The default value is1.0. For example,0.2 means20% of the sampled requests generate logs.

  8. To finish editing the backend service, clickUpdate.

  9. To finish editing the load balancer, clickUpdate.

gcloud: Global mode

Disable logging on a backend service with thegcloud compute backend-services update command.

Disabling logging entirely

gcloud compute backend-services updateBACKEND_SERVICE \    --global \    --no-enable-logging

where

  • --global indicates that the backend service is global. Use this fieldfor backend services used with global external Application Load Balancers.
  • --no-enable-logging disables logging for that backend service.

Enabling logging optional fields on an existing backend service

gcloud compute backend-services updateBACKEND_SERVICE \    --global \    --enable-logging \    --logging-sample-rate=VALUE \    --logging-optional=LOGGING_OPTIONAL_MODE \    --logging-optional-fields=OPTIONAL_FIELDS

where

  • --logging-sample-rate lets you specify a value from0.0 through1.0, where0.0 means that no requests are logged and1.0 means that 100% of the requests are logged. Only meaningful withthe--enable-logging parameter. Enabling logging but setting thesampling rate to0.0 is equivalent to disabling logging. The defaultvalue is1.0.

  • --logging-optional lets you specify the optional fields thatyou want to include in the logs:

    • INCLUDE_ALL_OPTIONAL to include all optional fields.

    • EXCLUDE_ALL_OPTIONAL (default) to exclude all optional fields.

    • CUSTOM to include a custom list of optional fields thatyou specify inOPTIONAL_FIELDS.

  • --logging-optional-fields lets you specify a comma-separated list of optional fields that you want to include in the logs.

    For example,tls.protocol,tls.cipher can only be set ifLOGGING_OPTIONAL_MODE is set toCUSTOM.If you usecustommetrics and want tolog elements of the ORCA load report, you setLOGGING_OPTIONAL_MODE toCUSTOM andspecify which elements must be logged in theOPTIONAL_FIELDS field. For example,orca_load_report.cpu_utilization,orca_load_report.mem_utilization.

Updating logging optional mode from CUSTOM to others

gcloud compute backend-services updateBACKEND_SERVICE \    --global \    --enable-logging \    --logging-sample-rate=VALUE \    --logging-optional=LOGGING_OPTIONAL_MODE \    --logging-optional-fields=

where

  • --logging-optional lets you specify the optional fields thatyou want to include in the logs:

    • INCLUDE_ALL_OPTIONAL to include all optional fields.

    • EXCLUDE_ALL_OPTIONAL (default) to exclude all optional fields.

  • --logging-optional-fields must be explicitly configured as shown toclear any existingCUSTOM fields. The API doesn't let youcombine a non-CUSTOM mode withCUSTOM fields.

Modifying the logging sample rate

gcloud compute backend-services updateBACKEND_SERVICE \    --global \    --logging-sample-rate=VALUE

gcloud: Classic mode

Disable logging on a backend service with thegcloud compute backend-services update command.

Disabling logging entirely

gcloud compute backend-services updateBACKEND_SERVICE \    --global \    --no-enable-logging

where

  • --global indicates that the backend service is global. Use this fieldfor backend services used with a classic Application Load Balancer.
  • --no-enable-logging disables logging for that backend service.

Modifying the logging sample rate

gcloud compute backend-services updateBACKEND_SERVICE \    --global \    --logging-sample-rate=VALUE

where

  • --global indicates that the backend service is global. Use this fieldfor backend services used with a classic Application Load Balancer.
  • --logging-sample-rate lets you specify a value from0.0 through1.0, where0.0 means that no requests are logged and1.0 means that 100% of the requests are logged. Only meaningful withthe--enable-logging parameter. Enabling logging but setting thesampling rate to0.0 is equivalent to disabling logging.

View logs

To follow step-by-step guidance for this task directly in the Google Cloud console, clickGuide me:

Guide me

HTTP(S) logs are indexed first by aforwarding rule,then by aURL map.

To view logs, go to theLogs Explorer page:

Go to Logs Explorer

  • To view all logs, in theResource filter menu, selectCloud HTTP LoadBalancer > All forwarding rules.

  • To view logs for one forwarding rule, select a single forwarding rule name.

  • To view logs for one URL map, select a forwarding rule, and then select a URLmap.

Log fields of typeboolean typically only appear if they have a value oftrue. If a boolean field has a value offalse, that field is omitted fromthe log.

UTF-8 encodingis enforced for log fields. Characters that are not UTF-8characters are replaced with question marks.For classic Application Load Balancers and global external Application Load Balancers, you can exportlogs-based metrics usingresource logs (resource.type="http_load_balancer"). The metricscreated are based on theApplication Load Balancer Rule(Logs-based Metrics) resource (l7_lb_rule), which is available underCloud Monitoring dashboards instead of under thehttps_lb_rule resource.

What is logged

External Application Load Balancer log entries contain information useful formonitoring and debugging your HTTP(S) traffic. Log records contain requiredfields, which are the default fields of every log record.

Log records contain optional fieldsthat add additional information about your HTTP(S) traffic.Optional fields can be omitted to save storage costs.

Some log fields are in a multi-field format, with more than one piece of datain a given field. For example, thetls field is of theTlsInfoformat, which contains theearlyDataRequest field.These multi-field fields are described in the following record format table.

FieldField formatField type: Required or OptionalDescription
severity
insertID
logName		LogEntryRequiredThe general fields as described in a log entry.
timestampstring (Timestamp format)OptionalThe time when the first layer GFE receives the request.
httpRequestHttpRequestRequiredA common protocol for logging HTTP requests.

HttpRequest.protocol isn't populated forresource.type="http_load_balancer"

.
resourceMonitoredResourceRequired

TheMonitoredResource is the resource type associated with a log entry.

TheMonitoredResourceDescriptor describes the schema of aMonitoredResource object by using a type name and a set of labels. For more information, seeResource labels.

jsonPayloadobject (Struct format)RequiredThe log entry payload that is expressed as a JSON object. The JSON object contains the following fields:
  • statusDetails
  • backendTargetProjectNumber
  • overrideResponseCode
  • errorService
  • errorBackendStatusDetails
  • authzPolicyInfo
  • loadBalancingScheme
  • tls
  • orca_load_report
stringRequiredThestatusDetails field holds a string that explains why the load balancer returned the HTTP status code that it did. For more information about these log strings, seestatusDetails HTTP success messages andstatusDetails HTTP failure messages.
stringRequiredThebackendTargetProjectNumber field holds the project number where the backend target—backend service or backend bucket—has been created. This field is in the format:"projects/PROJECT_NUMBER". This information is only available for global external Application Load Balancers usingcustom error responses.
integerRequiredTheoverrideResponseCode holds the override response code applied to the response sent to the client. This information is only available for global external Application Load Balancers usingcustom error responses.
stringRequiredTheerrorService field holds the backend service that provided the custom error response. This information is only available for global external Application Load Balancers usingcustom error responses.
stringRequiredTheerrorBackendStatusDetails field holds thestatusDetails of the final response served to the client. This information is only available for global external Application Load Balancers usingcustom error responses.
AuthzPolicyInfoRequiredTheauthzPolicyInfo field stores information about the authorization policy result. This information is only available for global external Application Load Balancers that have enabled authorization policies. For more information, see what is logged for authorization policies.
stringOptionalTheloadBalancingScheme field is only populated if you use the classic Application Load Balancer migration feature. This field holds a string that describes which load balancing scheme was used to route the request. The possible values are eitherEXTERNAL orEXTERNAL_MANAGED.
TlsInfoRequired

Thetls field holds theTlsInfo field that specifies the TLS metadata for the connection between the client and the load balancer. This field is only available if the client is using TLS/SSL encryption.

Use the--logging-optional-fields parameter to specify which elements must be logged:

  • Optional:tls.protocol
  • Optional:tls.cipher
  • Required:tls.earlyDataRequest

Youcan't set--logging-optional-fields totls to specify all elements.

OrcaLoadReportOptional

Theorca_load_report field contains some or all elements of the ORCA load report returned by the backend. This field is only present if the backend returns an ORCA load report and you configured the load balancer to log the ORCA load report.

Use the--logging-optional-fields parameter to specify which of the following elements of the ORCA load report must be logged:

  • orca_load_report.cpu_utilization
  • orca_load_report.mem_utilization
  • orca_load_report.request_cost
  • orca_load_report.utilization
  • orca_load_report.rps_fractional
  • orca_load_report.eps
  • orca_load_report.named_metrics
  • orca_load_report.application_utilization

You can also set--logging-optional-fields toorca_load_report to specify that all elements must be logged.

Note: Redirects (HTTP response status code302 Found) that are issued from the load balancer arenot logged. Redirects issued from the backend instancesare logged.

TlsInfo field format

FieldField formatField type: Required or OptionalDescription
protocolstringOptionalTLS protocol that clients use to establish a connection with the load balancer. Possible values areTLSv1,TLSv1.1,TLSv1.2,TLSv1.3, orQUIC. This value is set toNULL if the client is not using TLS/SSL encryption.
cipherstringOptionalTLS cipher that clients use to establish a connection with the load balancer. This value is set toNULL if the client isn't using HTTP(S) or the client isn't using TLS/SSL encryption.
earlyDataRequestbooleanRequiredThe request includes early data in the TLS handshake.

Resource labels

The following table lists the resource labels forresource.type="http_load_balancer".

FieldTypeDescription
backend_service_namestringThe name of the backend service.
forwarding_rule_namestringThe name of the forwarding rule object.
project_idstringThe identifier of the Google Cloud project associated with this resource.
target_proxy_namestringThe name of the target proxy object referenced by the forwarding rule.
url_map_namestringThe name of the URL map object configured to select a backend service.
zonestringThe zone in which the load balancer is running. The zone isglobal.

statusDetails HTTP success messages

statusDetails (successful)MeaningCommon accompanying response codes
byte_range_cachingThe HTTP request was served using Cloud CDNbyte range caching.Anycacheable response code is possible.
response_from_cacheThe HTTP request was served from a Cloud CDN cache.Anycacheable response code is possible.
response_from_cache_validatedThe return code was set from a Cloud CDN cached entry that was validated by a backend.Anycacheable response code is possible.
response_sent_by_backendThe HTTP request was proxied successfully to the backend, and the response was returned by the backend.The HTTP response code is set by the software running on the backend.

statusDetails HTTP failure messages

statusDetails (failure)MeaningCommon accompanying status codes
aborted_request_due_to_backend_early_responseA request with body was aborted due to the backend sending an early response with a status code. The response was forwarded to the client. The request was terminated.4XX or5XX
backend_connection_closed_after_partial_response_sentThe backend connection closed unexpectedly after a partial response had been sent to the client.

The HTTP status code is set by the software running on the backend. HTTP status code0 (zero) means that the backend sent incomplete HTTP headers.

The HTTP status code is101 if the HTTP(S) connection was upgraded to a websocket connection.

backend_connection_closed_before_data_sent_to_clientThe backend unexpectedly closed its connection to the load balancer before the response was proxied to the client.

502, 503

The HTTP status code is101 if the HTTP(S) connection was upgraded to a websocket connection.

backend_early_response_with_non_error_statusThe backend sent a non-error status code (1XX or2XX) to a request before receiving the whole request body.502,503
backend_interim_response_not_supportedThe backend sent an interim1XX status code to the request in a context where interim responses aren't supported.

502,503

backend_response_corruptedThe HTTP response body sent by the backend has invalid chunked transfer-encoding or is otherwise corrupted.Any status code possible depending on the nature of the corruption. Often502,503.
backend_response_headers_too_longThe HTTP response headers sent by the backend exceeded the allowed limit. See theHeader size for external Application Load Balancers section for more information.502,503
backend_timeout

The backend timed out while generating a response.

For a websocket connection:

  • For global external Application Load Balancer, a status code is generated when the GFE closes the websocket connection in idle state after the backend service timeout expires.
  • For classic Application Load Balancer, a status code is generated when the GFE closes the websocket connection in either idle or active state, after the backend service timeout expires.

502,503

The HTTP status code is101 if the HTTP(S) connection was upgraded to a websocket connection.

banned_by_security_policyThe request was banned by a Cloud Armor rate-based ban rule.429
body_not_allowedThe client sent an HTTP request with a body, but the HTTP method used doesn't allow a body.400
byte_range_caching_abortedThe load balancer previously received a response indicating that the resource was cacheable and supported byte ranges. Cloud CDN received an inconsistent response (for example, one with a status code other than the expected206 Partial Content). This happened when attempting to perform cache fill using abyte range request. As a result, the load balancer aborted the response to the client.2XX
byte_range_caching_forwarded_backend_responseThe load balancer previously received a response indicating that the resource was cacheable and supported byte ranges. Cloud CDN received an inconsistent response (for example, one with a status code other than the expected206 Partial Content). This happened when attempting to perform cache fill using abyte range request. The load balancer then forwarded the inconsistent response to the client.

Returned from the backend—any status code is possible.

byte_range_caching_retrieval_abandonedThe client canceled a byte range request or validation request initiated by Cloud CDN.

Returned from the backend—any status code is possible.

byte_range_caching_retrieval_from_backend_failed_after_partial_responseA byte range request or validation request initiated by Cloud CDN encountered an error. Refer to the corresponding Cloud Logging log entry for the request initiated by Cloud CDN for the detailed backend status.2XX
cache_lookup_failed_after_partial_responseThe load balancer failed to serve a full response from Cloud CDN cache due to an internal error.2XX
cache_lookup_timeout_after_partial_responseThe Cloud CDN cache lookup stream timed out because the client didn't retrieve the content in a timely manner.2XX
client_disconnected_after_partial_responseThe connection to the client was broken after the load balancer sent a partial response.

Returned from the backend—any status code is possible.

The HTTP status code is101 if the HTTP(S) connection was upgraded to a websocket connection.

client_disconnected_before_any_responseThe connection to the client was broken before the load balancer sent any response.

0

The HTTP status code is101 if the HTTP(S) connection was upgraded to a websocket connection.

client_timed_outThe Google Front End (GFE) idled out the client connection due to lack of progress while it was proxying either the request or the response.0 or408
client_cert_invalid_rsa_key_sizeA client leaf or intermediate certificate had an invalid RSA key size. For more information, seeLogged errors for closed connections.0
client_cert_unsupported_elliptic_curve_keyA client or intermediate certificate is using an unsupported elliptic curve. For more information, seeLogged errors for closed connections.0
client_cert_unsupported_key_algorithmA client or intermediate certificate is using a non-RSA or non-ECDSA algorithm. For more information, seeLogged errors for closed connections.0
client_cert_pki_too_largeThe PKI to be used for validation has more than ten intermediate certificates that share the same Subject and Subject Public Key Info. For more information, seeLogged errors for closed connections.0
client_cert_chain_max_name_constraints_exceededAn intermediate certificate provided for validation had more than ten name constraints. For more information, seeLogged errors for closed connections.0
client_cert_chain_invalid_ekuEither the client certificate or its issuer doesn't haveExtended Key Usage (EKU) that includesclientAuth. For more information, seeLogged errors for closed connections.0
client_cert_validation_timed_outTime limit exceeded while validating the certificate chain. For more information, seeLogged errors for closed connections.0
client_cert_validation_search_limit_exceededThe depth or iteration limit is reached while attempting to validate the certificate chain. For more information, seeLogged errors for closed connections.0
client_cert_validation_not_performedYou have configured mTLS without setting up aTrustConfig. For more information, seeLogged errors for closed connections.0
client_cert_not_providedThe client didn't provide the requested certificate during the handshake. For more information, seeLogged errors for closed connections.0
client_cert_validation_failedClient certificate fails validation with theTrustConfig when hashing algorithms such as MD4, MD5, and SHA-1 are used. For more information, seeLogged errors for closed connections.0
config_not_found

The load balancer is missing project configuration. This can occur intermittently after you've made configuration changes that add a new resource.

Another cause of the error is that the first-layer GFE fails to communicate with the second-layer GFE. This can be due to an internal error, such as an in-progress rollout, load balancer overload, or intermittent configuration issues.

These errors are transient in nature and are expected to fall well within theSLA. However, if the error rate exceeds 0.01%, please contactGoogle Cloud support for further assistance.

404,502,503
direct_responseThe load balancer overrode this request and returned a fixed response.You might see any HTTP status code depending on the nature of the issue. For example, the HTTP410 status code means that the backend is unavailable due to payment delinquency.
denied_by_security_policyThe load balancer denied this request because of aGoogle Cloud Armor security policy.Configured in the security policy.
error_uncompressing_gzipped_bodyThere was an error uncompressing a gzipped HTTP response.502,503
failed_parsing_client_headers

Requests that use methods (for example,GET orPOST) that aren't compliant withRFC 9110, Section 5.6.2 are rejected by a first-layer Google Front End (GFE).

This failure code applies only to global external Application Load Balancers and classic Application Load Balancers.

400
failed_to_connect_to_backendThe load balancer failed to connect to the backend. This includes timeouts during the connection phase.502,503
failed_to_pick_backendThe load balancer failed to pick a healthy backend to handle the request.502,503
failed_to_negotiate_alpnThe load balancer and the backend failed to negotiate an application layer protocol (such as HTTP/2) to use to communicate with each other over TLS.502,503
headers_too_longThe request headers were larger than the maximum allowed.413
http_version_not_supportedHTTP version not supported. Only HTTP 0.9, 1.0, 1.1, and 2.0 are supported.400
internal_errorInternal error at the load balancer. Normally represents a transient error in the load balancer infrastructure. Retry your query.4XX or5XX
invalid_chunk_framingRequests and responses sent with theTransfer-Encoding: Chunked header are not RFC 9112 compliant. As per the RFC, both thechunked_body and thelast-chunk fields must end inCRLF.400
invalid_external_origin_endpointThe configuration for the external backend is invalid. Review theinternet NEG configuration and ensure that it specifies a valid FQDN/IP address and port.4XX
invalid_request_headers

The HTTP request headers received from a client contain at least one character that isn't allowed under an applicable HTTP specification.

For example, header field names that include a double quotation mark (") or any characters outside of the standard ASCII range (that is, any byte >=0x80) are invalid.

For more information, see:

400
invalid_http2_client_header_formatThe HTTP/2 headers from a client are invalid. For more information, seeinvalid_request_headers.400
invalid_http2_client_request_path

The HTTP/2 request path from a client contains at least one character that isn't allowed under the URI specification.

For more information, seethe "3.3. Path" section of RFC 3986.

400
multiple_iap_policiesMultipleIdentity-Aware Proxy (IAP) policies cannot be combined. If you have an IAP policy attached to a backend service and another policy attached to a serverless object, remove one of the policies and try again. Serverless objects include App Engine, Cloud Run, and Cloud Run functions.500
malformed_chunked_bodyThe request body was improperly chunk encoded.411
request_loop_detectedThe load balancer detected a request loop. This loop might be caused by a misconfiguration where the backend forwarded the request back to the load balancer.502,503
required_body_but_no_content_lengthThe HTTP request requires a body but the request headers don't include a content length or transfer-encoding chunked header.400,403,411
retriable_error

The load balancer has encountered an infrastructure error and the request can be retried. This can occur intermittently after you've made configuration changes that add or reference a new resource.

This can also occur when the first-layer GFE fails to communicate with the second-layer GFE due to an internal error, such as an in-progress rollout, load balancer overload, or intermittent configuration issue.

These errors are transient in nature and are expected to fall well within theSLA. However, if the error rate exceeds 0.01% over a sustained period, please contactGoogle Cloud support for further assistance.

404,502,503
secure_url_rejectedA request with anhttps:// URL was received over a plaintext HTTP/1.1 connection.400
server_cert_chain_exceeded_limitThe server certificate chain is too long (more than 10 intermediate certificates included with the server certificate).502,503

server_cert_chain_invalid_eku

The server certificate has anExtended Key Usage (EKU) extension field but that field doesn't includeserverAuth.

server_cert_chain_max_name_constraints_exceeded

An intermediate certificate provided for validation had more than 10 name constraints.502,503
server_cert_exceeded_size_limitThe server certificate payload (including any intermediate certificates) is too large (more than 16 KB).503
server_cert_invalid_rsa_key_size

A server or an intermediate certificate has an invalid RSA key size.

No validation is performed.

RSA keys can range from 2048 to 4096 bits.

503
server_cert_not_providedThe server did not provide the requested certificate during the handshake.503
server_cert_pki_too_large

The PKI to be used for validation has more than ten intermediate certificates that share the same Subject and Subject Public Key Info.

No validation is performed.

503
server_cert_trust_config_not_foundMatchingTrustConfig not found.503
server_cert_unsupported_elliptic_curve_key

A server or an intermediate certificate is using an unsupported elliptic curve.

No validation is performed.

Valid curves are P-256 and P-384.

503
server_cert_unsupported_key_algorithm

A server or an intermediate certificate is using a non-RSA or non-ECDSA algorithm.

No validation is performed.

503
server_cert_validation_internal_errorInternal error validating certificate chain.503
server_cert_validation_not_performed

You configured mTLS without setting up aTrustConfig resource.

503
server_cert_validation_search_limit_exceeded

The depth or iteration limit is reached while attempting to validate the certificate chain.

The maximum depth for a certificate chain is ten, including the root and server certificates. The maximum number of iterations is 100 (certificates examined to validate the server certificate chain).

503
server_cert_validation_timed_outThe time limit is exceeded while attempting to validate the certificate chain.503
server_cert_validation_unavailableService is unable to perform certificate chain validation.503
ssl_certificate_san_verification_failedThe load balancer cannot find a Subject Alternative Name (SAN) in the SSL certificate presented by the backend that matches the configured hostname.502,503
ssl_certificate_chain_verification_failedThe SSL certificate presented by the backend failed SSL certificate verification.502,503
throttled_by_security_policyThe request was blocked by a Cloud Armor throttle rule.429
unsupported_methodThe client supplied an unsupported HTTP request method.400
unsupported_100_continueThe client request included the 'Expect: 100-continue' header on a protocol that doesn't support it.400
upgrade_header_rejectedThe client HTTP request contained the Upgrade header and was refused.400
websocket_closedThe websocket connection was closed.101
websocket_handshake_failedThe websocket handshake failed.Any status code possible depending on the nature of the handshake failure.
request_body_too_largeThe HTTP request body exceeded the maximum supported by the backend. Not applicable for VM backends.413
handled_by_identity_aware_proxyThis response was generated byIAP during identity verification of the client before allowing access.

200,302,400,401,403,500,502,503

429 (throttled by IAP)

serverless_neg_routing_failedThe serverless NEG request cannot be dispatched. This error can happen when the region specified in the NEG cannot be reached, or when the resource name (for example, the Cloud Run functions name) cannot be found.404,502,503
fault_filter_abortThis error can happen if the customer has configured a fault filter and the fault filter was triggered for the given request.The value must be from200 to599.
early_data_rejected

The request sent in TLS early data was invalid.

This might occur in the following cases but isn't limited to them:

  • TheTargetHttpsProxy has TLS early data set toSTRICT, but the request included query parameters.
  • TheTargetHttpsProxy has TLS early data set toSTRICT orPERMISSIVE, but the request used a nonidempotent HTTP method (such as POST or PUT).
425
service_extension_error

There was an error calling out to a service extension used by the load balancer.

This might occur if the Wasm plugin is slow to respond and exceeds the 1 millisecond limit to send its response.

425

View logs for mTLS client certificate validation

To view thelogged errors for closed connectionsduring mutual TLS client certificate validation, complete the following steps.

Console query

  1. In the Google Cloud console, go to the Logs Explorer page.

    Go to Logs Explorer

  2. Click theShow query toggle.

  3. Paste the following into the query field. ReplaceFORWARDING_RULE_NAME with the name of yourforwarding rule.

    jsonPayload.statusDetails=~"client_cert"jsonPayload.@type="type.googleapis.com/google.cloud.loadbalancing.type.LoadBalancerLogEntry"resource.labels.forwarding_rule_name=FORWARDING_RULE_NAME

  4. ClickRun query.

Authorization policy request logs

Theauthz_info object in the Load Balancer Log Entry JSON payload containsinformation about authorization policies. You can configure log-based metricsfor traffic allowed or denied by these policies. Check moreauthorization policies log details.

FieldTypeDescription
authz_info.policies[]objectThe list of policies that match the request.
authz_info.policies[].namestringThe name of the authorization policy that matches the request.

The name is empty for the following reasons:

  • NoALLOW policy matches the request and the request is denied.
  • NoDENY policy matches the request and the request is allowed.
authz_info.policies[].resultenumThe result can beALLOWED orDENIED.
authz_info.policies[].detailsstringThe details include the following:
  • allowed_as_no_deny_policies_matched_request
  • denied_as_no_allow_policies_matched_request
  • denied_by_authz_extension
  • denied_by_cloud_iap
authz_info.overall_resultenumThe result can beALLOWED orDENIED.

Logging for backend buckets

Logging is automatically enabled for load balancers with backend buckets.You cannot modify or disable logging forbackend buckets.

Logging for Cloud Armor

The table forstatusDetail HTTP failure messages contains some messages thatapply to Cloud Armor. For more information about whatCloud Armor logs, seeUse request logging.

Logging for Shared VPC deployments

Application Load Balancer logs and metrics are typically exported to the projectthat has the forwarding rule. Therefore, service admins—owners or users ofprojects where the backend service is created—won't have access to theload balancer's logs and metrics by default. You can use IAMroles to grant these permissions to service admins. To learn more about theIAM roles that are available, and the steps to provide access,seeGrant access toMonitoring.

Interacting with the logs

You can interact with the external Application Load Balancer logs by using theCloud Logging API. The Logging API provides ways to interactivelyfilter logs that have specific fields set. It exports matching logs toCloud Logging, Cloud Storage, BigQuery, or Pub/Sub.For more information about the Logging API, seeLogging API overview.

Monitoring

The load balancer exports monitoring data toMonitoring.

You can use monitoring metrics to do the following:

  • Evaluate a load balancer's configuration, usage, and performance
  • Troubleshoot problems
  • Improve resource utilization and user experience

In addition to the predefined dashboards in Monitoring, you can createcustom dashboards, set up alerts, and query the metrics through theCloud Monitoring API.

Note: Metrics are based on sampled traffic, that is, they won't correspondone-to-one with actual requests. The sampling rate is dynamic and cannot beadjusted.

Viewing predefined Cloud Monitoring dashboards

Cloud Monitoring provides predefined dashboards to monitor yourload balancers. These dashboards are automatically populated byMonitoring.

Load balancers don't appear as a resource that can be monitoredunless a load balancer exists in the current project.

Perform the following steps to access the predefined dashboards:

  1. In the Google Cloud console, go to theMonitoring page.

    Go to Monitoring

  2. In the Monitoring navigation panel, clickDashboards.

  3. UnderCategories, clickGCP.

    • To view a list of dashboards forall your Google Cloud loadbalancers, select the dashboard namedGoogle Cloud Load Balancers.To view a specific load balancer's dashboard, locate the load balancerin the list and click its name.

    • To view the predefined dashboards for only your external Application Load Balancers,select the dashboard namedExternal HTTP(S) Load Balancers.This page displays a dashboard that shows the 5XX response ratios andbackend latency for all external Application Load Balancers in your project. It also providesa list of dashboards for all the external Application Load Balancers in your project.

      You can click through to each load balancer's dashboard. Each dashboardincludes the following:

      • Pre-populated graphs that display breakdowns for responses bystatus code classes (5xx,4xx,3xx,2xx)
      • Total latency
      • Backend latency
      • Frontend RTT
      • Request count
      • A link to the logs for the load balancer

  4. To view dashboards for third-party services, go back to theDashboardspage. UnderCategories, clickOther.

    • To view a specific third-party service dashboard, locate it in the listand click its name.

Defining alerting policies

To follow step-by-step guidance for this task directly in the Google Cloud console, clickGuide me:

Guide me

You can create alerting policies to monitor the values of metrics and to notify you when those metrics violate a condition.

  1. In the Google Cloud console, go to the Alerting page:

    Go toAlerting

    If you use the search bar to find this page, then select the result whose subheading isMonitoring.

  2. If you haven't created your notification channels and if you want to be notified, then clickEdit Notification Channels and add your notification channels. Return to theAlerting page after you add your channels.
  3. From theAlerting page, selectCreate policy.
  4. To select the metric, expand theSelect a metric menu and then do the following:
    1. To limit the menu to relevant entries, enterGlobal External Application Load Balancer Rule into the filter bar. If there are no results after you filter the menu, then disable theShow only active resources & metrics toggle.
    2. For theResource type, selectGlobal External Application Load Balancer Rule.
    3. Select aMetric category and aMetric, and then selectApply.
  5. ClickNext.
  6. The settings in theConfigure alert trigger page determine when the alert is triggered. Select a condition type and, if necessary, specify a threshold. For more information, seeCreate metric-threshold alerting policies.
  7. ClickNext.
  8. Optional: To add notifications to your alerting policy, clickNotification channels. In the dialog, select one or more notification channels from the menu, and then clickOK.
  9. Optional: Update theIncident autoclose duration. This field determines when Monitoring closes incidents in the absence of metric data.
  10. Optional: ClickDocumentation, and then add any information that you want included in a notification message.
  11. ClickAlert name and enter a name for the alerting policy.
  12. ClickCreate Policy.
For more information, seeAlerting overview.

Defining Cloud Monitoring custom dashboards

You can create custom Cloud Monitoring dashboards for the load balancer'smetrics:

  1. In the Google Cloud console, go to theMonitoring page.

    Go to Monitoring

  2. SelectDashboards > Create Dashboard.

  3. ClickAdd Chart, and then give the chart a title.

  4. To identify the time series to be displayed, choose a resource type andmetric type:

    1. In theResource & Metric section, click the chart, and thenin theSelect a metric section, select from the available options:
      • For a global external Application Load Balancer, select the resource typeGlobal External Application Load Balancer Rule.
    2. ClickApply.

  5. To specify monitoring filters, clickFilters > Add filter.

  6. ClickSave.

Metric reporting frequency and retention

Metrics for the external Application Load Balancers are exported to Cloud Monitoring in1-minute granularity batches. Monitoring data is retained for six (6) weeks.

The dashboard provides data analysis in default intervals of 1H (one hour),6H (six hours), 1D (one day), 1W (one week), and 6W (six weeks). You canmanually request analysis in any interval from 6W to 1 minute.

Monitoring metrics

You can monitor the following metrics for external Application Load Balancers.

The following metrics for global external Application Load Balancers arereported into Cloud Monitoring.

Note: The Response code class fraction metric is available per entireload balancer only, with no further breakdowns available.
MetricNameDescription
Backend configured rate (Preview)network.googleapis.com/loadbalancer/backend/configured_rateThe maximum rate in requests per second configured per backend group. This is the result of scaling the target capacity by the (capacity scaler), if specified.
Backend configured utilization (Preview)network.googleapis.com/loadbalancer/backend/configured_utilizationThe maximum CPU utilization capacity as a fraction, configured per backend group. This is the result of scaling the target capacity by the capacity scaler, if specified.
Backend error rate (Preview)network.googleapis.com/loadbalancer/backend/error_rateThe errors served by each backend group per second.
Backend fullness (Preview)network.googleapis.com/loadbalancer/backend/fullnessThe current fullness of each backend group as a percentage, based on the load balancer'sbalancing mode.
Backend latencies†loadbalancing.googleapis.com/https/backend_latencies

A distribution of the backend latency. Backend latency is the timein milliseconds betweenthe last byte of the request sent to the backend and the last byte ofthe response received by the proxy. It includes the time taken by thebackend to process the request and the time taken for the response to besent back to the proxy.

Backend load balancing custom metrics (Preview)network.googleapis.com/loadbalancer/backend/lb_custom_metricThe current utilization by each backend group, based on your defined custom metrics.
Backend rate (Preview)network.googleapis.com/loadbalancer/backend/rateThe requests received by each backend group per second.
Backend utilization (Preview)network.googleapis.com/loadbalancer/backend/utilizationThe aggregate CPU utilization of the VMs in the group as a fraction.
Request countloadbalancing.googleapis.com/https/request_countThe number of requests served by the external Application Load Balancer
Request bytes countloadbalancing.googleapis.com/https/request_bytes_countThe number of bytes sent as requests from clients to the external Application Load Balancer
Response bytes countloadbalancing.googleapis.com/https/response_bytes_countThe number of bytes sent as responses from the external Application Load Balancer to clients
Total latenciesloadbalancing.googleapis.com/https/total_latencies

A distribution of the total latency. Total latency is the time inmilliseconds between thefirst byte of the request received by the proxy and the last byte of theresponse sent by the proxy. It includes: the time taken by the proxy to processthe request, the time taken for the request to be sent from the proxy to thebackend, the time taken by the backend to process the request, thetime taken for the response to be sent back to the proxy, and the time takenfor the proxy to process the response and send the response to the client.

It doesn't include the RTT between the client and the proxy. Additionally,pauses between requests on the same connection that useConnection:keep-alive do not affect the measurement. This measurement is typicallyreduced to the 95th percentile in Cloud Monitoring views.

For websocket connections, this field refers to the entire time duration of the connection.*

Example: A load balancer has 1 request per second from the UK, all with 100 ms latency, and 9 requests per second from the US, all with 50 ms latency. Over a certain minute there were 60 requests from the UK and 540 requests from the US. Monitoring metrics preserves the distribution over all dimensions. You can request information such as the following:

  • median overall latency (300/600) - 50 ms
  • median UK latency (30/60) - 100 ms
  • 95th percentile overall latency (570/600) - 100 ms
Frontend RTT†loadbalancing.googleapis.com/https/frontend_tcp_rtt

A distribution of the frontend RTT. Frontend RTT is the time inmilliseconds it takes for data to travel from the client to the proxyand back again. It includes the time taken for a request to travel fromthe client to the proxy and back from the proxy to the client. This isnot updated during the lifetime of the connection. For example, settingup a (TCP) connection with a 3-way handshake would take 1.5 RTTs.

When requests are processed, the load balancer samples and averages thetime it takes for data to travel back and forth between the client and theproxy, and then logs a smoothed RTT value. Smoothed RTT is an algorithm thatdeals with variations and anomalies that might occur in RTT measurements.

Response code class fraction Fraction of total external Application Load Balancer responses that are in each response code class (2xx,4xx, ...). In Monitoring, this value is only available on default dashboards. It isn't available for custom dashboards. You can use the Monitoring API to set alerts for it.
Backend request countloadbalancing.googleapis.com/https/backend_request_countThe number of requests sent from the external Application Load Balancer to the backends.
Backend request bytes countloadbalancing.googleapis.com/https/backend_request_bytes_countThe number of bytes sent as requests from the external Application Load Balancer to the backends.
Backend response bytes countloadbalancing.googleapis.com/https/backend_response_bytes_countThe number of bytes sent as responses from the backends (including cache) to the external Application Load Balancer.

* For monitoring websocket connections, create a backend servicespecifically for websockets.

† The sum of Frontend RTT and Backend latencies might not be lessthan or equal to Total latencies. This is because although we poll RTT over thesocket from the GFE to the client at the time the HTTP response is acknowledged,we rely on kernel reporting for some of these measurements, and we cannotbe sure that the kernel will have an RTT measurement for the givenHTTP response. The end result is a smoothed RTT value that is also affectedby previous HTTP responses, SYN/ACKs, and SSL handshakes that aren't affectingcurrent HTTP request actual timings.

Filtering dimensions for metrics

You can apply filters for metrics for external Application Load Balancers.

Metrics are aggregated for each classic Application Load Balancer andglobal external Application Load Balancer. You can filter aggregated metrics bythe following dimensions forresource.type="http_load_balancer" orresource.type="https_lb_rule". Note that not all dimensions are available onall metrics.

PropertyDescription
backend_scopeThe Google Cloud scope (region or zone) of the backend service instance group that served the connection.

If no instance group was available or if the request was served by another entity, you see one of the following values instead of the region or zone of the backend service instance group.

  • FRONTEND_5xx: an internal error occurred before the GFE could select a backend. The GFE returned5xx to the client.
  • INVALID_BACKEND: the GFE couldn't find a healthy backend to assign the request to, so it returned a5xx status code to the requestor.
  • NO_BACKEND_SELECTED: either an error or interruption occurred before a backend was selected, a URL redirect occurred, or a classic Application Load Balancer with serverless backends returned a200 OK response.
  • MULTIPLE_BACKENDS: the request was served by potentially multiple backends. This can happen when Cloud CDN has served the request partially from its cache and has also sent one or morebyte range requests to the backend. Use thebackend_scope breakdown to visualize each load balancer-to-backend request.

When this breakdown is chosen, the charts show backend metrics (load balancer-to-backends), not frontend metrics (client-to-load balancer).

backend_type

The name of the backend group that served the client's request. Can beINSTANCE GROUP,NETWORK_ENDPOINT_GROUP, orUNKNOWN is returned if the backend wasn't assigned. If no backend group was available or if the request was served by another entity, one of the following values is displayed instead of a backend group.

  • FRONTEND_5XX: an internal error occurred before the GFE could select a backend. The GFE returned5xx to the client.
  • INVALID_BACKEND: the GFE couldn't find a healthy backend to assign the request to, so it returned a5xx status code to the requestor.
  • NO_BACKEND_SELECTED: either an error or interruption occurred before a backend was selected, a URL redirect occurred, or a classic Application Load Balancer with serverless backends returned a200 OK response.
  • MULTIPLE_BACKENDS: the request was served by potentially multiple backends. This can happen when Cloud CDN has served the request partially from its cache and has also sent one or morebyte range requests to the backend. Use thebackend_scope breakdown to visualize each load balancer-to-backend request.
backend_target_typeThe name of the backend service that served the request. Can beBACKEND_SERVICE,BACKEND_BUCKET,UNKNOWN if the backend wasn't assigned, orNO_BACKEND_SELECTED if an error or interruption occurred before a backend was selected, a URL redirect occurred, or a classic Application Load Balancer with serverless backends returned a200 OK response.
matched_url_path_ruleThe URL map path rule that matched the prefix of the HTTP(S) request (up to 50 characters).
forwarding_rule_nameThe name of the forwarding rule used by the client to send the request.
url_map_name

The URL map path rule or route rule configured as part of the URL map key. Can beUNMATCHED orUNKNOWN as fallbacks.

  • UNMATCHED refers to a request that doesn't match any URL path rules, sourl_map_name uses the default path rule.
  • UNKNOWN indicates an internal error.
target_proxy_nameThe name of the target HTTP(S) proxy object referenced by the forwarding rule.
backend_target_nameThe name of the backend target. The target can be either a backend service or backend bucket.UNKNOWN is returned if a backend wasn't assigned.
backend_nameThe name of the backend instance group, bucket, or NEG.UNKNOWN is returned if the backend wasn't assigned, orNO_BACKEND_SELECTED if an error or interruption occurred before a backend was selected, a URL redirect occurred, or a classic Application Load Balancer with serverless backends returned a200 OK response.
backend_scope_type

The type of the scope of the backend group. Can beGLOBAL,REGION,ZONE,MULTIPLE_BACKENDS, orNO_BACKEND_SELECTED if an error or interruption occurred before a backend was selected, a URL redirect occurred, or a classic Application Load Balancer with serverless backends returned a200 OK response, or other possible backend_type outputs.

MULTIPLE_BACKENDS is used when chunk caching is used. Multiple queries are sent to the same backend for different chunks of data to support a single client request.

proxy_continentContinent of the HTTP(S) GFE that terminated the HTTP(S) connection—for example,America,Europe,Asia
protocolProtocol used by the client, one ofHTTP/1.0,HTTP/1.1,HTTP/2.0,QUIC/HTTP/2.0,UNKNOWN.
response_codeThe HTTP status code of the request.
response_code_classThe HTTP status code class of the request:200,300,400,500 or0 for none.
cache_resultCache result for serving HTTP request by proxy:HIT,MISS,DISABLED,PARTIAL_HIT (for a request served partially from cache and partially from backend), orUNKNOWN.
client_countryCountry of the client that issued the HTTP request—for example,United States orGermany.
load_balancing_schemeThe load balancing scheme used. If classic Application Load Balancer is used, the value isEXTERNAL. If global external Application Load Balancer is used, the value isEXTERNAL_MANAGED.

What's next

Except as otherwise noted, the content of this page is licensed under theCreative Commons Attribution 4.0 License, and code samples are licensed under theApache 2.0 License. For details, see theGoogle Developers Site Policies. Java is a registered trademark of Oracle and/or its affiliates.

Last updated 2025-12-17 UTC.