This document shows you how to configure and useCloud LoggingandCloud Monitoring for external passthrough Network Load Balancers.

Backend service-based external passthrough Network Load Balancers support both Loggingand Monitoring. However, target pool-based external passthrough Network Load Balancerssupport only Monitoring.

Logging

Logs provide useful information for troubleshootingand monitoring your backend service-based external passthrough Network Load Balancer. Logs areaggregated per connection and exported in near real time. Logs are generated foreach load-balanced instance's TCP, UDP, ESP, GRE, ICMP, and ICMPv6 flows for bothingress and egress traffic. For more information about the fields provided inthe log entry, seeLog fields.

There are no additional charges for using logs. Based on how you importlogs, standard pricing forCloud Logging,BigQuery, or Pub/Sub apply. Enabling logs has no effect on theperformance of the load balancer.

Logging provides the following benefits:

• External passthrough Network Load Balancer traffic monitoring. Per-connection logging gives you insight into how eachconnection is routed to serving backends.

• Network troubleshooting. You can utilize external passthrough Network Load Balancer logs for troubleshooting.For more information, seeTroubleshoot external passthrough Network Load Balancers.

Sample log format for an external client to VM flows

The following diagram shows the inbound and outbound traffic for an externalclient (203.0.113.7), external passthrough Network Load Balancer (198.51.100.99),and backend instance (10.240.0.2).

External passthrough Network Load Balancer logs for connections from the client to the backendinstance are formatted as follows:

• connection.clientIp: 203.0.113.7
• connection.serverIp: 198.51.100.99
• bytesSent: 1256
• bytesReceived: 4521

Logs sampling and collection

Google Cloud samples the packets that leave and enter load balancer backendVMs. Those sampled packets are processed to generate logs.

Not every packet is sampled. Google Cloud samples a variable subset ofpackets depending on the amount of traffic on the physical host.The lowest possible sampling rate is one out of 1,024 packets. The sampling rateis dynamically controlled by Google Cloud. You cannot adjust the samplingrate.

The packet sampling interacts with firewall rules in the following ways:

• Packets are sampledbefore egress firewall rules are applied.
• Packets are sampledafter ingress firewall rules are applied.

After packet sampling, Google Cloud processes the sampled packetsaccording to the following procedure:

1. Aggregation: Sampled packets are aggregated over a five-second intervalto produce a single flow entry.

2. Configurable (secondary) log sampling: This is a second samplingprocess, sampling the flows. You control the fraction of the flow entriesthat are emitted as log entries according to thelogConfig.sampleRateparameter. WhenlogConfig.sampleRate is1.0 (100%), this means thatallof the sampled packets are processed.

3. Write to logging: The log entries are written to Cloud Logging.

Optional fields

Log records contain required fields and optional fields. TheLogfields section lists which fields are optional and which are required. All requiredfields are always included. You can customize which optional fields you keep.

• If you selectinclude all optional, all optional fields in the logrecord format are included in the flow logs. When new optional fields are addedto the record format, the flow logs automatically include the new fields.

• If you selectexclude all optional, this omits all optional fields.

• If you selectcustom, you can specify the optional fieldsthat you want to include by the parent field, such asserverInstance, or bytheir full names, such asserverInstance.vm.

When new optional fields are added to the record format, the logs won'tinclude these fields, unless they are a new field within a parent field thatyou have specified to include.

If you specify a custom optional field using parent fields, when new optionalfields are added to the record format within that parent field, the logs willautomatically include the new fields. For example, if you choose to includeoptionalFieldA, then a new field added to the logs with the nameoptionalFieldA.subField1 will be included automatically.

For instructions about customizing optional fields, seeEnable logging on a new backend service.

Response packet source IP address requirements

Logging samples the response packets from the backendVMs only if the source IP address for those packets matches the load balancer'sforwarding rule IP address.For TCP connections, response packets must always have sources that match therequest packet's destination; however, for other protocols, it's possible forresponse packets to use a different source IP address.For more information, seeIP addresses for request and return packets.

The packet sampling process used by external passthrough Network Load Balancer logging omits anyresponse packets from backend VMs if those response packets have sources thatdon't match an IP address of a forwarding rule for an external passthrough Network Load Balancer.

Enable logging on a new backend service

    gcloud compute backend-services createBACKEND_SERVICE \      --region=REGION \      --enable-logging \      --logging-sample-rate=SAMPLE_RATE \      --logging-optional=LOGGING_OPTIONAL \      --logging-optional-fields=OPTIONAL_FIELDS

    POST https://compute.googleapis.com/compute/v1/projects/`PROJECT_ID`/regions/`REGION`/backendServices

    {    "name": "BACKEND_SERVICE",    "loadBalancingScheme": "EXTERNAL",    "logConfig": {       "enable": true,       "sampleRate":SAMPLE_RATE      }    }

    {    "name": "BACKEND_SERVICE",    "loadBalancingScheme": "EXTERNAL",    "logConfig": {       "enable": true,       "sampleRate":SAMPLE_RATE,       "optionalMode": "INCLUDE_ALL_OPTIONAL"      }    }

    {    "name": "BACKEND_SERVICE",    "loadBalancingScheme": "EXTERNAL",    "logConfig": {       "enable": true,       "sampleRate":SAMPLE_RATE,       "optionalMode": "CUSTOM",       "optionalFields": ["field1","field2",...]      }    }

Enable logging on an existing backend service

    gcloud compute backend-services updateBACKEND_SERVICE \      --region=REGION \      --enable-logging \      --logging-sample-rate=SAMPLE_RATE \      --logging-optional=LOGGING_OPTIONAL \      --logging-optional-fields=OPTIONAL_FIELDS

      PATCH https://compute.googleapis.com/compute/v1/projects/`PROJECT_ID`/regions/`REGION`/backendServices/`BACKEND_SERVICE`

     "logConfig": {       "enable": true,       "sampleRate":SAMPLE_RATE,       "optionalMode": "EXCLUDE_ALL_OPTIONAL"      }

    {    "name": "BACKEND_SERVICE",    "loadBalancingScheme": "EXTERNAL",    "logConfig": {       "enable": true,       "sampleRate":SAMPLE_RATE,       "optionalMode": "INCLUDE_ALL_OPTIONAL"      }    }

    {    "name": "BACKEND_SERVICE",    "loadBalancingScheme": "EXTERNAL",    "logConfig": {       "enable": true,       "sampleRate":SAMPLE_RATE,       "optionalMode": "CUSTOM",       "optionalFields": ["field1","field2",...]      }    }

Disable logging on an existing backend service

gcloud compute backend-services updateBACKEND_SERVICE \  --region=REGION \  --no-enable-logging

 PATCH https://compute.googleapis.com/compute/v1/projects/PROJECT_ID/regions/REGION/backendServices/BACKEND_SERVICE

 { "logConfig": {   "enable": false  } }

View logs

When logs are ingested into Cloud Logging and not excluded through aLog router sink, you can read logs by using theCloud Logging API and theGoogle Cloud CLI.

To view all the external passthrough Network Load Balancer logs:

View logs for a specific backend service

To view the external passthrough Network Load Balancer logs for a specific backend service:

View logs for a backend instance group

To view the external passthrough Network Load Balancer logs for a specific backend instance group:

Log fields

Log records contain required fields, which are the default fields of every logrecord, and optional fields that add additional information. Optional fields canbe omitted to save storage costs.

Some log fields are in a multi-field format, with more than one piece of datain a given field. For example, theconnection field is of theIpConnectionformat, which contains the source and destination IP address and port, plus theprotocol, in a single field. These multi-field fields are described in thefollowing record format table.

The monitored resource isloadbalancing.googleapis.com/ExternalNetworkLoadBalancerRule.

IpConnection field format

InstanceDetails field format

GeographicDetails field format

GkeDetails field format

ClusterDetails field format

PodDetails field format

ServiceDetails field format

NetworkTierDetails field format

Monitoring

External passthrough Network Load Balancers export key metrics toCloud Monitoring.

Monitoring metrics can be used for the following purposes:

• Evaluate configuration, usage, and performance of a load balancer
• Troubleshoot problems
• Improve resource utilization and user experience

In addition to the predefined dashboards in Monitoring, you cancreate custom dashboards, set up alerts, and query the metrics through theMonitoring API.

View Monitoring dashboards

Define Monitoring custom dashboards

You can create custom Monitoring dashboards overexternal passthrough Network Load Balancer metrics.

Refer toMetrics and resource types for a list ofcollected metrics. Refer toFilters for lists of attributes you canuse to filter your results.

Define Monitoring alerts

You can defineMonitoring alerts overvarious external passthrough Network Load Balancer metrics:

Metric reporting frequency and retention

Metrics for the external passthrough Network Load Balancers are exported to Monitoring inone-minute granularity batches. Monitoring data is retained for six weeks.Metrics are based on sampled traffic (sampling rate is dynamic and cannot beadjusted).The dashboard provides data analysis in default intervals of one hour (1H), sixhours (6H), one day (1D), one week (1W), and six weeks (6W). You can manuallyrequest analysis in any interval from six weeks to sixty seconds.

Metrics and resource types

This section describes the metrics and resource types for external passthrough Network Load Balancers.

Metrics

The following metrics for external passthrough Network Load Balancers arereported into Monitoring. You can usethese metric names when makingAPI requests.

Resource types

External passthrough Network Load Balancers use the following resource types:

• tcp_lb_rule
• udp_lb_rule
• loadbalancing.googleapis.com/ExternalNetworkLoadBalancerRule

An external passthrough Network Load Balancer can support TCP, UDP, ESP, GRE, ICMP, and ICMPv6 traffic.

You can specify the resource type astcp_lb_rule orudp_lb_rule to limitmonitored data to either TCP or UDP protocols.You can specify the resource type asloadbalancing.googleapis.com/ExternalNetworkLoadBalancerRuleto monitor all the supported protocols including TCP, UDP, ESP, GRE, ICMP, and ICMPv6protocols.

Filters

Metrics are aggregated for each external passthrough Network Load Balancer. You can filter aggregatedmetrics by the following dimensions.

Resource labels fortcp_lb_rule orudp_lb_rule

You can restrict your query by resource. You can also group results by thesevalues.

resource.label.<var>LABEL_KEY</var>:

Resource labels forloadbalancing.googleapis.com/ExternalNetworkLoadBalancerRule

You can restrict your query by resource. You can also group results by thesevalues.

resource.label.<var>LABEL_KEY</var>:

Metric labels

You can restrict your query by metric label. You can also group results by thesevalues.

metric.label.<var>LABEL_KEY</var>:

Monitoring API requests

You can construct arbitrary queries over external passthrough Network Load Balancer metricsby using theMonitoring API`s v3 projects.timeSeries.list request.The exact structure of the various definitions follows the genericmonitoring filters semantics.

Example API requests

• Get all bytes sent from all external passthrough Network Load Balancers in the project during 12:00-12:02 PM UTC on 2019-07-01with 1m aggregates.

  timeSeries.list parameters:

  • name: projects/PROJECT_ID
  • Filter:resource.type = "tcp_lb_rule" ANDmetric.name="loadbalancing.googleapis.com/l3/external/egress_bytes_count"
  • interval.start_time:2019-07-01T12:00:00Z
  • interval.end_time:2019-07-01T12:02:00Z
  • aggregation.alignmentPeriod:60s
  • aggregation.crossSeriesReducer:REDUCE_SUM
  • aggregation.perSeriesAligner:ALIGN_SUM

• Get median RTT measurement over all external passthrough Network Load Balancers in theproject during 12:19-12:20 PM UTC on 2019-07-01, broken down by client country.

  timeSeries.list parameters:

  • name: projects/PROJECT_ID
  • Filter:resource.type = "tcp_lb_rule" ANDmetric.name="loadbalancing.googleapis.com/l3/external/rtt_latencies"
  • interval.start_time:2019-07-01T12:19:00Z
  • interval.end_time:2019-07-01T12:20:00Z
  • aggregation.alignmentPeriod:60s
  • aggregation.crossSeriesReducer:REDUCE_SUM
  • aggregation.groupByFields:metric.label.client_country
  • aggregation.perSeriesAligner:ALIGN_PERCENTILE_50

• Get total bytes from a specific external passthrough Network Load Balancer during12:19-12:20 PM UTC on 2019-07-01, broken down by endpoint zone and client continent.

  timeSeries.list parameters:

  • name: projects/PROJECT_ID
  • Filter:resource.type = "tcp_lb_rule" ANDresource.label.load_balancer_name = "netlb-bs-1" ANDmetric.name="loadbalancing.googleapis.com/l3/external/ingress_bytes_count"
  • interval.start_time:2019-07-01T12:19:00Z
  • interval.end_time:2017-07-01T12:20:00Z
  • aggregation.alignmentPeriod:60s
  • aggregation.crossSeriesReducer:REDUCE_SUM
  • aggregation.groupByFields:metric.label.client_continent
  • aggregation.groupByFields:resource.label.endpoint_zone
  • aggregation.perSeriesAligner:ALIGN_SUM

What's next

• Readconceptual information about external passthrough Network Load Balancers
• Set up an external passthrough Network Load Balancer
• Read about MonitoringMetrics, time series, and resources
• Read aboutforwarding rules
• Read abouttroubleshoot logging issues