This document shows you how to configure and useCloud LoggingandCloud Monitoring for internal passthrough Network Load Balancers.

Logging

Logs provide useful information for troubleshootingand monitoring the pass-through Google Cloud load balancer. Logs areaggregated per connection and exported in near real time. Logs are generated foreach load-balanced instance TCP and UDP flows for bothingress and egress traffic. For more information about the fields provided inthe log entry, seeLog fields.

There are no additional charges for using logs. Based on how you ingestlogs, standard pricing forCloud Logging,BigQuery, or Pub/Sub apply. Enabling logs has no effect on theperformance of the load balancer.

Logging provides the following benefits:

• Internal passthrough Network Load Balancer traffic monitoring. Per-connection logging gives youinsight into how each connection is routed to serving backends.

• Network troubleshooting. You can utilize internal passthrough Network Load Balancer logs for troubleshooting.For more information, seeTroubleshoot internal passthrough Network Load Balancers.

Sample log format for VM to VM flows

The following diagram shows the inbound and outbound traffic for an internalclient (192.168.1.2), internal passthrough Network Load Balancer (10.240.0.200),and backend instance (10.240.0.3).

Internal passthrough Network Load Balancer logs for connections from the client to the backendinstance are formatted as follows:

• connection.clientIp: 192.168.1.2
• connection.serverIp: 10.240.0.200
• bytesSent: 1256
• bytesReceived: 4521

Logs sampling and collection

Google Cloud samples the packets that leave and enter load balancer backendVMs. Those sampled packets are processed to generate logs.

Not every packet is sampled. Google Cloud samples a variable subset ofpackets depending on the amount of traffic on the physical host.The lowest possible sampling rate is one out of 1,024 packets. The sampling rateis dynamically controlled by Google Cloud. You cannot adjust the samplingrate.

The packet sampling interacts with firewall rules in the following ways:

• Packets are sampledbefore egress firewall rules are applied.
• Packets are sampledafter ingress firewall rules are applied.

After packet sampling, Google Cloud processes the sampled packetsaccording to the following procedure:

1. Aggregation: Sampled packets are aggregated over a five-second intervalto produce a single flow entry.

2. Configurable (secondary) log sampling: This is a second samplingprocess, sampling the flows. You control the fraction of the flow entriesthat are emitted as log entries according to thelogConfig.sampleRateparameter. WhenlogConfig.sampleRate is1.0 (100%), this means thatall ofthe sampled packets are processed.

3. Write to logging: The log entries are written to Cloud Logging.

Optional fields

Log records contain required fields and optional fields. TheLogfields section lists which fields are optional and which are required. All requiredfields are always included. You can customize which optional fields you keep.

• If you selectinclude all optional, all optional fields in the logrecord format are included in the flow logs. When new optional fields are addedto the record format, the flow logs automatically include the new fields.

• If you selectexclude all optional, this omits all optional fields.

• If you selectcustom, you can specify the optional fieldsthat you want to include by the parent field, such asserverInstance, or bytheir full names, such asserverInstance.vm.

  When new optional fields are added to the record format, the logs do notinclude these fields unless they are a new field within a parent field thatyou have specified to include.

  If you specify a custom optional field by using parent fields, when new optionalfields are added to the record format within that parent field, the logsautomatically include the new fields. For example, if you choose to includeoptionalFieldA, then a new field added to the logs with the nameoptionalFieldA.subField1 is included automatically.

For instructions about customizing optional fields, seeEnable logging on a new backend service.

Response packet source IP address requirements

Internal passthrough Network Load Balancer logging samples the response packets from the backendVMs only if the source IP address for those packets matches the load balancer'sforwarding rule IP address.For TCP connections, response packets must always have sources that match therequest packet's destination; however, for other protocols, it's possible forresponse packets to use a different source IP address.For additional details, seeIP addresses for request and return packets.

The packet sampling process used by internal passthrough Network Load Balancers omits anyresponse packets from backend VMs if those response packets have sources that donot match an IP address of a forwarding rule for an internal passthrough Network Load Balancer.

Enable logging for a new backend service

gcloud compute backend-services createBACKEND_SERVICE \  --region=REGION \  --enable-logging \  --logging-sample-rate=SAMPLE_RATE  --logging-optional=LOGGING_OPTIONAL \  --logging-optional-fields=OPTIONAL_FIELDS

 POST https://compute.googleapis.com/compute/v1/projects/PROJECT_ID/regions/REGION/backendServices

 { "name": "BACKEND_SERVICE",  "loadBalancingScheme": "INTERNAL",  "logConfig": {   "enable": true,   "sampleRate":SAMPLE_RATE,   "optionalMode": "EXCLUDE_ALL_OPTIONAL"  } }

{ "name": "BACKEND_SERVICE", "loadBalancingScheme": "INTERNAL", "logConfig": {   "enable": true,   "sampleRate":SAMPLE_RATE,   "optionalMode": "INCLUDE_ALL_OPTIONAL"  }}

{ "name": "BACKEND_SERVICE", "loadBalancingScheme": "INTERNAL", "logConfig": {   "enable": true,   "sampleRate":SAMPLE_RATE,   "optionalMode": "CUSTOM",   "optionalFields": ["field1","field2",...]  }}

Enable logging on an existing backend service

gcloud compute backend-services updateBACKEND_SERVICE \--region=REGION \--enable-logging \--logging-sample-rate=SAMPLE_RATE \--logging-optional=LOGGING_OPTIONAL \--logging-optional-fields=OPTIONAL_FIELDS

 PATCH https://compute.googleapis.com/compute/v1/projects/PROJECT_ID/regions/REGION/backendServices/BACKEND_SERVICE

 "logConfig": {   "enable": true,   "sampleRate":SAMPLE_RATE,   "optionalMode": "EXCLUDE_ALL_OPTIONAL"  }

 "logConfig": {   "enable": true,   "sampleRate":SAMPLE_RATE,   "optionalMode": "INCLUDE_ALL_OPTIONAL"  }

 "logConfig": {   "enable": true,   "sampleRate":SAMPLE_RATE,   "optionalMode": "CUSTOM",   "optionalFields": ["field1","field2",...]  }

Disable logging on an existing backend service

gcloud compute backend-services updateBACKEND_SERVICE \  --region=REGION \  --no-enable-logging

 PATCH https://compute.googleapis.com/compute/v1/projects/PROJECT_ID/regions/REGION/backendServices/BACKEND_SERVICE

 { "logConfig": {   "enable": false  } }

View logs

When logs are ingested into Cloud Logging and not excluded through aLog router sink, you can read logs by using theCloud Logging API and theGoogle Cloud CLI.

To view all the internal passthrough Network Load Balancer logs:

View logs for a backend service

To view the internal passthrough Network Load Balancer logs for a specific backend service:

View logs for a backend instance group

To view the internal passthrough Network Load Balancer logs for a specific backend instance group or network endpoint group (NEG) withGCE_VM_IP endpoints, follow these steps:

Log fields

Log records contain required fields—the default fields of every logrecord—and optional fields that add additional information. Optional fields canbe omitted to save storage costs.

Some log fields are in a multi-field format, with more than one piece of datain a given field. For example, theconnection field is of theIpConnectionformat, which contains the source and destination IP address and port plus theprotocol in a single field. These multi-field fields are described in thefollowing record format table.

The monitored resource isloadbalancing.googleapis.com/InternalNetworkLoadBalancerRule.

IpConnection field format

InstanceDetails field format

VpcDetails field format

GkeDetails field format

ClusterDetails field format

PodDetails field format

ServiceDetails field format

Monitoring

An internal passthrough Network Load Balancer exports monitoring data toCloud Monitoring.

Monitoring metrics can be used for the following purposes:

• Evaluate an internal passthrough Network Load Balancer's configuration, usage, andperformance
• Troubleshoot problems
• Improve resource utilization and user experience

In addition to the predefined dashboards in Monitoring, you can create customdashboards, set up alerts, and query the metrics through theMonitoring API.

View Monitoring dashboards

Define Monitoring custom dashboards

You can create custom Monitoring dashboards forinternal passthrough Network Load Balancer metrics.

Define alerting policies

Metrics for internal passthrough Network Load Balancers

The following metrics for internal passthrough Network Load Balancers are reported intoMonitoring.

Filter dimensions for internal passthrough Network Load Balancer metrics

Metrics are aggregated for each internal passthrough Network Load Balancer. Metrics can befurther broken down by the following dimensions:

Metric reporting frequency and retention

Metrics for the internal passthrough Network Load Balancers are exported to Monitoring inone-minute granularity batches. Monitoring data is retained for six weeks.Metrics are based on sampled traffic (sampling rate is dynamic and cannot beadjusted).The dashboard provides data analysis in default intervals of one hour(1H), six hours (6H), one day (1D), one week (1W), and six weeks (6W). You canmanually request analysis in any interval from six weeks to one minute.

What's next

• For information about fundamentals, seeInternal passthrough Network Load Balancer overview.
• For information about failover, seeFailover for internal passthrough Network Load Balancers.
• For available DNS name options for your load balancer, seeInternal load balancing and DNS names.
• For an example internal passthrough Network Load Balancer configuration, seeSet up an internal passthrough Network Load Balancer with VM instance group backends.
• For information about how to configure and test an internal passthrough Network Load Balancer thatuses failover configuration, seeConfigure failover for internal passthrough Network Load Balancers.
• For information about accessing internal load balancers from peer networksconnected to your VPC network, seeInternal passthrough Network Load Balancers and connected networks.
• For information about how to troubleshoot,seeTroubleshoot internal passthrough Network Load Balancers.
• For information about how to troubleshoot issues with logs, seeTroubleshoot logging issues.