Control plane logging and monitoring

Note: This guide only supports Cloud Service Mesh with Google Cloud APIs anddoes not support Istio APIs. For more information see,Cloud Service Mesh overview.

This document demonstrates how to useCloud Logging andCloud Monitoring to view logs and metrics for theCloud Service Mesh control plane.

Using a service mesh gives you the ability to observe traffic to and fromservices, which allows for richer monitoring and debugging without code changesin the service itself. Log entries can provide important information fortroubleshooting your service mesh, including records of successful connectionsand disconnections, error reports for misconfigured clients, and alertsabout API resource conflicts.

Use cases

The following are three use cases for control plane logging and monitoring:

• Cloud Logging for Cloud Service Mesh control plane: You can securely store, search, analyze, and set alerts on all of yourCloud Service Mesh logging data and events using all the built-in features ofLogging. Cloud Service Mesh exports logs toLogging when an Envoy or gRPC client connects or disconnects,as well as when it detects configuration issues.
• Cloud Monitoring for Cloud Service Mesh control plane: Cloud Service Mesh exports a key metric indicating the number of clientsconnected to the Cloud Service Mesh control plane to Monitoring.You can set up a dashboard in Monitoring and visualize thismetric in real time to monitor the health of the mesh as clients connect anddisconnect. This also lets you set up an SLO for your mesh.
• Troubleshoot issues immediately: Cloud Service Mesh exports telemetry to Logging andMonitoring by default. No additional setup is necessary toconfigure logging and monitoring, enabling you to troubleshoot issues at anytime, including when you are setting up the mesh for the first time.

View logs

To view Cloud Service Mesh logs, use theLogs Explorer. The followingsection presents an example query to view Cloud Service Mesh logs, but you canuse the previous link to create your own query.

1. In the Google Cloud console, go to theLogs Explorer page.

   Go to the Logs Explorer

2. In theResource list,
   • If you use the service routing APIs, selectGateway Scope orMesh.
   • If you use the older APIs, selectGCE Network.
3. In theLog name list, selecttrafficdirector.googleapis.com/events.
4. ClickRun query.

Cloud Service Mesh log entry fields

Field	Description
node_id	ID of the xDS-client node, as provided by the xDS-client.
client_type	Type of xDS-client connected to Cloud Service Mesh. Possible values: ENVOY GRPC-JAVA GRPC-C++ GRPC-PYTHON GRPC-GO UNKNOWN
node_ip	IP address of the node as provided by the client.
api_version	The xDS API version used by xDS clients to connect to Cloud Service Mesh. Possible values areV2 andV3.
description	Text description of the event with additional details.

Example log entries

Example of the log entry	Description
"Cloud Service Mesh could not find any configuration for xDS client."	This log is generated when the xDS client is rejected by Cloud Service Mesh because no configuration exists. This might be due to the incomplete setup of Cloud Service Mesh-relevant API resources.
"Client has successfully connected."	This type of log message is generated every time a client connects successfully to Cloud Service Mesh.
"Client has successfully disconnected."	This type of log message is generated every time an established client is disconnected from Cloud Service Mesh.
"TRAFFICDIRECTOR_INTERCEPTION_PORT metadata variable is not set. Routing configuration for the interception listener exists, but will be ignored."	This type of log message is generated when Cloud Service Mesh resources are configured correctly, but theTRAFFICDIRECTOR_INTERCEPTION_PORT variable is not set in the xDS-client node metadata, so this configuration can not be added to the client.
"Interception listener is built on the given port 15001, but routing configuration does not exist for it."	This type of log message is generated when theTRAFFICDIRECTOR_INTERCEPTION_PORT variable is set in the xDS-client node metadata, but no resources were configured for Cloud Service Mesh to generate a complete xDS response.
"Sending ADS response from Cloud Service Mesh failed. Last discovery request from the node had an incorrect version and/or nonce."	This type of log message is generated when Cloud Service Mesh could not process xDS response correctly due to corrupted communication from the xDS client. This message indicates an implementation error in the client. We recommend checking the client's logs.
"Client sending cross-region traffic to backend servicebackend_service_id. Source region:source_region Destination region(s):destination_region1, destination_region2"	This type of log message is generated when a client reports to Cloud Service Mesh that it sent cross-region traffic from a source region to one or more destination regions.

View metrics

Cloud Service Mesh exports three metrics to Cloud Monitoring:xDS API Connected Streams,Request count, andRequest count by zone.xDS API Connected Streams indicates the number of clients that areconnected to your control plane;Request count indicates the number ofrequests sent to a backend service, grouped by source region, destinationregion, and request status.Request count by zone indicates the numberof requests sent to a backend service, grouped by source zone, destinationzone, and request status. To view these metrics, use theMetrics Explorer.

To view Cloud Service Mesh metrics, do the following:

1. In the Google Cloud console, go to theMetrics Explorer page.

   Go to the Metrics Explorer

2. In theResource type list, select a resource.
   • If you use the service routing APIs, selectGateway Scope orMesh.
   • If you use the older APIs, selectNetwork.
3. In theMetric list, selectconnected_clients.
4. Return to theResource type list, and then selectCompute Engine BackendService.
5. In theMetric list, select eitherRequest count orRequest count by zone.

Alternatively, you can use a query to view the cross-region request count:

1. SelectMQL.
2. In the field, enter the following example query:

   fetch gce_backend_service| metric 'trafficdirector.googleapis.com/xds/server/request_count'| filter ( ne(metric.source_region, metric.destination_region))| align rate(1m)| every 1m| group_by [metric.source_region, metric.destination_region, resource.backend_service_id],[value_request_count_aggregate: aggregate(value.request_count)]

3. ClickRun query.

Set up logs-based metrics and alerts

The following steps require that you set up logs-based metrics. For moreinformation about setting up logs-based metrics, see theoverview.

You can configure alerts to notify you when user-specified messages appear inyour included logs. These alerts can notify the operator when somethingunexpected occurs. For example, if a change in Cloud Service Mesh configurationresults in API resource conflicts, you can receive an alert on the errormessage. To set up alerts on your log-based metrics, seeConfigure charts and alerts.