Request proxy logs

Cloud Service Mesh supports two different types of access logs inCloud Logging:Traffic logs (alsoknown as Google Cloud Observability access logs) andEnvoy access logs. This page shows you how toenable, disable, view, and interpret these logs. Note that traffic logs areenabled by default.

Enabling and disabling access logs

Managed Cloud Service Mesh

Note: Cloud Service Mesh with a managed Cloud Service Mesh control plane onlysupports a cluster-wide telemetry API configuration, specifically, a singleIstio telemetry API CR in the root namespace. Don't deploy multiple Istiotelemetry API CRs in a cluster. If you deploy multiple Istio telemetry APICRs in a cluster, only the first one takes effect.If you already have an existing Istio telemetry API CR, to preserve yourprevious customization settings in your existing Istio telemetry CR, youmust include them within your new Istio telemetry API CR.

Envoy access logs

Run the following command to enable Envoy access logs and disable trafficlogs:

cat<<EOF|kubectlapply-nistio-system-f-apiVersion:telemetry.istio.io/v1alpha1kind:Telemetrymetadata:name:enable-envoy-disable-sdnamespace:istio-systemspec:accessLogging:-providers:-name:envoy-providers:-name:stackdriverdisabled:trueEOF

Note that the provider name for traffic log isstackdriver.

Traffic logs

By default, traffic logs are enabled and Envoy access logs are disabled. Ifyou previously enabled Envoy access logs and want to enable traffic logs anddisable Envoy access logs, run the following command:

cat<<EOF|kubectlapply-nistio-system-f-apiVersion:telemetry.istio.io/v1alpha1kind:Telemetrymetadata:name:disable-envoy-enable-sdnamespace:istio-systemspec:accessLogging:-providers:-name:envoydisabled:true-providers:-name:stackdriverEOF

Both

  • To enable both Envoy access logs and traffic logs, run the followingcommand:

    cat<<EOF|kubectlapply-nistio-system-f-apiVersion:telemetry.istio.io/v1alpha1kind:Telemetrymetadata:name:enable-envoy-and-sd-access-lognamespace:istio-systemspec:accessLogging:-providers:-name:envoy-name:stackdriverEOF

  • To disable both Envoy access logs and traffic logs, run the followingcommand:

    cat<<EOF|kubectlapply-nistio-system-f-apiVersion:telemetry.istio.io/v1alpha1kind:Telemetrymetadata:name:disable-envoy-and-sdnamespace:istio-systemspec:accessLogging:-providers:-name:envoydisabled:true-providers:-name:stackdriverdisabled:trueEOF

Managedistiod

Envoy access logs

Run the following commands to enable Envoy access logging:

Warning: You can overwrite your own changes. If you already have an existingconfigmap, you must include all previous customization settings within themesh: section to preserve your changes. This ensures no downtime when enabling optional features.

  1. Run the following command to addaccessLogFile: /dev/stdout:

    cat<<EOF|kubectlapply-f-apiVersion:v1data:mesh:|-accessLogFile:/dev/stdoutkind:ConfigMapmetadata:name:istio-release-channelnamespace:istio-systemEOF

    whererelease-channel is yourrelease channel(asm-managed,asm-managed-stable, orasm-managed-rapid).

  2. Run the following command to view the configmap:

     kubectl get configmap istio-release-channel -n istio-system -o yaml

  3. To verify that access logging is enabled, ensure theaccessLogFile: /dev/stdout line appears in themesh: section.

    ...apiVersion:v1data:mesh:|....accessLogFile: /dev/stdout...

Traffic logs

Traffic logs are enabled by default.

In-cluster

Envoy access logs

---apiVersion:install.istio.io/v1alpha1kind:IstioOperatorspec:meshConfig:accessLogFile:"/dev/stdout"

For more information, seeEnable Envoy's access logging.

Traffic logs

Traffic logs are enabled by default,unless Cloud Service Mesh is installed onGoogle Distributed Cloud with Istio CA (previously known as Citadel).

To enable traffic logs on Google Distributed Cloud with Istio CAwheninstalling in-cluster Cloud Service Mesh,use the flag--option stackdriver. Alternatively, you can enable trafficlogs on Google Distributed Cloud with Istio CAafter installing in-cluster Cloud Service Mesh.

Viewing access logs

Envoy access logs

Command line

To view Envoy access logs in the istio-proxy log, run the following command:

kubectllogsPOD_NAME-nNAMESPACE_NAME-cistio-proxy

Logs Explorer

To view Envoy access logs in theLogs Explorer:

  1. Navigate to the Logs Explorer:

    Go to the Logs Explorer

  2. Select the appropriate Google Cloud project.

  3. Run the following query:

resource.type="k8s_container"\resource.labels.container_name="istio-proxy"resource.labels.cluster_name="CLUSTER_NAME"\resource.labels.namespace_name="NAMESPACE_NAME"\resource.labels.pod_name="POD_NAME"

Traffic logs

To view traffic logs in theLogs Explorer:

  1. Navigate to the Logs Explorer:

    Go to the Logs Explorer

  2. Select the appropriate Google Cloud project.

  3. Run the following query depending on whether you are viewing client orserver access logs:

    Server logs

    resource.labels.cluster_name="CLUSTER_NAME"logName="projects/PROJECT_NAME/logs/server-accesslog-stackdriver"

    Client logs

    resource.labels.cluster_name="CLUSTER_NAME"logName="projects/PROJECT_NAME/logs/client-accesslog-stackdriver"

To view traffic logs in the Cloud Service Mesh page for a Service duringa specified time span, follow these steps:

  1. In Google Cloud console, go to theCloud Service Mesh page.

    Go to the Cloud Service Mesh page

  2. UnderServices, select the name of the Service you want to inspect.

  3. Go to theMetrics page.

  4. Specify a time span from theTime Span drop-down menu orset a custom span with the timeline.

  5. UnderSelect a filter option, clickView traffic logs.

The traffic log is named asserver-accesslog-stackdriver and is attachedto the corresponding monitored resource(k8s_container orgce_instance) your service isusing. The traffic log contains the following information:

  • HTTP request properties, such as ID, URL, size, latency, and commonheaders.

  • Source and destination workload information, such as name, namespace,identity, and common labels.

  • If tracing is enabled, trace information, such as sampling, trace ID, andspan ID.

An example log entry looks like the following:

{  insertId: "1awb4hug5pos2qi"  httpRequest: {    requestMethod: "GET"    requestUrl: "YOUR-INGRESS/productpage"    requestSize: "952"    status: 200    responseSize: "5875"    remoteIp: "10.8.0.44:0"    serverIp: "10.56.4.25:9080"    latency: "1.587232023s"    protocol: "http"  }  resource: {    type: "k8s_container"    labels: {      location: "us-central1-a"      project_id: "YOUR-PROJECT"      pod_name: "productpage-v1-76589d9fdc-ptnt9"      cluster_name: "YOUR-CLUSTER-NAME"      container_name: "productpage"      namespace_name: "default"    }  }  timestamp: "2020-04-28T19:55:21.056759Z"  severity: "INFO"  labels: {    destination_principal: "spiffe://cluster.local/ns/default/sa/bookinfo-productpage"    response_flag: "-"    destination_service_host: "productpage.default.svc.cluster.local"    source_app: "istio-ingressgateway"    service_authentication_policy: "MUTUAL_TLS"    source_name: "istio-ingressgateway-5ff85d8dd8-mwplb"    mesh_uid: "YOUR-MESH-UID"    request_id: "021ce752-9001-4ac6-b6d6-3b15f5d3632"    destination_namespace: "default"    source_principal:  "spiffe://cluster.local/ns/istio-system/sa/istio-ingressgateway-service-account"    destination_workload: "productpage-v1"    destination_version: "v1"    source_namespace: "istio-system"    source_workload: "istio-ingressgateway"    destination_name: "productpage-v1-76589d9fdc-ptnt9"    destination_app: "productpage"  }  trace: "projects/YOUR-PROJECT/traces/d4197f59b7a43e3aeff3571bac99d536"  receiveTimestamp: "2020-04-29T03:07:14.362416217Z"  spanId: "43226343ca2bb2b1"  traceSampled: true  logName: "projects/YOUR-PROJECT/logs/server-accesslog-stackdriver"  receiveTimestamp: "2020-04-28T19:55:32.185229100Z"}

Interpret Cloud Service Mesh telemetry

The following sections explain how to check the status of your mesh and reviewthe various telemetry that contain helpful details to assist yourtroubleshooting.

Interpret control plane metrics

Managed Cloud Service Mesh

Cloud Service Mesh with a managed Cloud Service Mesh control plane doesn't supportcontrol plane metrics.

Managedistiod

Cloud Service Mesh with a managedistiod control plane doesn't supportthe control plane metric inspection in this section.

In-cluster

When installing Cloud Service Mesh with the in-cluster control plane,istiodexports metrics to Google Cloud Observability for monitoring, by default.istiod prefixes these metrics withistio.io/control and give insights intothe control plane state, such as number of proxies connected to each controlplane instance, configuration events, pushes and validations.

Observe or troubleshoot the control plane, using the following steps.

  1. Load a sample dashboard:

    git clone https://github.com/GoogleCloudPlatform/monitoring-dashboard-samples && cd monitoring-dashboard-samples/dashboards && git checkout servicemesh

  2. Install the Cloud Service Mesh dashboard:

    gcloud monitoring dashboards create --config-from-file=dashboards/servicemesh/anthos-service-mesh-control-plane-monitoring.json

  3. Look for a dashboard namedIstio Control Plane Dashboard in the list.For more information, seeViewing the installed dashboard.

For the full list of metrics available, seeExported metrics.

Diagnose configuration delays

Managed Cloud Service Mesh

Cloud Service Mesh with a managed Cloud Service Mesh control plane doesn't supportdiagnosing configuration delays.

Managedistiod

Cloud Service Mesh with a managed istiod control plane doesn't supportdiagnosing configuration delays.

In-cluster

The following steps explain how to use thepilot_proxy_convergence_time metricto diagnose a delay between a configuration change and all proxies converging.

  1. Run a shell command in a pod:

    kubectl debug --image istio/base --target istio-proxy -it $(kubectl get pod -l app=pilot -o jsonpath='{.items[0].metadata.name}' -n istio-system) -n istio-system -- curl -s

  2. Accesslocalhost:15014 andgrep forconvergence in metrics:

    curl http://localhost:15014/metrics | grep convergence

Interpret traffic logs

The following information explains how to use the traffic logs to troubleshootconnection problems. Traffic logs are enabled by default.

Cloud Service Mesh exports data into traffic logs that can help you debug thefollowing types of problems:

  • Traffic flow and failures
  • End-to-end request routing

Traffic logs are enabled by default for Cloud Service Mesh installations onGoogle Kubernetes Engine. You can enable traffic logs by re-runningasmcli install. Usethe same options that you originally installed but omit the custom overlay thatdisabled Stackdriver.

There are two types of traffic logs:

  • Server access logs give a server-side view of requests. They are locatedunderserver-accesslog-stackdriver, attached to thek8s_container monitoredresource. Use the following URL syntax todisplay server-side access logs:

    https://console.cloud.google.com/logs/viewer?advancedFilter=logName="projects/PROJECT_ID/logs/server-accesslog-stackdriver"&project=PROJECT_ID

  • Client access logs give a client-side view of requests. They are located underclient-accesslog-stackdriver, attached to thek8s_pod monitored resource.Use the following URL syntax to display client-side access logs:

    https://console.cloud.google.com/logs/viewer?advancedFilter=logName="projects/PROJECT_ID/logs/client-accesslog-stackdriver"&project=PROJECT_ID

Access logs contain the following information:

  • HTTP request properties, such as ID, URL, size, latency, and common headers.
  • Source and destination workload information, such as name, namespace, identity,and common labels.
  • Source and destination canonical service and revision information.
  • If tracing is enabled, the logs contain trace information, such as sampling,trace ID, and span ID.

Traffic logs may contain the following labels:

  • route_name
  • upstream_cluster
  • X-Envoy-Original-Path

This is an example log entry:

{  "insertId": "1j84zg8g68vb62z",  "httpRequest": {    "requestMethod": "GET",    "requestUrl": "http://35.235.89.201:80/productpage",    "requestSize": "795",    "status": 200,    "responseSize": "7005",    "remoteIp": "10.168.0.26:0",    "serverIp": "10.36.3.153:9080",    "latency": "0.229384205s",    "protocol": "http"  },  "resource": {    "type": "k8s_container",    "labels": {      "cluster_name": "istio-e2e22",      "namespace_name": "istio-bookinfo-1-68819",      "container_name": "productpage",      "project_id": "***",      "location": "us-west2-a",      "pod_name": "productpage-v1-64794f5db4-8xbtf"    }  },  "timestamp": "2020-08-13T21:37:42.963881Z",  "severity": "INFO",  "labels": {    "protocol": "http",    "upstream_host": "127.0.0.1:9080",    "source_canonical_service": "istio-ingressgateway",    "source_namespace": "istio-system",    "x-envoy-original-path": "",    "source_canonical_revision": "latest",    "connection_id": "32",    "upstream_cluster": "inbound|9080|http|productpage.istio-bookinfo-1-68819.svc.cluster.local",    "requested_server_name": "outbound_.9080_._.productpage.istio-bookinfo-1-68819.svc.cluster.local",    "destination_version": "v1",    "destination_workload": "productpage-v1",    "source_workload": "istio-ingressgateway",    "destination_canonical_revision": "v1",    "mesh_uid": "cluster.local",    "source_principal": "spiffe://cluster.local/ns/istio-system/sa/istio-ingressgateway-service-account",    "x-envoy-original-dst-host": "",    "service_authentication_policy": "MUTUAL_TLS",    "destination_principal": "spiffe://cluster.local/ns/istio-bookinfo-1-68819/sa/bookinfo-productpage",    "response_flag": "-",    "log_sampled": "false",    "destination_service_host": "productpage.istio-bookinfo-1-68819.svc.cluster.local",    "destination_name": "productpage-v1-64794f5db4-8xbtf",    "destination_canonical_service": "productpage",    "destination_namespace": "istio-bookinfo-1-68819",    "source_name": "istio-ingressgateway-6845f6d664-lnfvp",    "source_app": "istio-ingressgateway",    "destination_app": "productpage",    "request_id": "39013650-4e62-9be2-9d25-78682dd27ea4",    "route_name": "default"  },  "logName": "projects/***/logs/server-accesslog-stackdriver",  "trace": "projects/***t/traces/466d77d15753cb4d7749ba5413b5f70f",  "receiveTimestamp": "2020-08-13T21:37:48.758673203Z",  "spanId": "633831cb1fda4fd5",  "traceSampled": true}

You can use this log in various ways:

  • Integrate with Cloud Trace, which is an optional feature inCloud Service Mesh.
  • Export traffic logs to BigQuery, where you can run queries likeselecting all requests take more than 5 seconds.
  • Create log-based metrics.
  • Troubleshoot404 and503 errors

Troubleshoot404 and503 errors

The following example explains how to use this log to troubleshoot when arequest fails with a404 or503 response code.

  1. In the client access log, search for an entry like the following:

    httpRequest: {requestMethod: "GET"requestUrl: "://IP_ADDRESS/src/Util/PHP/eval-stdin.php"requestSize: "2088"status: 404responseSize: "75"remoteIp: "10.168.0.26:34165"serverIp: "10.36.3.149:8080"latency: "0.000371440s"protocol: "http"}

  2. Navigate to the labels in the access log entry. Findtheresponse_flag field that looks like the following:

    response_flag: "NR"

    TheNR value is an acronym forNoRoute, which means no route was found for thedestination or there was no matching filter chain for a downstreamconnection. Similarly, you can use theresponse_flag label to troubleshoot503 errors also.

  3. If you see503 errors in both the client and server access logs, check thatthe port names set for each service match the name of the protocol in usebetween them. For example, if a golang binary client connects to a golang serverusing HTTP, but the port is namedhttp2, the protocol will not auto-negotiatecorrectly.

For more information, seeresponse flags.

Interpret Envoy access logs

The following steps explain how to use the Envoy access logs to show trafficbetween both ends of a connection for troubleshooting purposes.

Envoy access logs are useful for diagnosing issues like:

  • Traffic flow and failures
  • End-to-end request routing

Envoy access logs are not enabled by default in Cloud Service Mesh and can beenabled for the clusters in a mesh.

Note: Traffic logs (also known as theGoogle Cloud Observability logs) are the preferred logs formost troubleshooting scenarios. However, if the issue persists, enable Envoyaccess logs to troubleshoot further.

You can troubleshoot connection or request failures by generating activity inyour application that triggers an HTTP request, then inspecting the associatedrequest in the source or destination logs.

If you trigger a request appears and it appears in the source proxy logs, itindicates thatiptables traffic redirection is working correctly and the Envoyproxy is handling traffic. If you see errors in the logs, generate an Envoyconfiguration dump and check the Envoy cluster configuration to ensure it iscorrect. If you see the request but the log has no errors, check the destinationproxy logs instead.

If the request appears in the destination proxy logs, it indicates that the meshitself is working correctly. If you see an error instead, run an Envoyconfiguration dump and verify the correct values for the traffic port set inthe listener configuration.

If the problem persists after performing the previous steps, Envoy might beunable to auto-negotiate the protocol between the sidecar and its applicationpod. Ensure that the Kubernetes service port name, for examplehttp-80,matches the protocol that the application uses.

Use Logs Explorer to query logs

You can use theLogs Explorer interfaceto query specific Envoy access logs. For example, to query all requests thathaveMULTUAL_TLS enabled and use protocolgrpc, append following to theserver access logs query:

labels.protocol="grpc" labels.service_authentication_policy="MULTUAL_TLS"

Set an access log policy

Managed Cloud Service Mesh

To configure access logs for Cloud Service Mesh with a managedCloud Service Mesh control plane, seeEnabling access logs.

Managedistiod

To configure access logs for Cloud Service Mesh with a managedistiod control plane, seeEnabling access logs.

In-cluster

To set an access log policy for Cloud Service Mesh with the in-cluster controlplane:

  1. Create anIstioOperator custom overlay file that includes the applicableAccessLogPolicyConfigvalues for your scenario.

  2. Pass this file toasmcli using the--custom_overlay option to updatethe in-cluster control plane configuration. For information on runningasmcli install with a custom overlay file, seeInstall with optional features.

    Caution: If you customized the previous installation, you need the samecustomizations when you reinstall the control plane to include the filewith theAccessLogPolicyConfig settings. Omitting an overlay file resultsin a successful installation; however, your service mesh has a differentconfiguration than you intended because the omitted values are reset back tothe defaults when the control plane is redeployed.

View service or workload-specific information

If you have an issue with a specific service or workload rather than a mesh-wideproblem, inspect the individual Envoy proxies and gather relevant informationfrom them. To gather information about a particular workload and its proxies,you can usepilot-agent:

kubectl execPOD_NAME -nNAMESPACE_NAME -c istio-proxy -- pilot-agent request GETSCOPE

In the example,SCOPE is one of the following:

  • certs - Certificates within the Envoy instance
  • clusters - Clusters with Envoy configured
  • config_dump - Dumps the Envoy configuration
  • listeners - Listeners with Envoy configured
  • logging - View and change logging settings
  • stats - Envoy statistics
  • stats/prometheus - Envoy statistics as Prometheus records
Note:config_dump andstats are the options that are most likely to produceactionable information.

View proxy socket states

You can directly examine the state of Envoy proxy sockets by using the followingprocess.

  1. Display a list of established sockets, including sockets in theTIME_WAITstate, which can negatively affect scalability if their count is high:

    kubectl debug --image istio/base --target istio-proxy -itPOD_NAME -nNAMESPACE_NAME -- ss -anopim

  2. Display a summary of socket statistics:

    kubectl debug --image istio/base --target istio-proxy -itPOD_NAME -nNAMESPACE_NAME -- ss -s

For more information, seeAn Introduction to the ss Command.

istio-proxy andistio-init logs

In addition, retrieve theistio-proxy logs and review its contents for anyerrors that might suggest the cause of the problem:

kubectl logsPOD_NAME -nNAMESPACE_NAME -c istio-proxy

You can do the same for theinit container:

kubectl logsPOD_NAME -nNAMESPACE_NAME -c istio-init

What's next

Except as otherwise noted, the content of this page is licensed under theCreative Commons Attribution 4.0 License, and code samples are licensed under theApache 2.0 License. For details, see theGoogle Developers Site Policies. Java is a registered trademark of Oracle and/or its affiliates.

Last updated 2026-02-19 UTC.