Cloud Trace and Cloud Service Mesh

Cloud Trace is a distributed tracing system that collectslatency data from the applications and displays it in near real-time. It allowsyou to follow a sample request through your distributed system, observe thenetwork calls and profile your system end to end.

Cloud Trace is available with Cloud Service Mesh installations on thefollowing platforms:

• GKE on Google Cloud
• GKE Enterprise clusters on-premises if you installed withCloud Service Mesh certificate authority

Note that Cloud Trace is disabled by default. Once enabled,Cloud Service Mesh pages in the Google Cloud console provide a link to thetraces in the Cloud Trace page . For detailed pricing information, refer totheCloud Trace pricing page.

Enable Cloud Trace

This section shows you how to enable Cloud Trace.

 ./asmcli install \OTHER_FLAGS \    --option cloud-trace

apiVersion:install.istio.io/v1alpha1kind:IstioOperatorspec:meshConfig:enableTracing:truevalues:global:proxy:tracer:stackdriver

apiVersion:install.istio.io/v1alpha1kind:IstioOperatorspec:meshConfig:enableTracing:truedefaultConfig:tracing:sampling:100values:global:proxy:tracer:stackdriver

 ./asmcli install \OTHER_FLAGS \    --custom_overlayPATH_TO_FILE

kubectl rollout restart deployment -nNAMESPACEDEPLOYMENT_NAME

Trace context propagation

Although the sidecar proxies can automatically send trace spans, they need somehints to tie together the entire trace. Applications need to propagate theappropriate HTTP headers so that when the proxies send span information, thespans can be correlated correctly into a single trace.

To do this, an application needs to collect and propagate the appropriateheaders from the incoming request to any outgoing requests. TheCloud Service Mesh Stackdriver tracing configuration will accept any of thefollowing header formats, and will propagateall of the following formats:

• B3 (x-b3-traceid,x-b3-spanid,x-b3parentspanid,x-b3-sampled,x-b3-flags)
• W3C TraceContext (traceparent)
• Google Cloud Trace (x-cloud-trace-context)
• gRPC TraceBin (grpc-trace-bin)

This means that your applications can use any of those formats to propagatetracing context and the traces will be generated and set to Stackdriverappropriately.

Example

Here is an example HTTP-Get request with atraceparent header in the originalrequest. Notice the additional trace context headers added by the proxy.

$kubectlexec-itsleep-557747455f-n6flv--curl"httpbin:8000/anything?freeform="-H"accept: application/json"-H"Traceparent: 00-7543d15e09e5d61801d4f74cde1269b8-604ef051d35c5b3f-01"-vv*Trying10.12.3.52:8000...*Connectedtohttpbin(10.12.3.52)port8000(#0)>GET/anything?freeform=HTTP/1.1>Host:httpbin:8000>User-Agent:curl/7.80.0-DEV>accept:application/json>Traceparent:00-7543d15e09e5d61801d4f74cde1269b8-604ef051d35c5b3f-01>*Markbundleasnotsupportingmultiuse<HTTP/1.1200OK<server:envoy<date:Wed,10Nov202120:36:04GMT<content-type:application/json<content-length:1032<access-control-allow-origin:*<access-control-allow-credentials:true<x-envoy-upstream-service-time:5<{"args":{"freeform":""},"data":"","files":{},"form":{},"headers":{"Accept":"application/json","Grpc-Trace-Bin":"AAB1Q9FeCeXWGAHU90zeEmm4AaDHmGRtdM7wAgE","Host":"httpbin:8000","Traceparent":"00-7543d15e09e5d61801d4f74cde1269b8-a0c798646d74cef0-01","User-Agent":"curl/7.80.0-DEV","X-B3-Sampled":"1","X-B3-Spanid":"a0c798646d74cef0","X-B3-Traceid":"7543d15e09e5d61801d4f74cde1269b8","X-Cloud-Trace-Context":"7543d15e09e5d61801d4f74cde1269b8/11585396123534413552;o=1","X-Envoy-Attempt-Count":"1","X-Forwarded-Client-Cert":"<REDACTED>"},"json":null,"method":"GET","origin":"127.0.0.6","url":"http://httpbin:8000/anything?freeform="}

Notice that in the returned set of request headers, the full set of tracecontext headers is present.

For more examples propagating the headers, seeTrace context propagation.

Create a trace from client with custom ID

To create a trace from a client with a custom ID, use thecurl command tocreate a request with an external client and force it to show a trace. For example:

curl $URL --header "x-client-trace-id: 105445aa7843bc8bf206b12000100000"

For more information aboutx-client-trace-id, refer to theEnvoy documentation.

Access traces

View trace samples for a service

To view a sampling of traces for a service in your app, follow these steps:

1. Go to theCloud Service Mesh page in the Google Cloud console.

   Go to the Cloud Service Mesh page

2. UnderServices, select the name of the Service you want to inspect.

   The following screenshot shows an example of afrontend Service.

   

3. UnderRequest traces, click any trace to see more information.

   The following screenshot shows an example of the trace request subpanel.

   

View all traces

To view all traces for a Service, follow these steps:

1. Go to theCloud Service Mesh page in the Google Cloud console.

   Go to the Cloud Service Mesh page

2. UnderServices, select the name of the Service you want to inspect.

3. Go to theMetrics page.

4. Specify a time span from theTime Span dropdown menu orset a custom span with the timeline.

5. ClickView traces.

The traces for a service in Cloud Service Mesh contain following information:

• Request latencies across different services in the mesh.
• HTTP request properties, including ID, URL, size, latency, and protocol.
• Service name, namespace and mesh id as part of the labelsistio.canonical_service,istio.namespace, andistio.mesh_id,respectively.

What's next

• Viewing traces in the Google Cloud console