This document provides a brief overview on how to instrument your applicationfor Cloud Trace. For detailed instructions on setting up Cloud Trace,see the language-specific setup pages.

Cloud Trace provides distributed tracing data for your applications. Afterinstrumenting your application, you can inspect latency data for a singlerequest and view the aggregate latency for an entire application in theCloud Trace console.

When to instrument your application

Note: Trace data is automatically captured for some configurations and for some Google Cloud services, so you might not need to instrument your application. For more information, seeConfigurations with automatic tracing.

When trace data isn't automatically captured, you need to instrumentyour application to collect this data.

You can instrument your application so that it collectsapplication-specific information. Several open-sourceinstrumentation frameworks let you collect metrics, logs, andtraces from your application and send that data to any vendor,including Google Cloud. To instrument your application, we recommend that you use avendor-neutral instrumentation framework that is open source, such asOpenTelemetry, instead ofvendor- and product-specific APIs or client libraries.

For information about instrumenting your applications by usingvendor-neutral instrumentation frameworks, see Instrumentation and observability.

How to instrument applications

Note: Typically, we recommend that you useOpenTelemetry.You can export your data to your Google Cloud project by sending your data to theGoogle Cloud OTLP endpoint or by using the Cloud Trace exporter.For instrumentation recommendations, seeChoose an instrumentation approach.

To instrument your applications to collect trace data, you can doany of the following:

• You can useOpenTelemetry and the associatedCloud Trace exporter for the following programming languages:

  OpenTelemetry SDK	Example
  Go SDK	Go sample
  Java SDK	Java sample
  Node.js SDK	Node.js sample
  Python SDK	Python sample
  C++ SDK	C++ sample
  Ruby SDK	See OpenTelemetry documentation.

• You can useOpenTelemetry and send your data to theOTLP endpoint. To learn about this endpoint and why we recommend it, seeMigrate from the Trace exporter to the OTLP endpoint.

• If you are writing applications that run on Compute Engine, then you canuse the Ops Agent and the OpenTelemetry Protocol (OTLP) receiver to collecttraces and metrics from your application. The Ops Agent can also collectlogs, but not by using OTLP. For more information, seeUse the Ops Agent and OTLP andOps Agent overview.

• You can use the client libraries or directly call theCloud Trace APIto send tracing data to Cloud Trace. However, we recommend that youuse OpenTelemetry when your language is supported by that library.

• You can configureSpring Boot applications toforward the trace data it collects toCloud Trace. For information about this procedure, seeSpring Cloud for Google Cloud: Cloud Trace.

When to create spans

The Cloud Trace client libraries typically maintain a global tracecontext that holds information about the current span, including its trace IDand whether the trace is sampled. These libraries usually create spans on RPCboundaries. However, you might need to create spans if the default creationalgorithm isn't sufficient for your needs.

The current active span can be accessed by the global trace context, which issometimes wrapped in aTracer object. You can addinformation relevant to your application by using custom annotations and tagsto existing spans, or you can create new child spans with their own annotationsand tags to trace the application's behavior with finer granularity. Becausethe context is global, multi-threaded applications that update the context mustuse appropriate isolation.

When to provide authentication credentials

You generally don't need to provide authentication credentials toyour application or specify your Google Cloud project ID in yourapplication when you are running on Google Cloud. For some languages,you do need to specify your Google Cloud project ID even if you arerunning on Google Cloud. Also, if you use Autopilot mode forGoogle Kubernetes Engine, or if you enable Workload Identity Federation for GKE, then you mustconfigure your application to use Workload Identity Federation for GKE.

If you are running outside of Google Cloud, you need to provideauthentication credentials to your application. You also need to specify yourGoogle Cloud project ID in your application.

For details, go to the language-specific setup pages.

How to force a request to be traced

Unless your application always samples every span,it isn't possible, in general, to force a request to be traced end-to-endbecause each component in an end-to-end request makes its ownsampling decision. However, you can influence thedecision by adding to the trace header asampled flag,with this flag set totrue. This setting is a hint to child componentsto sample the request.For more information about trace headers, seeProtocols for context propagation.

For downstream components whose code you own, you must determine whetheryour instrumentation logic respects thesampled flag.For example, when usingOpenTelemetryfor instrumentation, you can use theParentBased samplerto ensure that the parent sampled flag is respected.

Google Cloud services that record tracing information to Cloud Tracetypically accept the parent sampling flag as a hint; however, mostservices also rate-limit sampling. Each Google Cloud servicedetermines whether it supports tracing, how the parent samplingflag is utilized, and the rate limit on sampling.

How to correlate metric and trace data

You can correlate distribution-valued metric datawith traces by attaching exemplars to the metric data points.Provided you complete the necessary configuration steps,OpenTelemetry, which is the recommended instrumentation library,automatically adds these exemplars. For more information, seeCorrelate metrics and traces by using exemplars.

Configure your project and platform

1. Ensure that the Cloud Trace API is enabled.

   By default, Google Cloud projects have the Cloud Trace API enabledand you don't need to take any action. However, security constraints definedby your organization might have disabled the API. For troubleshootinginformation, seeDevelop applications in a constrained Google Cloud environment.

   Enable the Cloud Trace API.

   Roles required to enable APIs

   To enable APIs, you need the Service Usage Admin IAM role (roles/serviceusage.serviceUsageAdmin), which contains theserviceusage.services.enable permission.Learn how to grant roles.

   Enable the API

2. Configure your platform.

   You can use Cloud Trace on Google Cloud and other platforms.

   • Google Cloud: When your application is running onGoogle Cloud, you don't need to provide authentication credentialsin the form of a service account to the client library. However, mustensure that your Google Cloud platform has theCloud Trace API access scopeenabled.

     For the following configurations, the default access-scope settingsinclude the Cloud Trace API access scope:

     • Compute Engine

     • Google Kubernetes Engine (GKE)

       Note: If you use Autopilot or if you enableWorkload Identity Federation for GKE for your GKE cluster,then ensure that youconfigure your application to use Workload Identity Federation for GKE.

     • App Engine flexible environment

     • App Engine standard environment

     • Cloud Run

     If you use custom access scopes, then you must ensure thatCloud Trace API access scope is enabled.For example, if you use the Google Cloud CLI to create aGKE cluster and if you specify the--scopes flag,then ensure the scope includestrace.append. The following commandillustrates setting the--scopes flag:

     gcloud container clusters create example-cluster-name --scopes=https://www.googleapis.com/auth/trace.append

   • Running locally and elsewhere: If your application is running outside ofGoogle Cloud, then you must provide authentication credentials inthe form of aservice account to the client library.The service account must be granted the Cloud Trace agent(roles/cloudtrace.agent) role. For information about roles, seeControl access with IAM.

     Google Cloud client libraries useApplication Default Credentials (ADC) to find yourapplication's credentials. You can provide these credentials in one ofthree ways:

     • Rungcloud auth application-default login

     • Place the service account in a default path for your operating system.The following lists the default paths for Windows and Linux:

       • Windows:%APPDATA%/gcloud/application_default_credentials.json

       • Linux:$HOME/.config/gcloud/application_default_credentials.json

     • Set theGOOGLE_APPLICATION_CREDENTIALS environment variable tothe path to your service account:

       Linux/macOS

       exportGOOGLE_APPLICATION_CREDENTIALS=path-to-your-service-accounts-private-key

       Windows

       setGOOGLE_APPLICATION_CREDENTIALS=path-to-your-service-accounts-private-key

       PowerShell:

       $env:GOOGLE_APPLICATION_CREDENTIALS="path-to-your-service-accounts-private-key"

What's next

For detailed configuration information, samples, and links to GitHub andother open source repositories, go to the setup page for your language.

• OpenTelemetry examples:

  • Python sample

  • Java sample

  • Node.js sample

  • Go sample

  • C++ sample