This document describes how to instrument a Java app to collect trace andmetric data using theOpenTelemetry SDK and an OpenTelemetry collector.It also describes how to write structured JSON logs tostandard output. To experiment with the instrumentation, download and run thesample app. This app uses theSpring Boot Framework andgenerates log, metric, and trace data.

When you use an OpenTelemetry collector, you instrument your application withthe SDK and the SDK's OTLP in-process exporter. This instrumentation isvendor neutral. You also deploy an OpenTelemetry collector that receives telemetryfrom the in-process exporter and then exports thattelemetry to your Google Cloud project. To learn more about collectors, seeGoogle-Built OpenTelemetry Collector.

We recommend that you use an OpenTelemetry collector to export your telemetrydata when your environment supports use of collector. For some environments,you must use an in-process exporter that directly sends data to yourGoogle Cloud project. To learn about in-process instrumentation, seeMigrate from the Trace exporter to the OTLP endpoint.

To learn more about instrumentation, see the following documents:

• Instrumentation and observability.
• Choose an instrumentation approach.

The instrumentation described in this document relies on OpenTelemetryzero-code instrumentation to send telemetry to your Google Cloud project.For Java,zero-code instrumentation refers to the practice ofdynamically injecting bytecode into libraries and frameworks to capturetelemetry. Zero-code instrumentation can collect telemetry for things likeinbound and outbound HTTP calls. For more information, seeJava Agent.

OpenTelemetry also provides an API for adding custom instrumentation toyour own code. OpenTelemetry refers to this asmanual instrumentation.This document doesn't describe manual instrumentation. For examples andinformation about that topic, seeManual instrumentation.

Before you begin

Sign in to your Google Cloud account. If you're new to Google Cloud, create an account to evaluate how our products perform in real-world scenarios. New customers also get $300 in free credits to run, test, and deploy workloads.

Install the Google Cloud CLI.

If you're using an external identity provider (IdP), you must first sign in to the gcloud CLI with your federated identity.

Toinitialize the gcloud CLI, run the following command:

gcloudinit

Create or select a Google Cloud project.

Roles required to select or create a project

• Select a project: Selecting a project doesn't require a specific IAM role—you can select any project that you've been granted a role on.
• Create a project: To create a project, you need the Project Creator role (roles/resourcemanager.projectCreator), which contains theresourcemanager.projects.create permission.Learn how to grant roles.

Note: If you don't plan to keep the resources that you create in this procedure, create a project instead of selecting an existing project. After you finish these steps, you can delete the project, removing all resources associated with the project.

• Create a Google Cloud project:

  gcloud projects createPROJECT_ID

  ReplacePROJECT_ID with a name for the Google Cloud project you are creating.

• Select the Google Cloud project that you created:

  gcloud config set projectPROJECT_ID

  ReplacePROJECT_ID with your Google Cloud project name.

Verify that billing is enabled for your Google Cloud project.

Enable the Cloud Logging, Cloud Monitoring, Cloud Trace, and Telemetry APIs:

Roles required to enable APIs

To enable APIs, you need the Service Usage Admin IAM role (roles/serviceusage.serviceUsageAdmin), which contains theserviceusage.services.enable permission.Learn how to grant roles.

gcloudservicesenablelogging.googleapis.com monitoring.googleapis.com cloudtrace.googleapis.com telemetry.googleapis.com

Install the Google Cloud CLI.

If you're using an external identity provider (IdP), you must first sign in to the gcloud CLI with your federated identity.

Toinitialize the gcloud CLI, run the following command:

gcloudinit

Create or select a Google Cloud project.

Roles required to select or create a project

• Select a project: Selecting a project doesn't require a specific IAM role—you can select any project that you've been granted a role on.
• Create a project: To create a project, you need the Project Creator role (roles/resourcemanager.projectCreator), which contains theresourcemanager.projects.create permission.Learn how to grant roles.

Note: If you don't plan to keep the resources that you create in this procedure, create a project instead of selecting an existing project. After you finish these steps, you can delete the project, removing all resources associated with the project.

• Create a Google Cloud project:

  gcloud projects createPROJECT_ID

  ReplacePROJECT_ID with a name for the Google Cloud project you are creating.

• Select the Google Cloud project that you created:

  gcloud config set projectPROJECT_ID

  ReplacePROJECT_ID with your Google Cloud project name.

Verify that billing is enabled for your Google Cloud project.

Enable the Cloud Logging, Cloud Monitoring, Cloud Trace, and Telemetry APIs:

Roles required to enable APIs

To enable APIs, you need the Service Usage Admin IAM role (roles/serviceusage.serviceUsageAdmin), which contains theserviceusage.services.enable permission.Learn how to grant roles.

gcloudservicesenablelogging.googleapis.com monitoring.googleapis.com cloudtrace.googleapis.com telemetry.googleapis.com

1. If you run the sample in the Cloud Shell, on Google Cloudresources, or on a local development environment, then the permissions listedin this section are sufficient. For production applications, typically aservice account provides the credentials to write log, metric, and tracedata.

   To get the permissions that you need to have the sample application to write log, metric, and trace data, ask your administrator to grant you the following IAM roles on your project:

   • Logs Writer (roles/logging.logWriter)
   • Monitoring Metric Writer (roles/monitoring.metricWriter)
   • Cloud Telemetry Traces Writer (roles/telemetry.tracesWriter)

   To get the permissions that you need to view your log, metric, and trace data, ask your administrator to grant you the following IAM roles on your project:

   • Logs Viewer (roles/logging.viewer)
   • Monitoring Viewer (roles/monitoring.viewer)
   • Cloud Trace User (roles/cloudtrace.user)

   For more information about granting roles, seeManage access to projects, folders, and organizations.

   You might also be able to get the required permissions throughcustom roles or otherpredefined roles.

Instrument your app to collect traces, metrics, and logs

To instrument your app to collect trace and metric data, and to writestructured JSON to standard out, perform the following steps as describedin subsequent sections of this document:

1. Configure your app to use the OpenTelemetry Java Agent
2. Configure OpenTelemetry
3. Configure structured logging
4. Write structured logs

Configure your app to use the OpenTelemetry Java Agent

To configure the app to write structured logs and to collect metrics and tracedata by using OpenTelemetry, update the invocation of your app to use theOpenTelemetry Java Agent. This method of instrumenting your app isknown asautomatic instrumentation because it doesn't require modifying yourapp code.

The following code sample illustrates a Dockerfile that downloads theOpenTelemetry Java Agent JAR file and updates the command line invocation to pass the-javaagent flag.

To view the full sample, clickmore_vertMore, and then selectView on GitHub.

RUNwget-O/opentelemetry-javaagent.jarhttps://github.com/open-telemetry/opentelemetry-java-instrumentation/releases/download/v1.31.0/opentelemetry-javaagent.jarCMDsh-c"java -javaagent:/opentelemetry-javaagent.jar -cp app:app/lib/* com.example.demo.DemoApplication \2>&1 | tee /var/log/app.log"

Alternatively, you can also set the-javaagent flag in theJAVA_TOOL_OPTIONSenvironment variable:

exportJAVA_TOOL_OPTIONS="-javaagent:PATH/TO/opentelemetry-javaagent.jar"

Configure OpenTelemetry

The default configuration for the OpenTelemetry Java Agent exports traces and metrics byusing theOTLP protocol. It also configures OpenTelemetry to use theW3C Trace Context format forpropagating tracecontext. This configuration ensures that spans have thecorrect parent-child relationship within a trace.

For more information and configuration options, seeOpenTelemetry Java automaticinstrumentation.

Configure structured logging

To include the trace information as part of the JSON-formatted logs written tostandard output, configure your app to output structured logs in JSONformat. We recommend usingLog4j2 as your logging implementation.The following code sample illustrates alog4j2.xml file configured to outputJSON structured logs using theJSON Template Layout:

<!--FormatJSONlogsfortheCloudLoggingagenthttps://cloud.google.com/logging/docs/structured-logging#special-payload-fields--><!--Log4j2'sJsonTemplateLayoutincludesatemplateforCloudLogging'sspecialJSONfieldshttps://logging.apache.org/log4j/2.x/manual/json-template-layout.html#event-templates--><JsonTemplateLayouteventTemplateUri="classpath:GcpLayout.json"><!--ExtendtheincludedGcpLayouttoincludethetraceandspanIDsfromMappedDiagnosticContext(MDC)sothatCloudLoggingcancorrelateLogsandSpansSincelog4j22.24.0,GcpLayout.jsonalreadyincludestracecontextloggingfromMDCandthebelowadditionalfieldsarenolongerneeded--><EventTemplateAdditionalFieldkey="logging.googleapis.com/trace"format="JSON"value='{"$resolver":"mdc","key":"trace_id"}'/><EventTemplateAdditionalFieldkey="logging.googleapis.com/spanId"format="JSON"value='{"$resolver":"mdc","key":"span_id"}'/><EventTemplateAdditionalFieldkey="logging.googleapis.com/trace_sampled"format="JSON"value="true"/></JsonTemplateLayout>

The previous configuration extracts information about the active span fromSLF4J'sMapped Diagnostic Context and adds that information as attributesto the log. These attributes can then be used to correlate a log with a trace:

• logging.googleapis.com/trace: Resource name of the trace associated withthe log entry.
• logging.googleapis.com/spanId: The span ID with the trace that isassociated with the log entry.
• logging.googleapis.com/trace_sampled: The value of this field must betrue orfalse.

For more information about these fields, see theLogEntrystructure.

Write structured logs

To write structured logs that link to a trace, use theSLF4J loggingAPI. For example, the following statement shows how to call theLogger.info()method:

logger.info("handle /multi request with subRequests={}",subRequests);

The OpenTelemetry Java Agent automatically populates SLF4J's Mapped Diagnostic Context withthespan context of the current active span in theOpenTelemetry Context. The Mapped Diagnostic Context is then includedin the JSON logs as described inConfigure structuredlogging.

Run a sample app configured to collect telemetry

The instrumentation in the sample app uses vendor-neutral formats, like JSONfor log data and OTLP for metric and trace data. The app also uses theSpring Boot Framework. The OpenTelemetryCollector sendslog and metric data to your project by using Google exporters. It sends yourtrace data to your project by using the Telemetry API, which uses OTLP.

The app has two endpoints:

• The/multi endpoint is handled by thehandleMulti function. The loadgenerator in the app issues requests to the/multi endpoint. When thisendpoint receives a request, it sends between three and seven requests tothe/single endpoint on the local server.

  /** * handleMulti handles an http request by making 3-7 http requests to the /single endpoint. * * <p>OpenTelemetry instrumentation requires no changes here. It will automatically generate a * span for the controller body. */@GetMapping("/multi")publicMono<String>handleMulti()throwsException{intsubRequests=ThreadLocalRandom.current().nextInt(3,8);// Write a structured log with the request context, which allows the log to// be linked with the trace for this request.logger.info("handle /multi request with subRequests={}",subRequests);// Make 3-7 http requests to the /single endpoint.returnFlux.range(0,subRequests).concatMap(i->client.get().uri("http://localhost:8080/single").retrieve().bodyToMono(Void.class)).then(Mono.just("ok"));}

• The/single endpoint is handled by thehandleSingle function. When thisendpoint receives a request, it sleeps for a short delay and then respondswith a string.

  /** * handleSingle handles an http request by sleeping for 100-200 ms. It writes the number of * milliseconds slept as its response. * * <p>OpenTelemetry instrumentation requires no changes here. It will automatically generate a * span for the controller body. */@GetMapping("/single")publicStringhandleSingle()throwsInterruptedException{intsleepMillis=ThreadLocalRandom.current().nextInt(100,200);logger.info("Going to sleep for {}",sleepMillis);Thread.sleep(sleepMillis);logger.info("Finishing the request");returnString.format("slept %s\n",sleepMillis);}

Download and deploy the app

1. In the Google Cloud console, activate Cloud Shell.

   Activate Cloud Shell

   At the bottom of the Google Cloud console, aCloud Shell session starts and displays a command-line prompt. Cloud Shell is a shell environment with the Google Cloud CLI already installed and with values already set for your current project. It can take a few seconds for the session to initialize.

2. Clone the repository:

   gitclonehttps://github.com/GoogleCloudPlatform/opentelemetry-operations-java

3. Go to the sample directory:

   cdopentelemetry-operations-java/examples/instrumentation-quickstart

4. Build and run the sample:

   dockercomposeup--abort-on-container-exit

   If you aren't running on Cloud Shell, then run the application with theGOOGLE_APPLICATION_CREDENTIALS environment variable pointing to acredentials file.Application DefaultCredentialsprovides a credentials file at$HOME/.config/gcloud/application_default_credentials.json.

   # Set environment variablesexportGOOGLE_CLOUD_PROJECT="PROJECT_ID"exportGOOGLE_APPLICATION_CREDENTIALS="$HOME/.config/gcloud/application_default_credentials.json"exportUSERID="$(id-u)"# Rundockercompose-fdocker-compose.yaml-fdocker-compose.creds.yamlup--abort-on-container-exit

View your metrics

The OpenTelemetry instrumentation in the sample app generates Prometheusmetrics that you can view by using theMetrics Explorer:

• Prometheus/http_server_duration_milliseconds/histogramrecords the duration of server requests and stores the resultsin a histogram.

• Prometheus/http_client_duration_milliseconds/histogramrecords the duration of client requests and stores the resultsin a histogram.

1. In the Google Cloud console, go to theleaderboard Metrics explorer page:

   Go toMetrics explorer

   If you use the search bar to find this page, then select the result whose subheading isMonitoring.

2. In the toolbar of the Google Cloud console, select your Google Cloud project. ForApp Hub configurations, select the App Hub host project or the app-enabled folder's management project.
3. In theMetric element, expand theSelect a metric menu, enterhttp_server in the filter bar, and then use the submenus to select a specific resource type and metric:
   1. In theActive resources menu, selectPrometheus Target.
   2. In theActive metric categories menu, selectHttp.
   3. In theActive metrics menu, select a metric.
   4. ClickApply.

4. To add filters, which remove time series from the query results, use theFilter element.

5. Configure how the data is viewed.
   

   When the measurements for a metric arecumulative, Metrics Explorer automatically normalizes the measured data bythe alignment period, which results in the chart displaying a rate. Formore information, seeKinds, types, and conversions.

   When integer or double values are measured, such as with the twocounter metrics, Metrics Explorer automatically sums all time series.To view the data for the/multi and/single HTTP routes,set the first menu of theAggregation entry toNone.

   For more information about configuring a chart, seeSelect metrics when using Metrics Explorer.

View your traces

It might take several minutes before your trace data is available. For example,when trace data is received by your project, Google Cloud Observability might need to createa database to store that data. The creation of the database can take a fewminutes and during that period, no trace data is available to view.

To view your trace data, do the following:

1. In the Google Cloud console, go to theTrace explorer page:

   Go toTrace explorer

   You can also find this page by using the search bar.

2. In the table section of the page, select a row with the span name/multi.

3. In the Gantt chart on theTrace details panel,select the span labeled/multi.

   A panel opens that displays information about the HTTP request. Thesedetails include the method, status code, number of bytes, and theuser agent of the caller.

4. To view the logs associated with this trace,select theLogs & Events tab.

   The tab shows individual logs. To view the details of the log entry,expand the log entry. You can also clickView Logs and view the logby using the Logs Explorer.

For more information about using the Cloud Trace explorer, seeFind and explore traces.

View your logs

From the Logs Explorer, you can inspect your logs, and you can alsoview associated traces, when they exist.

1. In the Google Cloud console, go to thesegmentLogs Explorer page:

   Go toLogs Explorer

   If you use the search bar to find this page, then select the result whose subheading isLogging.

2. Locate a log with the description ofhandle /multi request.

   To view the details of the log, expand the log entry.

3. ClickTraces on a log entry with the"handle /multi request" message, and then selectView trace details.

   ATrace details panel opens and displays the selected trace.

   Your log data might be available several minutes before your tracedata is available. If you encounter an error when viewing trace dataeither by searching for a trace by ID or by following the steps inthis task, then wait a minute or two and retry the action.

For more information about using the Logs Explorer, seeView logs by using the Logs Explorer.

What's next

• OpenTelemetry
• OTLP specification
• Structured logging
• Troubleshooting Managed Service for Prometheus
• Troubleshoot Cloud Trace