When you use an OpenTelemetry collector, you instrument your application withthe SDK and the SDK's OTLP in-process exporter. This instrumentation isvendor neutral. You also deploy an OpenTelemetry collector that receives telemetryfrom the in-process exporter and then exports thattelemetry to your Google Cloud project. To learn more about collectors, seeGoogle-Built OpenTelemetry Collector.

We recommend that you use an OpenTelemetry collector to export your telemetrydata when your environment supports use of collector. For some environments,you must use an in-process exporter that directly sends data to yourGoogle Cloud project. To learn about in-process instrumentation, seeMigrate from the Trace exporter to the OTLP endpoint.

To learn more about instrumentation, see the following documents:

• Instrumentation and observability.
• Choose an instrumentation approach.

This document doesn't describe zero-code instrumentation. For information aboutthat topic, seeJavaScript zero-code instrumentation.

For general information, seeOpenTelemetry Instrumentation for Node.

Before you begin

Sign in to your Google Cloud account. If you're new to Google Cloud, create an account to evaluate how our products perform in real-world scenarios. New customers also get $300 in free credits to run, test, and deploy workloads.

Install the Google Cloud CLI.

If you're using an external identity provider (IdP), you must first sign in to the gcloud CLI with your federated identity.

Toinitialize the gcloud CLI, run the following command:

gcloudinit

Create or select a Google Cloud project.

Roles required to select or create a project

• Select a project: Selecting a project doesn't require a specific IAM role—you can select any project that you've been granted a role on.
• Create a project: To create a project, you need the Project Creator role (roles/resourcemanager.projectCreator), which contains theresourcemanager.projects.create permission.Learn how to grant roles.

Note: If you don't plan to keep the resources that you create in this procedure, create a project instead of selecting an existing project. After you finish these steps, you can delete the project, removing all resources associated with the project.

• Create a Google Cloud project:

  gcloud projects createPROJECT_ID

  ReplacePROJECT_ID with a name for the Google Cloud project you are creating.

• Select the Google Cloud project that you created:

  gcloud config set projectPROJECT_ID

  ReplacePROJECT_ID with your Google Cloud project name.

Verify that billing is enabled for your Google Cloud project.

Enable the Cloud Logging, Cloud Monitoring, Cloud Trace, and Telemetry APIs:

Roles required to enable APIs

To enable APIs, you need the Service Usage Admin IAM role (roles/serviceusage.serviceUsageAdmin), which contains theserviceusage.services.enable permission.Learn how to grant roles.

gcloudservicesenablelogging.googleapis.com monitoring.googleapis.com cloudtrace.googleapis.com telemetry.googleapis.com

Install the Google Cloud CLI.

If you're using an external identity provider (IdP), you must first sign in to the gcloud CLI with your federated identity.

Toinitialize the gcloud CLI, run the following command:

gcloudinit

Create or select a Google Cloud project.

Roles required to select or create a project

• Select a project: Selecting a project doesn't require a specific IAM role—you can select any project that you've been granted a role on.
• Create a project: To create a project, you need the Project Creator role (roles/resourcemanager.projectCreator), which contains theresourcemanager.projects.create permission.Learn how to grant roles.

Note: If you don't plan to keep the resources that you create in this procedure, create a project instead of selecting an existing project. After you finish these steps, you can delete the project, removing all resources associated with the project.

• Create a Google Cloud project:

  gcloud projects createPROJECT_ID

  ReplacePROJECT_ID with a name for the Google Cloud project you are creating.

• Select the Google Cloud project that you created:

  gcloud config set projectPROJECT_ID

  ReplacePROJECT_ID with your Google Cloud project name.

Verify that billing is enabled for your Google Cloud project.

Enable the Cloud Logging, Cloud Monitoring, Cloud Trace, and Telemetry APIs:

Roles required to enable APIs

To enable APIs, you need the Service Usage Admin IAM role (roles/serviceusage.serviceUsageAdmin), which contains theserviceusage.services.enable permission.Learn how to grant roles.

gcloudservicesenablelogging.googleapis.com monitoring.googleapis.com cloudtrace.googleapis.com telemetry.googleapis.com

1. If you run the sample in the Cloud Shell, on Google Cloudresources, or on a local development environment, then the permissions listedin this section are sufficient. For production applications, typically aservice account provides the credentials to write log, metric, and tracedata.

   To get the permissions that you need to have the sample application to write log, metric, and trace data, ask your administrator to grant you the following IAM roles on your project:

   • Logs Writer (roles/logging.logWriter)
   • Monitoring Metric Writer (roles/monitoring.metricWriter)
   • Cloud Telemetry Traces Writer (roles/telemetry.tracesWriter)

   To get the permissions that you need to view your log, metric, and trace data, ask your administrator to grant you the following IAM roles on your project:

   • Logs Viewer (roles/logging.viewer)
   • Monitoring Viewer (roles/monitoring.viewer)
   • Cloud Trace User (roles/cloudtrace.user)

   For more information about granting roles, seeManage access to projects, folders, and organizations.

   You might also be able to get the required permissions throughcustom roles or otherpredefined roles.

Instrument your app to collect traces, metrics, and logs

To instrument your app to collect trace and metric data, and to writestructured JSON to standard out, perform the following steps as describedin subsequent sections of this document:

1. Configure OpenTelemetry
2. Configure your app to preload the OpenTelemetry configuration
3. Configure structured logging
4. Write structured logs

Configure OpenTelemetry

The default configuration for the OpenTelemetry Node.js SDK exports traces by usingtheOTLP protocol. It also configures OpenTelemetry to use theW3CTrace Context format forpropagating tracecontext. This configuration ensures that spans have thecorrect parent-child relationship within a trace.

The following code sample illustrates a JavaScript module to setup OpenTelemetry.

To view the full sample, clickmore_vertMore, and then selectView on GitHub.

diag.setLogger(newDiagConsoleLogger(),opentelemetry.core.diagLogLevelFromString(opentelemetry.core.getStringFromEnv('OTEL_LOG_LEVEL')));constsdk=newopentelemetry.NodeSDK({instrumentations:getNodeAutoInstrumentations({// Disable noisy instrumentations'@opentelemetry/instrumentation-fs':{enabled:false},}),resourceDetectors:getResourceDetectorsFromEnv(),metricReader:getMetricReader(),});try{sdk.start();diag.info('OpenTelemetry automatic instrumentation started successfully');}catch(error){diag.error('Error initializing OpenTelemetry SDK. Your application is not instrumented and will not produce telemetry',error);}// Gracefully shut down the SDK to flush telemetry when the program exitsprocess.on('SIGTERM',()=>{sdk.shutdown().then(()=>diag.debug('OpenTelemetry SDK terminated')).catch(error=>diag.error('Error terminating OpenTelemetry SDK',error));});

The previous code sample configures OpenTelemetry to export metrics using theOTLPprotocol, and it uses the@opentelemetry/auto-instrumentations-nodepackage to configure all available Node.js instrumentations.

To ensure that all pending telemetry is flushed and that connections are closedgracefully before the application shuts down, theSIGTERM handler callsshutdown.

For more information and configuration options, seeZero-Code Instrumentation Configuration.

Configure your app to preload the OpenTelemetry configuration

To configure the app to write structured logs and to collect metrics and tracedata by using OpenTelemetry, update the invocation of your app to preload theinstrumentation module with the Node.js--require flag. Usingthe--require flag ensures that OpenTelemetry is initialized before your appstarts. For more information, seeOpenTelemetry Node.js GettingStarted.

The following code sample illustrates a Dockerfile passing the--require flag:

CMDnode--require./build/src/instrumentation.jsbuild/src/index.js2>&1|tee/var/log/app.log

Configure structured logging

To include the trace information as part of the JSON-formatted logs written tostandard output, configure your app to output structured logs in JSON format.

The following code sample illustrates a PinoLoggerOptionsobject that configures the app to output JSON structured logs:

// Expected attributes that OpenTelemetry adds to correlate logs with spansinterfaceLogRecord{trace_id?:string;span_id?:string;trace_flags?:string;[key:string]:unknown;}// https://cloud.google.com/logging/docs/reference/v2/rest/v2/LogEntry#logseverityconstPinoLevelToSeverityLookup:Record<string,string|undefined>={trace:'DEBUG',debug:'DEBUG',info:'INFO',warn:'WARNING',error:'ERROR',fatal:'CRITICAL',};exportconstloggerConfig={messageKey:'message',// Same as pino.stdTimeFunctions.isoTime but uses "timestamp" key instead of "time"timestamp():string{return`,"timestamp":"${newDate(Date.now()).toISOString()}"`;},formatters:{log(object:LogRecord):Record<string,unknown>{// Add trace context attributes following Cloud Logging structured log format described// in https://cloud.google.com/logging/docs/structured-logging#special-payload-fieldsconst{trace_id,span_id,trace_flags,...rest}=object;return{'logging.googleapis.com/trace':trace_id,'logging.googleapis.com/spanId':span_id,'logging.googleapis.com/trace_sampled':trace_flags?trace_flags==='01':undefined,...rest,};},// See// https://getpino.io/#/docs/help?id=mapping-pino-log-levels-to-google-cloud-logging-stackdriver-severity-levelslevel(label:string){return{severity:PinoLevelToSeverityLookup[label]??PinoLevelToSeverityLookup['info'],};},},}satisfiesLoggerOptions;

The previous configuration extracts information about the active span from thelog message, and then adds that information as attributes to the JSON structuredlog. These attributes can then be used to correlate a log with a trace:

• logging.googleapis.com/trace: Resource name of the trace associated withthe log entry.
• logging.googleapis.com/spanId: The span ID with the trace that isassociated with the log entry.
• logging.googleapis.com/trace_sampled: The value of this field must betrue orfalse.

For more information about these fields, see theLogEntrystructure.

To use the Pino configuration with Fastify, pass the logger config object whencreating the Fastify app:

// Create the Fastify app providing the Pino logger configconstfastify=Fastify({logger:loggerConfig,});

Write structured logs

To write structured logs that link to a trace, use theFastify provided Pinologger. For example, the following statement shows how to calltheLogger.info() method:

request.log.info({subRequests},'handle /multi request');

OpenTelemetry automatically populates the Pino log entries with thespancontext of the current active span in theOpenTelemetryContext. This span context is then included in the JSON logs asdescribed inConfigure structured logging.

Run a sample app configured to collect telemetry

The instrumentation in the sample app uses vendor-neutral formats, like JSONfor log data and OTLP for metric and trace data. The app also uses theand theFastify framework. The OpenTelemetryCollector sendslog and metric data to your project by using Google exporters. It sends yourtrace data to your project by using the Telemetry API, which uses OTLP.

The app has two endpoints:

• The/multi endpoint is handled by thehandleMulti function. The loadgenerator in the app issues requests to the/multi endpoint. When thisendpoint receives a request, it sends between three and seven requests tothe/single endpoint on the local server.

  /** * handleMulti handles an http request by making 3-7 http requests to the /single endpoint. * * OpenTelemetry instrumentation requires no changes here. It will automatically generate a * span for the handler body. */fastify.get('/multi',asyncrequest=>{constsubRequests=randInt(3,8);request.log.info({subRequests},'handle /multi request');for(leti=0;i <subRequests;i++){awaitaxios.get(`http://localhost:${port}/single`);}return'ok';});

• The/single endpoint is handled by thehandleSingle function. When thisendpoint receives a request, it sleeps for a short delay and then respondswith a string.

  /** * handleSingle handles an http request by sleeping for 100-200 ms. It writes the number of * milliseconds slept as its response. */fastify.get('/single',asyncrequest=>{// Sleep between 100-200 millisecondsconstsleepMillis=randInt(100,200);request.log.info({sleepMillis},'Going to sleep');awaitsleep(sleepMillis);return`slept${sleepMillis}\n`;});

Download and deploy the app

1. In the Google Cloud console, activate Cloud Shell.

   Activate Cloud Shell

   At the bottom of the Google Cloud console, aCloud Shell session starts and displays a command-line prompt. Cloud Shell is a shell environment with the Google Cloud CLI already installed and with values already set for your current project. It can take a few seconds for the session to initialize.

2. Clone the repository:

   gitclonehttps://github.com/GoogleCloudPlatform/opentelemetry-operations-js

3. Go to the sample directory:

   cdopentelemetry-operations-js/samples/instrumentation-quickstart

4. Build and run the sample:

   dockercomposeup--abort-on-container-exit

   If you aren't running on Cloud Shell, then run the application with theGOOGLE_APPLICATION_CREDENTIALS environment variable pointing to acredentials file.Application DefaultCredentialsprovides a credentials file at$HOME/.config/gcloud/application_default_credentials.json.

   # Set environment variablesexportGOOGLE_CLOUD_PROJECT="PROJECT_ID"exportGOOGLE_APPLICATION_CREDENTIALS="$HOME/.config/gcloud/application_default_credentials.json"exportUSERID="$(id-u)"# Rundockercompose-fdocker-compose.yaml-fdocker-compose.creds.yamlup--abort-on-container-exit

View your metrics

The OpenTelemetry instrumentation in the sample app generates Prometheusmetrics that you can view by using theMetrics Explorer:

• Prometheus/http_server_duration_milliseconds/histogramrecords the duration of server requests and stores the resultsin a histogram.

• Prometheus/http_client_duration_milliseconds/histogramrecords the duration of client requests and stores the resultsin a histogram.

1. In the Google Cloud console, go to theleaderboard Metrics explorer page:

   Go toMetrics explorer

   If you use the search bar to find this page, then select the result whose subheading isMonitoring.

2. In the toolbar of the Google Cloud console, select your Google Cloud project. ForApp Hub configurations, select the App Hub host project or the app-enabled folder's management project.
3. In theMetric element, expand theSelect a metric menu, enterhttp_server in the filter bar, and then use the submenus to select a specific resource type and metric:
   1. In theActive resources menu, selectPrometheus Target.
   2. In theActive metric categories menu, selectHttp.
   3. In theActive metrics menu, select a metric.
   4. ClickApply.

4. To add filters, which remove time series from the query results, use theFilter element.

5. Configure how the data is viewed.
   

   When the measurements for a metric arecumulative, Metrics Explorer automatically normalizes the measured data bythe alignment period, which results in the chart displaying a rate. Formore information, seeKinds, types, and conversions.

   When integer or double values are measured, such as with the twocounter metrics, Metrics Explorer automatically sums all time series.To view the data for the/multi and/single HTTP routes,set the first menu of theAggregation entry toNone.

   For more information about configuring a chart, seeSelect metrics when using Metrics Explorer.

View your traces

It might take several minutes before your trace data is available. For example,when trace data is received by your project, Google Cloud Observability might need to createa database to store that data. The creation of the database can take a fewminutes and during that period, no trace data is available to view.

To view your trace data, do the following:

1. In the Google Cloud console, go to theTrace explorer page:

   Go toTrace explorer

   You can also find this page by using the search bar.

2. In the table section of the page, select a row with the span name/multi.

3. In the Gantt chart on theTrace details panel,select the span labeled/multi.

   A panel opens that displays information about the HTTP request. Thesedetails include the method, status code, number of bytes, and theuser agent of the caller.

4. To view the logs associated with this trace,select theLogs & Events tab.

   The tab shows individual logs. To view the details of the log entry,expand the log entry. You can also clickView Logs and view the logby using the Logs Explorer.

For more information about using the Cloud Trace explorer, seeFind and explore traces.

View your logs

From the Logs Explorer, you can inspect your logs, and you can alsoview associated traces, when they exist.

1. In the Google Cloud console, go to thesegmentLogs Explorer page:

   Go toLogs Explorer

   If you use the search bar to find this page, then select the result whose subheading isLogging.

2. Locate a log with the description ofhandle /multi request.

   To view the details of the log, expand the log entry.

3. ClickTraces on a log entry with the"handle /multi request" message, and then selectView trace details.

   ATrace details panel opens and displays the selected trace.

   Your log data might be available several minutes before your tracedata is available. If you encounter an error when viewing trace dataeither by searching for a trace by ID or by following the steps inthis task, then wait a minute or two and retry the action.

For more information about using the Logs Explorer, seeView logs by using the Logs Explorer.

What's next

• OpenTelemetry
• OTLP specification
• Structured logging
• Troubleshooting Managed Service for Prometheus
• Troubleshoot Cloud Trace