When you use an OpenTelemetry collector, you instrument your application withthe SDK and the SDK's OTLP in-process exporter. This instrumentation isvendor neutral. You also deploy an OpenTelemetry collector that receives telemetryfrom the in-process exporter and then exports thattelemetry to your Google Cloud project. To learn more about collectors, seeGoogle-Built OpenTelemetry Collector.

We recommend that you use an OpenTelemetry collector to export your telemetrydata when your environment supports use of collector. For some environments,you must use an in-process exporter that directly sends data to yourGoogle Cloud project. To learn about in-process instrumentation, seeMigrate from the Trace exporter to the OTLP endpoint.

To learn more about instrumentation, see the following documents:

• Instrumentation and observability.
• Choose an instrumentation approach.

OpenTelemetry'sContext is a mechanism for carrying execution-scopedvalues across APIs within a process. An important use of context is to carry thecurrent active span so it can be modified, or referenced as the parent of anynew spans when they are created. To summarize:

• Context refers to the mechanism to propagateexecution-scoped values, including the current active span, across APIswithin a process.

• Span Context is an immutable object on every span thatincludes the trace ID, the span ID, and flags and state for the trace.

• Propagation is the mechanism that moves contextbetween services and processes.

The Go standard library'scontext.Context also carries scopedvalues across API boundaries. Typically, handler functions in a server receivean incomingContext and pass it through the call chain to anyclients making outgoing requests.

Go's standard librarycontext.Context is used as the implementation ofOpenTelemetry Context in Go.

Before you begin

Sign in to your Google Cloud account. If you're new to Google Cloud, create an account to evaluate how our products perform in real-world scenarios. New customers also get $300 in free credits to run, test, and deploy workloads.

Install the Google Cloud CLI.

If you're using an external identity provider (IdP), you must first sign in to the gcloud CLI with your federated identity.

Toinitialize the gcloud CLI, run the following command:

gcloudinit

Create or select a Google Cloud project.

Roles required to select or create a project

• Select a project: Selecting a project doesn't require a specific IAM role—you can select any project that you've been granted a role on.
• Create a project: To create a project, you need the Project Creator role (roles/resourcemanager.projectCreator), which contains theresourcemanager.projects.create permission.Learn how to grant roles.

Note: If you don't plan to keep the resources that you create in this procedure, create a project instead of selecting an existing project. After you finish these steps, you can delete the project, removing all resources associated with the project.

• Create a Google Cloud project:

  gcloud projects createPROJECT_ID

  ReplacePROJECT_ID with a name for the Google Cloud project you are creating.

• Select the Google Cloud project that you created:

  gcloud config set projectPROJECT_ID

  ReplacePROJECT_ID with your Google Cloud project name.

Verify that billing is enabled for your Google Cloud project.

Enable the Cloud Logging, Cloud Monitoring, Cloud Trace, and Telemetry APIs:

Roles required to enable APIs

To enable APIs, you need the Service Usage Admin IAM role (roles/serviceusage.serviceUsageAdmin), which contains theserviceusage.services.enable permission.Learn how to grant roles.

gcloudservicesenablelogging.googleapis.com monitoring.googleapis.com cloudtrace.googleapis.com telemetry.googleapis.com

Install the Google Cloud CLI.

If you're using an external identity provider (IdP), you must first sign in to the gcloud CLI with your federated identity.

Toinitialize the gcloud CLI, run the following command:

gcloudinit

Create or select a Google Cloud project.

Roles required to select or create a project

• Select a project: Selecting a project doesn't require a specific IAM role—you can select any project that you've been granted a role on.
• Create a project: To create a project, you need the Project Creator role (roles/resourcemanager.projectCreator), which contains theresourcemanager.projects.create permission.Learn how to grant roles.

Note: If you don't plan to keep the resources that you create in this procedure, create a project instead of selecting an existing project. After you finish these steps, you can delete the project, removing all resources associated with the project.

• Create a Google Cloud project:

  gcloud projects createPROJECT_ID

  ReplacePROJECT_ID with a name for the Google Cloud project you are creating.

• Select the Google Cloud project that you created:

  gcloud config set projectPROJECT_ID

  ReplacePROJECT_ID with your Google Cloud project name.

Verify that billing is enabled for your Google Cloud project.

Enable the Cloud Logging, Cloud Monitoring, Cloud Trace, and Telemetry APIs:

Roles required to enable APIs

To enable APIs, you need the Service Usage Admin IAM role (roles/serviceusage.serviceUsageAdmin), which contains theserviceusage.services.enable permission.Learn how to grant roles.

gcloudservicesenablelogging.googleapis.com monitoring.googleapis.com cloudtrace.googleapis.com telemetry.googleapis.com

1. If you run the sample in the Cloud Shell, on Google Cloudresources, or on a local development environment, then the permissions listedin this section are sufficient. For production applications, typically aservice account provides the credentials to write log, metric, and tracedata.

   To get the permissions that you need to have the sample application to write log, metric, and trace data, ask your administrator to grant you the following IAM roles on your project:

   • Logs Writer (roles/logging.logWriter)
   • Monitoring Metric Writer (roles/monitoring.metricWriter)
   • Cloud Telemetry Traces Writer (roles/telemetry.tracesWriter)

   To get the permissions that you need to view your log, metric, and trace data, ask your administrator to grant you the following IAM roles on your project:

   • Logs Viewer (roles/logging.viewer)
   • Monitoring Viewer (roles/monitoring.viewer)
   • Cloud Trace User (roles/cloudtrace.user)

   For more information about granting roles, seeManage access to projects, folders, and organizations.

   You might also be able to get the required permissions throughcustom roles or otherpredefined roles.

Instrument your app to collect traces, metrics, and logs

To instrument your app to collect trace and metric data, and to writestructured JSON to standard out, perform the following steps as describedin subsequent sections of this document:

1. Configure the main function
2. Configure OpenTelemetry
3. Configure structured logging
4. Add instrumentation to the HTTP server
5. Link trace spans with logs and metrics
6. Add instrumentation to the HTTP client
7. Write structured logs

Configure the main function

To configure the app to write structured logs and to collect metrics andtrace data by using OpenTelemetry, update themain function toconfigure the Go structured logging package,slog, and to configure OpenTelemetry.

The following code sample illustrates amain function that calls twohelper functions,setupLogging() andsetupOpenTelemetry(). These helperfunctions configure the logging package and OpenTelemetry.

To view the full sample, clickmore_vertMore, and then selectView on GitHub.

funcmain(){ctx:=context.Background()// Setup loggingsetupLogging()// Setup metrics, tracing, and context propagationshutdown,err:=setupOpenTelemetry(ctx)iferr!=nil{slog.ErrorContext(ctx,"error setting up OpenTelemetry",slog.Any("error",err))os.Exit(1)}// Run the http server, and shutdown and flush telemetry after it exits.slog.InfoContext(ctx,"server starting...")iferr=errors.Join(runServer(),shutdown(ctx));err!=nil{slog.ErrorContext(ctx,"server exited with error",slog.Any("error",err))os.Exit(1)}}

After you configure the logging package, to link your logs to your tracedata, you must pass the GoContext to the logger. For more information,see theWrite structured logs section of thisdocument.

Configure OpenTelemetry

To collect and export traces and metrics by using theOTLP protocol, configure the globalTracerProviderandMeterProvider instances.The following code sample illustrates thesetupOpenTelemetry function,which is called from themain function:

funcsetupOpenTelemetry(ctxcontext.Context)(shutdownfunc(context.Context)error,errerror){varshutdownFuncs[]func(context.Context)error// shutdown combines shutdown functions from multiple OpenTelemetry// components into a single function.shutdown=func(ctxcontext.Context)error{varerrerrorfor_,fn:=rangeshutdownFuncs{err=errors.Join(err,fn(ctx))}shutdownFuncs=nilreturnerr}// Configure Context Propagation to use the default W3C traceparent formatotel.SetTextMapPropagator(autoprop.NewTextMapPropagator())// Configure Trace Export to send spans as OTLPtexporter,err:=autoexport.NewSpanExporter(ctx)iferr!=nil{err=errors.Join(err,shutdown(ctx))return}tp:=trace.NewTracerProvider(trace.WithBatcher(texporter))shutdownFuncs=append(shutdownFuncs,tp.Shutdown)otel.SetTracerProvider(tp)// Configure Metric Export to send metrics as OTLPmreader,err:=autoexport.NewMetricReader(ctx)iferr!=nil{err=errors.Join(err,shutdown(ctx))return}mp:=metric.NewMeterProvider(metric.WithReader(mreader),)shutdownFuncs=append(shutdownFuncs,mp.Shutdown)otel.SetMeterProvider(mp)returnshutdown,nil}

The previous code sample configures the globalTextMapPropagator to use theW3C Trace Context format forpropagating tracecontext. This configuration ensures that spans have thecorrect parent-child relationship within a trace.

To ensure that all pending telemetry is flushed and that connections are closedgracefully, thesetupOpenTelemetry function returns a function namedshutdown, which performs those actions.

Configure structured logging

To include the trace information as part of the JSON-formatted logs writtento standard output, configure the Go structured logging package,slog.The following code sample illustrates thesetupLogging function,which is called from themain function:

funcsetupLogging(){// Use json as our base logging format.jsonHandler:=slog.NewJSONHandler(os.Stdout,&slog.HandlerOptions{ReplaceAttr:replacer})// Add span context attributes when Context is passed to logging calls.instrumentedHandler:=handlerWithSpanContext(jsonHandler)// Set this handler as the global slog handler.slog.SetDefault(slog.New(instrumentedHandler))}

The previous code calls thehandlerWithSpanContext function, which extractsinformation from theContext instance and adds that information as attributesto a log. These attributes can then be used to correlate a log with a trace:

• logging.googleapis.com/trace: Resource name of the trace associated withthe log entry.
• logging.googleapis.com/spanId: The span ID with the trace that isassociated with the log entry.
• logging.googleapis.com/trace_sampled: The value of this field must betrue orfalse.

For more information about these fields, see theLogEntrystructure.

funchandlerWithSpanContext(handlerslog.Handler)*spanContextLogHandler{return&spanContextLogHandler{Handler:handler}}// spanContextLogHandler is a slog.Handler which adds attributes from the// span context.typespanContextLogHandlerstruct{slog.Handler}// Handle overrides slog.Handler's Handle method. This adds attributes from the// span context to the slog.Record.func(t*spanContextLogHandler)Handle(ctxcontext.Context,recordslog.Record)error{// Get the SpanContext from the context.ifs:=trace.SpanContextFromContext(ctx);s.IsValid(){// Add trace context attributes following Cloud Logging structured log format described// in https://cloud.google.com/logging/docs/structured-logging#special-payload-fieldsrecord.AddAttrs(slog.Any("logging.googleapis.com/trace",s.TraceID()),)record.AddAttrs(slog.Any("logging.googleapis.com/spanId",s.SpanID()),)record.AddAttrs(slog.Bool("logging.googleapis.com/trace_sampled",s.TraceFlags().IsSampled()),)}returnt.Handler.Handle(ctx,record)}funcreplacer(groups[]string,aslog.Attr)slog.Attr{// Rename attribute keys to match Cloud Logging structured log formatswitcha.Key{caseslog.LevelKey:a.Key="severity"// Map slog.Level string values to Cloud Logging LogSeverity// https://cloud.google.com/logging/docs/reference/v2/rest/v2/LogEntry#LogSeverityiflevel:=a.Value.Any().(slog.Level);level==slog.LevelWarn{a.Value=slog.StringValue("WARNING")}caseslog.TimeKey:a.Key="timestamp"caseslog.MessageKey:a.Key="message"}returna}

Add instrumentation to the HTTP server

To add trace and metric instrumentation to the requests handled by theHTTP server, use OpenTelemetry. The following sample uses theotelhttp handlerto propagate context, and for trace and metric instrumentation:

funcrunServer()error{handleHTTP("/single",handleSingle)handleHTTP("/multi",handleMulti)returnhttp.ListenAndServe(":8080",nil)}// handleHTTP handles the http HandlerFunc on the specified route, and uses// otelhttp for context propagation, trace instrumentation, and metric// instrumentation.funchandleHTTP(routestring,handleFnhttp.HandlerFunc){instrumentedHandler:=otelhttp.NewHandler(otelhttp.WithRouteTag(route,handleFn),route)http.Handle(route,instrumentedHandler)}

In the previous code, theotelhttp handler uses the globalTracerProvider,MeterProvider, andTextMapPropagator instances. ThesetupOpenTelemetry function configures these instances.

Link trace spans with logs and metrics

To link server and client spans, and to associate metrics and logs, pass theGoContext instance to the HTTP request and when you write logs.The following example illustrates a route handler that extracts theGoContext instance and the passes that instance to the logger and to thecallSingle function, which makes an outgoing HTTP request:

funchandleMulti(whttp.ResponseWriter,r*http.Request){subRequests:=3+rand.Intn(4)// Write a structured log with the request context, which allows the log to// be linked with the trace for this request.slog.InfoContext(r.Context(),"handle /multi request",slog.Int("subRequests",subRequests))err:=computeSubrequests(r,subRequests)iferr!=nil{http.Error(w,err.Error(),http.StatusBadGateway)return}fmt.Fprintln(w,"ok")}

In the previous code, the function callr.Context() retrieves the GoContextfrom the HTTP request.

Add instrumentation to the HTTP client

To inject the trace context into outgoing HTTP requests and to addtrace and metric instrumentation, call theotelhttp.Get function.In the following example, thecallSingle function performs this action:

funccallSingle(ctxcontext.Context)error{// otelhttp.Get makes an http GET request, just like net/http.Get.// In addition, it records a span, records metrics, and propagates context.res,err:=otelhttp.Get(ctx,"http://localhost:8080/single")iferr!=nil{returnerr}returnres.Body.Close()}

In the previous code, theotelhttp handler uses the globalTracerProvider,MeterProvider, andTextMapPropagator instances. ThesetupOpenTelemetry function configures these instances.

Write structured logs

To write structured logs that link to a trace, use Go's structured loggingpackage,slog, and pass the GoContext instance to the logger.The GoContext instance is required when you want to link a log to a span.For example, the following statement shows how to call theInfoContextmethod forslog, and it illustrates how to add the fieldsubRequeststo the JSON instance:

slog.InfoContext(r.Context(),"handle /multi request",slog.Int("subRequests",subRequests))

Run a sample app configured to collect telemetry

The instrumentation in the sample app uses vendor-neutral formats, like JSONfor log data and OTLP for metric and trace data. The OpenTelemetryCollector sendslog and metric data to your project by using Google exporters. It sends yourtrace data to your project by using the Telemetry API, which uses OTLP.The load generator in the app issues requests to the app's routes.

Download and deploy the app

1. In the Google Cloud console, activate Cloud Shell.

   Activate Cloud Shell

   At the bottom of the Google Cloud console, aCloud Shell session starts and displays a command-line prompt. Cloud Shell is a shell environment with the Google Cloud CLI already installed and with values already set for your current project. It can take a few seconds for the session to initialize.

2. Clone the repository:

   gitclonehttps://github.com/GoogleCloudPlatform/golang-samples

3. Go to the OpenTelemetry directory:

   cdgolang-samples/opentelemetry/instrumentation

4. Build and run the sample:

   dockercomposeup--abort-on-container-exit

   If you aren't running on Cloud Shell, then run the application with theGOOGLE_APPLICATION_CREDENTIALS environment variable pointing to acredentials file.Application DefaultCredentialsprovides a credentials file at$HOME/.config/gcloud/application_default_credentials.json.

   # Set environment variablesexportGOOGLE_CLOUD_PROJECT="PROJECT_ID"exportGOOGLE_APPLICATION_CREDENTIALS="$HOME/.config/gcloud/application_default_credentials.json"exportUSERID="$(id-u)"# Rundockercompose-fdocker-compose.yaml-fdocker-compose.creds.yamlup--abort-on-container-exit

View your metrics

The OpenTelemetry instrumentation in the sample app generates Prometheusmetrics that you can view by using theMetrics Explorer:

• Prometheus/http_server_duration_milliseconds/histogramrecords the duration of server requests and stores the resultsin a histogram.

• Prometheus/http_client_duration_milliseconds/histogramrecords the duration of client requests and stores the resultsin a histogram.

• Prometheus/http_server_response_size_bytes_total/counterrecords the response size length for the/multi and/single HTTP routes. The measurements for this metric are cumulative,which means that each value represents the total since collection of valuesbegan.

1. In the Google Cloud console, go to theleaderboard Metrics explorer page:

   Go toMetrics explorer

   If you use the search bar to find this page, then select the result whose subheading isMonitoring.

2. In the toolbar of the Google Cloud console, select your Google Cloud project. ForApp Hub configurations, select the App Hub host project or the app-enabled folder's management project.
3. In theMetric element, expand theSelect a metric menu, enterhttp_server in the filter bar, and then use the submenus to select a specific resource type and metric:
   1. In theActive resources menu, selectPrometheus Target.
   2. In theActive metric categories menu, selectHttp.
   3. In theActive metrics menu, select a metric.
   4. ClickApply.

4. To add filters, which remove time series from the query results, use theFilter element.

5. Configure how the data is viewed.
   

   When the measurements for a metric arecumulative, Metrics Explorer automatically normalizes the measured data bythe alignment period, which results in the chart displaying a rate. Formore information, seeKinds, types, and conversions.

   When integer or double values are measured, such as with the twocounter metrics, Metrics Explorer automatically sums all time series.To view the data for the/multi and/single HTTP routes,set the first menu of theAggregation entry toNone.

   For more information about configuring a chart, seeSelect metrics when using Metrics Explorer.

View your traces

It might take several minutes before your trace data is available. For example,when trace data is received by your project, Google Cloud Observability might need to createa database to store that data. The creation of the database can take a fewminutes and during that period, no trace data is available to view.

To view your trace data, do the following:

1. In the Google Cloud console, go to theTrace explorer page:

   Go toTrace explorer

   You can also find this page by using the search bar.

2. In the table section of the page, select a row with the span name/multi.

3. In the Gantt chart on theTrace details panel,select the span labeled/multi.

   A panel opens that displays information about the HTTP request. Thesedetails include the method, status code, number of bytes, and theuser agent of the caller.

4. To view the logs associated with this trace,select theLogs & Events tab.

   The tab shows individual logs. To view the details of the log entry,expand the log entry. You can also clickView Logs and view the logby using the Logs Explorer.

For more information about using the Cloud Trace explorer, seeFind and explore traces.

View your logs

From the Logs Explorer, you can inspect your logs, and you can alsoview associated traces, when they exist.

1. In the Google Cloud console, go to theLogs Explorer page:

   Go toLogs Explorer

   If you use the search bar to find this page, then select the result whose subheading isLogging.

2. Locate a log with the description ofhandle /multi request.

   To view the details of the log, expand the log entry. In thejsonPayload field, there is an entry labeledsubRequests.This entry was added by a statement in thehandleMulti function.

3. ClickTraces on a log entry with the"handle /multi request" message, and then selectView trace details.

   ATrace details panel opens and displays the selected trace.

   Your log data might be available several minutes before your tracedata is available. If you encounter an error when viewing trace dataeither by searching for a trace by ID or by following the steps inthis task, then wait a minute or two and retry the action.

For more information about using the Logs Explorer, seeView logs by using the Logs Explorer.

What's next

• OpenTelemetry
• OTLP specification
• Structured logging
• Troubleshooting Managed Service for Prometheus
• Troubleshoot Cloud Trace