This document shows how to run theGoogle-Built OpenTelemetry Collectoron Compute Engine to collect OTLP logs, metrics, and traces frominstrumented applications and then export that data to Google Cloud.

Before you begin

Running the Google-Built OpenTelemetry Collector requires the following resources:

Sign in to your Google Cloud account. If you're new to Google Cloud, create an account to evaluate how our products perform in real-world scenarios. New customers also get $300 in free credits to run, test, and deploy workloads.

In the Google Cloud console, on the project selector page, select or create a Google Cloud project.

Roles required to select or create a project

• Select a project: Selecting a project doesn't require a specific IAM role—you can select any project that you've been granted a role on.
• Create a project: To create a project, you need the Project Creator role (roles/resourcemanager.projectCreator), which contains theresourcemanager.projects.create permission.Learn how to grant roles.

Note: If you don't plan to keep the resources that you create in this procedure, create a project instead of selecting an existing project. After you finish these steps, you can delete the project, removing all resources associated with the project.

Go to project selector

Verify that billing is enabled for your Google Cloud project.

Enable the Cloud Logging, Cloud Monitoring, and Cloud Trace APIs.

Roles required to enable APIs

To enable APIs, you need the Service Usage Admin IAM role (roles/serviceusage.serviceUsageAdmin), which contains theserviceusage.services.enable permission.Learn how to grant roles.

Enable the APIs

In the Google Cloud console, on the project selector page, select or create a Google Cloud project.

Roles required to select or create a project

• Select a project: Selecting a project doesn't require a specific IAM role—you can select any project that you've been granted a role on.
• Create a project: To create a project, you need the Project Creator role (roles/resourcemanager.projectCreator), which contains theresourcemanager.projects.create permission.Learn how to grant roles.

Note: If you don't plan to keep the resources that you create in this procedure, create a project instead of selecting an existing project. After you finish these steps, you can delete the project, removing all resources associated with the project.

Go to project selector

Verify that billing is enabled for your Google Cloud project.

Enable the Cloud Logging, Cloud Monitoring, and Cloud Trace APIs.

Roles required to enable APIs

To enable APIs, you need the Service Usage Admin IAM role (roles/serviceusage.serviceUsageAdmin), which contains theserviceusage.services.enable permission.Learn how to grant roles.

Enable the APIs

1. A Compute Engine instance. If you don't have a Compute Engine instance, then follow the instructions atCreate and start a Compute Engine instance.
2. An installation ofgcloud. For information about installinggcloud, seeInstall thegcloud CLI.

Configure permissions for the Collector

By default, Compute Engine instances use theCompute Engine default service account,PROJECT_NUMBER-compute@developer.gserviceaccount.com.This service account usually has the Identity and Access Management (IAM) rolesnecessary to write the metrics and logs described in this document:

• Monitoring Metric Writer (roles/monitoring.metricWriter)
• Logs Writer (roles/logging.logWriter)
• Cloud Trace Agent

If you choose to use a user-managed service account instead of the default, itmust be assigned these same roles. For more information, seeService accounts.

Install the Collector

To install the Google-Built OpenTelemetry Collector as a package for your Compute Engine instance,complete the following steps.

Deploy the Collector

The Google-Built OpenTelemetry Collector includes a minimal configuration by default that does notexport telemetry. To export your telemetry, after installing the Collector,configure the Collector by overriding the defaultconfiguration. Put your configuration for the Collector in the following files:

• For Linux:/etc/otelcol-google/config.yaml
• For Windows:C:\Program Files\Google\OpenTelemetry Collector\config.yaml

Configure the Collector

We provide an OpenTelemetry Collector configuration for you to use with theGoogle-built Collector. This configuration is designed to deliver highvolumes of OTLP metrics, logs, andtraces.This configuration is also designed to prevent common ingestionissues. You can add to the configuration, but we strongly recommend that youdon't remove elements.

This section describes the provided configuration, the key componentslike exporters, processors, receivers, and other available components.

Provided Collector configuration

receivers:# Open two OTLP servers:# - On port 4317, open an OTLP GRPC server# - On port 4318, open an OTLP HTTP server## Docs:# https://github.com/open-telemetry/opentelemetry-collector/tree/main/receiver/otlpreceiverotlp:protocols:grpc:endpoint:localhost:4317http:cors:# This effectively allows any origin# to make requests to the HTTP server.allowed_origins:-http://*-https://*endpoint:localhost:4318processors:# The batch processor is in place to regulate both the number of requests# being made and the size of those requests.## Docs:# https://github.com/open-telemetry/opentelemetry-collector/tree/main/processor/batchprocessorbatch:send_batch_max_size:200send_batch_size:200timeout:5s# The memorylimiter will check the memory usage of the collector process.## Docs:# https://github.com/open-telemetry/opentelemetry-collector/tree/main/processor/memorylimiterprocessormemory_limiter:check_interval:1slimit_percentage:65spike_limit_percentage:20# The resourcedetection processor is configured to detect GCP resources.# Resource attributes that represent the GCP resource the collector is# running on will be attached to all telemetry that goes through this# processor.## Docs:# https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/resourcedetectionprocessor# https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/resourcedetectionprocessor#gcp-metadataresourcedetection:detectors:[gcp]timeout:10stransform/collision:metric_statements:-context:datapointstatements:-set(attributes["exported_location"], attributes["location"])-delete_key(attributes, "location")-set(attributes["exported_cluster"], attributes["cluster"])-delete_key(attributes, "cluster")-set(attributes["exported_namespace"], attributes["namespace"])-delete_key(attributes, "namespace")-set(attributes["exported_job"], attributes["job"])-delete_key(attributes, "job")-set(attributes["exported_instance"], attributes["instance"])-delete_key(attributes, "instance")-set(attributes["exported_project_id"], attributes["project_id"])-delete_key(attributes, "project_id")exporters:# The googlecloud exporter will export telemetry to different# Google Cloud services:# Logs -> Cloud Logging# Metrics -> Cloud Monitoring# Traces -> Cloud Trace## Docs:# https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/exporter/googlecloudexportergooglecloud:log:default_log_name:opentelemetry-collector# The googlemanagedprometheus exporter will send metrics to# Google Managed Service for Prometheus.## Docs:# https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/exporter/googlemanagedprometheusexportergooglemanagedprometheus:extensions:# Opens an endpoint on 13133 that can be used to check the# status of the collector. Since this does not configure the# `path` config value, the endpoint will default to `/`.## When running on Cloud Run, this extension is required and not optional.# In other environments it is recommended but may not be required for operation# (i.e. in Container-Optimized OS or other GCE environments).## Docs:# https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/extension/healthcheckextensionhealth_check:endpoint:0.0.0.0:13133service:extensions:-health_checkpipelines:logs:receivers:-otlpprocessors:-resourcedetection-memory_limiter-batchexporters:-googlecloudmetrics/otlp:receivers:-otlpprocessors:-resourcedetection-transform/collision-memory_limiter-batchexporters:-googlemanagedprometheustraces:receivers:-otlpprocessors:-resourcedetection-memory_limiter-batchexporters:-googlecloud# Internal telemetry for the collector supports both push and pull-based telemetry data transmission.# Leveraging the pre-configured OTLP receiver eliminates the need for an additional port.## Docs:# https://opentelemetry.io/docs/collector/internal-telemetry/telemetry:metrics:readers:-periodic:exporter:otlp:protocol:grpcendpoint:http://localhost:4317insecure:true

Exporters

The Collector configuration includes the following exporters:

• googlecloudexporter, for logs and traces. This exporter is configured with a defaultlog name.

• googlemanagedprometheusexporter, for metrics. This exporter does not require any configuration,but there are configuration options. For information about configurationoptions for thegooglemanagedprometheus exporter, seeGet started with theOpenTelemetry Collectorin the Google Cloud Managed Service for Prometheus documentation.

Processors

The Collector configuration includes the following processors:

• batch:Configured to batch telemetry requests at the Google Cloud maximum number ofentries per request, or at the Google Cloud minimum interval of every 5seconds (whichever comes first).

• memory_limiter:Caps the Collector's memory usage to prevent out-of-memory crashes bydropping data points when the limit is exceeded.

• resourcedetection:Automatically detects Google Cloud resource labels such asproject_id.

Receivers

The Collector configuration includes only theotlp receiver.For information about instrumenting your applications to send OTLP tracesand metrics to the Collector's OTLP endpoint, seeChoose an instrumentationapproach.

Available components

The Google-Built OpenTelemetry Collector contains the components that most users will need toenable a rich experience within Google Cloud Observability. For a complete list ofavailable components, seeComponentsin theopentelemetry-operations-collector repository.

To request any changes or additions to the available components,open a feature request.in theopentelemetry-operations-collector repository.

Generate telemetry

You can test your configuration by using the open-sourcetelemetrygen tool. Instructions for downloadingand running the tool are included at the link. To run the tool, you must installeither Go or Docker.

After a few minutes, telemetry generated by the application begins flowingthrough the Collector to the Google Cloud console for each signal.

View telemetry

The Google-Built OpenTelemetry Collector sends metrics, logs, and traces from your instrumentedapplications to Google Cloud Observability. The Collector also sends self-observabilitymetrics. The following sections describe how to view this telemetry.

View your metrics

The Google-Built OpenTelemetry Collector collects Prometheus metrics that you can view by usingtheMetrics Explorer. The metrics collected dependon the instrumentation of the app, although the Google-built Collector alsowrites some self-metrics.

1. In the Google Cloud console, go to theleaderboard Metrics explorer page:

   Go toMetrics explorer

   If you use the search bar to find this page, then select the result whose subheading isMonitoring.

2. In the toolbar of the Google Cloud console, select your Google Cloud project. ForApp Hub configurations, select the App Hub host project or the app-enabled folder's management project.
3. In theMetric element, expand theSelect a metric menu, enterPrometheus Target in the filter bar, and then use the submenus to select a specific resource type and metric:
   1. In theActive resources menu, selectPrometheus Target.
   2. To select a metric, use theActive metric categories andActive metrics menus. Metrics collected by the Google-Built OpenTelemetry Collector have theprefixprometheus.googleapis.com.
   3. ClickApply.

4. To add filters, which remove time series from the query results, use theFilter element.

5. Configure how the data is viewed.
   

   When the measurements for a metric arecumulative, Metrics Explorer automatically normalizes the measured data bythe alignment period, which results in the chart displaying a rate. Formore information, seeKinds, types, and conversions.

   When integer or double values are measured, such as withcounter metrics, Metrics Explorer automatically sums all time series.To change this behavior, set the first menu of theAggregation entrytoNone.

   For more information about configuring a chart, seeSelect metrics when using Metrics Explorer.

View your traces

To view your trace data, do the following:

1. In the Google Cloud console, go to theTrace explorer page:

   Go toTrace explorer

   You can also find this page by using the search bar.

2. In the toolbar of the Google Cloud console,select your Google Cloud project. ForApp Hubconfigurations, select the App Hub host project or management project.
3. In the table section of the page, select a row.

4. In the Gantt chart on theTrace details panel,select a span.

   A panel opens that displays information about the traced request. Thesedetails include the method, status code, number of bytes, and theuser agent of the caller.

5. To view the logs associated with this trace,select theLogs & Events tab.

   The tab shows individual logs. To view the details of the log entry,expand the log entry. You can also clickView Logs and view the logby using the Logs Explorer.

For more information about using the Cloud Trace explorer, seeFind and explore traces.

View your logs

From the Logs Explorer, you can inspect your logs, and you can alsoview associated traces, when they exist.

1. In the Google Cloud console, go to thesegmentLogs Explorer page:

   Go toLogs Explorer

   If you use the search bar to find this page, then select the result whose subheading isLogging.

2. Locate a log entry from your instrumented app.To view the details, expand the log entry.

3. ClickTraces on a log entry with a tracemessage, and then selectView trace details.

   ATrace details panel opens and displays the selected trace.

For more information about using the Logs Explorer, seeView logs by using the Logs Explorer.

Observe and debug the Collector

The Google-Built OpenTelemetry Collector automatically provides self-observability metrics tohelp you monitor its performance and ensure continued uptime of the OTLPingestion pipeline.

To monitor the Collector, install the sample dashboard for the Collector. Thisdashboard offers at-a-glance insights into several metrics from the Collector,including uptime, memory usage, and API calls to Google Cloud Observability.

To install the dashboard, do the following:

1. In the Google Cloud console, go to thedashboard Dashboards page:

   Go toDashboards

   If you use the search bar to find this page, then select the result whose subheading isMonitoring.

2. ClickDashboard Templates.
3. Search for theOpenTelemetry Collector dashboard.
4. Optional: To preview the dashboard, select it.

5. Clickplaylist_addAdd dashboard to your list andthen complete the dialog.

   The dialog lets you select the name of the dashboard,and add labels to the dashboard.

For more information about installing dashboards, seeInstall a dashboardtemplate.