This product or feature is subject to the "Pre-GA Offerings Terms" in the General Service Terms section of theService Specific Terms. Pre-GA products and features are available "as is" and might have limited support. For more information, see thelaunch stage descriptions.

This document explains how to set upManaged OpenTelemetry for GKE to sendOpenTelemetry Protocol (OTLP)traces, metrics, and logs to Google Cloud Observability fromapplications running on GKE.

For more details about how the Managed OpenTelemetry for GKEworks, seeManaged OpenTelemetry for GKE.

You can use Managed OpenTelemetry for GKE to do the following:

• Configure workloads running on GKE to send OpenTelemetryProtocol (OTLP) traces, metrics, and logs to the managed collector.
• Receive OpenTelemetry Protocol (OTLP) traces, metrics, andlogs from the applications running on GKE.
• Export that data to Google Cloud Observability.

If you need collector-level filtering andcontrols, use theGoogle-Built OpenTelemetry Collector instead of this managed offering.

Before you begin

Sign in to your Google Cloud account. If you're new to Google Cloud, create an account to evaluate how our products perform in real-world scenarios. New customers also get $300 in free credits to run, test, and deploy workloads.

Install the Google Cloud CLI.

Note: If you installed the gcloud CLI previously, make sure you have the latest version by runninggcloud components update.

If you're using an external identity provider (IdP), you must first sign in to the gcloud CLI with your federated identity.

Toinitialize the gcloud CLI, run the following command:

gcloudinit

Create or select a Google Cloud project.

Roles required to select or create a project

• Select a project: Selecting a project doesn't require a specific IAM role—you can select any project that you've been granted a role on.
• Create a project: To create a project, you need the Project Creator role (roles/resourcemanager.projectCreator), which contains theresourcemanager.projects.create permission.Learn how to grant roles.

Note: If you don't plan to keep the resources that you create in this procedure, create a project instead of selecting an existing project. After you finish these steps, you can delete the project, removing all resources associated with the project.

• Create a Google Cloud project:

  gcloud projects createPROJECT_ID

  ReplacePROJECT_ID with a name for the Google Cloud project you are creating.

• Select the Google Cloud project that you created:

  gcloud config set projectPROJECT_ID

  ReplacePROJECT_ID with your Google Cloud project name.

Verify that you have the permissions required to complete this guide.

Verify that billing is enabled for your Google Cloud project.

Enable the GKE, Telemetry (OTLP), Cloud Logging, Cloud Monitoring, Cloud Trace APIs:

Roles required to enable APIs

To enable APIs, you need the Service Usage Admin IAM role (roles/serviceusage.serviceUsageAdmin), which contains theserviceusage.services.enable permission.Learn how to grant roles.

gcloudservicesenablecontainer.googleapis.com telemetry.googleapis.com logging.googleapis.com monitoring.googleapis.com cloudtrace.googleapis.com

Install the Google Cloud CLI.

Note: If you installed the gcloud CLI previously, make sure you have the latest version by runninggcloud components update.

If you're using an external identity provider (IdP), you must first sign in to the gcloud CLI with your federated identity.

Toinitialize the gcloud CLI, run the following command:

gcloudinit

Create or select a Google Cloud project.

Roles required to select or create a project

• Select a project: Selecting a project doesn't require a specific IAM role—you can select any project that you've been granted a role on.
• Create a project: To create a project, you need the Project Creator role (roles/resourcemanager.projectCreator), which contains theresourcemanager.projects.create permission.Learn how to grant roles.

Note: If you don't plan to keep the resources that you create in this procedure, create a project instead of selecting an existing project. After you finish these steps, you can delete the project, removing all resources associated with the project.

• Create a Google Cloud project:

  gcloud projects createPROJECT_ID

  ReplacePROJECT_ID with a name for the Google Cloud project you are creating.

• Select the Google Cloud project that you created:

  gcloud config set projectPROJECT_ID

  ReplacePROJECT_ID with your Google Cloud project name.

Verify that you have the permissions required to complete this guide.

Verify that billing is enabled for your Google Cloud project.

Enable the GKE, Telemetry (OTLP), Cloud Logging, Cloud Monitoring, Cloud Trace APIs:

Roles required to enable APIs

To enable APIs, you need the Service Usage Admin IAM role (roles/serviceusage.serviceUsageAdmin), which contains theserviceusage.services.enable permission.Learn how to grant roles.

gcloudservicesenablecontainer.googleapis.com telemetry.googleapis.com logging.googleapis.com monitoring.googleapis.com cloudtrace.googleapis.com

Requirements

To use Managed OpenTelemetry for GKE, you must meet thefollowing requirements:

• The cluster must have GKE version 1.34.1-gke.2178000 or later.
• gcloud CLI enabled with version 551.0.0 or later.

Required roles

To get the permissions that you need to enable and use GKE managed OpenTelemetry, ask your administrator to grant you the following IAM roles on your project:

• Kubernetes Engine Cluster Admin (roles/container.clusterAdmin)
• Monitoring Viewer (roles/monitoring.viewer)
• Logs Viewer (roles/logging.viewer)
• Cloud Trace User (roles/cloudtrace.user)

For more information about granting roles, seeManage access to projects, folders, and organizations.

You might also be able to get the required permissions throughcustom roles or otherpredefined roles.

Costs

SeeBilling fordetails about costs connected to the use of Managed OpenTelemetry forGKE.

Enable Managed OpenTelemetry for GKE in a cluster

To set up Managed OpenTelemetry for GKE, you needto do the following:

• Enable Managed OpenTelemetry for GKE in a cluster.
• Configure the application that you are monitoring to send signals to themanaged collector's endpoint.

When you enable Managed OpenTelemetry forGKE, the following objects aredeployed to the cluster:

• A GKE Managed OpenTelemetry collector deployment thatis deployed within thegke-managed-otel namespace. The in-cluster managedOpenTelemetry collector HTTP endpoint for logs, metrics, and traces isthe following:http://opentelemetry-collector.gke-managed-otel.svc.cluster.local:4318.

• A custom resource definition,instrumentations.telemetry.googleapis.com,that you can use to set up automatic configuration of your workloads.

  For more details about custom resources, seecustom resourcein the Kubernetes documentation.

Enable on a new cluster

To enable Managed OpenTelemetry for GKE on a new cluster,follow these steps:

gcloudbetacontainerclusterscreate-autoCLUSTER_NAME\--project=PROJECT_ID\--managed-otel-scope=COLLECTION_AND_INSTRUMENTATION_COMPONENTS\--location=LOCATION\--cluster-version=VERSION

gcloudbetacontainerclusterscreateCLUSTER_NAME\--project=PROJECT_ID\--managed-otel-scope=COLLECTION_AND_INSTRUMENTATION_COMPONENTS\--location=LOCATION\--cluster-version=VERSION

Enable on an existing cluster

To enable the Managed OpenTelemetry for GKE on an existingcluster, follow these steps:

Configure your application to use the Managed OpenTelemetry collector

Applications need to be configured to be able to send signals to themanaged collector's endpoint.When the applications are configured, the Managed OpenTelemetry collectorreceives signals from the applications running on the cluster where thecollector is enabled. Signals from the applicationinclude traces, metrics, and logs.

To send OpenTelemetry signals, applications need to be already instrumentedto generate OpenTelemetry metrics. For details, seesupported workloads.

You can configure your application manually to send signals to the managedcollectorendpoint,or you can use automatic configuration. We don't recommend using both methodstogether for the same workload, because the automatic configuration canoverride manual changes.This combination might make it more difficult to track changes to theconfiguration.

The following sections describe how to configure applications to send signalsto the collector using the automatic configuration.

Set up automatic configuration

Automatic configuration uses environment variables to configure theworkloads to send signals to the managed collector's endpoint.

To enable automatic injection of environment variables into Pods, you use theInstrumentation custom resource. The environment variables have the OpenTelemetryconfiguration, and they can be injected into some Pods with matched labels in anamespace or all Pods in a namespace.

Then, when an application is deployed tothe namespace, GKE uses the configurationto automatically inject environment variables to the Podswhere the workloads run.

1. To configure theInstrumentation custom resource, do the following:

   1. Save the followingInstrumentation manifest in a file namedotlp-auto-config-namespace.yaml:

      apiVersion:telemetry.googleapis.com/v1alpha1kind:Instrumentationmetadata:namespace:NAMESPACEname:NAMEspec:selector:matchLabels:KEY:VALUEautoInstrumentationConfig:configInjection:enabled:trueotelSDKConfig:tracer_provider:sampler:parent_based:root:trace_id_ratio_based:ratio:"TRACE_RATIO"meter_provider:readers:-periodic:interval:METRICS_INTERVAL

      Replace the following:

      • NAMESPACE: the namespace thatcontains the Pods you want to target for auto-instrumentation.Usedefault to target the default namespace.
      • NAME: the name of the manifest file. In thisexample, the name isotlp-auto-config-namespace.yaml.
      • (Optional) The label attached to Pods to target. If an emptyselector is specified ({}), then all Pods in the namespaceare targeted.
        • KEY: the label's key.
        • VALUE: the label's value.
      • TRACE_RATIO: the ratio of trace data tocollect. If unspecified, the default is1.0.For more details, seeModify the trace sampling rate.
      • METRICS_INTERVAL: the interval, inmilliseconds, of monitoring data to collect. The default is60000.The value must be non-negative, with a minimum of 5,000 ms,maximum of 300,000 ms, and multiple of 5,000 ms.For more details, seeModify the metric export interval.

   2. If you want to modify any of the settings, then see the followingsection tomodify the configuration.

   3. Apply the configuration by running the following command:

      kubectlapply-fotlp-auto-config-namespace.yaml

2. To inject the environment variables automatically, you need to deploythe application to the namespace in your cluster that has theconfiguration applied.

   • To apply the configuration to a workload that is not yet running inthe namespace, deploy the workload using the followingcommand:

     kubectlapply-fDEPLOYMENT_NAME-nNAMESPACE

     Replace the following:

     • DEPLOYMENT_NAME: The name of the deployment.
     • NAMESPACE: The namespace.

   • To apply the configuration to a workload that is already running inthe namespace, redeploy the workload using the followingcommand:

     kubectlrolloutrestartdeploymentDEPLOYMENT_NAME-nNAMESPACE

     Replace the following:

     • DEPLOYMENT_NAME: The name of the deployment.
     • NAMESPACE: The namespace.

After you apply the configuration to the cluster, GKEautomatically configures all workloads when they are deployed to the cluster.The workloads are instrumented by injecting environment variables to the Podswhere the workloads run.

When a workload is configured with these environment variables is runningin a cluster where the managed collector is deployed, then as the workload runsit sends OpenTelemetry signals to the managed collector. These signals areavailable for you to view in Google Cloud Observability.

For more detail about viewingthe signals, seeView telemetry. For an example, seeGenerate sample telemetry.

Modify the configuration

To modify the configuration, you need to do the following:

1. Modify theInstrumentation manifest file.

2. Apply the modified configuration.

3. Redeploy or restart the applications in the corresponding namespace ofyour cluster after applying the modified configuration.

For more details about these steps, follow the instructions in the sectionCreate and deploy the configuration.

Modify the amount or frequency of data collection

You can modify the amount of trace data collected bymodifying the trace sampling rate.

You can modify the frequency that monitoring data is sent to Cloud Monitoringbymodifying the metric export interval.

You can't modify the amount or frequency of logging data collected. You can,however, disable all logging, metrics, or tracing data from being collected.For details, seeSelect the signal type to collect.

Modify the trace sampling rate

A workload can generate a large amount of trace data. For your own situation,it's important for you to determine the balance between the cost ofcollecting and storing data, and the level of detail that you need for thedata to be useful.

Thedefault OpenTelemetry SDK behaviorisalways_on, which is equivalent to a ratio of 1.

The following is an example of the configuration of the trace sample rate.In this example, the ratio is 0.25 and so trace data is collected at a rateof 25 percent. Modify this ratio number to change the sample rate.

tracer_provider:sampler:parent_based:root:trace_id_ratio_based:ratio:"0.25"

Modify the metric export interval

The metric export interval determines the granularity of data that you areable to see in the graphs in Cloud Monitoring.

The following is an example of the configuration of the metric exportinterval. In this example, the export interval is 60,000 ms.

Metric export interval is used to specify the delay intervalbetween the start of two consecutive exports of metrics from theOpenTelemetry SDK.

The value of this interval must be non-negative, with a minimum of 5,000 ms,maximum of 300,000 ms, and multiple of 5,000 ms. The value is expressedin milliseconds.

meter_provider:readers:-periodic:interval:60000

Select the signal types to collect

You can control which signal types are collected from a workload by disablingthe signal typesthat you don't want to collect. Signal types are traces, metrics, and logs.

You disable signal types using the environment variables in the containerwhere the workload runs. You modify environment variables by modifying theInstrumentation custom resource and then redeploying the workload tothe container.

The following example is anInstrumentation manifest file configured for thecollection of only trace data. The collection of logs andmetrics is disabled becausemeter_provider andlogger_provider are settonull.

apiVersion:telemetry.googleapis.com/v1alpha1kind:Instrumentationmetadata:namespace:defaultname:otlp-auto-config-disable-metrics-logsspec:selector:matchLabels:# Update the labels to match your workloadsapp:telemetrygen-appautoInstrumentationConfig:configInjection:enabled:trueotelSDKConfig:meter_provider:nulllogger_provider:null

Disable automatic configuration of workloads

To disable automatic instrumentation of workloads with the specifiedconfiguration, delete theInstrumentation custom resource from yourcluster. To do so, use the following command:

kubectldeleteinstrumentations.telemetry.googleapis.com<instrumentation-name>-n<namespace-name>

To disable automatic environment variable injection temporarily whilepreserving auto-instrumentation configuration for future use, setautoInstrumentationConfig.configInjection.enabled tofalse and apply theupdated custom resource.

The following is an example of the custom resource with the automaticenvironment variable injection temporarily disabled:

apiVersion:telemetry.googleapis.com/v1alpha1kind:Instrumentationmetadata:namespace:defaultname:otlp-auto-config-examplespec:selector:matchLabels:# Update the labels to match your workloadsapp:telemetrygen-appautoInstrumentationConfig:configInjection:enabled:false# disable environment variables config injectionotelSDKConfig:...# preserve OpenTelemetry configuration for future use

After you delete the custom resource or update it to disable automatic configinjection, GKE doesn't auto-instrument new workloadsthat are targeted by theInstrumentation custom resource.

To stop exporting OTLP signals to the managed collector from aworkload that was previously instrumented by the custom resource,you must restart the workload for the change to take effect. To do so,use the following command:

kubectlrolloutrestartdeployment<deployment-name>-n<namespace-name>

View telemetry

When a configured workload runs on GKE whereManaged OpenTelemetry for GKE is enabled, thenOpenTelemetry signals are sent to Google Cloud Observability.

For details about viewing data in Google Cloud Observability, see the following:

• Find and explore traces.
• Create charts with Metrics Explorer.
• View and analyze logs.

Generate sample telemetry

This section describes deploying a sample application and pointing thatapplication to the OTLP endpoint of the Managed OpenTelemetry collector.You can thenview the telemetry in Google Cloud.

The sample application is a small generator that exports traces,logs, and metrics to the in-cluster managed OpenTelemetry collector HTTPendpoint. The OTLP endpoint is hard-coded within the application, pointing tohttp://opentelemetry-collector.gke-managed-otel.svc.cluster.local:4318.

If you already have an application instrumented with an OpenTelemetry SDK, thenyou can generate telemetry from your application by pointing your applicationto the collector's endpoint, orconfiguring automatic instrumentationfor the application.

To deploy the sample application, do the following:

1. Connect to your cluster where you have enabled Managed OpenTelemetry. To doso, seeSet a default cluster forkubectl commands.

2. Run the following command:

   kubectlapply-fhttps://raw.githubusercontent.com/GoogleCloudPlatform/otlp-k8s-ingest/main/sample/gke-app.yaml

   After a few minutes, telemetry generated by the application begins flowingthrough the collector to the Google Cloud backend for each signal.

   1. Verify that the telemetry is ingested by viewing logs, metrics, andtraces from the demo application in Google Cloud console:

   2. To view metrics, do the following:

      1. In the Google Cloud console, go to the Metrics Explorer page:

         Go to Metrics Explorer

      2. Run the following PromQL query in Metrics Explorer:

         sum(avg_over_time({"__name__"="gen","namespace"="opentelemetry-demo","job"="telemetrygen"}[1h]))

   3. To view traces, do the following:

      1. In the Google Cloud console, go to theTrace Explorer page.

         Go to Trace Explorer

      2. Filter trace spans by span name equal tolets-go.

   4. To view logs, do the following:

      1. In the Google Cloud console, go to the Logs Explorer page.

         Go to Logs Explorer

      2. Run the following query:

         resource.type="k8s_pod"resource.labels.namespace_name="opentelemetry-demo"

Disable Managed OpenTelemetry for GKE

You can disable Managed OpenTelemetry for GKE in the cluster.When youdisable the collector, the Managed OpenTelemetry collector is removed fromthe cluster, and no new telemetry data is collected.

To disable the Managed OpenTelemetry for GKE, use the following steps.

gcloudbetacontainerclustersupdateCLUSTER_NAME\--project=PROJECT_ID\--managed-otel-scope=NONE\--location=LOCATION

When you disable the Managed OpenTelemetry for GKE, theInstrumentation custom resource definition and theInstrumentation customresources aren't removed from the cluster.If you re-enable managed OpenTelemetry, then it uses the configuration preservedin theInstrumentation custom resources.

If you have telemetry data that was already collected by Managed OpenTelemetryfor GKE, thendisabling the collector does not affect this data. Existing data is still storedin Google Cloud Observability, and no new telemetry data is collected.

Troubleshooting

Autopilot partner privileged workloads

If you try to use automatic configuration with an Autopilot partner privilegedworkload, then you might see that workload Pod was rejected.

OpenTelemetry config injection is not supported forprivileged workloads from GKE Autopilot partners.Targeting such workloads using anInstrumentation custom resource to enableOpenTelemetryenvironment variable injection may cause the workload to fail to match theAutopilot privileged workload allowlist, which means the config-injected Podwould be rejected by GKE Autopilot.

Logs, metrics, or traces are not visible in the Google Cloud console

Data might not be visible for many different reasons. These reasons includemissing permissions to view the data, or incorrect configuration thatprevent data from being collected.

Steps that you can take to resolve common issues are the following:

• Ensure you have allrequired APIs enabled in yourproject.

• Ensure that theInstrumentation custom resource is correctly configured,with namespace matching the namespace where the workload is running,and the selector matching the label of your workload.

• Inspect the workload's Pod to see if theenvironment variablesare injected correctly.

• Check the container logs of OpenTelemetry collector to see if there areerrors in the collector. To do so, run the following command:

  kubectllogs-ngke-managed-otel-lapp=opentelemetry-collector-copentelemetry-collector

Disabling a telemetry signal is not working

When you disable a telemetry signal using theInstrumentationcustom resource, make sure you apply the custom resource and redeploy theworkloads.

When applying the custom resource, useServer-Side Applyin thekubectl apply command when updating theInstrumentationcustom resource.

For details about disabling a telemetry signal, seeSelect the signal types to collect.

OpenTelemetry injected variables are not visible in my workload

The variables are injected into the containers of workload pods , not theworkload. Check the Pods,not the owner objects like ReplicaSets or Deployments.

For example, to confirm the variables are injected correctly for the sampleworkload in default namespace used in the previoussectionGenerate telemetry, do the following:

1. Run the following command:

   kubectlgetpods-ndefault-lapp=telemetrygen-app-oyaml

2. Examine thespec.containers[*].env of the Pods.

3. Ensure that there is anInstrumentation object in the same namespace andcheck that it is targeting the Pod and has config injection feature enabled.To do so, run the following command:

   kubectlgetinstrumentations.telemetry.googleapis.com-ndefault-oyaml

The variables are injected into the containers only when Pods are createdbecause theKubernetes API doesn't allow modifying most fields in the spec of an existingPod, such as environment variables. For the configuration to take effect onworkloads that were created before you created theInstrumentation object,restart the workload. For example, for a Deployment namedtelemetry-gen-app,run the following command:

kubectlrolloutrestartdeployment-ndefaulttelemetry-gen-app

An excessive amount of trace data in Cloud Trace

To reduce the data collected by Cloud Trace, you can configure aparent-based sampler with a trace ID ratio to only sample apercentage of your traces.

For example, add the following to theInstrumentation object:

spec:otelSDKConfig:tracer_provider:sampler:parent_based:root:trace_id_ratio_based:ratio:"0.01"

Thedefault OpenTelemetry SDK behavioris "always_on" tracing, which is equivalent to a ratio of 1.

Environment variables don't match the configuration

If you made an update to theInstrumentation object, check that you haverestarted your Pods as described in the sectionModify the configuration.

If you see the wrong configuration for your Pod, check that the Pod iscorrectly targeted by theInstrumentation object, and that you don't havemultipleInstrumentation objects targeting that same Pod:

kubectlgetinstrumentations--all-namespaces\-ocustom-columns=NAMESPACE:.metadata.namespace,NAME:.metadata.name,SELECTOR:.spec.selectorkubectlgetpod-n${NAMESPACE:?}${POD_NAME:?}--show-labels

Note that an empty selector targets all Pods in its namespace.

If multiple instrumentations target the same Pod when it is created, theinstrumentation that was last updated takes effect.

What's next

• For details about how Managed OpenTelemetry for GKE works,seeManaged OpenTelemertry for GKE.
• For a self-deployed alternative to the Managed OpenTelemetry forGKE, seeGoogle-Built OpenTelemetry Collector.