This document demonstrates how to automatically instrument Java applicationsrunning on Google Kubernetes Engine (GKE) by using OpenTelemetry zero-codeinstrumentation. Zero-code instrumentation gives you comprehensiveobservability into your Java workloads with minimal manual effort.

This walkthrough provides detailed steps for deploying an example Javaapplication to GKE, auto-instrumenting the application byusing OpenTelemetry, and visualizing the generated telemetry signals by usingGoogle Cloud Observability.

The walkthrough describes auto-instrumenting Java workloads running onGKE, but you can extend these instructions toother supportedlanguages.

Why use zero-code instrumentation?

Zero-code instrumentation (also calledauto-instrumentation)in OpenTelemetry is a powerful tool that lets application developers add configurableobservability to their code without needing to write any additional codeor modify any existing code. Zero-code instrumentation works by automaticallyadding instrumentation to actively deployed application libraries, givingreliable insights into application performance by emittingGolden Signals.This makes zero-code instrumentation an excellent starting point for anapplication's observability journey.

Zero-code instrumentation in Java emits metrics and trace spans byinstrumenting popular libraries that are widely used in the language'secosystem. For Java, the metrics include:

• JVM runtime metrics emitted by theJava Platform
• HttpClient metrics andHttpClient spans emitted from libraries likethe JavaHttp Client library andthe GoogleHttp Client library.
• Database Client metrics andDatabase Client spans emitted from libraries likeJDBC andLettuce.

The set of metrics emitted depends on the application being instrumentedand the libraries used by the application. For a full list of librariesand frameworks that can be instrumented by the Java zero-code instrumentationagent, seeSupported Libraries and Frameworks.

With auto-instrumentation, you can do things like:

• Get Golden Signals monitoring without having to make any code changes.
• Correlate high p99.9 latency to individual user traces, which span everyRPC call.
• Get consistent telemetry across metrics, logs, and traces from a widevariety of popular libraries and frameworks.

For more information about zero-code instrumentation,see OpenTelemetryZero-code.

Overview

Adding observability to a Java workload running on GKErequires deploying the following components:

• OpenTelemetry zero-code instrumentation, which generates the telemetry fromyour application.
• The OpenTelemetry Collector, which collects the telemetry generated by yourapplication, enriches it with critical metadata and context, and routesit to Google Cloud for consumption.

In this walkthrough, you deploy an instrumented Java application onGKE that emits metrics and traces to Google Cloud. You can thenview these telemetry signals by using Google Cloud's observability offeringssuch as Metrics Explorer, Logs Explorer, and Trace Explorer. Forinformation about how to view and use the generated telemetry, seeView your telemetry.

This walkthrough uses the following chargeable services to write signals:

• Cloud Monitoring's Managed Service for Prometheus
• Cloud Logging
• Cloud Trace
• Google Kubernetes Engine
• Cloud Build
• Artifact Registry

To generate a cost estimate based on your projected usage, use thepricing calculator.

Before you begin

This section describes how to set up your environment for deployingand instrumenting the app and then running it.

Select or create a Google Cloud project

Choose a Google Cloud project for this walkthrough. If you don't alreadyhave a Google Cloud project, then create one:

Sign in to your Google Cloud account. If you're new to Google Cloud, create an account to evaluate how our products perform in real-world scenarios. New customers also get $300 in free credits to run, test, and deploy workloads.

In the Google Cloud console, on the project selector page, select or create a Google Cloud project.

Roles required to select or create a project

• Select a project: Selecting a project doesn't require a specific IAM role—you can select any project that you've been granted a role on.
• Create a project: To create a project, you need the Project Creator role (roles/resourcemanager.projectCreator), which contains theresourcemanager.projects.create permission.Learn how to grant roles.

Note: If you don't plan to keep the resources that you create in this procedure, create a project instead of selecting an existing project. After you finish these steps, you can delete the project, removing all resources associated with the project.

Go to project selector

Verify that billing is enabled for your Google Cloud project.

In the Google Cloud console, on the project selector page, select or create a Google Cloud project.

Roles required to select or create a project

• Select a project: Selecting a project doesn't require a specific IAM role—you can select any project that you've been granted a role on.
• Create a project: To create a project, you need the Project Creator role (roles/resourcemanager.projectCreator), which contains theresourcemanager.projects.create permission.Learn how to grant roles.

Note: If you don't plan to keep the resources that you create in this procedure, create a project instead of selecting an existing project. After you finish these steps, you can delete the project, removing all resources associated with the project.

Go to project selector

Verify that billing is enabled for your Google Cloud project.

Install command-line tools

This document uses following command-line tools:

• gcloud
• kubectl

Thegcloud andkubectl tools are part of theGoogle Cloud CLI. For information about installing them, seeManaging Google Cloud CLI components. To see thegcloud CLI components you have installed, run the followingcommand:

gcloud components list

Set environment variables

To reduce the typing of frequent values, prevent typing errors, and make thecommands in this document runnable, set the following environment variables:

exportGOOGLE_CLOUD_PROJECT=PROJECT_IDexportCLUSTER_NAME=CLUSTER_NAMEexportCLUSTER_REGION=us-east1exportARTIFACT_REGISTRY=opentelemetry-autoinstrument-registryexportREGISTRY_LOCATION=us-east4

Before running the commands, replace the following variablesor use the defaults:

• PROJECT_ID: The identifier of the project.
• CLUSTER_NAME: The name for your cluster. This can refer to a newor existing cluster.
• us-east1: The region for your cluster. If you're using anexisting cluster, then use that cluster's region.
• opentelemetry-autoinstrument-registry: The name for your Artifact Registry repository.This can refer to a new or existing repository.
• us-east4: The region for your Artifact Registry repository.If you're using an existing repository, then use that repository's region.

If you encounter errors when running the commands in this document,check that these variables are set correctly in your terminal environment.

Enable APIs

To use the services in this document, you must have the following APIs enabledin your Google Cloud project:

• Artifact Registry API:artifactregistry.googleapis.com
• Cloud Build API:cloudbuild.googleapis.com
• Google Kubernetes Engine:container.googleapis.com
• Identity and Access Management API:iam.googleapis.com
• Cloud Logging API:logging.googleapis.com
• Cloud Monitoring API:monitoring.googleapis.com
• Telemetry API:telemetry.googleapis.com

These APIs, with the probable exception of the Telemetry API, mightalready be enabled in your Google Cloud project. Run the following command to listthe enabled APIs:

gcloudserviceslist

Enable any of the APIs that are not already enabled.Pay particular attention to theTelemetry API,telemetry.googleapis.com; this documentmight be the first time you've encountered this API.

To enable APIs that aren't already enabled, run thecommands for the corresponding APIs:

gcloudservicesenableartifactregistry.googleapis.comgcloudservicesenablecloudbuild.googleapis.comgcloudservicesenablecontainer.googleapis.comgcloudservicesenableiam.googleapis.comgcloudservicesenablelogging.googleapis.comgcloudservicesenablemonitoring.googleapis.comgcloudservicesenabletelemetry.googleapis.com

Set up your cluster

Set up a GKE cluster for the sample application and collector:

1. Sign in to Google Cloud by running the following commands:

   gcloudauthlogingcloudconfigsetprojectPROJECT_ID

2. If you don't already have a GKE cluster, then create an Autopilot cluster by running the following command:

   gcloudcontainerclusterscreate-auto--regionus-east1CLUSTER_NAME--projectPROJECT_ID

3. To interact with your cluster, connect your localkubectl utilityby running the following command:

   gcloudcontainerclustersget-credentialsCLUSTER_NAME--regionus-east1--projectPROJECT_ID

Deploy the Java application

The steps in this section deploy an uninstrumented sample Java application,theOpenTelemetry Spring Boot instrumentation example, on yourcluster. This section uses Cloud Build and Artifact Registry tocreate and store the application image.

1. If you don't already have an Artifact Registry repository, createa repository by running the following command:

   gcloudartifactsrepositoriescreateopentelemetry-autoinstrument-registry--repository-format=docker--location=us-east4--description="GKE Autoinstrumentation sample app"

2. Clone the sample application by running the following command:

   gitclonehttps://github.com/GoogleCloudPlatform/opentelemetry-operations-java.git

3. Set the following environment variables:

   exportGOOGLE_CLOUD_PROJECT=PROJECT_IDexportARTIFACT_REGISTRY=opentelemetry-autoinstrument-registryexportREGISTRY_LOCATION=us-east4

4. Build the application image and push it to your Artifact Registry repositoryby running the following commands:

   pushdopentelemetry-operations-java/examples/instrumentation-quickstart &&\gcloudbuildssubmit--config<(envsubst <cloudbuild-uninstrumented-app.yaml). &&\popd

   Note: If you are using a Mac, then running the previous commands mightrequire you to first install thegettext package from Homebrew. For moreinformation, see thisStack Overflow article.After installing the package, you might have to set your environmentvariables again.

5. Create a Kubernetes deployment configuration file for the application.Copy the following configuration and save it to a file calleddeployment.yaml:

   apiVersion:v1kind:Servicemetadata:name:quickstart-applabels:app:quickstart-appapp.kubernetes.io/part-of:gke-autoinstrument-guidespec:ports:-port:8080targetPort:8080name:quickstart-appselector:app:quickstart-app---apiVersion:apps/v1kind:Deploymentmetadata:name:quickstart-applabels:app:quickstart-appapp.kubernetes.io/part-of:gke-autoinstrument-guidespec:replicas:2selector:matchLabels:app:quickstart-apptemplate:metadata:labels:app:quickstart-appspec:containers:-name:quickstart-appimage:us-east4-docker.pkg.dev/PROJECT_ID/opentelemetry-autoinstrument-registry/java-quickstart:latestports:-containerPort:8080name:quickstart-app

6. Create a namespace for your application by running the following command:

   kubectlcreatenamespaceAPPLICATION_NAMESPACE

7. Apply the deployment configuration to your cluster by running thefollowing command:

   kubectlapply-fdeployment.yaml-nAPPLICATION_NAMESPACE

8. After you create the deployment, it might take a while for the podsto be created and start running. To check the status of the pods,run the following command:

   kubectlgetpo-nAPPLICATION_NAMESPACE-w

   To stop watching the pod status, enter Ctrl-C to stop the command.

Set up an OpenTelemetry Collector instance

In this section, you create another deployment that runs aGoogle-Built OpenTelemetry Collector instance in yourGKE cluster. The collector is configured to export data byusing the following exporters:

• Metrics by using theOTLP HTTP exporter
• Traces by using theOTLP HTTP exporter
• Logs by using theGoogle Cloud Exporter

Instead of following the manual steps in this document to deploy theOpenTelemetry Collector, you could followManaged OpenTelemetry for GKE, which usesthe Google-Built OpenTelemetry Collector in a managed environment.

1. Authorize the Kubernetes service account for the OpenTelemetry Collector'sdeployment by running the following commands:

   export PROJECT_NUMBER=$(gcloud projects describePROJECT_ID --format="value(projectNumber)")gcloud projects add-iam-policy-binding projects/PROJECT_ID \  --role=roles/logging.logWriter \  --member=principal://iam.googleapis.com/projects/$PROJECT_NUMBER/locations/global/workloadIdentityPools/PROJECT_ID.svc.id.goog/subject/ns/opentelemetry/sa/opentelemetry-collector \  --condition=Nonegcloud projects add-iam-policy-binding projects/PROJECT_ID \  --role=roles/monitoring.metricWriter \  --member=principal://iam.googleapis.com/projects/$PROJECT_NUMBER/locations/global/workloadIdentityPools/PROJECT_ID.svc.id.goog/subject/ns/opentelemetry/sa/opentelemetry-collector \  --condition=Nonegcloud projects add-iam-policy-binding projects/PROJECT_ID \  --role=roles/telemetry.tracesWriter \  --member=principal://iam.googleapis.com/projects/$PROJECT_NUMBER/locations/global/workloadIdentityPools/PROJECT_ID.svc.id.goog/subject/ns/opentelemetry/sa/opentelemetry-collector \  --condition=None

2. Deploy the Google-Built OpenTelemetry Collector instance on your GKE clusterwith the recommended, default configuration by running the followingcommand:

   kubectlkustomizehttps://github.com/GoogleCloudPlatform/otlp-k8s-ingest//k8s/base?ref=otlpmetric|envsubst|kubectlapply-f-

   This configuration deploys the collector to theopentelemetry namespace.

3. After you create the deployment, it might take a while for the podsto be created and start running. To check the status of the pods,run the following command:

   kubectlgetpo-nopentelemetry-w

   To stop watching the pod status, enter Ctrl-C to stop the command.

Configure OpenTelemetry zero-code instrumentation

To configure OpenTelemetry zero-code instrumentation for an application deployedon GKE, you install theOpenTelemetry Operatoron your GKE cluster. The OpenTelemetry Operator provides accessto a Kubernetes Custom Resource Definition (CRD), the Instrumentation Resource,that is used to inject auto-instrumentation into Kubernetes resources runningwithin a cluster.

The Instrumentation Resource is "injected" or "applied" to specificworkloads by using special annotations. The OpenTelemetry Operator observes theseannotations on resources running within the cluster and injects theappropriate instrumentation in the applications managed by those resources.

1. If you don't already have theHelm CLI installed,then install Helm by running the following commands:

   curl-fsSL-oget_helm.shhttps://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3chmod700get_helm.sh./get_helm.sh

2. If you don't already havecert-manager installed on the cluster,then install it by running the following commands:

   helmrepoaddjetstackhttps://charts.jetstack.iohelmrepoupdatehelminstall\--create-namespace\--namespacecert-manager\--setinstallCRDs=true\--setglobal.leaderElection.namespace=cert-manager\--setextraArgs={--issuer-ambient-credentials=true}\cert-managerjetstack/cert-manager

3. Install the OpenTelemetry Operator by running the following command andusing version 0.140.0 or newer:

   kubectlapply-fhttps://github.com/open-telemetry/opentelemetry-operator/releases/download/v0.140.0/opentelemetry-operator.yaml

4. Create a configuration file for theInstrumentation Custom Resourceby creating a file namedinstrumentation.yaml with thefollowing contents:

   apiVersion:opentelemetry.io/v1alpha1kind:Instrumentationmetadata:name:sample-java-auto-instrumentationspec:exporter:endpoint:http://opentelemetry-collector.opentelemetry.svc.cluster.local:4317sampler:type:parentbased_traceidratio# Adjust the sampling rate to control costargument:"0.01"java:env:-name:OTEL_EXPORTER_OTLP_PROTOCOLvalue:grpc-name:OTEL_LOGS_EXPORTERvalue:none

5. Create a namespace for the Operator. You can use the same value youused for the application namespace,APPLICATION_NAMESPACE.

   kubectlcreatenamespaceINSTRUMENTATION_NAMESPACE

6. Create the Instrumentation Custom Resource in your cluster andapply yourinstrumentation.yaml file by running the following command:

   kubectlapply-finstrumentation.yaml-nINSTRUMENTATION_NAMESPACE

7. Apply the auto-instrumentation annotation to the example Javaapplicationdeployed previously. Patch thethe original deployment with the annotation by running the followingcommand:

   kubectlpatchdeployment.apps/quickstart-app-nAPPLICATION_NAMESPACE-p'{"spec":{"template":{"metadata":{"annotations":{"instrumentation.opentelemetry.io/inject-java": "'"INSTRUMENTATION_NAMESPACE"'/sample-java-auto-instrumentation"}}}}}'

8. After patching the deployment, which causes the application deploymentto restart, you can check the status of the pods in the applicationby running the following command:

   kubectlgetpo-nAPPLICATION_NAMESPACE-w

   To stop watching the pod status, enter Ctrl-C to stop the command.

The Java application deployed in your GKE cluster is nowauto-instrumented.

Generate telemetry from the deployed application

To generate telemetry from the auto-instrumented application, you need tosend requests to the application to simulate user activity. You can generatea steady stream of traffic by running a traffic simulator. This sectiondescribes how download the simulator, build it by using Cloud Build,store in image in Artifact Registry, and deploy the simulator:

1. Download the traffic simulator application by running thefollowing commands:

   mkdirtraffic-simulator &&pushdtraffic-simulator &&\curl-Ohttps://raw.githubusercontent.com/GoogleCloudPlatform/opentelemetry-operator-sample/refs/heads/main/recipes/self-managed-otlp-ingest/traffic/hey.Dockerfile  &&\curl-Ohttps://raw.githubusercontent.com/GoogleCloudPlatform/opentelemetry-operator-sample/refs/heads/main/recipes/self-managed-otlp-ingest/traffic/cloudbuild-hey.yaml &&\popd

   The previous commands do the following:

   1. Create a separate directory for building a traffic-simulator application.
   2. Download the Dockerfile for the application.
   3. Download the Cloud Build configuration to build theapplication image.

2. Create the application image with Cloud Build andpush it to Artifact Registry by running the following commands:

   pushdtraffic-simulator &&\gcloudbuildssubmit--config<(envsubst <cloudbuild-hey.yaml). &&\popd

3. Create a deployment configuration file for the simulator by copying thefollowing content into a file in thetraffic-simulator directory namedquickstart-traffic.yaml:

   apiVersion:apps/v1kind:Deploymentmetadata:name:traffic-simulatorlabels:app.kubernetes.io/part-of:gke-autoinstrument-guidespec:replicas:1selector:matchLabels:app:traffic-simulatortemplate:metadata:labels:app:traffic-simulatorspec:containers:-name:traffic-simulatorimage:us-east4-docker.pkg.dev/PROJECT_ID/opentelemetry-autoinstrument-registry/hey:latestargs:--c=2--q=1--z=1h-http://quickstart-app:8080/multi

4. Deploy the traffic simulator by running the following commands:

   pushdtraffic-simulator &&\kubectlapply-fquickstart-traffic.yaml-nAPPLICATION_NAMESPACE &&\popd

5. After you create the deployment, it might take a while for the podsto be created and start running. To check the status of the pods,run the following command:

   kubectlgetpo-nAPPLICATION_NAMESPACE-w

   To stop watching the pod status, enter Ctrl-C to stop the command.

View your telemetry

After the traffic simulator starts sending requests to the application,the deployed Java application generates telemetry. It might take a fewminutes for the metrics and traces to start showing up in Google Cloud.

• The instrumentation injected by the OpenTelemetry Operator relies on theOpenTelemetry Java Agent. The OpenTelemetry Java Agent containsbuilt-in instrumentation for many popularJava libraries and frameworks that emit metrics.

• The injected instrumentation also automatically captures traces for anycalls made to any endpoint exposed by the sample application.The traffic simulator constantly sends requests to the/multi endpointexposed by the Java application at a fixed rate. The/multi endpoint inturn sends requests to/single endpoint to generate the final response.These interactions end up as traces that can be viewed using theTrace Explorer.

View metrics

You can view your auto-instrumented metrics in Cloud Monitoring'sMetrics Explorer. This section includes sample queries written inPromQL.

In the Google Cloud console, go to theleaderboard Metrics explorer page:

Go toMetrics explorer

If you use the search bar to find this page, then select the result whose subheading isMonitoring.

Some of the metrics you can view for the sample application include thefollowing:

• JVM Runtime metrics: The OpenTelemetry Java Agent emits metrics about theunderlying JVM. These metrics expose information about JVM memory,garbage collection, classes, threads, and other related concepts. For afull list of supported JVM metrics, see theJVM Runtime MetricsSemantic Conventions page.

  • JVM memory usage:paste the following PromQL query into Metrics Explorer:

    sum(avg_over_time({"process.runtime.jvm.memory.usage"}[${__interval}]))

  • JVM CPU utilization:paste the following PromQL query into Metrics Explorer:

    sum(avg_over_time({"process.runtime.jvm.cpu.utilization"}[${__interval}]))

    The following chart shows the JVM CPU utilization metric:

    

• Spring Boot metrics: Thesample applicationuses the Spring Boot framework, which is also supported by OpenTelemetryzero-code instrumentation for Java.

  • Mean HTTP client duration:paste the following PromQL query into Metrics Explorer:

    sum(rate({"http.client.duration_sum"}[${__interval}])) by (cluster, job) / sum(rate({"http.client.duration_count"}[${__interval}])) by (cluster, job)

  • Mean HTTP server duration:paste the following PromQL query into Metrics Explorer:

    sum(rate({"http.server.duration_sum"}[${__interval}])) by (cluster, job) / sum(rate({"http.server.duration_count"}[${__interval}])) by (cluster, job)

    The following chart shows the Spring Boot mean HTTP server duration metric:

    

For more information about using Metrics Explorer, seeSelect metrics when using Metrics Explorer.

View traces

You can view your auto-instrumented traces in Cloud Trace'sTrace Explorer. You can filter these traces in Trace Explorerby filtering on the "Service Name" and setting the service name toquickstart-app.

In the Google Cloud console, go to theTrace explorer page:

Go toTrace explorer

You can also find this page by using the search bar.

The following screenshot shows traces generated by the OpenTelemetryquickstart-app deployment:

For more information about using Trace Explorer, seeFind and explore traces.

Clean up

If you created a new Google Cloud project and you no longer need it, thenyou can delete it by running the following command:

gcloudprojectsdeletePROJECT_ID

If you used an existing project, you can delete the resources you createdas part of this walkthough to save costs:

• Delete the GKE cluster by running the following command:

  gcloudcontainerclustersdeleteCLUSTER_NAME--location=us-east1

• Delete the Artifact Registry repository by running the following command:

  gcloudartifactsrepositoriesdeleteopentelemetry-autoinstrument-registry--location=us-east4