This document describes how to configure anApp Hub application so that the metricsand trace data the application generates contains application-specific labels.It also describes how to have your application export a metric that is usedby Application Monitoring to report the traffic level, server error rate, or thelatency for HTTP requests workloads that run on Google Kubernetes Engine.

About application-specific labels

Application-specific labels refer to labels or attributes that is injectedinto log, metric, or trace data. These labels identify the service or workloadthat generated the data. You can use application-specific labels like any otherlabels. For example, you can filter data by the ID of an application.Telemetry generated bysupported infrastructureautomatically includes these labels. With instrumentation, metrics and traceswritten by your application can include these labels.

In this section,interpreta.b.{x,y} asa.b.x anda.b.y.

Metric data from instrumentation you added to your applications can include the followingmetric labels:

• metric.labels.apphub_application_{container,id,location}
• metric.labels.apphub_workload_{criticality_type,environment_type,id}

Trace spans generated by instrumentation you added to your applications can include the followingresource attributes:

• gcp.apphub.application.{container,id,location}
• gcp.apphub.{workload,service}.{criticality_type,environment_type,id}

About OpenTelemetry HTTP server metrics

There are no system metrics for workloads that run on Google Kubernetes Enginethat can report the traffic level, server error rate, or thelatency for HTTP requests. However, the values for these golden signalscan be derived from thehttp.server.request.duration,which is a metric automatically collected by theOpenTelemetry HTTP clientlibrary.

Thegooglemanagedprometheus exporter converts thehttp.server.request.duration OpenTelemetry metric into a Prometheus metric thathas the following properties:

• Name:prometheus/http_server_request_duration_seconds/histogram
• Metric:prometheus.googleapis.com/http_server_request_duration_seconds/histogram
• Resource types:prometheus_target
• Unit:s
• Kind:CUMULATIVE
• Value type:DISTRIBUTION

When your Google Cloud project contains thehttp_server_request_duration_secondsmetric, then your Application Monitoring dashboards display the traffic level,server error rate and the latency for HTTP requests.

To learn how to configure your application to export thehttp_server_request_duration_seconds metric,see theUse OpenTelemetry on Kubernetes section of this document.

Add application labels to metric data

Google Cloud Observability attempts to identify the source of Prometheus metrics by comparingthe attributes attached to Prometheus metrics that are sent to your projectwith data returned from the App Hub API.The remainder of this section lists configurations that letGoogle Cloud Observability identify the App Hub application.

Use OpenTelemetry on Kubernetes

Note: We recommend that you use this approach with the Google-Built OpenTelemetry Collector. If yourun workloads on Google Kubernetes Engine, then this approach exports the HTTP servermetric that Application Monitoring uses to generate values for the traffic level,server error rate, and the latency for HTTP requests.

To have Google Cloud Observability attach application labels to metric data generated by yourapplication's workloads that run in Google Kubernetes Engine, do the following:

1. Instrument the application with OpenTelemetry.

   • For general information, seeInstrumentation and observability.
   • For links to Go, Java, Node.js, and Python samples along with informationabout those samples, seeInstrumentation samples overview.

2. Deploy either the Google-built Collector or the OpenTelemetry Collector, andconfigure the collector as follows:

   • Export metric data with thegooglemanagedprometheus exporter.
   • Configure thek8sattributes processor to extract metadata,like thenamespace, from the environment.
   • Configure thetransform/collision processor to set theproject_id,location,cluster, andnamespace attributes.
   • Configure thetransform/aco-gke processor to populate thetop_level_controller_name andtop_level_controller_type labels.

   For an example, seeotlp-k8s-ingest/config/collector.yaml,which is the configuration file for the Google-built Collector. To learnmore about this collector, seeDeploy Google-Built OpenTelemetry Collector on GKE.

3. Register your application's workloads withApp Hub.

Use Google Cloud Managed Service for Prometheus on GKE

To have Google Cloud Observability attach application labels to metric data generated by yourapplication's workloads that runs on Google Kubernetes Engine clusters, do the following:

1. UseGoogle Cloud Managed Service for Prometheus with managed collection.

2. Deploy your workloads on a Google Kubernetes Engine cluster whose version is atleast1.32.1-gke.1439000.

3. Register your application's workloads withApp Hub.

Google Cloud Managed Service for Prometheus discoversthe values of the application labels throughservice discoverymetadata and then adds thetop_level_controller_{name,type} labels to thetargetLabels.metadata.During metric ingestion, Google Cloud Observability uses thetop_level_controller_{name,type} labels and the App Hub API to identifyyour App Hub application and to add the appropriate labels to yourmetric data.

Use Cloud Run

To have Google Cloud Observability attach application labels to metric data generated by yourCloud Run workloads, do the following:

1. Instrument your application by using either OpenTelemetry or theManaged Service for Prometheus sidecar for Cloud Run.For information about these approaches, see the following documents:

   • Deploy Google-Built OpenTelemetry Collector on Cloud Run
   • Use the Prometheus sidecar for Cloud Run

2. Register your application's workloads withApp Hub.

Verify metric labels

To verify that your application is sending Prometheus metrics, to your project,do the following:

1. Verify that your application is sending Prometheus metrics to your project:

   1. In the Google Cloud console, go to theleaderboard Metrics explorer page:

      Go toMetrics explorer

      If you use the search bar to find this page, then select the result whose subheading isMonitoring.

   2. In the toolbar of the Google Cloud console, select your Google Cloud project. ForApp Hub configurations, select the App Hub host project or the app-enabled folder's management project.
   3. In theMetric element, expand theSelect a metric menu, enterPrometheus Target in the filter bar, and then use the submenus to select a specific resource type and metric:
      1. In theActive resources menu, selectPrometheus Target.
      2. To select a metric, use theActive metric categories andActive metrics menus.
      3. ClickApply.

   4. To add filters, which remove time series from the query results, use theFilter element.

   5. Configure how the data is viewed.
      

      To view labels, expand theFilter menu.Each entry in theFilter corresponds to a label.

      For more information about configuring a chart, seeSelect metrics when using Metrics Explorer.

   If you aren't seeing any Prometheus metrics, then review your configuration.

If your Prometheus metrics don't contain application labels, then do thefollowing:

1. Verify that you registered your workload or service withApp Hub.

2. Examine your logs to determine whether there are any errors.

   For example, if you deploy an OpenTelemetry Collector or theGoogle-built Collector and are running on Google Kubernetes Engine, then you might dothe following:

   1. In the Google Cloud console, go to theWorkloads page:

      Go toWorkloads

      If you use the search bar to find this page, then select the result whose subheading isKubernetes Engine.

   2. Select your collector deployment, and then select theLogs tab.

3. If you are using an OpenTelemetry Collector or the Google-built Collector, thenreview the configuration of your collector. Your collector must do thefollowing:

   • Export metric data with thegooglemanagedprometheus exporter.
   • Configure thek8sattributes processor to extract metadata,like thenamespace, from the environment.
   • Configure thetransform/collision processor to set theproject_id,location,cluster, andnamespace attributes.
   • Configure thetransform/aco-gke processor to populate thetop_level_controller_name andtop_level_controller_type labels.

   For an example, see Google-built Collector configuration file:otlp-k8s-ingest/config/collector.yaml.

Add application attributes to trace spans

To have Cloud Trace attach application-specific resource attributes to tracedata generated by application's services and workloads, do the following:

1. Register your services and workloads withApp Hub.

2. Instrument your application with OpenTelemetry and send the trace data collectedfrom your application to your project by using theTelemetry (OTLP) API.

   Note: You must send trace data to your Google Cloud project by using theTelemetry API. If you use the Cloud Trace exporter or theCloud Trace API, then your trace data won't contain application attributes.To learn how to modify your application to use the Telemetry API, seeMigrate from the Cloud Trace exporter to the OTLP endpoint.

3. Configure the OpenTelemetry Collector or the Google-built Collectorsuch that the exported span data includes theOpenTelemetry resource attributesthat identify thesupported Google Cloud resourcesthat are used by your application. The resource attributes must includethe following:

   • cloud.account.id
   • One ofcloud.{availability_zone,region}
   • Resource-specific attributes. For example, for a Kubernetes workload,spans must have thek8s.cluster.name,k8s.namespace, and the type ofKubernetes deployment set.

   Use processors to have the collector attach resource-specific attributesto your spans. To learn more, seeresourcedetectionprocessorandk8sattributesprocessor.

   Google Cloud Observability uses the preceding resource attributes and the App Hub APIto identify your App Hub application. Application-specific attributesare added to trace data when an application is identified.

For more information about these attributes, seeGoogle Cloud App Hub semantic conventions.