Write Prometheus metrics by using the Prometheus sidecar

The Managed Service for Prometheus sidecar for Cloud Run is theGoogle-recommended way to getPrometheus-style monitoringfor Cloud Run services. If your Cloud Run servicewritesOTLP metrics, then you can use theOpenTelemetrysidecar. But for services that writePrometheusmetrics, use the Managed Service for Prometheus sidecar described in thisdocument.

This document describes how to do the following:

The sidecar usesGoogle Cloud Managed Service for Prometheus on the server sideand a distribution of the OpenTelemetry Collector, custom-built for serverlessworkloads, on the client side. This custom distributionincludes some changes to support Cloud Run. The collectorperforms a start-up scrape after 10 seconds and performs a shut-down scrape,no matter how short-lived the instance is. For maximum reliability, werecommend that Cloud Run services running the sidecaralso use the instance-based billing setting; for more information, seeBilling settings (services). Some scrape attempts mightfail if the CPU allocation is throttled due to a low number of queries persecond (QPS). An always-allocated CPU remains available.

The sidecar relies on the Cloud Run multi-container (sidecar)feature to run the collector as a sidecar container alongside your workloadcontainer. For more information about sidecars in Cloud Run, seeDeploying multiple containers to a service (sidecars).

The Managed Service for Prometheus sidecar for Cloud Run introduces a new configuration,RunMonitoring, a subset of the KubernetesPodMonitoring customresource. Most users can use the defaultRunMonitoringconfiguration, but you can also create custom configurations. For moreinformation about the default and customRunMonitoringconfigurations, seeAdd the sidecar to a Cloud Runservice.

Before you begin

This section describes the setup necessary to use this document.

Install and configure the gcloud CLI

Many of the steps described in this document use the Google Cloud CLI.For information about installing the gcloud CLI, seeManaging Google Cloud CLI components.

When invokinggcloud commands, you must specify the identifierfor your Google Cloud project. You can set a default project for theGoogle Cloud CLI and eliminate the need to enter it with every command.To set your project as the default, enter the following command:

gcloud config set projectPROJECT_ID

You can also set a default region for your Cloud Run service.To set a default region, enter the following command:

gcloud config set run/regionREGION

Enable APIs

You must enable the following APIs in your Google Cloud project:

  • Cloud Run Admin API:run.googleapis.com
  • Artifact Registry API:artifactregistry.googleapis.com
  • Cloud Monitoring API:monitoring.googleapis.com
  • Cloud Logging API:logging.googleapis.com
  • (Optional) If you create acustomRunMonitoringconfig, then you mustenable the Secret Manager API:secretmanager.googleapis.com
  • (Optional) If you choose to run the sample app by using Cloud Build,then you must also enable the following APIs:
    • Cloud Build API:cloudbuild.googleapis.com.
    • Identity and Access Management API:iam.googleapis.com.For more information, seeBuild and run the sampleapp.

To see the APIs enabled in your project, run the following command:

gcloud services list

To enable an API that isn't enabled, run one of following commands:

gcloud services enable run.googleapis.comgcloud services enable artifactregistry.googleapis.comgcloud services enable secretmanager.googleapis.comgcloud services enable monitoring.googleapis.comgcloud services enable logging.googleapis.comgcloud services enable cloudbuild.googleapis.comgcloud services enable iam.googleapis.com

Service account for Cloud Run

By default, Cloud Run jobs and services use theCompute Engine default service account,PROJECT_NUMBER-compute@developer.gserviceaccount.com.This service account usually has the Identity and Access Management (IAM) rolesnecessary to write the metrics and logs described in this document:

  • roles/monitoring.metricWriter
  • roles/logging.logWriter

If you create acustomRunMonitoring config,then your service account must also have the following roles:

  • roles/secretmanager.admin
  • roles/secretmanager.secretAccessor

You can also configure a user-managed service account forCloud Run. A user-managed service account must also have theseroles. For more information about service accounts for Cloud Run,seeConfiguring Service Identity.

Configure and add the sidecar to a Cloud Run service

The Managed Service for Prometheus sidecar for Cloud Run introduces a new configuration,RunMonitoring, which is a subset of the KubernetesPodMonitoring customresource. TheRunMonitoring configuration uses existingPodMonitoring options to support Cloud Runwhile eliminating some options that are specific to Kubernetes.

You can use the defaultRunMonitoring configuration, oryou can create a custom configuration. The following sections describe bothapproaches. You don't need to do both. For more information about usingdefault or custom configurations, select the corresponding tab.

Default configuration

Use the default RunMonitoring configuration

If you don't create a customRunMonitoring configuration, then thesidecar collector synthesizes the following default configuration to scrapemetrics from port 8080 at metrics path/metrics every 30 seconds:

apiVersion: monitoring.googleapis.com/v1betakind: RunMonitoringmetadata:  name: run-gmp-sidecarspec:  endpoints:  - port: 8080    path: /metrics    interval: 30s

Using the default configuration requires no additional setup beyondadding the sidecar to your Cloud Run service.

Add the sidecar to a Cloud Run service

To use the sidecar with the defaultRunMonitoringconfiguration, you need to modify your existing Cloud Runconfiguration to add the sidecar. To add the sidecar, do the following:

  • Add a container-dependency annotation that specifies the startup andshutdown order of the containers. In the following example, thesidecar container, named "collector", starts after and shuts downbefore the application container, named "app" in the example.
  • Create a container for the collector, named "collector" in thefollowing example.

For example, add the lines preceded by the+ (plus) character to yourCloud Run configuration, and then redeploy your service:

apiVersion: serving.knative.dev/v1kind: Servicemetadata:  annotations:    run.googleapis.com/launch-stage: ALPHA    run.googleapis.com/cpu-throttling: 'false'  name: my-cloud-run-servicespec:  template:    metadata:      annotations:        run.googleapis.com/execution-environment: gen2+       run.googleapis.com/container-dependencies: '{"collector":["app"]}'    spec:      containers:      - image: "REGION-docker.pkg.dev/PROJECT_ID/run-gmp/sample-app"        name: app        startupProbe:          httpGet:            path: /startup            port: 8000        livenessProbe:          httpGet:            path: /liveness            port: 8000        ports:        - containerPort: 8000+     - image: "us-docker.pkg.dev/cloud-ops-agents-artifacts/cloud-run-gmp-sidecar/cloud-run-gmp-sidecar:1.2.0"+       name: collector

To redeploy the Cloud Run service with the configuration filerun-service.yaml, run the following command:

gcloud run services replacerun-service.yaml --region=REGION
Note: If you'reusing Artifact Registry with VPC Service Controls,then you must also configure your egress rule to allow egress to thecloud-ops-agents-artifacts project. To do so, addprojects/1042947621568as a resource in your egress rule.

Custom configuration

Create a custom RunMonitoring configuration

You can create aRunMonitoring configuration for aCloud Run service if the default configuration is not sufficient.For example, you might create a configuration like the following:

apiVersion: monitoring.googleapis.com/v1betakind: RunMonitoringmetadata:  name: my-custom-cloud-run-jobspec:  endpoints:  - port: 8080    path: /metrics    interval: 10s    metricRelabeling:    - action: replace      sourceLabels:      - __address__      targetLabel: label_key      replacement: label_value  targetLabels:    metadata:    - service    - revision

This configuration does the following:

  • Scrapes metrics from the port 8080 and uses the default metrics path/metrics.
  • Uses a 10-second scrape interval.
  • Uses relabelling to add the labellabel_key with the valuelabel_value to every metric scraped.
  • Adds theservice andrevision metadata labels to every metric scraped.

For information on the available configuration options, seeRunMonitoring spec: configuration options.

When you use a customRunMonitoring configuration, you mustperform the following additional configuration:

Store the configuration by using Secret Manager

To provide a customRunMonitoring configuration, do the following:

  • Create a file that contains the custom configuration.
  • Create a Secret Manager secret that contains the customconfiguration.

The following command creates a secret namedmysecret from thecustom-config.yaml file:

gcloud secrets createmysecret --data-file=custom-config.yaml

To pick up a change to the configuration after you modify thecustom-config.yaml file, you must delete and recreatethe secret, and then redeploy the Cloud Run service.

Updating the configuration in Secret Manager

If you want to update the customRunMonitoring configuration at anytime, you can add a new version of the existing secret toSecret Manager.

For example, to updatemysecret with a new config stored in filecustom-config-updated.yaml, you can run:

gcloud secrets versions addmysecret --data-file=custom-config-updated.yaml

The sidecar automatically reloads and applies the changes to its configuration.

Add the sidecar to a Cloud Run service

After you have created a Secret Manager secret for yourRunMonitoring configuration, you must modify yourCloud Run configuration to do the following:

  • Add the Managed Service for Prometheus sidecar
    • Add a container-dependency annotation that specifies the startup andshutdown order of the containers. In the following example, thesidecar container, named "collector", starts after and shuts downbefore the application container, named "app" in the example.
    • Create a container for the collector, named "collector" in thefollowing example.
  • Add the secret by mounting thesecret at the location/etc/rungmp/config.yaml:
    • Add a secrets annotation that points to the secret holding yourcustomRunMonitoring configuration.
    • Create a volume, named "config" in the following example, for the secretthat points to the fileconfig.yaml.
    • Mount the volume as part of the collector image at the mount path/etc/rungmp.

For example, add the lines preceded by the+ (plus) character to yourCloud Run configuration, and then redeploy your service:

apiVersion: serving.knative.dev/v1kind: Servicemetadata:  annotations:    run.googleapis.com/launch-stage: ALPHA    run.googleapis.com/cpu-throttling: 'false'  name: my-cloud-run-servicespec:  template:    metadata:      annotations:        run.googleapis.com/execution-environment: gen2+       run.googleapis.com/container-dependencies: '{"collector":["app"]}'+       run.googleapis.com/secrets: 'mysecret:projects/PROJECT_ID/secrets/mysecret'    spec:      containers:      - image: "REGION-docker.pkg.dev/PROJECT_ID/run-gmp/sample-app"        name: app        startupProbe:          httpGet:            path: /startup            port: 8000        livenessProbe:          httpGet:            path: /liveness            port: 8000        ports:        - containerPort: 8000+     - image: "us-docker.pkg.dev/cloud-ops-agents-artifacts/cloud-run-gmp-sidecar/cloud-run-gmp-sidecar:1.2.0"+       name: collector+       volumeMounts:+       - mountPath: /etc/rungmp/+         name: config+     volumes:+     - name: config+       secret:+         items:+         - key: latest+           path: config.yaml+         secretName: 'mysecret'

To redeploy the Cloud Run service with the configuration filerun-service.yaml, run the following command:

gcloud run services replacerun-service.yaml --region=REGION

RunMonitoring spec: configuration options

This section describesRunMonitoring configuration spec for theManaged Service for Prometheus sidecar. The following shows an example custom configuration:

apiVersion: monitoring.googleapis.com/v1betakind: RunMonitoringmetadata:  name: my-custom-cloud-run-jobspec:  endpoints:  - port: 8080    path: /metrics    interval: 10s    metricRelabeling:    - action: replace      sourceLabels:      - __address__      targetLabel: label_key      replacement: label_value  targetLabels:    metadata:    - service    - revision

RunMonitoring

TheRunMonitoring configuration monitors a Cloud Run service.

FieldDescriptionSchemeRequired
metadataMetadata that all persisted resources must have. metav1.ObjectMetafalse
specSpecification of the Cloud Run deployment selected for target discovery by Prometheus.RunMonitoringSpectrue
RunMonitoringSpec

This spec describes how theRunMonitoring configurationmonitors a Cloud Run service.

FieldDescriptionSchemeRequired
endpointsEndpoints to scrape on the selected pods.[]ScrapeEndpointtrue
targetLabelsLabels to add to the Prometheus target for discovered endpoints. Theinstance label is always set to the Cloud Run instance ID.RunTargetLabelsfalse
limitsLimits to apply at scrape time.*ScrapeLimitsfalse
RunTargetLabels

TheRunTargetLabels configuration lets you includeCloud Run labels from the discovered Prometheus targets.

FieldDescriptionSchemeRequired
metadataCloud Run metadata labels that are set on all scraped targets.

Permitted keys areinstance,revision,service, andconfiguration.

If permitted keys are set, then the sidecar adds the corresponding value from the Cloud Run instance as metric labels to every metric:instanceID,revision_name,service_name, andconfiguration_name.

Defaults to all permitted keys being set.
*[]stringfalse

Build and run a sample app

This section describes how to run the Managed Service for Prometheus sidecar with a sample app.This section is optional. If you already have a Cloud Run serviceand want to deploy the sidecar with it, then seeAdd the Managed Service for Prometheus sidecar to a Cloud Run service.

Clone therun-gmp-sidecar repository

To get the sample app and its configuration files, clone therun-gmp-sidecarrepository by running the following command:

git clone https://github.com/GoogleCloudPlatform/run-gmp-sidecar/

After cloning the repository, change to therun-gmp-sidecar directory:

cdrun-gmp-sidecar

Build and run the sample app

The sample app requires Docker or a similar container build system for Linux.You can build and run the sample app in either of the following ways:

  • Use Cloud Build to run without local Docker support.If you use Cloud Build, then you must also enable theCloud Build API.
  • Build and run the sample app manually.

The following sections describe both approaches. You don't need to do both.For more information, select the tab for one of the approaches.

Use Cloud Build

Use Cloud Build

Therun-gmp-sidecar repository includes configuration files forCloud Build. These files bundle together the steps neededto build and deploy the sample app.

You can also manually perform the steps handled by Cloud Build.For more information, seeBuild and run manually.

Set up for Cloud Build

These configuration files require a Cloud Build service accountand an Artifact Registry repository. Therun-gmp-sidecar repository also includes ascript,create-sa-and-ar.sh that does the following:

  • Creates a service account,run-gmp-sa@PROJECT_ID.iam.gserviceaccount.com.
  • Grants the following roles to the service account:
    • roles/iam.serviceAccountUser
    • roles/storage.objectViewer
    • roles/logging.logWriter
    • roles/artifactregistry.createOnPushWriter
    • roles/secretmanager.admin
    • roles/secretmanager.secretAccessor
    • roles/run.admin
  • Creates an Artifact Registry repository,run-gmp, for yourcontainer images.

To create therun-gmp-sa service account andrun-gmp repository,run the following command:

./create-sa-and-ar.sh

It takes some time for the permissions to propagate, so we recommend waitingabout 30 seconds before proceeding to the next step. Otherwise, you mightsee authorization errors.

Submit Cloud Build request

After you've set up therun-gmp-sa service account andrun-gmprepository, you can start a Cloud Build job by submitting theCloud Build configuration file. You can use the followinggcloud builds submit command:

gcloud builds submit . --config=cloudbuild-simple.yaml --region=REGION

This command takes several minutes to run, but it almost immediatelyindicates you where you can find the Cloud Build build details.The message looks like the following:

Logs are available at [https://console.cloud.google.com/cloud-build/builds/637860fb-7a14-46f2-861e-09c1dc4cea6b?project=PROJECT_NUMBER].

To watch the build progress, navigate to the returned URL in your browser.You can alsoview the sidecar logs in Cloud Logging.

Check the service URL

After the Cloud Build job ends, check the Cloud Runservice endpoint URL by running the following command:

gcloud run services describe my-cloud-run-service --region=REGION --format="value(status.url)"

This command returns the service URL, which looks like the following:

https://my-cloud-run-service-abcdefghij-ue.a.run.app

Build and run manually

Build and run manually

You can manually perform the steps handled by Cloud Build,described inUse Cloud Build. The following sectionsdescribe how to manually perform the same tasks.

Set variables used by later steps

Several of the later steps use environment variables for common values.Set up these variables by using the following commands:

export GCP_PROJECT=PROJECT_IDexport REGION=REGION

Create a container image repository

Create an Artifact Registry repository for the container image by runningthe following command:

gcloud artifacts repositories create run-gmp \    --repository-format=docker \    --location=${REGION}

Build and push the sample app

This example uses Docker on Linux to build the sample app and push itto an Artifact Registry repository. You might need to adapt these commands if youare working in a Windows or macOS environment.

Note: If you needsudo toexecutedocker commands, then you also need to authenticate byusingsudo as well. Failure to do so results in the wrong systemuser being configured, and requests to Cloud Run are notauthenticated.
sudo gcloud auth login --no-launch-browsersudo gcloud auth configure-docker

To build the sample app and push it to Artifact Registry, do the following:

  1. Authenticate your Docker client with the Google Cloud CLI:

    gcloud auth configure-docker ${REGION}-docker.pkg.dev
  2. Go to thesimple-app directory in therun-gmp-sidecar repository:

    pushd sample-apps/simple-app

  3. Build thesimple-app app by running the following command:

    docker build -t ${REGION}-docker.pkg.dev/${GCP_PROJECT}/run-gmp/sample-app .
  4. Push the image for the built app by running the following command:

    docker push ${REGION}-docker.pkg.dev/${GCP_PROJECT}/run-gmp/sample-app

  5. Return to therun-gmp-sidecar directory:

    popd

Configure the Cloud Run service

Therun-service-simple.yaml file in therun-gmp-sidecar repository defines amulti-container Cloud Run service that uses the sample app andcollector images that you built in previous steps. Therun-service-simple.yaml file contains the following Service specification:

apiVersion: serving.knative.dev/v1kind: Servicemetadata:  annotations:    run.googleapis.com/launch-stage: ALPHA    run.googleapis.com/cpu-throttling: 'false'  name: my-cloud-run-servicespec:  template:    metadata:      annotations:        run.googleapis.com/execution-environment: gen2        run.googleapis.com/container-dependencies: '{"collector":["app"]}'    spec:      containers:      - image: "%SAMPLE_APP_IMAGE%"        name: app        startupProbe:          httpGet:            path: /startup            port: 8000        livenessProbe:          httpGet:            path: /liveness            port: 8000        ports:        - containerPort: 8000      - image: "us-docker.pkg.dev/cloud-ops-agents-artifacts/cloud-run-gmp-sidecar/cloud-run-gmp-sidecar:1.2.0"        name: collector

This file contains a placeholder value for the images you've built in previoussteps, so you must update the placeholders with the actual values for yourGoogle Cloud project.

Replace the%SAMPLE_APP_IMAGE% placeholder by running the following command:

sed -i s@%SAMPLE_APP_IMAGE%@REGION-docker.pkg.dev/${GCP_PROJECT}/run-gmp/sample-app@g run-service-simple.yaml

Deploy the Cloud Run service

After you've updated therun-service-simple.yaml file with your values, youcan create and deploy the Cloud Run service by running thefollowing command:

gcloud run services replace run-service-simple.yaml --region=REGION

This command returns the service URL, which looks like the following:

https://my-cloud-run-service-abcdefghij-ue.a.run.app

Allow unauthenticated HTTP access

Before you can make a request to the service URL, you need to changethe Cloud Run service policy to accept unauthenticatedHTTP access. Thepolicy.yaml file in therun-gmp-sidecar repositorycontains the necessary change.

To change the service policy, run the following command:

gcloud run services set-iam-policy my-cloud-run-service policy.yaml --region=REGION

Verify that your app is running

To verify that the Cloud Run service has correctly run thesample app, use thecurl utility to access the service'smetricsendpoint.

  1. Run the following command to obtain your service URL:

    SERVICE_URL=$(gcloud run services describe my-cloud-run-service --region=REGION --format 'value(status.url)')

    ReplaceREGION with the Cloud Run region of your service.

  2. Send a request to the service URL by running the following command:

    curl $SERVICE_URL

If your app was successfully started, you see the following response:

User request received!

View app metrics in Metrics Explorer

The sampleapp container writes the following metrics:

  • foo_metric: The current time as a floating-point value (gauge).
  • bar_metric: The current time as a floating-point value (counter).

To view these metrics in Metrics Explorer, do the following:

  1. In the Google Cloud console, go to the Metrics explorer page:

    Go toMetrics explorer

    If you use the search bar to find this page, then select the result whose subheading isMonitoring.

  2. In the toolbar of thequery-builder pane, select the button whose name is either MQL or PromQL.

  3. Verify thatPromQL is selectedin theLanguage toggle. The language toggle is in the same toolbar thatlets you format your query.

  4. Enter the following query into the editor pane:

    foo_metric
  5. ClickAdd query.

  6. In the toolbar of thequery-builder pane, select the button whose name is either MQL or PromQL.

  7. Verify thatPromQL is selectedin theLanguage toggle. The language toggle is in the same toolbar thatlets you format your query.

  8. Enter the following query into the second editor pane:

    bar_metric
  9. ClickRun query.

  10. To see details about the metrics, in thetoggle labeledChart Table Both, selectBoth.

Clean up

When you are finished experimenting with the sample app, you canuse theclean-up-cloud-run.sh script in therun-gmp-sidecar repository todelete the following resources you might have created for the sample:

  • The Cloud Run service.
  • The Artifact Registry repository.
  • The service account created for Cloud Build.

Deleting these resources ensures that you don't incur costs afterrunning the sample.

To clean up the sample app, run theclean-up-cloud-run.sh script:

./clean-up-cloud-run.sh

View self metrics in Metrics Explorer

The Managed Service for Prometheus sidecar reports the following metrics about itself toCloud Monitoring:

  • agent_uptime: The uptime of the sidecar collector (counter).
  • agent_memory_usage: The memory being consumed by the sidecar collector(gauge).
  • agent_api_request_count: The number of API requests from the sidecarcollector (counter).
  • agent_monitoring_point_count: The number of metric points written toCloud Monitoring by the sidecar collector (counter).

To view the sidecar's self metrics in Metrics Explorer, do thefollowing:

  1. In the Google Cloud console, go to the Metrics explorer page:

    Go toMetrics explorer

    If you use the search bar to find this page, then select the result whose subheading isMonitoring.

  2. In the toolbar of thequery-builder pane, select the button whose name is either MQL or PromQL.

  3. Verify thatPromQL is selectedin theLanguage toggle. The language toggle is in the same toolbar thatlets you format your query.

  4. Enter the name of the metric you want to query into the editor pane, forexample:

    agent_api_request_count
  5. ClickRun query.

  6. To see details about the metrics, in thetoggle labeledChart Table Both, selectBoth.

View self logs in Logs Explorer

The Managed Service for Prometheus sidecar writes logs to Cloud Logging. The sidecar writeslogs against the Logging monitored resource typecloud_run_revision.

To view the sidecar's logs in Logs Explorer, do the following:

  1. In the Google Cloud console, go to theLogs Explorer page:

    Go toLogs Explorer

    If you use the search bar to find this page, then select the result whose subheading isLogging.

  2. Select theCloud Run Revision resource type, or enter the followingquery and clickRun query:

    resource.type="cloud_run_revision"

About the collected metrics

When the metrics emitted by the sidecar are ingested into Cloud Monitoring,the metrics are written against the Cloud Monitoringprometheus_target monitored resource type.This resource type is used for both application metrics and the sidecar's selfmetrics.

When writing the metrics, the sidecar sets the resource labels as follows:

  • project_id: The ID of the Google Cloud project in which the container isrunning.
  • location: The Google Cloud region in which the container is running.
  • cluster: The value__run__.
  • namespace: The name of the Cloud Run service being run;comes from theK_SERVICE environment variable.
  • job: Name from theRunMonitoring configuration; defaults torun-gmp-sidecar.
  • instance: The valuefaas.ID:PORT of thelocal workload from which the container is configured to pull metrics. Thefaas.ID value is the instance ID of the Cloud Runinstance.

The sidecar also adds the following metric labels:

  • instanceId: The ID of the Cloud Run instance.
  • service_name: The name of the Cloud Run service being run.
  • revision_name: The name of the Cloud Run revision being run.
  • configuration_name: The name of the Cloud Run configurationbeing run.

These metric labels are all added by default. If you use acustomRunMonitoring configuration, you canomit theservice_name,revision_name, andconfiguration_name labels byusing thetargetLabels option in theRunMonitoring spec. You canalso use a custom configuration to relabel the values of theservice_name,revision_name, andconfiguration_name labels.

All other labels that appear with ingested metrics come from your metric.If a user-defined label conflicts with one of the system-provided labels,the user-defined label is prefixed with the stringexported_. For example,a user-specified labelnamespace="playground" conflicts with the system-definednamespace label, so the user label appears asexported_namespace="playground".

Metric type

When the metrics emitted by the sidecar are ingested into Cloud Monitoring,the metrics are written asprometheus.googleapis.com metrics, and the typeof the Prometheus metric is appended to the end of the name. For example,the sample app emits a Prometheus gauge metric namedfoo_metric. InCloud Monitoring, the metric is stored as the metric typeprometheus.googleapis.com/foo_metric/gauge.

When querying the metric with PromQL, you can use the Prometheus name,as illustrated inView app metrics in Metrics ExplorerandView self metrics in Metrics Explorer.When using tools like the query builder or Monitoring Query Language (MQL) inMetrics Explorer, the Cloud Monitoring metric type is relevant.

Billing

Metrics emitted by the sidecar are ingested into Cloud Monitoring withthe prefixprometheus.googleapis.com. Metrics with this prefix are chargedby the number of samples ingested. For more information about billing andManaged Service for Prometheus, seeBilling.

Except as otherwise noted, the content of this page is licensed under theCreative Commons Attribution 4.0 License, and code samples are licensed under theApache 2.0 License. For details, see theGoogle Developers Site Policies. Java is a registered trademark of Oracle and/or its affiliates.

Last updated 2026-02-18 UTC.