After you have deployed Google Cloud Managed Service for Prometheus,you can query the data sent to the managed service anddisplay the results in charts and dashboards.

This document describesmetrics scopes, which determine thedata you can query, and how to use Grafana to retrieve and use thedata you've collected.

All query interfaces for Managed Service for Prometheus are configured toretrieve data from Monarch using the Cloud Monitoring API. Byquerying Monarch instead of querying data fromlocal Prometheus servers, you get global monitoring at scale.

Before you begin

If you have not already deployed the managed service,then set upmanaged collection orself-deployedcollection. You can skip this if you're only interested inquerying Cloud Monitoring metrics using PromQL.

Configure your environment

To avoid repeatedly entering your project ID or cluster name,perform the following configuration:

• Configure the command-line tools as follows:

  • Configure the gcloud CLI to refer to the ID of yourGoogle Cloud project:

    gcloud config set projectPROJECT_ID

  • If running on GKE, use gcloud CLI to setyour cluster:

    gcloud container clusters get-credentialsCLUSTER_NAME --locationLOCATION --projectPROJECT_ID

  • Otherwise, use thekubectl CLI to set your cluster:

    kubectl config set-clusterCLUSTER_NAME

  For more information about these tools, see the following:

  • gcloud CLI overview
  • kubectl commands

Set up a namespace

Create theNAMESPACE_NAME Kubernetes namespace for resources you createas part of the example application. We recommend using the namespace namegmp-test when using this documentation to configure an example Prometheussetup.

Create the namespace by running the following:

kubectl create nsNAMESPACE_NAME

Verify service account credentials

If your Kubernetes cluster hasWorkload Identity Federation for GKE enabled,then you can skip this section.

When running on GKE, Managed Service for Prometheusautomatically retrieves credentials from the environment based on theCompute Engine default service account. The default service account has thenecessary permissions. If you don't use Workload Identity Federation for GKE and if you havepreviously removed either themonitoring.metricWriter andmonitoring.viewerrole grant from the default node service account, then youhave tore-add those missing roles before continuing.

Configure a service account for Workload Identity Federation for GKE

If your Kubernetes cluster doesn't haveWorkload Identity Federation for GKEenabled, then you can skip this section.

Managed Service for Prometheus captures metric data by using theCloud Monitoring API. If your cluster is using Workload Identity Federation for GKE,you must grant your Kubernetes service account permission to theMonitoring API. This section describes the following:

• Creating a dedicatedGoogle Cloud service account,gmp-test-sa.
• Binding the Google Cloud service account to the defaultKubernetesservice account in a test namespace,NAMESPACE_NAME.
• Granting the necessary permission to the Google Cloud service account.

Create and bind the service account

This step appears in several places in the Managed Service for Prometheusdocumentation. If you have already performed this step as part of a priortask, then you don't need to repeat it. Skip ahead toAuthorize theservice account.

First, create a service account if you haven't yet done so:

gcloud config set projectPROJECT_ID \&&gcloud iam service-accounts creategmp-test-sa

Then use the following command sequence to bind thegmp-test-sa serviceaccount to the default Kubernetes service account in theNAMESPACE_NAME namespace:

gcloud config set projectPROJECT_ID \&&gcloud iam service-accounts add-iam-policy-binding \  --role roles/iam.workloadIdentityUser \  --condition=None \  --member "serviceAccount:PROJECT_ID.svc.id.goog[NAMESPACE_NAME/default]" \gmp-test-sa@PROJECT_ID.iam.gserviceaccount.com \&&kubectl annotate serviceaccount \  --namespaceNAMESPACE_NAME \  default \  iam.gke.io/gcp-service-account=gmp-test-sa@PROJECT_ID.iam.gserviceaccount.com

If you are using a different GKE namespace or service account,adjust the commands appropriately.

Authorize the service account

Groups of related permissions are collected intoroles, andyou grant the roles to a principal, in this example, the Google Cloudservice account. For more information about Monitoring roles,seeAccess control.

The following command grants the Google Cloud service account,gmp-test-sa, the Monitoring API roles it needs toreadmetric data.

If you have already granted the Google Cloud service accounta specific role as part of prior task, then you don't need to do it again.

gcloud projects add-iam-policy-bindingPROJECT_ID \  --member=serviceAccount:gmp-test-sa@PROJECT_ID.iam.gserviceaccount.com \  --role=roles/monitoring.viewer \  --condition=None \&& \gcloud projects add-iam-policy-bindingPROJECT_ID \  --member=serviceAccount:gmp-test-sa@PROJECT_ID.iam.gserviceaccount.com \  --role=roles/iam.serviceAccountTokenCreator \  --condition=None

Debug your Workload Identity Federation for GKE configuration

If you are having trouble getting Workload Identity Federation for GKE to work, see thedocumentation forverifying your Workload Identity Federation for GKE setupand theWorkload Identity Federation for GKE troubleshooting guide.

As typos and partial copy-pastes are the most common sources of errors whenconfiguring Workload Identity Federation for GKE, westrongly recommend using the editablevariables and clickable copy-paste icons embedded in the code samples in theseinstructions.

Workload Identity Federation for GKE in production environments

The example described in this document binds the Google Cloud serviceaccount to the default Kubernetes service account and gives the Google Cloudservice account all necessary permissions to use the Monitoring API.

In a production environment, you might want to use a finer-grained approach,with a service account for each component, each with minimal permissions.For more information on configuring service accounts forworkload-identity management, seeUsing Workload Identity Federation for GKE.

Queries and metrics scopes

The data you can query is determined by the Cloud Monitoring constructmetrics scope, regardless of the method you use to query the data.For example, if you use Grafana to query Managed Service for Prometheusdata, then each metrics scope must be configured as a separate data source.

A Monitoring metrics scope is a read-time-only construct thatlets you query metric data belonging to multiple Google Cloud projects. Everymetrics scope is hosted by a designated Google Cloud project, called thescoping project.

By default, a project is the scoping project for its own metrics scope,and the metrics scope contains the metrics and configuration for thatproject. A scoping project can have more than one monitored project in itsmetrics scope, and the metrics and configurations from all the monitoredprojects in the metrics scope are visible to the scoping project. Amonitored project can also belong to more than one metrics scope.

When you query the metrics in a scoping project, and if thatscoping project hosts a multi-project metrics scope, you can retrievedata from multiple projects. If your metrics scope contains all yourprojects, then your queries and rules evaluate globally.

For more information about scoping projects and metrics scope,seeMetrics scopes. For information about configuringmulti-project metrics scope, seeView metrics for multipleprojects.

Managed Service for Prometheus data in Cloud Monitoring

The simplest way to verify that your Prometheus data is being exportedis to use the Cloud Monitoring Metrics Explorer pagein the Google Cloud console, which supports PromQL. For instructions, seeQuerying using PromQL in Cloud Monitoring.

You can alsoimport yourGrafana dashboards into Cloud Monitoring. Thisenables you to keep using community-created or personal Grafana dashboardswithout having to configure or deploy a Grafana instance.

Grafana

Managed Service for Prometheus uses the built-in Prometheus data source forGrafana, meaning that you can keep using any community-created or personalGrafana dashboards without any changes.

Deploy Grafana, if needed

If you don't have a running Grafana deployment in your cluster, then you cancreate an ephemeral test deployment to experiment with.

To create an ephemeral Grafana deployment, apply theManaged Service for Prometheusgrafana.yaml manifest to yourcluster, and port-forward thegrafana service to your local machine.Dueto CORS restrictions, you can't access a Grafana deployment usingCloud Shell.

1. Apply thegrafana.yaml manifest:

   kubectl -nNAMESPACE_NAME apply -f https://raw.githubusercontent.com/GoogleCloudPlatform/prometheus-engine/v0.17.2/examples/grafana.yaml

2. Port-forward thegrafana service to your local machine. This exampleforwards the service to port 3000:

   kubectl -nNAMESPACE_NAME port-forward svc/grafana 3000

   This command does not return, and while it is running, it reportsaccesses to the URL.

   You can access Grafana in your browser at the URLhttp://localhost:3000with the username:passwordadmin:admin.

Then add a new Prometheus data source to Grafana by doing the following:

1. Go to your Grafana deployment, for example, by browsing to theURLhttp://localhost:3000 to reach the Grafana welcome page.

2. SelectConnections from the main Grafana menu, then selectData Sources.

   

3. SelectAdd data source, and select Prometheus as the time seriesdatabase.

   

4. Give the data source a name, set theURL field tohttp://localhost:9090, then selectSave & Test. You can ignoreany errors saying that the data source is not configured correctly.

5. Copy down the local service URL for your deployment, which will look like thefollowing:

   http://grafana.NAMESPACE_NAME.svc:3000

Configure and authenticate the Grafana data source

Google Cloud APIs all require authentication using OAuth2; however, Grafanadoesn't support OAuth2 authentication for service accounts used with Prometheusdata sources. To use Grafana withManaged Service for Prometheus,you use thedata source syncer to generate OAuth2credentials for your service account and sync them to Grafana through theGrafana data sourceAPI.

You must use the data source syncer to configure and authorize Grafana toquery data globally. If you don't follow these steps, then Grafana onlyexecutes queries against data in the local Prometheus server.

The data source syncer is a command-line interface tool that remotely sendsconfiguration values to a given Grafana Prometheus data source. This ensuresthat your Grafana data source has the following configured correctly:

• Authentication, done by refreshing an OAuth2 access token periodically
• The Cloud Monitoring API set as the Prometheus server URL
• The HTTP method set to GET
• The Prometheus type and version set to a minimum of 2.40.x
• The HTTP and Query timeout values set to 2 minutes

The data source syncer must run repeatedly. Asservice account access tokens have a default lifetime of one hour,running the data source syncer every 10 minutes ensures you have anuninterrupted authenticated connection between Grafana and theCloud Monitoring API.

You can choose to run the data source syncer either by using a KubernetesCronJob or by using Cloud Run and Cloud Scheduler for a fullyserverless experience. If you are deploying Grafana locally such as withopen-source Grafana or Grafana Enterprise, we recommend running the data sourcesyncer in the same cluster where Grafana is running. If you are using GrafanaCloud, we recommend choosing the fully serverless option.

Run queries by using Grafana

You can now create Grafana dashboards and run queries using the configured datasource. The following screenshot shows a Grafana chart that displays theupmetric:

For information about queryingGoogle Cloud system metrics using PromQL, seePromQL forCloud Monitoring metrics.

Running the data source syncer outside of GKE

If you are running the data source syncer in a Google Kubernetes Engine cluster or ifyou are using the serverless option, then you can skip this section.If you are having authentication issues on GKE, seeVerify service account credentials.

When running on GKE, the data source syncerautomatically retrieves credentials from the environment based on thenode's service account or the Workload Identity Federation for GKE setup.In non-GKE Kubernetes clusters, credentials must be explicitlyprovided to the data source syncer by using theGOOGLE_APPLICATION_CREDENTIALS environment variable.

1. Set the context to your target project:

   gcloud config set projectPROJECT_ID

2. Create a service account:

   gcloud iam service-accounts creategmp-test-sa

   This step creates the service account that you might havealready created in theWorkload Identity Federation for GKE instructions.

3. Grant the required permissions to the service account:

   gcloud projects add-iam-policy-bindingPROJECT_ID \  --member=serviceAccount:gmp-test-sa@PROJECT_ID.iam.gserviceaccount.com \  --role=roles/monitoring.viewer \&& \gcloud projects add-iam-policy-bindingPROJECT_ID \  --member=serviceAccount:gmp-test-sa@PROJECT_ID.iam.gserviceaccount.com \  --role=roles/iam.serviceAccountTokenCreator

4. Create and download a key for the service account:

   gcloud iam service-accounts keys creategmp-test-sa-key.json \  --iam-account=gmp-test-sa@PROJECT_ID.iam.gserviceaccount.com

5. Set the key-file path by using theGOOGLE_APPLICATION_CREDENTIALSenvironment variable.

Authorize the data source syncer to get multi-project monitoring

Managed Service for Prometheus supports multi-project monitoring by usingmetrics scopes.

For those using the serverless option, you get multi-project querying if yourchosen project is the scoping project of a multi-project metric scope.

For those deploying the data source syncer on Kubernetes, if your local projectis your scoping project,and you have followed theinstructions for verifying or configuring a serviceaccount for the local project, then multi-project queryingshould work with no further configuration.

If your local project is not your scoping project, then you need to authorizeeither the local project'sdefault compute service account oryourWorkload Identity Federation for GKE service account to havemonitoring.viewer access to the scoping project. Then pass in the scopingproject's ID as the value of thePROJECT_ID environmentvariable.

If you use the Compute Enginedefault service account,you can do one of the following:

• Deploy the data source syncer in a cluster that belongs to your scopingproject.

• Enable Workload Identity Federation for GKE for your cluster and follow theconfiguration steps.

• Provide anexplicit service-account key.

To grant a service account the permissions needed to access adifferent Google Cloud project, do the following:

1. Grant the service account permission to read from the target project youwant to query:

   gcloud projects add-iam-policy-bindingSCOPING_PROJECT_ID \  --member=serviceAccount:gmp-test-sa@PROJECT_ID.iam.gserviceaccount.com \  --role=roles/monitoring.viewer

2. When configuring the data source syncer, pass in the scoping project's IDas the value of thePROJECT_ID environmentvariable.

Inspect the Kubernetes CronJob

If you are deploying the data source syncer on Kubernetes, you can inspect theCronJob and ensure that all variables are correctly set by running thefollowing command:

kubectl describe cronjob datasource-syncer

To see logs for the Job that initially configures Grafana, run the followingcommand immediately after applying thedatasource-syncer.yaml file:

kubectl logs job.batch/datasource-syncer-init

Teardown

To disable the data source syncer Cronjob on Kubernetes, run the followingcommand:

kubectl delete -f https://raw.githubusercontent.com/GoogleCloudPlatform/prometheus-engine/v0.17.2/cmd/datasource-syncer/datasource-syncer.yaml

Disabling the data source syncer stops updating the linked Grafana withfresh authentication credentials, and as a consequence queryingManaged Service for Prometheus no longer works.

API compatibility

The followingPrometheus HTTP API endpoints aresupported by Managed Service for Prometheus under the URL prefixed byhttps://monitoring.googleapis.com/v1/projects/PROJECT_ID/location/global/prometheus/api/v1/.

For full documentation, see theCloud Monitoring API referencedocumentation. The Promethus HTTP endpoints aren'tavailable in the Cloud Monitoring language-specific client libraries.

For information about PromQL compatibility, seePromQLsupport.

• The following endpoints are fully supported:

  • /api/v1/query
  • /api/v1/query_range
  • /api/v1/metadata
  • /api/v1/labels
  • /api/v1/query_exemplars

• The/api/v1/label/<label_name>/valuesendpoint only works if the__name__label is provided either by using it as the<label_name> value or by exactlymatching on it using a series selector. For example, the followingcalls are fully supported:

  • /api/v1/label/__name__/values
  • /api/v1/label/__name__/values?match[]={__name__=~".*metricname.*"}
  • /api/v1/label/labelname/values?match[]={__name__="metricname"}

  This limitation causeslabel_values($label) variable queries in Grafana tofail. Instead, you can uselabel_values($metric, $label). This type of queryis recommended because it avoids fetching values for labels on metrics thatare not relevant to the given dashboard.

• The/api/v1/series endpoint is supported forGETbut notPOST requests.When you use thedata source syncer orfrontendproxy, this restriction is managed for you. You can alsoconfigure your Prometheus data sources in Grafana to issue onlyGETrequests. Thematch[] parameter does not support regular expression matchingon the__name__ label.

What's next

• UsePromQL alerts in Cloud Monitoring.
• Set upmanaged rule evaluation.
• Set upcommonly used exporters.