This page describes how to configure a Google Kubernetes Engine (GKE) cluster tosend a curated set of cAdvisor/Kubelet metrics toCloud Monitoring using Google Cloud Managed Service for Prometheus. This pagealso describes how these metrics are formatted when they are written toMonitoring, and how to query metrics.

The curated set of metrics is a subset of the set ofcAdvisor/Kubelet metrics built into every Kubernetes deploymentby default and includes metrics related to reducing ingestion volume andassociated costs.

Before you begin

Before you start, make sure that you have performed the following tasks:

• Enable the Google Kubernetes Engine API.
Enable Google Kubernetes Engine API
• If you want to use the Google Cloud CLI for this task,install and theninitialize the gcloud CLI. If you previously installed the gcloud CLI, get the latest version by running thegcloud components update command. Earlier gcloud CLI versions might not support running the commands in this document.Note: For existing gcloud CLI installations, make sure to set thecompute/regionproperty. If you use primarily zonal clusters, set thecompute/zone instead. By setting a default location, you can avoid errors in the gcloud CLI like the following:One of [--zone, --region] must be supplied: Please specify location. You might need to specify the location in certain commands if the location of your cluster differs from the default that you set.

Requirements

To collect cAdvisor/Kubelet metrics, your GKEcluster must meet the following requirements:

• The cluster must be running GKE version1.29.3-gke.1093000 or later. cAdvisor/Kubeletpackages are enabled by default when you create a cluster running version1.29.3-gke.1093000 or later.
• The cluster must havesystem metricsenabled.
• The cluster hasGoogle Cloud Managed Service for Prometheus managed collectionenabled. Google Cloud Managed Service for Prometheus managed collectionis enabled by default for new clusters.

gcloudcontainerclustersupdateCLUSTER_NAME\--location=COMPUTE_LOCATION\--enable-managed-prometheus\--monitoring=SYSTEM,CADVISOR,KUBELET

Metric format

All Kubernetes cAdvisor/Kubelet metrics written to Cloud Monitoringuse the resource typeprometheus_target.Each metric name is prefixed withprometheus.googleapis.com/ and has a suffix indicating thePrometheus metric type, such as/gauge,/histogram,or/counter. Otherwise, each metric name isidentical to the metric name exposed by open source Kubernetes.

Exporting from Cloud Monitoring

The cAdvisor/Kubelet metrics can be exported from Cloud Monitoring byusing theCloud Monitoring API.Because all cAdvisor/Kubelet metrics are ingested by usingGoogle Cloud Managed Service for Prometheus,cAdvisor/Kubelet metrics can be queriedby using Prometheus Query Language (PromQL). They can also be queried byusingby using Monitoring Query Language (MQL).

Querying metrics

When you query cAdvisor/Kubelet metrics, the name you use depends onwhether you are using PromQL or Cloud Monitoring-based features likeMQL or the Metrics Explorermenu-driven interface.

The following tables of cAdvisor/Kubelet metrics show two versions of each metric name:

• PromQL metric name: Whenusing PromQL in Cloud Monitoring pages of the Google Cloud console or in PromQL fields of theCloud Monitoring API, use the PromQL metric name.
• Cloud Monitoring metric name When using other Cloud Monitoring features, use the Cloud Monitoring metric name in the tables below. This name must be prefixed withprometheus.googleapis.com/, which has been omitted from the entries in the table.

cAdvisor metrics

The Cloud Monitoring metric names in this table must be prefixed withprometheus.googleapis.com/. That prefix has been omitted from theentries in the table.

Kubelet metrics

The Cloud Monitoring metric names in this table must be prefixed withprometheus.googleapis.com/. That prefix has been omitted from theentries in the table.