You can save time and effort configuring and maintaining monitoringapplications running on Google Kubernetes Engine (GKE) by enablingautomatic application monitoring for supported workloads.

How automatic application monitoring works

When you enable automatic application monitoring, GKE detectsdeployed instances ofsupported workloadsand deploysPodMonitoring resourcesfor each detected workload instance.

Automatic application monitoring also installs out-of-the-box dashboards formonitoring applications. The metrics are collected byGoogle Cloud Managed Service for Prometheus.

Supported workloads

Automatic application monitoring supports the following workloads:

• Apache Airflow
• Istio
• RabbitMQ
• AI model servers:
  • JetStream
  • NVIDIA Triton
  • TensorFlow Serving
  • Text Generation Inference
  • TorchServe
  • vLLM

The metrics scraping interval for AI model servers is 15s. For non-AI workloads,the scraping interval is 30s.

For more information about other out-of-the-box observability solutions that youcan configure manually, seeIntroduction to exporter configurationin the Google Cloud Observability documentation.

Before you begin

Before you start, make sure that you have performed the following tasks:

• Enable the Google Kubernetes Engine API.
Enable Google Kubernetes Engine API
• If you want to use the Google Cloud CLI for this task,install and theninitialize the gcloud CLI. If you previously installed the gcloud CLI, get the latest version by running thegcloud components update command. Earlier gcloud CLI versions might not support running the commands in this document.Note: For existing gcloud CLI installations, make sure to set thecompute/regionproperty. If you use primarily zonal clusters, set thecompute/zone instead. By setting a default location, you can avoid errors in the gcloud CLI like the following:One of [--zone, --region] must be supplied: Please specify location. You might need to specify the location in certain commands if the location of your cluster differs from the default that you set.

• Enable theCloud Monitoring API.

Requirements for automatic application monitoring

To enable automatic application monitoring, your GKE clustermust meet the following requirements:

• Your cluster must be running GKE version 1.28 or later.
• You must have gcloud CLI version 492.0.0 or later.
• Your cluster must haveGoogle Cloud Managed Service for Prometheus managed collectionenabled. Google Cloud Managed Service for Prometheus managed collection is enabled bydefault for new clusters.

Enable automatic application monitoring

You can enable automatic application monitoring for new or existing GKEclusters using the Google Cloud console, the Google Cloud CLI or theGKE API.

Metrics scraping takes up to 10 minutes to start forsupported workloads after you deploy a newworkload or after you enable automatic application monitoring for a cluster.

gcloudcontainerclusterscreate-autoCLUSTER_NAME\--location=COMPUTE_LOCATION\--auto-monitoring-scope=ALL

POSThttps://container.googleapis.com/v1/projects/PROJECT_ID/locations/COMPUTE_LOCATION/clusters{"cluster":{"name":"CLUSTER_NAME","initialNodeCount":1,"monitoringConfig":{"managedPrometheusConfig":{"enabled":true,"autoMonitoringConfig":{"scope":"ALL"}}}}}

Determine if automatic application monitoring is enabled

You can determine if automatic application monitoring is enabled or disabledfor a cluster using the Google Cloud console, the gcloud CLI, orthe GKE API.

gcloudcontainerclustersdescribeCLUSTER_NAME\--location=COMPUTE_LOCATION\--format='value(monitoringConfig.managedPrometheusConfig.autoMonitoringConfig.scope)'

autoMonitoringConfig.scope: ALL

GEThttps://container.googleapis.com/v1/projects/PROJECT_ID/locations/COMPUTE_LOCATION/clusters/CLUSTER_NAME

autoMonitoringConfig.scope: ALL

View dashboards

When you enable automatic application monitoring, GKE installsout-of-the-box dashboards for monitoring applications forsupported workloads that are deployed to your cluster.You won't see dashboards for supported workloads that have never run on acluster.

Dashboards with telemetry from automatically monitored workloads

To view the dashboards with telemetry from automatically monitored workloads inthe Google Kubernetes Engine page, do the following:

1. Go to theGoogle Kubernetes Engine page in the Google Cloud console:

   Go to Google Kubernetes Engine

2. Click the name of the cluster where you want to view dashboards forautomatically monitored workloads.

3. Click theObservability tab.TheIntegrations section shows configured dashboards forsupported workloads running on the cluster.

To view the dashboards with telemetry from automatically monitored workloads inthe Cloud Monitoring page, do the following:

1. Go to theDashboards page.

   Go to Dashboards

2. Click theDashboard List tab.

3. Select theIntegrations category.

4. Click the name of the dashboard. For example,RabbitMQ Prometheus Overview.

Dashboards for AI/ML observability metrics

To view dashboards for AI/ML observability metrics, do the following:

1. In the Google Cloud console, go to theDeployed Models page:

   Go to Deployed Models

2. To view details about a specific deployment, including its metrics, logs,and dashboards, click the model name in the list.

3. To view the observability dashboards, click theObservability tab. Ifprompted, clickEnable to enable metricscollection for the cluster.

   • TheInfrastructure usage dashboard displays utilization metrics.
   • TheDCGM dashboard displays NVIDIA Data Center GPU Manager (DCGM) metrics.
   • If the deployment uses vLLM, then theModel performance dashboard isavailable and displays metrics for vLLM model performance.

Disable automatic application monitoring

If you disable automatic application monitoring on a cluster, thePodMonitoring resources that GKE created remain intact andGKE continues to monitor existingsupported workloads. GKE stopsautomatically monitoring new instances of supported workloads that you deployto the cluster. To stop monitoring existing supported workloads, you mustdelete thePodMonitoring resourcesthat GKE created. If you re-enable automatic applicationmonitoring for a cluster, GKE detects and reconciles thePodMonitoring resources that it previously created.

You can disable automatic application monitoring for a cluster using theGoogle Cloud console, the gcloud CLI, or theGKE API.

gcloudcontainerclustersupdateCLUSTER_NAME\--location=COMPUTE_LOCATION\--auto-monitoring-scope=NONE

PUThttps://container.googleapis.com/v1/projects/PROJECT_ID/locations/COMPUTE_LOCATION/clusters/CLUSTER_NAME{"update":{"desiredMonitoringConfig":{"managedPrometheusConfig":{"autoMonitoringConfig":{"scope":"NONE"}}}}}

Customize automatic monitoring of supported workloads

You can customize the monitoring configuration for individual instancesofsupported workloads or opt out of monitoringindividual instances of supported workloads.

To customize thePodMonitoring resource for a deployed instance of asupported workload without impacting the automatic application monitoring of otherworkloads, you mustexclude the instance fromautomatic application monitoring and then update the configuration or replace thePodMonitoring resource associated with the target workload instance.

If you update or delete aPodMonitoring resource that GKEcreated without excluding the associated workload, GKE restoresthePodMonitoring resource to ensure continuity while monitoring othersupported workloads.

Find thePodMonitoring resource for a workload

You can identify thePodMonitoring resource that GKE createdfor a workload using the Google Cloud console:

1. Go to theObject browser page.

   Go to Object browser

2. In theCluster field, enter the name of the cluster.

3. In theNamespace field, enter the name of the namespace.

4. In theObject Kinds filter, selectPodMonitoring.

5. Undermonitoring.googleapis.com, selectPodMonitoring.

6. Click an object name to inspect the object.The metadata labelsource:gke-auto-monitoring indicates thatGKE created thePodMonitoring resource.

Exclude deployed instances from automatic monitoring

If you don't want to monitor a deployed instance of asupported workload, or if you want to customize themonitoring configuration of a workload, you can exclude the workload fromautomatic application monitoring.

To exclude a workload, you must add the metadata labelallow-gke-auto-monitoring: false to the target workload configuration.For example, the following manifest describes a RabbitMQ StatefulSet thatis not monitored by automatic application monitoring:

apiVersion:apps/v1kind:StatefulSetmetadata:name:rabbitmqlabels:allow-gke-auto-monitoring:"false"spec:serviceName:rabbitmqreplicas:3

After you exclude a workload instance, you can delete thePodMonitoringresource that GKE created for the workload. GKEdoes not attempt to restore thePodMonitoring resource if you delete ormodify it. If you deploy a new workload with the metadata labelallow-gke-auto-monitoring: false, GKE does not create aPodMonitoring resource for the workload.

Delete the monitoring configuration for a workload

If you want to stop monitoring a workload or if you want to deploy your owncustomizedPodMonitoring resource for a workload, you can delete themonitoring configuration.

To delete the automatic application monitoring configuration of an individual workload,do the following:

1. Exclude the workload from automatic application monitoring.
2. Identify thePodMonitoring resource thatGKE created for the workload.

3. Delete thePodMonitoring resource:

   kubectldeletepodmonitoringPOD_MONITORING_NAME-nNAMESPACE

   Replace the following:

   • POD_MONITORING_NAME: the name of thePodMonitoring resource.
   • NAMESPACE: the namespace of thePodMonitoringresource.

To delete the automatic application monitoring configuration of all workloads in acluster, do the following:

1. Disable automatic application monitoring.

2. Delete allPodMonitoring resources in the cluster with the metadata labelsource:gke-auto-monitoring:

   kubectldeletepodmonitorings-lsource=gke-auto-monitoring--all-namespaces

Pricing

The automatic application monitoring configuration and installation of out-of-the-boxdashboards for supported workloads by automatic application monitoring isavailable at no additional charge. However, Monitoring chargesfor the ingestion of metrics by Google Cloud Managed Service for Prometheus according toMonitoring pricing.

What's next

• Learn how toView observability metrics.