Reacting to issues after they occur can lead to downtime. To maintain aresilient system in Google Kubernetes Engine (GKE), you need to identifypotential problemsbefore they affect your users.

Use this page to proactively monitor your GKE environment withCloud Monitoring by tracking key performance indicators, visualizing trends,and setting up alerts to detect issues like rising error rates or resourceconstraints.

This information is important for Platform admins and operators responsible forensuring the health, reliability, and efficiency of the GKEenvironment. It also helps Application developers understand theirapp's performance in real-world conditions, detect regressions acrossdeployments, and gain insights for optimization. For more information about thecommon roles and example tasks that we reference in Google Cloud content, seeCommon GKE user roles and tasks.

Review useful metrics

GKE automatically sends a set of metrics to Cloud Monitoring.The following sections list some of the most important metrics fortroubleshooting:

• Container performance and health metrics
• Node performance and health metrics
• Pod performance and health metrics

For a complete list of GKE metrics, seeGKE system metrics.

Container performance and health metrics

Start with these metrics when you suspect a problem with a specific app. Thesemetrics help you monitor the health of your app, including discovering if acontainer is restarting frequently, running out of memory, or being throttled byCPU limits.

Node performance and health metrics

Examine these metrics when you need to diagnose issues with the underlyingGKE infrastructure. These metrics are crucial for understandingthe overall health and capacity of your nodes, helping you investigate whetherthe node is unhealthy or under pressure, or whether the node has enough memoryto schedule new Pods.

Pod performance and health metrics

These metrics help you troubleshoot issues related to a Pod's interaction withits environment, such as networking and storage. Use these metrics when you needto diagnose slow-starting Pods, investigate potential network connectivityissues, or proactively manage storage to prevent write failures from fullvolumes.

Visualize metrics with Metrics Explorer

To visualize the state of your GKE environment, createcharts based on metrics with Metrics Explorer.

To use Metrics Explorer, complete the following steps:

1. In the Google Cloud console, go to theMetrics Explorer page.

   Go to Metrics Explorer

2. In theMetrics field, select or enter the metric that you want toinspect.

3. View the results and observe any trends over time.

For example, to investigate the memory consumption of Pods in a specificnamespace, you can do the following:

1. In theSelect a metric list, choose the metrickubernetes.io/container/memory/used_bytes and clickApply.
2. ClickAdd filter and selectnamespace_name.
3. In theValue list, select the namespace you want to investigate.
4. In theAggregation field, selectSum >pod_name and clickOK. This setting displays a separate time series line for each Pod.
5. ClickSave chart.

The resulting chart shows you the memory usage for each Pod over time, which canhelp you visually identify any Pods with unusually high or spiking memoryconsumption.

Metrics Explorer has a great deal of flexibility in how to construct themetrics that you want to view. For more information about advancedMetrics Explorer options, seeCreate charts with Metrics Explorerin the Cloud Monitoring documentation.

Create alerts for proactive issue detection

To receive notifications when things go wrong or when metrics breach certainthresholds, set up alerting policies in Cloud Monitoring.

For example, to set up an alerting policy that notifies you when the containerCPU limit is over 80% for five minutes, do the following:

1. In the Google Cloud console, go to theAlerting page.

   Go to Alerting

2. ClickCreate policy.

3. In theSelect a metric box, filter forCPU limit utilizationand then select the following metric:kubernetes.io/container/cpu/limit_utilization.

4. ClickApply.

5. Leave theAdd a filter field blank. This setting triggers an alertwhen any cluster violates your threshold.

6. In theTransform data section, do the following:

   1. In theRolling window list, select1 minute. Thissetting means that Google Cloud calculates an average value every minute.

   2. In theRolling window function list, selectmean.

      Both of these settings average the CPU limit utilization for eachcontainer every minute.

7. ClickNext.

8. In theConfigure alert section, do the following:

   1. ForCondition type, selectThreshold.
   2. ForAlert trigger, selectAny time series violates.
   3. ForThreshold position, selectAbove threshold.
   4. ForThreshold value, enter0.8. This value represents the80% threshold that you want to monitor for.
   5. ClickAdvanced options.
   6. In theRetest window list, select5 min. This setting meansthat the alert triggers only if the CPU utilization stays over 80%for a continuous five-minute period, which reduces false alarms frombrief spikes.
   7. In theCondition name field, give the condition a descriptive name.
   8. ClickNext.

9. In theConfigure the notifications and finalize the alert section,do the following:

   1. In theNotification channels list, select the channel whereyou want to receive the alert. If you don't have a channel, clickManage notification channels to create one.
   2. In theName the alert policy field, give the policy a clearand descriptive name.
   3. Leave all other fields with their default values.
   4. ClickNext.

10. Review your policy, and if it all looks correct, clickCreate policy.

To learn about the additional ways that you can create alerts, seeAlertingoverview in the Cloud Monitoring documentation.

What's next

• ReadAccelerate diagnosis with Gemini Cloud Assist(the next page in this series).

• See these concepts applied in theexample troubleshooting scenario.

• For advice about resolving specific problems, reviewGKE's troubleshooting guides.

• If you can't find a solution to your problem in the documentation, seeGet support for further help,including advice on the following topics:

  • Opening a support case by contactingCloud Customer Care.
  • Getting support from the community byasking questions on StackOverflow and using thegoogle-kubernetes-engine tag to search for similarissues. You can also join the#kubernetes-engine Slack channel for more community support.
  • Opening bugs or feature requests by using thepublic issue tracker.