Resource metrics pipeline

For Kubernetes, theMetrics API offers a basic set of metrics to support automatic scaling andsimilar use cases. This API makes information available about resource usage for node and pod,including metrics for CPU and memory. If you deploy the Metrics API into your cluster, clients ofthe Kubernetes API can then query for this information, and you can use Kubernetes' access controlmechanisms to manage permissions to do so.

TheHorizontalPodAutoscaler (HPA) andVerticalPodAutoscaler (VPA)use data from the metrics API to adjust workload replicas and resources to meet customer demand.

You can also view the resource metrics using thekubectl topcommand.

Figure 1 illustrates the architecture of the resource metrics pipeline.

Figure 1. Resource Metrics Pipeline

The architecture components, from right to left in the figure, consist of the following:

• cAdvisor: Daemon for collecting, aggregating and exposingcontainer metrics included in Kubelet.

• kubelet: Node agent for managing containerresources. Resource metrics are accessible using the/metrics/resource and/stats kubeletAPI endpoints.

• node level resource metrics: API provided by the kubelet for discovering and retrievingper-node summarized stats available through the/metrics/resource endpoint.

• metrics-server: Cluster addon component that collects and aggregates resourcemetrics pulled from each kubelet. The API server serves Metrics API for use by HPA, VPA, and bythekubectl top command. Metrics Server is a reference implementation of the Metrics API.

• Metrics API: Kubernetes API supporting access to CPU and memory used forworkload autoscaling. To make this work in your cluster, you need an API extension server thatprovides the Metrics API.

  Note:

  cAdvisor supports reading metrics from cgroups, which works with typical container runtimes on Linux.If you use a container runtime that uses another resource isolation mechanism, for examplevirtualization, then that container runtime must supportCRI Container Metricsin order for metrics to be available to the kubelet.

Metrics API

The metrics-server implements the Metrics API. This API allows you to access CPU and memory usagefor the nodes and pods in your cluster. Its primary role is to feed resource usage metrics to K8sautoscaler components.

Here is an example of the Metrics API request for aminikube node piped throughjq for easierreading:

kubectl get --raw"/apis/metrics.k8s.io/v1beta1/nodes/minikube" | jq'.'

Here is the same API call usingcurl:

curl http://localhost:8080/apis/metrics.k8s.io/v1beta1/nodes/minikube

Sample response:

{"kind":"NodeMetrics","apiVersion":"metrics.k8s.io/v1beta1","metadata": {"name":"minikube","selfLink":"/apis/metrics.k8s.io/v1beta1/nodes/minikube","creationTimestamp":"2022-01-27T18:48:43Z"  },"timestamp":"2022-01-27T18:48:33Z","window":"30s","usage": {"cpu":"487558164n","memory":"732212Ki"  }}

Here is an example of the Metrics API request for akube-scheduler-minikube pod contained in thekube-system namespace and piped throughjq for easier reading:

kubectl get --raw"/apis/metrics.k8s.io/v1beta1/namespaces/kube-system/pods/kube-scheduler-minikube" | jq'.'

Here is the same API call usingcurl:

curl http://localhost:8080/apis/metrics.k8s.io/v1beta1/namespaces/kube-system/pods/kube-scheduler-minikube

Sample response:

{"kind":"PodMetrics","apiVersion":"metrics.k8s.io/v1beta1","metadata": {"name":"kube-scheduler-minikube","namespace":"kube-system","selfLink":"/apis/metrics.k8s.io/v1beta1/namespaces/kube-system/pods/kube-scheduler-minikube","creationTimestamp":"2022-01-27T19:25:00Z"  },"timestamp":"2022-01-27T19:24:31Z","window":"30s","containers": [    {"name":"kube-scheduler","usage": {"cpu":"9559630n","memory":"22244Ki"      }    }  ]}

The Metrics API is defined in thek8s.io/metricsrepository. You must enable theAPI aggregation layerand register anAPIServicefor themetrics.k8s.io API.

To learn more about the Metrics API, seeresource metrics API design,themetrics-server repository and theresource metrics API.

Measuring resource usage

CPU

CPU is reported as the average core usage measured in cpu units. One cpu, in Kubernetes, isequivalent to 1 vCPU/Core for cloud providers, and 1 hyper-thread on bare-metal Intel processors.

This value is derived by taking a rate over a cumulative CPU counter provided by the kernel (inboth Linux and Windows kernels). The time window used to calculate CPU is shown under window fieldin Metrics API.

To learn more about how Kubernetes allocates and measures CPU resources, seemeaning of CPU.

Memory

Memory is reported as the working set, measured in bytes, at the instant the metric was collected.

In an ideal world, the "working set" is the amount of memory in-use that cannot be freed undermemory pressure. However, calculation of the working set varies by host OS, and generally makesheavy use of heuristics to produce an estimate.

The Kubernetes model for a container's working set expects that the container runtime countsanonymous memory associated with the container in question. The working set metric typically alsoincludes some cached (file-backed) memory, because the host OS cannot always reclaim pages.

To learn more about how Kubernetes allocates and measures memory resources, seemeaning of memory.

Metrics Server

The metrics-server fetches resource metrics from the kubelets and exposes them in the KubernetesAPI server through the Metrics API for use by the HPA and VPA. You can also view these metricsusing thekubectl top command.

The metrics-server uses the Kubernetes API to track nodes and pods in your cluster. Themetrics-server queries each node over HTTP to fetch metrics. The metrics-server also builds aninternal view of pod metadata, and keeps a cache of pod health. That cached pod health informationis available via the extension API that the metrics-server makes available.

For example with an HPA query, the metrics-server needs to identify which pods fulfill the labelselectors in the deployment.

The metrics-server calls thekubelet APIto collect metrics from each node. Depending on the metrics-server version it uses:

• Metrics resource endpoint/metrics/resource in version v0.6.0+ or
• Summary API endpoint/stats/summary in older versions

What's next

To learn more about the metrics-server, see themetrics-server repository.

You can also check out the following:

• metrics-server design
• metrics-server FAQ
• metrics-server known issues
• metrics-server releases
• Horizontal Pod Autoscaling

To learn about how the kubelet serves node metrics, and how you can access those viathe Kubernetes API, readNode Metrics Data.