Repository files navigation

STATUS: stable, maintained

cadvisor is great, but missing a few important metrics, that every serious devops person wants to know about.This is a secondary process to export all the missingPrometheus metrics:

• OOM-kill
• number of container restarts
• last exit code

This was motivated by hunting down a OOM kills in a large Kubernetes cluster.It's possible for containers to keep running, even after a OOM-kill, if asub-process got affect for example. Without this metric, it becomes much moredifficult to find the root cause of the issue.

True story; after this was deployed, a recurring OOM-kill in Fluentd wasquickly discovered on one of the nodes. It turns out that the resource limitswere set too low to process logs on that node. Logs werenot being forwarded because the Fluentd worker process kept being OOM-kill andthen restarted by the main process. A fix was then deployed 10 minutes later.

• Docker
• Containerd

Kubernetes 1.20 has deprecated Docker container runtime, so we have added support for Containerd since the version0.21.0 ofmissing-container-metrics.Both options should cover most of common use cases (EKS, GKE, K3S, Digital Ocean Kubernetes, ...).

The easiest way of installingmissing-container-metrics in your kubernetes cluster is using ourhelm chart.

$ docker run -d -p 3001:3001 -v /var/run/docker.sock:/var/run/docker.sock dmilhdef/missing-container-metrics:v0.14.0

Exposes metrics about Docker/Containerd containers.Every metric contains following labels:

Each of those metrics, are published with the labels from the next section.

Number of restarts of the container.

Number of OOM kills for the container. This covers OOM kill of any process in the container cgroup.

Last exit code of the container.

Full id of the container.

First 6 bytes of the Docker container id.

Container id represented in the same format as in metrics of kubernetes pods - prefixed withdocker:// andcontainerd:// depending on the container runtime. This enables easy joins in Prometheus tokube_pod_container_info metric.

Name of the container.

Image id represented in the same format as in metrics of k8s pod. This enables easy joins in Prometheus tokube_pod_container_info metric.

Ifio.kubernetes.pod.name label is set on the container, it's valuewill be set as thepod label in the metric

Ifio.kubernetes.pod.namespace label is set on the container, it's valuewill be set as thenamespace label of the metric.

Together withpod, this label is useful in the context of Kubernetes deployments, to determine namespace/pod to which the container is part of.One can see it as a shortcut to joining with thekube_pod_container_info metric to determine those values.

Contributions are welcome, send your issues and PRs to this repo.

MIT - Copyright Dragan Milic and contributors