Admin tasks and tools
Toolforge components
Packaging
Runbooks and emergency guides
Other
[edit]
This happens when prometheus has no data from k8s on the builds-api pod.
This usually comes in the form of analert in alertmanager.
There you will get which project (tools, toolsbeta, ...) is the one it's failing for.
This is a tricky one and it will be related to the way we gather metrics on tools/toolsbeta.
Note that this is not directly related to themetricsinfra monitoring project, but toolforge's own setup.
You can start by going to the project's prometheus page and trying to get the stats there, example for tools:
Add new issues here when you encounter them!
If build-api seems up, you can check if the certificates that prometheus uses to connect to k8s have expired (there should have been another alert though)Portal:Toolforge/Admin/Runbooks/PrometheusK8sCertExpirySoon.
project=tools orproject=toolsbeta for filtering)Add any incident tasks here!