- Notifications
You must be signed in to change notification settings - Fork6.3k
monitoring: make cluster matcher backward compatible for pre-reef metrics#66661
Conversation
cloudbehl left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
Ceph 18.* adds a cluster label to all Prometheus metrics. When upgrading from earlier releases, historical metrics lack this label and are excluded by Grafana queries that strictly match on cluster. Update the shared Grafana matcher logic to use a regex matcher that also matches series without the cluster label, restoring visibility of pre-upgrade metrics while preserving multi-cluster behavior.
I'm not sure why do they lack the cluster label as the cluster metric and the cluster label should be part of upgrade. Once back can you explain, what exactly happens and why?
Uh oh!
There was an error while loading.Please reload this page.
afreen23 commentedDec 23, 2025
Prometheus creates new time series on addition of a new label, so the old ones no longer match the query and appears to be missing. |
5fbbf99 to164559aCompare
cloudbehl left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
I have not tested it, but the changes look good to me!
aaSharma14 commentedJan 6, 2026
jenkins test windows |
164559a toaeeea1bCompareaaSharma14 commentedJan 7, 2026
jenkins test make check |
cloudbehl commentedJan 19, 2026
|
Ceph 18.* adds a `cluster` label to all Prometheus metrics. Whenupgrading from earlier releases, historical metrics lack this labeland are excluded by Grafana queries that strictly match on `cluster`.Update the shared Grafana matcher logic to use a regex matcher thatalso matches series without the `cluster` label, restoring visibilityof pre-upgrade metrics while preserving multi-cluster behavior.Fixes:https://tracker.ceph.com/issues/74342Signed-off-by: Aashish Sharma <aasharma@redhat.com>
aeeea1b tof7f74e4Compare
cloudbehl left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
LGTM.
af3099e intoceph:mainUh oh!
There was an error while loading.Please reload this page.
Uh oh!
There was an error while loading.Please reload this page.
Ceph 18.* adds a
clusterlabel to all Prometheus metrics. When upgrading from earlier releases, historical metrics lack this label and are excluded by Grafana queries that strictly match oncluster. Update the shared Grafana matcher logic to use a regex matcher that also matches series without theclusterlabel, restoring visibility of pre-upgrade metrics while preserving multi-cluster behavior.Fixes:https://tracker.ceph.com/issues/74342
Contribution Guidelines
To sign and title your commits, please refer toSubmitting Patches to Ceph.
If you are submitting a fix for a stable branch (e.g. "quincy"), please refer toSubmitting Patches to Ceph - Backports for the proper workflow.
When filling out the below checklist, you may click boxes directly in the GitHub web UI. When entering or editing the entire PR message in the GitHub web UI editor, you may also select a checklist item by adding an
xbetween the brackets:[x]. Spaces and capitalization matter when checking off items this way.Checklist
Show available Jenkins commands
jenkins test classic perfJenkins Job |Jenkins Job Definitionjenkins test crimson perfJenkins Job |Jenkins Job Definitionjenkins test signedJenkins Job |Jenkins Job Definitionjenkins test make checkJenkins Job |Jenkins Job Definitionjenkins test make check arm64Jenkins Job |Jenkins Job Definitionjenkins test submodulesJenkins Job |Jenkins Job Definitionjenkins test dashboardJenkins Job |Jenkins Job Definitionjenkins test dashboard cephadmJenkins Job |Jenkins Job Definitionjenkins test apiJenkins Job |Jenkins Job Definitionjenkins test docsReadTheDocs |Github Workflow Definitionjenkins test ceph-volume allJenkins Jobs |Jenkins Jobs Definitionjenkins test windowsJenkins Job |Jenkins Job Definitionjenkins test rook e2eJenkins Job |Jenkins Job DefinitionYou must only issue one Jenkins command per-comment. Jenkins does not understand
comments with more than one command.