- Notifications
You must be signed in to change notification settings - Fork1k
chore: add tx metrics and logs for serialization errors#15215
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to ourterms of service andprivacy statement. We’ll occasionally send you account related emails.
Already on GitHub?Sign in to your account
Merged
Uh oh!
There was an error while loading.Please reload this page.
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.Learn more about bidirectional Unicode characters
DB metrics were all or nothing. This changes that to alwaysinclude some information (tx data).
coadler approved these changesOct 25, 2024
ccfffc6 intomain 27 checks passed
Uh oh!
There was an error while loading.Please reload this page.
Sign up for freeto subscribe to this conversation on GitHub. Already have an account?Sign in.
Labels
None yet
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading.Please reload this page.
Before db_metrics were all or nothing. Now
InTxmetrics are always recorded, and query metrics are opt in.What this does
Adds instrumentation & logging around serialization failures in the database. When the database hits a serialization error, we retry the transaction up to 3 times. This mostly works, however in production we have seen all 3 retries fail.
We have 0 information on the frequency of failures if they retry once or twice. Or where they are located. Making debugging challenging.
The goal of this PR is not to solve the issue, but begin to investigate the frequency and location of the failures. This is assuming sometimes retries are successful.
Example metric of a failure.
Log line:
Prometheus
Context
For investigatinghttps://github.com/coder/customers/issues/683