Uh oh!
There was an error while loading.Please reload this page.
- Notifications
You must be signed in to change notification settings - Fork8.1k
feat(CI): add Codecov Test Analytics for flaky and failed tests#29881
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to ourterms of service andprivacy statement. We’ll occasionally send you account related emails.
Already on GitHub?Sign in to your account
base:main
Are you sure you want to change the base?
feat(CI): add Codecov Test Analytics for flaky and failed tests#29881
Uh oh!
There was an error while loading.Please reload this page.
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
Thank you for opening your first PR into Matplotlib!
If you have not heard from us in a week or so, please leave a new comment below and that should bring it to our attention. Most of our reviewers are volunteers and sometimes things fall through the cracks.
You can also join uson gitter for real-time discussion.
For details on testing, writing docs, and our review process, please seethe developer guide
We strive to be a welcoming and open project. Please follow ourCode of Conduct.
ps.: the action that uploads the test results doesn't have a great timedifferentiating 'linux' from 'linux-arm64', so I had to add the matrixvalues for the `os:` option
fa88a74 to3c13d7fCompare
QuLogic left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
I guess the results are here:https://app.codecov.io/gh/matplotlib/matplotlib/tests/giovanni-guidini%3Agio%2Fcodecov-test-analytics
But it doesn't say too much since it's just the one commit, and there are no trends to speak of.
Uh oh!
There was an error while loading.Please reload this page.
41ad7d5 to887c1e9Compare
dstansby left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
This looks like a nice new feature, and I'd definitely be 👍 to trying it out.
The other comment I have, is would you recommend us adding any tags? The codecov docs seem to recommend the Python version, so perhaps we should add that as a tag? We have definitely had tests fail on certain Python versions before, so I think adding that tag could be helpful.
.github/workflows/tests.yml Outdated
| uses:codecov/test-results-action@v1 | ||
| with: | ||
| token:${{ secrets.CODECOV_TOKEN }} | ||
| os:${{ matrix.codecov-test-results-action-os }} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
Is there a reason we need to manually specify the os - the docs say "Override the assumed OS.", so I would have assumed that the os can be autodetected somehow? I was thinking of saving having to hard code the os as a variable for every element of the test matrix.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
I completely agree with you there, it would be better to rely on the autodetect. It works mostly fine, but it's not perfect.
Originally I was relying on that, butthis run shows that specifically theubuntu-22.04-arm is being detected as "linux" not "linux-arm64". Hopefully this will be addressed in the Action's side, of course, but in the mean time using theos var can side step this limitation and make sure all uploads are being done correctly.
An alternative approach that I didn't think would work is to conditionally pass theos var only in that one case that would need it. This would be a much smaller footprint in the yaml. I can try that approach.
giovanni-guidini commentedApr 10, 2025
We appreciate the interest in this feature! About the flags, I think the python version is indeed a good candidate for flags. The |
dstansby commentedApr 13, 2025
Can you explain where the Python tag can be used on codecov to filter results (or for other functionality)? I tried looking athttps://app.codecov.io/gh/matplotlib/matplotlib/commit/bc70afb98830826cfd18ee24dc185bcc1af16a66 andhttps://app.codecov.io/gh/matplotlib/matplotlib/tests/giovanni-guidini%3Agio%2Fcodecov-test-analytics, but I can't obviously see where the tags show up. |
giovanni-guidini commentedApr 14, 2025 • edited
Loading Uh oh!
There was an error while loading.Please reload this page.
edited
Uh oh!
There was an error while loading.Please reload this page.
Good observation. I've confirmed with the team that at this time you can only filter the test results by flag in the The team did say they are considering expanding filtering to any branch, but I won't make promises, nor do I have a timeframe for such feature. Sorry :E |
PR summary
Hi, Gio from Codecov here. I saw that you used Codecov already, and I thought you could benefit from catching any flaky or failed tests. We released aTest Analytics product that might be helpful and would love your feedback.
Changes:
PR checklist