- Notifications
You must be signed in to change notification settings - Fork1.1k
fix(site): fix flaky Chromatic tests#20808
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to ourterms of service andprivacy statement. We’ll occasionally send you account related emails.
Already on GitHub?Sign in to your account
fix(site): fix flaky Chromatic tests#20808
Conversation
efeb716 to83c9847Compare83c9847 to0bb3de1CompareEmyrk commentedNov 17, 2025
I am also seeing |
aslilac left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
I think a better solution might be to remove this logic all together. stories are all snapshot tests, so we already know that the names are present and in the right order because they're in the screenshot. if this logic is flakey, then lets just rely on the snapshot instead.
…-orgs-sorted-alphabetically
ssncferreira commentedNov 18, 2025
@aslilac I think the test might actually make sense. It validates that the organizations are listed in the correct order: first the active organization (Omega org), then the remaining organizations alphabetically. AFAIU, if a bug is introduced where the order of the orgs changes, Chromatic will detect those visual changes in the screenshot, but it won't fail automatically, a reviewer would need to catch it manually. The programmatic assertion ensures the sorting logic works correctly and fails immediately if it breaks. This new logic fixes the flakiness of the test by querying the |
ssncferreira commentedNov 19, 2025
After discussing with@aslilac on the best practices around Chromatic tests, the focus should be on visual validation through snapshots rather than programmatic assertions. The snapshot will catch visual changes resulting from any programmatic changes during the Chromatic review process. The programmatic assertions on these tests were redundant and introduced flakiness, so removing them makes the tests more stable while still catching issues through visual diffs. |
aslilac left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
thank you!
17edeea intomainUh oh!
There was an error while loading.Please reload this page.
Uh oh!
There was an error while loading.Please reload this page.
Problem
The
OrgsSortedAlphabeticallytest fromOrganizationSidebarView.stories.tsxwas failing on Chromatic. Test logic was attempting to verify organization sorting order programmatically. This was identified in the Chromatic build of PR:https://www.chromatic.com/build?appId=624de63c6aacee003aa84340&number=26015After fixing this test, two additional tests started failing:
VanillaJavascriptErrorfromGlobalErrorBoundary.stories.tsx: Test was making incorrect assertions about stack trace contentMarkAllNotificationsAsReadErrorfromNotificationsInbox.stories.tsx: Test was flaky due to competing WebSocket error messagesSolution
These are Chromatic snapshot tests, so implementation details (like sorting order or exact error message content) are already validated by the visual snapshots. Programmatic assertions were causing flakiness and are redundant.