- Notifications
You must be signed in to change notification settings - Fork433
fix(storage): gRPC misuse causing crashes due to concurrent writes from Flush() and Write()#15802
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to ourterms of service andprivacy statement. We’ll occasionally send you account related emails.
Already on GitHub?Sign in to your account
base:main
Are you sure you want to change the base?
Conversation
…om Flush() and Write()
Summary of ChangesHello@v-pratap, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request resolves a critical race condition within the gRPC async writer that could lead to application crashes. The issue arose from Highlights
Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on ourdocumentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on@gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign uphere. You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with theGemini Code Assist IDE Extension. Footnotes
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
Code Review
This pull request effectively resolves a critical race condition that could cause crashes due to concurrent writes on a gRPC stream. The solution, which involves introducing awriting_ flag for serialization and adjusting the order of asynchronous operations, is sound and well-implemented. The new test case,NoConcurrentWritesWhenFlushAndWriteRace, is particularly valuable as it specifically targets the race condition and confirms the fix. The changes significantly enhance the thread safety and reliability of the async writer. I have one minor style suggestion.
| { | ||
| std::unique_lock<std::mutex>lk(mu_); | ||
| was_finalizing = finalizing_; | ||
| was_finalizing = finalizing_; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@## main #15802 +/- ##==========================================- Coverage 92.94% 92.93% -0.01%========================================== Files 2457 2457 Lines 227128 227190 +62 ==========================================+ Hits 211099 211148 +49- Misses 16029 16042 +13 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
Uh oh!
There was an error while loading.Please reload this page.
A gRPC misuse issue can lead to crashes when using the async writer. The issue occurs due to a race condition between user code and the SDK's internal handling of Flush() operations.
Here's a breakdown of the problem:
When a Flush() is issued, the buffer is cleaned up asynchronously via OnQuery() after the flush future is marked complete. OnQuery() can then call WriteLoop(), which issues a write. However, user code can also call Write() after the Flush() future is complete, which can also call WriteLoop(). This can result in two concurrent writes to the same gRPC stream, which is against the gRPC API contract and can cause a crash.