- Notifications
You must be signed in to change notification settings - Fork3.6k
Concurrent safe writes#980
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to ourterms of service andprivacy statement. We’ll occasionally send you account related emails.
Already on GitHub?Sign in to your account
base:main
Are you sure you want to change the base?
Conversation
ghost commentedMar 10, 2025
The PR does not enable concurrent writes. This PR replaces the best-effort race detector with a mutex. The scope of the state that must be protected by a mutex is greater than the scope of the race detector. The function TestConcurrentWrites doesnot call WriteMessage concurrently. The tests will fail under the race detector when the test is modified to allow concurrent calls. There is discussion in issues on why the Conn methods are not not amenable to allowing concurrent writes. This PR does not address any of those problems. There are other problems with the PR, but I will not discuss them here because the points above are fatal. |
Ale-Cas commentedMar 10, 2025 • edited
Loading Uh oh!
There was an error while loading.Please reload this page.
edited
Uh oh!
There was an error while loading.Please reload this page.
Hi @hulkingshtick, you're right this PR is not aiming to enable concurrent writes on the connection, the purpose was just to enable thread-safety in write functions using a mutex, and to remove the Maybe I misunderstood your commenthere:
Isn't this what you had in mind?
If there are other problems, I'm open to feedback and happy to improve the PR. Otherwise if you still think it does not bring any value I'll close it up. |
ghost commentedMar 10, 2025
The concurrency issue is higher in the call stack. Any fix for the problem will not require a mutex in
This panic is useful because the panic message explicitly tells the programmer what the problem is. The alternative is to panic in less obvious ways or to corrupt data.
This PR does not relax theconcurrency requirements because the PR does not fix the concurrent write problem. Update the your test to call WriteMessage concurrently and
I said that mutual exclusion lock in
|
ghost commentedMar 11, 2025 • edited by ghost
Loading Uh oh!
There was an error while loading.Please reload this page.
edited by ghost
Uh oh!
There was an error while loading.Please reload this page.
This test below fails when run with This test starts all goroutines before waiting. The test in the PR waits on the previously started goroutine before starting the next goroutine. |
Ale-Cas commentedMar 11, 2025
I understand now, thank you very much for clarifying, I'm working on it and will follow your suggestions |
Ale-Cas commentedMar 11, 2025 • edited
Loading Uh oh!
There was an error while loading.Please reload this page.
edited
Uh oh!
There was an error while loading.Please reload this page.
Btw if you're referring to the changes in the files other than conn/conn_test, they've been included here just because I run |
frankjkelly commentedMar 12, 2025
Very interested in this . . . . . following. . . . |
ghost commentedMar 12, 2025
The following test fails when run with the race detector. There is discussion in issues on why the Conn methods are not not amenable to allowing concurrent writes. This PR does not address any of those problems. |
Ale-Cas commentedMar 15, 2025
@hulkingshtick Thank you very much for the review and for providing that failing test! Looking again at both your comment and the code, I agree that there's no good way to make NextWriter thread-safe. Still, I'm wondering if making the WriteMessage and WriteJSON methods thread-safe would reduce issues related to concurrent writes. An alternative,as you suggested would be to add a new package that encapsulates this common use case, which would provide developers with a robust higher-level API. Based on the outcome of this discussion, I will either update or close this PR. |
ghost commentedMar 16, 2025
Those methods are often on the stack in issues reporting the concurrent write panic, so yes, adding a mutex will cut down on reported issues. The goal of my suggestion is to reduce the support burden for the maintainers, not to make those applications correct. I would hold off on any changes until the maintainers show that they are still interested in maintaining this package. |
liggitt commentedMar 19, 2025
If the underlying Conn won't support concurrent writes well, is it a good idea to allow concurrency in calls to higher level methods? That seems like it will invite concurrent use that will be non-deterministic and cause more problems. |
frankjkelly commentedMar 19, 2025 • edited
Loading Uh oh!
There was an error while loading.Please reload this page.
edited
Uh oh!
There was an error while loading.Please reload this page.
It would be great to have concurrency but even if we can't if there were examples that showed recommended "templates" of how to achieve concurrency (with locks? channels?) that would be great. We have a use-case where we have multiple writers and we can figure out the locks but I'm sure whatever we will figure out will likely be suboptimal given our (current) shallow understanding of concurrency / goroutines / locking versus what the great folks here on in the open source world have. Either way, appreciative of the effort here by all sides. |
ghost commentedMar 19, 2025
The current code ensures that there are no concurrent writes to the underlying net.Conn.
|
frankjkelly commentedMar 19, 2025
Thanks @hulkingshtick but maybe I am missing something? I see some |
ghost commentedMar 19, 2025
@frankjkelly The examples use goroutines and channels to ensure a single concurrent writer. This approach is the simplest way to make an application robust against slow or dead peers. |
What type of PR is this? (check all applicable)
Description
Implements concurrency-safe write functions, using a mutex, as suggested in#826 and#828
Related Tickets & Documents
Added/updated tests?
have not been included
Run verifications and test
make verifyis passingmake testis passing