Uh oh!
There was an error while loading.Please reload this page.
- Notifications
You must be signed in to change notification settings - Fork32.2k
gh-87135: threading.Lock: Raise rather than hang on Python finalization#135991
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to ourterms of service andprivacy statement. We’ll occasionally send you account related emails.
Already on GitHub?Sign in to your account
base:main
Are you sure you want to change the base?
Conversation
…lizationAfter Python finalization gets to the point where no other threadcan attach thread state, attempting to acquire a Python lock musthang.Raise PythonFinalizationError instead of hanging.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
I think this approach makes sense. A comment about the flags below
@@ -95,6 +95,18 @@ _PyMutex_LockTimed(PyMutex *m, PyTime_t timeout, _PyLockFlags flags) | |||
if (timeout == 0) { | |||
return PY_LOCK_FAILURE; | |||
} | |||
if ((flags & _PY_LOCK_PYTHONLOCK) && Py_IsFinalizing()) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
The first part of the condition here is checking if any of the bits of_PY_LOCK_PYTHONLOCK
are set. In other words, it will pass even if only_PY_LOCK_DETACH
is set fromPyMutex_Lock()
.
It could be:
if ((flags & _PY_LOCK_PYTHONLOCK) == _PY_LOCK_PYTHONLOCK && Py_IsFinalizing()) {
But given that this is an easy mistake to make and hard to spot, I'd rather just have a separate bit for fail-on-finalization and check that individual bit here.
If we want a constant to combine those three bits, let's keep that constant it in_threadmodule.c
.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
🤦
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
Won't this be prone to TOCTOU issues?Py_IsFinalizing
could pass, but once the thread state detaches in the parking lot, the interpreter is free to start finalizing. The parking lot will then hang the thread upon reattachment in_PySemaphore_Wait
, right?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
No - this check is only relevant for the main thread (the one that performsPy_Finalize()
) so there's no TOCTOU. There are two "hanging" threads behaviors that you may be confusing:
- Non-main threads, typically daemon threads (and occasionally other non-daemon threads), hang when they try to attach during Python finalization. This PR doesn't affect that behavior.
- The main thread, when performing finalization, may acquire
threading.Lock/RLock
called via destructors / GC. If those locks happen to be held by other (now hung) threads, then the shutdown would deadlock. This instead raises a Python exception.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
Hm, yeah, I was confusing it with that. But, you've mentioned before that acquiring Python locks in finalizers isn't safe regardless of finalization, because the GC might run in any code where the lock is held and deadlock while reacquiring it. Looking at the test case, I think that still applies. What's the use-case that this PR is fixing?
There is a suspicious error when running test.test_multiprocessing_spawn.test_processes:
|
Uh oh!
There was an error while loading.Please reload this page.
After Python finalization gets to the point where no other thread can attach thread state, attempting to acquire a Python lock will hang.
Raise PythonFinalizationError instead of hanging.
@colesbury, does this look maintainable (and correct) to you?