Uh oh!
There was an error while loading.Please reload this page.
- Notifications
You must be signed in to change notification settings - Fork33.3k
gh-137433: Fix deadlock with stop-the-world and daemon threads#137735
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to ourterms of service andprivacy statement. We’ll occasionally send you account related emails.
Already on GitHub?Sign in to your account
Merged
colesbury merged 2 commits intopython:mainfromcolesbury:gh-137433-stop-the-world-daemon-deadlockSep 16, 2025
Merged
gh-137433: Fix deadlock with stop-the-world and daemon threads#137735
colesbury merged 2 commits intopython:mainfromcolesbury:gh-137433-stop-the-world-daemon-deadlockSep 16, 2025
Uh oh!
There was an error while loading.Please reload this page.
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.Learn more about bidirectional Unicode characters
There was a deadlock originally seen by Memray when a daemon threadenabled or disabled profiling while the interpreter was shutting down.I think this could also happen with garbage collection, but I haven'tseen that in practice.The daemon thread could be hung while trying acquire the global rwmutexthat prevents overlapping global and per-interpreter stop-the-world events.Since it already held the main interpreter's stop-the-world lock, italso deadlocked the main thread, which is trying to perform interpreterfinalization.Swap the order of lock acquisition to prevent this deadlock.Additionally, refactor `_PyParkingLot_Park` so that the global bucketshashtable is left in a clean state if the thread is hung in`PyEval_AcquireThread`.
gpshead approved these changesSep 14, 2025
ZeroIntensity approved these changesSep 15, 2025
Member
ZeroIntensity left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
LGTM. I'm not a huge fan of usingPyEval_AcquireThread/PyEval_ReleaseThread over_PyThreadState_Attach/_PyThreadState_Detach, but since those were there already I'm not going to bother complaining about it.
90fe325 intopython:main 85 of 87 checks passed
Uh oh!
There was an error while loading.Please reload this page.
Thanks@colesbury for the PR 🌮🎉.. I'm working now to backport this PR to: 3.14. |
miss-islington pushed a commit to miss-islington/cpython that referenced this pull requestSep 16, 2025
…ythongh-137735)There was a deadlock originally seen by Memray when a daemon threadenabled or disabled profiling while the interpreter was shutting down.I think this could also happen with garbage collection, but I haven'tseen that in practice.The daemon thread could be hung while trying acquire the global rwmutexthat prevents overlapping global and per-interpreter stop-the-world events.Since it already held the main interpreter's stop-the-world lock, italso deadlocked the main thread, which is trying to perform interpreterfinalization.Swap the order of lock acquisition to prevent this deadlock.Additionally, refactor `_PyParkingLot_Park` so that the global bucketshashtable is left in a clean state if the thread is hung in`PyEval_AcquireThread`.(cherry picked from commit90fe325)Co-authored-by: Sam Gross <colesbury@gmail.com>
GH-138965 is a backport of this pull request to the3.14 branch. |
encukou pushed a commit that referenced this pull requestOct 7, 2025
…gh-137735) (GH-138965)There was a deadlock originally seen by Memray when a daemon threadenabled or disabled profiling while the interpreter was shutting down.I think this could also happen with garbage collection, but I haven'tseen that in practice.The daemon thread could be hung while trying acquire the global rwmutexthat prevents overlapping global and per-interpreter stop-the-world events.Since it already held the main interpreter's stop-the-world lock, italso deadlocked the main thread, which is trying to perform interpreterfinalization.Swap the order of lock acquisition to prevent this deadlock.Additionally, refactor `_PyParkingLot_Park` so that the global bucketshashtable is left in a clean state if the thread is hung in`PyEval_AcquireThread`.(cherry picked from commit90fe325)Co-authored-by: Sam Gross <colesbury@gmail.com>
Sign up for freeto join this conversation on GitHub. Already have an account?Sign in to comment
Labels
None yet
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading.Please reload this page.
There was a deadlock originally seen by Memray when a daemon thread enabled or disabled profiling while the interpreter was shutting down. I think this could also happen with garbage collection, but I haven't seen that in practice.
The daemon thread could be hung while trying acquire the global rwmutex that prevents overlapping global and per-interpreter stop-the-world events. Since it already held the main interpreter's stop-the-world lock, it also deadlocked the main thread, which is trying to perform interpreter finalization.
Swap the order of lock acquisition to prevent this deadlock. Additionally, refactor
_PyParkingLot_Parkso that the global buckets hashtable is left in a clean state if the thread is hung inPyEval_AcquireThread.