Movatterモバイル変換


[0]ホーム

URL:


Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

Sign up
Appearance settings

gh-100227: Make the Global PyModuleDef Cache Safe for Isolated Interpreters#103084

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to ourterms of service andprivacy statement. We’ll occasionally send you account related emails.

Already on GitHub?Sign in to your account

Conversation

ericsnowcurrently
Copy link
Member

@ericsnowcurrentlyericsnowcurrently commentedMar 28, 2023
edited
Loading

Sharing mutable (or non-immortal) objects between interpreters is generally not safe. We can work around that but not easily.
There are two restrictions that are critical for objects that break interpreter isolation.

The first is that the object's state be guarded by a global lock. For now the GIL meets this requirement, but a granular global lock is needed once we have a per-interpreter GIL.

The second restriction is that the object (and, for a container, its items) be deallocated/resized only when the interpreter in which it was allocated is the current one. This is because every interpreter has (or will have, seegh-101660) its own object allocator. Deallocating an object with a different allocator can cause crashes.

The dict for the cache of module defs is completely internal, which simplifies what we have to do to meet those requirements. To do so, we do the following:

  • add a mechanism for re-using a temporary thread state tied to the main interpreter in an arbitrary thread
    • add_PyRuntime.imports.extensions.main_tstate
    • add_PyThreadState_InitDetached() and_PyThreadState_ClearDetached() (pystate.c)
    • add_PyThreadState_BindDetached() and_PyThreadState_UnbindDetached() (pystate.c)
  • make sure the cache dict (_PyRuntime.imports.extensions.dict) and its items are all owned by the main interpreter)
  • add a placeholder using for a granular global lock

Note that the cache is only used for legacy extension modules and not for multi-phase init modules.

(This PR was derived fromgh-102925, which I recently reverted. Also, there was a similar, more complex PR that I'veclosed:gh-102938.)

@ericsnowcurrentlyericsnowcurrently added the 🔨 test-with-refleak-buildbotsTest PR w/ refleak buildbots; report in status section labelMar 29, 2023
@bedevere-bot
Copy link

🤖 New build scheduled with the buildbot fleet by@ericsnowcurrently for commite9c37fe 🤖

If you want to schedule another build, you need to add the🔨 test-with-refleak-buildbots label again.

@bedevere-botbedevere-bot removed the 🔨 test-with-refleak-buildbotsTest PR w/ refleak buildbots; report in status section labelMar 29, 2023
@ericsnowcurrentlyericsnowcurrently merged commitdcd6f22 intopython:mainMar 29, 2023
@ericsnowcurrentlyericsnowcurrently deleted the extensions-dict-owned-by-main-interpreter-2 branchMarch 29, 2023 23:15
ericsnowcurrently added a commit that referenced this pull requestMar 31, 2023
Decref the key in the right interpreter in _extensions_cache_set().This is a follow-up togh-103084. I found the bug while working ongh-101660.
warsaw pushed a commit to warsaw/cpython that referenced this pull requestApr 11, 2023
…Interpreters (pythongh-103084)Sharing mutable (or non-immortal) objects between interpreters is generally not safe.  We can work around that but not easily.  There are two restrictions that are critical for objects that break interpreter isolation.The first is that the object's state be guarded by a global lock.  For now the GIL meets this requirement, but a granular global lock is needed once we have a per-interpreter GIL.The second restriction is that the object (and, for a container, its items) be deallocated/resized only when the interpreter in which it was allocated is the current one.  This is because every interpreter has (or will have, seepythongh-101660) its own object allocator.  Deallocating an object with a different allocator can cause crashes.The dict for the cache of module defs is completely internal, which simplifies what we have to do to meet those requirements.  To do so, we do the following:* add a mechanism for re-using a temporary thread state tied to the main interpreter in an arbitrary thread   * add _PyRuntime.imports.extensions.main_tstate`    * add _PyThreadState_InitDetached() and _PyThreadState_ClearDetached() (pystate.c)   * add _PyThreadState_BindDetached() and _PyThreadState_UnbindDetached() (pystate.c)* make sure the cache dict (_PyRuntime.imports.extensions.dict) and its items are all owned by the main interpreter)* add a placeholder using for a granular global lockNote that the cache is only used for legacy extension modules and not for multi-phase init modules.python#100227
warsaw pushed a commit to warsaw/cpython that referenced this pull requestApr 11, 2023
Decref the key in the right interpreter in _extensions_cache_set().This is a follow-up topythongh-103084. I found the bug while working onpythongh-101660.
Sign up for freeto join this conversation on GitHub. Already have an account?Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

2 participants
@ericsnowcurrently@bedevere-bot

[8]ページ先頭

©2009-2025 Movatter.jp