Movatterモバイル変換

[0]ホーム
URL:

Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

 Sign up
Appearance settings

gh-99113: A Per-Interpreter GIL!#104210

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to ourterms of service andprivacy statement. We’ll occasionally send you account related emails.

Already on GitHub?Sign in to your account

Merged

Conversation

ericsnowcurrently
Copy link
Member

@ericsnowcurrentlyericsnowcurrently commentedMay 5, 2023•
edited
Loading

This is the culmination of PEP 684 (and of my 8-year long multi-core Python project)!

Each subinterpreter may now be created with its own GIL (viaPy_NewInterpreterFromConfig()). If not so configured then the interpreter will share with the main interpreter--the status quo since the subinterpreters were added decades ago. The main interpreter always has its own GIL and subinterpreters fromPy_NewInterpreter() will always share with the main interpreter.

(FYI, I've split up the originalgh-99114 into multiple PRs, with this one being the final one in the chain.)

primal100, w9ip, 01101sam, viseshrp, BkrmDahal, xsa-dev, Hananonamida, jgcb00, itskatt, audiodude, and IdoBloch4 reacted with thumbs up emojiEclips4, barneygale, erlend-aasland, hauntsaninja, itamaro, thinkwelltwd, wookie184, AA-Turner, encukou, giusepper11, and 27 more reacted with hooray emojistonebig, manodeep, viseshrp, martinohansen, abodacs, jgcb00, tkdchen, jeandut, and gdubicki reacted with heart emojiEclips4, erlend-aasland, alexprengere, arhadthedev, TTimerkhanov, ljarosch, andreia-oca, alcysec, Hananonamida, abodacs, and 7 more reacted with rocket emoji
@bedevere-bot
Copy link

🤖 New build scheduled with the buildbot fleet by@ericsnowcurrently for commit289d050 🤖

If you want to schedule another build, you need to add the🔨 test-with-buildbots label again.

@bedevere-botbedevere-bot removed the 🔨 test-with-buildbotsTest PR w/ buildbots; report in status section labelMay 6, 2023
@ericsnowcurrentlyericsnowcurrently marked this pull request as ready for reviewMay 6, 2023 22:03
@ericsnowcurrently
Copy link
MemberAuthor

FYI, there are plenty of little things to clean up after this lands, particularly for PEP 684. There are also some corner-case crashes I need to chase down, related to extreme races.

Also, I still can't believe I've finally reached this point with this project!!! 😭

erlend-aasland, Eclips4, klausschuch, itamaro, kumaraditya303, madhavajay, ZYM66, audiodude, and gabus reacted with hooray emojierlend-aasland, DBJim, bgruening, Eclips4, h-vetinari, arhadthedev, Fidget-Spinner, itamaro, kumaraditya303, encukou, and 6 more reacted with heart emoji

Copy link
Contributor

@erlend-aaslanderlend-aasland left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

LGTM 🎉 BTW, I think the massive top-level comment inPython/ceval_gil.c needs an overhaul now.

Copy link
Contributor

@kumaraditya303kumaraditya303 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

LGTM is short for the amount of effort you put into it.

erlend-aasland, viseshrp, itskatt, and thijstriemstra reacted with thumbs up emoji
@ericsnowcurrently
Copy link
MemberAuthor

Thanks for all the great work and support, everyone!

There are plenty of small things to wrap up, but this is definitely the climax.

thijstriemstra reacted with thumbs up emojijakirkham, Signko, itamaro, h-vetinari, encukou, erlend-aasland, Eclips4, AA-Turner, Ali-Fani, temeddix, and 2 more reacted with hooray emojijakirkham, erlend-aasland, Eclips4, tkanhe, and war59312 reacted with heart emojijakirkham, encukou, erlend-aasland, Eclips4, and audiodude reacted with rocket emoji

carljm added a commit to carljm/cpython that referenced this pull requestMay 9, 2023
* main: (47 commits)pythongh-97696 Remove unnecessary check for eager_start kwarg (python#104188)pythonGH-104308: socket.getnameinfo should release the GIL (python#104307)pythongh-104310: Add importlib.util.allowing_all_extensions() (pythongh-104311)pythongh-99113: A Per-Interpreter GIL! (pythongh-104210)pythonGH-104284: Fix documentation gettext build (python#104296)pythongh-89550: Buffer GzipFile.write to reduce execution time by ~15% (python#101251)pythongh-104223: Fix issues with inheriting from buffer classes (python#104227)pythongh-99108: fix typo in Modules/Setup (python#104293)pythonGH-104145: Use fully-qualified cross reference types for the bisect module (python#104172)pythongh-103193: Improve `getattr_static` test coverage (python#104286)  Trim trailing whitespace and test on CI (python#104275)pythongh-102500: Remove mention of bytes shorthand (python#104281)pythongh-97696: Improve and fix documentation for asyncio eager tasks (python#104256)pythongh-99108: Replace SHA3 implementation HACL* version (python#103597)pythongh-104273: Remove redundant len() calls in argparse function (python#104274)pythongh-64660: Don't hardcode Argument Clinic return converter result variable name (python#104200)pythongh-104265 Disallow instantiation of `_csv.Reader` and `_csv.Writer` (python#104266)pythonGH-102613: Improve performance of `pathlib.Path.rglob()` (pythonGH-104244)pythongh-103650: Fix perf maps address format (python#103651)pythonGH-89812: Churn `pathlib.Path` methods (pythonGH-104243)  ...
carljm added a commit to carljm/cpython that referenced this pull requestMay 9, 2023
* main: (29 commits)pythongh-104276: Make `_struct.unpack_iterator` type use type flag instead of custom constructor (python#104277)pythongh-97696: Move around and update the whatsnew entry for asyncio eager task factory (python#104298)pythongh-103193: Fix refleaks in `test_inspect` and `test_typing` (python#104320)  require-pr-label.yml: Add missing "permissions:" (python#104309)pythongh-90656: Add platform triplets for 64-bit LoongArch (LA64) (python#30939)pythongh-104180: Read SOCKS proxies from macOS System Configuration (python#104181)pythongh-97696 Remove unnecessary check for eager_start kwarg (python#104188)pythonGH-104308: socket.getnameinfo should release the GIL (python#104307)pythongh-104310: Add importlib.util.allowing_all_extensions() (pythongh-104311)pythongh-99113: A Per-Interpreter GIL! (pythongh-104210)pythonGH-104284: Fix documentation gettext build (python#104296)pythongh-89550: Buffer GzipFile.write to reduce execution time by ~15% (python#101251)pythongh-104223: Fix issues with inheriting from buffer classes (python#104227)pythongh-99108: fix typo in Modules/Setup (python#104293)pythonGH-104145: Use fully-qualified cross reference types for the bisect module (python#104172)pythongh-103193: Improve `getattr_static` test coverage (python#104286)  Trim trailing whitespace and test on CI (python#104275)pythongh-102500: Remove mention of bytes shorthand (python#104281)pythongh-97696: Improve and fix documentation for asyncio eager tasks (python#104256)pythongh-99108: Replace SHA3 implementation HACL* version (python#103597)  ...
carljm added a commit to carljm/cpython that referenced this pull requestMay 9, 2023
* main: (156 commits)pythongh-97696 Add documentation for get_coro() behavior with eager tasks (python#104304)pythongh-97933: (PEP 709) inline list/dict/set comprehensions (python#101441)pythongh-99889: Fix directory traversal security flaw in uu.decode() (python#104096)pythongh-104184: fix building --with-pydebug --enable-pystats (python#104217)pythongh-104139: Add itms-services to uses_netloc urllib.parse. (python#104312)pythongh-104240: return code unit metadata from codegen (python#104300)pythongh-104276: Make `_struct.unpack_iterator` type use type flag instead of custom constructor (python#104277)pythongh-97696: Move around and update the whatsnew entry for asyncio eager task factory (python#104298)pythongh-103193: Fix refleaks in `test_inspect` and `test_typing` (python#104320)  require-pr-label.yml: Add missing "permissions:" (python#104309)pythongh-90656: Add platform triplets for 64-bit LoongArch (LA64) (python#30939)pythongh-104180: Read SOCKS proxies from macOS System Configuration (python#104181)pythongh-97696 Remove unnecessary check for eager_start kwarg (python#104188)pythonGH-104308: socket.getnameinfo should release the GIL (python#104307)pythongh-104310: Add importlib.util.allowing_all_extensions() (pythongh-104311)pythongh-99113: A Per-Interpreter GIL! (pythongh-104210)pythonGH-104284: Fix documentation gettext build (python#104296)pythongh-89550: Buffer GzipFile.write to reduce execution time by ~15% (python#101251)pythongh-104223: Fix issues with inheriting from buffer classes (python#104227)pythongh-99108: fix typo in Modules/Setup (python#104293)  ...
carljm added a commit to carljm/cpython that referenced this pull requestMay 9, 2023
* main: (35 commits)pythongh-97696 Add documentation for get_coro() behavior with eager tasks (python#104304)pythongh-97933: (PEP 709) inline list/dict/set comprehensions (python#101441)pythongh-99889: Fix directory traversal security flaw in uu.decode() (python#104096)pythongh-104184: fix building --with-pydebug --enable-pystats (python#104217)pythongh-104139: Add itms-services to uses_netloc urllib.parse. (python#104312)pythongh-104240: return code unit metadata from codegen (python#104300)pythongh-104276: Make `_struct.unpack_iterator` type use type flag instead of custom constructor (python#104277)pythongh-97696: Move around and update the whatsnew entry for asyncio eager task factory (python#104298)pythongh-103193: Fix refleaks in `test_inspect` and `test_typing` (python#104320)  require-pr-label.yml: Add missing "permissions:" (python#104309)pythongh-90656: Add platform triplets for 64-bit LoongArch (LA64) (python#30939)pythongh-104180: Read SOCKS proxies from macOS System Configuration (python#104181)pythongh-97696 Remove unnecessary check for eager_start kwarg (python#104188)pythonGH-104308: socket.getnameinfo should release the GIL (python#104307)pythongh-104310: Add importlib.util.allowing_all_extensions() (pythongh-104311)pythongh-99113: A Per-Interpreter GIL! (pythongh-104210)pythonGH-104284: Fix documentation gettext build (python#104296)pythongh-89550: Buffer GzipFile.write to reduce execution time by ~15% (python#101251)pythongh-104223: Fix issues with inheriting from buffer classes (python#104227)pythongh-99108: fix typo in Modules/Setup (python#104293)  ...
@aotto1968
Copy link

well , i changed my library to use this, but end up in ...

I started using Py_NewInterpreterFromConfig to add an PER-THREAD interpreter with mylibrary. The library is thread safe and supports many languages-> C C++ JAVA C# VB.Net Perl Python GO TCL Ruby Phpother languages with thread support-> C C++ JAVA C# TCL GOworks fine.even python works fine without thread support.Task: now I started thread support in python, I had to change to Multi-phase initializationisolate the python data on an per-thread-level and finally gent an easy thread example working-> ok    static PyInterpreterConfig config = {        .use_main_obmalloc = 0,        .allow_fork = 0,        .allow_exec = 0,        .allow_threads = 1,        .allow_daemon_threads = 0,        .check_multi_interp_extensions = 1,        .gil = PyInterpreterConfig_OWN_GIL,    };    PyThreadState *tstate = NULL;    PyStatus status = Py_NewInterpreterFromConfig(&tstate, &config);NOW I start with an more complex example and this stuff fails with a python INTERNAL corethat I can't debug. The stack-trace comes from my MqDisasterSignal handler.-> question: what can I do to debug this issue?-> debugging with valgrind is not possible because with valgrind this example works fine.Debug memory block at address p=0x7f11884def70: API '^@'    18302063728033398269 bytes originally requested    The 7 pad bytes at p-7 are not all FORBIDDENBYTE (0xfd):        at p-7: 0x00 *** OUCH        at p-6: 0x00 *** OUCH        at p-5: 0x00 *** OUCH        at p-4: 0x00 *** OUCH        at p-3: 0x00 *** OUCH        at p-2: 0x00 *** OUCH        at p-1: 0x00 *** OUCH    Because memory is corrupted at the start, the count of bytes requested       may be bogus, and checking the trailing pad bytes may segfault.    The 8 pad bytes at tail=0xfdfe7d0f864bed6d are X> {PRINT               :pid(60831):tid(0x7f118819c700):X:dlv(0):ctxId( 0):rc(1):ctx(0x1cb99b8     ):MqDisasterSignal              }: BackTrace {:pid(60831):tid(0x7f118819c700):   [ library              : filename                       : lineno ] function:pid(60831):tid(0x7f118819c700):   [ -------              : --------                       : ------ ] --------:pid(60831):tid(0x7f118819c700):   [ theLink              : c/sys_mq.c                     : 705    ] MqDisasterSignal:pid(60831):tid(0x7f118819c700):   [ unknown              : unknown                        : 0      ] unknown:pid(60831):tid(0x7f118819c700):   [ system               : Objects/obmalloc.c             : 2408   ] _PyObject_DebugDumpAddress:pid(60831):tid(0x7f118819c700):   [ system               : Objects/obmalloc.c             : 2326   ] _PyMem_DebugCheckAddress:pid(60831):tid(0x7f118819c700):   [ system               : Objects/obmalloc.c             : 2159   ] _PyMem_DebugRawFree:pid(60831):tid(0x7f118819c700):   [ system               : Objects/obmalloc.c             : 685    ] PyMem_RawFree:pid(60831):tid(0x7f118819c700):   [ system               : Objects/obmalloc.c             : 1853   ] _PyObject_Free:pid(60831):tid(0x7f118819c700):   [ system               : Objects/obmalloc.c             : 2163   ] _PyMem_DebugRawFree:pid(60831):tid(0x7f118819c700):   [ system               : Objects/obmalloc.c             : 2296   ] _PyMem_DebugFree:pid(60831):tid(0x7f118819c700):   [ system               : Objects/obmalloc.c             : 830    ] PyObject_Free:pid(60831):tid(0x7f118819c700):   [ system               : Objects/dictobject.c           : 1569   ] dictresize:pid(60831):tid(0x7f118819c700):   [ system               : Objects/dictobject.c           : 1194   ] insertion_resize:pid(60831):tid(0x7f118819c700):   [ system               : Objects/dictobject.c           : 1261   ] insertdict:pid(60831):tid(0x7f118819c700):   [ system               : Objects/dictobject.c           : 1865   ] _PyDict_SetItem_Take2:pid(60831):tid(0x7f118819c700):   [ system               : Objects/dictobject.c           : 1883   ] PyDict_SetItem:pid(60831):tid(0x7f118819c700):   [ system               : Objects/typeobject.c           : 7618   ] add_subclass:pid(60831):tid(0x7f118819c700):   [ system               : Objects/typeobject.c           : 7352   ] type_ready_add_subclasses:pid(60831):tid(0x7f118819c700):   [ system               : Objects/typeobject.c           : 7515   ] type_ready:pid(60831):tid(0x7f118819c700):   [ system               : Objects/typeobject.c           : 7553   ] PyType_Ready:pid(60831):tid(0x7f118819c700):   [ system               : Objects/typeobject.c           : 3795   ] type_new_impl:pid(60831):tid(0x7f118819c700):   [ system               : Objects/typeobject.c           : 3929   ] type_new:pid(60831):tid(0x7f118819c700):   [ system               : Objects/typeobject.c           : 1664   ] type_call:pid(60831):tid(0x7f118819c700):   [ system               : Objects/call.c                 : 240    ] _PyObject_MakeTpCall:pid(60831):tid(0x7f118819c700):   [ system               : Objects/typeobject.c           : 3949   ] type_vectorcall:pid(60831):tid(0x7f118819c700):   [ system               : Objects/call.c                 : 133    ] _PyObject_FastCallDictTstate:pid(60831):tid(0x7f118819c700):   [ system               : Objects/call.c                 : 157    ] PyObject_VectorcallDict:pid(60831):tid(0x7f118819c700):   [ system               : Python/bltinmodule.c           : 208    ] builtin___build_class__:pid(60831):tid(0x7f118819c700):   [ system               : Objects/methodobject.c         : 438    ] cfunction_vectorcall_FASTCALL_KEYWORDS:pid(60831):tid(0x7f118819c700):   [ system               : Include/internal/pycore_call.h : 92     ] _PyObject_VectorcallTstate:pid(60831):tid(0x7f118819c700):   [ system               : Objects/call.c                 : 325    ] PyObject_Vectorcall:pid(60831):tid(0x7f118819c700):   [ system               : Python/bytecodes.c             : 2714   ] _PyEval_EvalFrameDefault:pid(60831):tid(0x7f118819c700):   [ system               : nclude/internal/pycore_ceval.h : 89     ] _PyEval_EvalFrame:pid(60831):tid(0x7f118819c700):   [ system               : Python/ceval.c                 : 1683   ] _PyEval_Vector:pid(60831):tid(0x7f118819c700):   [ system               : Python/ceval.c                 : 578    ] PyEval_EvalCode:pid(60831):tid(0x7f118819c700):   [ system               : Python/pythonrun.c             : 1722   ] run_eval_code_obj:pid(60831):tid(0x7f118819c700):   [ system               : Python/pythonrun.c             : 1743   ] run_mod:pid(60831):tid(0x7f118819c700):   [ system               : Python/pythonrun.c             : 1643   ] pyrun_file:pid(60831):tid(0x7f118819c700):   [ system               : Python/pythonrun.c             : 433    ] _PyRun_SimpleFileObject:pid(60831):tid(0x7f118819c700):   [ system               : Python/pythonrun.c             : 466    ] PyRun_SimpleFileExFlags:pid(60831):tid(0x7f118819c700):   [ theLink              : py/MqFactoryC_py.c             : 194    ] py_mqmsgque_sFactoryCTor:pid(60831):tid(0x7f118819c700):   [ theLink              : c/MqFactoryS_mq.c              : 289    ] MqFactoryInvoke_RT:pid(60831):tid(0x7f118819c700):   [ theLink              : c/sys_mq.c                     : 269    ] MqSysServerThreadMain:pid(60831):tid(0x7f118819c700):   [ theLink              : c/sys_mq.c                     : 324    ] sSysServerThreadInit:pid(60831):tid(0x7f118819c700):   [ unknown              : unknown                        : 0      ] unknown:pid(60831):tid(0x7f118819c700): }:pid(60831):tid(0x7f118819c700): ^[[1;36mglobal data -----------------------------------------------------^[[0;m:pid(60831):tid(0x7f118819c700): ^[[1;36mruntime  : MkRuntimeRLS=0x1cb93b0, MqRuntimeRLS=0x7f11879539c8^[[0;m:pid(60831):tid(0x7f118819c700): ^[[1;36mgeDat    : geDat=0x7f1188499de0, geMax=16, geCur=2^[[0;m:pid(60831):tid(0x7f118819c700): ^[[1;36mevent    : event=0x7f1187954610, DataLNum=16, DataLCur=0, geNum=1, fdmax=0^[[0;m.../NHI1/bin/Nhi1Exec: line 870: 60831 Aborted                 '.../x86_64-suse-linux-gnu/debug/bin/python3' '.../NHI1/example/py/server.py' '--thread' '--uds' '--file' '/tmp/test.uds'

Sign up for freeto join this conversation on GitHub. Already have an account?Sign in to comment
Reviewers

@erlend-aaslanderlend-aaslanderlend-aasland approved these changes

@kumaraditya303kumaraditya303kumaraditya303 approved these changes

@markshannonmarkshannonAwaiting requested review from markshannon

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

5 participants
@ericsnowcurrently@bedevere-bot@aotto1968@erlend-aasland@kumaraditya303
[8]ページ先頭
©2009-2025 Movatter.jp