Movatterモバイル変換


[0]ホーム

URL:


Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

Sign up
Appearance settings

[3.12] gh-116510: Fix a crash due to shared immortal interned strings.#124541

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to ourterms of service andprivacy statement. We’ll occasionally send you account related emails.

Already on GitHub?Sign in to your account

Merged
nascheme merged 2 commits intopython:3.12fromnascheme:intern_dict_share_main
Sep 27, 2024

Conversation

nascheme
Copy link
Member

@naschemenascheme commentedSep 25, 2024
edited by bedevere-appbot
Loading

Fix a crash caused by immortal interned strings being shared between sub-interpreters that use basic single-phase init. In that case, the string can be used by an interpreter that outlives the interpeter that created and
interned it. For interpreters that share obmalloc state, also share the interned dict with the main interpreter.

Fix a crash caused by immortal interned strings being shared betweensub-interpreters that use basic single-phase init.  In that case, the stringcan be used by an interpreter that outlives the interpeter that created andinterned it.  For interpreters that share obmalloc state, also share theinterned dict with the main interpreter.
@naschemenascheme changed the titlegh-116510: Fix a crash due to shared immortal interned strings.[3.12] gh-116510: Fix a crash due to shared immortal interned strings.Sep 25, 2024
@nascheme
Copy link
MemberAuthor

This is a small change code-wise but I'm not sure about the full implications of it. It does fix a bug but might create other bugs or problems, I'm not confident.

One downside to this change is that if many sub-interpreters are created and they create a lot of immortal interned strings, those strings will remain alive in the main interpreter state. So, it could cause a lot more memory to be used. My gut feeling is that's okay since we shouldn't be immortalizing strings unless we are fairly sure they won't cause memory bloat, even in a single interpreter case. E.g. it happens to variable names appearing in Python source code.

ericsnowcurrently reacted with thumbs up emoji

Copy link
Member

@ericsnowcurrentlyericsnowcurrently left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

Mostly LGTM

I think there are just a couple spots where we need to clear thePyInterpreterState field for the shared interned dict case.

@bedevere-app
Copy link

When you're done making the requested changes, leave the comment:I have made the requested changes; please review again.

@nascheme
Copy link
MemberAuthor

I have made the requested changes; please review again

@bedevere-app
Copy link

Thanks for making the requested changes!

@ericsnowcurrently: please review the changes made to this pull request.

Copy link
Member

@ericsnowcurrentlyericsnowcurrently left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

LGTM

@naschemenascheme merged commit5dd07eb intopython:3.12Sep 27, 2024
29 of 32 checks passed
@bedevere-bot
Copy link

⚠️⚠️⚠️ Buildbot failure⚠️⚠️⚠️

Hi! The buildbotAMD64 Arch Linux TraceRefs 3.12 has failed when building commit5dd07eb.

What do you need to do:

  1. Don't panic.
  2. Checkthe buildbot page in the devguide if you don't know what the buildbots are or how they work.
  3. Go to the page of the buildbot that failed (https://buildbot.python.org/#/builders/1197/builds/893) and take a look at the build logs.
  4. Check if the failure is related to this commit (5dd07eb) or if it is a false positive.
  5. If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/#/builders/1197/builds/893

Failed tests:

  • test_embed
  • test_syslog
  • test_threading
  • test_import
  • test_multibytecodec
  • test_atexit
  • test_importlib
  • test_int
  • test_capi

Failed subtests:

  • test_subinterps_different_ids - test.test_embed.EmbeddingTests.test_subinterps_different_ids
  • test_single_init_extension_compat - test.test_import.SubinterpImportTests.test_single_init_extension_compat
  • test_python_compat - test.test_import.SubinterpImportTests.test_python_compat
  • test_subinterps_distinct_state - test.test_embed.EmbeddingTests.test_subinterps_distinct_state
  • test_subinterps_main - test.test_embed.EmbeddingTests.test_subinterps_main
  • test_multi_init_extension_compat - test.test_import.SubinterpImportTests.test_multi_init_extension_compat

Summary of the results of the build (if available):

==

Click to see traceback logs
TracebackTests.test_broken_parent_from) ... okTracebackTests.test_broken_submodule) ... okTracebackTests.test_import_bug) ... okTracebackTests.test_syntax_error) ... okTraceback (most recent call last):  File"/buildbot/buildarea/3.12.pablogsal-arch-x86_64/build/Lib/test/test_embed.py", line174, intest_subinterps_mainfor runinself.run_repeated_init_and_subinterpreters():^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^  File"/buildbot/buildarea/3.12.pablogsal-arch-x86_64/build/Lib/test/test_embed.py", line119, inrun_repeated_init_and_subinterpreters    out, err=self.run_embedded_interpreter("test_repeated_init_and_subinterpreters")^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^  File"/buildbot/buildarea/3.12.pablogsal-arch-x86_64/build/Lib/test/test_embed.py", line113, inrun_embedded_interpreterself.assertEqual(p.returncode, returncode,AssertionError:-6 != 0 : bad returncode -6, stderr is 'Objects/object.c:2231: _Py_ForgetReference: Assertion failed: invalid object chain\nEnable tracemalloc to get the memory block allocation traceback\n\nobject address  : 0x7f22e05ffe80\nobject refcount : 0\nobject type     : 0x55b39aff42c0\nobject type name: str\nobject repr     : <refcnt 0 at 0x7f22e05ffe80>\n\nFatal Python error: _PyObject_AssertFailed: _PyObject_AssertFailed\nPython runtime state: finalizing (tstate=0x000055b39b168908)\n\nCurrent thread 0x00007f22e0989740 (most recent call first):\n  <no Python frame>\n'Traceback (most recent call last):  File"/buildbot/buildarea/3.12.pablogsal-arch-x86_64/build/Lib/test/test_embed.py", line180, intest_subinterps_different_idsfor runinself.run_repeated_init_and_subinterpreters():^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^  File"/buildbot/buildarea/3.12.pablogsal-arch-x86_64/build/Lib/test/test_embed.py", line119, inrun_repeated_init_and_subinterpreters    out, err=self.run_embedded_interpreter("test_repeated_init_and_subinterpreters")^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^  File"/buildbot/buildarea/3.12.pablogsal-arch-x86_64/build/Lib/test/test_embed.py", line113, inrun_embedded_interpreterself.assertEqual(p.returncode, returncode,AssertionError:-6 != 0 : bad returncode -6, stderr is 'Objects/object.c:2231: _Py_ForgetReference: Assertion failed: invalid object chain\nEnable tracemalloc to get the memory block allocation traceback\n\nobject address  : 0x7f48cf32fe80\nobject refcount : 0\nobject type     : 0x557a9680d2c0\nobject type name: str\nobject repr     : <refcnt 0 at 0x7f48cf32fe80>\n\nFatal Python error: _PyObject_AssertFailed: _PyObject_AssertFailed\nPython runtime state: finalizing (tstate=0x0000557a96981908)\n\nCurrent thread 0x00007f48cf6b9740 (most recent call first):\n  <no Python frame>\n'TracebackTests.test_exec_failure_nested) ... okTracebackTests.test_nonexistent_module_nested) ... okTraceback (most recent call last):  File"/buildbot/buildarea/3.12.pablogsal-arch-x86_64/build/Lib/test/test_import/__init__.py", line1897, intest_multi_init_extension_compatself.check_compatible_fresh(module,strict=True)  File"/buildbot/buildarea/3.12.pablogsal-arch-x86_64/build/Lib/test/test_import/__init__.py", line1817, incheck_compatible_fresh    _, out, err= script_helper.assert_python_ok('-c', textwrap.dedent(f'''^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^  File"/buildbot/buildarea/3.12.pablogsal-arch-x86_64/build/Lib/test/support/script_helper.py", line166, inassert_python_okreturn _assert_python(True,*args,**env_vars)^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^  File"/buildbot/buildarea/3.12.pablogsal-arch-x86_64/build/Lib/test/support/script_helper.py", line151, in_assert_python    res.fail(cmd_line)  File"/buildbot/buildarea/3.12.pablogsal-arch-x86_64/build/Lib/test/support/script_helper.py", line76, infailraiseAssertionError("Process return code is%d\n"AssertionError:Process return code is -11command line: ['/buildbot/buildarea/3.12.pablogsal-arch-x86_64/build/python', '-X', 'faulthandler', '-I', '-c', '\nimport _testcapi, sys\nassert (\n    \'_testmultiphase\' in sys.builtin_module_names or\n    \'_testmultiphase\' not in sys.modules\n), repr(\'_testmultiphase\')\nret = _testcapi.run_in_subinterp_with_config(\n    "\\nimport os, sys\\n\\ntry:\\n    import _testmultiphase\\nexcept ImportError as exc:\\n    text = \'ImportError: \' + str(exc)\\nelse:\\n    text = \'okay\'\\nos.write(sys.stdout.fileno(), text.encode(\'utf-8\'))\\n",\n    **{\'allow_fork\': False, \'allow_exec\': False, \'allow_threads\': True, \'allow_daemon_threads\': False, \'use_main_obmalloc\': True, \'gil\': 1, \'check_multi_interp_extensions\': True},\n)\nassert ret == 0, ret\n']Traceback (most recent call last):  File"/buildbot/buildarea/3.12.pablogsal-arch-x86_64/build/Lib/test/test_import/__init__.py", line1886, intest_single_init_extension_compatself.check_incompatible_fresh(module)  File"/buildbot/buildarea/3.12.pablogsal-arch-x86_64/build/Lib/test/test_import/__init__.py", line1841, incheck_incompatible_fresh    _, out, err= script_helper.assert_python_ok('-c', textwrap.dedent(f'''^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^  File"/buildbot/buildarea/3.12.pablogsal-arch-x86_64/build/Lib/test/support/script_helper.py", line166, inassert_python_okreturn _assert_python(True,*args,**env_vars)^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^  File"/buildbot/buildarea/3.12.pablogsal-arch-x86_64/build/Lib/test/support/script_helper.py", line151, in_assert_python    res.fail(cmd_line)  File"/buildbot/buildarea/3.12.pablogsal-arch-x86_64/build/Lib/test/support/script_helper.py", line76, infailraiseAssertionError("Process return code is%d\n"AssertionError:Process return code is -11command line: ['/buildbot/buildarea/3.12.pablogsal-arch-x86_64/build/python', '-X', 'faulthandler', '-I', '-c', '\nimport _testcapi, sys\nassert \'_testsinglephase\' not in sys.modules, \'_testsinglephase\'\nret = _testcapi.run_in_subinterp_with_config(\n    "\\nimport os, sys\\n\\ntry:\\n    import _testsinglephase\\nexcept ImportError as exc:\\n    text = \'ImportError: \' + str(exc)\\nelse:\\n    text = \'okay\'\\nos.write(sys.stdout.fileno(), text.encode(\'utf-8\'))\\n",\n    **{\'allow_fork\': False, \'allow_exec\': False, \'allow_threads\': True, \'allow_daemon_threads\': False, \'use_main_obmalloc\': True, \'gil\': 1, \'check_multi_interp_extensions\': True},\n)\nassert ret == 0, ret\n']Traceback (most recent call last):  File"/buildbot/buildarea/3.12.pablogsal-arch-x86_64/build/Lib/test/test_embed.py", line188, intest_subinterps_distinct_statefor runinself.run_repeated_init_and_subinterpreters():^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^  File"/buildbot/buildarea/3.12.pablogsal-arch-x86_64/build/Lib/test/test_embed.py", line119, inrun_repeated_init_and_subinterpreters    out, err=self.run_embedded_interpreter("test_repeated_init_and_subinterpreters")^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^  File"/buildbot/buildarea/3.12.pablogsal-arch-x86_64/build/Lib/test/test_embed.py", line113, inrun_embedded_interpreterself.assertEqual(p.returncode, returncode,AssertionError:-6 != 0 : bad returncode -6, stderr is 'Objects/object.c:2231: _Py_ForgetReference: Assertion failed: invalid object chain\nEnable tracemalloc to get the memory block allocation traceback\n\nobject address  : 0x7fef8cfbfe80\nobject refcount : 0\nobject type     : 0x55da79f132c0\nobject type name: str\nobject repr     : <refcnt 0 at 0x7fef8cfbfe80>\n\nFatal Python error: _PyObject_AssertFailed: _PyObject_AssertFailed\nPython runtime state: finalizing (tstate=0x000055da7a087908)\n\nCurrent thread 0x00007fef8d34b740 (most recent call first):\n  <no Python frame>\n'TracebackTests.test_broken_from) ... okTracebackTests.test_nonexistent_module) ... okTraceback (most recent call last):  File"/buildbot/buildarea/3.12.pablogsal-arch-x86_64/build/Lib/test/test_embed.py", line174, intest_subinterps_mainfor runinself.run_repeated_init_and_subinterpreters():^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^  File"/buildbot/buildarea/3.12.pablogsal-arch-x86_64/build/Lib/test/test_embed.py", line119, inrun_repeated_init_and_subinterpreters    out, err=self.run_embedded_interpreter("test_repeated_init_and_subinterpreters")^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^  File"/buildbot/buildarea/3.12.pablogsal-arch-x86_64/build/Lib/test/test_embed.py", line113, inrun_embedded_interpreterself.assertEqual(p.returncode, returncode,AssertionError:-6 != 0 : bad returncode -6, stderr is 'Objects/object.c:2231: _Py_ForgetReference: Assertion failed: invalid object chain\nEnable tracemalloc to get the memory block allocation traceback\n\nobject address  : 0x7fc0394bfe80\nobject refcount : 0\nobject type     : 0x55c76866b2c0\nobject type name: str\nobject repr     : <refcnt 0 at 0x7fc0394bfe80>\n\nFatal Python error: _PyObject_AssertFailed: _PyObject_AssertFailed\nPython runtime state: finalizing (tstate=0x000055c7687df908)\n\nCurrent thread 0x00007fc03984b740 (most recent call first):\n  <no Python frame>\n'Traceback (most recent call last):  File"/buildbot/buildarea/3.12.pablogsal-arch-x86_64/build/Lib/test/test_import/__init__.py", line1945, intest_python_compatself.check_compatible_fresh(module,strict=True)  File"/buildbot/buildarea/3.12.pablogsal-arch-x86_64/build/Lib/test/test_import/__init__.py", line1817, incheck_compatible_fresh    _, out, err= script_helper.assert_python_ok('-c', textwrap.dedent(f'''^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^  File"/buildbot/buildarea/3.12.pablogsal-arch-x86_64/build/Lib/test/support/script_helper.py", line166, inassert_python_okreturn _assert_python(True,*args,**env_vars)^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^  File"/buildbot/buildarea/3.12.pablogsal-arch-x86_64/build/Lib/test/support/script_helper.py", line151, in_assert_python    res.fail(cmd_line)  File"/buildbot/buildarea/3.12.pablogsal-arch-x86_64/build/Lib/test/support/script_helper.py", line76, infailraiseAssertionError("Process return code is%d\n"AssertionError:Process return code is -11command line: ['/buildbot/buildarea/3.12.pablogsal-arch-x86_64/build/python', '-X', 'faulthandler', '-I', '-c', '\nimport _testcapi, sys\nassert (\n    \'threading\' in sys.builtin_module_names or\n    \'threading\' not in sys.modules\n), repr(\'threading\')\nret = _testcapi.run_in_subinterp_with_config(\n    "\\nimport os, sys\\n\\ntry:\\n    import threading\\nexcept ImportError as exc:\\n    text = \'ImportError: \' + str(exc)\\nelse:\\n    text = \'okay\'\\nos.write(sys.stdout.fileno(), text.encode(\'utf-8\'))\\n",\n    **{\'allow_fork\': False, \'allow_exec\': False, \'allow_threads\': True, \'allow_daemon_threads\': False, \'use_main_obmalloc\': True, \'gil\': 1, \'check_multi_interp_extensions\': True},\n)\nassert ret == 0, ret\n']TracebackTests.test_unencodable_filename) ... okTraceback (most recent call last):  File"/buildbot/buildarea/3.12.pablogsal-arch-x86_64/build/Lib/test/test_embed.py", line188, intest_subinterps_distinct_statefor runinself.run_repeated_init_and_subinterpreters():^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^  File"/buildbot/buildarea/3.12.pablogsal-arch-x86_64/build/Lib/test/test_embed.py", line119, inrun_repeated_init_and_subinterpreters    out, err=self.run_embedded_interpreter("test_repeated_init_and_subinterpreters")^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^  File"/buildbot/buildarea/3.12.pablogsal-arch-x86_64/build/Lib/test/test_embed.py", line113, inrun_embedded_interpreterself.assertEqual(p.returncode, returncode,AssertionError:-6 != 0 : bad returncode -6, stderr is 'Objects/object.c:2231: _Py_ForgetReference: Assertion failed: invalid object chain\nEnable tracemalloc to get the memory block allocation traceback\n\nobject address  : 0x7f14a6de80a0\nobject refcount : 0\nobject type     : 0x5567decca2c0\nobject type name: str\nobject repr     : <refcnt 0 at 0x7f14a6de80a0>\n\nFatal Python error: _PyObject_AssertFailed: _PyObject_AssertFailed\nPython runtime state: finalizing (tstate=0x00005567dee3e908)\n\nCurrent thread 0x00007f14a7168740 (most recent call first):\n  <no Python frame>\n'Traceback (most recent call last):  File"/buildbot/buildarea/3.12.pablogsal-arch-x86_64/build/Lib/test/test_embed.py", line180, intest_subinterps_different_idsfor runinself.run_repeated_init_and_subinterpreters():^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^  File"/buildbot/buildarea/3.12.pablogsal-arch-x86_64/build/Lib/test/test_embed.py", line119, inrun_repeated_init_and_subinterpreters    out, err=self.run_embedded_interpreter("test_repeated_init_and_subinterpreters")^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^  File"/buildbot/buildarea/3.12.pablogsal-arch-x86_64/build/Lib/test/test_embed.py", line113, inrun_embedded_interpreterself.assertEqual(p.returncode, returncode,AssertionError:-6 != 0 : bad returncode -6, stderr is 'Objects/object.c:2231: _Py_ForgetReference: Assertion failed: invalid object chain\nEnable tracemalloc to get the memory block allocation traceback\n\nobject address  : 0x7f06ef537e80\nobject refcount : 0\nobject type     : 0x5610d3a722c0\nobject type name: str\nobject repr     : <refcnt 0 at 0x7f06ef537e80>\n\nFatal Python error: _PyObject_AssertFailed: _PyObject_AssertFailed\nPython runtime state: finalizing (tstate=0x00005610d3be6908)\n\nCurrent thread 0x00007f06ef8be740 (most recent call first):\n  <no Python frame>\n'TracebackTests.test_broken_parent) ... okTracebackTests.test_exec_failure) ... ok

nascheme added a commit to nascheme/cpython that referenced this pull requestOct 1, 2024
@bedevere-app
Copy link

GH-124814 is a backport of this pull request to the3.12 branch.

Yhg1s pushed a commit that referenced this pull requestOct 1, 2024
… immortal interned strings. (gh-124541)" (#124814)Revert "[3.12]gh-116510: Fix a crash due to shared immortal interned strings. (gh-124541)"This reverts commit5dd07eb.
Sign up for freeto join this conversation on GitHub. Already have an account?Sign in to comment
Reviewers

@ericsnowcurrentlyericsnowcurrentlyericsnowcurrently approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

3 participants
@nascheme@bedevere-bot@ericsnowcurrently

[8]ページ先頭

©2009-2025 Movatter.jp