Uh oh!
There was an error while loading.Please reload this page.
- Notifications
You must be signed in to change notification settings - Fork32.1k
gh-104690 Disallow thread creation and fork at interpreter finalization#104826
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to ourterms of service andprivacy statement. We’ll occasionally send you account related emails.
Already on GitHub?Sign in to your account
gh-104690 Disallow thread creation and fork at interpreter finalization#104826
Uh oh!
There was an error while loading.Please reload this page.
Conversation
in the following functions, check if interpreter is finalizing and raise `RuntimeError` with appropriate message:* `_thread.start_new_thread`* `posix.fork`* `posix.fork1`* `posix.forkpty`* `_posixsubprocess.fork_exec`
Uh oh!
There was an error while loading.Please reload this page.
Misc/NEWS.d/next/Core and Builtins/2023-05-24-12-10-54.gh-issue-104690.HX3Jou.rst OutdatedShow resolvedHide resolved
Uh oh!
There was an error while loading.Please reload this page.
Uh oh!
There was an error while loading.Please reload this page.
bedevere-bot commentedMay 26, 2023
A Python core developer has requested some changes be made to your pull request before we can consider merging it. If you could please address their requests along with any other requests in other reviews from core developers that would be appreciated. Once you have made the requested changes, please leave a comment on this pull request containing the phrase And if you don't make the requested changes, you will be put in the comfy chair! |
…thub.com:chgnrdv/cpython into disallow-thread-creation-and-fork-at-interp-exit
* removed excess test case from `test_threading`* changed NEWS entry to not mention internal function* allowed `subprocess.Popen` at shutdown if `preexec_fn` is `None`
I have made the requested changes; please review again |
bedevere-bot commentedMay 26, 2023
Thanks for making the requested changes! @gpshead: please review the changes made to this pull request. |
FYI - if you want an example of real world code that was spawning threads from atexit (I'm not claiming that this was agood idea... just that it inadvertently happened): See the example given in#86813. |
Sorry@chgnrdv and@gpshead, I had trouble checking out the |
…lization (pythonGH-104826)Disallow thread creation and fork at interpreter finalization.in the following functions, check if interpreter is finalizing and raise `RuntimeError` with appropriate message:* `_thread.start_new_thread` and thus `threading`* `posix.fork`* `posix.fork1`* `posix.forkpty`* `_posixsubprocess.fork_exec` when a `preexec_fn=` is supplied.---------(cherry picked from commitce558e6)Co-authored-by: chgnrdv <52372310+chgnrdv@users.noreply.github.com>Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com>Co-authored-by: Gregory P. Smith <greg@krypto.org>
bedevere-bot commentedJun 4, 2023
GH-105277 is a backport of this pull request to the3.12 branch. |
…alization (GH-104826) (#105277)gh-104690 Disallow thread creation and fork at interpreter finalization (GH-104826)Disallow thread creation and fork at interpreter finalization.in the following functions, check if interpreter is finalizing and raise `RuntimeError` with appropriate message:* `_thread.start_new_thread` and thus `threading`* `posix.fork`* `posix.fork1`* `posix.forkpty`* `_posixsubprocess.fork_exec` when a `preexec_fn=` is supplied.---------(cherry picked from commitce558e6)Co-authored-by: chgnrdv <52372310+chgnrdv@users.noreply.github.com>Co-authored-by: Gregory P. Smith <greg@krypto.org>
In Python 3.12, you cannot pass a preexec_fn to subprocess.Popenduring interpreter shutdown:python/cpython#104826This avoids the problem by giving startProgram an arg to notuse a preexec_fn, and passing that all the way from anaconda'sexitHandler through execWithRedirect and _run_program.Signed-off-by: Adam Williamson <awilliam@redhat.com>
In Python 3.12, you cannot pass a preexec_fn to subprocess.Popenduring interpreter shutdown:python/cpython#104826This avoids the problem by giving startProgram an arg to notuse a preexec_fn, and passing that all the way from anaconda'sexitHandler through execWithRedirect and _run_program.Signed-off-by: Adam Williamson <awilliam@redhat.com>
This change introduced a test_concurrent_futures regression: issue#109047. Well, right now I'm not sure about the root cause analysis. |
… python 3.12 (#6859)## Motivation- Make the instrumentation telemetry client compatible with python3.12:python/cpython#104826## Description - Start telemetry worker thread as early as possible. - Delays sending all telemetry events until app-started is queued. - Refactors tests to align with this new logic. ## Risk - Telemetry events (metrics/logs/integrations) are queued as early aspossible but these events are only sent when the trace agent writer isstarted. This **may** result in a memory leak if high cardinalitytelemetry metrics and logs are added in the future. This is not aconcern right now.## Checklist- [x] Change(s) are motivated and described in the PR description.- [x] Testing strategy is described if automated tests are not includedin the PR.- [x] Risk is outlined (performance impact, potential for breakage,maintainability, etc).- [x] Change is maintainable (easy to change, telemetry, documentation).- [x] [Library release noteguidelines](https://ddtrace.readthedocs.io/en/stable/releasenotes.html)are followed. If no release note is required, add label`changelog/no-changelog`.- [x] Documentation is included (in-code, generated user docs, [publiccorp docs](https://github.com/DataDog/documentation/)).- [x] Backport labels are set (if[applicable](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting))## Reviewer Checklist- [x] Title is accurate.- [x] No unnecessary changes are introduced.- [x] Description motivates each change.- [x] Avoids breaking[API](https://ddtrace.readthedocs.io/en/stable/versioning.html#interfaces)changes unless absolutely necessary.- [x] Testing strategy adequately addresses listed risk(s).- [x] Change is maintainable (easy to change, telemetry, documentation).- [x] Release note makes sense to a user of the library.- [x] Reviewer has explicitly acknowledged and discussed the performanceimplications of this PR as reported in the benchmarks PR comment.- [x] Backport labels are set in a manner that is consistent with the[release branch maintenancepolicy](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting)- [x] If this PR touches code that signs or publishes builds orpackages, or handles credentials of any kind, I've requested a reviewfrom `@DataDog/security-design-and-guidance`.- [x] This PR doesn't touch any of that.---------Co-authored-by: Yun Kim <yun.kim@datadoghq.com>Co-authored-by: Tahir H. Butt <tahir.butt@datadoghq.com>Co-authored-by: Gabriele N. Tornetta <gabriele.tornetta@datadoghq.com>Co-authored-by: Yun Kim <35776586+Yun-Kim@users.noreply.github.com>Co-authored-by: Emmett Butler <emmett.butler321@gmail.com>Co-authored-by: ZStriker19 <zach.groves@datadoghq.com>
… python 3.12 (#6859)- Make the instrumentation telemetry client compatible with python3.12:python/cpython#104826 - Start telemetry worker thread as early as possible. - Delays sending all telemetry events until app-started is queued. - Refactors tests to align with this new logic.- Telemetry events (metrics/logs/integrations) are queued as early aspossible but these events are only sent when the trace agent writer isstarted. This **may** result in a memory leak if high cardinalitytelemetry metrics and logs are added in the future. This is not aconcern right now.- [x] Change(s) are motivated and described in the PR description.- [x] Testing strategy is described if automated tests are not includedin the PR.- [x] Risk is outlined (performance impact, potential for breakage,maintainability, etc).- [x] Change is maintainable (easy to change, telemetry, documentation).- [x] [Library release noteguidelines](https://ddtrace.readthedocs.io/en/stable/releasenotes.html)are followed. If no release note is required, add label`changelog/no-changelog`.- [x] Documentation is included (in-code, generated user docs, [publiccorp docs](https://github.com/DataDog/documentation/)).- [x] Backport labels are set (if[applicable](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting))- [x] Title is accurate.- [x] No unnecessary changes are introduced.- [x] Description motivates each change.- [x] Avoids breaking[API](https://ddtrace.readthedocs.io/en/stable/versioning.html#interfaces)changes unless absolutely necessary.- [x] Testing strategy adequately addresses listed risk(s).- [x] Change is maintainable (easy to change, telemetry, documentation).- [x] Release note makes sense to a user of the library.- [x] Reviewer has explicitly acknowledged and discussed the performanceimplications of this PR as reported in the benchmarks PR comment.- [x] Backport labels are set in a manner that is consistent with the[release branch maintenancepolicy](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting)- [x] If this PR touches code that signs or publishes builds orpackages, or handles credentials of any kind, I've requested a reviewfrom `@DataDog/security-design-and-guidance`.- [x] This PR doesn't touch any of that.---------Co-authored-by: Yun Kim <yun.kim@datadoghq.com>Co-authored-by: Tahir H. Butt <tahir.butt@datadoghq.com>Co-authored-by: Gabriele N. Tornetta <gabriele.tornetta@datadoghq.com>Co-authored-by: Yun Kim <35776586+Yun-Kim@users.noreply.github.com>Co-authored-by: Emmett Butler <emmett.butler321@gmail.com>Co-authored-by: ZStriker19 <zach.groves@datadoghq.com>
… python 3.12 (#6859)## Motivation- Make the instrumentation telemetry client compatible with python3.12:python/cpython#104826## Description - Start telemetry worker thread as early as possible. - Delays sending all telemetry events until app-started is queued. - Refactors tests to align with this new logic. ## Risk - Telemetry events (metrics/logs/integrations) are queued as early aspossible but these events are only sent when the trace agent writer isstarted. This **may** result in a memory leak if high cardinalitytelemetry metrics and logs are added in the future. This is not aconcern right now.## Checklist- [x] Change(s) are motivated and described in the PR description.- [x] Testing strategy is described if automated tests are not includedin the PR.- [x] Risk is outlined (performance impact, potential for breakage,maintainability, etc).- [x] Change is maintainable (easy to change, telemetry, documentation).- [x] [Library release noteguidelines](https://ddtrace.readthedocs.io/en/stable/releasenotes.html)are followed. If no release note is required, add label`changelog/no-changelog`.- [x] Documentation is included (in-code, generated user docs, [publiccorp docs](https://github.com/DataDog/documentation/)).- [x] Backport labels are set (if[applicable](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting))## Reviewer Checklist- [x] Title is accurate.- [x] No unnecessary changes are introduced.- [x] Description motivates each change.- [x] Avoids breaking[API](https://ddtrace.readthedocs.io/en/stable/versioning.html#interfaces)changes unless absolutely necessary.- [x] Testing strategy adequately addresses listed risk(s).- [x] Change is maintainable (easy to change, telemetry, documentation).- [x] Release note makes sense to a user of the library.- [x] Reviewer has explicitly acknowledged and discussed the performanceimplications of this PR as reported in the benchmarks PR comment.- [x] Backport labels are set in a manner that is consistent with the[release branch maintenancepolicy](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting)- [x] If this PR touches code that signs or publishes builds orpackages, or handles credentials of any kind, I've requested a reviewfrom `@DataDog/security-design-and-guidance`.- [x] This PR doesn't touch any of that.---------Co-authored-by: Yun Kim <yun.kim@datadoghq.com>Co-authored-by: Tahir H. Butt <tahir.butt@datadoghq.com>Co-authored-by: Gabriele N. Tornetta <gabriele.tornetta@datadoghq.com>Co-authored-by: Yun Kim <35776586+Yun-Kim@users.noreply.github.com>Co-authored-by: Emmett Butler <emmett.butler321@gmail.com>Co-authored-by: ZStriker19 <zach.groves@datadoghq.com>
Python 3.12 introduced a bug during the interpreter shutdown, see:- issuepython/cpython#104826- bugfixpython/cpython#117029With Python 3.12.3, this change could potentially be reverted (after testing).
Python 3.12 introduced a bug during the interpreter shutdown, see:- issuepython/cpython#104826- bugfixpython/cpython#117029With Python 3.12.3, this change could potentially be reverted (after testing).
Uh oh!
There was an error while loading.Please reload this page.
Fixes#104690
In the following functions, check if interpreter is finalizing and in this case raise
RuntimeError
with appropriate message:_thread.start_new_thread
posix.fork
posix.fork1
posix.forkpty
_posixsubprocess.fork_exec
when a preexec_fn is used.