Uh oh!
There was an error while loading.Please reload this page.
- Notifications
You must be signed in to change notification settings - Fork32k
gh-113628: Fix test_site test with long stdlib paths#113640
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to ourterms of service andprivacy statement. We’ll occasionally send you account related emails.
Already on GitHub?Sign in to your account
Merged
Merged
Uh oh!
There was an error while loading.Please reload this page.
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.Learn more about bidirectional Unicode characters
carljm approved these changesJan 3, 2024
miss-islington pushed a commit to miss-islington/cpython that referenced this pull requestJan 3, 2024
GH-113671 is a backport of this pull request to the3.12 branch. |
miss-islington pushed a commit to miss-islington/cpython that referenced this pull requestJan 3, 2024
GH-113672 is a backport of this pull request to the3.11 branch. |
carljm pushed a commit that referenced this pull requestJan 3, 2024
carljm pushed a commit that referenced this pull requestJan 3, 2024
facebook-github-bot pushed a commit to facebookincubator/cinder that referenced this pull requestJan 3, 2024
Summary:upstream issue:python/cpython#113628upstream PR:python/cpython#113640upstream commit:python/cpython@5dc79e3 ---The `getpath` module [tries to read certain `_pth` files during initialization](https://github.com/python/cpython/blob/b4b2cc101216ae1017898dfbe43c90da2fd0a308/Modules/getpath.py#L462)).This is tested in `test_site` with generated `_pth` files that [include the stdlib path 200 times](https://github.com/python/cpython/blob/b4b2cc101216ae1017898dfbe43c90da2fd0a308/Lib/test/test_site.py#L669).The `getpath` module [disallows reading files over 32KB during initialization](https://github.com/python/cpython/blob/b4b2cc101216ae1017898dfbe43c90da2fd0a308/Modules/getpath.c#L375).If the test suite runs from a very long base path, 200 repetitions of the stdlib path in the `_pth` file would be enough to exceed the 32KB limit.To demonstrate, artificially increase the number of repetitions to some high number that would exceed 32KB, e.g. this patch:``` diff --git a/Lib/test/test_site.py b/Lib/test/test_site.py --- a/Lib/test/test_site.py+++ b/Lib/test/test_site.py@@ -631,5 +631,5 @@ pth_lines = [ 'fake-path-name',- *[libpath for _ in range(200)],+ *[libpath for _ in range(5000)], '', '# comment',```and observe the test failing with the following error:```python3.12 -m test test_site -v -m '*_pthFileTests.test_underpth_nosite_file'== CPython 3.12.1+meta (3.12:2305ca5, Dec 07 2023, 21:46:47) [Clang 15.0.7 (mononoke://mononoke.internal.tfbnw.net/fbsource 3e29b2044f484840f== Linux-5.12.0-0_fbk16_hardened_7661_geb00762ce6d2-x86_64-with-glibc2.34 little-endian== Python build: release ThinLTO dtrace== cwd: /tmp/test_python_worker_2504793æ== CPU count: 72== encodings: locale=UTF-8 FS=utf-8== resources: all test resources are disabled, use -u option to unskip testsUsing random seed: 30016444980:00:00 load avg: 6.52 Run 1 test sequentially0:00:00 load avg: 6.52 [1/1] test_sitetest_underpth_nosite_file (test.test_site._pthFileTests.test_underpth_nosite_file) ... Exception ignored error evaluating path:Traceback (most recent call last): File "<frozen getpath>", line 463, in <module>MemoryError: cannot read file larger than 32KB during initializationFatal Python error: error evaluating pathPython runtime state: core initializedCurrent thread 0x00007fb01f2d0740 (most recent call first): <no Python frame>ERROR======================================================================ERROR: test_underpth_nosite_file (test.test_site._pthFileTests.test_underpth_nosite_file)----------------------------------------------------------------------Traceback (most recent call last): File "/data/users/itamaro/fbsource/buck-out/v2/gen/fbsource/da203790281a65b9/third-party/python/3.12/__install-base__/out/install/lib/python3.12/test/test_site.py", line 647, in test_underpth_nosite_file output = subprocess.check_output([exe_file, '-c', ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/data/users/itamaro/fbsource/buck-out/v2/gen/fbsource/da203790281a65b9/third-party/python/3.12/__install-base__/out/install/lib/python3.12/subprocess.py", line 466, in check_output return run(*popenargs, stdout=PIPE, timeout=timeout, check=True, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/data/users/itamaro/fbsource/buck-out/v2/gen/fbsource/da203790281a65b9/third-party/python/3.12/__install-base__/out/install/lib/python3.12/subprocess.py", line 571, in run raise CalledProcessError(retcode, process.args,subprocess.CalledProcessError: Command '['/tmp/tmpsq9c4frs/python3.12', '-c', 'import sys; print("\\n".join(sys.path) if sys.flags.no_site else "")']' returned non-zero exit status 1.----------------------------------------------------------------------Ran 1 test in 0.019sFAILED (errors=1)test test_site failedtest_site failed (1 error)== Tests result: FAILURE ==1 test failed: test_siteTotal duration: 118 msTotal tests: run=1 (filtered)Total test files: run=1/1 (filtered) failed=1Result: FAILURE```Seepython/cpython#113628 for an upstream issue report.Reviewed By: carljmDifferential Revision: D52475050fbshipit-source-id: 3d8d19bc9058d8c78eb65fd4477e0c7e848c7e81
kulikjak pushed a commit to kulikjak/cpython that referenced this pull requestJan 22, 2024
aisk pushed a commit to aisk/cpython that referenced this pull requestFeb 11, 2024
Glyphack pushed a commit to Glyphack/cpython that referenced this pull requestSep 2, 2024
Sign up for freeto join this conversation on GitHub. Already have an account?Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading.Please reload this page.
As suggested by@terryjreedyon the issue, adapt the number of
libpathlines in pth lines based on the length oflibpathto avoid exceeding a pth file that exceeds the 32KB limit.I (arbitrarily) added a skip test if we end up with 2 or fewer repetitions (which is highly unlikely, considering this would mean a stdlib path of over 10k bytes)