Movatterモバイル変換

[0]ホーム
URL:

Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

 Sign up
Appearance settings

WIP: Python 3.13 support#6

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to ourterms of service andprivacy statement. We’ll occasionally send you account related emails.

Already on GitHub?Sign in to your account

Closed
zanieb wants to merge33 commits intomainfromzb/3.13
Closed

WIP: Python 3.13 support#6

zanieb wants to merge33 commits intomainfromzb/3.13

Conversation

@zanieb
Copy link
Owner

@zaniebzanieb commentedSep 4, 2024•
edited
Loading

Replacing#5 — need a different branch name for my sanity.

Personal branch ofastral-sh#264 for testing.

@zaniebzanieb changed the titlePython 3.13 supportWIP: Python 3.13 supportSep 4, 2024
@zaniebzaniebforce-pushed thezb/3.13 branch 2 times, most recently fromb33ec8d toa02d8fbCompareSeptember 4, 2024 18:11
@zanieb
Copy link
OwnerAuthor

zanieb commentedSep 4, 2024•
edited
Loading

Okay(aarch64-apple-darwin, macos-14, cpython-3.13, pgo) is failing ontest_embed which lives inCPython.

Unfortunately the only output istest_embed failed (41 failures)

We'll need to get more output from the test script.

(aarch64-apple-darwin, macos-14, cpython-3.13, pgo+lto) is the same.

A local attempt to reproduce this succeeded. I tried settingPROFILE_TASK to include-v to seewhy the tests are failing and stumbled my way over to aCPython commit which does not show failures or allow verbose output with the PGO flag.

Furthermore, I think this same test fails in Python 3.12 and we're just cool with that. I'm not sure why it's a hard-failure here. I don't actually see this test fail on 3.12 now.

Finally stumbled my way to a verbose failure

cpython-3.13> ======================================================================cpython-3.13> FAIL: test_preinit_python_config (test.test_embed.InitConfigTests.test_preinit_python_config)cpython-3.13> ----------------------------------------------------------------------cpython-3.13> Traceback (most recent call last):cpython-3.13>   File "/private/var/folders/6p/k5sd5z7j31b31pq4lhn0l8d80000gn/T/tmpp72efx2c/Python-3.13.0/Lib/test/test_embed.py", line 1179, in test_preinit_python_configcpython-3.13>     self.check_all_configs("test_preinit_python_config", api=API_PYTHON)cpython-3.13>     ~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^cpython-3.13>   File "/private/var/folders/6p/k5sd5z7j31b31pq4lhn0l8d80000gn/T/tmpp72efx2c/Python-3.13.0/Lib/test/test_embed.py", line 899, in check_all_configscpython-3.13>     self.assertEqual(err.rstrip(), stderr)cpython-3.13>     ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^cpython-3.13> AssertionError: 'objc[48331]: Class TKNSString is implemen[3073 chars]ned.' != ''cpython-3.13> - objc[48331]: Class TKNSString is implemented in both /private/var/folders/6p/k5sd5z7j31b31pq4lhn0l8d80000gn/T/tmpp72efx2c/Python-3.13.0/libpython3.13.dylib (0x1032eef50) and /private/var/folders/6p/k5sd5z7j31b31pq4lhn0l8d80000gn/T/tmpp72efx2c/Python-3.13.0/Programs/_testembed (0x1003ede58). One of the two will be used. Which one is undefined.cpython-3.13> - objc[48331]: Class TKApplication is implemented in both /private/var/folders/6p/k5sd5z7j31b31pq4lhn0l8d80000gn/T/tmpp72efx2c/Python-3.13.0/libpython3.13.dylib (0x1032eefc8) and /private/var/folders/6p/k5sd5z7j31b31pq4lhn0l8d80000gn/T/tmpp72efx2c/Python-3.13.0/Programs/_testembed (0x1003eded0). One of the two will be used. Which one is undefined.cpython-3.13> - objc[48331]: Class TKContentView is implemented in both /private/var/folders/6p/k5sd5z7j31b31pq4lhn0l8d80000gn/T/tmpp72efx2c/Python-3.13.0/libpython3.13.dylib (0x1032ef018) and /private/var/folders/6p/k5sd5z7j31b31pq4lhn0l8d80000gn/T/tmpp72efx2c/Python-3.13.0/Programs/_testembed (0x1003edf20). One of the two will be used. Which one is undefined.cpython-3.13> - objc[48331]: Class TKBackgroundLoop is implemented in both /private/var/folders/6p/k5sd5z7j31b31pq4lhn0l8d80000gn/T/tmpp72efx2c/Python-3.13.0/libpython3.13.dylib (0x1032ef068) and /private/var/folders/6p/k5sd5z7j31b31pq4lhn0l8d80000gn/T/tmpp72efx2c/Python-3.13.0/Programs/_testembed (0x1003edf70). One of the two will be used. Which one is undefined.cpython-3.13> - objc[48331]: Class TKMenu is implemented in both /private/var/folders/6p/k5sd5z7j31b31pq4lhn0l8d80000gn/T/tmpp72efx2c/Python-3.13.0/libpython3.13.dylib (0x1032ef090) and /private/var/folders/6p/k5sd5z7j31b31pq4lhn0l8d80000gn/T/tmpp72efx2c/Python-3.13.0/Programs/_testembed (0x1003edf98). One of the two will be used. Which one is undefined.cpython-3.13> - objc[48331]: Class TkService is implemented in both /private/var/folders/6p/k5sd5z7j31b31pq4lhn0l8d80000gn/T/tmpp72efx2c/Python-3.13.0/libpython3.13.dylib (0x1032ef108) and /private/var/folders/6p/k5sd5z7j31b31pq4lhn0l8d80000gn/T/tmpp72efx2c/Python-3.13.0/Programs/_testembed (0x1003ee010). One of the two will be used. Which one is undefined.cpython-3.13> - objc[48331]: Class TKPanel is implemented in both /private/var/folders/6p/k5sd5z7j31b31pq4lhn0l8d80000gn/T/tmpp72efx2c/Python-3.13.0/libpython3.13.dylib (0x1032ef158) and /private/var/folders/6p/k5sd5z7j31b31pq4lhn0l8d80000gn/T/tmpp72efx2c/Python-3.13.0/Programs/_testembed (0x1003ee060). One of the two will be used. Which one is undefined.cpython-3.13> - objc[48331]: Class TKDrawerWindow is implemented in both /private/var/folders/6p/k5sd5z7j31b31pq4lhn0l8d80000gn/T/tmpp72efx2c/Python-3.13.0/libpython3.13.dylib (0x1032ef1a8) and /private/var/folders/6p/k5sd5z7j31b31pq4lhn0l8d80000gn/T/tmpp72efx2c/Python-3.13.0/Programs/_testembed (0x1003ee0b0). One of the two will be used. Which one is undefined.cpython-3.13> - objc[48331]: Class TKWindow is implemented in both /private/var/folders/6p/k5sd5z7j31b31pq4lhn0l8d80000gn/T/tmpp72efx2c/Python-3.13.0/libpython3.13.dylib (0x1032ef1f8) and /private/var/folders/6p/k5sd5z7j31b31pq4lhn0l8d80000gn/T/tmpp72efx2c/Python-3.13.0/Programs/_testembed (0x1003ee100). One of the two will be used. Which one is undefined.

with

diff --git a/pythonbuild/utils.py b/pythonbuild/utils.pyindex bf05d0d..83227f5 100644--- a/pythonbuild/utils.py+++ b/pythonbuild/utils.py@@ -495,6 +495,7 @@ def add_env_common(env):     """Adds extra keys to environment variables."""      cpu_count = multiprocessing.cpu_count()+    env["PROFILE_TASK"] = "-m test --pgo -W"     env["NUM_CPUS"] = "%d" % cpu_count     env["NUM_JOBS_AGGRESSIVE"] = "%d" % max(cpu_count + 2, cpu_count * 2)

and

diff --git a/Lib/test/libregrtest/cmdline.py b/Lib/test/libregrtest/cmdline.pyindex 8bef04cba81..0bff5eaff5b 100644--- a/Lib/test/libregrtest/cmdline.py+++ b/Lib/test/libregrtest/cmdline.py@@ -480,8 +480,6 @@ def _parse_args(args, **kwargs):         ns.python = shlex.split(ns.python)     if ns.failfast and not (ns.verbose or ns.verbose3):         parser.error("-G/--failfast needs either -v or -W")-    if ns.pgo and (ns.verbose or ns.rerun or ns.verbose3):-        parser.error("--pgo/-v don't go together!")     if ns.pgo_extended:         ns.pgo = True  # pgo_extended implies pgo

I was also able to get past the test failure during PGO with this patch

diff --git a/Lib/test/libregrtest/main.py b/Lib/test/libregrtest/main.pyindex 5148d307051..85b9e46f946 100644--- a/Lib/test/libregrtest/main.py+++ b/Lib/test/libregrtest/main.py@@ -555,6 +555,9 @@ def _run_tests(self, selected: TestTuple, tests: TestList | None) -> int:         self.display_summary()         self.finalize_tests(coverage)+        if self.pgo:+            return 0+         return self.results.get_exitcode(self.fail_env_changed,                                          self.fail_rerun)

but that doesn't seem ideal.

See#6 (comment) for a resolution.

@zanieb
Copy link
OwnerAuthor

zanieb commentedSep 4, 2024•
edited
Loading

(x86_64-apple-darwin, macos-13, cpython-3.13, debug) is failing with

cpython-3.13> ./Modules/posixmodule.c:8659:11: error: 'ptsname_r' is only available on macOS 10.13.4 or newer [-Werror,-Wunguarded-availability-new]cpython-3.13>  8659 |     ret = ptsname_r(fd, name, sizeof(name));cpython-3.13>       |           ^~~~~~~~~cpython-3.13> /Applications/Xcode_15.2.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX14.2.sdk/usr/include/stdlib.h:220:5: note: 'ptsname_r' has been marked as being introduced in macOS 10.13.4 here, but the deployment target is macOS 10.9.0cpython-3.13>   220 | int ptsname_r(int fildes, char *buffer, size_t buflen) __API_AVAILABLE(macos(10.13.4), ios(11.3), tvos(11.3), watchos(4.3));cpython-3.13>       |     ^cpython-3.13> ./Modules/posixmodule.c:8659:11: note: enclose 'ptsname_r' in a __builtin_available check to silence this warningcpython-3.13>  8659 |     ret = ptsname_r(fd, name, sizeof(name));cpython-3.13>       |           ^~~~~~~~~

which seems relatively straight-forward in that we set a target min deployment of macOS 10.9.0 but they're using an item that's declared in macOS 10.13.4. I'm not yet sure what the pattern is to resolve this.

We define the min version at

-'-mmacosx-version-min=10.9'

10ee020 unconditionally bumps the minimum version — which is probably wrong but hopefully unblocks the issue for now.

Looking into a better solution at#8 — bumping the minimum version unleashed additional complexity.

@zanieb
Copy link
OwnerAuthor

zanieb commentedSep 4, 2024•
edited
Loading

The Windows failure is a bit unhinged

Building Project "C:\Users\RUNNER~1\AppData\Local\Temp\python-build-u2rnjqy8\Python-3.13.0rc1\PCbuild\pcbuild.proj" (Build target(s)) -- FAILED.

then lots of

      "C:\Users\RUNNER~1\AppData\Local\Temp\python-build-u2rnjqy8\Python-3.13.0rc1\PCbuild\pcbuild.proj" (Build target) (1) ->cpython>        "C:\Users\RUNNER~1\AppData\Local\Temp\python-build-u2rnjqy8\Python-3.13.0rc1\PCbuild\_freeze_module.vcxproj" (Build target) (2) ->cpython>        (_CheckWindowsSDKFound target) ->cpython>          C:\Users\RUNNER~1\AppData\Local\Temp\python-build-u2rnjqy8\Python-3.13.0rc1\PCbuild\python.props(132,5): warning : Failed to locate a Windows SDK installation. If the build fails, please use the Visual Studio Installer to install the Windows SDK. (Ignore the version number specified in the error message and select the latest.) [C:\Users\RUNNER~1\AppData\Local\Temp\python-build-u2rnjqy8\Python-3.13.0rc1\PCbuild\_freeze_module.vcxproj]cpython>

and finally

cpython>        "C:\Users\RUNNER~1\AppData\Local\Temp\python-build-u2rnjqy8\Python-3.13.0rc1\PCbuild\_decimal.vcxproj" (Build target) (18) ->cpython>        (ClCompile target) ->cpython>          c1 : fatal error C1083: Cannot open source file: 'C:\Users\runneradmin\AppData\Local\Temp\python-build-u2rnjqy8\Python-3.13.0rc1\externals\mpdecimal-4.0.0\libmpdec\constants.c': No such file or directory [C:\Users\RUNNER~1\AppData\Local\Temp\python-build-u2rnjqy8\Python-3.13.0rc1\PCbuild\_decimal.vcxproj]cpython>          c1 : fatal error C1083: Cannot open source file: 'C:\Users\runneradmin\AppData\Local\Temp\python-build-u2rnjqy8\Python-3.13.0rc1\externals\mpdecimal-4.0.0\libmpdec\context.c': No such file or directory [C:\Users\RUNNER~1\AppData\Local\Temp\python-build-u2rnjqy8\Python-3.13.0rc1\PCbuild\_decimal.vcxproj]cpython>          c1 : fatal error C1083: Cannot open source file: 'C:\Users\runneradmin\AppData\Local\Temp\python-build-u2rnjqy8\Python-3.13.0rc1\externals\mpdecimal-4.0.0\libmpdec\basearith.c': No such file or directory [C:\Users\RUNNER~1\AppData\Local\Temp\python-build-u2rnjqy8\Python-3.13.0rc1\PCbuild\_decimal.vcxproj]cpython>          c1 : fatal error C1083: Cannot open source file: 'C:\Users\runneradmin\AppData\Local\Temp\python-build-u2rnjqy8\Python-3.13.0rc1\externals\mpdecimal-4.0.0\libmpdec\convolute.c': No such file or directory [C:\Users\RUNNER~1\AppData\Local\Temp\python-build-u2rnjqy8\Python-3.13.0rc1\PCbuild\_decimal.vcxproj]cpython>          c1 : fatal error C1083: Cannot open source file: 'C:\Users\runneradmin\AppData\Local\Temp\python-build-u2rnjqy8\Python-3.13.0rc1\externals\mpdecimal-4.0.0\libmpdec\difradix2.c': No such file or directory [C:\Users\RUNNER~1\AppData\Local\Temp\python-build-u2rnjqy8\Python-3.13.0rc1\PCbuild\_decimal.vcxproj]cpython>          c1 : fatal error C1083: Cannot open source file: 'C:\Users\runneradmin\AppData\Local\Temp\python-build-u2rnjqy8\Python-3.13.0rc1\externals\mpdecimal-4.0.0\libmpdec\crt.c': No such file or directory [C:\Users\RUNNER~1\AppData\Local\Temp\python-build-u2rnjqy8\Python-3.13.0rc1\PCbuild\_decimal.vcxproj]cpython>          c1 : fatal error C1083: Cannot open source file: 'C:\Users\runneradmin\AppData\Local\Temp\python-build-u2rnjqy8\Python-3.13.0rc1\externals\mpdecimal-4.0.0\libmpdec\fnt.c': No such file or directory [C:\Users\RUNNER~1\AppData\Local\Temp\python-build-u2rnjqy8\Python-3.13.0rc1\PCbuild\_decimal.vcxproj]cpython>          c1 : fatal error C1083: Cannot open source file: 'C:\Users\runneradmin\AppData\Local\Temp\python-build-u2rnjqy8\Python-3.13.0rc1\externals\mpdecimal-4.0.0\libmpdec\io.c': No such file or directory [C:\Users\RUNNER~1\AppData\Local\Temp\python-build-u2rnjqy8\Python-3.13.0rc1\PCbuild\_decimal.vcxproj]cpython>          c1 : fatal error C1083: Cannot open source file: 'C:\Users\runneradmin\AppData\Local\Temp\python-build-u2rnjqy8\Python-3.13.0rc1\externals\mpdecimal-4.0.0\libmpdec\fourstep.c': No such file or directory [C:\Users\RUNNER~1\AppData\Local\Temp\python-build-u2rnjqy8\Python-3.13.0rc1\PCbuild\_decimal.vcxproj]cpython>          c1 : fatal error C1083: Cannot open source file: 'C:\Users\runneradmin\AppData\Local\Temp\python-build-u2rnjqy8\Python-3.13.0rc1\externals\mpdecimal-4.0.0\libmpdec\mpalloc.c': No such file or directory [C:\Users\RUNNER~1\AppData\Local\Temp\python-build-u2rnjqy8\Python-3.13.0rc1\PCbuild\_decimal.vcxproj]cpython>          c1 : fatal error C1083: Cannot open source file: 'C:\Users\runneradmin\AppData\Local\Temp\python-build-u2rnjqy8\Python-3.13.0rc1\externals\mpdecimal-4.0.0\libmpdec\mpdecimal.c': No such file or directory [C:\Users\RUNNER~1\AppData\Local\Temp\python-build-u2rnjqy8\Python-3.13.0rc1\PCbuild\_decimal.vcxproj]cpython>          c1 : fatal error C1083: Cannot open source file: 'C:\Users\runneradmin\AppData\Local\Temp\python-build-u2rnjqy8\Python-3.13.0rc1\externals\mpdecimal-4.0.0\libmpdec\numbertheory.c': No such file or directory [C:\Users\RUNNER~1\AppData\Local\Temp\python-build-u2rnjqy8\Python-3.13.0rc1\PCbuild\_decimal.vcxproj]cpython>          c1 : fatal error C1083: Cannot open source file: 'C:\Users\runneradmin\AppData\Local\Temp\python-build-u2rnjqy8\Python-3.13.0rc1\externals\mpdecimal-4.0.0\libmpdec\sixstep.c': No such file or directory [C:\Users\RUNNER~1\AppData\Local\Temp\python-build-u2rnjqy8\Python-3.13.0rc1\PCbuild\_decimal.vcxproj]cpython>          c1 : fatal error C1083: Cannot open source file: 'C:\Users\runneradmin\AppData\Local\Temp\python-build-u2rnjqy8\Python-3.13.0rc1\externals\mpdecimal-4.0.0\libmpdec\transpose.c': No such file or directory [C:\Users\RUNNER~1\AppData\Local\Temp\python-build-u2rnjqy8\Python-3.13.0rc1\PCbuild\_decimal.vcxproj]cpython>          C:\Users\runneradmin\AppData\Local\Temp\python-build-u2rnjqy8\Python-3.13.0rc1\Modules\_decimal\windows\mpdecimal.h(13,14): fatal error C1083: Cannot open include file: 'mpdecimal32vc.h': No such file or directory (compiling source file ..\Modules\_decimal\_decimal.c) [C:\Users\RUNNER~1\AppData\Local\Temp\python-build-u2rnjqy8\Python-3.13.0rc1\PCbuild\_decimal.vcxproj]

The last error looks related to a change todisable bundled libmpdecimal by default which is okay in Unix because we use an external distribution

but it doesn't look like we use that in Windows.

Ind33c58b, I try to just turn this off and use the bundled one again since it'll be around until Python 3.15. If that works, we can follow up with using an externallibmpdecimal as we do in Unix — I'm just not familiar with the Windows build code yet. Exploring the build in#7 — which is looking promising.

Resolved by#7

@zanieb
Copy link
OwnerAuthor

zanieb commentedSep 5, 2024•
edited
Loading

(x86_64_v4-unknown-linux-musl, cpython-3.13, noopt) is failing with

cpython-3.13> checking for multiarch...cpython-3.13> configure: error: internal configure error for the platform triplet, please file a bug reportTraceback (most recent call last):  File "/home/runner/work/python-build-standalone/python-build-standalone/cpython-unix/build.py", line 1219, in <module>    sys.exit(main())             ^^^^^^  File "/home/runner/work/python-build-standalone/python-build-standalone/cpython-unix/build.py", line 1201, in main    build_cpython(  File "/home/runner/work/python-build-standalone/python-build-standalone/cpython-unix/build.py", line 803, in build_cpython    build_env.run("build-cpython.sh", environment=env)  File "/home/runner/work/python-build-standalone/python-build-standalone/pythonbuild/buildenv.py", line 95, in run    container_exec(self.container, program, user=user, environment=environment)  File "/home/runner/work/python-build-standalone/python-build-standalone/pythonbuild/docker.py", line 140, in container_exec    raise Exception("exit code %d from %s" % (inspect_res["ExitCode"], command))Exception: exit code 1 from /build/build-cpython.shmake: *** [Makefile:325: /home/runner/work/python-build-standalone/python-build-standalone/build/cpython-3.13.0rc1-x86_64_v4-unknown-linux-musl-noopt.tar] Error 1

As well as(x86_64_v4-unknown-linux-musl, cpython-3.13, lto)

This looks related topython/cpython#107221 which changed the platform detection on 3.13

With7c3c95d this is resolved and we're encountering

2024-09-05T14:23:59.6617409Z cpython-3.13> copying /tmp/tmp5nezm6gr to container:/build/out/python/PYTHON.json2024-09-05T14:24:03.4884042Z Traceback (most recent call last):2024-09-05T14:24:03.4894989Z   File "/home/runner/work/python-build-standalone/python-build-standalone/cpython-unix/build.py", line 1219, in <module>2024-09-05T14:24:03.4896273Z     sys.exit(main())2024-09-05T14:24:03.4896672Z              ^^^^^^2024-09-05T14:24:03.4897940Z   File "/home/runner/work/python-build-standalone/python-build-standalone/cpython-unix/build.py", line 1201, in main2024-09-05T14:24:03.4898797Z     build_cpython(2024-09-05T14:24:03.4899748Z   File "/home/runner/work/python-build-standalone/python-build-standalone/cpython-unix/build.py", line 898, in build_cpython2024-09-05T14:24:03.4900543Z     fh.write(build_env.get_output_archive("python"))2024-09-05T14:24:03.4900934Z              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^2024-09-05T14:24:03.4901818Z   File "/home/runner/work/python-build-standalone/python-build-standalone/pythonbuild/buildenv.py", line 122, in get_output_archive2024-09-05T14:24:03.4902594Z     data = normalize_tar_archive(data)2024-09-05T14:24:03.4902911Z            ^^^^^^^^^^^^^^^^^^^^^^^^^^^2024-09-05T14:24:03.4903753Z   File "/home/runner/work/python-build-standalone/python-build-standalone/pythonbuild/utils.py", line 356, in normalize_tar_archive2024-09-05T14:24:03.4904496Z     filedata = tf.extractfile(ti)2024-09-05T14:24:03.4904793Z                ^^^^^^^^^^^^^^^^^^2024-09-05T14:24:03.4905375Z   File "/opt/hostedtoolcache/Python/3.11.9/x64/lib/python3.11/tarfile.py", line 2381, in extractfile2024-09-05T14:24:03.5511063Z     return self.extractfile(self._find_link_target(tarinfo))2024-09-05T14:24:03.5512044Z                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^2024-09-05T14:24:03.5513593Z   File "/opt/hostedtoolcache/Python/3.11.9/x64/lib/python3.11/tarfile.py", line 2723, in _find_link_target2024-09-05T14:24:03.5516017Z     raise KeyError("linkname %r not found" % linkname)2024-09-05T14:24:03.5517440Z KeyError: "linkname 'python/install/lib/python3.13/config-3.13-x86_64-linux-gnu/libpython3.13.a' not found"2024-09-05T14:24:03.5972138Z make: *** [Makefile:325: /home/runner/work/python-build-standalone/python-build-standalone/build/cpython-3.13.0rc1-x86_64_v4-unknown-linux-musl-noopt.tar] Error 1

Which seems to indicate that the link created with

ln -sf python3.13/config-3.13-x86_64-linux-gnu/libpython3.13.a /build/out/python/install/lib/libpython3.13.a

Does not have a valid target. However... the logs look about the same as the successful 3.12 build so I'm not sure what's going on yet.

I'm not sure what changed upstream.

931009e had no affect. Not did1d8da17 because the target triple includes GNU.4a78a31 did it though.

@zanieb
Copy link
OwnerAuthor

zanieb commentedSep 5, 2024•
edited
Loading

(x86_64-unknown-linux-gnu, cpython-3.13, pgo, true) is failing with

cpython-3.13> 0:00:21 load avg: 2.28 [16/44] test_embedcpython-3.13> Fatal Python error: Segmentation faultcpython-3.13> cpython-3.13> Current thread 0x00007fe7f6c36740 (most recent call first):cpython-3.13>   File "/build/Python-3.13.0rc1/Lib/test/test_embed.py", line 1676 in test_global_pathconfigcpython-3.13>   File "/build/Python-3.13.0rc1/Lib/unittest/case.py", line 606 in _callTestMethodcpython-3.13>   File "/build/Python-3.13.0rc1/Lib/unittest/case.py", line 651 in runcpython-3.13>   File "/build/Python-3.13.0rc1/Lib/unittest/case.py", line 707 in __call__cpython-3.13>   File "/build/Python-3.13.0rc1/Lib/unittest/suite.py", line 122 in runcpython-3.13>   File "/build/Python-3.13.0rc1/Lib/unittest/suite.py", line 84 in __call__cpython-3.13>   File "/build/Python-3.13.0rc1/Lib/unittest/suite.py", line 122 in runcpython-3.13>   File "/build/Python-3.13.0rc1/Lib/unittest/suite.py", line 84 in __call__cpython-3.13>   File "/build/Python-3.13.0rc1/Lib/test/libregrtest/testresult.py", line 148 in runcpython-3.13>   File "/build/Python-3.13.0rc1/Lib/test/libregrtest/single.py", line 57 in _run_suitecpython-3.13>   File "/build/Python-3.13.0rc1/Lib/test/libregrtest/single.py", line 37 in run_unittestcpython-3.13>   File "/build/Python-3.13.0rc1/Lib/test/libregrtest/single.py", line 135 in test_funccpython-3.13>   File "/build/Python-3.13.0rc1/Lib/test/libregrtest/single.py", line 91 in regrtest_runnercpython-3.13>   File "/build/Python-3.13.0rc1/Lib/test/libregrtest/single.py"cpython-3.13> , line 138 in _load_run_testcpython-3.13>   File "/build/Python-3.13.0rc1/Lib/test/libregrtest/single.py", line 181 in _runtest_env_changed_exccpython-3.13>   File "/build/Python-3.13.0rc1/Lib/test/libregrtest/single.py", line 281 in _runtestcpython-3.13>   File "/build/Python-3.13.0rc1/Lib/test/libregrtest/single.py", line 310 in run_single_testcpython-3.13>   Filecpython-3.13> "/build/Python-3.13.0rc1/Lib/test/libregrtest/main.py", line 355 incpython-3.13> run_testcpython-3.13>   File "/build/Python-3.13.0rc1/Lib/test/libregrtest/main.py", linecpython-3.13> 389 in run_tests_sequentiallycpython-3.13>   File "/build/Python-3.13.0rc1/Lib/test/libregrtest/main.pycpython-3.13> ", linecpython-3.13> 533 in _run_testscpython-3.13>   File "/build/Python-3.13.0rc1/Lib/test/libregrtest/main.py"cpython-3.13> , line 568 in run_testscpython-3.13>   Filecpython-3.13> "/build/Python-3.13.0rc1/Lib/test/libregrtest/main.py", line 731 incpython-3.13> maincpython-3.13>   File "/build/Python-3.13.0rc1/Lib/test/libregrtest/main.py", line 739cpython-3.13> in maincpython-3.13>   File "/build/Python-3.13.0rc1/Lib/test/__main__.pycpython-3.13> ", line 2 in <module>cpython-3.13>   Filecpython-3.13> "<frozen runpy>", line 88cpython-3.13> in _run_codecpython-3.13>   File "<frozen runpy>cpython-3.13> ", line 198 in _run_module_as_maincpython-3.13> Extension modules: _testinternalcapicpython-3.13> (total: 1)cpython-3.13> Segmentation fault (core dumped)cpython-3.13> make[1]: *** [profile-bolt-stamp] Error 139cpython-3.13> Makefile:884: recipe for target 'profile-bolt-stamp' failedcpython-3.13> make[1]: Leaving directory '/build/Python-3.13.0rc1'cpython-3.13> make: *** [bolt-opt] Error 2cpython-3.13> Makefile:915: recipe for target 'bolt-opt' failed

Much discussion about this follows in the thread, concluding at#6 (comment)

* Attempt to download and build `mpdecimal` on Windows* Patch `mpdecimal` in `python.props`* Remove `_msi` from expected global extensions* Allow `_msi` to be missing during `collect_python_build_artifacts`* Add `_testclinic_limited` and `_testlimitedcapi` to ignored extensions* Add `python313.dll` to allowed libraries* Fix expected global extensions for 3.13* Allow `_PyWarnings_Init` to be missing on 3.13* Include 3.8 in `_PyWarnings_Init` case* Include `_xxsubinterpreters` in 3.8 extensions
@vstinner
Copy link

I don't know thepython-build-standalone project, but it seems like it has tons of patches on top of Python. Maybe one of them is causing your failures (like the segfault).

See alsopython/cpython#119696 which discuss making these patches upstream. I asked basic questions, but so far, no one replied:

Can you please describe what is a "relocatable Python installation"? What are the current issues that you're trying to solve? Why is it important for Python to support such installation?

@vstinner
Copy link

You could try building with the --no-docker flag, but I've never tried that.

It fails differently:

make: *** No rule to make target '/home/vstinner/dev/python-build-standalone/build/image-gcc.tar', needed by '/home/vstinner/dev/python-build-standalone/build/binutils-2.42-linux64.tar'.  Stop.

@zanieb
Copy link
OwnerAuthor

Here's a patch to actually show the Docker error correctlyastral-sh#315

I can go through the CPython patches and see if one seems relevant. Generally I'm not familiar with contents of the patches though.

See alsopython/cpython#119696 ...

We'll chime in there. I don't understand why nobody has answered your basic questions — but we are happy to engage.

vstinner reacted with thumbs up emojiAlexWaygood reacted with heart emoji

@vstinner
Copy link

I failed to reproduce the segfault on Linux without optimization (without PGO).

I built Python 3.13 withpython build-linux.py --python cpython-3.13 on Linux. I decompressed it in RELOCATE/:

mkdir RELOCATEcd RELOCATEtar -xf ../dist/cpython-3.13.0rc1-x86_64-unknown-linux-gnu-noopt-20240905T1235.tar.zstcd python/

I createdscript.py:

import os.pathimport _testinternalcapidef assertEqual(a, b):    assert a == b, "%r != %r" % (a, b)def test_global_pathconfig():    # Test C API functions getting the path configuration:    #    # - Py_GetExecPrefix()    # - Py_GetPath()    # - Py_GetPrefix()    # - Py_GetProgramFullPath()    # - Py_GetProgramName()    # - Py_GetPythonHome()    #    # The global path configuration (_Py_path_config) must be a copy    # of the path configuration of PyInterpreter.config (PyConfig).    import ctypes    def get_func(name):        func = getattr(ctypes.pythonapi, name)        func.argtypes = ()        func.restype = ctypes.c_wchar_p        return func    Py_GetPath = get_func('Py_GetPath')    Py_GetPrefix = get_func('Py_GetPrefix')    Py_GetExecPrefix = get_func('Py_GetExecPrefix')    Py_GetProgramName = get_func('Py_GetProgramName')    Py_GetProgramFullPath = get_func('Py_GetProgramFullPath')    Py_GetPythonHome = get_func('Py_GetPythonHome')    config = _testinternalcapi.get_configs()['config']    assertEqual(list(Py_GetPath().split(os.path.pathsep)),                     config['module_search_paths'])    assertEqual(Py_GetPrefix(), config['prefix'])    assertEqual(Py_GetExecPrefix(), config['exec_prefix'])    assertEqual(Py_GetProgramName(), config['program_name'])    assertEqual(Py_GetProgramFullPath(), config['executable'])    assertEqual(Py_GetPythonHome(), config['home'])    print("OK!")test_global_pathconfig()

The script runs successfully:

$ ./install/bin/python script.py OK!

Note: I cannot runpython -m test -v test_embed: all tests are skipped since the_testembed program is missing from the archive (well, it's only used for tests, so it's normal that it's missing).

@zanieb
Copy link
OwnerAuthor

Interesting! Thank you for sharing those details. I am a bit surprised by this — I was about to conjecture that we only see failures in the PGO builds because we don't actually (as far as I know) run the tests in other builds. It also seems surprising that the build that we're profiling for PGO would be different than the base build? I'll see if I can reproduce the macOS test failures locally.

@vstinner
Copy link

Ah, I can reproduce the crash when building with PGO optimization on my Linux Fedora 40:

$ python build-linux.py --python cpython-3.13 --optimizations pgo(...)cpython-3.13> 0:00:19 load avg: 5.15 [14/44] test_decimalcpython-3.13> 0:00:24 load avg: 4.82 [15/44] test_difflibcpython-3.13> 0:00:25 load avg: 4.51 [16/44] test_embedcpython-3.13> Fatal Python error: Segmentation faultcpython-3.13> cpython-3.13> Current thread 0x00007fa38fd1b740 (most recent call first):cpython-3.13>   File "/build/Python-3.13.0rc1/Lib/test/test_embed.py", line 1676 in test_global_pathconfigcpython-3.13>   File "/build/Python-3.13.0rc1/Lib/unittest/case.py", line 606 in _callTestMethod(...)

All containers are destroyed whenbuild-linux.py completes, so I don't know how to debug that.

@zanieb
Copy link
OwnerAuthor

Okay, similarly if I build with

❯ ./build-macos.py \    --target-triple aarch64-apple-darwin \    --python cpython-3.13 --optimizations debug

Then follow the instructions you wrote there the test passes without issue. Of course, the test I pulled a failure for on macOS is different (test_preinit_python_config) and I'm not sure how to transpose that into a script like you did. There were 40 other failing tests though.

I'm not sure if I should spend time trying to getPrograms/_testembed to be exported with the distribution so we can test against it?

@zanieb
Copy link
OwnerAuthor

You may be able to keep the container around by applying

diff --git a/pythonbuild/buildenv.py b/pythonbuild/buildenv.pyindex 0ce3cd0..03860e9 100644--- a/pythonbuild/buildenv.py+++ b/pythonbuild/buildenv.py@@ -274,7 +274,6 @@ def build_environment(client, image):         yield context     finally:         if container:-            container.stop(timeout=0)-            container.remove()+            pass         else:             td.cleanup()

@zanieb
Copy link
OwnerAuthor

If the test passes in non-PGO build but fails in a PGO build does that mean that the BOLT instrumentation is causing the problem? Does applying this to the passing debug binary cause it to fail the test?

2024-09-05T18:56:18.3656553Z cpython-3.13> # Instrument each binary.2024-09-05T18:56:18.3666980Z cpython-3.13> for bin in python libpython3.13.so.1.0; do \2024-09-05T18:56:18.3669004Z cpython-3.13>   /tools/llvm/bin/llvm-bolt "${bin}" -instrument -instrumentation-file-append-pid -instrumentation-file=/build/Python-3.13.0rc1/${bin}.bolt -o ${bin}.bolt_inst ; \2024-09-05T18:56:18.3670071Z cpython-3.13>   mv "${bin}.bolt_inst" "${bin}"; \2024-09-05T18:56:18.3670477Z cpython-3.13> done

@vstinner
Copy link

I built Python 3.13.0rc1 on Fedora 40 with the following patches but I reverted configure (to avoid enforced cross-compilation).

patch-always-build-python-for-freeze.patchpatch-checksharedmods-disable.patchpatch-configure-disable-stdlib-mod-3.12.patchpatch-ctypes-callproc.patchpatch-ctypes-static-binary.patchpatch-disable-multiarch-13.patchpatch-dont-clear-runshared-13.patchpatch-force-cross-compile.patchpatch-macos-link-extension-modules-13.patchpatch-makesetup-deduplicate-objs.patchpatch-pwd-remove-conditional.patchpatch-python-link-modules-3.11.patchpatch-readline-libedit.patchpatch-write-python-for-build-3.12.patch

I fail to reproduce thetest_embed crash with these patches. I built Python with./configure --enable-optimizations && make.

Note: It seems like python-build-standalone is using musl C library instead of the glibc C library. I tried building Python withmusl-gcc but it failed with:

./Modules/_bz2module.c:9:10: fatal error: bzlib.h: No such file or directory./Modules/binascii.c:64:12: fatal error: zlib.h: No such file or directory./Modules/_lzmamodule.c:18:10: fatal error: lzma.h: No such file or directory

@vstinner
Copy link

If the test passes in non-PGO build but fails in a PGO build does that mean that the BOLT instrumentation is causing the problem?

Maybe. Try to disable BOLT and see if you can still reproduce the issue? Then try to disable some patches and see if you can reproduce the issue. If yes, disable more patches. etc.

I think that I spent enough time attempting to reproduce the issue, I'm clueless at this point. Good luck to identify the root cause :-)

@zanieb
Copy link
OwnerAuthor

Thanks for trying! I appreciate it. I'll ping you if I learn anything interesting.

@zanieb
Copy link
OwnerAuthor

zanieb commentedSep 5, 2024•
edited
Loading

It looks like I was correct and BOLT is the problem, the following succeeded for me:

diff --git a/cpython-unix/build-cpython.sh b/cpython-unix/build-cpython.shindex 8cb47ae..da40703 100755--- a/cpython-unix/build-cpython.sh+++ b/cpython-unix/build-cpython.sh@@ -352,9 +352,6 @@ fi  if [ -n "${CPYTHON_OPTIMIZED}" ]; then     CONFIGURE_FLAGS="${CONFIGURE_FLAGS} --enable-optimizations"-    if [[ -n "${PYTHON_MEETS_MINIMUM_VERSION_3_12}" && -n "${BOLT_CAPABLE}" ]]; then-        CONFIGURE_FLAGS="${CONFIGURE_FLAGS} --enable-bolt"-    fi fi  if [ -n "${CPYTHON_LTO}" ]; then
./build-linux.py --python cpython-3.13 --optimizations pgo --target-triple x86_64-unknown-linux-gnu

Maybe helpfulllvm/llvm-project#59025

Unfortunately this isn't the problem on macOS so I'll need to figure something else out there still.

@zaniebzanieb mentioned this pull requestSep 5, 2024
@zanieb
Copy link
OwnerAuthor

zanieb commentedSep 5, 2024•
edited
Loading

Okay the issue on aarch64-macos with the duplicate implementations is that the tcl/tk libraries are linked twice. This ugly patch resolves it

diff --git a/Makefile.pre.in b/Makefile.pre.inindex f9932ddab9d..ba73e3d62a0 100644--- a/Makefile.pre.in+++ b/Makefile.pre.in@@ -1432,6 +1432,12 @@ regen-re: $(BUILDPYTHON)        $(RUNSHARED) ./$(BUILDPYTHON) $(srcdir)/Tools/build/generate_re_casefix.py $(srcdir)/Lib/re/_casefix.py  Programs/_testembed: Programs/_testembed.o $(LINK_PYTHON_DEPS)+       $(eval SYSLIBS := $(subst -Xlinker -hidden-ltcl8.6, , $(SYSLIBS)))+       $(eval SYSLIBS := $(subst -Xlinker -hidden-ltk8.6, , $(SYSLIBS)))+       $(eval LIBS := $(subst -Xlinker -hidden-ltcl8.6, , $(LIBS)))+       $(eval LIBS := $(subst -Xlinker -hidden-ltk8.6, , $(LIBS)))+       $(eval MODLIBS := $(subst -Xlinker -hidden-ltcl8.6, , $(MODLIBS)))+       $(eval MODLIBS := $(subst -Xlinker -hidden-ltk8.6, , $(MODLIBS)))        $(LINKCC) $(PY_CORE_LDFLAGS) $(LINKFORSHARED) -o $@ Programs/_testembed.o $(LINK_PYTHON_OBJS) $(LIBS) $(MODLIBS) $(SYSLIBS)  ############################################################################

I'll narrow this down to a single variable...

charliermarsh pushed a commit to astral-sh/python-build-standalone that referenced this pull requestSep 6, 2024
A continuation of the wonderful work by@kpfleming in#264Adds initial support for Python 3.13 using CPython 3.13.0rc1.There are a few caveats and interesting details:- BOLT is disabled. There's a segmentation fault in the tests with the BOLT instrumented binaries. The BOLT optimizations are not critical, so we'll follow up on this separately. [See more context](zanieb#6 (comment)).- `mpdecimal` is now built from source on Windows. We already did this in Unix builds, but in Windows we were still using the bundled library. The bundled library is no longer used upstream and it seemed prudent to switch though it will remain available until 3.15. [See more context](zanieb#6 (comment)).- Apple cross-compilation is not available. I have a patch that adds support, but need to test it and it's not needed for these builds. [See more context](zanieb@447fb86).- `run_tests.py` was removed upstream. We provide a compatibility script that calls the appropriate command still so that our distributions are stable. We may want to change how `run_tests.py` is declared in the distribution metadata though. [See more context](#319 (comment)) .
@zaniebzanieb closed thisSep 16, 2024
Sign up for freeto join this conversation on GitHub. Already have an account?Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@zanieb@vstinner@kpfleming
[8]ページ先頭
©2009-2025 Movatter.jp