Movatterモバイル変換

[0]ホーム
URL:

Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

 Sign up
Appearance settings

gh-90905: Allow cross-compilation on macOS#128385

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to ourterms of service andprivacy statement. We’ll occasionally send you account related emails.

Already on GitHub?Sign in to your account

Merged
ned-deily merged 3 commits intopython:mainfromzanieb:zb/cross-apple
Jan 13, 2025

Conversation

@zanieb
Copy link
Contributor

@zaniebzanieb commentedDec 31, 2024•
edited
Loading

Adds support for cross-compiling x86_64 from arm64 macOS. This is particularly useful due to the decreasing availability of x86_64 machines to perform builds on.

The changes here are loosely based on thepatch thatpython-build-standalone uses for cross-compiling.

I tested compiling x86_64 from an arm64 machine with the following script:

#!/usr/bin/env bashset -exROOT=$(pwd)OUT=${ROOT}/../cpython-out# Assume that x86_64 Homebrew is installedibrew() { arch -x86_64 /usr/local/bin/brew"$@"; }export -f ibrewmkdir -p${OUT}# Compile the "build" aarch64 Pythonmake cleanrm -rf${OUT}/cpython-aarch64# Exclude modules we don't need that require system dependencies./configure \    py_cv_module__openssl=n/a \    py_cv_module__hashlib=n/a \    py_cv_module__gdbm=n/a \    --without-ensurepip \    --prefix${OUT}/cpython-aarch64make -j8make -j sharedinstallmake -j install# Build the "host" x86_64 Pythonmake cleanrm -rf${OUT}/cpython-x86_64CFLAGS="-arch x86_64" \LDFLAGS="-arch x86_64" \MACOSX_DEPLOYMENT_TARGET=10.15 \GDBM_CFLAGS="-I$(ibrew --prefix gdbm)/include" \GDBM_LIBS="-L$(ibrew --prefix gdbm)/lib -lgdbm" \PKG_CONFIG="$(ibrew --prefix pkg-config)/bin/pkg-config" \./configure \    --with-pydebug \    --with-system-libmpdec \    --with-openssl="$(ibrew --prefix openssl@3)" \    --build=aarch64-apple-darwin \    --host=x86_64-apple-darwin \    --prefix${OUT}/cpython-x86_64 \    --with-build-python=${OUT}/cpython-aarch64/bin/python3 \    --disable-ipv6 \    ac_cv_file__dev_ptc=no \    ac_cv_file__dev_ptmx=no \    ac_cv_func_sendfile=nomake -j8make -j sharedinstallmake -j install${OUT}/cpython-x86_64/bin/python3 -c"import platform; print(platform.machine())"

@zanieb
Copy link
ContributorAuthor

zanieb commentedJan 1, 2025•
edited
Loading

This seems a bit too easy. I'm curious if I'm missing something?

The justification for the configure flags are as follows:

--disable-ipv6

configure: error: You must get working getaddrinfo() function or pass the "--disable-ipv6" option to configure.

ac_cv_file__dev_ptmx=no

checking for /dev/ptmx... not setconfigure: error: set ac_cv_file__dev_ptmx to yes/no in your CONFIG_SITE file when cross compiling

ac_cv_file__dev_ptc=no

checking for /dev/ptc... not setconfigure: error: set ac_cv_file__dev_ptc to yes/no in your CONFIG_SITE file when cross compiling

ac_cv_func_sendfile=no

./Modules/posixmodule.c:11871:15: error: call to undeclared function 'sendfile'; ISO C99 and later do not support implicit function declarations [-Wimplicit-function-declaration]    ret = sendfile(in_fd, out_fd, offset, &sbytes, &sf, flags);            ^

It's unclear to me if it's expected that the-arch needs to be added toCFLAGS andLDFLAGS explicitly.

I also checked thePlatform in thesysconfig, as I know we have an extra patch for this inpython-build-standalone but it doesn't look necessary here:

❯ ../cpython-out/cpython-x86_64/bin/python3 -m sysconfig | grep PlatformPlatform: "macosx-10.15-x86_64"

@zaniebzanieb marked this pull request as ready for reviewJanuary 1, 2025 04:59
@corona10corona10 requested a review froma teamJanuary 1, 2025 09:11
@corona10
Copy link
Member

I think that@ned-deily and@ronaldoussoren have interests with this patch.

Copy link
Member

@FFY00FFY00 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

The change looks correct to me.

I'll approve, but give@ned-deily some time to look it over before merging.@zanieb if this doesn't happen within a week, please ping me to move it forward. Thanks!

zanieb reacted with heart emoji
@ned-deily
Copy link
Member

Thanks for the PR. The change looks OK so I'm merging it for 3.14.

But, IMO, I wouldn't recommend using this approach for most uses. There is a reason that we haven't bothered to support the "traditional" style of cross-building for macOS builds and that is because essentially the same result can be accomplished through the use of macOS universal builds, something that has been around in Python for many years and which is robust and well-tested. For example, the currentuniversal2 flavor of universal builds produces fat builds including botharm64 andx86_64 architectures that can be built on either kind of Mac and will then run natively on both archs. The main differences are that the installed binary files are larger, likely negligible in most cases although the resultant binaries could be lipo-ed into single arch binaries, and it is necessary to build with universal builds of the third-party libraries used by the standard library that are not provided by macOS (i.e. openssl, xz, gdbm, tcl-tk, mpdecimal). AFAIK, Homebrew does not support universal builds of those libs; MacPorts does. We have had discussions about providing pre-built universal macOS and iOS binaries of at least some of these libs; that may yet happen in the coming months.

In the meantime, it might be a very useful interim experiment for someone to try to provide a script thatlipo's and, as needed,install_name_tool's the Homebrew arm64 (/opt/homebrew) and x86_64 (/usr/local) versions of those few libs we need so that they could be cached and used in a GHA CI job to add macOS universal testing.

@ned-deilyned-deily merged commit6ecb620 intopython:mainJan 13, 2025
41 checks passed
@ned-deily
Copy link
Member

As an example, a very simple universal build rough equivalent:

make clean || true# If you need GDBM, uncomment the next two lines for MacPorts and/or edit.# GDBM_CFLAGS="-I$(dirname $(dirname $(which port)))/include" \# GDBM_LIBS="-L$(dirname $(dirname $(which port)))/lib -lgdbm" \MACOSX_DEPLOYMENT_TARGET=10.15 \./configure \    --with-pydebug \    --enable-universalsdk \    --with-universal-archs=universal2 \    --with-system-libmpdec \    --prefix ${OUT}/cpython-x86_64 \make -j8make -j install${OUT}/cpython-x86_64/bin/python3  -c "import platform; print(platform.machine())"${OUT}/cpython-x86_64/bin/python3-intel64  -c "import platform; print(platform.machine())"

And, the one-time commands to build and install the universal libs from MacPorts:

#!/bin/shset -ex# Assume that MacPorts base is installed and that "port" is on $PATH#   (https://www.macports.org/install.php)# Update ports infosudo port selfupdate# Install Python dependenciessudo port -N install \    pkgconfig \    openssl +universal \    xz +universal \    gdbm +universal \    mpdecimal +universal \    # Tk disabled at the moment due to open MacPorts issue with univeral variant:    # https://trac.macports.org/ticket/71415    # tk +quartz +universal
zanieb reacted with heart emoji

@bedevere-bot
Copy link

⚠️⚠️⚠️ Buildbot failure⚠️⚠️⚠️

Hi! The buildbotAMD64 Debian root 3.x has failed when building commit6ecb620.

What do you need to do:

  1. Don't panic.
  2. Checkthe buildbot page in the devguide if you don't know what the buildbots are or how they work.
  3. Go to the page of the buildbot that failed (https://buildbot.python.org/#/builders/345/builds/9923) and take a look at the build logs.
  4. Check if the failure is related to this commit (6ecb620) or if it is a false positive.
  5. If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/#/builders/345/builds/9923

Failed tests:

  • test.test_multiprocessing_forkserver.test_misc

Failed subtests:

  • test_large_pool - test.test_multiprocessing_forkserver.test_misc.MiscTestCase.test_large_pool

Summary of the results of the build (if available):

==

Click to see traceback logs
Traceback (most recent call last):  File"/root/buildarea/3.x.angelico-debian-amd64/build/Lib/test/_test_multiprocessing.py", line6607, intest_large_pool    rc, out, err= script_helper.assert_python_ok(testfn)~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^  File"/root/buildarea/3.x.angelico-debian-amd64/build/Lib/test/support/script_helper.py", line182, inassert_python_okreturn _assert_python(True,*args,**env_vars)  File"/root/buildarea/3.x.angelico-debian-amd64/build/Lib/test/support/script_helper.py", line167, in_assert_python    res.fail(cmd_line)~~~~~~~~^^^^^^^^^^  File"/root/buildarea/3.x.angelico-debian-amd64/build/Lib/test/support/script_helper.py", line80, infailraiseAssertionError(f"Process return code is{exitcode}\n"...<10 lines>...f"---")AssertionError:Process return code is 1command line: ['/root/buildarea/3.x.angelico-debian-amd64/build/python', '-X', 'faulthandler', '-I', '@test_3507502_tmpæ']Traceback (most recent call last):  File"<string>", line1, in<module>from multiprocessing.forkserverimport main; main(10,11, ['__main__'],**{'sys_path': ['/root/buildarea/3.x.angelico-debian-amd64/build/target/lib/python314.zip','/root/buildarea/3.x.angelico-debian-amd64/build/Lib','/root/buildarea/3.x.angelico-debian-amd64/build/build/lib.linux-x86_64-3.14'],'authkey_r':13})~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^  File"/root/buildarea/3.x.angelico-debian-amd64/build/Lib/multiprocessing/forkserver.py", line324, inmain    pid= os.fork()OSError:[Errno 12] Cannot allocate memoryTraceback (most recent call last):  File"/root/buildarea/3.x.angelico-debian-amd64/build/build/test_python_3517661æ/@test_3517661_tmpæ", line4, in<module>with multiprocessing.Pool(200)as p:~~~~~~~~~~~~~~~~~~~~^^^^^  File"/root/buildarea/3.x.angelico-debian-amd64/build/Lib/multiprocessing/context.py", line119, inPoolreturn Pool(processes, initializer, initargs, maxtasksperchild,                context=self.get_context())  File"/root/buildarea/3.x.angelico-debian-amd64/build/Lib/multiprocessing/pool.py", line215, in__init__self._repopulate_pool()~~~~~~~~~~~~~~~~~~~~~^^  File"/root/buildarea/3.x.angelico-debian-amd64/build/Lib/multiprocessing/pool.py", line306, in_repopulate_poolreturnself._repopulate_pool_static(self._ctx,self.Process,~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^self._processes,^^^^^^^^^^^^^^^^...<3 lines>...self._maxtasksperchild,^^^^^^^^^^^^^^^^^^^^^^^self._wrap_exception)^^^^^^^^^^^^^^^^^^^^^  File"/root/buildarea/3.x.angelico-debian-amd64/build/Lib/multiprocessing/pool.py", line329, in_repopulate_pool_static    w.start()~~~~~~~^^  File"/root/buildarea/3.x.angelico-debian-amd64/build/Lib/multiprocessing/process.py", line121, instartself._popen=self._Popen(self)~~~~~~~~~~~^^^^^^  File"/root/buildarea/3.x.angelico-debian-amd64/build/Lib/multiprocessing/context.py", line300, in_Popenreturn Popen(process_obj)  File"/root/buildarea/3.x.angelico-debian-amd64/build/Lib/multiprocessing/popen_forkserver.py", line35, in__init__super().__init__(process_obj)~~~~~~~~~~~~~~~~^^^^^^^^^^^^^  File"/root/buildarea/3.x.angelico-debian-amd64/build/Lib/multiprocessing/popen_fork.py", line20, in__init__self._launch(process_obj)~~~~~~~~~~~~^^^^^^^^^^^^^  File"/root/buildarea/3.x.angelico-debian-amd64/build/Lib/multiprocessing/popen_forkserver.py", line59, in_launchself.pid= forkserver.read_signed(self.sentinel)~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^  File"/root/buildarea/3.x.angelico-debian-amd64/build/Lib/multiprocessing/forkserver.py", line390, inread_signedraiseEOFError('unexpected EOF')EOFError:unexpected EOF---Traceback (most recent call last):  File"/root/buildarea/3.x.angelico-debian-amd64/build/Lib/test/_test_multiprocessing.py", line6607, intest_large_pool    rc, out, err= script_helper.assert_python_ok(testfn)~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^  File"/root/buildarea/3.x.angelico-debian-amd64/build/Lib/test/support/script_helper.py", line182, inassert_python_okreturn _assert_python(True,*args,**env_vars)  File"/root/buildarea/3.x.angelico-debian-amd64/build/Lib/test/support/script_helper.py", line167, in_assert_python    res.fail(cmd_line)~~~~~~~~^^^^^^^^^^  File"/root/buildarea/3.x.angelico-debian-amd64/build/Lib/test/support/script_helper.py", line80, infailraiseAssertionError(f"Process return code is{exitcode}\n"...<10 lines>...f"---")AssertionError:Process return code is 1command line: ['/root/buildarea/3.x.angelico-debian-amd64/build/python', '-X', 'faulthandler', '-I', '@test_3517661_tmpæ']Traceback (most recent call last):  File"<string>", line1, in<module>from multiprocessing.forkserverimport main; main(10,11, ['__main__'],**{'sys_path': ['/root/buildarea/3.x.angelico-debian-amd64/build/target/lib/python314.zip','/root/buildarea/3.x.angelico-debian-amd64/build/Lib','/root/buildarea/3.x.angelico-debian-amd64/build/build/lib.linux-x86_64-3.14'],'authkey_r':13})~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^  File"/root/buildarea/3.x.angelico-debian-amd64/build/Lib/multiprocessing/forkserver.py", line324, inmain    pid= os.fork()OSError:[Errno 12] Cannot allocate memoryTraceback (most recent call last):  File"/root/buildarea/3.x.angelico-debian-amd64/build/build/test_python_3507502æ/@test_3507502_tmpæ", line4, in<module>with multiprocessing.Pool(200)as p:~~~~~~~~~~~~~~~~~~~~^^^^^  File"/root/buildarea/3.x.angelico-debian-amd64/build/Lib/multiprocessing/context.py", line119, inPoolreturn Pool(processes, initializer, initargs, maxtasksperchild,                context=self.get_context())  File"/root/buildarea/3.x.angelico-debian-amd64/build/Lib/multiprocessing/pool.py", line215, in__init__self._repopulate_pool()~~~~~~~~~~~~~~~~~~~~~^^  File"/root/buildarea/3.x.angelico-debian-amd64/build/Lib/multiprocessing/pool.py", line306, in_repopulate_poolreturnself._repopulate_pool_static(self._ctx,self.Process,~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^self._processes,^^^^^^^^^^^^^^^^...<3 lines>...self._maxtasksperchild,^^^^^^^^^^^^^^^^^^^^^^^self._wrap_exception)^^^^^^^^^^^^^^^^^^^^^  File"/root/buildarea/3.x.angelico-debian-amd64/build/Lib/multiprocessing/pool.py", line329, in_repopulate_pool_static    w.start()~~~~~~~^^  File"/root/buildarea/3.x.angelico-debian-amd64/build/Lib/multiprocessing/process.py", line121, instartself._popen=self._Popen(self)~~~~~~~~~~~^^^^^^  File"/root/buildarea/3.x.angelico-debian-amd64/build/Lib/multiprocessing/context.py", line300, in_Popenreturn Popen(process_obj)  File"/root/buildarea/3.x.angelico-debian-amd64/build/Lib/multiprocessing/popen_forkserver.py", line35, in__init__super().__init__(process_obj)~~~~~~~~~~~~~~~~^^^^^^^^^^^^^  File"/root/buildarea/3.x.angelico-debian-amd64/build/Lib/multiprocessing/popen_fork.py", line20, in__init__self._launch(process_obj)~~~~~~~~~~~~^^^^^^^^^^^^^  File"/root/buildarea/3.x.angelico-debian-amd64/build/Lib/multiprocessing/popen_forkserver.py", line59, in_launchself.pid= forkserver.read_signed(self.sentinel)~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^  File"/root/buildarea/3.x.angelico-debian-amd64/build/Lib/multiprocessing/forkserver.py", line390, inread_signedraiseEOFError('unexpected EOF')EOFError:unexpected EOF---

@zanieb
Copy link
ContributorAuthor

Thanks for the explanation@ned-deily — there is some discussion on universal builds downstreamastral-sh/python-build-standalone#140

@Andrej730
Copy link
Contributor

So#90905 can be closed?

Sign up for freeto join this conversation on GitHub. Already have an account?Sign in to comment

Reviewers

@FFY00FFY00FFY00 approved these changes

@erlend-aaslanderlend-aaslandAwaiting requested review from erlend-aaslanderlend-aasland is a code owner

@corona10corona10Awaiting requested review from corona10corona10 is a code owner

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@zanieb@corona10@ned-deily@bedevere-bot@Andrej730@FFY00
[8]ページ先頭
©2009-2025 Movatter.jp