This PR lays the groundwork for introducing iOS/tvOS/watchOS frameworks. It is a subset of#115063; that PR may be helpful for seeing the full context for some of the changes that have been made. This PR doesn't add anything iOS specific - it only includes the structural refactoring needed so that iOS branches can be added into in a subsequent PR.

Summary of changes:

Updatesconfig.sub to the 2024-01-01 release. This is the "as released" version ofconfig.sub.
Adds aRESSRCDIR variable to allow sharing of macOS and iOS Makefile steps
Adds anINSTALLTARGETS variable so platforms can customise which targets are actually installed. This will be used to exclude certain targets (e.g., binaries, manfiles) from iOS framework installs.
Adds aPYTHONFRAMEWORKINSTALLNAMEPREFIX variable; this is used as the install name for the library. This is needed to allow for iOS frameworks to specify an@rpath-based install name.
EvaluatesMACHDEP earlier in the configure process so thatac_sys_system is available
Modifies_PYTHON_HOST_PLATFORM evaluation for cross-platform builds so that the CPU architecture is differentiated from the host identifier. This will be used to generate a_PYTHON_HOST_PLATFORM definition that includes ABI information, not just CPU architecture.
Differentiates betweenSOABI_PLATFORM andPLATFORM_TRIPLET.SOABI_PLATFORM is used in binary module names, and includes the ABI, but not the OS or CPU architecture (e.g.,math.cpython-313-iphonesimulator.dylib).PLATFORM_TRIPLET is used as thesys._multiarch value, and on iOS will contains the ABI and architecture (e.g.,iphoneos-arm64). This differentiation hasn't historically been needed because while macOS is a multiarch platform, it uses a baredarwin asPLATFORM_TRIPLE.
Removes the use of the deprecated-Wl,-single_module flag when compiling macOS frameworks.
Some whitespace normalisation where there was a mix of spaces and tabs in a single block.

Issue:Add support for iOS as a target platform #114099

freakboy3742 requested review fromerlend-aasland andcorona10 ascode owners

February 7, 2024 01:56

bedevere-appbot added the awaiting review label

Feb 7, 2024

bedevere-appbot mentioned this pull request

Feb 7, 2024

Add support for iOS as a target platform#114099

Closed

freakboy3742 requested a review fromned-deily

February 7, 2024 01:57

erlend-aasland approved these changes

Feb 7, 2024

View reviewed changes

Copy link

Contributor

erlend-aasland left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

Thanks!

bedevere-appbot added awaiting merge and removed awaiting review labels

Feb 7, 2024

Copy link

Contributor

erlend-aasland commentedFeb 7, 2024

I'll wait for Ned's thumbs up before merging, since you requested his review :)

erlend-aasland self-assigned this

Feb 7, 2024

mhsmith reviewed

Feb 7, 2024

View reviewed changes

Copy link

Member

mhsmith left a comment•
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

Differentiates betweenSOABI_PLATFORM andPLATFORM_TRIPLET.SOABI_PLATFORM is used in binary module names, and includes the ABI, but not the OS or CPU architecture (e.g.,math.cpython-313-iphonesimulator.dylib).PLATFORM_TRIPLET is used as thesys._multiarch value, and on iOS will contains the ABI and architecture (e.g.,iphoneos-arm64). This differentiation hasn't historically been needed because while macOS is a multiarch platform, it uses a baredarwin asPLATFORM_TRIPLE.

Why is this differentiation needed now? Using the triplet directly would avoid introducing yet another platform identification variable on top of the dozen or more that already exist in this script.

Also, looking ahead to the next PR in this series, I'm not sure about the PLATFORM_TRIPLET format. The existing platforms all use an actual autoconf triplet with the architecture first, except for macOS which just returns "darwin" on all architectures.iphoneos-arm64 doesn't follow either of those conventions.

Using an actual autoconf triplet would cause "ios" to be duplicated in the sysconfigdata name, but that's already the case on Linux and macOS, and in such a complex area it's helpful to be consistent as much as possible.

configure.ac OutdatedShow resolvedHide resolved

Copy link

ContributorAuthor

freakboy3742 commentedFeb 8, 2024

Differentiates betweenSOABI_PLATFORM andPLATFORM_TRIPLET.SOABI_PLATFORM is used in binary module names, and includes the ABI, but not the OS or CPU architecture (e.g.,math.cpython-313-iphonesimulator.dylib).PLATFORM_TRIPLET is used as thesys._multiarch value, and on iOS will contains the ABI and architecture (e.g.,iphoneos-arm64). This differentiation hasn't historically been needed because while macOS is a multiarch platform, it uses a baredarwin asPLATFORM_TRIPLE.
Why is this differentiation needed now? Using the triplet directly would avoid introducing yet another platform identification variable on top of the dozen or more that already exist in this script.

The tl;dr is "because this is what the PEP proposed", with a side order of "that's what I've got working"; but I'll grant that dogmatic adherence to a spec or implementation isn't a very satisfying answer :-)

The longer answer is that iOS has some peculiarities that mean it doesn't fit neatly into the "GNU triple" nomenclature, most of which have existing analogs in the CPython codebase. Also, while the iOS changes aroundPLATFORM_TRIPLET et al do introduce some additional logic and temporary working variables, the set of variables used at runtime haven't changed.SOABI_PLATFORM isn't used externally - it's only used as a component of the previously existingSOABI.

iOS (and tvOS/watchOS) is a sort of hybrid between the Linux and macOS situation. Unlike Linux, it's a platform where a single binary can/will contain multiple architectures. Unlike macOS, there's multiple incompatible ABIs - the device ABI and the simulator ABI.

Using a true "triple" inPLATFORM_TRIPLET (and thusMULTIARCH andsysconfigdata) would be redundant. The triple is needed for Unix platforms because thereare different Unix kernels and C library implementations, in addition to CPU differences.x86_64-linux-gnu tells you three useful things about the platform, and as an added benefit, the rest of the Unix compiler ecosystem uses these triples as configuration values.arm64-apple-ios tells you 2 things, because there isn't a non-Apple iOS; and the only parts of the Apple compiler ecosystem that honor the "triple" form are the parts that are trying to fit into the Unix compiler ecosystem (usually badly, because of the single CPU architecture assumptions baked into the triple format). Native iOS tooling usesiphoneos oriphonesimulator descriptors.iphoneos-arm64 tells you everything you need to know about a single architecture artefact.

It's not just macOS that breaks this "triple" convention, either. WASM/Emscripten and GNU Hurd both avoid the "triple" form (usingwasm-emscripten et al, andx86_64-gnu et al respectively), for largely analogous reasons, AFAICT.

You might note that macOSdoesn't include the CPU architecture inMULTIARCH. That's because it is possible to compile a universal2 build of macOS in a single pass; and as a result, a singlesysconfigdata module can be used. This isn't true of iOS; you can't (to the best of my knowledge) do a single-pass, multi-CPU architecture compilation of an iOS target, so we need to differentiate whichsysconfigdata module is appropriate at runtime.

Including the CPU architecture in theSOABI tag is problematic because of how the binary is used at runtime. We can't distribute multiple binaries to support multiple CPU architectures on iOS - we have to use fat binaries that include multiple CPU architectures. When loading a binary module, we need to provide a single unique name to load, and that name can't include a CPU architecture.

This means we're left with needing anSOABI thatdoesn't include an architecture description. An alternate approach might be to include the architecture in the SOABI name, but havean installation process that strips the architecture off the binary name as part of compile/link process, and a runtime environment that knows that the runtime SOABI is different from the compile-time SOABI. However, that would involve a lot more moving parts and iOS eccentricities than just using an architecture neutral, ABI-specific SOABI tag in the first place. This also mirrors the macOS experience, which usesdarwin as the SOABI, regardless of whether it's a x86, arm64, or universal2 binary.

Admittedly, theSOABI issue isslightly muted at present because there's only 1 CPU architecture on physical devices at present, and simulator binaries don't have to be distributed. However, given that the platform has a history of supporting multiple CPU architectures, it seems unwise to bake the assumption that there will only ever be a single CPU architecture into the naming scheme for iOS/tvOS/watchOS binaries.

With all that said - one inconsistency in the PEP/#115063 that might be worth addressing is theordering of the iOSPLATFORM_TRIPLET value. The convention on every other platform is to put the CPU architecture before the ABI/kernel details, soarm64-iphoneos would be more consistent. That doesn't actually change that much, other than the exact name of of some binary modules.

Copy link

ContributorAuthor

freakboy3742 commentedFeb 8, 2024

I'll wait for Ned's thumbs up before merging, since you requested his review :)

FWIW - I tagged@ned-deily for review because it seemed appropriate given he's the core team sponsor for this work, and he weighed in on the#115063 version of this patch. While I'd certainly value his input, if it's not needed in terms of CPython core's process, I don't have any inherent reason to need Ned's sign-off specifically. If y'all are happy with me sticking to the default Bevedere reviewers, I'm happy to do that (or, if you'd like me to keep tagging Ned on the iOS patches, I'm happy to do that too).

Copy link

Member

ned-deily commentedFeb 8, 2024

Thanks for the opportunity to review it. I will be looking at it shortly.

ned-deily requested changes

Feb 8, 2024

View reviewed changes

Copy link

Member

ned-deily left a comment•
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

As you know, there are a lot of configure options for framework builds - on top of the enormous number of other configure options - and we currently do very little if any CI or buildbot testing of them. So my highest review priority is to try to ensure there are no regressions with existing configurations of interest (on macOS). So I think you should test a few of these on macOS manually when making updates of PRs affectingconfigure.ac andMakefile.pre.in. What would be ideal would be to add at least some of these configurations to our standard CI and/or buildbots but that is a bit out of scope here (although PRs welcome).

As the PR stands at this moment, these all fail on my current macOS (14.3, case-sensitive) systems:

# build but avoid installing into /Library and /usr/local1. ./configure --enable-framework && make -j4 && make pythoninfo# alternate install location2. ./configure --enable-framework=$PWD/root/Library/Frameworks && make -j4 && make install && $PWD/root/bin/python3 -m test.pythoninfo# alternate install location with debug build and with alternate framework name3. ./configure --enable-framework=$PWD/root/Library/Frameworks --with-framework-name=pytest --with-pydebug && make -j4 && make install && $PWD/root/bin/python3 -m test.pythoninfo

I'd suggest running on a currentHEAD of main to see the results without the PR and then with the PR. Once those tests pass, then we should run the whole test suite before and after. And then finally add in auniversal2 configuration though that is more tricky because it requires universal builds of the third-party dependencies; that is something I can test at that point.

Once we feel there are no macOS regressions, we can dig deeper into the iOS changes.

bedevere-appbot added awaiting changes and removed awaiting merge labels

Feb 8, 2024

Copy link

bedevere-appbot commentedFeb 8, 2024

A Python core developer has requested some changes be made to your pull request before we can consider merging it. If you could please address their requests along with any other requests in other reviews from core developers that would be appreciated.

Once you have made the requested changes, please leave a comment on this pull request containing the phraseI have made the requested changes; please review again. I will then notify any core developers who have left a review that you're ready for them to take another look at this pull request.

Copy link

Member

mhsmith commentedFeb 8, 2024•
edited
Loading

The longer answer is ...

Thanks, that all makes sense. Let's just make sure that in the next PR when we add the iOS-specific stuff, that this is explained clearly enough in the code comments so it doesn't get forgotten.

The convention on every other platform is to put the CPU architecture before the ABI/kernel details, soarm64-iphoneos would be more consistent. That doesn't actually change that much, other than the exact name of of some binary modules.

The only thing it would change is the sysconfigdata filename, right? In which case, it probably isn't worth changing now when it's already specified in the PEP, and the wheel tag uses the same order.

freakboy3742 added3 commits

February 9, 2024 09:22

Refactor configure and Makefile to accomodate non-macOS frameworks.

28150e7

Removed macOS from the list of recognized cross-compile targets.

caf24d7

Corrected the construction of the Framework install name.

b2bed5a

freakboy3742 force-pushed theiOS-configure-reorg branch fromb1fd709 tob2bed5aCompare

February 9, 2024 02:58

Copy link

ContributorAuthor

freakboy3742 commentedFeb 9, 2024•
edited
Loading

As the PR stands at this moment, these all fail on my current macOS (14.3, case-sensitive) systems:

I've found the source of the problem - all three seem to have the same (obvious in hindsight) cause. I've updated the PR with the fix.

Once those tests pass, then we should run the whole test suite before and after. And then finally add in auniversal2 configuration though that is more tricky because it requires universal builds of the third-party dependencies; that is something I can test at that point.

I've run an additional universal2 test building on case (3):

4. ./configure --enable-universalsdk="`xcrun --show-sdk-path`" --with-universal-archs=universal2 --enable-framework=$PWD/root/Library/Frameworks --with-framework-name=pytest --with-pydebug && make -j4 && make install && $PWD/root/bin/python3 -m test.pythoninfo

This succeeded. I then used this configuration to run$PWD/root/bin/python3.13 -m test -j 4. This wasn't completely clean - I got 3 test failures, one each in test_site, test_sys, and test_venv - but as far as I can make out, they all seemed to be due to the test case not fully accomodating the custom installation path and framework name. The same failures occur on the equivalent build & run on a checkout of main.

Lastly, I did a non-framework, installed, universal2 build and test:

5. ./configure --enable-universalsdk="`xcrun --show-sdk-path`" --with-universal-archs=universal2 --prefix=/Users/rkm/projects/python/host && make -j 4 all install && /Users/rkm/projects/python/host/bin/python3.13 -m test -j 4

This passed without error.

Copy link

ContributorAuthor

freakboy3742 commentedFeb 9, 2024•
edited
Loading

The longer answer is ...
Thanks, that all makes sense. Let's just make sure that in the next PR when we add the iOS-specific stuff, that this is explained clearly enough in the code comments so it doesn't get forgotten.

Will do.

[Edit - I've included a description in this PR]

The convention on every other platform is to put the CPU architecture before the ABI/kernel details, soarm64-iphoneos would be more consistent. That doesn't actually change that much, other than the exact name of of some binary modules.
The only thing it would change is the sysconfigdata filename, right? In which case, it probably isn't worth changing now when it's already specified in the PEP, and the wheel tag uses the same order.

Correct, those are the only two locations it would manifest (or, at least, the only places I can think that it would manifest). Ultimately, it doesn'treally matter; I just hate seeing inconsistencies in patterns.

Plus, there will need to be a final PEP update to reflect the "implemented" status - AIUI, this update usually includes any tweaks and modifications identified during the final implementation process (seePEP Maintenance in PEP1). If that process suggests a name change is appropriate, the overhead of updating the PEP to reflect that shouldn't be a major consideration, IMHO.

Document some edge cases in how PLATFORM_TRIPLET, MULTIARCH and SOABI…

d714196

… are used.

Copy link

ContributorAuthor

freakboy3742 commentedFeb 9, 2024

I have made the requested changes; please review again.

Copy link

Member

mhsmith commentedFeb 9, 2024

When a PR has already been reviewed, it's probably better to bring in changes from the main branch with a merge rather than a rebase, so the reviewer can use the "changes since last review" feature.

Copy link

Contributor

erlend-aasland commentedFeb 9, 2024

When a PR has already been reviewed, it's probably better to bring in changes from the main branch with a merge rather than a rebase, so the reviewer can use the "changes since last review" feature.

True, we try to avoid force-pushing. This is also mentioned in the devguide (Life of a Pull Request).

Copy link

ContributorAuthor

freakboy3742 commentedFeb 9, 2024

True, we try to avoid force-pushing. This is also mentioned in the devguide (Life of a Pull Request).

Apologies - I'll avoid force-pushes in future.

On the matter of process - have I misunderstood the instructions provided by Bevedere? I've left a comment as requested (originally as the last comment in my most recent response to Ned (since edited), then as a standalone comment), but Bevedere doesn't appear to have responded. Have I misunderstood what is needed here?

Copy link

Member

ned-deily commentedFeb 9, 2024

I'm reviewing the changes even without Bev's help.

Merge branch 'main' into iOS-configure-reorg

00c8965

Copy link

Member

ned-deily commentedFeb 10, 2024•
edited
Loading

My apologies, I didn't mean to push that merge to your branch but hopefully there should be no harm done: it just updates to the current HEAD. Feel free to force purge it!

ned-deily approved these changes

Feb 10, 2024

View reviewed changes

Copy link

Member

ned-deily left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

Thanks for the fixes and the additional testing. My use cases of interest now show no obvious regressions and other changes LGTM.

bedevere-appbot added awaiting merge and removed awaiting changes labels

Feb 10, 2024

erlend-aasland approved these changes

Feb 11, 2024

View reviewed changes

Merge branch 'main' into iOS-configure-reorg

17f6028

Copy link

ContributorAuthor

freakboy3742 commentedFeb 12, 2024

Just checking - Is there anything else I need to do on this PR? Is the merge an oversight, or is there something procedural that we're waiting on?

(Apologies for any impatience on my part - I definitely appreciate the volunteer efforts of the core team that are keeping the iOS work chugging along)

Copy link

Contributor

erlend-aasland commentedFeb 12, 2024

Sorry; merge oversight!

erlend-aasland merged commit2f07786 intopython:main

Feb 12, 2024

bedevere-appbot removed the awaiting merge label

Feb 12, 2024

freakboy3742 deleted the iOS-configure-reorg branch

February 12, 2024 23:11

freakboy3742 mentioned this pull request

Feb 13, 2024

gh-114099: Add configure and Makefile targets to support iOS compilation.#115390

Merged

fsc-eriker pushed a commit to fsc-eriker/cpython that referenced this pull request

Feb 14, 2024

pythongh-114099: Refactor configure and Makefile to accomodate non-ma…

571e83b

…cOS frameworks (python#115120)Part of the PEP 730 work to add iOS support.This change lays the groundwork for introducing iOS/tvOS/watchOS frameworks; it includes the structural refactoring needed so that iOSbranches can be added into in a subsequent PR.Summary of changes:* Updates config.sub to the 2024-01-01 release. This is the "as   released" version of config.sub.* Adds a RESSRCDIR variable to allow sharing of macOS and iOS Makefile   steps.* Adds an INSTALLTARGETS variable so platforms can customise which  targets are actually installed. This will be used to exclude certain  targets (e.g., binaries, manfiles) from iOS framework installs.* Adds a PYTHONFRAMEWORKINSTALLNAMEPREFIX variable; this is used as  the install name for the library. This is needed to allow for iOS  frameworks to specify an @rpath-based install name.* Evaluates MACHDEP earlier in the configure process so that  ac_sys_system is available.* Modifies _PYTHON_HOST_PLATFORM evaluation for cross-platform builds  so that the CPU architecture is differentiated from the host  identifier. This will be used to generate a _PYTHON_HOST_PLATFORM  definition that includes ABI information, not just CPU architecture.* Differentiates between SOABI_PLATFORM and PLATFORM_TRIPLET.  SOABI_PLATFORM is used in binary module names, and includes the ABI,  but not the OS or CPU architecture (e.g.,  math.cpython-313-iphonesimulator.dylib). PLATFORM_TRIPLET is used  as the sys._multiarch value, and on iOS will contains the ABI and  architecture (e.g., iphoneos-arm64). This differentiation hasn't  historically been needed because while macOS is a multiarch platform,  it uses a bare darwin as PLATFORM_TRIPLE.* Removes the use of the deprecated -Wl,-single_module flag when  compiling macOS frameworks.* Some whitespace normalisation where there was a mix of spaces and tabs   in a single block.

ned-deily mentioned this pull request

Feb 21, 2024

gh-115737: Correct the install name for non-framework macOS libPython builds.#115750

Merged