Movatterモバイル変換

[0]ホーム
URL:

Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

 Sign up
Appearance settings

bpo-44689: ctypes.util.find_library() now finds macOS 11+ system libraries when built on older macOS systems#27251

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to ourterms of service andprivacy statement. We’ll occasionally send you account related emails.

Already on GitHub?Sign in to your account

Merged
ned-deily merged 7 commits intopython:mainfrombergkvist:macos-ctypes-find-library
Aug 30, 2021

Conversation

@bergkvist
Copy link
Contributor

@bergkvistbergkvist commentedJul 20, 2021•
edited
Loading

Improve binary portability between MacOS versions

https://bugs.python.org/issue44689

When compiling CPython using a MacOS Catalina build server, it won't work as expected when trying to run it on MacOS Big Sur. In particular,ctypes.util.find_library(name) will always return None. This PR adds support for compiling on an older MacOS version.

Background

Checking if a shared library exists on MacOS Big Sur is no longer possible by looking at the file system. Instead, Apple recommends the use of dlopen to check if a shared library exists (MacOS Big Sur 11.0.1 changenotes). Note that using dlopen to check for library existence is not ideal, since this might cause arbitrary side effects.

_dyld_shared_cache_contains_path and build time requirements

The current solution (introduced in#22855) to make find_library work on MacOS Big Sur does not use dlopen, but rather _dyld_shared_cache_contains_path from#include <mach-io/dyld>. This function/symbol is only available on MacOS Big Sur, meaning it will only be included if you compile CPython on MacOS Big Sur.

In other words, if we compile CPython on MacOS Catalina, and move the binary to MacOS Big Sur,from _ctypes import _dyld_shared_cache_contains_path will raise an ImportError - and find_library will not work, always returning None.

# ctypes/macholib/dyld.py# ...+try:+    from _ctypes import _dyld_shared_cache_contains_path+except ImportError:+    def _dyld_shared_cache_contains_path(*args):+        raise NotImplementedError# ...def dyld_find(name, executable_path=None, env=None):    """    Find a library or framework using dyld semantics    """    for path in dyld_image_suffix_search(chain(                dyld_override_search(name, env),                dyld_executable_path_search(name, executable_path),                dyld_default_search(name, env),            ), env):        if os.path.isfile(path):            return path+        try:+            if _dyld_shared_cache_contains_path(path):+                return path+        except NotImplementedError:+            pass# ...

Method/Verification

In order to support building for MacOS >= 11.0 (Big Sur or later) with a build server running MacOS < 11.0 (Catalina or earlier), we usedynamic loading to avoid errors when compiling. _dyld_shared_cache_contains_path will be resolved at runtime instead of compile-time, since it is not available at compile-time in this case.

The downside to doing this at runtime instead of compile-time is that the compiler won't give us useful error messages. This is why we want to keep using theweak linking approach when compiling on MacOS >= 11.0 (Big Sur or later) - and only use dynamic loading to support thedeprecated use case of compiling on an older MacOS version.

To test that this works, I've set up 2 virtual machines usingOSX-KVM on a Manjaro Linux host:

  • Catalina (OS: macOS Catalina 10.15.7 19H15 x86_64, Kernel 19.6.0)
  • Big Sur (OS: macOS 11.4 20F71 x86_64, Kernel: 20.5.0)

Files are copied between the 2 virtual machines using rsync:

#!/usr/bin/env bashrm -rf"./local"echo"Pulling from Catalina... (/usr/local/ -> ./local)"rsync -a -e"ssh -p 2222" tobias@localhost:/usr/local.echo"Pushing to BigSur... (./local/ -> /usr/local)"rsync -a -e"ssh -p 3222" ./local/* tobias@localhost:/usr/local/echo"Done!"

Checks

  • Compile onCatalina and install, copy/usr/local/* over toBigSur, and callfind_library('c')

Results

Before this change:

❌ Compile onCatalina and install, copy/usr/local/* over toBigSur, and callfind_library('c')

Python 3.11.0a0 (heads/main:635bfe8162, Jul 19 2021, 08:09:05) [Clang 12.0.0 (clang-1200.0.32.29)] on darwinType "help", "copyright", "credits" or "license" for more information.>>> from ctypes.util import find_library; print(find_library('c'))None

After this change:

✔️ Compile onCatalina and install, copy/usr/local/* over toBigSur, and callfind_library('c')

Python 3.11.0a0 (heads/macos-ctypes-find-library:0858c0c9ff, Jul 19 2021, 17:52:24) [Clang 12.0.0 (clang-1200.0.32.29)] on darwinType "help", "copyright", "credits" or "license" for more information.>>> from ctypes.util import find_library; print(find_library('c'))/usr/lib/libc.dylib

Related issues/Symptoms of the issue

Note that some of these are closed due to the symptom having been treated - rather than the underlying cause.

https://bugs.python.org/issue44689

@the-knights-who-say-ni

Hello, and thanks for your contribution!

I'm a bot set up to make sure that the project can legally accept this contribution by verifying everyone involved has signed thePSF contributor agreement (CLA).

Recognized GitHub username

We couldn't find abugs.python.org (b.p.o) account corresponding to the following GitHub usernames:

@bergkvist

This might be simply due to a missing "GitHub Name" entry in one's b.p.o account settings. This is necessary for legal reasons before we can look at this contribution. Please followthe steps outlined in the CPython devguide to rectify this issue.

You cancheck yourself to see if the CLA has been received.

Thanks again for the contribution, we look forward to reviewing it!

@bergkvistbergkvist changed the titleImprove binary portability between MacOS versionsbpo-44689: Improve binary portability between MacOS versionsJul 20, 2021
@bergkvist
Copy link
ContributorAuthor

Since this PR suggests changes to code that was merged in#22855, I'd love a review from@ronaldoussoren
This might also be relevant to@ned-deily,@serhiy-storchaka,@vstinner - since you guys were involved with that PR.

Copy link
Contributor

@erlend-aaslanderlend-aasland left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

I'll leave the technical review to Ronald, but I've left some remarks regarding code style (PEP 7).

BTW, did you read the discussion inGH-21241 regarding_dyld_shared_cache_contains_path?

@bergkvist
Copy link
ContributorAuthor

Thanks a lot for the feedback@erlend-aasland - and for linking the discussion about _dyld_shared_cache_contains_path. Do you know if there a VSCode plugin for checking or auto-formatting C code according to PEP 7?

I discussed a bit with Ronald onthe issue linked to this PR, and he made me realize that using dlopen to check for library existence is not a good idea because it can have side effects. A shared library can execute arbitrary code when opened with dlopen. This is why _dyld_shared_cache_contains_path is currently being used instead.

Instead of checking for _dyld_shared_cache_contains_path at compile time - it is possible to resolve it at run-time (dynamic loading) - and provide a fallback function which always returns false if the symbol does not exist. Then it won't matter whether the compilation happened on Catalina or Big Sur. I'll make an attempt at implementing this over the next few days.

erlend-aasland reacted with thumbs up emoji

bergkvistand others added2 commitsJuly 28, 2021 01:12
Co-authored-by: Erlend Egeberg Aasland <erlend.aasland@innova.no>
@erlend-aasland
Copy link
Contributor

Do you know if there a VSCode plugin for checking or auto-formatting C code according to PEP 7?

Sorry, I do not; I don't use VSCode, I use Vim.

bergkvist reacted with thumbs up emoji

@ned-deilyned-deily requested a review froma teamJuly 29, 2021 17:19
@yaitskov
Copy link

I would like to see this merged too, because having custom build of cpython in nix is a bit pain - some functions are private and has to be copy-pasted.

Copy link
Member

@ned-deilyned-deily left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

Thanks for the updated PR. Please see my recent detailed response on the bug tracker issue for comments that apply to this version.

@bedevere-bot
Copy link

A Python core developer has requested some changes be made to your pull request before we can consider merging it. If you could please address their requests along with any other requests in other reviews from core developers that would be appreciated.

Once you have made the requested changes, please leave a comment on this pull request containing the phraseI have made the requested changes; please review again. I will then notify any core developers who have left a review that you're ready for them to take another look at this pull request.

@bergkvist
Copy link
ContributorAuthor

I have made the requested changes; please review again

@bedevere-bot
Copy link

Thanks for making the requested changes!

@ned-deily: please review the changes made to this pull request.

Copy link
Contributor

@ronaldoussorenronaldoussoren left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

The PR looks good to me, from staring at the code. I have not yet tried building on an older macOS version and running tests on Big Sur.

@ned-deily
Copy link
Member

I will review and test it next week.

@ned-deilyned-deily changed the titlebpo-44689: Improve binary portability between MacOS versionsbpo-44689: ctypes.util.find_library() now finds macOS 11+ system libraries when built on older macOS systemsAug 30, 2021
@miss-islington
Copy link
Contributor

Thanks@bergkvist for the PR, and@ned-deily for merging it 🌮🎉.. I'm working now to backport this PR to: 3.8, 3.9, 3.10.
🐍🍒⛏🤖

@bedevere-bot
Copy link

@ned-deily: Please replace# withGH- in the commit message next time. Thanks!

miss-islington pushed a commit to miss-islington/cpython that referenced this pull requestAug 30, 2021
…aries when built on older macOS systems (pythonGH-27251)Previously, when built on older macOS systems, `find_library` was not able to find macOS system libraries when running on Big Sur due to changes in how system libraries are stored.(cherry picked from commit71853a7)Co-authored-by: Tobias Bergkvist <tobias@bergkv.ist>
@bedevere-bot
Copy link

GH-28052 is a backport of this pull request to the3.10 branch.

@bedevere-botbedevere-bot removed the needs backport to 3.10only security fixes labelAug 30, 2021
@bedevere-bot
Copy link

GH-28053 is a backport of this pull request to the3.9 branch.

@bedevere-bot
Copy link

GH-28054 is a backport of this pull request to the3.8 branch.

miss-islington pushed a commit to miss-islington/cpython that referenced this pull requestAug 30, 2021
…aries when built on older macOS systems (pythonGH-27251)Previously, when built on older macOS systems, `find_library` was not able to find macOS system libraries when running on Big Sur due to changes in how system libraries are stored.(cherry picked from commit71853a7)Co-authored-by: Tobias Bergkvist <tobias@bergkv.ist>
ambv pushed a commit that referenced this pull requestAug 30, 2021
…aries when built on older macOS systems (GH-27251) (GH-28054)Previously, when built on older macOS systems, `find_library` was not able to find macOS system libraries when running on Big Sur due to changes in how system libraries are stored.(cherry picked from commit71853a7)Co-authored-by: Tobias Bergkvist <tobias@bergkv.ist>
ambv pushed a commit that referenced this pull requestAug 30, 2021
…aries when built on older macOS systems (GH-27251) (GH-28053)Previously, when built on older macOS systems, `find_library` was not able to find macOS system libraries when running on Big Sur due to changes in how system libraries are stored.(cherry picked from commit71853a7)Co-authored-by: Tobias Bergkvist <tobias@bergkv.ist>
miss-islington added a commit that referenced this pull requestAug 30, 2021
…aries when built on older macOS systems (GH-27251)Previously, when built on older macOS systems, `find_library` was not able to find macOS system libraries when running on Big Sur due to changes in how system libraries are stored.(cherry picked from commit71853a7)Co-authored-by: Tobias Bergkvist <tobias@bergkv.ist>
@teaketeake mentioned this pull requestDec 2, 2021
4 tasks
Sign up for freeto join this conversation on GitHub. Already have an account?Sign in to comment

Reviewers

@erlend-aaslanderlend-aaslanderlend-aasland left review comments

@ronaldoussorenronaldoussorenronaldoussoren approved these changes

@ned-deilyned-deilyned-deily approved these changes

Assignees

No one assigned

Labels

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

8 participants

@bergkvist@the-knights-who-say-ni@erlend-aasland@yaitskov@bedevere-bot@ned-deily@miss-islington@ronaldoussoren
[8]ページ先頭
©2009-2025 Movatter.jp