You mentioned yesterday that this logic is difficult to test completely when the stdlib doesn't have any extension modules inside packages. But couldn't it be tested with any directory containing files with the correct suffixes? They don't actually have to be dylibs, and the test doesn't actually have to load them, it just needs to verify that the correct spec is returned.

I actually found a way to test this by faking the request; seetest_file_origin in the loader tests.

That tests the loader, and I see extension/test_finder.py gives some coverage of the finder for top-level modules, but the distinction betweenfullname andname for modules inside a package should be tested somehow as well.

This code no longer exists, so I guess extra tests aren't needed either.

I've reworked this sentence, so the suggestion no longer applies.

I don't see this in the importlib docs. It would probably make sense to have it there.

Makes sense; I'll add it in.

Why is this a meta-path finder and not a path-entry finder? In other words, why don't you subclassFileFinder and then plug this in tosys.path_hooks? This is certainly a file-focused finder.

FWIW, I think I know the answer already. 😄 Correct me if I'm wrong, but that would requireFRAMEWORKS/FULLNAME.framework to be onsys.path. It would also mean that it would be checked (unnecessarily) for any othersys.path entries and that any other path-entry finders would check the framework-specificsys.path entries unnecessarily.

That said, would it be feasible to write a customFileFinder subclass that operated relative to the app's frameworks directory. Then you'd put that onsys.path and wouldn't need a custom__init__(). I'm not saying you need to do this (or even that it is doable, though my gut says it is). Mostly I'm asking if you had considered this and, if not, recommend that you take a look.

One reason I bring it up is because, any time we step outside the normal flow of things, we run the risk of disrupting users. For example, in this case a user might expect extension modules to be found via a path-entry finder rather than a meta-path finder. They might mess with things under that assumption and then get grumpy when things break (most likely mysteriously). Of course the risk here is supremely small, but users have a habit of surprising us. 😄

It would also mean that it would be checked (unnecessarily) for any othersys.path entries

Out of curiosity, what should one expectsys.path to look like on iOS?

Also, if I've understood correctly then extension modules can only be found under the app's frameworks folder and never on any othersys.path entry. If that's the case then should we remove theExtensionFileLoader entry from_get_supported_file_loaders() in Lib/importlib/_bootstrap_external.py? Also, are the restrictions on extension modules (location, suffix) technical restrictions or strictly a consequence of the fact that apps cannot ship with non-conforming binary files, as enforced by the app store police? Is there a possibility that one could find an extension file out of place somehow, like in some system directory?

IIUC, .py files may still be imported using the normal path-entry finder (FileFinder), right? I'm not familiar with iOS apps to even imagine when the finder might be looking (i.e. what's insys.path).

Relatedly, what happens if an app maintainer (or, somehow, a user) registers additional importers or messes withsys.path? Could that cause problems for this finder? Could it circumvent the app store rules?

and that any other path-entry finders would check the framework-specificsys.path entries unnecessarily.

FYI, the import machinery already makes this (almost) a non-issue with path hooks andsys.path_importer_cache.

You're correct that the reason for the metapath finder is the structure of the Frameworks folder. I hadn't considered using a custom FileFinder; I agree we should do anything we can to avoid breaking user expectations (however eccentric or unexpected those might be), so I'll take a look and see if I can make this work.

As for ExtensionFileLoader - that's an interesting case. iOS apps are entirelycapable of loading binary modules from the PYTHONPATH; it's entirely an issue of the resulting app being acceptable to the App Store. If you've got a project thatdoesn't have the "move dylibs to the Frameworks folder" build step, the app will stillrun - it will just be rejected when you try to submit it to the App store (and it will be rejected by an automated check at time of submission, not hours/days/weeks later as a result of the review process). I've also had a couple of bugs (including one I'm still chasing with cryptography) where it's useful to be able to confirm if the problem you're seeing is because the .dylib has been moved, or because the .dylib isn't working. I therefore opted to retain the ExtensionFileLoader, just in case it was useful.

The SourceFileLoader and SourcelessFileLoader both work exactly the same, however. Loading.py files and.pyc files from anywhere in the app bundle isn't an issue; it's only binary modules that the App Store restricts.sys.path for a running app will contain{dirname(sys.executable)}/python/lib/python3.X at a minimum (i.e.,prefix=={dirname(sys.executable)}/python, but the bin/include/man/share folders aren't installed). BeeWare apps usually end up with{dirname(sys.executable)}/app and{dirname(sys.executable)}/app_packages as well as the location of app code and app dependencies, but that's entirely up to the app itself.

If a user installs an additional importer... I guess that depends on what the importer is doing. The entire contents of the app bundle is fair game for reading; so a novel mechanism for finding .py files shouldn't be an issue. The only place I could force a problem is if the importer is expecting to find an executable binary in a specific location outside the Frameworks folder - but the app won't be accepted into the App Store in the first place if they try to do this. As for circumventing the App Store rules - it might be a lack of imagination on my part, but I'm having difficulty imaging what you'd be able to do here. Normal filesystem write protections should prevent even a malicious user from doing anything damaging, and you won't be able to use anything binary that isn't in the app bundle and signed on that basis.

Sounds good. Thanks for the detailed explanation!

I hadn't considered using a custom FileFinder; I agree we should do anything we can to avoid breaking user expectations (however eccentric or unexpected those might be), so I'll take a look and see if I can make this work.

Let me know if you have any questions on this.

Out of curiosity, is there really strictly no way any extension module could have a.so suffix? What about in places the app wouldn't be expected to look but might still have access to (if that's even possible)?

(To be clear, I'm not opposed to restricting the suffix here if that satisfies conforming to the iOS app rules.)

That's a good question - I don't think I've actually tried. iOS App Store policies definitely restrict thetype of binary you can use in a framework, and I assumed that would extend to using the default extension as well - but this might not be true. It's definitely worth confirming, though - I agree it would be a lot nicer if we could use the.so extension consistently.

So... I feel like a bit of an idiot. It doesn't matter what file extension is used when the module is compiled - we can rename it when it's moved to the final Frameworks location. The extension referenced in this list is only used for finder purposes... but the FileFinder won't ever find a.so or.dylib file in the normal PYTHONPATH, because the App Store prohibits that.

For bonus facepalm points: the default extension used by an Apple framework is... no extension at all - so when the binary is moved, we can strip off any extension, and rename it to the fully-qualified Python module name.

its modules

This comment no longer exists, as there isn't a need for a separate finder.

If I understand correctly, when importing a top-level module,paths is None, so the__file__ attribute will not be rewritten.

In fact, I don't think itcould be rewritten with the information available. If an app has both the standard library and some other source of binary modules onsys.path (whether it'sapp_packages,site_packages or anything else), then the current framework naming convention makes it impossible to know which path entry the "original" location was.

Maybe the framework name needs some additional dotted segments at the start, so it effectively represents the "original" path relative todirname(sys.executable)? This would also ensure there are no name clashes with any other frameworks that happen to be in the app.

The new loader implementation I've just pushed should resolve this. It leaves a.fwork breadcrumb which is used to anchor the discovery of the module, which allows us to canonically know where a binary came from, without any guesswork, even if it's inlib-dynload or somewhere else at a root of asys.path entry.

Should useunittest assertion methods.

Good note; but this test no longer exist in the new implementation.