This PEP attempts to document the status of cross-compilation of downstreamprojects.
It should give an overview of the approaches currently used by distributors(Linux distros, WASM environment providers, etc.) to cross-compile downstreamprojects (3rd party extensions, etc.).
We write this PEP to express the challenges in cross-compilation and act as asupporting document in future improvement proposals.
There are a couple different approaches being used to tackle this, withdifferent levels of interaction required from the user, but they all require asignificant amount of effort. This is due to the lack of standardizedcross-compilation infrastructure on the Python packaging ecosystem, which itselfstems from the complexity of cross-builds, making it a huge undertaking.
Some major projects like CPython, setuptools, etc. provide some support to helpwith cross-compilation, but it’s unofficial and at a best-effort basis. Forexample, thesysconfig module allows overwriting the data module name viathe_PYTHON_SYSCONFIGDATA_NAME environment variable, something that isrequired for cross-builds, and setuptoolsaccepts patches[1] to tweak/fixits logic to be compatible with popular“environment faking” workflows[2].
The lack of first-party support in upstream projects leads to cross-compilationbeing fragile and requiring a significant effort from users, but at the sametime, the lack of standardization makes it harder for upstreams to improvesupport as there’s no clarity on how this feature should be provided.
It seems relevant to point out that there are a few modern Python packagebuild-backends with, at least, decent cross-compilation support, those beingscikit-build andmeson-python. Both these projects integrate external maturebuild-systems into Python packaging —CMake andMeson, respectively — socross-build support is inherited from them.
Cross-compilation approaches fall in a spectrum that goes from, by design,requiring extensive user interaction to (ideally) almost none. Usually, they’llbe based on one of two main strategies, using across-build environment,orfaking the target environment.
This consists of running the Python interpreter normally and utilizing thecross-build provided by the projects’ build-system. However, as we saw above,upstream support is lacking, so this approach only works for a small-ish set ofprojects. When this fails, the usual strategy is to patch the build-system codeto build use the correct toolchain, system details, etc.[3].
Since this approach often requires package-specific patching, it requires a lotof user interaction.
Examples
python-for-android,kivy-ios, etc.
Aiming to drop the requirement for user input, a popular approach is trying tofake the target environment. It generally consists of monkeypatching the Pythoninterpreter to get it to mimic the interpreter on the target system, whichconstitutes of changing many of thesys module attributes, thesysconfigdata, etc. Using this strategy, build-backends do not need to have anycross-build support, and should just work without any code changes.
Unfortunately, though, it isn’t possible to truly fake the target environment.There are many reasons for this, one of the main ones being that it breaks codethat actually needs to introspect the running interpreter. As a result,monkeypatching Python to look like target is very tricky — to achieve the lessamount of breakage, we can only patch certain aspects of the interpreter.Consequently, build-backends may need some code changes, but these are generallymuch smaller than the previous approach. This is an inherent limitation of thetechnique, meaning this strategy still requires some user interaction.
Nonetheless, this strategy still works out-of-the-box with significantly moreprojects than the approach above, and requires much less effort in these cases.It is successful in decreasing the amount of user interaction needed, eventhough it doesn’t succeed in being generic.
Examples
crossenv,conda-forge, etc.
As explained above, most build system code is written with the assumption thatthe target system is the same as where the build is occurring, so introspectionis usually used to guide the build.
In this section, we try to document most of the ways this is accomplished. Itshould give a decent overview of of environment details that are required bybuild systems.
| Snippet | Description | Variance |
|---|---|---|
>>>importlib.machinery.EXTENSION_SUFFIXES[ '.cpython-311-x86_64-linux-gnu.so', '.abi3.so', '.so',] | Extension (native module) suffixes supported by this interpreter. | This is implementation-defined, but itusually differs based on theimplementation, system architecture, build configuration, Pythonlanguage version, and implementation version — if one exists. |
>>>importlib.machinery.SOURCE_SUFFIXES['.py'] | Source (pure-Python) suffixes supported by this interpreter. | This is implementation-defined, but itusually doesn’t differ(outside exotic implementations or systems). |
>>>importlib.machinery.all_suffixes()[ '.py', '.pyc', '.cpython-311-x86_64-linux-gnu.so', '.abi3.so', '.so',] | All module file suffixes supported by this interpreter. Itshould be theunion of allimportlib.machinery.*_SUFFIXES attributes. | This is implementation-defined, but itusually differs based on theimplementation, system architecture, build configuration, Pythonlanguage version, and implementation version — if one exists. See theentries above for more information. |
>>>sys.abiflags'' | ABI flags, as specified inPEP 3149. | Differs based on the build configuration. |
>>>sys.api_version1013 | C API version. | Differs based on the Python installation. |
>>>sys.base_prefix/usr | Prefix of the installation-wide directories where platform independentfiles are installed. | Differs based on the platform, and installation. |
>>>sys.base_exec_prefix/usr | Prefix of the installation-wide directories where platform dependentfiles are installed. | Differs based on the platform, and installation. |
>>>sys.byteorder'little' | Native byte order. | Differs based on the platform. |
>>>sys.builtin_module_names('_abc', '_ast', '_codecs', ...) | Names of all modules that are compiled into the Python interpreter. | Differs based on the platform, system architecture, and buildconfiguration. |
>>>sys.exec_prefix/usr | Prefix of the site-specific directories where platform independent filesare installed. Because it concerns the site-specific directories, instandard virtual environment implementation, it will be avirtual-environment-specific path. | Differs based on the platform, installation, and environment. |
>>>sys.executable'/usr/bin/python' | Path of the Python interpreter being used. | Differs based on the installation. |
>>>withopen(sys.executable,'rb')asf:...header=f.read(4)...ifis_elf:=(header==b'\x7fELF'):...elf_class=int(f.read(1))...size={1:52,2:64}.get(elf_class)...elf_header=f.read(size-5) | Whether the Python interpreter is an ELF file, and the ELF header. Thisapproach is something used to identify the target architecture of theinstallation (example). | Differs based on the installation. |
>>>sys.float_infosys.float_info( max=1.7976931348623157e+308, max_exp=1024, max_10_exp=308, min=2.2250738585072014e-308, min_exp=-1021, min_10_exp=-307, dig=15, mant_dig=53, epsilon=2.220446049250313e-16, radix=2, rounds=1,) | Low level information about the float type, as defined byfloat.h. | Differs based on the architecture, and platform. |
>>>sys.getandroidapilevel()21 | Integer representing the Android API level. | Differs based on the platform. |
>>>sys.getwindowsversion()sys.getwindowsversion( major=10, minor=0, build=19045, platform=2, service_pack='',) | Windows version of the system. | Differs based on the platform. |
>>>sys.hexversion0x30b03f0 | Python version encoded as an integer. | Differs based on the Python language version. |
>>>sys.implementationnamespace( name='cpython', cache_tag='cpython-311', version=sys.version_info( major=3, minor=11, micro=3, releaselevel='final', serial=0, ), hexversion=0x30b03f0, _multiarch='x86_64-linux-gnu',) | Interpreter implementation details. | Differs based on the interpreter implementation, Python languageversion, and implementation version — if one exists. It may also includearchitecture-dependent information, so it may also differ based on thesystem architecture. |
>>>sys.int_infosys.int_info( bits_per_digit=30, sizeof_digit=4, default_max_str_digits=4300, str_digits_check_threshold=640,) | Low level information about Python’s internal integer representation. | Differs based on the architecture, platform, implementation, build, andruntime flags. |
>>>sys.maxsize0x7fffffffffffffff | Maximum value a variable of typePy_ssize_t can take. | Differs based on the architecture, platform, and implementation. |
>>>sys.maxunicode0x10ffff | Value of the largest Unicode code point. | Differs based on the implementation, and on Python versions older than3.3, the build. |
>>>sys.platformlinux | Platform identifier. | Differs based on the platform. |
>>>sys.prefix/usr | Prefix of the site-specific directories where platform dependent filesare installed. Because it concerns the site-specific directories, instandard virtual environment implementation, it will be avirtual-environment-specific path. | Differs based on the platform, installation, and environment. |
>>>sys.platlibdirlib | Platform-specific library directory. | Differs based on the platform, and vendor. |
>>>sys.version_infosys.version_info( major=3, minor=11, micro=3, releaselevel='final', serial=0,) | Python language version implemented by the interpreter. | Differs if the target Python version is not the same[4]. |
>>>sys.thread_infosys.thread_info( name='pthread', lock='semaphore', version='NPTL 2.37',) | Information about the thread implementation. | Differs based on the platform, and implementation. |
>>>sys.winver3.8-32 | Version number used to form Windows registry keys. | Differs based on the platform, and implementation. |
>>>sysconfig.get_config_vars(){ ... }>>>sysconfig.get_config_var(...)... | Python distribution configuration variables. It includes a set ofvariables[5] — likeprefix,exec_prefix, etc. — based on therunning context[6], and may include some extra variables based on thePython implementation and system.In CPython and most other implementations that use the samebuild-system, the “extra” variables mention above are: on POSIX, allvariables from the | This is implementation-defined, but itusually differs betweennon-identical builds. Please refer to thesysconfig configuration variables table for a overview of the differentconfiguration variable that are usually present. |
sys.prefix and otherattributes accordingly.Makefile, and insteaduse the Visual Studio build system. A subset of the most relevantMakefile variables is provided to make user code that uses themsimpler.| Name | Example Value | Description | Variance |
|---|---|---|---|
SOABI | cpython-311-x86_64-linux-gnu | ABI string — defined byPEP 3149. | Differs based on the implementation, system architecture, Pythonlanguage version, and implementation version — if one exists. |
SHLIB_SUFFIX | .so | Shared library suffix. | Differs based on the platform. |
EXT_SUFFIX | .cpython-311-x86_64-linux-gnu.so | Interpreter-specific Python extension (native module) suffix — generallydefined as.{SOABI}.{SHLIB_SUFFIX}. | Differs based on the implementation, system architecture, Pythonlanguage version, and implementation version — if one exists. |
LDLIBRARY | libpython3.11.so | Sharedlibpython library name — if available. If unavailable[8],the variable will be empty, if available, the library should be locatedinLIBDIR. | Differs based on the implementation, system architecture, buildconfiguration, Python language version, and implementation version — ifone exists. |
PY3LIBRARY | libpython3.so | Shared Python 3 only (major version bound only)[9]libpythonlibrary name — if available. If unavailable[8], the variable will beempty, if available, the library should be located inLIBDIR. | Differs based on the implementation, system architecture, buildconfiguration, Python language version, and implementation version — ifone exists. |
LIBRARY | libpython3.11.a | Staticlibpython library name — if available. If unavailable[8],the variable will be empty, if available, the library should be locatedinLIBDIR. | Differs based on the implementation, system architecture, buildconfiguration, Python language version, and implementation version — ifone exists. |
Py_DEBUG | 0 | Whether this is adebug build. | Differs based on the build configuration. |
WITH_PYMALLOC | 1 | Whether this build haspymalloc support. | Differs based on the build configuration. |
Py_TRACE_REFS | 0 | Whether reference tracing (debug build only) is enabled. | Differs based on the build configuration. |
Py_UNICODE_SIZE | Size of thePy_UNICODE object, in bytes. This variable is onlypresent in CPython versions older than 3.3, and was commonly used todetect if the build uses UCS2 or UCS4 for unicode objects — beforePEP 393. | Differs based on the build configuration. | |
Py_ENABLE_SHARED | 1 | Whether a sharedlibpython is available. | Differs based on the build configuration. |
PY_ENABLE_SHARED | 1 | Whether a sharedlibpython is available. | Differs based on the build configuration. |
CC | gcc | The C compiler used to build the Python distribution. | Differs based on the build configuration. |
CXX | g++ | The C compiler used to build the Python distribution. | Differs based on the build configuration. |
CFLAGS | -DNDEBUG-g-fwrapv... | The C compiler flags used to build the Python distribution. | Differs based on the build configuration. |
py_version | 3.11.3 | Full form of the Python version. | Differs based on the Python language version. |
py_version_short | 3.11 | Custom form of the Python version, containing only the major and minornumbers. | Differs based on the Python language version. |
py_version_nodot | 311 | Custom form of the Python version, containing only the major and minornumbers, and no dots. | Differs based on the Python language version. |
prefix | /usr | Same assys.prefix, please refer to the entry in table above. | Differs based on the platform, installation, and environment. |
base | /usr | Same assys.prefix, please refer to the entry in table above. | Differs based on the platform, installation, and environment. |
exec_prefix | /usr | Same assys.exec_prefix, please refer to the entry in table above. | Differs based on the platform, installation, and environment. |
platbase | /usr | Same assys.exec_prefix, please refer to the entry in table above. | Differs based on the platform, installation, and environment. |
installed_base | /usr | Same assys.base_prefix, please refer to the entry in table above. | Differs based on the platform, and installation. |
installed_platbase | /usr | Same assys.base_exec_prefix, please refer to the entry in tableabove. | Differs based on the platform, and installation. |
platlibdir | lib | Same assys.platlibdir, please refer to the entry in table above. | Differs based on the platform, and vendor. |
SIZEOF_* | 4 | Size of a certain C type (double,float, etc.). | Differs based on the system architecture, and build details. |
libpythonsupport, respectively.libpython library that users of thestable ABI shouldlink against, if they need to link againstlibpython.There are some bits of information required by build systems — eg. platformparticularities — scattered across many places, and it often is difficult toidentify code with assumptions based on them. In this section, we try todocument the most relevant cases.
libpython?When building extensions for dynamic loading, depending on the target platform,they may need to be linked againstlibpython.
On Windows, extensions need to link againstlibpython, because all symbolsmust be resolvable at link time. POSIX-like platforms based on Windows — likeCygwin, MinGW, or MSYS — will also require linking againstlibpython.
On most POSIX platforms, it is not necessary to link againstlibpython, asthe symbols will already be available due to the interpreter — or, whenembedding, the executable/library in question — already linking tolibpython. Not linking an extension module againstlibpython will allowit to be loaded by static Python builds, so when possible, it is desirable to doso (seeGH-65735).
This might not be the case on all POSIX platforms, so make sure you check. Oneexample is Android, where only the main executable andLD_PRELOAD entriesare considered to beRTLD_GLOBAL (meaning dependencies areRTLD_LOCAL)[10], which causes thelibpython symbols be unavailable when loading theextension.
prefix,exec_prefix,base_prefix, andbase_exec_prefix?These aresys attributesset in the Python initialization that describethe running environment. They refer to the prefix of directories whereinstallation/environment files are installed, according to the table below.
| Name | Target files | Environment Scope |
|---|---|---|
prefix | platform independent (eg. pure Python) | site-specific |
exec_prefix | platform dependent (eg. native code) | site-specific |
base_prefix | platform independent (eg. pure Python) | installation-wide |
base_exec_prefix | platform dependent (eg. native code) | installation-wide |
Because the site-specific prefixes will be different inside virtualenvironments, checkingsys.prexix!=sys.base_prefix is commonly used tocheck if we are in a virtual environment.
crossenv is a tool to create a virtual environment with a monkeypatchedPython installation that tries to emulate the target machine in certainscenarios. More about this approach can be found in theFaking the target environment section.
XXX: Jaime will write a quick summary once the PEP draft is public.
XXXUses a modified crossenv.
XXX: Sent email to the mailing list.
TODO
TODO
XXX: Hood should review/expand this section.
Pyodide is a provides a Python distribution compiled toWebAssemblyusing theEmscripten toolchain.
It patches several aspects of the CPython installation and some externalcomponents. A custom package manager —micropip — supporting both Pure andwasm32/Emscripten wheels, is also provided as a part of the distribution. On topof this, a repo with aselected set of 3rd party packages is also providedand enabled by default.
TODO
resourcehttps://github.com/Android-for-Python/Android-for-Python-Users
python-for-android is a tool to package Python apps on Android. It creates aPython distribution with your app and its dependencies.
Pure-Python dependencies are handled automatically and in a generic way, butnative dependencies needrecipes. A set of recipes forpopular dependencies is provided, but users need to provide their ownrecipes for any other native dependencies.
kivy-ios is a tool to package Python apps on iOS. It provides a toolchain tobuild a Python distribution with your app and its dependencies, as well as a CLIto create and manage Xcode projects that integrate with the toolchain.
It uses the same approach aspython-for-android (also maintained by theKivy project) for app dependencies — pure-Python dependencies are handledautomatically, but native dependencies needrecipes, and the project providesrecipes forpopular dependencies.
TODO
TODO
contacthttps://www.riverbankcomputing.com/pipermail/pyqt/2023-May/thread.htmlcontacted Phil, the maintainer
TODO
TODO
TODO
TODO
TODO
Source:https://github.com/python/peps/blob/main/peps/pep-0720.rst
Last modified:2025-05-06 20:54:51 GMT