A Forward Looking View on Free-Threaded Python

PEP 703, the accepted proposal for free threading, states that the rolloutof free-threaded Python should be gradual, which has been clarified by thePython Steering Council inthe PEP 703 acceptance post to mean a three stageprocess over multiple releases. It is therefore important to make sure that themechanisms in this PEP are useable for Python interpreters where eitherfree-threading or non-free-threading could be the default or the only option.

At the time of writing, free-threaded Python is in Phase I: experimental phase.In this phase, there is an acute need for the proposed environment markers tohelp with the transition to free-threaded Python as package authors graduallyadd support.

As the number of packages with support increases, and particularly duringPhase II: Supported-but-not-default phase, we still anticipate a strong needfor the environment markers to help with the transition.

As free-threaded Python enters into Phase III: Default phase, the need for theenvironment markers will decrease, though at this point it is not clear thatthe GIL-enabled Python will be completely phased out (it may remain availableas a non standard build option). If it persists, the inverse need for the ABIfeature detection may arise.

Indeed, in all three phases it may be necessary for package authors to choosespecific versions of their dependencies based on the ABI features, with a shiftfrom GIL-enabled as default to free-threading as default over time.

The ABI features are designed with this in mind to guarantee usefulness andsimplicity for the foreseeable future in a changing Python ecosystem.

Relation to Other PEPs

This PEP extends environment markers with set semantics for ABI features.PEP 751 includes a similar extensionfor lock file specific environment markers; although the two have beendeveloped indepedently, they are compatible where they overlap in terms of thenew set semantics.

ABI Features

ABI features are intrinsic properties of the Python interpreter, expressed assimple, understandable strings. However, not all features are equallyapplicable to all Python interpreters or Python versions. For example, thedistinction between free-threaded and GIL-enabled interpreters is only relevantfor CPython 3.13 onwards, but the bitness of the interpreter is relevant forall interpreters.

All interpreters MUST handle the following ABI features as stated. ABI featuresthat are restricted to particular interpreters MUST NOT be provided by otherinterpreters. The features are subdivided into groups and for each group thereMUST be exactly one feature present, except when the group is marked asoptional, in which case there MUST be at most one feature present.

free-threading orgil-enabled (only CPython)

If the Python interpreter is free-threaded, thefree-threading featureMUST be present and thegil-enabled feature MUST NOT be present.Otherwise, thegil-enabled feature MUST be present and thefree-threading feature MUST NOT be present.

debug (only CPython, optional)

This ABI feature is reserved for the--with-pydebug build of CPython.If the interpreter is a CPython interpreter withPy_DEBUG capabilities,thedebug feature MUST be present. On POSIX systems, this correspondsto the Python expression"d"insys.abiflags.

32-bit or64-bit (optional)

The bitness of the interpreter, that is, whether it is a 32-bit or 64-bitbuild[2]. If the bitness is unknown or neither 32-bit nor 64-bit,this feature MUST NOT be present.

Thesys_abi_features Environment Marker

To make ABI features available in dependency specifications, a new environmentmarker variable,sys_abi_features, is added to the format of dependencyspecifiers.

To do this, we need to extend the grammar laid out inPEP 508 and maintainedin theDependency specifiers and document the possible values.

The grammar is extended to include thesys_abi_features marker variable byaugmenting the definition ofenv_var as follows:

env_var=('python_version'|'python_full_version'|'os_name'|'sys_platform'|'platform_release'|'platform_system'|'platform_version'|'platform_machine'|'platform_python_implementation'|'implementation_name'|'implementation_version'|'sys_abi_features'|'extra'# ONLY when defined by a containing layer)

Like the grammar, also the overview table of environment markers inDependency specifiers is augmented to add the following row:

With these additions, ABI features can be used in dependency specifications viathein operator to test for the presence of a feature, or thenotinoperator to test for the absence of a feature.

Require Cython Pre-release for Free-Threaded Python

To require a pre-release of Cython only for a free-threaded Python interpreter,the following dependency specification can be used:

cython>3.1.0a1;"free-threading"insys_abi_featurescython==3.0.*;"free-threading"notinsys_abi_features

Require SciPy Unless on 32-bitwin32

To require SciPy unless on a 32-bit interpreter on Windows, the followingdependency specification can be used:

scipy;platform_system!="Windows"or"32-bit"notinsys_abi_features

Require NumPy for a Free-Threaded Interpreter With Debugging Capabilities

To require NumPy only for a free-threaded interpreter with debuggingcapabilities, the following dependency can be used:

numpy;"free-threading"insys_abi_featuresand"debug"insys_abi_features

Audit and Update Tools

A wide range of tools understand Python dependency data as expressed inrequirements.txt files. (e.g., Dependabot, Tidelift, etc)

Such tools inspect dependency data and, in some cases, offer tool-assisted orfully automated updates. It is our expectation that no such tools would supportthe new environment markers at first, and broad ecosystem support could takemany months or even some number of years to arrive.

As a result, users of the new environment markers would experience adegradation in their workflows and tool support at the time that they startusing them. This is true of any new standard for where and how dependency dataare encoded.

Extension Mechanism

In an early discussion of the topic (Environment marker for free-threading),the idea of a general extension mechanism for environment markers was broughtup. While it is appealing to forego a whole PEP process should the need fornew environment markers arise in the future, there are two main challenges.

First, a completely dynamic mechanism would present difficulties for tools thatrely on static analysis of dependency specifications.

This means that even if a dynamic mechanism were to be adopted, new environmentmarkers would likely still need to be spelled out in a PEP.

Second, the introduction of a dynamic mechanism would require a more compleximplementation in the packaging library, which would be a significant departurefrom the current approach.

Other Environment Markers

If other environment markers are needed right now, this PEP could be extendedto include them.

Other Tooling

The reference implementation is based on thepackaging library andpip.We have confirmed that this allows for building and installing packages withseveral build backends. It is possible that other tools should be added to thereference implementation.