PEP 602 proposes reducing the feature delivery latency for the Pythonstandard library and CPython reference interpreter by increasing the frequencyof CPython feature releases from every 18-24 months to instead occur every 9-12months.
This PEP proposes to insteadreduce the frequency of new baseline featurereleases (with the associated filesystem layout changes, bytecode formatchanges, and C ABI compatibility breaks) to occur only every other year (2020,2022, 2024, etc), but to combine that change with a new policy and approach thatallows the introduction of backwards compatible features in the initial set ofpoint releases within a given release series.
This PEP has been withdrawn in favour of the rolling beta release streamproposal inPEP 605.
However, the concerns raised in this PEP are likely to apply to any other“Long Term Support branch” proposals that allow feature backports to improvethe developer experience of supporting such releases (such as the EL Pythondraft at[3]), so the ideas presented here may provide useful designsuggestions for such proposals.
The proposal to keep the current CPython release compatibility managementprocess, but go through it more often has significant practical downsides,as a CPython feature release carries certain expectations (most notably, a 5-yearmaintenance lifecycle, support for parallel installation with the previousfeature release, and the possibility of breaking changes to the CPython-specificABI, requiring recompilation of all extension modules) that mean faster featurereleases in their current form have the potential to significantly increase theburden of maintaining 3rd party Python libraries and applications across allactively supported CPython releases.
It’s also arguable whether such an approach would noticeably reduce the typicalfeature delivery latency in practice for most end users, as the adoption cyclefor new feature releases is typically measured in months or years, so morefrequent releases may just lead to end users updating to every 3rd or 4thfeature release, rather than every 2nd or 3rd feature release (as often happenstoday).
This PEP presents a competing proposal to insteadslow down the frequency ofparallel installable feature releases that change the filesystem layoutand CPython ABI to a consistent 24-month cycle, but to compensate for this byintroducing the notion of build compatible incremental feature releases, andthen deferring the full feature freeze of a given feature release series fromthe initial baseline X.Y.0 release to a subsequent X.Y.Z feature completerelease that occurs ~12 months after the initial baseline feature release.
A newfeature_complete attribute on thesys.version_info structure willprovide a programmatic indicator as to whether or not a release series remainsopen to further incremental feature releases. Alternate implementations ofPython would also be free to clear this flag to indicate that their support fortheir nominal Python version may still be a work in progress.
For compatibility testing purposes, and to maintain pickle compatibility inmixed version environments, a newsys.feature_limit attribute (andcorresponding CPython CLI parameter,--feature-limitX.Y.Z, and environmentvariable,PYTHONFEATURELIMIT) will provide a way to limit the runtimeavailability of features added in incremental feature releases.
The existing cycle and the new cycle would be synchronised on their featurefreeze release dates, so the full Python 3.9.x feature freeze would occur inOctober 2021, 24 months after the Python 3.8.0 feature release, with the initialPython 3.9.0 release taking place in October 2020.
Under this proposal, Python 3.9.0a1 would be released in November 2019, shortlyafter the Python 3.8.0 feature complete release in October 2019.
The 3.9.0b1 release would then follow 6 months later in May 2020, with 3.9.0itself being released in October 2020.
Assuming micro releases of 3.9.x were to occur quarterly, then the overallrelease timeline would look like:
Feature complete release numbers would typically be written without anyqualifier (as they are today), while the baseline and incremental featurereleases would be expected to have a qualifier appended indicating that theyaren’t a traditional CPython release (3.9.0(BFR),3.9.1(IFR), etc).
The Python 3.10 release series would start being published the month after thefirst Python 3.9 feature complete release, in parallel with the final 12 monthsof routine Python 3.9 maintenance releases:
In this model, there are always two or three active branches:
(Pre-alpha and pre-beta development occurs on the main git branch, all otherdevelopment occurs on a release specific branch with changes typicallybackported from the main git branch)
TODO: this really needs a diagram to help explain it, so I’ll add a pictureonce I have one to add.
This is quite similar to the status quo, but with a more consistent cadence,alternating between baseline feature release years (2020, 2022, etc) that focuson the alpha and beta cycle for a new baseline feature release (while continuingto publish maintenance releases for the previous feature release series), andfeature complete release years (2021, 2023, etc), that focus on makingsmaller improvements to the current feature release series (while making plansfor the next feature release series the following year).
Excluding alpha and beta releases, CPython currently has 3 different kindsof release increment:
Feature freeze takes place at the time of the X.Y.0b1 release.Build compatibility freeze now takes place at the time of the last beta release(providing time for projects to upload wheel archives to PyPI prior to thefirst release candidate).
This then creates the following periods in the lifecycle of a release series:
The proposal in this PEP is that the “Feature release” category be split up intothree different kinds of feature release:
X.Y.Z releases)This would then introduce a new “Feature releases” phase in the release serieslifecycle:
The pre-release beta period would be relaxed to use the incremental featurerelease policy for changes, rather than the stricter maintenance release policy.
For governance purposes, baseline feature releases are the only releases thatwould qualify as a “feature release” in thePEP 13 sense (incremental featurereleases wouldn’t count).
Baseline feature releases are essentially just the existing feature releases,given a new name to help distinguish them from the new incremental featurereleases, and also to help indicate that unlike their predecessors, they areno longer considered feature complete at release.
Baseline feature releases would continue to define a new feature release series,locking in the following language, build, and installation compatibilityconstraints for the remainder of that series:
ast module AST formatpyc file magic number and filename compatibility tagsBaseline feature releases would also continue to be the only releases where:
Key characteristics of a feature release series:
Key characteristics of a baseline feature release:
sys.version_info.feature_complete==Falsesys.version_info.micro==0sys.version_info as their sole runtime indicatorof the feature’s availabilityKey expectations around feature release series and baseline feature releases:
The key motivation of the proposal in this PEP is that the public and privateproject behaviours described above aren’tnew expectations: they’redescriptions of the way CPython release series are already handled by the widercommunity today. As such, the PEP represents an attempt to adjust our releasepolicies and processes to better match the way the wider community alreadyhandles them, rather than changing our processes in a way that then means thewider community needs to adjust to us rather than the other way around.
Incremental feature releases are the key new process addition being proposed bythis PEP. They are subject to the same strict runtime compatibility requirementsas the existing maintenance releases, but would have the following morerelaxed policies around API additions and enhancements:
The intent of this change in policy is to allow usability improvements for new(and existing!) language features to be delivered in a more timely fashion,rather than requiring users to incur the inherent delay and costs of waiting forand then upgrading to the next feature release series.
It is also designed such that the approval to add a feature to the next baselinefeature release can be considered separately from the question of whether or notto make it available in the next incremental feature release for the currentrelease series, potentially allowing the first task to be completed by volunteercontributors, while the latter activity could be handled by paid contributors(e.g. customers of commercial Python redistributors could potentially requestthat their vendor backport a feature, or core developers could offer toundertake specific backports on a contract basis). (There would be potentialethical concerns with gating bug fixes this way, but those concerns don’t applyfor backports of new features)
Key characteristics of an incremental feature release:
sys.version_info.feature_complete==Falsesys.version_info.micro!=0sys.version_info or runtime code object introspection. In most cases, asimplehasattr check on the affected module will serve this purpose, butwhen it doesn’t, an alternative approach will need to be implemented as partof the feature addition. Prior art in this area includes thepickle.HIGHEST_PROTOCOL attribute, thehashlib.algorithms_availableset, and the variousos.supports_* sets that theos module alreadyoffers for platform dependent capability detectionsys.feature_limit setting as described in the nextsectionKey expectations around incremental feature releases:
Python-Requires metadata appropriately(projects already do this when necessary - e.g.aiohttp specificallyrequires 3.5.3 or later due to an issue withasyncio.get_event_loop()in earlier versions)Some standard library modules may also impose their own restrictions onacceptable changes in incremental feature releases (for example, only abaseline feature release should ever add new hash algorithms tohashlib.algorithms_guaranteed - incremental feature releases would only bepermitted to add algorithms tohashlib.algorithms_available)
It is a common practice to use Python’spickle module to exchangeinformation between Python processes running on different versions of Python.Between release series, this compatibility is expected to only run one way(i.e. excluding deprecated APIs, Python “X.Y+1” processes should be able toread pickle archives produced by Python “X.Y” processes, but the reverse doesnot hold, as the newer archives may reference attributes and parameters thatdon’t exist in the older version).
Within a release series, however, it is expected to hold in both directions,as the “No new features” policy means that almost all pickle archives createdon Python “X.Y.Z+1” will be readable by Python “X.Y.Z” processes.
Similarly, Python libraries and applications are often only tested againstthe latest version in a release series, and this is usually sufficient to keepcode working on earlier releases in that same series.
Allowing feature additions in later “X.Y.Z” releases with no way to turn themoff would pose a problem for these common practices, as a library or applicationthat works fine when tested on CPython version “X.Y.Z” would fail on earlierversions if it used a feature newly introduced in “X.Y.Z”, and any picklearchives it creates that rely on those new interfaces may also not be readableon the older versions.
To help address these problems, a newsys.feature_limit attribute would beadded, as a structured sequence corresponding to the first 3 fields insys.version_info (major,minor,micro).
A new CLI option (--feature-limitX.Y.Z) and environment variable(PYTHONFEATURELIMIT=X.Y.Z) would be used to set this attribute. ThePyCoreConfig struct would also gain a new field:
wchar_t*feature_limit;
If the limit is not set explicitly, it would default to the first 3 fields insys.version_info. If the limit is set to a value outside the lower bound ofsys.version_info[:2] and the upper bound ofsys.version_info[:3], itwill be clamped to those bounds, padding with zeroes if necessary.
For example, given a current version of “3.9.3”, nominal limits would beconverted to runtimesys.feature_limit values as follows:
3=>(3,9,0)3.8.1=>(3,9,0)3.9=>(3,9,0)3.9.2=>(3,9,2)<unset>=>(3,9,3)3.9.3=>(3,9,3)3.9.4=>(3,9,3)4=>(3,9,3)
New APIs backported to an incremental feature release would be expected toinclude a guard that deletes the API from the module if the feature limit istoo low:
deffeature_api():..._version_feature_api_added=(3,9,1)if_version_feature_api_added>sys.feature_limit:delfeature_api
Similarly, new parameters would be expected to include a guard that adjusts thefunction signature to match the old one:
deffeature_api(old_param1,old_param2,new_param=default):"""Updated API docstring"""..._version_feature_api_changed=(3,9,1)if_version_feature_api_changed>sys.feature_limit:_new_feature_api=feature_apideffeature_api(old_param1,old_param2):"""Legacy API docstring"""return_new_feature_api(old_param1,old_param2)
Structuring the guards this way would keep the code structure as similar aspossible between the main development branch and the backport branches, sofuture bug fixes can still be backported automatically.
It is expected that convenience functions and/or additional automated testswould eventually be added to help ensure these backported APIs are guardedappropriately, but it seems reasonable to wait until specific concreteexamples are available to drive the design of those APIs and automated tests,rather than designing them solely on the basis of hypothetical examples.
The feature complete release for a given feature release series would bedeveloped under the normal policy for an incremental feature release, butwould have one distinguishing feature:
sys.version_info.feature_complete==TrueAny subsequent maintenance and security fix only releases would also have thatflag set, and may informally be referred to as “feature complete releases”.For release series definition purposes though, the feature complete releaseis the first one that sets that flag to “True”.
To help improve consistency in management of provisional APIs, this PEP proposesthat provisional APIs be subject to regular backwards compatibility requirementsfollowing the feature complete release for a given release series.
Other aspects of managing provisional APIs would remain as they are today, so aslong as an API remains in the provisional state, regular backwards compatibilityrequirements would not apply to that API in baseline and incremental featurereleases.
This policy is expected to provide increased clarity to end users (as evenprovisional APIs will become stable for that release series in the featurecomplete release), with minimal practical downsides for standard librarymaintainers, based on the following analysis of documented API additions andchanges in micro releases of CPython since 3.0.0:
When post-baseline-release changes need to be made, the majority of them occurwithin the first two maintenance releases, which have always occurred within 12months of the baseline release.
(Note: these counts are not solely for provisional APIs - they cover all APIswhere semantic changes were made after the baseline release that were considerednecessary to cover in the documentation. To avoid double counting changes, thenumbers exclude any change markers from the What’s New section)
The motivation for change in this PEP is essentially the same as the motivationfor change inPEP 596: the current 18-24 month gap between feature releases hasa lot of undesirable consequences, especially for the standard library (seePEP 596 for further articulation of the details).
This PEP’s concern with the specific proposal inPEP 596 is that it doubles thenumber of actively supported Python branches, increasing the complexity ofcompatibility testing matrices for the entire Python community, increasing thenumber of binary Python wheels to be uploaded to PyPI when not using the stableABI, and just generally having a high chance of inflicting a relatively highlevel of additional cost across the entire Python ecosystem.
The view taken in this PEP is that there’s an alternative approach that providesmost of the benefits of a faster feature release without actually incurring theassociated costs: we can split the current X.Y.0 “feature freeze” into twoparts, such that the baseline X.Y.0 release only imposes a“runtime compatibility freeze”, and the full standard library feature freezeis deferred until later in the release series lifecycle.
This proposal does NOT retroactively apply to Python 3.8 - it is being proposedfor Python 3.9 and later releases only.
Actual release dates may be adjusted up to a month earlier or later atthe discretion of the release manager, based on release team availability, andthe timing of other events (e.g. PyCon US, or the annual core developmentsprints). However, part of the goal of this proposal is to provide a consistentannual cadence for both contributors and end users, so adjustments ideally wouldbe rare.
This PEP does not dictate a specific cadence for micro releases within a releaseseries - it just specifies the rough timelines for transitions between therelease series lifecycle phases (pre-alpha, alpha, beta, feature releases,bug fixes, security fixes). The number of micro releases within each phase isdetermined by the release manager for that series based on how frequently theyand the rest of the release team for that series are prepared to undertake theassociated work.
However, for the sake of the example timelines, the PEP assumes quarterlymicro releases (the cadence used for Python 3.6 and 3.7, splitting thedifference between the twice yearly cadence used for some historical releaseseries, and the monthly cadence planned for Python 3.8 and 3.9).
The filesystem layout changes and other inherently incompatible changes involvedin a baseline feature release create additional work for large sections of thewider Python community.
Decoupling those layout changes from the Python version numbering scheme is alsosomething that would in and of itself involve making backwards incompatiblechanges, as well as adjusting community expectations around which versions willinstall over the top of each other, and which can be installed in parallel ona single system.
We also don’t have a straightforward means to communicate to the communityvariations in support periods like “Only support Python version X.Y untilX.Y+1 is out, but support X.Z until X.Z+2 is out”.
So this PEP takes as its starting assumption that the vast majority of Pythonusers simplyshouldn’t need to care that we’re changing our release policy,and the only folks that should be affected are those that are eagerly waitingfor standard library improvements (and other backwards compatible interpreterenhancements), and those that need to manage mission critical applications incomplex deployment environments.
Many Python libraries (both open source and proprietary) currently adopt thepractice of testing solely against the latest micro release within each featurerelease series that the project still supports.
The design assumption in this PEP is that this practice will continue to befollowed during the feature release phase of a release series, with theexpectation being that anyone choosing to adopt a new release series before itis feature complete will closely track the incremental feature releases.
Libraries that support a previous feature release series are unlikely to adoptfeatures added in an incremental feature release, and if they do adopt sucha feature, then any associated fallback compatibility strategies should beimplemented in such a way that they’re also effective on the earlier releasesin that release series.
Based on discussions at SciPy 2019, a NEP is currently being drafted[2] todefine a common convention across the Scientific Python ecosystem for droppingsupport for older Python versions.
While the exact formulation of that policy is still being discussed, the initialproposal was very simple: support any Python feature release published withinthe last 42 months.
For an 18-month feature release cadence, that works out to always supporting atleast the two most recent feature releases, and then dropping support for allX.Y.z releases around 6 months after X.(Y+2).0 is released. This means there isa 6-month period roughly every other year where the three most recent featurereleases are supported.
For a 12-month release cadence, it would work out to always supporting atleast the three most recent feature releases, and then dropping support for allX.Y.z releases around 6 months after X.(Y+3).0 is released. This means thatfor half of each year, the four most recent feature releases would be supported.
For a 24-month release cadence, a 42-month support cycle works out to alwayssupporting at least the most recent feature release, and then dropping supportfor all X.Y.z feature releases around 18 months after X.(Y+1).0 is released.This means there is a 6-month period every other year where only one featurerelease is supported (and that period overlaps with the pre-release testingperiod for the X.(Y+2).0 baseline feature release).
Importantly for the proposal in this PEP, that support period would abide bythe recommendation that library developers maintain support for the previousrelease series until the latest release series has attained feature completestatus: dropping support 18 months after the baseline feature release will beroughly equivalent to dropping support 6 months after the feature completerelease, without needing to track exactlywhich release marked the series asfeature complete.
For the purposes of this PEP, a “simple” deployment environment is any use casewhere it is straightforward to ensure that all target environments are updatedto a new Python micro version at the same time (or at least in advance of therollout of new higher level application versions), and there isn’t anyrequirement for older Python versions to be able to reliably read pickle streamsgenerated with the newer Python version, such that any pre-release testing thatoccurs need only target a single Python micro version.
The simplest such case would be scripting for personal use, where the testingand target environments are the exact same environment.
Similarly simple environments would be containerised web services, where thesame Python container is used in the CI pipeline as is used on deployment, andany application that bundles its own Python runtime, rather than relying on apre-existing Python deployment on the target system.
For these use cases, this PEP shouldn’t have any significant implications - onlya single micro version needs to be tested, independently of whether thatversion is feature complete or not.
For the purposes of this PEP, “complex” deployment environments are use caseswhich don’t meet the “simple deployment” criterion above: new applicationversions are combined with two or more distinct micro versions withinthe same release series as part of the deployment process, rather than alwaystargeting exactly one micro version at a time.
If the proposal in this PEP has the desired effect of reducing feature deliverylatency, then it can be expected that developers using a release series that isnot yet feature complete will actually make use of the new features as they’remade available. This then means that testing against a newer incremental featurerelease becomes an even less valid test of compatibility with the baselinefeature release and older incremental feature releases than testing against anewer maintenance release is for older maintenance releases.
One option for handling such cases is to simply prohibit the use of new Pythonversions until the series has reached “feature complete” status. Such a policyis effectively already adopted by many organisations when it comes to newfeature release series, with acceptance into operational environments occurringmonths or years after the original release. If this policy is adopted, then suchorganisations could potentially still adopt a new Python version every otheryear - it would just be based on the availability of the feature completereleases, rather than the baseline feature releases.
A less strict alternative to outright prohibition would be to make use of theproposedPYTHONFEATURELIMIT setting to enable phased migrations to newincremental feature releases:
PYTHONFEATURELIMIT=X.Y.0 set in CIand on deploymentPYTHONFEATURELIMIT=X.Y.0 settingPYTHONFEATURELIMIT=X.Y.1PYTHONFEATURELIMIT=X.Y.1 in CI only after all deployment environmentshave been updatedPYTHONFEATURELIMIT and leaveit at the feature complete version numberThis PEP proposes that feature additions be limited to 12 months after theinitial baseline feature release.
The primary motivation for that is specifically to sync up with the Ubuntu LTStiming, such that the feature complete release for the Python 3.9.x series getspublished in October 2021, ready for inclusion in the Ubuntu 22.04 release.(other LTS Linux distributions like RHEL, SLES, and Debian don’t have a fixedpublishing cadence, so they can more easily tweak their LTS timing a bit toalign with stable versions of their inputs. Canonical deliberately haven’tgiven themselves that flexibility with their own release cycle).
The 12 month feature addition period then arises from splitting the timefrom the 2019-10 release of Python 3.8.0 and a final Python 3.9.x incrementalfeature release in 2021-10 evenly between pre-release development and subsequentincremental feature releases.
This is an area where this PEP could adopt part of the proposal inPEP 596,by instead making that split ~9 months of pre-release development, and ~15months of incremental feature releases:
This approach would mean there were still always two or three active branches,it’s just that proportionally more time would be spent with a branch in the“feature releases” phase, as compared to the “pre-alpha”, “pre-beta”, and“pre-release” phases:
In the baseline proposal in this PEP, the proposed timelines still includeperiods where we go for 18 months without making a release from the main gitbranch (e.g. 3.9.0b1 would branch off in 2020-05, and 3.10.0a1 wouldn’t bepublished until 2021-11). They just allow for a wider variety of changes tobe backported to the most recent maintenance branch for 12 of those months.
The variant of the proposal that moves the beta branch point earlier in therelease series lifecycle would increase that period of no direct releases to21 months - the only period where releases were made directly from the mainbranch would be during the relatively short window between the last incrementalfeature release of the previous release series, and the beta branch point afew months later.
While alternating the annual cadence between “big foundational enhancements”and “targeted low risk API usability improvements” is a deliberate feature ofthis proposal, it still seems strange to wait that long for feedback in theevent that changesare made shortly after the previous release series isbranched.
An alternative way of handling this would be to start publishing alpha releasesfor the next baseline feature release during the feature addition period (similarto the way thatPEP 596 proposes to starting publishing Python 3.9.0 alphareleases during the Python 3.8.0 release candidate period).
However, rather than setting specific timelines for that at a policy level,it may make sense to leave that decision to individual release managers, basedon the specific changes that are being proposed for the release they’remanaging.
If this were a versioning design document for a new language, itwould usesemantic versioning: the policies described above for baseline feature releaseswould be applied to X.0.0 releases, the policies for incremental featurereleases would be applied to X.Y.0 releases, and the policies for maintenancereleases would be applied to X.Y.Z releases.
The problem for Python specifically is that all the policies and properties forparallel installation support and ABI compatibility definitions are currentlyassociated with the firsttwo fields of the version number, and it has beenthat way for the better part of thirty years.
As a result, it makes sense to split out the policy question of introducingincremental feature releases in the first place from the technical question ofmaking the version numbering scheme better match the semantics of the differentrelease types.
If the proposal in this PEP were to be accepted by the Steering Council forPython 3.9, then a better time to tackle that technical question would be forthe subsequent October 2022 baseline feature release, as there are already inherentcompatibility risks associated with the choice of either “Python 4.0” (erroneouschecks for the major version being exactly 3 rather than 3 or greater), or“Python 3.10” (code incorrectly assuming that the minor version will alwayscontain exactly one decimal digit)[1].
While the text of this PEP assumes that the release published in 2022 will be3.10 (as the PEP author personally considers that the more reasonable and mostlikely choice), there are complex pros and cons on both sides of that decision,and this PEP does arguably add a potential pro in favour of choosing the“Python 4.0” option (with the caveat that we would also need to amend theaffected installation layout and compatibility markers to only consider themajor version number, rather than both the major and minor version).
If such a version numbering change were to be proposed and accepted, then theexample 3.10.x timeline given above would instead become the following 4.xseries timeline:
And the 5 year schedule forecast would look like:
This document has been placed in the public domain.
Source:https://github.com/python/peps/blob/main/peps/pep-0598.rst
Last modified:2025-02-01 08:59:27 GMT