With the acceptance ofPEP 453 meaning thatpip will be available tomost new Python users by default, this will hopefully reduce the pressureto add new modules to the standard library before they are sufficientlymature.
The last couple of years have also seen increased usage of the model wherea standard library package also has an equivalent available from the PythonPackage Index that also supports older versions of Python.
Given these two developments and the level of engagement throughout thePython 3.4 release cycle, the PEP author no longer feels it would beappropriate to make such a fundamental change to the standard librarydevelopment process.
This PEP proposes the adoption of a separate versioning scheme for thestandard library (distinct from, but coupled to, the existing languageversioning scheme) that allows accelerated releases of the Python standardlibrary, while maintaining (or even slowing down) the current rate ofchange in the core language definition.
LikePEP 407, it aims to adjust the current balance between measuredchange that allows the broader community time to adapt and being able tokeep pace with external influences that evolve more rapidly than the currentrelease cycle can handle (this problem is particularly notable forstandard library elements that relate to web technologies).
However, it’s more conservative in its aims thanPEP 407, seeking torestrict the increased pace of development to builtin and standard libraryinterfaces, without affecting the rate of change for other elements suchas the language syntax and version numbering as well as the CPythonbinary API and bytecode format.
To quote thePEP 407 abstract:
Finding a release cycle for an open-source project is a delicate exercisein managing mutually contradicting constraints: developer manpower,availability of release management volunteers, ease of maintenance forusers and third-party packagers, quick availability of new features (andbehavioural changes), availability of bug fixes without pulling in newfeatures or behavioural changes.The current release cycle errs on the conservative side. It is adequatefor people who value stability over reactivity. This PEP is an attempt tokeep the stability that has become a Python trademark, while offering amore fluid release of features, by introducing the notion of long-termsupport versions.
I agree with thePEP 407 authors that the current release cycle of thestandard library is too slow to effectively cope with the pace of changein some key programming areas (specifically, web protocols and relatedtechnologies, including databases, templating and serialisation formats).
However, I have written this competing PEP because I believe that theapproach proposed inPEP 407 of offering full, potentially binaryincompatible releases of CPython every 6 months places too great a burdenon the wider Python ecosystem.
Under the current CPython release cycle, distributors of key binaryextensions will often support Python releases even after the CPython branchesenter “security fix only” mode (for example, Twisted currently ships binariesfor 2.5, 2.6 and 2.7, NumPy and SciPy support those 3 along with 3.1 and 3.2,PyGame adds a 2.4 binary release, wxPython provides both 32-bit and 64-bitbinaries for 2.6 and 2.7, etc).
If CPython were to triple (or more) its rate of releases, the developers ofthose libraries (many of which are even more resource starved than CPython)would face an unpalatable choice: either adopt the faster release cyclethemselves (up to 18 simultaneous binary releases for PyGame!), dropolder Python versions more quickly, or else tell their users to stick to theCPython LTS releases (thus defeating the entire point of speeding up theCPython release cycle in the first place).
Similarly, many support tools for Python (e.g. syntax highlighters) can takequite some time to catch up with language level changes.
At a cultural level, the Python community is also accustomed to a certainmeaning for Python version numbers - they’re linked to deprecation periods,support periods, all sorts of things.PEP 407 proposes that collectiveknowledge all be swept aside, without offering a compelling rationale for whysuch a course of action is actuallynecessary (aside from, perhaps, makingthe lives of the CPython core developers a little easier at the expense ofeveryone else).
However, if we go back to the primary rationale for increasing the pace ofchange (i.e. more timely support for web protocols and related technologies),we can note that those only requirestandard library changes. That meansmany (perhaps even most) of the negative effects on the wider community canbe avoided by explicitly limiting which parts of CPython are affected by thenew release cycle, and allowing other parts to evolve at their current, moresedate, pace.
This PEP proposes the introduction of a new kind of CPython release:“standard library releases”. As withPEP 407, this will give CPython 3 kindsof release:
Under this scheme, an unqualified version reference (such as “3.3”) wouldalways refer to the most recent corresponding language or maintenancerelease. It will never be used without qualification to refer to a standardlibrary release (at least, not by python-dev - obviously, we can only set anexample, not force the rest of the Python ecosystem to go along with it).
Language releases will continue as they are now, as new versions of thePython language definition, along with a new version of the CPythoninterpreter and the Python standard library. Accordingly, a languagerelease may contain any and all of the following changes:
Maintenance releases will also continue as they do today, being strictlylimited to bug fixes for the corresponding language release. No new featuresor radical internal changes are permitted.
The new standard library releases will occur in parallel with eachmaintenance release and will be qualified with a new version identifierdocumenting the standard library version. Standard library releases mayinclude the following changes:
Standard library version identifiers are constructed by combining the majorand minor version numbers for the Python language release into a single twodigit number and then appending a sequential standard library versionidentifier.
When maintenance releases are created,two new versions of Python wouldactually be published on python.org (using the first 3.3 maintenance release,planned for February 2013 as an example):
3.3.1# Maintenance release3.3(33.1)# Standard library release
A further 6 months later, the next 3.3 maintenance release would again beaccompanied by a new standard library release:
3.3.2# Maintenance release3.3(33.2)# Standard library release
Again, the standard library release would be binary compatible with theprevious language release, merely offering additional features at thePython level.
Finally, 18 months after the release of 3.3, a new language release wouldbe made around the same time as the final 3.3 maintenance and standardlibrary releases:
3.3.3# Maintenance release3.3(33.3)# Standard library release3.4.0# Language release
The 3.4 release cycle would then follow a similar pattern to that for 3.3:
3.4.1# Maintenance release3.4(34.1)# Standard library release3.4.2# Maintenance release3.4(34.2)# Standard library release3.4.3# Maintenance release3.4(34.3)# Standard library release3.5.0# Language release
To expose the new version details programmatically, this PEP proposes theaddition of a newsys.stdlib_info attribute that records the newstandard library version above and beyond the underlying interpreterversion. Using the initial Python 3.3 release as an example:
sys.stdlib_info(python=33,version=0,releaselevel='final',serial=0)
This information would also be included in thesys.version string:
Python3.3.0(33.0,default,Feb172012,23:03:41)[GCC4.6.1]
For maintenance releases the process of handling out-of-cycle releases (forexample, to fix a security issue or resolve a critical bug in a new release),remains the same as it is now: the minor version number is incremented and anew release is made incorporating the required bug fixes, as well as anyother bug fixes that have been committed since the previous release.
For standard library releases, the process is essentially the same, but thecorresponding “What’s New?” document may require some tidying up for therelease (as the standard library release may incorporate new features,not just bug fixes).
The versioning scheme proposed above is based on a number of user scenariosthat are likely to be encountered if this scheme is adopted. In each case,the scenario is described for both the status quo (i.e. slow release cycle)the versioning scheme in this PEP and the free wheeling minor version numberscheme proposed inPEP 407.
To give away the ending, the point of using a separate version number is thatfor almost all scenarios, the important number is thelanguage version, notthe standard library version. Most users won’t even need to care that thestandard library version number exists. In the two identified cases whereit matters, providing it as a separate number is actually clearer and moreexplicit than embedding the two different kinds of number into a singlesequence and then tagging some of the numbers in the unified sequence asspecial.
Status quo: must choose between 3.3 and 2.7
This PEP: must choose between 3.3 (33.1), 3.3 and 2.7.
PEP 407: must choose between 3.4, 3.3 (LTS) and 2.7.
Verdict: explaining the meaning of a Long Term Support release is about ascomplicated as explaining the meaning of the proposed standard library releaseversion numbers. I call this a tie.
Status quo: minor version differences indicate 18-24 months oflanguage evolution
This PEP: same as status quo for language core, standard library versionnumbers indicate 6 months of standard library evolution.
PEP 407: minor version differences indicate 18-24 months of languageevolution up to 3.3, then 6 months of language evolution thereafter.
Verdict: Since language changes and deprecations can have a much biggereffect on the accuracy of third party documentation than the addition of newfeatures to the standard library, I’m calling this a win for the schemein this PEP.
Status quo: look for the binary corresponding to the Python version you arerunning.
This PEP: same as status quo.
PEP 407 (full releases): same as status quo, but corresponding binary versionis more likely to be missing (or, if it does exist, has to be found amongsta much larger list of alternatives).
PEP 407 (ABI updates limited to LTS releases): all binary release pages willneed to tell users that Python 3.3, 3.4 and 3.5 all need the 3.3 binary.
Verdict: I call this a clear win for the scheme in this PEP. Absolutelynothing changes from the current situation, since the standard libraryversion is actually irrelevant in this case (only binary extensioncompatibility is important).
Status quo: unless using thePEP 384 stable ABI, a new binary release isneeded every time the minor version number changes.
This PEP: same as status quo.
PEP 407 (full releases): same as status quo, but becomes a far morefrequent occurrence.
PEP 407 (ABI updates limited to LTS releases): before deciding, must firstlook up whether the new release is an LTS release or an interim release. Ifit is an LTS release, then a new build is necessary.
Verdict: I call this another clear win for the scheme in this PEP. As withthe end user facing side of this problem, the standard library version isactually irrelevant in this case. Moving that information out to aseparate number avoids creating unnecessary confusion.
Status quo: code that triggers deprecation warnings is not guaranteed torun on a version of Python with a higher minor version number.
This PEP: same as status quo
PEP 407: unclear, as the PEP doesn’t currently spell this out. Assuming thedeprecation cycle is linked to LTS releases, then upgrading to a non-LTSrelease is safe but upgrading to the next LTS release may require avoidingthe deprecated construct.
Verdict: another clear win for the scheme in this PEP since, once again, thestandard library version is irrelevant in this scenario.
Status quo: new Python versions arrive infrequently, but are a mish-mash ofstandard library updates and core language definition and interpreterchanges.
This PEP: standard library updates, which are easier to integrate, aremade available more frequently in a form that is clearly and explicitlycompatible with the previous version of the language definition. This meansthat, once an alternative implementation catches up to Python 3.3, theyshould have a much easier time incorporating standard library features asthey happen (especially pure Python changes), leaving minor version numberupdates as the only task that requires updates to their core compilation andexecution components.
PEP 407 (full releases): same as status quo, but becomes a far morefrequent occurrence.
PEP 407 (language updates limited to LTS releases): unclear, as the PEPdoesn’t currently spell out a specific development strategy. Assuming a3.3 compatibility branch is adopted (as proposed in this PEP), then theoutcome would be much the same, but the version number signalling would beslightly less clear (since you would have to check to see if a particularrelease was an LTS release or not).
Verdict: while not as clear cut as some previous scenarios, I’m stillcalling this one in favour of the scheme in this PEP. Explicit is better thanimplicit, and the scheme in this PEP makes a clear split between the twodifferent kinds of update rather than adding a separate “LTS” tag to anotherwise ordinary release number. Tagging a particular version as beingspecial is great for communicating with version control systems and associatedautomated tools, but it’s a lousy way to communicate information to otherhumans.
Status quo: look for “version added” or “version changed” markers in thedocumentation, check againstsys.version_info
This PEP: look for “version added” or “version changed” markers in thedocumentation. If written as a bare Python version, such as “3.3”, checkagainstsys.version_info. If qualified with a standard library version,such as “3.3 (33.1)”, check againstsys.stdlib_info.
PEP 407: same as status quo
Verdict: the scheme in this PEP actually allows third party libraries to bemore explicit about their rate of adoption of standard library features. Moreconservative projects will likely pin their dependency to the languageversion and avoid features added in the standard library releases. Fastermoving projects could instead declare their dependency on a particularstandard library version. However, sincePEP 407 does have the advantage ofpreserving the status quo, I’m calling this one forPEP 407 (albeit with aslim margin).
Status quo: if not already provided, ask the reporter which version ofPython they’re using. This is often done by asking for the first two linesdisplayed by the interactive prompt or the value ofsys.version.
This PEP: same as the status quo (assys.version will be updated toalso include the standard library version), but may be needed on additionaloccasions (where the user knew enough to state their Python version, but thatproved to be insufficient to reproduce the fault).
PEP 407: same as the status quo
Verdict: another marginal win forPEP 407. The new standard library versionis an extra piece of information that users may need to pass back todevelopers when reporting issues with Python libraries (or Python itself,on our own tracker). However, by including it insys.version, manyfault reports will already include it, and it is easy to request if needed.
Status quo: create a new maintenance release incorporating the securityfix and any other bug fixes under source control. Also create source releasesfor any branches open solely for security fixes.
This PEP: same as the status quo for maintenance branches. Also create anew standard library release (potentially incorporating new features alongwith the security fix). For security branches, create source releases forboth the former maintenance branch and the standard library update branch.
PEP 407: same as the status quo for maintenance and security branches,but handling security fixes for non-LTS releases is currently an openquestion.
Verdict: untilPEP 407 is updated to actually address this scenario, aclear win for this PEP.
Similar toPEP 407, this PEP will break up the delivery of new features intomore discrete chunks. Instead of a whole raft of changes landing all at oncein a language release, each language release will be limited to 6 monthsworth of standard library changes, as well as any changes associated withnew syntax.
This PEP proposes the creation of a single additional branch for use in thenormal workflow. After the release of 3.3, the following branches would bein use:
2.7# Maintenance branch, no change3.3# Maintenance branch, as for 3.23.3-compat# New branch, backwards compatible changesdefault# Language changes, standard library updates that depend on them
When working on a new feature, developers will need to decide whether or notit is an acceptable change for a standard library release. If so, then itshould be checked in on3.3-compat and then merged todefault.Otherwise it should be checked in directly todefault.
The “version added” and “version changed” markers for any changes made onthe3.3-compat branch would need to be flagged with both the languageversion and the standard library version. For example: “3.3 (33.1)”.
Any changes made directly on thedefault branch would just be flaggedwith “3.4” as usual.
The3.3-compat branch would be closed to normal development at thesame time as the3.3 maintenance branch. The3.3-compat branch wouldremain open for security fixes for the same period of time as the3.3maintenance branch.
The effect on the bug fix workflow is essentially the same as that on theworkflow for new features - there is one additional branch to pass throughbefore the change reaches thedefault branch.
If critical bugs are found in a maintenance release, then new maintenance andstandard library releases will be created to resolve the problem. The finalpart of the version number will be incremented for both the language versionand the standard library version.
If critical bugs are found in a standard library release that do not affectthe associated maintenance release, then only a new standard library releasewill be created and only the standard library’s version number will beincremented.
Note that in these circumstances, the standard library releasemay includeadditional features, rather than just containing the bug fix. It isassumed that anyone that cares about receivingonly bug fixes without anynew features mixed in will already be relying strictly on the maintenancereleases rather than using the new standard library releases.
PEP 407 has this to say about the effects on the community:
People who value stability can just synchronize on the LTS releases which,with the proposed figures, would give a similar support cycle (both induration and in stability).
I believe this statement is just plain wrong. Life isn’t that simple. Instead,developers of third party modules and frameworks will come under pressure tosupport the full pace of the new release cycle with binary updates, teachersand book authors will receive complaints that they’re only covering an “old”version of Python (“You’re only using 3.3, the latest is 3.5!”), etc.
As the minor version number starts climbing 3 times faster than it has in thepast, I believe perceptions of language stability would also fall (whethersuch opinions were justified or not).
I believe isolating the increased pace of change to the standard library,and clearly delineating it with a separate version number will greatlyreassure the rest of the community that no, we’re not suddenlyasking them to triple their own rate of development. Instead, we’re merelygoing to ship standard library updates for the next language release in6-monthly installments rather than delaying them all until the next languagedefinition update, even those changes that are backwards compatible with thepreviously released version of Python.
The community benefits listed inPEP 407 are equally applicable to this PEP,at least as far as the standard library is concerned:
People who value reactivity and access to new features (without taking therisk to install alpha versions or Mercurial snapshots) would get much morevalue from the new release cycle than currently.People who want to contribute new features or improvements would be moremotivated to do so, knowing that their contributions will be more quicklyavailable to normal users.
If the faster release cycle encourages more people to focus on contributingto the standard library rather than proposing changes to the languagedefinition, I don’t see that as a bad thing.
The “What’s New” documents would be split out into separate documents forstandard library releases and language releases. So, during the 3.3 releasecycle, we would see:
And then finally, we would see the next language release:
For the benefit of users that ignore standard library releases, the 3.4What’s New would link back to the What’s New documents for each of thestandard library releases in the 3.3 series.
Merge conflicts on the NEWS file are already a hassle. Since this PEPproposes introduction of an additional branch into the normal workflow,resolving this becomes even more critical. While Mercurial phases mayhelp to some degree, it would be good to eliminate the problem entirely.
One suggestion from Barry Warsaw is to adopt a non-conflictingseparate-files-per-change approach, similar to that used by Twisted[2].
Given that the current manually updated NEWS file will be used for the 3.3.0release, one possible layout for such an approach might look like:
Misc/NEWS# Now autogenerated from news_entriesnews_entries/3.3/NEWS# Original 3.3 NEWS filemaint.1/# Maintenance branch changescore/<newsentries>builtins/<newsentries>extensions/<newsentries>library/<newsentries>documentation/<newsentries>tests/<newsentries>compat.1/# Compatibility branch changesbuiltins/<newsentries>extensions/<newsentries>library/<newsentries>documentation/<newsentries>tests/<newsentries># Add maint.2, compat.2 etc as releases are made3.4/core/<newsentries>builtins/<newsentries>extensions/<newsentries>library/<newsentries>documentation/<newsentries>tests/<newsentries># Add maint.1, compat.1 etc as releases are made
Putting the version information in the directory hierarchy isn’t strictlynecessary (since the NEWS file generator could figure out from the versionhistory), but does make it easier forhumans to keep the different versionsin order.
The current release cycle is a compromise between the desire for stabilityin the core language definition and C extension ABI, and the desire to getnew features (most notably standard library updates) into user’s hands morequickly.
With the standard library release cycle decoupled (to some degree) from thatof the core language definition, it provides an opportunity to actuallyslow down the rate of change in the language definition. The languagemoratorium for Python 3.2 effectively slowed that cycle down tomore than 3years (3.1: June 2009, 3.3: August 2012) without causing any majorproblems or complaints.
The NEWS file management scheme described above is actually designed toallow us the flexibility to slow down language releases at the same timeas standard library releases become more frequent.
As a simple example, if a full two years was allowed between 3.3 and 3.4,the 3.3 release cycle would end up looking like:
3.2.4# Maintenance release3.3.0# Language release3.3.1# Maintenance release3.3(33.1)# Standard library release3.3.2# Maintenance release3.3(33.2)# Standard library release3.3.3# Maintenance release3.3(33.3)# Standard library release3.3.4# Maintenance release3.3(33.4)# Standard library release3.4.0# Language release
The elegance of the proposed branch structure and NEWS entry layout is thatthis decision wouldn’t really need to be made until shortly before the planned3.4 release date. At that point, the decision could be made to postpone the3.4 release and keep the3.3 and3.3-compat branches open after the3.3.3 maintenance release and the 3.3 (33.3) standard library release, thusadding another standard library release to the cycle. The choice betweenanother standard library release or a full language release would then beavailable every 6 months after that.
As noted in the previous section, one benefit of the scheme proposed in thisPEP is that it largely decouples the language release cycle from thestandard library release cycle. The standard library could be updated every3 months, or even once a month, without having any flow on effects on thelanguage version numbering or the perceived stability of the core language.
While that pace of development isn’t practical as long as the binaryinstaller creation for Windows and Mac OS X involves several manual steps(including manual testing) and for as long as we don’t have separate“<branch>-release” trees that only receive versions that have been marked asgood by the stable buildbots, it’s still a useful criterion to keep in mindwhen considering proposed new versioning schemes: what if we eventually wantto make standard library releases evenfaster than every 6 months?
If the practical issues were ever resolved, then the separate standardlibrary versioning scheme in this PEP could handle it. The tagged versionnumber approach proposed inPEP 407 could not (at least, not without a lotof user confusion and uncertainty).
The simplest and most logical solution would actually be to map themajor.minor.micro version numbers to the language version, stdlib versionand maintenance release version respectively.
Instead of releasing Python 3.3.0, we would instead release Python 4.0.0and the release cycle would look like:
4.0.0# Language release4.0.1# Maintenance release4.1.0# Standard library release4.0.2# Maintenance release4.2.0# Standard library release4.0.3# Maintenance release4.3.0# Standard library release5.0.0# Language release
However, the ongoing pain of the Python 2 -> Python 3 transition (andassociated workarounds like thepython3 andpython2 symlinks torefer directly to the desired release series) means that this simple optionisn’t viable for historical reasons.
One way that this simple approachcould be made to work is to merge thecurrent major and minor version numbers directly into a 2-digit majorversion number:
33.0.0# Language release33.0.1# Maintenance release33.1.0# Standard library release33.0.2# Maintenance release33.2.0# Standard library release33.0.3# Maintenance release33.3.0# Standard library release34.0.0# Language release
Another simple versioning scheme would just add a “standard library” versioninto the existing versioning scheme:
3.3.0.0# Language release3.3.0.1# Maintenance release3.3.1.0# Standard library release3.3.0.2# Maintenance release3.3.2.0# Standard library release3.3.0.3# Maintenance release3.3.3.0# Standard library release3.4.0.0# Language release
However, this scheme isn’t viable due to backwards compatibility constraintson thesys.version_info structure.
Earlier versions of this PEP proposed a date-based versioning scheme forthe standard library. However, such a scheme made it very difficult tohandle out-of-cycle releases to fix security issues and other criticalbugs in standard library releases, as it required the following steps:
With the sequential scheme now proposed, such releases should at most requirea little tidying up of the What’s New document before making the release.
PEP 384 introduced the notion of a “Stable ABI” for CPython, a limitedsubset of the full C ABI that is guaranteed to remain stable. Extensionsbuilt against the stable ABI should be able to support all subsequentPython versions with the same binary.
This will help new projects to avoid coupling their C extension modules tooclosely to a specific version of CPython. For existing modules, however,migrating to the stable ABI can involve quite a lot of work (especially forextension modules that define a lot of classes). With limited developmentresources available, any time spent on such a change is time that couldotherwise have been spent working on features that offer more direct benefitsto end users.
There are also other benefits to separate versioning (as described above)that are not directly related to the question of binary compatibility withthird party C extensions.
There’s a case to be made thatadditions to the CPython C ABI couldreasonably be permitted in standard library releases. This would give Cextension authors the same freedom as any other package or module authorto depend either on a particular language version or on a standard libraryversion.
The PEP currently associates the interpreter version with the languageversion, and therefore limits major interpreter changes (including C ABIadditions) to the language releases.
An alternative, internally consistent, approach would be to link theinterpreter version with the standard library version, with only changes thatmay affect backwards compatibility limited to language releases.
Under such a scheme, the following changes would be acceptable in standardlibrary releases:
And the following changes would be acceptable in language releases:
While such an approach could probably be made to work, there does not appearto be a compelling justification for it, and the approach currently describedin the PEP is simpler and easier to explain.
A concept that is occasionally discussed is the idea of making the standardlibrary truly independent from the CPython reference implementation.
My personal opinion is that actually making such a change would involve alot of work for next to no pay-off. CPython without the standard library isuseless (the build chain won’t even run, let alone the test suite). You alsocan’t create a standalone pure Python standard library either, because toomany “standard library modules” are actually tightly linked in to theinternal details of their respective interpreters (for example, the builtins,weakref,gc,sys,inspect,ast).
Creating a separate CPython development branch that is kept compatible withthe previous language release, and making releases from that branch that areidentified with a separate standard library version number should providemost of the benefits of a separate standard library repository with only afraction of the pain.
Thanks go to thePEP 407 authors for starting this discussion, as well asto those authors and Larry Hastings for initial discussions of the proposalmade in this PEP.
This document has been placed in the public domain.
Source:https://github.com/python/peps/blob/main/peps/pep-0413.rst
Last modified:2025-02-01 08:59:27 GMT