The need to evolve frequently

To remain relevant and useful, Python has to evolve frequently; someenhancements require incompatible changes. Any incompatible change canbreak an unknown number of Python projects. Developers can decide tonot implement a feature because of that.

Users want to get the latest Python version to obtain new features andbetter performance. A few incompatible changes can prevent them from using theirapplications on the latest Python version.

This PEP proposes to add a partial compatibility with old Pythonversions as a tradeoff to fit both use cases.

The main issue with the migration from Python 2 to Python 3 is not thatPython 3 is backward incompatible, but how incompatible changes wereintroduced.

Partial compatibility to minimize the Python maintenance burden

While technically it would be possible to provide full compatibilitywith old Python versions, this PEP proposes to minimize the number offunctions handling backward compatibility to reduce the maintenanceburden of the Python project (CPython).

Each change introducing backport compatibility to a function should beproperly discussed to estimate the maintenance cost in the long-term.

Backward compatibility code will be dropped on each Python release, on acase-by-case basis. Each compatibility function can be supported for adifferent number of Python releases depending on its maintenance costand the estimated risk (number of broken projects) if it’s removed.

The maintenance cost does not only come from the code implementing thebackward compatibility, but also comes from the additional tests.

Cases excluded from backward compatibility

The performance overhead of any compatibility code must be low whensys.set_python_compat_version() is not called.

The C API is out of the scope of this PEP:Py_LIMITED_API macro andthe stable ABI are solving this problem differently, see thePEP 384:Defining a Stable ABI.

Security fixes which break backward compatibility on purpose willnot get a compatibility layer; security matters more than compatibility.For example,http.client.HTTPSConnection was modified in Python3.4.3 to performs all the necessary certificate and hostname checks bydefault. It was a deliberate change motivated byPEP 476: Enablingcertificate verification by default for stdlib http clients (bpo-22417).

The Python language does not provide backward compatibility.

Changes which are not clearly incompatible are not covered by this PEP.For example, Python 3.9 changed the default protocol in thepicklemodule to Protocol 4 which was first introduced in Python 3.4. Thischange is backward compatible up to Python 3.4. There is no need to usethe Protocol 3 by default when compatibility with Python 3.8 isrequested.

The newDeprecationWarning andPendingDeprecatingWarning warningsin Python 3.9 will not be disabled in Python 3.8 compatibility mode.If a project runs its test suite using-Werror (treat any warning asan error), these warnings must be fixed, or specific deprecationwarnings must be ignored on a case-by-case basis.

Upgrading a project to a newer Python

Without backward compatibility, all incompatible changes must be fixedat once, which can be a blocker issue. It is even worse when a projectis upgraded to a newer Python which is separated by multiple releasesfrom the old Python.

Postponing an upgrade only makes things worse: each skipped release addsmore incompatible changes. The technical debt only steadilyincreases over time.

With backward compatibility, it becomes possible to upgrade Pythonincrementally in a project, without having to fix all of the issues at once.

The “all-or-nothing” is a showstopper to port large Python 2 code basesto Python 3. The list of incompatible changes between Python 2 andPython 3 is long, and it’s getting longer with each Python 3.x release.

Cleaning up Python and DeprecationWarning

One of theZen of Python (PEP 20) motto is:

There should be one– and preferably only one –obvious way to doit.

When Python evolves, new ways inevitably emerge.DeprecationWarningsare emitted to suggest using the new way, but many developers ignorethese warnings, which are silent by default (except in the__main__module: see thePEP 565).Some developers simply ignore all warnings when there are too manywarnings, thus only bother with exceptions when the deprecated code isremoved.

Sometimes, supporting both ways has a minor maintenance cost, butdevelopers prefer to drop the old way to clean up their code. These kinds ofchanges are backward incompatible.

Some developers can take the end of the Python 2 support as anopportunity to push even more incompatible changes than usual.

Adding an opt-in backward compatibility prevents the breaking ofapplications and allows developers to continue doing these cleanups.

Redistribute the maintenance burden

The backward compatibility involves authors of incompatiblechanges more in the upgrade path.

collections ABC aliases

collections.abc aliases to ABC classes have been removed from thecollections module in Python 3.9, after being deprecated sincePython 3.3. For example,collections.Mapping no longer exists.

In Python 3.6, aliases were created incollections/__init__.py byfrom_collections_abcimport*.

In Python 3.7, a__getattr__() has been added to thecollectionsmodule to emit a DeprecationWarning upon first access to anattribute:

def__getattr__(name):# For backwards compatibility, continue to make the collections ABCs# through Python 3.6 available through the collections module.# Note: no new collections ABCs were added in Python 3.7ifnamein_collections_abc.__all__:obj=getattr(_collections_abc,name)importwarningswarnings.warn("Using or importing the ABCs from 'collections' instead ""of from 'collections.abc' is deprecated since Python 3.3, ""and in 3.9 it will be removed.",DeprecationWarning,stacklevel=2)globals()[name]=objreturnobjraiseAttributeError(f'module{__name__!r} has no attribute{name!r}')

Compatibility with Python 3.8 can be restored in Python 3.9 by addingback the__getattr__() function, but only when backwardcompatibility is requested:

def__getattr__(name):if(sys.get_python_compat_version()<(3,9)andnamein_collections_abc.__all__):...raiseAttributeError(f'module{__name__!r} has no attribute{name!r}')

Deprecated open() “U” mode

The"U" mode ofopen() is deprecated since Python 3.4 and emits aDeprecationWarning.bpo-37330 proposes to drop this mode:open(filename,"rU") would raise an exception.

This change falls into the “cleanup” category: it is not required toimplement a feature.

A backward compatibility mode would be trivial to implement and would bewelcomed by users.

sys functions

Add 3 functions to thesys module:

• sys.set_python_compat_version(version): set the Pythoncompatibility version. If it has been called previously, use theminimum of requested versions. Raise an exception ifsys.set_python_min_compat_version(min_version) has been called andversion<min_version.version must be greater than or equal to(3,0).
• sys.set_python_min_compat_version(min_version): set theminimum compatibility version. Raise an exception ifsys.set_python_compat_version(old_version) has been calledpreviously andold_version<min_version.min_version must be greater than or equal to(3,0).
• sys.get_python_compat_version(): get the Python compatibilityversion. Return atuple of 3 integers.

Aversion must a tuple of 2 or 3 integers.(major,minor) versionis equivalent to(major,minor,0).

By default,sys.get_python_compat_version() returns the currentPython version.

For example, to request compatibility with Python 3.8.0:

importcollectionssys.set_python_compat_version((3,8))# collections.Mapping alias, removed from Python 3.9, is available# again, even if collections has been imported before calling# set_python_compat_version().parent=collections.Mapping

Obviously, callingsys.set_python_compat_version(version) has noeffect on code executed before the call. Use-Xcompat_version=VERSION command line option orPYTHONCOMPATVERSIONVERSION=VERSION environment variable to set thecompatibility version at Python startup.

Command line

Add-Xcompat_version=VERSION and-Xmin_compat_version=VERSIONcommand line options: call respectivelysys.set_python_compat_version() andsys.set_python_min_compat_version().VERSION is a version stringwith 2 or 3 numbers (major.minor.micro ormajor.minor). Forexample,-Xcompat_version=3.8 callssys.set_python_compat_version((3,8)).

AddPYTHONCOMPATVERSIONVERSION=VERSION andPYTHONCOMPATMINVERSION=VERSION=VERSION environment variables: callrespectivelysys.set_python_compat_version() andsys.set_python_min_compat_version().VERSION is a versionstring with the same format as the command line options.

Provide a workaround for each incompatible change

An application can work around most incompatible changes whichimpacts it.

For example,collections aliases can be added back using:

importcollections.abccollections.Mapping=collections.abc.Mappingcollections.Sequence=collections.abc.Sequence

Handle backward compatibility in the parser

The parser is modified to support multiple versions of the Pythonlanguage (grammar).

The current Python parser cannot be easily modified for that. AST andgrammar are hardcoded to a single Python version.

In Python 3.8,compile() has an undocumented_feature_version to not considerasync andawait askeywords.

The latest major language backward incompatible change was Python 3.7which madeasync andawait real keywords. It seems like Twistedwas the only affected project, and Twisted had a single affectedfunction (it used a parameter calledasync).

Handling backward compatibility in the parser seems quite complex, notonly to modify the parser, but also for developers who have to checkwhich version of the Python language is used.

from__future__importpython35_syntaxasync=1await=2

Temporary moratorium on incompatible changes

In 2009,PEP 3003 “Python Language Moratorium” proposed atemporary moratorium (suspension) of all changes to the Python languagesyntax, semantics, and built-ins for Python 3.1 and Python 3.2.

In May 2018, during thePEP 572 discussions, it was also proposed to slowdown Python changes: see the python-dev threadSlow down…

Barry Warsaw’s call on this:

I don’t believe that the way for Python to remain relevant anduseful for the next 10 years is to cease all language evolution.Who knows what the computing landscape will look like in 5 years,let alone 10? Something as arbitrary as a 10-year moratorium is(again, IMHO) a death sentence for the language.

PEP 387

PEP 387 – Backwards Compatibility Policy proposes a process to makeincompatible changes. The main point is the 4th step of the process:

See if there’s any feedback. Users not involved in the originaldiscussions may comment now after seeing the warning. Perhapsreconsider.

PEP 497

PEP 497 – A standard mechanism for backward compatibility proposes differentsolutions to provide backward compatibility.

Except for the__past__ mechanism idea,PEP 497 does not proposeconcrete solutions:

When an incompatible change to core language syntax or semantics isbeing made, Python-dev’s policy is to prefer and expect that,wherever possible, a mechanism for backward compatibility beconsidered and provided for future Python versions after thebreaking change is adopted by default, in addition to any mechanismsproposed for forward compatibility such as new future_statements.

Python 3.8

Examples of Python 3.8 incompatible changes:

• (During beta phase)PyCode_New() required a new parameter: itbroke all Cython extensions (all projects distributing precompiledCython code). This change has been reverted during the 3.8 beta phaseand a newPyCode_NewWithPosOnlyArgs() function was added instead.
• types.CodeType requires an additional mandatory parameter.TheCodeType.replace() function was added to help projects to nolonger depend on the exact signature of theCodeType constructor.
• C extensions are no longer linked to libpython.
• sys.abiflags changed from'm' to an empty string.For example,python3.8m program is gone.
• The C structurePyInterpreterState was made opaque.
  • Blender:
    • https://bugzilla.redhat.com/show_bug.cgi?id=1734980#c6
    • https://developer.blender.org/D6038
• XML attribute order:bpo-34160. Broken projects:
  • coverage
  • docutils
  • pcs
  • python-glyphsLib

Backward compatibility cannot be added for all these changes. Forexample, changes in the C API and in the build system are out of thescope of this PEP.

SeeWhat’s New In Python 3.8: API and Feature Removalsfor all changes.

See also thePorting to Python 3.8section of What’s New In Python 3.8.

Python 3.7

Examples of Python 3.7 incompatible changes:

• async andawait are now reserved keywords.
• Several undocumented internal imports were removed. One example isthatos.errno is no longer available; useimporterrnodirectly instead. Note that such undocumented internal imports may beremoved any time without notice, even in micro version releases.
• Unknown escapes consisting of'\' and an ASCII letter inreplacement templates forre.sub() were deprecated in Python 3.5,and will now cause an error.
• Theasyncio.windows_utils.socketpair() function has been removed:it was an alias tosocket.socketpair().
• asyncio no longer exports theselectors and_overlappedmodules asasyncio.selectors andasyncio._overlapped. Replacefromasyncioimportselectors withimportselectors.
• PEP 479 is enabled for all code in Python 3.7, meaning thatStopIteration exceptions raised directly or indirectly incoroutines and generators are transformed intoRuntimeErrorexceptions.
• socketserver.ThreadingMixIn.server_close() now waits until allnon-daemon threads complete. Set the newblock_on_close classattribute toFalse to get the pre-3.7 behaviour.
• Thestruct.Struct.format type is nowstr instead ofbytes.
• repr fordatetime.timedelta has changed to include the keywordarguments in the output.
• tracemalloc.Traceback frames are now sorted from oldest to mostrecent to be more consistent withtraceback.

Adding backward compatibility for most of these changes would be easy.

See also thePorting to Python 3.7section of What’s New In Python 3.7.

Micro releases

Sometimes, incompatible changes are introduced in micro releases(micro inmajor.minor.micro) to fix bugs or securityvulnerabilities. Examples include:

• Python 3.7.2,compileall andpy_compile module: theinvalidation_mode parameter’s default value is updated toNone;theSOURCE_DATE_EPOCH environment variable no longeroverrides the value of theinvalidation_mode argument, anddetermines its default value instead.
• Python 3.7.1,xml modules: the SAX parser no longer processesgeneral external entities by default to increase security by default.
• Python 3.5.2,os.urandom(): on Linux, if thegetrandom()syscall blocks (the urandom entropy pool is not initialized yet), fallback on reading/dev/urandom.
• Python 3.5.1,sys.setrecursionlimit(): aRecursionErrorexception is now raised if the new limit is too low at the currentrecursion depth.
• Python 3.4.4,ssl.create_default_context(): RC4 was dropped fromthe default cipher string.
• Python 3.4.3,http.client:HTTPSConnection now performs allthe necessary certificate and hostname checks by default.
• Python 3.4.2,email.message:EmailMessage.is_attachment() isnow a method instead of a property, for consistency withMessage.is_multipart().
• Python 3.4.1,os.makedirs(name,mode=0o777,exist_ok=False):Before Python 3.4.1, ifexist_ok wasTrue and the directoryexisted,makedirs() would still raise an error ifmode did notmatch the mode of the existing directory. Since this behavior wasimpossible to implement safely, it was removed in Python 3.4.1(bpo-21082).

Examples of changes made in micro releases which are not backwardincompatible:

• ssl.OP_NO_TLSv1_3 constant was added to 2.7.15, 3.6.3 and 3.7.0for backwards compatibility with OpenSSL 1.0.2.
• typing.AsyncContextManager was added to Python 3.6.2.
• Thezipfile module accepts a path-like object since Python 3.6.2.
• loop.create_future() was added to Python 3.5.2 in theasynciomodule.

No backward compatibility code is needed for these kinds of changes.