Too few projects are involved in the Python beta phase

Currently, Python beta versions are available four months before thefinal 3.x.0 release.

Bugs reported during the beta phase can be easily fixed and can block arelease if they are serious enough.

Incompatible changes are discussed during the beta phase: enhancedocumentation explaining how to update code, or consider to revert thesechanges.

Even if more and more projects are tested on the master branch of Pythonin their CI, too many projects of the top 50 PyPI projects are onlycompatible with the new Python a few weeks, or even months, after thefinal Python release.

DeprecationWarning is being ignored

Python has well defined process to deprecate features. ADeprecationWarning must be emitted during at least one Python release,before a feature can be removed.

In practice, DeprecationWarning warnings are ignored for years in majorPython projects. Usually, maintainers explain that there are too manywarnings and so they simply ignore warnings. Moreover, DeprecationWarningis silent by default (except in the__main__ module:PEP 565).

Even if more and more projects are running their test suite withwarnings treated as errors (-Werror), Python core developers stillhave no idea how many projects are broken when a feature is removed.

Need to coordinate

When issues and incompatible changes are discovered and discussed afterthe final Python release, it becomes way more complicated and expensiveto fix Python. Once an API is part of an official final release, Pythonshould provide backward compatibility for the whole 3.x releaselifetime. Some operating systems can be shipped with the buggy finalrelease and can take several months before being updated.

Too many projects are only updated to the new Python after the finalPython release, which makes this new Python version barely usable to runlarge applications when Python is released.

It is proposed to block a Python release until a compatible version ofall selected projects is available.

Shorter Python release schedule

ThePEP 602: Annual Release Cycle for Python and thePEP 605: Arolling feature release stream for CPython would like to releasePython more often to ship new features more quickly.

The problem is that each Python3.x release breaks many projects.

Coordinated Python releases reduces the number of broken projects andmakes new Python release more usable.

Limit the delay

When a build or test issue with the next Python version is reported to aproject, maintainers have one month to answer. With no answer, theproject can be excluded from the list of projects blocking the Pythonrelease.

Multiple projects are already tested on the master branch of Python in aCI. Problems can be detected very early in a Python release which shouldprovide enough time to handle them. More CI can be added for projectswhich are not tested on the next Python yet.

Once selected projects issues are known, exceptions can be discussedbetween the Python release manager and involved project maintainers on acase-by-case basis. Not all issues deserve to block a Python release.

Selected projects

List of projects blocking a Python release (total: 27):

• Projects (13):
  • aiohttp
  • cryptography
  • Cython
  • Django
  • numpy
  • pandas
  • pip
  • requests
  • scipy
  • Sphinx (needed to build Python)
  • sqlalchemy
  • pytest
  • tox
• Direct and indirect dependencies (14):
  • certifi (needed by urllib3)
  • cffi (needed by cryptography)
  • chardet (needed by Sphinx)
  • colorama (needed by pip)
  • docutils (needed by Sphinx)
  • idna (needed by Sphinx and requests)
  • jinja2 (needed by Sphinx)
  • MarkupSafe (needed by Sphinx)
  • psycopg2 (needed by Django)
  • pycparser (needed by cffi)
  • setuptools (needed by pip and tons of Python projects)
  • six (needed by tons of Python projects)
  • urllib3 (needed by requests)
  • wheel (needed by pip)

How projects are selected

Projects used by to build Python should be in the list, like Sphinx.

Most popular projects are picked from the most downloaded PyPI projects.

Most of project dependencies are included in the list as well, since asingle incompatible dependency can block a whole project. Somedependencies are excluded to reduce the list length.

Test dependencies as pytest and tox should be included as well. If aproject cannot be tested, a new version cannot be shipped neither.

The list should be long enough to have a good idea of the cost ofporting a project to the next Python, but small enough to not block aPython release for too long.

Obviously, projects which are not part of the list also are encouragedto report issues with the next Python and to have a CI running on thenext Python version.

Examples

There are different kinds of incompatible changes:

• Change in the Python build. For example, Python 3.8 removed'm'(which stands for pymalloc) fromsys.abiflags which impacts Pythonvendors like Linux distributions.
• Change in the C extensions build. For example, Python 3.8 no longerlinks C extensions to libpython, and Python 3.7 removedos.errno alias to theerrno module.
• Removed function. For example, collections aliases to ABC classeshave been removed in Python 3.9.
• Changed function signature:
  • Reject a type which was previously accepted (ex: only acceptint,rejectfloat).
  • Add a new mandatory parameter.
  • Convert a positional-or-keyword parameter to positional-only.
• Behavior change. For example, Python 3.8 now serializes XML attributesin their insertion order, rather than sorting them by name.
• New warning. Since more and more projects are tested with all warningstreated as errors, any new warning can cause a project test to fail.
• Function removed from the C API.
• Structure made opaque in the C API. For example, PyInterpreterStatebecame opaque in Python 3.8 which broke projects accessinginterp->modules (PyImport_GetModuleDict() should be usedinstead).

Cleaning up Python and DeprecationWarning

One of theZen of Python (PEP 20) motto is:

There should be one– and preferably only one –obvious way to doit.

When Python evolves, new ways emerge inevitably.DeprecationWarningare emitted to suggest to use the new way, but many developers ignorethese warnings which are silent by default.

Sometimes, supporting both ways has a minor maintenance cost, but Pythoncore developers prefer to drop the old way to clean up the Python codebase and standard library. Such kind of change is backward incompatible.

More incompatible changes than usual should be expected with the end ofthe Python 2 support which is a good opportunity to cleaning up oldPython code.