PEP 543

PEP 543 defines a new TLS API for Python which would enhance Pythonsecurity by giving Python applications access to the native TLS implementationson Windows and macOS, instead of using OpenSSL. A side effect is that it givesaccess to the system trust store and certificates installedlocally by system administrators, enabling Python applications to use “companycertificates” without having to modify each application and so to correctlyvalidate TLS certificates (instead of having to ignore or bypass TLScertificate validation).

For practical reasons, Cory Benfield would like to first implement anI/O-less class similar to ssl.MemoryBIO and ssl.SSLObject forPEP 543, and to provide a second class based on the first one to usesockets or file descriptors. This design would help to structure the codeto support more backends and simplify testing and auditing, as well asimplementation. Later, optimized classes using directly sockets or filedescriptors may be added for performance.

WhilePEP 543 defines an API, the PEP would only make sense if itcomes with at least one complete and good implementation. The firstimplementation would ideally be based on thessl module of the Pythonstandard library, as this is shipped to all users by default and can be used asa fallback implementation in the absence of anything more targeted.

If this backport is not performed, the only baseline implementation that couldbe used would be pyOpenSSL. This is problematic, however, because of theinteraction with pip, which is shipped with CPython on all supported versions.

requests, pip and ensurepip

There are plans afoot to look at moving Requests to a more event-loop-ymodel. The Requests team does not feel at this time it is possible to abandonsupport for Python 2.7, so doing so would require using either Twisted orTornado, or writing their own asynchronous abstraction.

For asynchronous code, a MemoryBIO provides substantial advantages over using awrapped socket. It reduces the amount of buffering that must be done, works onIOCP-based reactors as well as select/poll based ones, and also greatlysimplifies the reactor and implementation code. For this reason, Requests isdisinclined to use a wrapped-socket-based implementation. In the absence of abackport to Python 2.7, Requests is required to use the same solution thatTwisted does: namely, a mandatory dependency onpyOpenSSL.

Thepip program has to embed all itsdependencies for practical reasons: namely, that it cannot rely on any otherinstallation method being present. Since pip depends on requests, it meansthat it would have to embed a copy of pyOpenSSL. That would imply substantialusability pain to install pip. Currently, pip doesn’t support embeddingC extensions which must be compiled on each platform and so require a Ccompiler.

Since Python 2.7.9, Python embeds a copy of pip both for defaultinstallation and for use in virtual environments via the newensurepipmodule. If pip ends up bundling PyOpenSSL, then CPython will end upbundling PyOpenSSL. Only backportingssl.MemoryBIO andssl.SSLObject would avoid the need to embed pyOpenSSL, and would fix thebootstrap issue (python -> ensurepip -> pip -> requests -> MemoryBIO).

This situation is less problematic than the barrier to adoption ofPEP 543, asnaturally Requests does not have to move to an event loop model before it dropssupport for Python 2.7. However, it does make it painful for Requests (and pip)to embrace both asyncio and theasync andawait keywords for as long asit continues to support Python 2.

Other Benefits

Adopting this PEP would have other smaller ecosystem benefits. For example,Twisted would be able to reduce its dependency on third-party C extensions.Additionally, the PyOpenSSL development team would like to sunset the module,and this backport would free them up to do so in a graceful manner withoutleaving their users in the lurch.

Each of these fringe benefits, while small, also provides value to the widerPython ecosystem.

What About Old Python 2?

A number of the Python 2 users in the world are not keeping pace with Python 2releases. This is most usually because they are using LTS releases that are notkeeping pace with the minor releases of Python 2. These users would not be ableto use the MemoryBIO, and so projects concerned with Python 2 compatibility maybe unable to rely on the MemoryBIO being present on most of their user’ssystems.

This concern is reasonable. How critical it is depends on the likelihood ofcurrent users of Python 2 migrating to Python 3, or just trying to use the mostrecent Python 2 release. Put another way, at some point libraries will want todrop Python 2 support: the question is only whether a significant majority oftheir Python 2 users have moved to whatever Python 2 release contains thisbackport before they do so.

Ultimately, the authors of this PEP believe that the burden of this backport issufficiently minimal to justify backporting despite this concern. If it turnsout that migration to newer 2.7 releases is too slow, then the value of thework will be minimal, but if the migration to newer 2.7 releases is anythinglike reasonable then there will be substantial value gained.