Most CPython tracker issues are classified as errors in behaviour orproposed enhancements. Most patches to fix behavioural errors areapplied to all active maintenance branches. Enhancement patches arerestricted to the default branch that becomes the next Python version.
This cadence works reasonably well during Python’s normal 18-24 monthfeature release cycle, which is still applicable to the Python 3 series.However, the age of the standard library in Python 2 has now reached a pointwhere it is sufficiently far behind the state of the art in network securityprotocols for it to be causing real problems in use cases where upgrading toPython 3 in the near term may not be feasible.
In recognition of the additional practical considerations that have arisenduring the 4+ year maintenance cycle for Python 2.7, this PEP allows acritical set of network security related features to be backported fromPython 3.4 to upcoming Python 2.7.x maintenance releases.
While this PEP does not make any changes to the core development team’shandling of security-fix-only branches that are no longer in activemaintenance, itdoes recommend that commercial redistributors providingextended support periods for the Python standard library either backportthese features to their supported versions, or else explicitly disclaimsupport for the use of older versions in roles that involve connectingdirectly to the public internet.
Under this proposal, the following features will be backported from Python3.4 to upcoming Python 2.7.x maintenance releases:
os module:os.urandom().hmac module:hmac.compare_digest()).hashlib module:hashlib.pbkdf2_hmac()).hashlib.algorithms_guaranteedandhashlib.algorithms_available).ssl module:ssl module featuresnot backported under this policy arethessl.RAND_* functions that provide access to OpenSSL’s randomnumber generation capabilities - useos.urandom() instead.As a general change in maintenance policy, permission is also granted toupgrade to newer feature releases of OpenSSL when preparing the binaryinstallers for new maintenance releases of Python 2.7.
This PEP does NOT propose a general exception for backporting new featuresto Python 2.7 - every new feature proposed for backporting will still needto be justified independently. In particular, it will need to be explainedwhy relying on an independently updated backport on the Python Package Indexinstead is not an acceptable solution.
This PEP originally proposed adding all listed features to the Python 2.7.7maintenance release. That approach proved to be too ambitious given thelimited time frame between the original creation and acceptance of the PEPand the release of Python 2.7.7rc1. Instead, the progress of each individualaccepted feature backport is being tracked as an independent enhancementtargeting Python 2.7.
Implemented for Python 2.7.7:
hmac.compare_digestImplemented for Python 2.7.8:
hashlib.pbkdf2Implemented for Python 2.7.9 (in development):
ssl module featureshashlib module featuresos.urandom shared file descriptor changeAs in the Python 3 series, the backportedssl.create_default_context()API is granted a backwards compatibility exemption that permits theprotocol, options, cipher and other settings of the created SSL context tobe updated in maintenance releases to use higher default security settings.This allows them to appropriately balance compatibility and security at thetime of the maintenance release, rather than at the time of the originalfeature release.
This PEP doesnot grant any other exemptions to the usual backwardscompatibility policy for maintenance releases. Instead, by explicitlyencouraging the use of feature based checks, it is designed to make it easierto write more secure cross-version compatible Python software, while stilllimiting the risk of breaking currently working software when upgrading toa new Python 2.7 maintenance release.
In all cases where this proposal allows new features to be backported tothe Python 2.7 release series, it is possible to write cross-versioncompatible code that operates by “feature detection” (for example, checkingfor particular attributes in a module), without needing to explicitly checkthe Python version.
It is then up to library and framework code to provide an appropriate warningand fallback behaviour if a desired feature is found to be missing. Whilesome especially security sensitive software MAY fail outright if a desiredsecurity feature is unavailable, most software SHOULD instead emit a warningand continue operating using a slightly degraded security configuration.
The backported APIs allow library and application code to perform thefollowing actions after detecting the presence of a relevantnetwork security related feature:
Security related changes to other modules (such as higher level networkinglibraries and data format processing libraries) will continue to be madeavailable as backports and new modules on the Python Package Index, asindependent distribution remains the preferred approach to handlingsoftware that must continue to evolve to handle changing developmentrequirements independently of the Python 2 standard library. Refer totheMotivation and Rationale section for a review of the characteristicsthat make the secure networking infrastructure worthy of specialconsideration.
Under this proposal, OpenSSL may be upgraded to more recent feature releasesin Python 2.7 maintenance releases. On Linux and most other POSIX systems,the specific version of OpenSSL used already varies, as CPython dynamicallylinks to the system provided OpenSSL library by default.
For the Windows binary installers, the_ssl and_hashlib modules arestatically linked with OpenSSL and the associated symbols are not exported.Marc-Andre Lemburg indicates that updating to newer OpenSSL releases in theegenix-pyopenssl binaries has not resulted in any reported compatibilityissues[3]
The Mac OS X binary installers historically followed the same policy asother POSIX installations and dynamically linked to the Apple providedOpenSSL libraries. However, Apple has now ceased updating thesecross-platform libraries, instead requiring that even cross-platformdevelopers adopt Mac OS X specific interfaces to access up to date securityinfrastructure on their platform. Accordingly, and independently of thisPEP, the Mac OS X binary installers were already going to be switched tostatically linker newer versions of OpenSSL[4]
A number of developers, including Alex Gaynor and Donald Stufft, haveexpressed interest in carrying out the feature backports covered by thispolicy, and assisting with any additional maintenance burdens that arisein the Python 2 series as a result.
Steve Dower and Brian Curtin have offered to help with the creation of theWindows installers, allowing Martin von Löwis the opportunity to step backfrom the task of maintaining the 2.7 Windows installer.
This PEP is primarily about establishing the consensus needed to allow themto carry out this work. For other core developers, this policy changeshouldn’t impose any additional effort beyond potentially reviewing theresulting patches for those developers specifically interested in theaffected modules.
This PEP does not propose any changes to the handling of securityreleases - those will continue to be source only releases thatinclude only critical security fixes.
However, the recommendations for library and application developers aredeliberately designed to accommodate commercial redistributors that chooseto apply these changes to additional Python release series that are eitherin security fix only mode, or have been declared “end of life” by the coredevelopment team.
Whether or not redistributors choose to exercise that option will be upto the individual redistributor.
Third party integration testing services should offer users the abilityto test against multiple Python 2.7 maintenance releases (at least 2.7.6and 2.7.7+), to ensure that libraries, frameworks and applications can stilltest their handling of the legacy security infrastructure correctly (eitherfailing or degrading gracefully, depending on the security sensitivity ofthe software), even after the features covered in this proposal have beenbackported to the Python 2.7 series.
For better or for worse (mostly worse), there are some environments wherethe risk of latent security defects is more tolerated than even a slightlyincreased risk of regressions in maintenance releases. This proposal largelyexcludes these environments from consideration where the modules covered bythe exemption are concerned - this approach is entirely inappropriate forsoftware connected to the public internet, and defence in depth securityprinciples suggest that it is not appropriate for most private networkseither.
Downstream redistributors may still choose to cater to such environments,but they will need to handle the process of downgrading the securityrelated modules and doing the associated regression testing themselves.The main CPython continuous integration infrastructure will not cover thisscenario.
The creation of this PEP was prompted primarily by the aging SSL support inthe Python 2 series. As of March 2014, the Python 2.7 SSL module isapproaching four years of age, and the SSL support in the still popularPython 2.6 release had its feature set locked six years ago.
These are simply too old to provide a foundation that can be recommendedin good conscience for secure networking software that operates over thepublic internet, especially in an era where it is becoming quite clearlyevident that advanced persistent security threats are even more widespreadand more indiscriminate in their targeting than had previously beenunderstood. While they represented reasonable security infrastructure intheir time, the state of the art has moved on, and we need to investigatemechanisms for effectively providing more up to date network securityinfrastructure for users that, for whatever reason, are not currently ina position to migrate to Python 3.
While the use of the system OpenSSL installation addresses many of theseconcerns on Linux platforms, it doesn’t address all of them (in particular,it is still difficult for sotware to explicitly require some higher levelsecurity settings). The standard library support can be bypassed by using athird party library like PyOpenSSL or Pycurl, but this still results in asecurity problem, as these can be difficult dependencies to deploy, and manyusers will remain unaware that they might want them. Rather than explainingto potentially naive users how to obtain and use these libraries, it seemsbetter to just fix the included batteries.
In the case of the binary installers for Windows and Mac OS X that arepublished on python.org, the version of OpenSSL used is entirely withinthe control of the Python core development team, but is currently limitedto OpenSSL maintenance releases for the version initially shipped with thecorresponding Python feature release.
With increased popularity comes increased responsibility, and this proposalaims to acknowledge the fact that Python’s popularity and adoption is at asufficiently high level that some of our design and policy decisions havesignificant implications beyond the Python development community.
As one example, the Python 2ssl module does not support the ServerName Indication standard. While it is possible to obtain SNI supportby using the third partyrequests client library, actually doing socurrently requires using not onlyrequests and its embedded dependencies,but also half a dozen or more additional libraries. The lack of supportin the Python 2 series thus serves as an impediment to making effectiveuse of SNI on servers, as Python 2 clients will frequently fail to handleit correctly.
Another more critical example is the lack of SSL hostname matching in thePython 2 standard library - it is currently necessary to rely on a thirdparty library, such asrequests orbackports.ssl_match_hostname toobtain that functionality in Python 2.
The Python 2 series also remains more vulnerable to remote timing attackson security sensitive comparisons than the Python 3 series, as it lacks astandard library equivalent to the timing attack resistanthmac.compare_digest() function. While appropriate secure comparisonfunctions can be implemented in third party extensions, many users don’teven consider the issue and use ordinary equality comparisons instead- while a standard library solution doesn’t automatically fix that problem,itdoes make the barrier to resolution much lower once the problem ispointed out.
Python 2.7 represents the only long term maintenance release the coredevelopment team has provided, and it is natural that there will be thingsthat worked over a historically shorter maintenance lifespan that don’t workover this longer support period. In the specific case of the problemdescribed in this PEP, the simplest available solution is to acknowledgethat long term maintenance of network security related modulesrequiresthe ability to add new features, even while retaining backwards compatibilityfor existing interfaces.
For those familiar with it, it is worth comparing the approach described inthis PEP with Red Hat’s handling of its long term open source supportcommitments: it isn’t the RHEL 6.0 release itself that receives 10 yearsworth of support, but the overall RHEL 6series. The individual RHEL 6.xpoint releases within the series then receive a wide variety of newfeatures, including security enhancements, all while meeting strictbackwards compatibility guarantees for existing software. The proposalcovered in this PEP brings our approach to long term maintenance more intoline with this precedent - we retain our strict backwards compatibilityrequirements, but make an exception to the restriction against adding newfeatures.
To date, downstream redistributors have respected our upstream policy of“no new features in Python maintenance releases”. This PEP explicitlyaccepts that a more nuanced policy is appropriate in the case of networksecurity related features, and the specific change it describes isdeliberately designed such that it is potentially suitable for Red HatEnterprise Linux and its downstream derivatives.
The key requirement for a feature to be considered for inclusion in thisproposal was that it must have security implicationsbeyond the specificapplication that is written in Python and the system that application isrunning on. Thus the focus on network security protocols, password storageand related cryptographic infrastructure - Python is a popular choice forthe development of web services and clients, and thus the capabilities ofwidely used Python versions have implications for the security design ofother services that may themselves be using newer versions of Python orother development languages, but need to interoperate with clients orservers written using older versions of Python.
The intent behind this requirement was to minimise any impact that theintroduction of this policy may have on the stability and compatibility ofmaintenance releases, while still addressing some key security concernsrelating to the particular aspects of Python 2.7. It would be thoroughlycounterproductive if end users became as cautious about updating to newPython 2.7 maintenance releases as they are about updating to new featurereleases within the same release series.
Thessl module changes are included in this proposal to bring thePython 2 series up to date with the past 4 years of evolution in networksecurity standards, and make it easier for those standards to be broadlyadopted in both servers and clients. Similarly the hash algorithmavailability indicators inhashlib are included to make it easier forapplications to detect and employ appropriate hash definitions across bothPython 2 and 3.
Thehmac.compare_digest() andhashlib.pbkdf2_hmac() are included tohelp lower the barriers to secure password storage and checking in Python 2server applications.
Theos.urandom() change has been included in this proposal to furtherencourage users to leave the task of providing high quality random numbersfor cryptographic use cases to operating system vendors. The use ofinsufficiently random numbers has the potential to compromiseanycryptographic system, and operating system developers have more toolsavailable to address that problem adequately than the typical Pythonapplication runtime.
This alternative represents the status quo. Unfortunately, it has provento be unworkable in practice, as the backwards compatibility implicationsmean that this is a non-trivial migration process for large applicationsand integration projects. While the tools for migration have evolved toa point where it is possible to migrate even large applicationsopportunistically and incrementally (rather than all at once) by updatingcode to run in the large common subset of Python 2 and Python 3, using themost recent technology often isn’t a priority in commercial environments.
Previously, this was considered an acceptable harm, as while it was anunfortunate problem for the affected developers to have to face, it wasseen as an issue between them and their management chain to make the casefor infrastructure modernisation, and this case would become naturallymore compelling as the Python 3 series evolved.
However, now that we’re fully aware of the impact the limitations of thePython 2 standard library may be having on the evolution of internetsecurity standards, I no longer believe that it is reasonable to expectplatform and application developers to resolve all of the latent defectsin an application’s Unicode correctness solely in order to gain access tothe network security enhancements already available in Python 3.
While Ubuntu (and to some extent Debian as well) are committed to porting alldefault system services and scripts to Python 3, and to removing Python 2from its default distribution images (but not from its archives), this isa mammoth task and won’t be completed for the Ubuntu 14.04 LTS release(at least for the desktop image - it may be achieved for the mobile andserver images).
Fedora has even more work to do to migrate, and it will take a non-trivialamount of time to migrate the relevant infrastructure components. WhileRed Hat are also actively working to make it easier for users to use morerecent versions of Python on our stable platforms, it’s going to take timefor those efforts to start having an impact on end users’ choice of version,and any such changes also don’t benefit the core platform infrastructurethat runs in the integrated system Python by necessity.
The OpenStack migration to Python 3 is also still in its infancy, and eventhough that’s a project with an extensive and relatively robust automatedtest suite, it’s still large enough that it is going to take quite some timeto migrate fully to a Python 2/3 compatible code base.
And that’s just three of the highest profile open source projects thatmake heavy use of Python. Given the likely existence of large amounts oflegacy code that lacks the kind of automated regression test suite neededto help support a migration from Python 2 to Python 3, there are likely tobe many cases where reimplementation (perhaps even in Python 3) proveseasier than migration. The key point of this PEP is that those situationsaffect more people than just the developers and users of the affectedapplication: the existence of clients and servers with outdated networksecurity infrastructure becomes something that developers of securenetworked services need to take into account as part of their securitydesign, and that’s a problem that inhibits the adoption of better securitystandards.
As Terry Reedy noted, if we try to persist with the status quo, the likelyoutcome is that commercial redistributors will attempt to do somethinglike this on behalf of their customersanyway, but in a potentiallyinconsistent and ad hoc manner. By drawing the scope definition processinto the upstream project we are in a better position to influence theapproach taken to address the situation and to help ensure some consistencyacross redistributors.
The problem is real, sosomething needs to change, and this PEP describesmy preferred approach to addressing the situation.
With sufficient corporate support, it likelywould be possible to createand release Python 2.8 (it’s highly unlikely such a project would garnerenough interest to be achievable with only volunteers). However, thiswouldn’t actually solve the problem, as the aim is to provide arelativelylow impact way to incorporate enhanced security features into integratedproducts and deployments that make use of Python 2.
Upgrading to a new Python feature release would mean both more work for thecore development team, as well as a more disruptive update that mostpotential end users would likely just skip entirely.
Attempting to create a Python 2.8 release would also bring in suggestionsto backport many additional features from Python 3 (such astracemallocand the improved coroutine support), making the migration from Python 2.7to this hypothetical 2.8 release even riskier and more disruptive.
This is not a recommended approach, as it would involve substantialadditional work for a result that is actually less effective in achievingthe original aim (which is to eliminate the current widespread use of theaging network security infrastructure in the Python 2 series).
Furthermore, while I can’t make any commitments to actually addressingthis issue on Red Hat platforms, Ican categorically rule out the ideaof a Python 2.8 being of any use to me in even attempting to get itaddressed.
While this initially appears to be an attractive and easier to manageapproach, it actually suffers from several significant problems.
Firstly, this is complex, low level, cross-platform code that integrateswith the underlying operating system across a variety of POSIX platforms(including Mac OS X) and Windows. The CPython BuildBot fleet is already setup to handle continuous integration in that context, but most of thefreely available continuous integration services just offer Linux, andperhaps paid access to Windows. Those services work reasonably well forsoftware that largely runs on the abstraction layers offered by Python andother dynamic languages, as well as the more comprehensive abstractionoffered by the JVM, but won’t suffice for the kind of code involved here.
The OpenSSL dependency for the network security support also qualifies asthe kind of “complex binary dependency” that isn’t yet handled well by thepip based software distribution ecosystem. Relying on a third partybinary dependency also creates potential compatibility problems forpipwhen running on other interpreters likePyPy.
Another practical problem with the idea is the fact thatpip itselfrelies on thessl support in the standard library (with some additionalsupport from a bundled copy ofrequests, which in turn bundlesbackport.ssl_match_hostname), and hence would require any replacementmodule to also be bundled withinpip. This wouldn’t pose anyinsurmountable difficulties (it’s just another dependency to vendor), butitwould mean yet another copy of OpenSSL to keep up to date.
This approach also has the same flaw as all other “improve security byrenaming things” approaches: they completely miss the users who most needhelp, and raise significant barriers against being able to encourage usersto do the right thing when their infrastructure supports it (since“use this other module” is a much higher impact change than “turn on thishigher security setting”). Deprecating the aging SSL infrastructure in thestandard library in favour of an external module would be even more userhostile than accepting the slightly increased risk of regressions associatedwith upgrading it in place.
Last, but certainly not least, this approach suffers from the same problemas the idea of doing a Python 2.8 release: likely not solving the actualproblem. Commercial redistributors of Python are set up to redistributePython, and a pre-existing set of additional packages. Getting newpackages added to the pre-existing setcan be done, but means approachingeach and every redistributor and asking them to update theirrepackaging process accordingly. By contrast, the approach described inthis PEP would require redistributors to deliberatelyopt out of thesecurity enhancements by deliberately downgrading the provided networksecurity infrastructure, which most of them are unlikely to do.
Earlier versions of this PEP included the concept of a2.7-legacy-sslbranch that preserved the exact feature set of the Python 2.7.6 networksecurity infrastructure.
In my opinion, anyone that actually wants this is almost certainly making amistake, and if they insist they really do want it in their specificsituation, they’re welcome to either make it themselves or arrange for adownstream redistributor to make it for them.
If they are made publicly available, any such rebuilds should be referred toas “Python 2.7 with Legacy SSL” to clearly distinguish them from the officialPython 2.7 releases that include more up to date network securityinfrastructure.
After the first Python 2.7 maintenance release that implements this PEP, itwould also be appropriate to refer to Python 2.7.6 and earlier releases as“Python 2.7 with Legacy SSL”.
Earlier versions of this PEP suggested synchronising thehmac,hashlib andssl modules entirely with their Python 3 counterparts.
This approach proved too vague to build a compelling case for the exception,and has thus been replaced by the current more explicit proposal.
Earlier versions of this PEP suggested a general policy change related tofuture Python 3 enhancements that impact the general security of theinternet.
That approach created unnecessary uncertainty, so it has been simplified topropose backport a specific concrete set of changes. Future featurebackport proposals can refer back to this PEP as precedent, but it willstill be necessary to make a specific case for each feature addition tothe Python 2.7 long-term support release.
The author of this PEP currently works for Red Hat on test automation tools.If this proposal is accepted, I will be strongly encouraging Red Hat to takeadvantage of the resulting opportunity to help improve the overall securityof the Python ecosystem. However, I do not speak for Red Hat in this matter,and cannot make any commitments on Red Hat’s behalf.
Thanks to Christian Heimes and other for their efforts in greatly improvingPython’s SSL support in the Python 3 series, and a variety of members ofthe Python community for helping me to better understand the implicationsof the default settings we provide in our SSL modules, and the impact thattolerating the use of SSL infrastructure that was defined in 2010(Python 2.7) or even 2008 (Python 2.6) potentially has for the securityof the web as a whole.
Thanks to Donald Stufft and Alex Gaynor for identifying a more limited setof essential security features that allowed the proposal to be made morefine-grained than backporting entire modules from Python 3.4 ([7],[8]).
Christian and Donald also provided valuable feedback on a preliminarydraft of this proposal.
Thanks also to participants in the python-dev mailing list threads([1],[2],[5],[6]), as well as the various folks I discussed this issue with atPyCon 2014 in Montreal.
This document has been placed in the public domain.
Source:https://github.com/python/peps/blob/main/peps/pep-0466.rst
Last modified:2025-02-01 08:59:27 GMT