Abstract

The Python standard library under CPython contains various instancesof modules implemented in both pure Python and C (either entirely orpartially). This PEP requires that in these instances that theC codemust pass the test suite used for the pure Python codeso as to act as much as a drop-in replacement as reasonably possible(C- and VM-specific tests are exempt). It is also required that newC-based modules lacking a pure Python equivalent implementation getspecial permission to be added to the standard library.

Rationale

Python has grown beyond the CPython virtual machine (VM).IronPython,Jython, andPyPy are all currently viable alternatives to theCPython VM. The VM ecosystem that has sprung up around the Pythonprogramming language has led to Python being used in many differentareas where CPython cannot be used, e.g., Jython allowing Python to beused in Java applications.

A problem all of the VMs other than CPython face is handling modulesfrom the standard library that are implemented (to some extent) in C.Since other VMs do not typically support the entireC API of CPythonthey are unable to use the code used to create the module. Oftentimesthis leads these other VMs to either re-implement the modules in purePython or in the programming language used to implement the VM itself(e.g., in C# for IronPython). This duplication of effort betweenCPython, PyPy, Jython, and IronPython is extremely unfortunate asimplementing a moduleat least in pure Python would help mitigatethis duplicate effort.

The purpose of this PEP is to minimize this duplicate effort bymandating that all new modules added to Python’s standard librarymust have a pure Python implementationunless special dispensationis given. This makes sure that a module in the stdlib is available toall VMs and not just to CPython (pre-existing modules that do not meetthis requirement are exempt, although there is nothing preventingsomeone from adding in a pure Python implementation retroactively).

Re-implementing parts (or all) of a module in C (in the caseof CPython) is still allowed for performance reasons, but any suchaccelerated code must pass the same test suite (sans VM- or C-specifictests) to verify semantics and prevent divergence. To accomplish this,the test suite for the module must have comprehensive coverage of thepure Python implementation before the acceleration code may be added.

Details

Starting in Python 3.3, any modules added to the standard library musthave a pure Python implementation. This rule can only be ignored ifthe Python development team grants a special exemption for the module.Typically the exemption will be granted only when a module wraps aspecific C-based library (e.g.,sqlite3). In granting an exemption itwill be recognized that the module will be considered exclusive toCPython and not part of Python’s standard library that other VMs areexpected to support. Usage ofctypes to provide anAPI for a C library will continue to be frowned upon asctypeslacks compiler guarantees that C code typically relies upon to preventcertain errors from occurring (e.g., API changes).

Even though a pure Python implementation is mandated by this PEP, itdoes not preclude the use of a companion acceleration module. If anacceleration module is provided it is to be named the same as themodule it is accelerating with an underscore attached as a prefix,e.g.,_warnings forwarnings. The common pattern to accessthe accelerated code from the pure Python implementation is to importit with animport*, e.g.,from_warningsimport*. This istypically done at the end of the module to allow it to overwritespecific Python objects with their accelerated equivalents. This kindof import can also be done before the end of the module when needed,e.g., an accelerated base class is provided but is then subclassed byPython code. This PEP does not mandate that pre-existing modules inthe stdlib that lack a pure Python equivalent gain such a module. Butif people do volunteer to provide and maintain a pure Pythonequivalent (e.g., the PyPy team volunteering their pure Pythonimplementation of thecsv module and maintaining it) then suchcode will be accepted. In those instances the C version is consideredthe reference implementation in terms of expected semantics.

Any new accelerated code must act as a drop-in replacement as closeto the pure Python implementation as reasonable. Technical details ofthe VM providing the accelerated code are allowed to differ asnecessary, e.g., a class being atype when implemented in C. Toverify that the Python and equivalent C code operate as similarly aspossible, both code bases must be tested using the same tests whichapply to the pure Python code (tests specific to the C code or any VMdo not follow under this requirement). The test suite is expected tobe extensive in order to verify expected semantics.

Acting as a drop-in replacement also dictates that no public API beprovided in accelerated code that does not exist in the pure Pythoncode. Without this requirement people could accidentally come to relyon a detail in the accelerated code which is not made available toother VMs that use the pure Python implementation. To help verifythat the contract of semantic equivalence is being met, a module mustbe tested both with and without its accelerated code as thoroughly aspossible.

As an example, to write tests which exercise both the pure Python andC accelerated versions of a module, a basic idiom can be followed:

fromtest.supportimportimport_fresh_moduleimportunittestc_heapq=import_fresh_module('heapq',fresh=['_heapq'])py_heapq=import_fresh_module('heapq',blocked=['_heapq'])classExampleTest:deftest_example(self):self.assertTrue(hasattr(self.module,'heapify'))classPyExampleTest(ExampleTest,unittest.TestCase):module=py_heapq@unittest.skipUnless(c_heapq,'requires the C _heapq module')classCExampleTest(ExampleTest,unittest.TestCase):module=c_heapqif__name__=='__main__':unittest.main()

The test module defines a base class (ExampleTest) with test methodsthat access theheapq module through aself.heapq class attribute,and two subclasses that set this attribute to either the Python or the Cversion of the module. Note that only the two subclasses inherit fromunittest.TestCase – this prevents theExampleTest class frombeing detected as aTestCase subclass byunittest test discovery.AskipUnless decorator can be added to the class that tests the C codein order to have these tests skipped when the C module is not available.

If this test were to provide extensive coverage forheapq.heappop() in the pure Python implementation then theaccelerated C code would be allowed to be added to CPython’s standardlibrary. If it did not, then the test suite would need to be updateduntil proper coverage was provided before the accelerated C codecould be added.

To also help with compatibility, C code should use abstract APIs onobjects to prevent accidental dependence on specific types. Forinstance, if a function accepts a sequence then the C code shoulddefault to usingPyObject_GetItem() instead of something likePyList_GetItem(). C code is allowed to have a fast path if theproperPyList_CheckExact() is used, but otherwise APIs should workwith any object that duck types to the proper interface instead of aspecific type.

Copyright

This document has been placed in the public domain.