Python Enhancement Proposals

Python »
PEP Index »
PEP 743

PEP 743 – Add Py_OMIT_LEGACY_API to the Python C API

Author:: Victor Stinner <vstinner at python.org>,Petr Viktorin <encukou at gmail.com>
Discussions-To:

Table of Contents

Rejection Notice

The Steering Council has rejected this PEP in its current form:

While we agree that the problem the PEP tries to solve is one worth solving,we’re not convinced this is the correct approach.

Seethe full post.

Abstract

AddPy_OMIT_LEGACY_API C macro that hides deprecated andsoft-deprecated symbols, allowing users to opt out of using API with knownissues that other API solves.

Also, add namespaced alternatives for API without thePy_ prefix,and soft-deprecate the original names.

Motivation

Some of Python’s C API has flaws that are only obvious in hindsight.

If an API prevents adding features or optimizations, or presents a serioussecurity risk or maintenance burden, we can deprecate and remove it asdescribed inPEP 387.

However, this leaves us with some API that has “sharp edges” – it works finefor its current users, but should be avoided in new code.For example:

API that cannot signal an exception, so failures are either ignored orexit the process with a fatal error. For examplePyObject_HasAttr.
API that is not thread-safe, for example by borrowing references frommutable objects, or exposing unfinished mutable objects. For examplePyDict_GetItemWithError.
API with names that don’t use thePy/_Py prefix, and so can clashwith other code. For example:setter.

It is important to note that despite such flaws, it’s usually possibleto use the API correctly. For example, in a single-threaded environment,thread safety is not an issue.We do not want to break working code, even if it uses API that would be wrongin some – or evenmost – other contexts.

On the other hand, we want to steer users away from such “undesirable” APIinnew code, especially if a safer alternative exists.

Adding the`Py` prefix

Some names defined in CPython headers is not namespaced: it that lacks thePy prefix (or a variant:_Py, and alternative capitalizations).For example, we declare a function type named simplysetter.

While such names are not exported in the ABI (as checked bymakesmelly),they can clash with user code and, more importantly, with libraries linkedto third-party extensions.

While it would be possible to provide namespaced aliases and (soft-)deprecatethese names, the only way to make them not clash with third-party code is tonot define them in Python headers at all.

Rationale

We want to allow an easy way for users to avoid “undesirable” API if theychoose to do so.

It might be be sufficient to leave this to third-party linters.For that we’d need a good way to expose a list of (soft-)deprecatedAPI to such linters.While adding that, we can – rather easily – do the linter’s job directlyin CPython headers, avoiding the need for an extra tool.Unlike Python, C makes it rather easy to limit available API – for a wholeproject or for each individual source file – by having users definean “opt-in” macro.

We already do something similar withPy_LIMITED_API, which limits theavailable API to a subset that compiles to stable ABI. (In hindsight, we shouldhave used a different macro name for that particular kind of limiting, but it’stoo late to change that now.)

To be clear, this mechanism isnot a replacement for deprecation.Deprecation is for API that prevents new features or optimizations, orpresents a security risk or maintenance burden.This mechanism, on the other hand, is meant for cases where “we founda slightly better way of doing things” – perhaps one that’s harder to misuse,or just has a less misleading name.(On a lighter note: many people configure a code quality checker to shout atthem about the number of blank lines between functions. Let’s help themidentify more substantial “code smells”!)

The proposed macro does notchange any API definitions; it onlyhides them.So, if code compiles with the macro, it’ll also compile without it, withidentical behaviour.This has implications for core devs: to deal with undesirable behaviour,we’ll need to introduce new, better API, andthen discourage the old one.In turn, this implies that we should look at an individual API and fix all itsknown issues at once, rather than do codebase-wide sweeps for a single kind ofissue, so that we avoid multiple renames of the same function.

Specification

We introduce aPy_OMIT_LEGACY_API macro.If this macro is defined before#include<Python.h>, some API definitions– as described below – will be omitted from the Python header files.

The macro only omits complete top-level definitions exposed from<Python.h>.Other things (the ABI, structure definitions, macro expansions, static inlinefunction bodies, etc.) are not affected.

The C API working group (PEP 731) has authority over the set of omitteddefinitions.

The set of omitted definitions will be tied to a particular feature releaseof CPython, and is finalized in each 3.x.0 Beta 1 release.In rare cases, entries can be removed (i.e. made available for use) at anytime.

Requirements for omitted API

An API that is omitted withPy_OMIT_LEGACY_API must:

be soft-deprecated (seePEP 387);
for all known use cases of the API, have a documented alternativeor workaround;
have tests to ensure it keeps working (except for 1:1 renames using#define ortypedef);
be documented (except if it was never mentioned in previous versions of thedocumentation); and
be approved by the C API working group. (The WG may give blanket approvalsfor groups of related API; seeInitial set below for examples.)

Note thatPy_OMIT_LEGACY_API is meant for API that can be triviallyreplaced by a better alternative.API without a replacement should generally be deprecated instead.

Location

All API definitions omitted byPy_OMIT_LEGACY_API will be moved toa new header,Include/legacy.h.

This is meant to help linter authors compile lists, so they can flag the APIwith warnings rather than errors.

Note that for simple renaming of source-only constructs (macros, types), weexpect names to be omitted in the same version – or the same PR – that addsa replacement.This means that the original definition will be renamed, and atypedefor#define for the old name added toInclude/legacy.h.

Documentation

Documentation for omitted API should generally:

appear after the recommended replacement,
reference the replacement (e.g. “Similar to X, but…”), and
focus on differences from the replacement and migration advice.

Exceptions are possible if there is a good reason for them.

Initial set

The following API will be omitted withPy_OMIT_LEGACY_API set:

Omit API returning borrowed references:
Omitted API Replacement
PyDict_GetItem() PyDict_GetItemRef()
PyDict_GetItemString() PyDict_GetItemStringRef()
PyImport_AddModule() PyImport_AddModuleRef()
PyList_GetItem() PyList_GetItemRef()

Omitted API	Replacement
`PyDict_GetItem()`	`PyDict_GetItemRef()`
`PyDict_GetItemString()`	`PyDict_GetItemStringRef()`
`PyImport_AddModule()`	`PyImport_AddModuleRef()`
`PyList_GetItem()`	`PyList_GetItemRef()`

Omit deprecated APIs:

Omitted Deprecated API	Replacement
`PY_FORMAT_SIZE_T`	`"z"`
`PY_UNICODE_TYPE`	`wchar_t`
`PyCode_GetFirstFree()`	`PyUnstable_Code_GetFirstFree()`
`PyCode_New()`	`PyUnstable_Code_New()`
`PyCode_NewWithPosOnlyArgs()`	`PyUnstable_Code_NewWithPosOnlyArgs()`
`PyImport_ImportModuleNoBlock()`	`PyImport_ImportModule()`
`PyMem_DEL()`	`PyMem_Free()`
`PyMem_Del()`	`PyMem_Free()`
`PyMem_FREE()`	`PyMem_Free()`
`PyMem_MALLOC()`	`PyMem_Malloc()`
`PyMem_NEW()`	`PyMem_New()`
`PyMem_REALLOC()`	`PyMem_Realloc()`
`PyMem_RESIZE()`	`PyMem_Resize()`
`PyModule_GetFilename()`	`PyModule_GetFilenameObject()`
`PyOS_AfterFork()`	`PyOS_AfterFork_Child()`
`PyObject_DEL()`	`PyObject_Free()`
`PyObject_Del()`	`PyObject_Free()`
`PyObject_FREE()`	`PyObject_Free()`
`PyObject_MALLOC()`	`PyObject_Malloc()`
`PyObject_REALLOC()`	`PyObject_Realloc()`
`PySlice_GetIndicesEx()`	(two calls; see current docs)
`PyThread_ReInitTLS()`	(no longer needed)
`PyThread_create_key()`	`PyThread_tss_alloc()`
`PyThread_delete_key()`	`PyThread_tss_free()`
`PyThread_delete_key_value()`	`PyThread_tss_delete()`
`PyThread_get_key_value()`	`PyThread_tss_get()`
`PyThread_set_key_value()`	`PyThread_tss_set()`
`PyUnicode_AsDecodedObject()`	`PyUnicode_Decode()`
`PyUnicode_AsDecodedUnicode()`	`PyUnicode_Decode()`
`PyUnicode_AsEncodedObject()`	`PyUnicode_AsEncodedString()`
`PyUnicode_AsEncodedUnicode()`	`PyUnicode_AsEncodedString()`
`PyUnicode_IS_READY()`	(no longer needed)
`PyUnicode_READY()`	(no longer needed)
`PyWeakref_GET_OBJECT()`	`PyWeakref_GetRef()`
`PyWeakref_GetObject()`	`PyWeakref_GetRef()`
`Py_UNICODE`	`wchar_t`
`_PyCode_GetExtra()`	`PyUnstable_Code_GetExtra()`
`_PyCode_SetExtra()`	`PyUnstable_Code_SetExtra()`
`_PyDict_GetItemStringWithError()`	`PyDict_GetItemStringRef()`
`_PyEval_RequestCodeExtraIndex()`	`PyUnstable_Eval_RequestCodeExtraIndex()`
`_PyHASH_BITS`	`PyHASH_BITS`
`_PyHASH_IMAG`	`PyHASH_IMAG`
`_PyHASH_INF`	`PyHASH_INF`
`_PyHASH_MODULUS`	`PyHASH_MODULUS`
`_PyHASH_MULTIPLIER`	`PyHASH_MULTIPLIER`
`_PyObject_EXTRA_INIT`	(no longer needed)
`_PyThreadState_UncheckedGet()`	`PyThreadState_GetUnchecked()`
`_PyUnicode_AsString()`	`PyUnicode_AsUTF8()`
`_Py_HashPointer()`	`Py_HashPointer()`
`_Py_T_OBJECT`	(`tp_getset`; docs to be written)
`_Py_WRITE_RESTRICTED`	(no longer needed)

Soft-deprecate and omit APIs:

Omitted Deprecated API	Replacement
`PyDict_GetItemWithError()`	`PyDict_GetItemRef()`
`PyDict_SetDefault()`	`PyDict_SetDefaultRef()`
`PyMapping_HasKey()`	`PyMapping_HasKeyWithError()`
`PyMapping_HasKeyString()`	`PyMapping_HasKeyStringWithError()`
`PyObject_HasAttr()`	`PyObject_HasAttrWithError()`
`PyObject_HasAttrString()`	`PyObject_HasAttrStringWithError()`

Omit<structmember.h> legacy API:

The header filestructmember.h, which is not included from<Python.h>and must be included separately, will#error ifPy_OMIT_LEGACY_API is defined.This affects the following API:

Omitted Deprecated API	Replacement
`T_SHORT`	`Py_T_SHORT`
`T_INT`	`Py_T_INT`
`T_LONG`	`Py_T_LONG`
`T_FLOAT`	`Py_T_FLOAT`
`T_DOUBLE`	`Py_T_DOUBLE`
`T_STRING`	`Py_T_STRING`
`T_OBJECT`	(`tp_getset`; docs to be written)
`T_CHAR`	`Py_T_CHAR`
`T_BYTE`	`Py_T_BYTE`
`T_UBYTE`	`Py_T_UBYTE`
`T_USHORT`	`Py_T_USHORT`
`T_UINT`	`Py_T_UINT`
`T_ULONG`	`Py_T_ULONG`
`T_STRING_INPLACE`	`Py_T_STRING_INPLACE`
`T_BOOL`	`Py_T_BOOL`
`T_OBJECT_EX`	`Py_T_OBJECT_EX`
`T_LONGLONG`	`Py_T_LONGLONG`
`T_ULONGLONG`	`Py_T_ULONGLONG`
`T_PYSSIZET`	`Py_T_PYSSIZET`
`T_NONE`	(`tp_getset`; docs to be written)
`READONLY`	`Py_READONLY`
`PY_AUDIT_READ`	`Py_AUDIT_READ`
`READ_RESTRICTED`	`Py_AUDIT_READ`
`PY_WRITE_RESTRICTED`	(no longer needed)
`RESTRICTED`	`Py_AUDIT_READ`

Omit soft deprecated macros:
Omitted Macros Replacement
Py_IS_NAN() isnan() (C99+<math.h>)
Py_IS_INFINITY() isinf(X) (C99+<math.h>)
Py_IS_FINITE() isfinite(X) (C99+<math.h>)
Py_MEMCPY() memcpy() (C<string.h>)
Soft-deprecate and omit typedefs without thePy/_Py prefix(getter,setter,allocfunc, …), in favour ofnew onesthat add the prefix (Py_getter , etc.)
Soft-deprecate and omit macros without thePy/_Py prefix(METH_O,CO_COROUTINE,FUTURE_ANNOTATIONS,WAIT_LOCK, …),favour ofnew ones that add the prefix (Py_METH_O , etc.).
Any others approved by the C API workgroup

Omitted Macros	Replacement
`Py_IS_NAN()`	`isnan()` (C99+`<math.h>`)
`Py_IS_INFINITY()`	`isinf(X)` (C99+`<math.h>`)
`Py_IS_FINITE()`	`isfinite(X)` (C99+`<math.h>`)
`Py_MEMCPY()`	`memcpy()` (C`<string.h>`)

If any of these proposed replacements, or associated documentation,are not added in time for 3.14.0b1, they’ll be omitted with later versionsofPy_OMIT_LEGACY_API.(We expect this for macros generated byconfigure:HAVE_*,WITH_*,ALIGNOF_*,SIZEOF_*, and several without a common prefix.)