Unstable C API tier

The Python C-API is currently divided intothree stability tiers:

• Limited API, with high compatibility expectations
• Public API, which follows thebackwards compatibility policy, and requires deprecation warnings before changes
• Internal (private) API, which can change at any time.

Tools requiring access to CPython internals (e.g. advanceddebuggers and JIT compilers) are often built for minor series releasesof CPython, and assume that the C-API internals used do not changein patch releases. To support these tools, we need a tier between thePublic and Private C-API, with guarantees on stability throughoutthe minor-series release: the proposedUnstable tier.

Some functions, likePyCode_New(), are documented as unstable(“Calling [it] directly can bind you to a precise Python version”),and also often change in practice.The unstable tier should make their status obvious even to people who don’tread the docs carefully enough, making them hard to use accidentally.

Reserving leading underscores for Private API

Currently, CPython developers don’t agree on the exact meaning of a leadingunderscore in API names.It is used to mean two different things:

• API that may change between minor releases, as in the Unstable tier proposedhere (e.g. functions introduced inPEP 523).
• API that isprivate and should not be used outside of CPython at all(e.g. because it may change without notice, or it relies on undocumentedassumptions that non-CPython code cannot guarantee).

The unclear meaning makes the underscore less useful than it could be.If it only markedprivate API, CPython developers could change underscoredfunctions, or remove unused ones, without researching how they’redocumented or used outside CPython.

With the introduction of a dedicated unstable tier, we can clarify the meaningof the leading underscore. It should mark private API only.

Not breaking code unnecessarily

This PEP specifies that API in the unstable tier should have a special nameprefix. This means functions (macros, etc.) will need to be renamed.After a rename, the old name should continue to be available untilan incompatible change is made (i.e. until call sites need to be updatedanyway).In other words, just changing the tier of a function shouldn’t break users’code.

Leading underscore

C API named with a leading underscore, as well as API only available withPy_BUILD_CORE, will be consideredinternal.This means:

• It may change or be removedwithout notice in minorreleases (3.x.0, including Alpha and Beta releases of 3.x.0).API changes in patch releases or Release Candidates should only be done ifabsolutely necessary.
• It should be documented in source comments or Devguide only, not in thepublic documentation.
• API introduced before Python 3.12 that is documented or widely usedexternally should be moved to the Unstable tier as explained above.

  This might happen long after this PEP is accepted.Consequently, for a few years core devs should do some research beforechanging underscored API, especially if it doesn’t needPy_BUILD_CORE.

Users of the C API are encouraged to search their codebase for_Py and_PY identifier prefixes, and treat any hits as issues to be eventuallyfixed – either by switching to an existing alternative, or by openinga CPython issue to request exposing public API for their use case,and eventually switching to that.

Initial unstable API

The following API will be moved to the Unstable tier in the initialimplementation as proof of the concept.

Code object constructors:

• PyUnstable_Code_New() (renamed fromPyCode_New)
• PyUnstable_Code_NewWithPosOnlyArgs() (renamed fromPyCode_NewWithPosOnlyArgs)

Code extra information (PEP 523):

• PyUnstable_Eval_RequestCodeExtraIndex() (renamed from_PyEval_RequestCodeExtraIndex)
• PyUnstable_Code_GetExtra() (renamed from_PyCode_GetExtra)
• PyUnstable_Code_SetExtra() (renamed from_PyCode_SetExtra)

More are expected in Python 3.12, without the need for another PEP.

No special prefix

In the initial version of this PEP, unstable API didn’t have thePyUnstableprefix.Instead, definingPy_USING_UNSTABLE_API made the API available in a givensource file, signifying acknowledgement that the file as a whole willpotentially need to be revisited for each Python release.

However, it was decided that unstable-ness needs to be exposedin the individual names.

Underscore prefix

It would be possible to mark both private and unstable API withleading underscores.However, that would dilute the meaning of_Py prefix.Reserving the prefix for internal API only makes it trivial to search for.

New header directory

Other API tiers have dedicated directories for headers(Include/cpython/,Include/internal/).

Since the unstable tier uses a very obvious naming conventionand the names are always available,a directory likeInclude/unstable/ is unnecessary.

Python API

It might be good to add a similar tier in the Python (not C) API,e.g. fortypes.CodeType.However, the mechanism for that would need to be different.This is outside the scope of the PEP.