Using slots

The main alternative to slots is using a versionedstruct for input.

There are two variants of such a design:

• A large struct with fields for all info. As we can see withPyTypeObject, most of such a struct tends to be NULLs in practice.As more fields become obsolete, either the wastage grows, or we introduce newstruct layouts (while keeping compatibility with the old ones for a while).
• A small struct with only the info necessary for initial creation, with otherinfo added afterwards (with dedicated function calls, or Python-levelsetattr). This design:
  • makes it cumbersome to add/obsolete/adjust the required info (for example,inPEP 697 I gave meaning to negative values of an existing field;adding a new field would be cleaner in similar situations);
  • increases the number of API calls between an extension and the interpreter.

  We believe that “batch” API for type/module creation makes sense,even if it partially duplicates an API to modify “live” objects.

Using slotsonly

The classesPyType_Spec andPyModuleDef have explicit fieldsin addition to a slots array. These include:

• Required information, such as the class name (PyType_Spec.name).This proposal adds aslot ID for the name, and makes it required.
• Non-pointers (basicsize,flags).Originally, slots were intended toonly containfunction pointers; they now containdata pointers as well asintegers or flags. This proposal uses a union to handle types cleanly.
• Items added before the slots mechanism. ThePyModuleDef.m_slotsitself was repurposed fromm_reload which was always NULL;the optionalm_traverse orm_methods members predate it.

We can do without these fields, and haveonly an array of slots.A wrapper class around the array would complicate the design.If fields in such a class ever become obsolete, they are hard to remove orrepurpose.

Nested slot tables

In this proposal, the array of slots can reference another array of slots,which is treated as if it was merged into its “parent”, recursively.This complicates slot handling inside the interpreter, but allows:

• Mixing dynamically allocated (or stack-allocated) slots withstatic ones.This solves the issue that lead to thePyType_From* family offunctions expanding with values that typically can’t bestatic.For example, themodule argument toPyType_FromModuleAndSpec()should be a heap-allocated module object.
• Sharing a subset of the slots to implement functionalitycommon to several classes/modules.
• Easily including some slots conditionally, e.g. based on the Python version.

Nested “legacy” slot tables

Similarly to nested arrays ofPyType_Slot, we also propose supportingarrays of “legacy” slots (PyType_Slot andPyModuleDef_Slot) inthe “new” slots, and vice versa.

This way, users can reuse code they already have written withoutrewriting/reformatting,and only use the “new” slots if they need any new features.

Fixed-width integers

This proposal uses fixed-width integers (uint16_t) for slot IDs andflags.With the Cint type, using more than 16 bits would not be portable,but it would silently work on common platforms.Usingint but avoiding values overUINT16_MAX wastes 16 bitson common platforms.

Memory layout

On common 64-bit platforms, we can keep the size of the new struct the sameas the existingPyType_Slot andPyModuleDef_Slot. (The existingstruct waste 6 out of 16 bytes due toint portability and padding;this proposal puts some of those bits to use for new features.)On 32-bit platforms, this proposal calls for the same layout as on 64-bit,doubling the size compared to the existing structs (from 8 bytes to 16).For “configuration” data that’s usuallystatic, it should be OK.

The proposal does not use bit-fields and enums, whose memory representation iscompiler-dependent, causing issues when using the API from languages otherthan C.

The structure is laid out assuming that a type’s alignment matches its size.

Single ID space

Currently, the numeric values ofmodule andtype slots overlap:

• Py_bf_getbuffer ==Py_mod_create == 1
• Py_bf_releasebuffer ==Py_mod_exec == 2
• Py_mp_ass_subscript ==Py_mod_multiple_interpreters == 3
• Py_mp_length ==Py_mod_gil == 4
• and similar for module slots added in CPython 3.15

This proposal use a single sequence for both, so future slots avoid thisoverlap. This is to:

• Avoidaccidentally using type slots for modules, and vice versa
• Allow external libraries or checkers to determine a slot’s meaning(and type) based on the ID.

The 4 existing overlaps means we don’t reach these goals right now,but we can gradually migrate to new numeric IDs in a way that’s transparentto the user.

The main disadvantage is that any internal lookup tables will be either bigger(if we use separate ones for types & modules, so they’ll contain blanks),or harder to manage (if they’re merged).

Deprecation warnings

Multiple slots are documented to not allow NULL values, but CPython allowsNULL for backwards compatibility.Similarly, multiple slot IDs should not appear more than once in a singlearray, but CPython allows such duplicates.

This is a maintenance issue, as CPython should preserve its undocumented(and often untested) behaviour in these cases as the implementation is changed.

It also prevents API extensions.For example, instead of adding thePy_TPFLAGS_DISALLOW_INSTANTIATIONflag in 3.10, we could have allowed settning thePy_tp_new slot to NULL forthe same effect.

To allow changing the edge case behaviour in the (far) future,and to allow freedom for possible alternative implementations of the C API,we’ll start issuing runtime deprecation warnings in these cases.

New API

The following function will be added.It will create the corresponding Python type object from the givenarray of slots:

PyObject*PyType_FromSlots(constPySlot*slots);

With this function, thePy_tp_token slot may not be set toPy_TP_USE_SPEC (i.e.NULL).

Changed API

ThePyModule_FromSlotsAndSpec function (added in CPython 3.15 inPEP 793) will bechanged to take the new slot structure:

PyObject*PyModule_FromSlotsAndSpec(constPySlot*slots,PyObject*spec)

Theextension module export hookadded inPEP 793 (PyModExport_<name>) will bechanged toreturn the new slot structure.ThePyMODEXPORT_FUNC macro willbe updated accordingly.

General slot semantics

When slots are passed to a function that applies them, the function will notmodify the slot array, nor any data it points to (recursively).

After the function is done, the user is allowed to modify or deallocate thearray, and any data it points to (recursively), unless it’s explicitly markedas “static” (seePySlot_STATIC below).This means the interpreter typically needs to make a copy of all datain the struct, includingchar* text.

Flags

sl_flags may set the following bits. Unassigned bits must be set to zero.

• PySlot_OPTIONAL: If the slot ID is unknown, the interpreter shouldignore the slot entirely. (For example, ifnb_matrix_multiply was beingadded to CPython now, your type could use this.)
• PySlot_STATIC: All data the slot points to is statically allocatedand constant.Thus, the interpreter does not need to copy the information.This flag is implied for function pointers.

  The flag applies even to data the slot points to “indirectly”, except fornested slots – seeNested slot tables below – which can have theirownPySlot_STATIC flag.For example, if applied to aPy_tp_members slot that points to anarray ofPyMemberDef structures, then the entire array, as well as thename anddoc strings in its elements, must be static and constant.

• PySlot_HAS_FALLBACK: If the slot ID is unknown, the interpreter willignore the slot.If it’s known, the interpreter should ignore subsequent slots up to(and including) the first one without HAS_FALLBACK.

  Effectively, consecutive slots with the HAS_FALLBACK flag, plus the firstnon-HAS_FALLBACK slot after them, form a “block” where the the interpreterwill only consider thefirst slot in the block that it understands.If the entire block is to be optional, it should end with aslot with the OPTIONAL flag.

• PySlot_INTPTR: The data is stored insl_ptr, and must be cast tothe appropriate type.

  This flag simplifies porting from the existingPyType_Slot andPyModuleDef_Slot, where all slots work this way.

Convenience macros

The following macros will be added to the API to simplify slot definition:

#define PySlot_DATA(NAME, VALUE) \   {.sl_id=NAME, .sl_ptr=(void*)(VALUE)}#define PySlot_FUNC(NAME, VALUE) \   {.sl_id=NAME, .sl_func=(VALUE)}#define PySlot_SIZE(NAME, VALUE) \   {.sl_id=NAME, .sl_size=(VALUE)}#define PySlot_INT64(NAME, VALUE) \   {.sl_id=NAME, .sl_int64=(VALUE)}#define PySlot_UINT64(NAME, VALUE) \   {.sl_id=NAME, .sl_uint64=(VALUE)}#define PySlot_STATIC_DATA(NAME, VALUE) \   {.sl_id=NAME, .sl_flags=PySlot_STATIC, .sl_ptr=(VALUE)}#define PySlot_END {0}

We’ll also add two more macros that avoid named initializers,for use in C++11-compatibile code.Note that these cast the value tovoid*, so they do not improve type safetyover existing slots:

#define PySlot_PTR(NAME, VALUE) \   {NAME, PySlot_INTPTR, {0}, {(void*)(VALUE)}}#define PySlot_PTR_STATIC(NAME, VALUE) \   {NAME, PySlot_INTPTR|Py_SLOT_STATIC, {0}, {(void*)(VALUE)}}

Nested slot tables

A new slot,Py_slot_subslots, will be added to allow nesting slot tables.Its value (sl_ptr) should point to an array ofPySlot structures,which will be treated as if they were part of the current slot array.sl_ptr can beNULL to indicate that there are no slots.

Two more slots will allow similar nesting for existing slot structures:

• Py_tp_slots for an array ofPyType_Slot
• Py_mod_slots for an array ofPyModuleDef_Slot

EachPyType_Slot in the array will be converted to(PySlot){.sl_id=slot,.sl_flags=PySlot_INTPTR,.sl_ptr=func},and similar withPyModuleDef_Slot.

The initial implementation will have restrictions that may be liftedin the future:

• Py_slot_subslots,Py_tp_slots andPy_mod_slots cannot usePySlot_HAS_FALLBACK (the flag cannot be set on them nor a slot thatprecedes them).
• Nesting depth will be limited to 5 levels.

New slot IDs

The following new slot IDs, usable for both type and moduledefinitions, will be added:

• Py_slot_end (defined as0): Marks the end of a slots array.
  • ThePySlot_INTPTR andPySlot_STATIC flags are ignored.
  • ThePySlot_OPTIONAL andPySlot_HAS_FALLBACK flags are notallowed withPy_slot_end.
• Py_slot_subslots,Py_tp_slots,Py_mod_slots: seeNested slot tables above
• Py_slot_invalid (defined asUINT16_MAX, i.e.-1): treated as anunknown slot ID.

The following new slot IDs will be added to cover existingmembers ofPyModuleDef:

• Py_tp_name (mandatory for type creation)
• Py_tp_basicsize (of typePy_ssize_t)
• Py_tp_extra_basicsize (equivalent to settingPyType_Spec.basicsizeto-extra_basicsize)
• Py_tp_itemsize
• Py_tp_flags

The following new slot IDs will be added to coverarguments ofPyType_FromMetaclass:

• Py_tp_metaclass (used to setob_type after metaclass calculation)
• Py_tp_module

Note thatPy_tp_base andPy_tp_bases already exist.The interpreter will treat them identically: either can specify a classobject or a tuple of them.Py_tp_base will be soft-deprecated in favour ofPy_tp_bases.Specifying both in a single definition will be deprecated (currently,Py_tp_bases overridesPy_tp_base).

None of the new slots will be usable withPyType_GetSlot.(This limitation may be lifted in the future, with C API WG approval.)

Of the new slots, onlyPy_slot_end,Py_slot_subslots,Py_tp_slots,Py_mod_slots will be allowed inPyType_Spec and/orPyModuleDef.

Slot renumbering

New slots IDs will have unique numeric values (that is,Py_slot_*,Py_tp_* andPy_mod_* won’t share IDs).

Slots numbered 1 through 4 (Py_bf_getbuffer…Py_mp_length andPy_mod_create…Py_mod_gil) will be redefined as new(larger) numbers.The old numbers will remain as aliases, and will be used when compiling forStable ABI versions below 3.15.

Slots for members ofPyModuleDef, which were added inPEP 793, will be renumbered so that they haveunique IDs:

• Py_mod_name
• Py_mod_doc
• Py_mod_state_size
• Py_mod_methods
• Py_mod_state_traverse
• Py_mod_state_clear
• Py_mod_state_free

Soft deprecation

These existing functions will besoft-deprecated:

• PyType_FromSpec
• PyType_FromSpecWithBases
• PyType_FromModuleAndSpec
• PyType_FromMetaclass
• PyModule_FromDefAndSpec
• PyModule_FromDefAndSpec2
• PyModule_ExecDef

(As a reminder: soft-deprecated API is not scheduled for removal, does notraise warnings, and remains documented and tested. However, no newfunctionality will be added to it.)

Arrays ofPyType_Slot orPyModuleDef_Slot, which are accepted bythese functions, can contain any slots, including “new” ones definedin this PEP.This includes nested “new-style” slots (Py_slot_subslots).

Deprecation warnings

CPython will emit runtime deprecation warnings for the following cases,for slots where the case is currently disallowed in documentation but allowedby the runtime:

• setting a slot value to NULL:
  • all type slots exceptPy_tp_doc
  • Py_mod_create
  • Py_mod_exec
• repeating a slot ID in a single slots array (including sub-slot arraysadded in this PEP):
  • all type slots, except slots where this is already a runtime error(Py_tp_doc,Py_tp_members)
  • Py_mod_create
  • Py_mod_abi

Third-party slot ID allocation

It was suggested to allow third parties to reserve slot IDs for their own use.This would be mainly useful for alternate implementations. For example,something like GraalPy might want custom type slots (e.g. an “inheritsfrom this Java class” slot).Similarly, at one point PyPy had an extratp_pypy_flags in theirtypeobject struct.

This PEP does not specify a namespace mechanism.One can be added in the future.We’re also free to reserve individual slot IDs for alternate implementations.

Note that slots are not a good way forextension modules to add extra datato types or modules, as there is no API to retrieve the slots used to createa specific object.

Avoiding anonymous unions

This PEP proposes a struct withanonymous unions, which are not yet used inCPython’s documented public API.

There is no known issue with adding these, but the following notes maybe relevant:

• Anonymous unions are only supported in C since C11.But, CPython already requires the feature, and uses it for internal membersof thePyObject struct.
• Until C++20, which adds C-style designated initializers, C++ initializersonly allow setting the first member of a union.However, this is an issue fornamed unions as well.Avoiding unions entirely would mean losing most of the type-safetyimprovements of this PEP.

  Note that the proposed flagPySlot_INTPTR, and the workaround macrosPySlot_PTR &PySlot_PTR_STATIC, allow using this API incode that needs to be compatible with C++11 or has similar union-relatedlimitations.

• C++ doesn’t have anonymousstructs.This might surprise C programmers for whom anonymous structs/unions area single language feature.
• Non-C/C++ language wrappers may need to give the union a name.This is fine.(Dear reader: if you need this, please open a CPython issue aboutexposing a preferred name in headers and documentation.)

For a bigger picture: anonymous unions can be a helpful tool for implemetingtagged unions and for evolving public API in backwards-compatible ways.This PEP intentionally opens the door to using them more often.