Extending opaque types

In the Limited API, moststructs are opaque: their size and memory layoutare not exposed, so they can be changed in new versions of CPython (oralternate implementations of the C API).

This means that the usual subclassing pattern – making thestructused for instances of thebase type be the first element of thestructused for instances of thederived type – does not work.To illustrate with code, theexample from the tutorialextendsPyListObject (list)using the followingstruct:

typedefstruct{PyListObjectlist;intstate;}SubListObject;

This won’t compile in the Limited API, sincePyListObject is opaque (toallow changes as features and optimizations are implemented).

Instead, this PEP proposes using astruct with only the state neededin the subclass, that is:

typedefstruct{intstate;}SubListState;// (or just `typedef int SubListState;` in this case)

The subclass can now be completely decoupled from the memory layout (and size)of the superclass.

This is possible today. To use such a struct:

• when creating the class, usePyListObject->tp_basicsize+sizeof(SubListState)asPyType_Spec.basicsize;
• when accessing the data, usePyListObject->tp_basicsize as the offsetinto the instance (PyObject*).

However, this has disadvantages:

• The base’sbasicsize may not be properly aligned, causing issueson some architectures if not mitigated. (These issues can be particularlynasty if alignment changes in a new release.)
• PyTypeObject.tp_basicsize is not exposed in theLimited API, so extensions that support Limited API need tousePyObject_GetAttrString(obj,"__basicsize__").This is cumbersome, and unsafe in edge cases (the Python attribute canbe overridden).
• Variable-size objects are not handled(seeExtending variable-size objects below).

To make this easy (and evenbest practice for projects that choose loosecoupling over maximum performance), this PEP proposes an API to:

1. During class creation, specify thatSubListStateshould be “appended” toPyListObject, without passing any additionaldetails aboutlist. (The interpreter itself gets all necessary info,liketp_basicsize, from the base).

   This will be specified by a negativePyType_Spec.basicsize:-sizeof(SubListState).

2. Given an instance, and the subclassPyTypeObject*,get a pointer to theSubListState.A new function,PyObject_GetTypeData, will be added for this.

The base class is not limited toPyListObject, of course: it can be used toextend any base class whose instancestruct is opaque, unstable acrossreleases, or not exposed at all – includingtype(PyHeapTypeObject) or third-party extensions(for example, NumPy arrays[1]).

For cases where no additional state is needed, a zerobasicsize will beallowed: in that case, the base’stp_basicsize will be inherited.(This currently works, but lacks explicit documentation and tests.)

Thetp_basicsize of the new class will be set to the computed total size,so code that inspects classes will continue working as before.

Extending variable-size objects

Additional considerations are needed to subclassvariable-sizedobjectswhile maintaining loose coupling:the variable-sized data can collide with subclass data (SubListState inthe example above).

Currently, CPython doesn’t provide a way to prevent such collisions.So, the proposed mechanism of extending opaque classes (negativebase->tp_itemsize) willfail by default.

We could stop there, but since the motivating type —PyHeapTypeObject —is variable sized, we need a safe way to allow subclassing it.A bit of background first:

PyTupleObject:┌───────────────────┬───┬───┬╌╌╌╌┐│ PyObject_VAR_HEAD │var. data   │└───────────────────┴───┴───┴╌╌╌╌┘tuple subclass:┌───────────────────┬───┬───┬╌╌╌╌┬─────────────┐│ PyObject_VAR_HEAD │var. data   │subclass data│└───────────────────┴───┴───┴╌╌╌╌┴─────────────┘

heap type:┌───────────────────┬──────────────┬───┬───┬╌╌╌╌┐│ PyObject_VAR_HEAD │Heap type data│var. data   │└───────────────────┴──────────────┴───┴───┴╌╌╌╌┘type subclass:┌───────────────────┬──────────────┬─────────────┬───┬───┬╌╌╌╌┐│ PyObject_VAR_HEAD │Heap type data│subclass data│var. data   │└───────────────────┴──────────────┴─────────────┴───┴───┴╌╌╌╌┘

Relative member offsets

One more piece of the puzzle isPyMemberDef.offset.Extensions that use a subclass-specificstruct (SubListState above)will get a way to specify “relative” offsets (offsets based from thisstruct) rather than “absolute” ones (based off thePyObject struct).

One way to do it would be to automatically assume “relative” offsetswhen creating a class using the new API.However, this implicit assumption would be too surprising.

To be more explicit, this PEP proposes a new flag for “relative” offsets.At least initially, this flag will serve only as a check against misuse(and a hint for reviewers).It must be present if used with the new API, and must not be used otherwise.

Relativebasicsize

Thebasicsize member ofPyType_Spec will be allowed to be zero ornegative.In that case, its absolute value will specify how muchextra storage spaceinstances of the new class require, in addition to the basicsize of thebase class.That is, the basicsize of the resulting class will be:

type->tp_basicsize=_align(base->tp_basicsize)+_align(-spec->basicsize);

where_align rounds up to a multiple ofalignof(max_align_t).

Whenspec->basicsize is zero, basicsize will be inheriteddirectly instead, i.e. set tobase->tp_basicsize without aligning.(This already works; explicit tests and documentation will be added.)

On an instance, the memory area specific to a subclass – that is, the“extra space” that subclass reserves in addition its base – will be availablethrough a new function,PyObject_GetTypeData.In CPython, this function will be defined as:

void*PyObject_GetTypeData(PyObject*obj,PyTypeObject*cls){return(char*)obj+_align(cls->tp_base->tp_basicsize);}

Another function will be added to retrieve the size of this memory area:

Py_ssize_tPyType_GetTypeDataSize(PyTypeObject*cls){returncls->tp_basicsize-_align(cls->tp_base->tp_basicsize);}

The result may be higher than requested by-basicsize. It is safe touse all of it (e.g. withmemset).

The new*Get* functions come with an important caveat, which will bepointed out in documentation: They may only be used for classes created usingnegativePyType_Spec.basicsize. For other classes, their behavior isundefined.(Note that this allows the above code to assumecls->tp_base is notNULL.)

Inheritingitemsize

Whenspec->itemsize is zero,tp_itemsize will be inheritedfrom the base.(This already works; explicit tests and documentation will be added.)

A new type flag,Py_TPFLAGS_ITEMS_AT_END, will be added.This flag can only be set on types with non-zerotp_itemsize.It indicates that the variable-sized portion of an instanceis stored at the end of the instance’s memory.

The default metatype (PyType_Type) will set this flag.

A new function,PyObject_GetItemData, will be added to access thememory reserved for variable-sized content of types with the new flag.In CPython it will be defined as:

void*PyObject_GetItemData(PyObject*obj){if(!PyType_HasFeature(Py_TYPE(obj),Py_TPFLAGS_ITEMS_AT_END){<failwithTypeError>}return(char*)obj+Py_TYPE(obj)->tp_basicsize;}

This function will initiallynot be added to the Limited API.

Extending a class with positivebase->itemsize usingnegativespec->basicsize will fail unlessPy_TPFLAGS_ITEMS_AT_ENDis set, either on the base or inspec->flags.(SeeExtending variable-size objects for a full explanation.)

Extending a class with positivespec->itemsize using negativespec->basicsize will fail.

Relative member offsets

In types defined using negativePyType_Spec.basicsize, the offsets ofmembers defined viaPy_tp_members must be relative to theextra subclass data, rather than the fullPyObject struct.This will be indicated by a new flag inPyMemberDef.flags:Py_RELATIVE_OFFSET.

In the initial implementation, the new flag will be redundant. It only servesto make the offset’s changed meaning clear, and to help avoid mistakes.It will be an error tonot usePy_RELATIVE_OFFSET with negativebasicsize, and it will be an error to use it in any other context(i.e. direct or indirect calls toPyDescr_NewMember,PyMember_GetOne,PyMember_SetOne).

CPython will adjust the offset and clear thePy_RELATIVE_OFFSET flag wheninitializing a type.This means that:

• the created type’stp_members will not match the inputdefinition’sPy_tp_members slot, and
• any code that readstp_members will not need to handle the flag.

Alignment & Performance

The proposed implementation may waste some space if instance structsneed smaller alignment thanalignof(max_align_t).Also, dealing with alignment makes the calculation slower than it could beif we could rely onbase->tp_basicsize being properly aligned for thesubtype.

In other words, the proposed implementation focuses on safety and ease of use,and trades space and time for it.If it turns out that this is a problem, the implementation can be adjustedwithout breaking the API:

• The offset to the type-specific buffer can be stored, soPyObject_GetTypeData effectively becomes(char*)obj+cls->ht_typedataoffset, possibly speeding things up atthe cost of an extra pointer in the class.
• Then, a newPyType_Slot can specify the desired alignment, toreduce space requirements for instances.

Other layouts for variable-size types

A flag likePy_TPFLAGS_ITEMS_AT_END could be added to signal the“tuple-like” layout described inExtending variable-size objects,and all mechanisms this PEP proposes could be adapted to support it.Other layouts could be added as well.However, it seems there’d be very little practical benefit,so it’s just a theoretical possibility.