Invalid class definitions

The following class definition raises an error at runtime, becauseint andstr are distinct disjoint bases:

classC(int,str):pass

Without knowledge of disjoint bases, type checkers are not currently able to detect the reason why this classdefinition is invalid, though they may detect that if this class were to exist, some of its methods would be incompatible.(When it sees this class definition, mypy will point at incompatible definitions of__add__ and several othermethods.)

This is not a particularly compelling problem by itself, as the error would usually be caught the first time the codeis imported, but it is mentioned here for completeness.

Reachability

We already mentioned the reachability of code usingisinstance(). Similar issues arise with other typenarrowing constructs such asmatch statements: correct inference of reachability requires an understanding ofdisjoint bases.

classA:passclassB:passdeff(x:A):matchx:caseB():# reachableprint("It's both!")defg(x:int):matchx:casestr():# unreachableprint("It's both!")

Overloads

Functions decorated with@overload may be unsafe if the parameter types of some overloads overlap, but the return typesdo not. For example, the following set of overloads could be exploited toachieve unsound behavior:

fromtypingimportoverloadclassA:passclassB:pass@overloaddeff(x:A)->str:...@overloaddeff(x:B)->int:...

If a class exists that inherits from bothA andB, then type checkers could pick the wrong overload on acall tof().

Type checkers could detect this source of unsafety and warn about it, but a correct implementation requires an understanding of disjoint bases,because it relies on knowing whether values that are instances of bothA andB can exist.Although many type checkers already perform a version of this check for overlapping overloads, the typing specification does notcurrently prescribe how this check should work. This PEP does not propose to change that, but it helps provide a building block fora sound check for overlapping overloads.

Intersection types

Explicit intersection types, denoting a type that contains values that are instances of all of thegiven types, are not currently part of the type system. They do, however, arise naturally in a set-theoretic type systemlike Python’s as a result of type narrowing, and future extensions to the type system may add support for explicit intersection types.

With intersection types, it is often important to know whether a particular intersection is inhabited, that is, whetherthere are values that can be members of that intersection. This allows type checkers to understand reachability andprovide more precise type information to users.

As a concrete example, a possible implementation of assignability with intersection types could be thatgiven an intersection typeA&B, a typeC is assignable to it ifC is assignable to at least one ofA andB, and overlaps with all ofA andB. (“Overlaps” here means that at least one runtime value could existthat would be a member of both types. That is,A andB overlap ifA&B is inhabited.) The second part of the rule ensures thatstr is not assignable to a type likeint&Any: whilestr is assignable toAny,it does not overlap withint. But of course, we can only know thatstr andint do not overlap if we knowthat both classes are disjoint bases.

Overview

Disjoint bases can be helpful in many corners of the type system. Though some of these corners are underspecified,speculative, or of marginal importance, in each case the concept of disjoint bases enables type checkers to gain a moreprecise understanding than the current type system allows. Thus, disjoint bases provide a firm foundation(a solid base, if you will) for improving the Python type system.

Naming

The initial version of this PEP used the name “solid base”, following the terminology used in CPython’s implementation.However, this term is somewhat vague. The alternative term “disjoint base” suggests that a class with this decoratoris disjoint from other bases, which is a good first-order description of the concept. (The exact semantics are more subtleand are described below.)

Runtime restrictions on multiple inheritance

While Python generally allows multiple inheritance, the runtime imposes various restrictions, asdocumented in CPython.Two sets of restrictions, around a consistent MRO and a consistent metaclass, can already be implemented bytype checkers using information available in the type system. The third restriction, around instance layout,is the one that requires knowledge of disjoint bases. Classes that contain a non-empty__slots__ definitionare automatically disjoint bases, as are many built-in classes implemented in C.

Alternative implementations of Python, such as PyPy, tend to behave similarly to CPython but may differ in details,such as exactly which standard library classes are disjoint bases. As the type system does not currently contain anyexplicit support for alternative Python implementations, this PEP recommends that stub libraries such as typesheduse CPython’s behavior to determine when to use the@disjoint_base decorator. If future extensions to the type systemadd support for alternative implementations (for example, branching on the value ofsys.implementation.name),stubs could condition the presence of the@disjoint_base decorator on the implementation where necessary.

Although the concept of “disjoint bases” (referred to as “solid bases”) in the CPython implementation has existedfor decades, the rules for deciding which classes are disjoint bases have occasionally changed.Before Python 3.12, adding a__dict__ or support for weakrefs relative to the base class could make aclass a disjoint base. In practice, this often meant that Python-implemented classes inheriting fromclasses implemented in C, such asnamedtuple() classes, were themselves disjoint bases.This behavior was changed in Python 3.12 bypython/cpython#96028.This PEP focuses on supporting the behavior of Python 3.12 and later, which is simpler and easier to understand.Type checkers may choose to implement a version of the pre-3.12 behavior if they wish, but doing this correctlyrequires information that is not currently available in the type system.

The exact set of classes that are disjoint bases at runtime may change again in future versions of Python.If this were to happen, the type stubs used by type checkers could be updated to reflect this new reality.In other words, this PEP adds the concept of disjoint bases to the type system, but it does not prescribe exactlywhich classes are disjoint bases.

@disjoint_base in implementation files

The most obvious use case for the@disjoint_base decorator will be in stub files for C libraries, such as the standard library,for marking disjoint bases implemented in C.

However, there are also use cases for marking disjoint bases in implementation files, where the effect would be to disallowthe existence of child classes that inherit from the decorated class and another disjoint base, such as a standard library classor another user class decorated with@disjoint_base. For example, this could allow type checkers to flag code that can onlybe reachable if a class exists that inherits from both a user class and a standard library class such asint orstr,which may be technically possible but not practically plausible.

@disjoint_baseclassBaseModel:# ... General logic for model classespassclassSpecies(BaseModel):name:str# ... more fieldsdefprocess_species(species:Species):ifisinstance(species,str):# oops, forgot `.name`pass# type checker should warn about this branch being unreachable# BaseModel and str are disjoint bases, so a class that inherits from both cannot exist

This is similar in principle to the existing@final decorator, which also acts to restrict subclassing: in stubs, itis used to mark classes that programmatically disallow subclassing, but in implementation files, it is often used toindicate that a class is not intended to be subclassed, without runtime enforcement.

@disjoint_base on special classes

The@disjoint_base decorator is primarily intended for nominal classes, but the type system contains some other constructs thatsyntactically use class definitions, so we have to consider whether the decorator should be allowed on them as well, and if so,what it would mean.

ForProtocol definitions, the most consistent interpretation would be that the only classes that can implement theprotocol would be classes that use nominal inheritance from the protocol, or@final classes that implement the protocol.Other classes either have or could potentially have a disjoint base that is not the protocol. This is convoluted and not useful,so we disallow@disjoint_base onProtocol definitions.

Similarly, the concept of a “disjoint base” is not meaningful onTypedDict definitions, as TypedDicts are purely structural types.

Although they receive some special treatment in the type system,NamedTuple definitions create real nominal classes that canhave child classes, so it makes sense to allow@disjoint_base on them and treat them like regular classes for the purposesof the disjoint base mechanism. AllNamedTuple classes havetuple, a disjoint base, in their MRO, so theycannot multiple inherit from other disjoint bases.

Solid bases in CPython

The concept of “solid bases” has been part of the CPython implementation for a long time;the concept dates back toa 2001 commit.Nevertheless, the concept has received little attention in the documentation.Although details of the mechanism are closely tied to CPython’s internal object representation,it is useful to explain at a high level how and why CPython works this way.

Every object in CPython is essentially a pointer to a C struct, a contiguous piece of memory thatcontains information about the object. Some information is managed by the interpreter and sharedby many or all objects, such as a reference to the type of the object, and the attribute__dict__for user-defined objects. Some classes contain additional information that is specific to that class.For example, user-defined classes with__slots__ contain a place in memory for each slot,and the built-infloat class contains a Cdouble value that stores the value of the float.This memory layout must be preserved for all instances of the class: C code thatinteracts with afloat expects to find the value at a particular offset in the object’s memory.

When a child class is created, CPython must create a memory layout for the new class thatis compatible with all of its parent classes. For example, when a child class offloatis created, it must be possible to pass instances of the child class to C code that interactsdirectly with the underlying struct for thefloat class. Therefore, such a subclass must storethedouble value at the same offset as the parentfloat class does. It may, however, addadditional fields at the end of the struct. CPython knows how to do this with the__dict__attribute, which is why it is possible to create a child class offloat that adds a__dict__.

However, there is no way to combine afloat, which must have adouble in its struct,with another C type likeint, which stores different data at the same spot. Therefore,a common subclass offloat andint cannot exist. We say thatfloat andintare solid bases.

A class implemented in C is a solid base if it has an underlying struct that storesdata at a fixed offset, and that struct is different from the struct of its parent class.A C class may also store a variable-size array of data (such as the contents of a string);if this differs from the parent class, the class also becomes a solid base.CPython’s implementation deduces this from thetp_itemsizeandtp_basicsize fields of the type object, which are alsoaccessible from Python code as the undocumented attributes__itemsize__ and__basicsize__on type objects.

Similarly, classes implemented in Python are solid bases if they have__slots__, becauseslots force a particular memory layout.

Mypy’s incompatibility check

The mypy type checker considers two classes to be incompatible if they haveincompatible methods. For example, mypy considers theint andstr classes to be incompatiblebecause they have incompatible definitions of various methods. Given a class definition like:

classC(int,str):pass

Mypy will outputDefinitionof"__add__"inbaseclass"int"isincompatiblewithdefinitioninbaseclass"str",and similar errors for a number of other methods. These errors are correct, because the definitions of__add__ in the two classes are indeed incompatible:int.__add__ expects anint argument, whilestr.__add__ expects astr. If this class were to exist, at runtime__add__ would resolve toint.__add__. Instances ofC would also be members of thestr type, but they would not supportsome of the operations thatstr supports, such as concatenation with anotherstr.

So far, so good. But mypy also uses very similar logic to conclude that no classcan inherit from bothint andstr.Nevertheless, it accepts the following class definition without error:

fromtypingimportNeverclassC(int,str):def__add__(self,other:object)->Never:raiseTypeErrordef__mod__(self,other:object)->Never:raiseTypeErrordef__mul__(self,other:object)->Never:raiseTypeErrordef__rmul__(self,other:object)->Never:raiseTypeErrordef__ge__(self,other:int|str)->bool:returnint(self)>otherifisinstance(other,int)elsestr(self)>otherdef__gt__(self,other:int|str)->bool:returnint(self)>=otherifisinstance(other,int)elsestr(self)>=otherdef__lt__(self,other:int|str)->bool:returnint(self)<otherifisinstance(other,int)elsestr(self)<otherdef__le__(self,other:int|str)->bool:returnint(self)<=otherifisinstance(other,int)elsestr(self)<=otherdef__getnewargs__(self)->Never:raiseTypeError

There is a similar situation with attributes. Given two classes with incompatibleattributes, mypy claims that a common subclass cannot exist, yet it acceptsa subclass that overrides these attributes to make them compatible:

fromtypingimportNeverclassX:a:intclassY:a:strclassZ(X,Y):@propertydefa(self)->Never:raiseRuntimeError("no luck")@a.setterdefa(self,value:int|str)->None:pass

While the examples given so far rely on overrides that returnNever, mypy’s rulecan also reject classes that have more practically useful implementations:

fromtypingimportLiteralclassCarnivore:defeat(self,food:Literal["meat"])->None:print("devouring meat")classHerbivore:defeat(self,food:Literal["plants"])->None:print("nibbling on plants")classOmnivore(Carnivore,Herbivore):defeat(self,food:str)->None:print(f"eating{food}")defis_it_both(obj:Carnivore):# mypy --warn-unreachable:# Subclass of "Carnivore" and "Herbivore" cannot exist: would have incompatible method signaturesifisinstance(obj,Herbivore):pass

Mypy’s rule works reasonably well in practice for deducing whether an intersection of twoclasses is inhabited. Most builtin classes that are disjoint bases happen to implement common dundermethods such as__add__ and__iter__ in incompatible ways, so mypy will consider themincompatible. There are some exceptions: mypy allowsclassC(BaseException,int):...,though both of these classes are disjoint bases and the class definition is rejected at runtime.Conversely, when multiple inheritance is used in practice, usually the parent classes will nothave incompatible methods.

Thus, mypy’s approach to deciding that two classes cannot intersect is both too broad(it incorrectly considers some intersections to be uninhabited) and too narrow (it missessome intersections that are uninhabited because of disjoint bases). This is discussed inan issue on the mypy tracker.