Generics¶
Introduction¶
Since type information about objects kept in containers cannot bestatically inferred in a generic way, abstract base classes have beenextended to support subscription to denote expected types for containerelements. Example:
fromcollections.abcimportMappingdefnotify_by_email(employees:set[Employee],overrides:Mapping[str,str])->None:...
Generics can be parameterized by using a factory available intyping calledTypeVar. Example:
fromcollections.abcimportSequencefromtypingimportTypeVarT=TypeVar('T')# Declare type variabledeffirst(l:Sequence[T])->T:# Generic functionreturnl[0]
Or, since Python 3.12 (PEP 695), by using the new syntax forgeneric functions:
fromcollections.abcimportSequencedeffirst[T](l:Sequence[T])->T:# Generic functionreturnl[0]
The two syntaxes are equivalent.In either case the contract is that the returned value is consistent withthe elements held by the collection.
ATypeVar() expression must always directly be assigned to avariable (it should not be used as part of a larger expression). Theargument toTypeVar() must be a string equal to the variable nameto which it is assigned. Type variables must not be redefined.
TypeVar supports constraining parametric types to a fixed set of possibletypes (note: those types cannot be parameterized by type variables). Forexample, we can define a type variable that ranges over juststr andbytes. By default, a type variable ranges over all possible types.Example of constraining a type variable:
fromtypingimportTypeVarAnyStr=TypeVar('AnyStr',str,bytes)defconcat(x:AnyStr,y:AnyStr)->AnyStr:returnx+y
Or using the built-in syntax (3.12 and higher):
defconcat[AnyStr:(str,bytes)](x:AnyStr,y:AnyStr)->AnyStr:returnx+y
The functionconcat can be called with either twostr argumentsor twobytes arguments, but not with a mix ofstr andbytesarguments.
There should be at least two constraints, if any; specifying a singleconstraint is disallowed.
Subtypes of types constrained by a type variable should be treatedas their respective explicitly listed base types in the context of thetype variable. Consider this example:
classMyStr(str):...x=concat(MyStr('apple'),MyStr('pie'))
The call is valid but the type variableAnyStr will be set tostr and notMyStr. In effect, the inferred type of the returnvalue assigned tox will also bestr.
Additionally,Any is a valid value for every type variable.Consider the following:
defcount_truthy(elements:list[Any])->int:returnsum(1foreleminelementsifelem)
This is equivalent to omitting the generic notation and just sayingelements:list.
User-defined generic types¶
There are several ways to define a user-defined class as generic:
Include a
Genericbase class.Use the new generic class syntax in Python 3.12 and higher.
Include a
Protocolbase class parameterized with type variables. Thisapproach also marks the class as a protocol - seegeneric protocols for more information.Include a generic base class parameterized with type variables.
Example usingGeneric:
fromtypingimportTypeVar,GenericfromloggingimportLoggerT=TypeVar('T')classLoggedVar(Generic[T]):def__init__(self,value:T,name:str,logger:Logger)->None:self.name=nameself.logger=loggerself.value=valuedefset(self,new:T)->None:self.log('Set '+repr(self.value))self.value=newdefget(self)->T:self.log('Get '+repr(self.value))returnself.valuedeflog(self,message:str)->None:self.logger.info('{}:{}'.format(self.name,message))
Or, using the new generic class syntax:
classLoggedVar[T]:# methods as in previous example
This implicitly addsGeneric[T] as a base class, and type checkersshould treat the two definitions ofLoggedVar largely equivalently (exceptfor variance, see below).
Generic[T] as a base class defines that the classLoggedVartakes a single type parameterT. This also makesT valid asa type within the class body.
TheGeneric base class uses a metaclass that defines__getitem__so thatLoggedVar[t] is valid as a type:
fromcollections.abcimportIterabledefzero_all_vars(vars:Iterable[LoggedVar[int]])->None:forvarinvars:var.set(0)
A generic type can have any number of type variables, and type variablesmay be constrained. This is valid:
fromtypingimportTypeVar,GenericT=TypeVar('T')S=TypeVar('S')classPair(Generic[T,S]):...
Each type variable argument toGeneric must be distinct. This isthus invalid:
fromtypingimportTypeVar,GenericT=TypeVar('T')classPair(Generic[T,T]):# INVALID...
All arguments toGeneric orProtocol must be type variables:
fromtypingimportGeneric,ProtocolclassBad1(Generic[int]):# INVALID...classBad2(Protocol[int]):# INVALID...
When aGeneric or parameterizedProtocol base class is present, all typeparameters for the class must appear within theGeneric orProtocol type argument list, respectively. A type checker should report anerror if a type variable that is not included in the type argument list appearselsewhere in the base class list:
fromtypingimportGeneric,Protocol,TypeVarfromcollections.abcimportIterableT=TypeVar('T')S=TypeVar('S')classBad1(Iterable[T],Generic[S]):# INVALID...classBad2(Iterable[T],Protocol[S]):# INVALID...
Note that the above rule does not apply to a bareProtocol base class. Thisis valid (see below):
fromtypingimportProtocol,TypeVarfromcollections.abcimportIteratorT=TypeVar('T')classMyIterator(Iterator[T],Protocol):...
When noGeneric or parameterizedProtocol base class is present, adefined class is generic if you subclass one or more other generic classes andspecify type variables for their parameters. SeeArbitrary generic types as base classesfor details.
You can use multiple inheritance withGeneric:
fromtypingimportTypeVar,Genericfromcollections.abcimportSized,Iterable,ContainerT=TypeVar('T')classLinkedList(Sized,Generic[T]):...K=TypeVar('K')V=TypeVar('V')classMyMapping(Iterable[tuple[K,V]],Container[tuple[K,V]],Generic[K,V]):...
Subclassing a generic class without specifying type parameters assumesAny for each position unless the type parameter has a default value.In the following example,MyIterable is not generic but implicitly inheritsfromIterable[Any]:
fromcollections.abcimportIterableclassMyIterable(Iterable):# Same as Iterable[Any]...
Generic metaclasses are not supported.
Scoping rules for type variables¶
When using the generic class syntax introduced in Python 3.12, the location ofits declaration defines its scope. When using the older syntax, the scopingrules are more subtle and complex:
A type variable used in a generic function could be inferred to representdifferent types in the same code block. Example:
fromtypingimportTypeVar,GenericT=TypeVar('T')deffun_1(x:T)->T:...# T heredeffun_2(x:T)->T:...# and here could be differentfun_1(1)# This is OK, T is inferred to be intfun_2('a')# This is also OK, now T is str
A type variable used in a method of a generic class that coincideswith one of the variables that parameterize this class is always boundto that variable. Example:
fromtypingimportTypeVar,GenericT=TypeVar('T')classMyClass(Generic[T]):defmeth_1(self,x:T)->T:...# T heredefmeth_2(self,x:T)->T:...# and here are always the samea:MyClass[int]=MyClass()a.meth_1(1)# OKa.meth_2('a')# This is an error!
A type variable used in a method that does not match any of the variablesthat parameterize the class makes this method a generic function in thatvariable:
T=TypeVar('T')S=TypeVar('S')classFoo(Generic[T]):defmethod(self,x:T,y:S)->S:...x:Foo[int]=Foo()y=x.method(0,"abc")# inferred type of y is str
Unbound type variables should not appear in the bodies of generic functions,or in the class bodies apart from method definitions:
T=TypeVar('T')S=TypeVar('S')defa_fun(x:T)->None:# this is OKy:list[T]=[]# but below is an error!y:list[S]=[]classBar(Generic[T]):# this is also an erroran_attr:list[S]=[]defdo_something(self,x:S)->S:# this is OK though...
A generic class definition that appears inside a generic functionshould not use type variables that parameterize the generic function:
defa_fun(x:T)->None:# This is OKa_list:list[T]=[]...# This is however illegalclassMyGeneric(Generic[T]):...
A generic class nested in another generic class cannot use the same typevariables. The scope of the type variables of the outer classdoesn’t cover the inner one:
T=TypeVar('T')S=TypeVar('S')classOuter(Generic[T]):classBad(Iterable[T]):# Error...classAlsoBad:x:list[T]# Also an errorclassInner(Iterable[S]):# OK...attr:Inner[T]# Also OK
Instantiating generic classes and type erasure¶
User-defined generic classes can be instantiated. Suppose we writeaNode class:
classNode[T]:...
To createNode instances you callNode() just as for a regularclass. At runtime the type (class) of the instance will beNode.But what type does it have to the type checker? The answer depends onhow much information is available in the call. If the constructor(__init__ or__new__) usesT in its signature, and acorresponding argument value is passed, the type of the correspondingargument(s) is substituted. Otherwise, the default value for the typeparameter (orAny, if no default is provided) is assumed. Example:
classNode[T]:x:T# Instance attribute (see below)def__init__(self,label:T|None=None)->None:...x=Node('')# Inferred type is Node[str]y=Node(0)# Inferred type is Node[int]z=Node()# Inferred type is Node[Any]
In case the inferred type uses[Any] but the intended type is morespecific, you can use an annotation (see below) to force the type ofthe variable, e.g.:
# (continued from previous example)a:Node[int]=Node()b:Node[str]=Node()
Alternatively, you can instantiate a specific concrete type, e.g.:
# (continued from previous example)p=Node[int]()q=Node[str]()r=Node[int]('')# Errors=Node[str](0)# Error
Note that the runtime type (class) ofp andq is still justNode–Node[int] andNode[str] are distinguishable class objects, butthe runtime class of the objects created by instantiating them doesn’trecord the distinction. This behavior is called “type erasure”; it iscommon practice in languages with generics (e.g. Java, TypeScript).
Using generic classes (parameterized or not) to access attributes will resultin type check failure. Outside the class definition body, a class attributecannot be assigned, and can only be looked up by accessing it through aclass instance that does not have an instance attribute with the same name:
# (continued from previous example)Node[int].x=1# ErrorNode[int].x# ErrorNode.x=1# ErrorNode.x# Errortype(p).x# Errorp.x# Ok (evaluates to int)Node[int]().x# Ok (evaluates to int)p.x=1# Ok, but assigning to instance attribute
Generic versions of abstract collections likeMapping orSequenceand generic versions of built-in classes –List,Dict,Set,andFrozenSet – cannot be instantiated. However, concrete user-definedsubclasses thereof and generic versions of concrete collections can beinstantiated:
data=DefaultDict[int,bytes]()
Note that one should not confuse static types and runtime classes.The type is still erased in this case and the above expression isjust a shorthand for:
data:DefaultDict[int,bytes]=collections.defaultdict()
It is not recommended to use the subscripted class (e.g.Node[int])directly in an expression – using a type alias (e.g.IntNode=Node[int])instead is preferred. (First, creating the subscripted class,e.g.Node[int], has a runtime cost. Second, using a type aliasis more readable.)
Arbitrary generic types as base classes¶
Generic[T] is only valid as a base class – it’s not a proper type.However, user-defined generic types such asLinkedList[T] from theabove example and built-in generic types and ABCs such aslist[T]andIterable[T] are valid both as types and as base classes. Forexample, we can define a subclass ofdict that specializes typearguments:
classNode:...classSymbolTable(dict[str,list[Node]]):defpush(self,name:str,node:Node)->None:self.setdefault(name,[]).append(node)defpop(self,name:str)->Node:returnself[name].pop()deflookup(self,name:str)->Node|None:nodes=self.get(name)ifnodes:returnnodes[-1]returnNone
SymbolTable is a subclass ofdict and a subtype ofdict[str,list[Node]].
If a generic base class has a type variable as a type argument, thismakes the defined class generic. For example, we can define a genericLinkedList class that is iterable and a container:
fromtypingimportTypeVarfromcollections.abcimportIterable,ContainerT=TypeVar('T')classLinkedList(Iterable[T],Container[T]):...
NowLinkedList[int] is a valid type. Note that we can useTmultiple times in the base class list, as long as we don’t use thesame type variableT multiple times withinGeneric[...].
Also consider the following example:
fromtypingimportTypeVarfromcollections.abcimportMappingT=TypeVar('T')classMyDict(Mapping[str,T]):...
In this caseMyDict has a single type parameter,T.
Type variables are applied to the defined class in the order in whichthey first appear in any generic base classes:
fromtypingimportGeneric,TypeVarT1=TypeVar('T1')T2=TypeVar('T2')T3=TypeVar('T3')classParent1(Generic[T1,T2]):...classParent2(Generic[T1,T2]):...classChild(Parent1[T1,T3],Parent2[T2,T3]):...
ThatChild definition is equivalent to:
classChild(Parent1[T1,T3],Parent2[T2,T3],Generic[T1,T3,T2]):...
A type checker should report an error when the type variable order isinconsistent:
fromtypingimportGeneric,TypeVarT1=TypeVar('T1')T2=TypeVar('T2')T3=TypeVar('T3')classGrandparent(Generic[T1,T2]):...classParent(Grandparent[T1,T2]):...classChild(Parent[T1,T2],Grandparent[T2,T1]):# INVALID...
Abstract generic types¶
The metaclass used byGeneric is a subclass ofabc.ABCMeta.A generic class can be an ABC by including abstract methodsor properties, and generic classes can also have ABCs as baseclasses without a metaclass conflict.
Type variables with an upper bound¶
A type variable may specify an upper bound usingbound=<type> (when usingtheTypeVar constructor) or using:<type> (when using the nativesyntax for generics). The bound itself cannot be parameterized by typevariables. This means that an actual type substituted (explicitly orimplicitly) for the type variable must beassignable to the bound.Example:
fromtypingimportTypeVarfromcollections.abcimportSizedST=TypeVar('ST',bound=Sized)deflonger(x:ST,y:ST)->ST:iflen(x)>len(y):returnxelse:returnylonger([1],[1,2])# ok, return type list[int]longer({1},{1,2})# ok, return type set[int]longer([1],{1,2})# ok, return type a supertype of list[int] and set[int]
An upper bound cannot be combined with type constraints (as used inAnyStr,see the example earlier); type constraints cause the inferred type to beexactly one of the constraint types, while an upper bound just requires thatthe actual type isassignable to the bound.
Variance¶
Consider a classEmployee with a subclassManager. Nowsuppose we have a function with an argument annotated withlist[Employee]. Should we be allowed to call this function with avariable of typelist[Manager] as its argument? Many people wouldanswer “yes, of course” without even considering the consequences.But unless we know more about the function, a type checker shouldreject such a call: the function might append anEmployee instanceto the list, which would violate the variable’s type in the caller.
It turns out such an argument actscontravariantly, whereas theintuitive answer (which is correct in case the function doesn’t mutateits argument!) requires the argument to actcovariantly. A longerintroduction to these concepts can be found onWikipedia and inPEP 483; here we just show how to controla type checker’s behavior.
By default generic types declared using the oldTypeVar syntax areconsideredinvariant in all type variables, which means that e.g.list[Manager] is neither a supertype nor a subtype oflist[Employee].
See below for the behavior when using the built-in generic syntax in Python3.12 and higher.
To facilitate the declaration of container types where covariant orcontravariant type checking is acceptable, type variables accept keywordargumentscovariant=True orcontravariant=True. At most one of thesemay be passed. Generic types defined with such variables are consideredcovariant or contravariant in the corresponding variable. By convention,it is recommended to use names ending in_co for type variablesdefined withcovariant=True and names ending in_contra for thatdefined withcontravariant=True.
A typical example involves defining an immutable (or read-only)container class:
fromtypingimportTypeVar,Genericfromcollections.abcimportIterable,IteratorT_co=TypeVar('T_co',covariant=True)classImmutableList(Generic[T_co]):def__init__(self,items:Iterable[T_co])->None:...def__iter__(self)->Iterator[T_co]:......classEmployee:...classManager(Employee):...defdump_employees(emps:ImmutableList[Employee])->None:forempinemps:...mgrs:ImmutableList[Manager]=ImmutableList([Manager()])dump_employees(mgrs)# OK
The read-only collection classes intyping are all declaredcovariant in their type variable (e.g.Mapping andSequence). Themutable collection classes (e.g.MutableMapping andMutableSequence) are declared invariant. The one example ofa contravariant type is theGenerator type, which is contravariantin thesend() argument type (see below).
Variance is meaningful only when a type variable is bound to a generic class.If a type variable declared as covariant or contravariant is bound to a genericfunction or type alias, type checkers may warn users about this. However, anysubsequent type analysis involving such functions or aliases should ignore thedeclared variance:
T=TypeVar('T',covariant=True)classA(Generic[T]):# T is covariant in this context...deff(x:T)->None:# Variance of T is meaningless in this context...Alias=list[T]|set[T]# Variance of T is meaningless in this context
ParamSpec¶
(Originally specified byPEP 612.)
ParamSpec Variables¶
Declaration¶
In Python 3.12 and newer, a parameter specification variable can be introducedinline by prefixing its name with** inside a type parameter list(for a function, class, or type alias).
defdecorator[**P](func:Callable[P,int])->Callable[P,str]:...classCallbackWrapper[T,**P]:callback:Callable[P,T]
Prior to 3.12, theParamSpec constructor can be used.
fromtypingimportParamSpecP=ParamSpec("P")# AcceptedP=ParamSpec("WrongName")# Rejected because P =/= WrongName
The runtime should acceptbounds andcovariant andcontravariantarguments in the declaration just astyping.TypeVar does, but for now wewill defer the standardization of the semantics of those options to a later PEP.
Valid use locations¶
Previously only a list of parameter arguments ([A,B,C]) or an ellipsis(signifying “undefined parameters”) were acceptable as the first “argument” totyping.Callable . We now augment that with two new options: a parameterspecification variable (Callable[P,int]) or a concatenation on aparameter specification variable (Callable[Concatenate[int,P],int]).
callable::=Callable"["parameters_expression,type_expression"]"parameters_expression::=|"..."|"["[type_expression(","type_expression)*]"]"|parameter_specification_variable|concatenate"["type_expression(","type_expression)*","parameter_specification_variable"]"
whereparameter_specification_variable is introduced either inline (using** in a type parameter list) or viatyping.ParamSpec as shown above,andconcatenate istyping.Concatenate.
As before,parameters_expressions by themselves are not acceptable inplaces where a type is expected
deffoo[**P](x:P)->P:...# Rejecteddeffoo[**P](x:Concatenate[int,P])->int:...# Rejecteddeffoo[**P](x:list[P])->None:...# Rejecteddeffoo[**P](x:Callable[[int,str],P])->None:...# Rejected
User-Defined Generic Classes¶
Just as defining a class withclassC[T]:... makes that class generic overthe type parameterT, adding**P makes it generic over a parameterspecification.
fromcollections.abcimportCallablefromtypingimportConcatenateclassX[T,**P]:f:Callable[P,int]x:Tdefaccept_params[**P](x:X[int,P])->str:...# Accepteddefaccept_concatenate[**P](x:X[int,Concatenate[int,P]])->str:...# Accepteddefaccept_concrete(x:X[int,[int,bool]])->str:...# Accepteddefaccept_unspecified(x:X[int,...])->str:...# Accepteddefreject_bad(x:X[int,int])->str:...# Rejected
By the rules defined above, spelling a concrete instance of a class genericwith respect to only a singleParamSpec would require unsightly doublebrackets. For aesthetic purposes we allow these to be omitted.
classZ[**P]:f:Callable[P,int]defaccept_list(x:Z[[int,str,bool]])->str:...# Accepteddefaccept_flat(x:Z[int,str,bool])->str:...# Equivalent# Both Z[[int, str, bool]] and Z[int, str, bool] express this:classZ_instantiated:f:Callable[[int,str,bool],int]
Semantics¶
The inference rules for the return type of a function invocation whose signaturecontains aParamSpec variable are analogous to those aroundevaluating ones withTypeVars.
defchanges_return_type_to_str[**P](x:Callable[P,int])->Callable[P,str]:...defreturns_int(a:str,b:bool)->int:...f=changes_return_type_to_str(returns_int)# f should have the type:# (a: str, b: bool) -> strf("A",True)# Acceptedf(a="A",b=True)# Acceptedf("A","A")# Rejectedexpects_str(f("A",True))# Acceptedexpects_int(f("A",True))# Rejected
Just as with traditionalTypeVars, a user may include the sameParamSpec multiple times in the arguments of the same function,to indicate a dependency between multiple arguments. In these cases a typechecker may choose to solve to a common behavioral supertype (i.e. a set ofparameters for which all of the valid calls are valid in both of the subtypes),but is not obligated to do so.
deffoo[**P](x:Callable[P,int],y:Callable[P,int])->Callable[P,bool]:...defx_y(x:int,y:str)->int:...defy_x(y:int,x:str)->int:...foo(x_y,x_y)# Should return (x: int, y: str) -> bool# (a callable with two positional-or-keyword parameters)foo(x_y,y_x)# Could return (a: int, b: str, /) -> bool# (a callable with two positional-only parameters)# This works because both callables have types that are# behavioral subtypes of Callable[[int, str], int]defkeyword_only_x(*,x:int)->int:...defkeyword_only_y(*,y:int)->int:...foo(keyword_only_x,keyword_only_y)# Rejected
The constructors of user-defined classes generic onParamSpecs should beevaluated in the same way.
classY[U,**P]:f:Callable[P,str]prop:Udef__init__(self,f:Callable[P,str],prop:U)->None:self.f=fself.prop=propdefa(q:int)->str:...Y(a,1)# Should resolve to Y[int, (q: int)]Y(a,1).f# Should resolve to (q: int) -> str
The semantics ofConcatenate[X,Y,P] are that it represents the parametersrepresented byP with two positional-only parameters prepended. This meansthat we can use it to represent higher order functions that add, remove ortransform a finite number of parameters of a callable.
defbar(x:int,*args:bool)->int:...defadd[**P](x:Callable[P,int])->Callable[Concatenate[str,P],bool]:...add(bar)# Should return (a: str, /, x: int, *args: bool) -> booldefremove[**P](x:Callable[Concatenate[int,P],int])->Callable[P,bool]:...remove(bar)# Should return (*args: bool) -> booldeftransform[**P](x:Callable[Concatenate[int,P],int])->Callable[Concatenate[str,P],bool]:...transform(bar)# Should return (a: str, /, *args: bool) -> bool
This also means that while any function that returns anR can satisfytyping.Callable[P,R], only functions that can be called positionally intheir first position with aX can satisfytyping.Callable[Concatenate[X,P],R].
defexpects_int_first[**P](x:Callable[Concatenate[int,P],int])->None:...@expects_int_first# Rejecteddefone(x:str)->int:...@expects_int_first# Rejecteddeftwo(*,x:int)->int:...@expects_int_first# Rejecteddefthree(**kwargs:int)->int:...@expects_int_first# Accepteddeffour(*args:int)->int:...
There are still some classes of decorators still not supported with thesefeatures:
those that add/remove/change avariable number of parameters (forexample,
functools.partialremains untypable even usingParamSpec)those that add/remove/change keyword-only parameters.
The components of aParamSpec¶
AParamSpec captures both positional and keyword accessibleparameters, but there unfortunately is no object in the runtime that capturesboth of these together. Instead, we are forced to separate them into*argsand**kwargs, respectively. This means we need to be able to split aparta singleParamSpec into these two components, and then bringthem back together into a call. To do this, we introduceP.args torepresent the tuple of positional arguments in a given call andP.kwargs to represent the correspondingMapping of keywords tovalues.
Valid use locations¶
These “properties” can only be used as the annotated types for*args and**kwargs, accessed from a ParamSpec already in scope.
defp_in_scope[**P](f:Callable[P,int])->None:definner(*args:P.args,**kwargs:P.kwargs)->None:# Acceptedpassdefmixed_up(*args:P.kwargs,**kwargs:P.args)->None:# Rejectedpassdefmisplaced(x:P.args)->None:# Rejectedpassdefp_not_in_scope(*args:P.args,**kwargs:P.kwargs)->None:# Rejectedpass
Furthermore, because the default kind of parameter in Python ((x:int))may be addressed both positionally and through its name, two valid invocationsof a(*args:P.args,**kwargs:P.kwargs) function may give differentpartitions of the same set of parameters. Therefore, we need to make sure thatthese special types are only brought into the world together, and are usedtogether, so that our usage is valid for all possible partitions.
defp_in_scope[**P](f:Callable[P,int])->None:stored_args:P.args# Rejectedstored_kwargs:P.kwargs# Rejecteddefjust_args(*args:P.args)->None:# Rejectedpassdefjust_kwargs(**kwargs:P.kwargs)->None:# Rejectedpass
Semantics¶
With those requirements met, we can now take advantage of the unique propertiesafforded to us by this set up:
Inside the function,
argshas the typeP.args, nottuple[P.args,...]as would be with a normal annotation(and likewise with the**kwargs)This special case is necessary to encapsulate the heterogeneous contentsof the
args/kwargsof a given call, which cannot be expressedby an indefinite tuple/dictionary type.
A function of type
Callable[P,R]can be called with(*args,**kwargs)if and only ifargshas the typeP.argsandkwargshas the typeP.kwargs, and that those types both originated from the same functiondeclaration.A function declared as
definner(*args:P.args,**kwargs:P.kwargs)->Xhas typeCallable[P,X].
With these three properties, we now have the ability to fully type checkparameter preserving decorators.
defdecorator[**P](f:Callable[P,int])->Callable[P,None]:deffoo(*args:P.args,**kwargs:P.kwargs)->None:f(*args,**kwargs)# Accepted, should resolve to intf(*kwargs,**args)# Rejectedf(1,*args,**kwargs)# Rejectedreturnfoo# Accepted
To extend this to includeConcatenate, we declare the following properties:
A function of type
Callable[Concatenate[A,B,P],R]can only becalled with(a,b,*args,**kwargs)whenargsandkwargsare therespective components ofP,ais of typeAandbis oftypeB.A function declared as
definner(a:A,b:B,*args:P.args,**kwargs:P.kwargs)->Rhas typeCallable[Concatenate[A,B,P],R]. Placing keyword-onlyparameters between the*argsand**kwargsis forbidden.
defadd[**P](f:Callable[P,int])->Callable[Concatenate[str,P],None]:deffoo(s:str,*args:P.args,**kwargs:P.kwargs)->None:# Acceptedpassdefbar(*args:P.args,s:str,**kwargs:P.kwargs)->None:# Rejectedpassreturnfoo# Accepteddefremove[**P](f:Callable[Concatenate[int,P],int])->Callable[P,None]:deffoo(*args:P.args,**kwargs:P.kwargs)->None:f(1,*args,**kwargs)# Acceptedf(*args,1,**kwargs)# Rejectedf(*args,**kwargs)# Rejectedreturnfoo
Note that the names of the parameters preceding theParamSpeccomponents are not mentioned in the resultingConcatenate. This means thatthese parameters can not be addressed via a named argument:
defouter[**P](f:Callable[P,None])->Callable[P,None]:deffoo(x:int,*args:P.args,**kwargs:P.kwargs)->None:f(*args,**kwargs)defbar(*args:P.args,**kwargs:P.kwargs)->None:foo(1,*args,**kwargs)# Acceptedfoo(x=1,*args,**kwargs)# Rejectedreturnbar
This is not an implementation convenience, but a soundness requirement. If wewere to allow that second calling style, then the following snippet would beproblematic.
@outerdefproblem(*,x:object)->None:passproblem(x="uh-oh")
Inside ofbar, we would getTypeError:foo()gotmultiplevaluesforargument'x'. Requiring theseconcatenated arguments to be addressed positionally avoids this kind of problem,and simplifies the syntax for spelling these types. Note that this also why wehave to reject signatures of the form(*args:P.args,s:str,**kwargs:P.kwargs).
If one of these prepended positional parameters contains a freeParamSpec,we consider that variable in scope for the purposes of extracting the componentsof thatParamSpec. That allows us to spell things like this:
deftwice[**P](f:Callable[P,int],*args:P.args,**kwargs:P.kwargs)->int:returnf(*args,**kwargs)+f(*args,**kwargs)
The type oftwice in the above example isCallable[Concatenate[Callable[P,int],P],int], whereP is bound by theouterCallable. This has the following semantics:
defa_int_b_str(a:int,b:str)->int:returnatwice(a_int_b_str,1,"A")# Acceptedtwice(a_int_b_str,b="A",a=1)# Acceptedtwice(a_int_b_str,"A",1)# Rejected
TypeVarTuple¶
(Originally specified inPEP 646.)
ATypeVarTuple serves as a placeholder not for a single typebut for atuple of types.
In addition, we introduce a new use for the star operator: to ‘unpack’TypeVarTuple instances and tuple types such astuple[int,str]. Unpacking aTypeVarTuple or tuple type is the typingequivalent of unpacking a variable or a tuple of values.
Type Variable Tuples¶
In the same way that a normal type variable is a stand-in for a singletype such asint, a type variabletuple is a stand-in for atuple type such astuple[int,str].
In Python 3.12 and newer, type variable tuples can be introduced inline by prefixingtheir name with* inside a type parameter list.
classArray[*Ts]:...defpack[*Ts](*values:*Ts)->tuple[*Ts]:...
Prior to 3.12, theTypeVarTuple constructor can be used.
fromtypingimportTypeVarTupleTs=TypeVarTuple('Ts')classArray(Generic[*Ts]):...
Using Type Variable Tuples in Generic Classes¶
Type variable tuples behave like a number of individual type variables packed in atuple. To understand this, consider the following example:
fromtypingimportNewTypeclassArray[*Shape]:...Height=NewType('Height',int)Width=NewType('Width',int)x:Array[Height,Width]=Array()
TheShape type variable tuple here behaves liketuple[T1,T2],whereT1 andT2 are type variables. To use these type variablesas type parameters ofArray, we mustunpack the type variable tuple usingthe star operator:*Shape. The signature ofArray then behavesas if we had simply writtenclassArray[T1,T2]:....
In contrast toclassArray[T1,T2], however,classArray[*Shape] allowsus to parameterize the class with anarbitrary number of type parameters.That is, in addition to being able to define rank-2 arrays such asArray[Height,Width], we could also define rank-3 arrays, rank-4 arrays,and so on:
Time=NewType('Time',int)Batch=NewType('Batch',int)y:Array[Batch,Height,Width]=Array()z:Array[Time,Batch,Height,Width]=Array()
Using Type Variable Tuples in Functions¶
Type variable tuples can be used anywhere a normalTypeVar can.This includes class definitions, as shown above, as well as functionsignatures and variable annotations:
classArray[*Shape]:def__init__(self,shape:tuple[*Shape]):self._shape:tuple[*Shape]=shapedefget_shape(self)->tuple[*Shape]:returnself._shapeshape=(Height(480),Width(640))x:Array[Height,Width]=Array(shape)y=abs(x)# Inferred type is Array[Height, Width]z=x+x# ... is Array[Height, Width]
Type Variable Tuples Must Always be Unpacked¶
Note that in the previous example, theshape argument to__init__was annotated astuple[*Shape]. Why is this necessary - ifShapebehaves liketuple[T1,T2,...], couldn’t we have annotated theshapeargument asShape directly?
This is, in fact, deliberately not possible: type variable tuples mustalways be used unpacked (that is, prefixed by the star operator). This isfor two reasons:
To avoid potential confusion about whether to use a type variable tuplein a packed or unpacked form (“Hmm, should I write ‘
->Shape’,or ‘->tuple[Shape]’, or ‘->tuple[*Shape]’…?”)To improve readability: the star also functions as an explicit visualindicator that the type variable tuple is not a normal type variable.
Variance, Type Constraints and Type Bounds: Not (Yet) Supported¶
TypeVarTuple does not yet support specification of:
Variance (e.g.
TypeVar('T',covariant=True))Type constraints (
TypeVar('T',int,float))Type bounds (
TypeVar('T',bound=ParentClass))
We leave the decision of how these arguments should behave to a future PEP, when variadic generics have been tested in the field. As of PEP 646, type variable tuples areinvariant.
Type Variable Tuple Equality¶
If the sameTypeVarTuple instance is used in multiple places in a signatureor class, a valid type inference might be to bind theTypeVarTuple toatuple of a union of types:
deffoo(arg1:tuple[*Ts],arg2:tuple[*Ts]):...a=(0,)b=('0',)foo(a,b)# Can Ts be bound to tuple[int | str]?
We donot allow this; type unions maynot appear within thetuple.If a type variable tuple appears in multiple places in a signature,the types must match exactly (the list of type parameters must be the samelength, and the type parameters themselves must be identical):
defpointwise_multiply(x:Array[*Shape],y:Array[*Shape])->Array[*Shape]:...x:Array[Height]y:Array[Width]z:Array[Height,Width]pointwise_multiply(x,x)# Validpointwise_multiply(x,y)# Errorpointwise_multiply(x,z)# Error
Multiple Type Variable Tuples: Not Allowed¶
Only a single type variable tuple may appear in a type parameter list:
classArray[*Ts1,*Ts2]:...# Error
The reason is that multiple type variable tuples make it ambiguouswhich parameters get bound to which type variable tuple:
x:Array[int,str,bool]# Ts1 = ???, Ts2 = ???
Type Concatenation¶
Type variable tuples don’t have to be alone; normal types can beprefixed and/or suffixed:
Batch=NewType('Batch',int)Channels=NewType('Channels',int)defadd_batch_axis[*Shape](x:Array[*Shape])->Array[Batch,*Shape]:...defdel_batch_axis[*Shape](x:Array[Batch,*Shape])->Array[*Shape]:...defadd_batch_channels[*Shape](x:Array[*Shape])->Array[Batch,*Shape,Channels]:...a:Array[Height,Width]b=add_batch_axis(a)# Inferred type is Array[Batch, Height, Width]c=del_batch_axis(b)# Array[Height, Width]d=add_batch_channels(a)# Array[Batch, Height, Width, Channels]
NormalTypeVar instances can also be prefixed and/or suffixed:
defprefix_tuple[T,*Ts](x:T,y:tuple[*Ts])->tuple[T,*Ts]:...z=prefix_tuple(x=0,y=(True,'a'))# Inferred type of z is tuple[int, bool, str]
Unpacking Tuple Types¶
We mentioned that aTypeVarTuple stands for a tuple of types.Since we can unpack aTypeVarTuple, for consistency, we alsoallow unpacking a tuple type. As we shall see, this also enables anumber of interesting features.
Unpacking Unbounded Tuple Types¶
Unpacking unbounded tuples is useful in function signatures wherewe don’t care about the exact elements and don’t want to define anunnecessaryTypeVarTuple:
defprocess_batch_channels(x:Array[Batch,*tuple[Any,...],Channels])->None:...x:Array[Batch,Height,Width,Channels]process_batch_channels(x)# OKy:Array[Batch,Channels]process_batch_channels(y)# OKz:Array[Batch]process_batch_channels(z)# Error: Expected Channels.
We can also pass a*tuple[Any,...] wherever a*Ts isexpected. This is useful when we have particularly dynamic code andcannot state the precise number of dimensions or the precise types foreach of the dimensions. In those cases, we can smoothly fall back toan unbounded tuple:
y:Array[*tuple[Any,...]]=read_from_file()defexpect_variadic_array(x:Array[Batch,*Shape])->None:...expect_variadic_array(y)# OKdefexpect_precise_array(x:Array[Batch,Height,Width,Channels])->None:...expect_precise_array(y)# OK
Array[*tuple[Any,...]] stands for an array with an arbitrarynumber of dimensions of typeAny. This means that, in the call toexpect_variadic_array,Batch is bound toAny andShapeis bound totuple[Any,...]. In the call toexpect_precise_array, the variablesBatch,Height,Width, andChannels are all bound toAny.
This allows users to handle dynamic code gracefully while stillexplicitly marking the code as unsafe (by usingy:Array[*tuple[Any,...]]). Otherwise, users would face noisy errors from the typechecker every time they tried to use the variabley, which wouldhinder them when migrating a legacy code base to useTypeVarTuple.
*args as a Type Variable Tuple¶
this specification states that when atype annotation is provided for*args, every argumentmust be of the type annotated. That is, if we specify*args to be typeint,thenall arguments must be of typeint. This limits our ability to specifythe type signatures of functions that take heterogeneous argument types.
If*args is annotated as a type variable tuple, however, the types of theindividual arguments become the types in the type variable tuple:
Ts=TypeVarTuple('Ts')defargs_to_tuple(*args:*Ts)->tuple[*Ts]:...args_to_tuple(1,'a')# Inferred type is tuple[int, str]
In the above example,Ts is bound totuple[int,str]. If noarguments are passed, the type variable tuple behaves like an emptytuple,tuple[()].
As usual, we can unpack any tuple types. For example, by using a typevariable tuple inside a tuple of other types, we can refer to prefixesor suffixes of the variadic argument list. For example:
# os.execle takes arguments 'path, arg0, arg1, ..., env'defexecle(path:str,*args:*tuple[*Ts,Env])->None:...
Note that this is different to
defexecle(path:str,*args:*Ts,env:Env)->None:...
as this would makeenv a keyword-only argument.
Using an unpacked unbounded tuple is equivalent to thebehaviorof*args:int, which accepts zero ormore values of typeint:
deffoo(*args:*tuple[int,...])->None:...# equivalent to:deffoo(*args:int)->None:...
Unpacking tuple types also allows more precise types for heterogeneous*args. The following function expects anint at the beginning,zero or morestr values, and astr at the end:
deffoo(*args:*tuple[int,*tuple[str,...],str])->None:...
For completeness, we mention that unpacking a concrete tuple allows usto specify*args of a fixed number of heterogeneous types:
deffoo(*args:*tuple[int,str])->None:...foo(1,"hello")# OK
Note that, in keeping with the rule that type variable tuples must alwaysbe used unpacked, annotating*args as being a plain type variable tupleinstance isnot allowed:
deffoo(*args:Ts):...# NOT valid
*args is the only case where an argument can be annotated as*Ts directly;other arguments should use*Ts to parameterize something else, e.g.tuple[*Ts].If*args itself is annotated astuple[*Ts], the old behavior still applies:all arguments must be atuple parameterized with the same types.
deffoo(*args:tuple[*Ts]):...foo((0,),(1,))# Validfoo((0,),(1,2))# Errorfoo((0,),('1',))# Error
Finally, note that a type variable tuple maynot be used as the type of**kwargs. (We do not yet know of a use case for this feature, so we preferto leave the ground fresh for a potential future PEP.)
# NOT validdeffoo(**kwargs:*Ts):...
Type Variable Tuples withCallable¶
Type variable tuples can also be used in the arguments section of aCallable:
classProcess:def__init__(self,target:Callable[[*Ts],None],args:tuple[*Ts],)->None:...deffunc(arg1:int,arg2:str)->None:...Process(target=func,args=(0,'foo'))# ValidProcess(target=func,args=('foo',0))# Error
Other types and normal type variables can also be prefixed/suffixedto the type variable tuple:
T=TypeVar('T')deffoo(f:Callable[[int,*Ts,T],tuple[T,*Ts]]):...
The behavior of a Callable containing an unpacked item, whether theitem is aTypeVarTuple or a tuple type, is to treat the elementsas if they were the type for*args. So,Callable[[*Ts],None]is treated as the type of the function:
deffoo(*args:*Ts)->None:...
Callable[[int,*Ts,T],tuple[T,*Ts]] is treated as the type ofthe function:
deffoo(*args:*tuple[int,*Ts,T])->tuple[T,*Ts]:...
Behavior when Type Parameters are not Specified¶
When a generic class parameterized by a type variable tuple is used withoutany type parameters and the TypeVarTuple has no default value, it behaves asif the type variable tuple was substituted withtuple[Any,...]:
deftakes_any_array(arr:Array):...# equivalent to:deftakes_any_array(arr:Array[*tuple[Any,...]]):...x:Array[Height,Width]takes_any_array(x)# Validy:Array[Time,Height,Width]takes_any_array(y)# Also valid
This enables gradual typing: existing functions accepting, for example,a plain TensorFlowTensor will still be valid even ifTensor is madegeneric and calling code passes aTensor[Height,Width].
This also works in the opposite direction:
deftakes_specific_array(arr:Array[Height,Width]):...z:Array# equivalent to Array[*tuple[Any, ...]]takes_specific_array(z)
(For details, see the section onUnpacking Unbounded Tuple Types.)
This way, even if libraries are updated to use types likeArray[Height,Width],users of those libraries won’t be forced to also apply type annotations toall of their code; users still have a choice about what parts of their codeto type and which parts to not.
Aliases¶
Generic aliases can be created using a type variable tuple ina similar way to regular type variables:
typeIntTuple[*Ts]=tuple[int,*Ts]typeNamedArray[*Ts]=tuple[str,Array[*Ts]]IntTuple[float,bool]# Equivalent to tuple[int, float, bool]NamedArray[Height]# Equivalent to tuple[str, Array[Height]]
As this example shows, all type parameters passed to the alias arebound to the type variable tuple.
This allows us to define convenience aliases for arrays of a fixed shapeor datatype:
classArray[DType,*Shape]:...# E.g. Float32Array[Height, Width, Channels]typeFloat32Array[*Shape]=Array[np.float32,*Shape]# E.g. Array1D[np.uint8]typeArray1D[DType]=Array[DType,Any]
If an explicitly empty type parameter list is given, the type variabletuple in the alias is set empty:
IntTuple[()]# Equivalent to tuple[int]NamedArray[()]# Equivalent to tuple[str, Array[()]]
If the type parameter list is omitted entirely, the unspecified typevariable tuples are treated astuple[Any,...] (similar toBehavior when Type Parameters are not Specified):
deftakes_float_array_of_any_shape(x:Float32Array):...x:Float32Array[Height,Width]=Array()takes_float_array_of_any_shape(x)# Validdeftakes_float_array_with_specific_shape(y:Float32Array[Height,Width]):...y:Float32Array=Array()takes_float_array_with_specific_shape(y)# Valid
NormalTypeVar instances can also be used in such aliases:
typeFoo[T,*Ts]=tuple[T,*Ts]# T bound to str, Ts to tuple[int]Foo[str,int]# T bound to float, Ts to tuple[()]Foo[float]# T bound to Any, Ts to an tuple[Any, ...]Foo
Substitution in Aliases¶
In the previous section, we only discussed simple usage of generic aliasesin which the type arguments were just simple types. However, a number ofmore exotic constructions are also possible.
Type Arguments can be Variadic¶
First, type arguments to generic aliases can be variadic. For example, aTypeVarTuple can be used as a type argument:
typeIntTuple[*Ts1]=tuple[int,*Ts1]typeIntFloatTuple[*Ts2]=IntTuple[float,*Ts2]# Valid
Here,*Ts1 in theIntTuple alias is bound totuple[float,*Ts2],resulting in an aliasIntFloatTuple equivalent totuple[int,float,*Ts2].
Unpacked arbitrary-length tuples can also be used as type arguments, withsimilar effects:
IntFloatsTuple=IntTuple[*tuple[float,...]]# Valid
Here,*Ts1 is bound to*tuple[float,...], resulting inIntFloatsTuple being equivalent totuple[int,*tuple[float,...]]: a tupleconsisting of anint then zero or morefloats.
Variadic Arguments Require Variadic Aliases¶
Variadic type arguments can only be used with generic aliases that arethemselves variadic. For example:
typeIntTuple[T]=tuple[int,T]IntTuple[str]# ValidIntTuple[*Ts]# NOT validIntTuple[*tuple[float,...]]# NOT valid
Here,IntTuple is anon-variadic generic alias that takes exactly onetype argument. Hence, it cannot accept*Ts or*tuple[float,...] as typearguments, because they represent an arbitrary number of types.
Aliases with Both TypeVars and TypeVarTuples¶
InAliases, we briefly mentioned that aliases can be generic in bothTypeVars andTypeVarTuples:
T=TypeVar('T')Foo=tuple[T,*Ts]Foo[str,int]# T bound to str, Ts to tuple[int]Foo[str,int,float]# T bound to str, Ts to tuple[int, float]
In accordance withMultiple Type Variable Tuples: Not Allowed, at most oneTypeVarTuple may appear in the type parameters to an alias. However, aTypeVarTuple can be combined with an arbitrary number ofTypeVars,both before and after:
T1=TypeVar('T1')T2=TypeVar('T2')T3=TypeVar('T3')tuple[*Ts,T1,T2]# Validtuple[T1,T2,*Ts]# Validtuple[T1,*Ts,T2,T3]# Valid
In order to substitute these type variables with supplied type arguments,any type variables at the beginning or end of the type parameter list firstconsume type arguments, and then any remaining type arguments are boundto theTypeVarTuple:
Shrubbery=tuple[*Ts,T1,T2]Shrubbery[str,bool]# T2=bool, T1=str, Ts=tuple[()]Shrubbery[str,bool,float]# T2=float, T1=bool, Ts=tuple[str]Shrubbery[str,bool,float,int]# T2=int, T1=float, Ts=tuple[str, bool]Ptang=tuple[T1,*Ts,T2,T3]Ptang[str,bool,float]# T1=str, T3=float, T2=bool, Ts=tuple[()]Ptang[str,bool,float,int]# T1=str, T3=int, T2=float, Ts=tuple[bool]
Note that the minimum number of type arguments in such cases is set bythe number ofTypeVars:
Shrubbery[int]# Not valid; Shrubbery needs at least two type arguments
Splitting Arbitrary-Length Tuples¶
A final complication occurs when an unpacked arbitrary-length tuple is usedas a type argument to an alias consisting of bothTypeVars and aTypeVarTuple:
Elderberries=tuple[*Ts,T1]Hamster=Elderberries[*tuple[int,...]]# valid
In such cases, the arbitrary-length tuple is split between theTypeVarsand theTypeVarTuple. We assume the arbitrary-length tuple containsat least as many items as there areTypeVars, such that individualinstances of the inner type - hereint - are bound to anyTypeVarspresent. The ‘rest’ of the arbitrary-length tuple - here*tuple[int,...],since a tuple of arbitrary length minus two items is still arbitrary-length -is bound to theTypeVarTuple.
Here, therefore,Hamster is equivalent totuple[*tuple[int,...],int]:a tuple consisting of zero or moreints, then a finalint.
Of course, such splitting only occurs if necessary. For example, if we insteaddid:
Elderberries[*tuple[int,...],str]
Then splitting would not occur;T1 would be bound tostr, andTs to*tuple[int,...].
In particularly awkward cases, aTypeVarTuple may consume both a typeand a part of an arbitrary-length tuple type:
Elderberries[str,*tuple[int,...]]
Here,T1 is bound toint, andTs is bound totuple[str,*tuple[int,...]]. This expression is therefore equivalent totuple[str,*tuple[int,...],int]: a tuple consisting of astr, thenzero or moreints, ending with anint.
TypeVarTuples Cannot be Split¶
Finally, although any arbitrary-length tuples in the type argument list can besplit between the type variables and the type variable tuple, the same is nottrue ofTypeVarTuples in the argument list:
typeCamelot[T,*Ts]=tuple[T,*Ts]Camelot[int,*tuple[str,...]]# ValidCamelot[*tuple[str,...]]# NOT valid
This is not possible because, unlike in the case of an unpacked arbitrary-lengthtuple, there is no way to ‘peer inside’ theTypeVarTuple to see what itsindividual types are.
Overloads for Accessing Individual Types¶
For situations where we require access to each individual type in the type variable tuple,overloads can be used with individualTypeVar instances in place of the type variable tuple:
Axis1=TypeVar('Axis1')Axis2=TypeVar('Axis2')Axis3=TypeVar('Axis3')classArray[*Shape]:@overloaddeftranspose(self:Array[Axis1,Axis2])->Array[Axis2,Axis1]:...@overloaddeftranspose(self:Array[Axis1,Axis2,Axis3])->Array[Axis3,Axis2,Axis1]:...
(For array shape operations in particular, having to specifyoverloads for each possible rank is, of course, a rather cumbersomesolution. However, it’s the best we can do without additional typemanipulation mechanisms.)
Defaults for Type Parameters¶
(Originally specified inPEP 696.)
Default values can be provided for a TypeVar, ParamSpec, or TypeVarTuple.
Default Ordering and Subscription Rules¶
The order for defaults should follow the standard function parameterrules, so a type parameter with nodefault cannot follow one withadefault value. Doing so may raise aTypeError at runtime,and a type checker should flag this as an error.
DefaultStrT=TypeVar("DefaultStrT",default=str)DefaultIntT=TypeVar("DefaultIntT",default=int)DefaultBoolT=TypeVar("DefaultBoolT",default=bool)T=TypeVar("T")T2=TypeVar("T2")classNonDefaultFollowsDefault(Generic[DefaultStrT,T]):...# Invalid: non-default TypeVars cannot follow ones with defaultsclassNoNonDefaults(Generic[DefaultStrT,DefaultIntT]):...(NoNonDefaults==NoNonDefaults[str]==NoNonDefaults[str,int])# All validclassOneDefault(Generic[T,DefaultBoolT]):...OneDefault[float]==OneDefault[float,bool]# Validreveal_type(OneDefault)# type is type[OneDefault[T, DefaultBoolT = bool]]reveal_type(OneDefault[float]())# type is OneDefault[float, bool]classAllTheDefaults(Generic[T1,T2,DefaultStrT,DefaultIntT,DefaultBoolT]):...reveal_type(AllTheDefaults)# type is type[AllTheDefaults[T1, T2, DefaultStrT = str, DefaultIntT = int, DefaultBoolT = bool]]reveal_type(AllTheDefaults[int,complex]())# type is AllTheDefaults[int, complex, str, int, bool]AllTheDefaults[int]# Invalid: expected 2 arguments to AllTheDefaults(AllTheDefaults[int,complex]==AllTheDefaults[int,complex,str]==AllTheDefaults[int,complex,str,int]==AllTheDefaults[int,complex,str,int,bool])# All valid
With the new Python 3.12 syntax for generics (introduced byPEP 695), this canbe enforced at compile time:
typeAlias[DefaultT=int,T]=tuple[DefaultT,T]# SyntaxError: non-default TypeVars cannot follow ones with defaultsdefgeneric_func[DefaultT=int,T](x:DefaultT,y:T)->None:...# SyntaxError: non-default TypeVars cannot follow ones with defaultsclassGenericClass[DefaultT=int,T]:...# SyntaxError: non-default TypeVars cannot follow ones with defaults
ParamSpec Defaults¶
ParamSpec defaults are defined using the same syntax asTypeVar s but use alist of types or an ellipsisliteral “...” or another in-scopeParamSpec (seeScoping Rules).
classFoo[**P=[str,int]]:...reveal_type(Foo)# type is type[Foo[str, int]]reveal_type(Foo())# type is Foo[str, int]reveal_type(Foo[[bool,bool]]())# type is Foo[bool, bool]
TypeVarTuple Defaults¶
TypeVarTuple defaults are defined using the same syntax asTypeVar s, but instead of a single type, they use an unpacked tuple oftypes or an unpacked, in-scopeTypeVarTuple (seeScoping Rules).
classFoo[*Ts=*tuple[str,int]]:...reveal_type(Foo)# type is type[Foo[str, int]]reveal_type(Foo())# type is Foo[str, int]reveal_type(Foo[int,bool]())# type is Foo[int, bool]
Using Another Type Parameter asdefault¶
This allows for a value to be used again when the type parameter to ageneric is missing but another type parameter is specified.
To use another type parameter as a default thedefault and thetype parameter must be the same type (aTypeVar’s default must beaTypeVar, etc.).
StartT=TypeVar("StartT",default=int)StopT=TypeVar("StopT",default=StartT)StepT=TypeVar("StepT",default=int|None)classslice(Generic[StartT,StopT,StepT]):...reveal_type(slice)# type is type[slice[StartT = int, StopT = StartT, StepT = int | None]]reveal_type(slice())# type is slice[int, int, int | None]reveal_type(slice[str]())# type is slice[str, str, int | None]reveal_type(slice[str,bool,timedelta]())# type is slice[str, bool, timedelta]T2=TypeVar("T2",default=DefaultStrT)classFoo(Generic[DefaultStrT,T2]):def__init__(self,a:DefaultStrT,b:T2)->None:...reveal_type(Foo(1,""))# type is Foo[int, str]Foo[int](1,"")# Invalid: Foo[int, str] cannot be assigned to self: Foo[int, int] in Foo.__init__Foo[int]("",1)# Invalid: Foo[str, int] cannot be assigned to self: Foo[int, int] in Foo.__init__
When using a type parameter as the default to another type parameter, thefollowing rules apply, whereT1 is the default forT2.
Scoping Rules¶
T1 must be used beforeT2 in the parameter list of the generic.
T2=TypeVar("T2",default=T1)classFoo(Generic[T1,T2]):...# ValidStartT=TypeVar("StartT",default="StopT")# Swapped defaults around from previous exampleStopT=TypeVar("StopT",default=int)classslice(Generic[StartT,StopT,StepT]):...# ^^^^^^ Invalid: ordering does not allow StopT to be bound
Using a type parameter from an outer scope as a default is not supported.
classFoo(Generic[T1]):classBar(Generic[T2]):...# Type Error
Bound Rules¶
T1’s bound must beassignable toT2’s bound.
T1=TypeVar("T1",bound=int)TypeVar("Ok",default=T1,bound=float)# ValidTypeVar("AlsoOk",default=T1,bound=int)# ValidTypeVar("Invalid",default=T1,bound=str)# Invalid: int is not assignable to str
Constraint Rules¶
The constraints ofT2 must be a superset of the constraints ofT1.
T1=TypeVar("T1",bound=int)TypeVar("Invalid",float,str,default=T1)# Invalid: upper bound int is incompatible with constraints float or strT1=TypeVar("T1",int,str)TypeVar("AlsoOk",int,str,bool,default=T1)# ValidTypeVar("AlsoInvalid",bool,complex,default=T1)# Invalid: {bool, complex} is not a superset of {int, str}
Type Parameters as Parameters to Generics¶
Type parameters are valid as parameters to generics inside of adefault when the first parameter is in scope as determined by theprevious section.
T=TypeVar("T")ListDefaultT=TypeVar("ListDefaultT",default=list[T])classBar(Generic[T,ListDefaultT]):def__init__(self,x:T,y:ListDefaultT):...reveal_type(Bar)# type is type[Bar[T, ListDefaultT = list[T]]]reveal_type(Bar[int])# type is type[Bar[int, list[int]]]reveal_type(Bar[int](0,[]))# type is Bar[int, list[int]]reveal_type(Bar[int,list[str]](0,[]))# type is Bar[int, list[str]]reveal_type(Bar[int,str](0,""))# type is Bar[int, str]
Specialization Rules¶
Generic Type Aliases¶
A generic type alias can be further subscripted following normal subscriptionrules. If a type parameter has a default that hasn’t been overridden, it shouldbe treated like it was substituted into the type alias.
classSomethingWithNoDefaults(Generic[T,T2]):...MyAlias:TypeAlias=SomethingWithNoDefaults[int,DefaultStrT]# Validreveal_type(MyAlias)# type is type[SomethingWithNoDefaults[int, DefaultStrT]]reveal_type(MyAlias[bool]())# type is SomethingWithNoDefaults[int, bool]MyAlias[bool,int]# Invalid: too many arguments passed to MyAlias
Subclassing¶
Generic classes with type parameters that have defaults behave similarlyto generic type aliases. That is, subclasses can be further subscripted followingnormal subscription rules, non-overridden defaults should be substituted.
classSubclassMe(Generic[T,DefaultStrT]):x:DefaultStrTclassBar(SubclassMe[int,DefaultStrT]):...reveal_type(Bar)# type is type[Bar[DefaultStrT = str]]reveal_type(Bar())# type is Bar[str]reveal_type(Bar[bool]())# type is Bar[bool]classFoo(SubclassMe[float]):...reveal_type(Foo().x)# type is strFoo[str]# Invalid: Foo cannot be further subscriptedclassBaz(Generic[DefaultIntT,DefaultStrT]):...classSpam(Baz):...reveal_type(Spam())# type is <subclass of Baz[int, str]>
Usingbound anddefault¶
If bothbound anddefault are passed,default must beassignable tobound. If not, the type checker should generate anerror.
TypeVar("Ok",bound=float,default=int)# ValidTypeVar("Invalid",bound=str,default=int)# Invalid: the bound and default are incompatible
Constraints¶
For constrainedTypeVars, the default needs to be one of theconstraints. A type checker should generate an error even if it is asubtype of one of the constraints.
TypeVar("Ok",float,str,default=float)# ValidTypeVar("Invalid",float,str,default=int)# Invalid: expected one of float or str got int
Function Defaults¶
In generic functions, type checkers may use a type parameter’s default when thetype parameter cannot be solved to anything. We leave the semantics of thisusage unspecified, as ensuring thedefault is returned in every code pathwhere the type parameter can go unsolved may be too hard to implement. Typecheckers are free to either disallow this case or experiment with implementingsupport.
T=TypeVar('T',default=int)deffunc(x:int|set[T])->T:...reveal_type(func(0))# a type checker may reveal T's default of int here
Defaults followingTypeVarTuple¶
ATypeVar that immediately follows aTypeVarTuple is not allowedto have a default, because it would be ambiguous whether a type argumentshould be bound to theTypeVarTuple or the defaultedTypeVar.
classFoo[*Ts,T=bool]:...# Type checker error# Could be reasonably interpreted as either Ts = (int, str, float), T = bool# or Ts = (int, str), T = floatFoo[int,str,float]
It is allowed to have aParamSpec with a default following aTypeVarTuple with a default, as there can be no ambiguity between a type argumentfor theParamSpec and one for theTypeVarTuple.
classFoo[*Ts=*tuple[int,str],**P=[float,bool]]:...# ValidFoo[int,str]# Ts = (int, str), P = [float, bool]Foo[int,str,[bytes]]# Ts = (int, str), P = [bytes]
Binding rules¶
Type parameter defaults should be bound by attribute access(including call and subscript).
classFoo[T=int]:defmeth(self)->Self:returnselfreveal_type(Foo.meth)# type is (self: Foo[int]) -> Foo[int]
Self¶
(Originally specified inPEP 673.)
Use in Method Signatures¶
Self used in the signature of a method is treated as if it were aTypeVar bound to the class.
fromtypingimportSelfclassShape:defset_scale(self,scale:float)->Self:self.scale=scalereturnself
is treated equivalently to:
fromtypingimportTypeVarSelfShape=TypeVar("SelfShape",bound="Shape")classShape:defset_scale(self:SelfShape,scale:float)->SelfShape:self.scale=scalereturnself
This works the same for a subclass too:
classCircle(Shape):defset_radius(self,radius:float)->Self:self.radius=radiusreturnself
which is treated equivalently to:
SelfCircle=TypeVar("SelfCircle",bound="Circle")classCircle(Shape):defset_radius(self:SelfCircle,radius:float)->SelfCircle:self.radius=radiusreturnself
One implementation strategy is to simply desugar the former to the latter in apreprocessing step. If a method usesSelf in its signature, the type ofself within a method will beSelf. In other cases, the type ofself will remain the enclosing class.
Use in Classmethod Signatures¶
TheSelf type annotation is also useful for classmethods that returnan instance of the class that they operate on. For example,from_config inthe following snippet builds aShape object from a givenconfig.
classShape:def__init__(self,scale:float)->None:...@classmethoddeffrom_config(cls,config:dict[str,float])->Shape:returncls(config["scale"])
However, this means thatCircle.from_config(...) is inferred to return avalue of typeShape, when in fact it should beCircle:
classCircle(Shape):defcircumference(self)->float:...shape=Shape.from_config({"scale":7.0})# => Shapecircle=Circle.from_config({"scale":7.0})# => *Shape*, not Circlecircle.circumference()# Error: `Shape` has no attribute `circumference`
The current workaround for this is unintuitive and error-prone:
Self=TypeVar("Self",bound="Shape")classShape:@classmethoddeffrom_config(cls:type[Self],config:dict[str,float])->Self:returncls(config["scale"])
Instead,Self can be used directly:
fromtypingimportSelfclassShape:@classmethoddeffrom_config(cls,config:dict[str,float])->Self:returncls(config["scale"])
This avoids the complicatedcls:type[Self] annotation and theTypeVardeclaration with abound. Once again, the latter code behaves equivalentlyto the former code.
Use in Parameter Types¶
Another use forSelf is to annotate parameters that expect instances ofthe current class:
Self=TypeVar("Self",bound="Shape")classShape:defdifference(self:Self,other:Self)->float:...defapply(self:Self,f:Callable[[Self],None])->None:...
Self can be used directly to achieve the same behavior:
fromtypingimportSelfclassShape:defdifference(self,other:Self)->float:...defapply(self,f:Callable[[Self],None])->None:...
Note that specifyingself:Self is harmless, so some users may find itmore readable to write the above as:
classShape:defdifference(self:Self,other:Self)->float:...
Use in Attribute Annotations¶
Another use forSelf is to annotate attributes. One example is where wehave aLinkedList whose elements must beassignable to the currentclass.
fromdataclassesimportdataclassfromtypingimportGeneric,TypeVarT=TypeVar("T")@dataclassclassLinkedList(Generic[T]):value:Tnext:LinkedList[T]|None=None# OKLinkedList[int](value=1,next=LinkedList[int](value=2))# Not OKLinkedList[int](value=1,next=LinkedList[str](value="hello"))
However, annotating thenext attribute asLinkedList[T] allows invalidconstructions with subclasses:
@dataclassclassOrdinalLinkedList(LinkedList[int]):defordinal_value(self)->str:returnas_ordinal(self.value)# Should not be OK because LinkedList[int] is not assignable to# OrdinalLinkedList, but the type checker allows it.xs=OrdinalLinkedList(value=1,next=LinkedList[int](value=2))ifxs.next:print(xs.next.ordinal_value())# Runtime Error.
This constraint can be expressed usingnext:Self|None:
fromtypingimportSelf@dataclassclassLinkedList(Generic[T]):value:Tnext:Self|None=None@dataclassclassOrdinalLinkedList(LinkedList[int]):defordinal_value(self)->str:returnas_ordinal(self.value)xs=OrdinalLinkedList(value=1,next=LinkedList[int](value=2))# Type error: Expected OrdinalLinkedList, got LinkedList[int].ifxs.nextisnotNone:xs.next=OrdinalLinkedList(value=3,next=None)# OKxs.next=LinkedList[int](value=3,next=None)# Not OK
The code above is semantically equivalent to treating each attributecontaining aSelf type as aproperty that returns that type:
fromdataclassesimportdataclassfromtypingimportAny,Generic,TypeVarT=TypeVar("T")Self=TypeVar("Self",bound="LinkedList")classLinkedList(Generic[T]):value:T@propertydefnext(self:Self)->Self|None:returnself._next@next.setterdefnext(self:Self,next:Self|None)->None:self._next=nextclassOrdinalLinkedList(LinkedList[int]):defordinal_value(self)->str:returnstr(self.value)
Use in Generic Classes¶
Self can also be used in generic class methods:
classContainer(Generic[T]):value:Tdefset_value(self,value:T)->Self:...
This is equivalent to writing:
Self=TypeVar("Self",bound="Container[Any]")classContainer(Generic[T]):value:Tdefset_value(self:Self,value:T)->Self:...
The behavior is to preserve the type argument of the object on which themethod was called. When called on an object with concrete typeContainer[int],Self is bound toContainer[int]. When called withan object of generic typeContainer[T],Self is bound toContainer[T]:
defobject_with_concrete_type()->None:int_container:Container[int]str_container:Container[str]reveal_type(int_container.set_value(42))# => Container[int]reveal_type(str_container.set_value("hello"))# => Container[str]defobject_with_generic_type(container:Container[T],value:T,)->Container[T]:returncontainer.set_value(value)# => Container[T]
The PEP doesn’t specify the exact type ofself.value within the methodset_value. Some type checkers may choose to implementSelf types usingclass-local type variables withSelf=TypeVar(“Self”,bound=Container[T]), which will infer a precise typeT. However, giventhat class-local type variables are not a standardized type system feature, itis also acceptable to inferAny forself.value. We leave this up tothe type checker.
Note that we reject usingSelf with type arguments, such asSelf[int].This is because it creates ambiguity about the type of theself parameterand introduces unnecessary complexity:
classContainer(Generic[T]):deffoo(self,other:Self[int],other2:Self,)->Self[str]:# Rejected...
In such cases, we recommend using an explicit type forself:
classContainer(Generic[T]):deffoo(self:Container[T],other:Container[int],other2:Container[T])->Container[str]:...
Use in Protocols¶
Self is valid within Protocols, similar to its use in classes:
fromtypingimportProtocol,SelfclassShapeProtocol(Protocol):scale:floatdefset_scale(self,scale:float)->Self:self.scale=scalereturnself
is treated equivalently to:
fromtypingimportTypeVarSelfShape=TypeVar("SelfShape",bound="ShapeProtocol")classShapeProtocol(Protocol):scale:floatdefset_scale(self:SelfShape,scale:float)->SelfShape:self.scale=scalereturnself
SeePEP 544 fordetails on the behavior of TypeVars bound to protocols.
Checking a class for assignability to a protocol: If a protocol usesSelfin methods or attribute annotations, then a classFoo isassignableto the protocol if its corresponding methods and attribute annotations useeitherSelf orFoo or any ofFoo’s subclasses. See the examplesbelow:
fromtypingimportProtocolclassShapeProtocol(Protocol):defset_scale(self,scale:float)->Self:...classReturnSelf:scale:float=1.0defset_scale(self,scale:float)->Self:self.scale=scalereturnselfclassReturnConcreteShape:scale:float=1.0defset_scale(self,scale:float)->ReturnConcreteShape:self.scale=scalereturnselfclassBadReturnType:scale:float=1.0defset_scale(self,scale:float)->int:self.scale=scalereturn42classReturnDifferentClass:scale:float=1.0defset_scale(self,scale:float)->ReturnConcreteShape:returnReturnConcreteShape(...)defaccepts_shape(shape:ShapeProtocol)->None:y=shape.set_scale(0.5)reveal_type(y)defmain()->None:return_self_shape:ReturnSelfreturn_concrete_shape:ReturnConcreteShapebad_return_type:BadReturnTypereturn_different_class:ReturnDifferentClassaccepts_shape(return_self_shape)# OKaccepts_shape(return_concrete_shape)# OKaccepts_shape(bad_return_type)# Not OK# Not OK because it returns a non-subclass.accepts_shape(return_different_class)
Valid Locations forSelf¶
ASelf annotation is only valid in class contexts, and will always referto the encapsulating class. In contexts involving nested classes,Selfwill always refer to the innermost class.
The following uses ofSelf are accepted:
classReturnsSelf:deffoo(self)->Self:...# Accepted@classmethoddefbar(cls)->Self:# Acceptedreturncls()def__new__(cls,value:int)->Self:...# Accepteddefexplicitly_use_self(self:Self)->Self:...# Accepted# Accepted (Self can be nested within other types)defreturns_list(self)->list[Self]:...# Accepted (Self can be nested within other types)@classmethoddefreturn_cls(cls)->type[Self]:returnclsclassChild(ReturnsSelf):# Accepted (we can override a method that uses Self annotations)deffoo(self)->Self:...classTakesSelf:deffoo(self,other:Self)->bool:...# AcceptedclassRecursive:# Accepted (treated as an @property returning ``Self | None``)next:Self|NoneclassCallableAttribute:deffoo(self)->int:...# Accepted (treated as an @property returning the Callable type)bar:Callable[[Self],int]=fooclassHasNestedFunction:x:int=42deffoo(self)->None:# Accepted (Self is bound to HasNestedFunction).defnested(z:int,inner_self:Self)->Self:print(z)print(inner_self.x)returninner_selfnested(42,self)# OKclassOuter:classInner:deffoo(self)->Self:...# Accepted (Self is bound to Inner)
The following uses ofSelf are rejected.
deffoo(bar:Self)->Self:...# Rejected (not within a class)bar:Self# Rejected (not within a class)classFoo:# Rejected (Self is treated as unknown).defhas_existing_self_annotation(self:T)->Self:...classFoo:defreturn_concrete_type(self)->Self:returnFoo()# Rejected (see FooChild below for rationale)classFooChild(Foo):child_value:int=42defchild_method(self)->None:# At runtime, this would be Foo, not FooChild.y=self.return_concrete_type()y.child_value# Runtime error: Foo has no attribute child_valueclassBar(Generic[T]):defbar(self)->T:...classBaz(Bar[Self]):...# Rejected
We reject type aliases containingSelf. SupportingSelfoutside class definitions can require a lot of special-handling intype checkers. Given that it also goes against the rest of the PEP touseSelf outside a class definition, we believe the addedconvenience of aliases is not worth it:
TupleSelf=Tuple[Self,Self]# RejectedclassAlias:defreturn_tuple(self)->TupleSelf:# Rejectedreturn(self,self)
Note that we rejectSelf in staticmethods.Self does not add muchvalue since there is noself orcls to return. The only possible usecases would be to return a parameter itself or some element from a containerpassed in as a parameter. These don’t seem worth the additional complexity.
classBase:@staticmethoddefmake()->Self:# Rejected...@staticmethoddefreturn_parameter(foo:Self)->Self:# Rejected...
Likewise, we rejectSelf in metaclasses.Self consistently refers to thesame type (that ofself). But in metaclasses, it would have to refer todifferent types in different method signatures. For example, in__mul__,Self in the return type would refer to the implementing classFoo, not the enclosing classMyMetaclass. But, in__new__,Selfin the return type would refer to the enclosing classMyMetaclass. Toavoid confusion, we reject this edge case.
classMyMetaclass(type):def__new__(cls,*args:Any)->Self:# Rejectedreturnsuper().__new__(cls,*args)def__mul__(cls,count:int)->list[Self]:# Rejectedreturn[cls()]*countclassFoo(metaclass=MyMetaclass):...
Variance Inference¶
(Originally specified byPEP 695.)
The introduction of explicit syntax for generic classes in Python 3.12eliminates the need for variance to be specified for typeparameters. Instead, type checkers will infer the variance of type parametersbased on their usage within a class. Type parameters are inferred to beinvariant, covariant, or contravariant depending on how they are used.
Python type checkers already include the ability to determine the variance oftype parameters for the purpose of validating variance within a genericprotocol class. This capability can be used for all classes (whether or notthey are protocols) to calculate the variance of each type parameter.
The algorithm for computing the variance of a type parameter is as follows.
For each type parameter in a generic class:
1. If the type parameter is variadic (TypeVarTuple) or a parameterspecification (ParamSpec), it is always considered invariant. No furtherinference is needed.
2. If the type parameter comes from a traditionalTypeVar declaration andis not specified asinfer_variance (see below), its variance is specifiedby theTypeVar constructor call. No further inference is needed.
3. Create two specialized versions of the class. We’ll refer to these asupper andlower specializations. In both of these specializations,replace all type parameters other than the one being inferred by a dummy typeinstance (a concrete anonymous class that is assumed to meet the bounds orconstraints of the type parameter). In theupper specialized class,specialize the target type parameter with anobject instance. Thisspecialization ignores the type parameter’s upper bound or constraints. In thelower specialized class, specialize the target type parameter with itself(i.e. the corresponding type argument is the type parameter itself).
4. Determine whetherlower can be assigned toupper using normalassignability rules. If so, the target type parameter is covariant. If not,determine whetherupper can be assigned tolower. If so, the targettype parameter is contravariant. If neither of these combinations areassignable, the target type parameter is invariant.
Here is an example.
classClassA[T1,T2,T3](list[T1]):defmethod1(self,a:T2)->None:...defmethod2(self)->T3:...
To determine the variance ofT1, we specializeClassA as follows:
upper=ClassA[object,Dummy,Dummy]lower=ClassA[T1,Dummy,Dummy]
We find thatupper is not assignable tolower. Likewise,lower isnot assignable toupper, so we conclude thatT1 is invariant.
To determine the variance ofT2, we specializeClassA as follows:
upper=ClassA[Dummy,object,Dummy]lower=ClassA[Dummy,T2,Dummy]
Sinceupper is assignable tolower,T2 is contravariant.
To determine the variance ofT3, we specializeClassA as follows:
upper=ClassA[Dummy,Dummy,object]lower=ClassA[Dummy,Dummy,T3]
Sincelower is assignable toupper,T3 is covariant.
Auto Variance For TypeVar¶
The existingTypeVar class constructor accepts keyword parameters namedcovariant andcontravariant. If both of these areFalse, thetype variable is assumed to be invariant. PEP 695 adds another keywordparameter namedinfer_variance indicating that a type checker should useinference to determine whether the type variable is invariant, covariant orcontravariant. A corresponding instance variable__infer_variance__ can beaccessed at runtime to determine whether the variance is inferred. Typevariables that are implicitly allocated using the new syntax will alwayshave__infer_variance__ set toTrue.
A generic class that uses the traditional syntax may include combinations oftype variables with explicit and inferred variance.
T1=TypeVar("T1",infer_variance=True)# Inferred varianceT2=TypeVar("T2")# InvariantT3=TypeVar("T3",covariant=True)# Covariant# A type checker should infer the variance for T1 but use the# specified variance for T2 and T3.classClassA(Generic[T1,T2,T3]):...
Compatibility with Traditional TypeVars¶
The existing mechanism for allocatingTypeVar,TypeVarTuple, andParamSpec is retained for backward compatibility. However, these“traditional” type variables should not be combined with type parametersallocated using the new syntax. Such a combination should be flagged asan error by type checkers. This is necessary because the type parameterorder is ambiguous.
It is OK to combine traditional type variables with new-style type parametersif the class, function, or type alias does not use the new syntax. Thenew-style type parameters must come from an outer scope in this case.
K=TypeVar("K")classClassA[V](dict[K,V]):...# Type checker errorclassClassB[K,V](dict[K,V]):...# OKclassClassC[V]:# The use of K and V for "method1" is OK because it uses the# "traditional" generic function mechanism where type parameters# are implicit. In this case V comes from an outer scope (ClassC)# and K is introduced implicitly as a type parameter for "method1".defmethod1(self,a:V,b:K)->V|K:...# The use of M and K are not allowed for "method2". A type checker# should generate an error in this case because this method uses the# new syntax for type parameters, and all type parameters associated# with the method must be explicitly declared. In this case, ``K``# is not declared by "method2", nor is it supplied by a new-style# type parameter defined in an outer scope.defmethod2[M](self,a:M,b:K)->M|K:...