Important
This PEP is a historical document. The up-to-date, canonical documentation can now be found atTypeVarTuple andtyping.TypeVarTuple.
×
SeePEP 1 for how to propose changes.
PEP 484 introducedTypeVar, enabling creation of generics parameterisedwith a single type. In this PEP, we introduceTypeVarTuple, enabling parameterisationwith anarbitrary number of types - that is, avariadic type variable,enablingvariadic generics. This enables a wide variety of use cases.In particular, it allows the type of array-like structuresin numerical computing libraries such as NumPy and TensorFlow to beparameterised with the arrayshape, enabling static type checkersto catch shape-related bugs in code that uses these libraries.
This PEP was accepted for Python 3.11, with the caveat that details aroundmultiple unpackings in a type expression aren’t specified precisely.This gives individual type checkers some leeway, but can be tightenedin future PEPs.
Variadic generics have long been a requested feature, for a myriad ofuse cases[4]. One particular use case - a use case with potentiallylarge impact, and the main case this PEP targets - concerns typing innumerical libraries.
In the context of numerical computation with libraries such as NumPy and TensorFlow,theshape of variables is often just as important as the variabletype.For example, consider the following function which converts a batch[1]of videos to grayscale:
defto_gray(videos:Array):...
From the signature alone, it is not obvious what shape of array[2]we should pass for thevideos argument. Possibilities include, forexample,
batch × time × height × width × channels
and
time × batch × channels × height × width.[3]
This is important for three reasons:
Ideally, we should have some way of making shape requirements explicit intype signatures. Multiple proposals[6][7][9] have suggested the use of the standard generics syntax forthis purpose. We would write:
defto_gray(videos:Array[Time,Batch,Height,Width,Channels]):...
However, note that arrays can be of arbitrary rank -Array as used above isgeneric in an arbitrary number of axes. One way around this would be to use a differentArray class for each rank…
Axis1=TypeVar('Axis1')Axis2=TypeVar('Axis2')classArray1(Generic[Axis1]):...classArray2(Generic[Axis1,Axis2]):...
…but this would be cumbersome, both for users (who would have to sprinkle 1s and 2sand so on throughout their code) and for the authors of array libraries (who would have to duplicate implementations throughout multiple classes).
Variadic generics are necessary for anArray that is generic in an arbitrarynumber of axes to be cleanly defined as a single class.
Cutting right to the chase, this PEP allows anArray class that is genericin its shape (and datatype) to be defined using a newly-introducedarbitrary-length type variable,TypeVarTuple, as follows:
fromtypingimportTypeVar,TypeVarTupleDType=TypeVar('DType')Shape=TypeVarTuple('Shape')classArray(Generic[DType,*Shape]):def__abs__(self)->Array[DType,*Shape]:...def__add__(self,other:Array[DType,*Shape])->Array[DType,*Shape]:...
Such anArray can be used to support a number of different kinds ofshape annotations. For example, we can add labels describing thesemantic meaning of each axis:
fromtypingimportNewTypeHeight=NewType('Height',int)Width=NewType('Width',int)x:Array[float,Height,Width]=Array()
We could also add annotations describing the actual size of each axis:
fromtypingimportLiteralasLx:Array[float,L[480],L[640]]=Array()
For consistency, we use semantic axis annotations as the basis of the examplesin this PEP, but this PEP is agnostic about which of these two (or possibly other)ways of usingArray is preferable; that decision is left to library authors.
(Note also that for the rest of this PEP, for conciseness of example, we usea simpler version ofArray which is generic only in the shape -not thedata type.)
In order to support the above use cases, we introduceTypeVarTuple. This serves as a placeholder not for a single typebut for atuple of types.
In addition, we introduce a new use for the star operator: to ‘unpack’TypeVarTuple instances and tuple types such asTuple[int,str]. Unpacking aTypeVarTuple or tuple type is the typingequivalent of unpacking a variable or a tuple of values.
In the same way that a normal type variable is a stand-in for a singletype such asint, a type variabletuple is a stand-in for atuple type such asTuple[int,str].
Type variable tuples are created with:
fromtypingimportTypeVarTupleTs=TypeVarTuple('Ts')
Type variable tuples behave like a number of individual type variables packed in aTuple. To understand this, consider the following example:
Shape=TypeVarTuple('Shape')classArray(Generic[*Shape]):...Height=NewType('Height',int)Width=NewType('Width',int)x:Array[Height,Width]=Array()
TheShape type variable tuple here behaves likeTuple[T1,T2],whereT1 andT2 are type variables. To use these type variablesas type parameters ofArray, we mustunpack the type variable tuple usingthe star operator:*Shape. The signature ofArray then behavesas if we had simply writtenclassArray(Generic[T1,T2]):....
In contrast toGeneric[T1,T2], however,Generic[*Shape] allowsus to parameterise the class with anarbitrary number of type parameters.That is, in addition to being able to define rank-2 arrays such asArray[Height,Width], we could also define rank-3 arrays, rank-4 arrays,and so on:
Time=NewType('Time',int)Batch=NewType('Batch',int)y:Array[Batch,Height,Width]=Array()z:Array[Time,Batch,Height,Width]=Array()
Type variable tuples can be used anywhere a normalTypeVar can.This includes class definitions, as shown above, as well as functionsignatures and variable annotations:
classArray(Generic[*Shape]):def__init__(self,shape:Tuple[*Shape]):self._shape:Tuple[*Shape]=shapedefget_shape(self)->Tuple[*Shape]:returnself._shapeshape=(Height(480),Width(640))x:Array[Height,Width]=Array(shape)y=abs(x)# Inferred type is Array[Height, Width]z=x+x# ... is Array[Height, Width]
Note that in the previous example, theshape argument to__init__was annotated asTuple[*Shape]. Why is this necessary - ifShapebehaves likeTuple[T1,T2,...], couldn’t we have annotated theshapeargument asShape directly?
This is, in fact, deliberately not possible: type variable tuples mustalways be used unpacked (that is, prefixed by the star operator). This isfor two reasons:
->Shape’,or ‘->Tuple[Shape]’, or ‘->Tuple[*Shape]’…?”)Unpack for Backwards CompatibilityNote that the use of the star operator in this context requires a grammar change,and is therefore available only in new versions of Python. To enable use of typevariable tuples in older versions of Python, we introduce theUnpack typeoperator that can be used in place of the star operator:
# Unpacking using the star operator in new versions of PythonclassArray(Generic[*Shape]):...# Unpacking using ``Unpack`` in older versions of PythonclassArray(Generic[Unpack[Shape]]):...
To keep this PEP minimal,TypeVarTuple does not yet support specification of:
TypeVar('T',covariant=True))TypeVar('T',int,float))TypeVar('T',bound=ParentClass))We leave the decision of how these arguments should behave to a future PEP, when variadic generics have been tested in the field. As of this PEP, type variable tuples areinvariant.
If the sameTypeVarTuple instance is used in multiple places in a signatureor class, a valid type inference might be to bind theTypeVarTuple toaTuple of aUnion of types:
deffoo(arg1:Tuple[*Ts],arg2:Tuple[*Ts]):...a=(0,)b=('0',)foo(a,b)# Can Ts be bound to Tuple[int | str]?
We donot allow this; type unions maynot appear within theTuple.If a type variable tuple appears in multiple places in a signature,the types must match exactly (the list of type parameters must be the samelength, and the type parameters themselves must be identical):
defpointwise_multiply(x:Array[*Shape],y:Array[*Shape])->Array[*Shape]:...x:Array[Height]y:Array[Width]z:Array[Height,Width]pointwise_multiply(x,x)# Validpointwise_multiply(x,y)# Errorpointwise_multiply(x,z)# Error
As of this PEP, only a single type variable tuple may appear in a type parameter list:
classArray(Generic[*Ts1,*Ts2]):...# Error
The reason is that multiple type variable tuples make it ambiguouswhich parameters get bound to which type variable tuple:
x:Array[int,str,bool]# Ts1 = ???, Ts2 = ???
Type variable tuples don’t have to be alone; normal types can beprefixed and/or suffixed:
Shape=TypeVarTuple('Shape')Batch=NewType('Batch',int)Channels=NewType('Channels',int)defadd_batch_axis(x:Array[*Shape])->Array[Batch,*Shape]:...defdel_batch_axis(x:Array[Batch,*Shape])->Array[*Shape]:...defadd_batch_channels(x:Array[*Shape])->Array[Batch,*Shape,Channels]:...a:Array[Height,Width]b=add_batch_axis(a)# Inferred type is Array[Batch, Height, Width]c=del_batch_axis(b)# Array[Height, Width]d=add_batch_channels(a)# Array[Batch, Height, Width, Channels]
NormalTypeVar instances can also be prefixed and/or suffixed:
T=TypeVar('T')Ts=TypeVarTuple('Ts')defprefix_tuple(x:T,y:Tuple[*Ts])->Tuple[T,*Ts]:...z=prefix_tuple(x=0,y=(True,'a'))# Inferred type of z is Tuple[int, bool, str]
We mentioned that aTypeVarTuple stands for a tuple of types.Since we can unpack aTypeVarTuple, for consistency, we alsoallow unpacking a tuple type. As we shall see, this also enables anumber of interesting features.
Unpacking a concrete tuple type is analogous to unpacking a tuple ofvalues at runtime.Tuple[int,*Tuple[bool,bool],str] isequivalent toTuple[int,bool,bool,str].
Unpacking an unbounded tuple preserves the unbounded tuple as it is.That is,*Tuple[int,...] remains*Tuple[int,...]; there’s nosimpler form. This enables us to specify types such asTuple[int,*Tuple[str,...],str] - a tuple type where the first element isguaranteed to be of typeint, the last element is guaranteed to beof typestr, and the elements in the middle are zero or moreelements of typestr. Note thatTuple[*Tuple[int,...]] isequivalent toTuple[int,...].
Unpacking unbounded tuples is also useful in function signatures wherewe don’t care about the exact elements and don’t want to define anunnecessaryTypeVarTuple:
defprocess_batch_channels(x:Array[Batch,*Tuple[Any,...],Channels])->None:...x:Array[Batch,Height,Width,Channels]process_batch_channels(x)# OKy:Array[Batch,Channels]process_batch_channels(y)# OKz:Array[Batch]process_batch_channels(z)# Error: Expected Channels.
We can also pass a*Tuple[int,...] wherever a*Ts isexpected. This is useful when we have particularly dynamic code andcannot state the precise number of dimensions or the precise types foreach of the dimensions. In those cases, we can smoothly fall back toan unbounded tuple:
y:Array[*Tuple[Any,...]]=read_from_file()defexpect_variadic_array(x:Array[Batch,*Shape])->None:...expect_variadic_array(y)# OKdefexpect_precise_array(x:Array[Batch,Height,Width,Channels])->None:...expect_precise_array(y)# OK
Array[*Tuple[Any,...]] stands for an array with an arbitrarynumber of dimensions of typeAny. This means that, in the call toexpect_variadic_array,Batch is bound toAny andShapeis bound toTuple[Any,...]. In the call toexpect_precise_array, the variablesBatch,Height,Width, andChannels are all bound toAny.
This allows users to handle dynamic code gracefully while stillexplicitly marking the code as unsafe (by usingy:Array[*Tuple[Any,...]]). Otherwise, users would face noisy errors from the typechecker every time they tried to use the variabley, which wouldhinder them when migrating a legacy code base to useTypeVarTuple.
As withTypeVarTuples,only one unpacking may appear in a tuple:
x:Tuple[int,*Ts,str,*Ts2]# Errory:Tuple[int,*Tuple[int,...],str,*Tuple[str,...]]# Error
*args as a Type Variable TuplePEP 484 states that when a type annotation is provided for*args, every argumentmust be of the type annotated. That is, if we specify*args to be typeint,thenall arguments must be of typeint. This limits our ability to specifythe type signatures of functions that take heterogeneous argument types.
If*args is annotated as a type variable tuple, however, the types of theindividual arguments become the types in the type variable tuple:
Ts=TypeVarTuple('Ts')defargs_to_tuple(*args:*Ts)->Tuple[*Ts]:...args_to_tuple(1,'a')# Inferred type is Tuple[int, str]
In the above example,Ts is bound toTuple[int,str]. If noarguments are passed, the type variable tuple behaves like an emptytuple,Tuple[()].
As usual, we can unpack any tuple types. For example, by using a typevariable tuple inside a tuple of other types, we can refer to prefixesor suffixes of the variadic argument list. For example:
# os.execle takes arguments 'path, arg0, arg1, ..., env'defexecle(path:str,*args:*Tuple[*Ts,Env])->None:...
Note that this is different to
defexecle(path:str,*args:*Ts,env:Env)->None:...
as this would makeenv a keyword-only argument.
Using an unpacked unbounded tuple is equivalent to thePEP 484behavior of*args:int, which accepts zero ormore values of typeint:
deffoo(*args:*Tuple[int,...])->None:...# equivalent to:deffoo(*args:int)->None:...
Unpacking tuple types also allows more precise types for heterogeneous*args. The following function expects anint at the beginning,zero or morestr values, and astr at the end:
deffoo(*args:*Tuple[int,*Tuple[str,...],str])->None:...
For completeness, we mention that unpacking a concrete tuple allows usto specify*args of a fixed number of heterogeneous types:
deffoo(*args:*Tuple[int,str])->None:...foo(1,"hello")# OK
Note that, in keeping with the rule that type variable tuples must alwaysbe used unpacked, annotating*args as being a plain type variable tupleinstance isnot allowed:
deffoo(*args:Ts):...# NOT valid
*args is the only case where an argument can be annotated as*Ts directly;other arguments should use*Ts to parameterise something else, e.g.Tuple[*Ts].If*args itself is annotated asTuple[*Ts], the old behaviour still applies:all arguments must be aTuple parameterised with the same types.
deffoo(*args:Tuple[*Ts]):...foo((0,),(1,))# Validfoo((0,),(1,2))# Errorfoo((0,),('1',))# Error
Finally, note that a type variable tuple maynot be used as the type of**kwargs. (We do not yet know of a use case for this feature, so we preferto leave the ground fresh for a potential future PEP.)
# NOT validdeffoo(**kwargs:*Ts):...
CallableType variable tuples can also be used in the arguments section of aCallable:
classProcess:def__init__(self,target:Callable[[*Ts],None],args:Tuple[*Ts],)->None:...deffunc(arg1:int,arg2:str)->None:...Process(target=func,args=(0,'foo'))# ValidProcess(target=func,args=('foo',0))# Error
Other types and normal type variables can also be prefixed/suffixedto the type variable tuple:
T=TypeVar('T')deffoo(f:Callable[[int,*Ts,T],Tuple[T,*Ts]]):...
The behavior of a Callable containing an unpacked item, whether theitem is aTypeVarTuple or a tuple type, is to treat the elementsas if they were the type for*args. So,Callable[[*Ts],None]is treated as the type of the function:
deffoo(*args:*Ts)->None:...
Callable[[int,*Ts,T],Tuple[T,*Ts]] is treated as the type ofthe function:
deffoo(*args:*Tuple[int,*Ts,T])->Tuple[T,*Ts]:...
When a generic class parameterised by a type variable tuple is used withoutany type parameters, it behaves as if the type variable tuple wassubstituted withTuple[Any,...]:
deftakes_any_array(arr:Array):...# equivalent to:deftakes_any_array(arr:Array[*Tuple[Any,...]]):...x:Array[Height,Width]takes_any_array(x)# Validy:Array[Time,Height,Width]takes_any_array(y)# Also valid
This enables gradual typing: existing functions accepting, for example,a plain TensorFlowTensor will still be valid even ifTensor is madegeneric and calling code passes aTensor[Height,Width].
This also works in the opposite direction:
deftakes_specific_array(arr:Array[Height,Width]):...z:Array# equivalent to Array[*Tuple[Any, ...]]takes_specific_array(z)
(For details, see the section onUnpacking Unbounded Tuple Types.)
This way, even if libraries are updated to use types likeArray[Height,Width],users of those libraries won’t be forced to also apply type annotations toall of their code; users still have a choice about what parts of their codeto type and which parts to not.
Generic aliases can be created using a type variable tuple ina similar way to regular type variables:
IntTuple=Tuple[int,*Ts]NamedArray=Tuple[str,Array[*Ts]]IntTuple[float,bool]# Equivalent to Tuple[int, float, bool]NamedArray[Height]# Equivalent to Tuple[str, Array[Height]]
As this example shows, all type parameters passed to the alias arebound to the type variable tuple.
Importantly for our originalArray example (seeSummary Examples), thisallows us to define convenience aliases for arrays of a fixed shapeor datatype:
Shape=TypeVarTuple('Shape')DType=TypeVar('DType')classArray(Generic[DType,*Shape]):# E.g. Float32Array[Height, Width, Channels]Float32Array=Array[np.float32,*Shape]# E.g. Array1D[np.uint8]Array1D=Array[DType,Any]
If an explicitly empty type parameter list is given, the type variabletuple in the alias is set empty:
IntTuple[()]# Equivalent to Tuple[int]NamedArray[()]# Equivalent to Tuple[str, Array[()]]
If the type parameter list is omitted entirely, the unspecified typevariable tuples are treated asTuple[Any,...] (similar toBehaviour when Type Parameters are not Specified):
deftakes_float_array_of_any_shape(x:Float32Array):...x:Float32Array[Height,Width]=Array()takes_float_array_of_any_shape(x)# Validdeftakes_float_array_with_specific_shape(y:Float32Array[Height,Width]):...y:Float32Array=Array()takes_float_array_with_specific_shape(y)# Valid
NormalTypeVar instances can also be used in such aliases:
T=TypeVar('T')Foo=Tuple[T,*Ts]# T bound to str, Ts to Tuple[int]Foo[str,int]# T bound to float, Ts to Tuple[()]Foo[float]# T bound to Any, Ts to an Tuple[Any, ...]Foo
In the previous section, we only discussed simple usage of generic aliasesin which the type arguments were just simple types. However, a number ofmore exotic constructions are also possible.
First, type arguments to generic aliases can be variadic. For example, aTypeVarTuple can be used as a type argument:
Ts1=TypeVar('Ts1')Ts2=TypeVar('Ts2')IntTuple=Tuple[int,*Ts1]IntFloatTuple=IntTuple[float,*Ts2]# Valid
Here,*Ts1 in theIntTuple alias is bound toTuple[float,*Ts2],resulting in an aliasIntFloatTuple equivalent toTuple[int,float,*Ts2].
Unpacked arbitrary-length tuples can also be used as type arguments, withsimilar effects:
IntFloatsTuple=IntTuple[*Tuple[float,...]]# Valid
Here,*Ts1 is bound to*Tuple[float,...], resulting inIntFloatsTuple being equivalent toTuple[int,*Tuple[float,...]]: a tupleconsisting of anint then zero or morefloats.
Variadic type arguments can only be used with generic aliases that arethemselves variadic. For example:
T=TypeVar('T')IntTuple=Tuple[int,T]IntTuple[str]# ValidIntTuple[*Ts]# NOT validIntTuple[*Tuple[float,...]]# NOT valid
Here,IntTuple is anon-variadic generic alias that takes exactly onetype argument. Hence, it cannot accept*Ts or*Tuple[float,...] as typearguments, because they represent an arbitrary number of types.
InAliases, we briefly mentioned that aliases can be generic in bothTypeVars andTypeVarTuples:
T=TypeVar('T')Foo=Tuple[T,*Ts]Foo[str,int]# T bound to str, Ts to Tuple[int]Foo[str,int,float]# T bound to str, Ts to Tuple[int, float]
In accordance withMultiple Type Variable Tuples: Not Allowed, at most oneTypeVarTuple may appear in the type parameters to an alias. However, aTypeVarTuple can be combined with an arbitrary number ofTypeVars,both before and after:
T1=TypeVar('T1')T2=TypeVar('T2')T3=TypeVar('T3')Tuple[*Ts,T1,T2]# ValidTuple[T1,T2,*Ts]# ValidTuple[T1,*Ts,T2,T3]# Valid
In order to substitute these type variables with supplied type arguments,any type variables at the beginning or end of the type parameter list firstconsume type arguments, and then any remaining type arguments are boundto theTypeVarTuple:
Shrubbery=Tuple[*Ts,T1,T2]Shrubbery[str,bool]# T2=bool, T1=str, Ts=Tuple[()]Shrubbery[str,bool,float]# T2=float, T1=bool, Ts=Tuple[str]Shrubbery[str,bool,float,int]# T2=int, T1=float, Ts=Tuple[str, bool]Ptang=Tuple[T1,*Ts,T2,T3]Ptang[str,bool,float]# T1=str, T3=float, T2=bool, Ts=Tuple[()]Ptang[str,bool,float,int]# T1=str, T3=int, T2=float, Ts=Tuple[bool]
Note that the minimum number of type arguments in such cases is set bythe number ofTypeVars:
Shrubbery[int]# Not valid; Shrubbery needs at least two type arguments
A final complication occurs when an unpacked arbitrary-length tuple is usedas a type argument to an alias consisting of bothTypeVars and aTypeVarTuple:
Elderberries=Tuple[*Ts,T1]Hamster=Elderberries[*Tuple[int,...]]# valid
In such cases, the arbitrary-length tuple is split between theTypeVarsand theTypeVarTuple. We assume the arbitrary-length tuple containsat least as many items as there areTypeVars, such that individualinstances of the inner type - hereint - are bound to anyTypeVarspresent. The ‘rest’ of the arbitrary-length tuple - here*Tuple[int,...],since a tuple of arbitrary length minus two items is still arbitrary-length -is bound to theTypeVarTuple.
Here, therefore,Hamster is equivalent toTuple[*Tuple[int,...],int]:a tuple consisting of zero or moreints, then a finalint.
Of course, such splitting only occurs if necessary. For example, if we insteaddid:
Elderberries[*Tuple[int,...],str]
Then splitting would not occur;T1 would be bound tostr, andTs to*Tuple[int,...].
In particularly awkward cases, aTypeVarTuple may consume both a typeand a part of an arbitrary-length tuple type:
Elderberries[str,*Tuple[int,...]]
Here,T1 is bound toint, andTs is bound toTuple[str,*Tuple[int,...]]. This expression is therefore equivalent toTuple[str,*Tuple[int,...],int]: a tuple consisting of astr, thenzero or moreints, ending with anint.
Finally, although any arbitrary-length tuples in the type argument list can besplit between the type variables and the type variable tuple, the same is nottrue ofTypeVarTuples in the argument list:
Ts1=TypeVarTuple('Ts1')Ts2=TypeVarTuple('Ts2')Camelot=Tuple[T,*Ts1]Camelot[*Ts2]# NOT valid
This is not possible because, unlike in the case of an unpacked arbitrary-lengthtuple, there is no way to ‘peer inside’ theTypeVarTuple to see what itsindividual types are.
For situations where we require access to each individual type in the type variable tuple,overloads can be used with individualTypeVar instances in place of the type variable tuple:
Shape=TypeVarTuple('Shape')Axis1=TypeVar('Axis1')Axis2=TypeVar('Axis2')Axis3=TypeVar('Axis3')classArray(Generic[*Shape]):@overloaddeftranspose(self:Array[Axis1,Axis2])->Array[Axis2,Axis1]:...@overloaddeftranspose(self:Array[Axis1,Axis2,Axis3])->Array[Axis3,Axis2,Axis1]:...
(For array shape operations in particular, having to specifyoverloads for each possible rank is, of course, a rather cumbersomesolution. However, it’s the best we can do without additional typemanipulation mechanisms. We plan to introduce these in a future PEP.)
Considering the use case of array shapes in particular, note that as ofthis PEP, it is not yet possible to describe arithmetic transformationsof array dimensions - for example,defrepeat_each_element(x:Array[N])->Array[2*N]. We considerthis out-of-scope for the current PEP, but plan to propose additionalmechanisms thatwill enable this in a future PEP.
As noted in the introduction, itis possible to avoid variadic genericsby simply defining aliases for each possible number of type parameters:
classArray1(Generic[Axis1]):...classArray2(Generic[Axis1,Axis2]):...
However, this seems somewhat clumsy - it requires users to unnecessarilypepper their code with 1s, 2s, and so on for each rank necessary.
TypeVarTupleTypeVarTuple began asListVariadic, based on its naming inan early implementation in Pyre.
We then changed this toTypeVar(list=True), on the basis that a)it better emphasises the similarity toTypeVar, and b) the meaningof ‘list’ is more easily understood than the jargon of ‘variadic’.
Once we’d decided that a variadic type variable should behave like aTuple,we also consideredTypeVar(bound=Tuple), which is similarly intuitiveand accomplishes most what we wanted without requiring any new arguments toTypeVar. However, we realised this may constrain us in the future, iffor example we want type bounds or variance to function slightly differentlyfor variadic type variables than what the semantics ofTypeVar mightotherwise imply. Also, we may later wish to support arguments that should not be supported by regular type variables (such asarbitrary_len[10]).
We therefore settled onTypeVarTuple.
In order to support gradual typing, this PEP states thatbothof the following examples should type-check correctly:
deftakes_any_array(x:Array):...x:Array[Height,Width]takes_any_array(x)deftakes_specific_array(y:Array[Height,Width]):...y:Arraytakes_specific_array(y)
Note that this is in contrast to the behaviour of the only currently-existingvariadic type in Python,Tuple:
deftakes_any_tuple(x:Tuple):...x:Tuple[int,str]takes_any_tuple(x)# Validdeftakes_specific_tuple(y:Tuple[int,str]):...y:Tupletakes_specific_tuple(y)# Error
The rules forTuple were deliberately chosen such that the latter caseis an error: it was thought to be more likely that the programmer has made amistake than that the function expects a specific kind ofTuple but thespecific kind ofTuple passed is unknown to the type checker. Additionally,Tuple is something of a special case, in that it is used to representimmutable sequences. That is, if an object’s type is inferred to be anunparameterisedTuple, it is not necessarily because of incomplete typing.
In contrast, if an object’s type is inferred to be an unparameterisedArray,it is much more likely that the user has simply not yet fully annotated theircode, or that the signature of a shape-manipulating library function cannot yetbe expressed using the typing system and therefore returning a plainArrayis the only option. We rarely deal with arrays of truly arbitrary shape;in certain cases,some parts of the shape will be arbitrary - for example,when dealing with sequences, the first two parts of the shape are often‘batch’ and ‘time’ - but we plan to support these cases explicitly in afuture PEP with a syntax such asArray[Batch,Time,...].
We therefore made the decision to have variadic genericsother thanTuple behave differently, in order to give the user more flexibilityin how much of their code they wish to annotate, and to enable compatibilitybetween old unannotated code and new versions of libraries which do usethese type annotations.
It should be noted that the approach outlined in this PEP to solve theissue of shape checking in numerical libraries isnot the only approachpossible. Examples of lighter-weight alternatives based onruntime checking includeShapeGuard[13], tsanley[11], and PyContracts[12].
While these existing approaches improve significantly on the defaultsituation of shape checking only being possible through lengthy and verboseassert statements, none of them enablestatic analysis of shape correctness.As mentioned inMotivation, this is particularly desirable formachine learning applications where, due to library and infrastructure complexity,even relatively simple programs must suffer long startup times; iteratingby running the program until it crashes, as is necessary with theseexisting runtime-based approaches, can be a tedious and frustratingexperience.
Our hope with this PEP is to begin to codify generic type annotations asan official, language-supported way of dealing with shape correctness.With something of a standard in place, in the long run, this willhopefully enable a thriving ecosystem of tools for analysing and verifyingshape properties of numerical computing programs.
This PEP requires two grammar changes.
The first grammar change enables use of star expressions in index operations (that is,within square brackets), necessary to support star-unpacking of TypeVarTuples:
DType=TypeVar('DType')Shape=TypeVarTuple('Shape')classArray(Generic[DType,*Shape]):...
Before:
slices: | slice !',' | ','.slice+ [',']
After:
slices: | slice !',' | ','.(slice | starred_expression)+ [',']
As with star-unpacking in other contexts, the star operator calls__iter__on the callee, and adds the contents of the resulting iterator to the argumentpassed to__getitem__. For example, if we dofoo[a,*b,c], andb.__iter__ produces an iterator yieldingd ande,foo.__getitem__ would receive(a,d,e,c).
To put it another way, note thatx[...,*a,...] produces the same resultasx[(...,*a,...)] (with any slicesi:j in... replaced withslice(i,j), with the one edge case thatx[*a] becomesx[(*a,)]).
With this grammar change,TypeVarTuple is implemented as follows.Note that this implementation is useful only for the benefit of a) correctrepr() and b) runtime analysers; static analysers would not use theimplementation.
classTypeVarTuple:def__init__(self,name):self._name=nameself._unpacked=UnpackedTypeVarTuple(name)def__iter__(self):yieldself._unpackeddef__repr__(self):returnself._nameclassUnpackedTypeVarTuple:def__init__(self,name):self._name=namedef__repr__(self):return'*'+self._name
This grammar change implies a number of additional changes in behaviour notrequired by this PEP. We choose to allow these additional changes rather thandisallowing them at a syntax level in order to keep the syntax change as smallas possible.
First, the grammar change enables star-unpacking of other structures, suchas lists, within indexing operations:
idxs=(1,2)array_slice=array[0,*idxs,-1]# Equivalent to [0, 1, 2, -1]array[0,*idxs,-1]=array_slice# Also allowed
Second, more than one instance of a star-unpack can occur within an index:
array[*idxs_to_select,*idxs_to_select]# Equivalent to array[1, 2, 1, 2]
Note that this PEP disallows multiple unpacked TypeVarTuples within a singletype parameter list. This requirement would therefore need to be implementedin type checking tools themselves rather than at the syntax level.
Third, slices may co-occur with starred expressions:
array[3:5,*idxs_to_select]# Equivalent to array[3:5, 1, 2]
However, note that slices involving starred expressions are still invalid:
# Syntax errorarray[*idxs_start:*idxs_end]
*args as a TypeVarTupleThe second change enables use of*args:*Ts in function definitions.
Before:
star_etc:|'*'param_no_defaultparam_maybe_default*[kwds]|'*'','param_maybe_default+[kwds]|kwds
After:
star_etc:|'*'param_no_defaultparam_maybe_default*[kwds]|'*'param_no_default_star_annotationparam_maybe_default*[kwds]# New|'*'','param_maybe_default+[kwds]|kwds
Where:
param_no_default_star_annotation:| param_star_annotation ',' TYPE_COMMENT?| param_star_annotation TYPE_COMMENT? &')'param_star_annotation: NAME star_annotationstar_annotation: ':' star_expression
We also need to deal with thestar_expression that results from thisconstruction. Normally, astar_expression occurs within the contextof e.g. a list, so astar_expression is handled by essentiallycallingiter() on the starred object, and inserting the resultsof the resulting iterator into the list at the appropriate place. For*args:*Ts, however, we must process thestar_expression in adifferent way.
We do this by instead making a special case for thestar_expressionresulting from*args:*Ts, emitting code equivalent to[annotation_value]=[*Ts]. That is, we create an iterator fromTs by callingTs.__iter__, fetch a single value from the iterator,verify that the iterator is exhausted, and set that value as the annotationvalue. This results in the unpackedTypeVarTuple being set directlyas the runtime annotation for*args:
>>>Ts=TypeVarTuple('Ts')>>>deffoo(*args:*Ts):pass>>>foo.__annotations__{'args': *Ts}# *Ts is the repr() of Ts._unpacked, an instance of UnpackedTypeVarTuple
This allows the runtime annotation to be consistent with an AST representationthat uses aStarred node for the annotations ofargs - in turn importantfor tools that rely on the AST such as mypy to correctly recognise the construction:
>>>print(ast.dump(ast.parse('def foo(*args: *Ts): pass'),indent=2))Module( body=[ FunctionDef( name='foo', args=arguments( posonlyargs=[], args=[], vararg=arg( arg='args', annotation=Starred( value=Name(id='Ts', ctx=Load()), ctx=Load())), kwonlyargs=[], kw_defaults=[], defaults=[]), body=[ Pass()], decorator_list=[])], type_ignores=[])
Note that the only scenario in which this grammar change allows*Ts to beused as a direct annotation (rather than being wrapped in e.g.Tuple[*Ts])is*args. Other uses are still invalid:
x:*Ts# Syntax errordeffoo(x:*Ts):pass# Syntax error
As with the first grammar change, this change also has a number of side effects.In particular, the annotation of*args could be set to a starred objectother than aTypeVarTuple - for example, the following nonsensicalannotations are possible:
>>>foo=[1]>>>defbar(*args:*foo):pass>>>bar.__annotations__{'args': 1}>>>foo=[1,2]>>>defbar(*args:*foo):passValueError: too many values to unpack (expected 1)
Again, prevention of such annotations will need to be done by, say, staticcheckers, rather than at the level of syntax.
Unpack?)If these grammar changes are considered too burdensome, there are twoalternatives.
The first would be tosupport change 1 but not change 2. Variadic genericsare more important to us than the ability to annotate*args.
The second alternative would be touse ``Unpack`` instead, requiring nogrammar changes. However, we regard this as a suboptimal solution for tworeasons:
classArray(Generic[DType,Unpack[Shape]]) is a bitof a mouthful; the flow of reading is interrupted by length ofUnpack andthe extra set of square brackets.classArray(Generic[DType,*Shape])is much easier to skim, while still markingShape as special.*Ts - especially when they see thatTs is aTypeVar**Tuple** - than the meaning ofUnpack[Ts]. (This assumesthe user is familiar with star-unpacking in other contexts; if theuser is reading or writing code that uses variadic generics, this seemsreasonable.)If even change 1 is thought too significant a change, therefore, it might bebetter for us to reconsider our options before going ahead with this secondalternative.
TheUnpack version of the PEP should be back-portable to previousversions of Python.
Gradual typing is enabled by the fact that unparameterised variadic classesare compatible with an arbitrary number of type parameters. This meansthat if existing classes are made generic, a) all existing (unparameterised)uses of the class will still work, and b) parameterised and unparameterisedversions of the class can be used together (relevant if, for example, librarycode is updated to use parameters while user code is not, or vice-versa).
Two reference implementations of type-checking functionality exist:one in Pyre, as of v0.9.0, and one in Pyright, as of v1.1.108.
A preliminary implementation of theUnpack version of the PEP in CPythonis available incpython/23527. A preliminary version of the versionusing the star operator, based on an early implementation ofPEP 637,is also available atmrahtz/cpython/pep637+646.
To give this PEP additional context for those particularly interested in thearray typing use case, in this appendix we expand on the different waysthis PEP can be used for specifying shape-based subtypes.
The simplest way to parameterise array types is usingLiteraltype parameters - e.g.Array[Literal[64],Literal[64]].
We can attach names to each parameter using normal type variables:
K=TypeVar('K')N=TypeVar('N')defmatrix_vector_multiply(x:Array[K,N],y:Array[N])->Array[K]:...a:Array[Literal[64],Literal[32]]b:Array[Literal[32]]matrix_vector_multiply(a,b)# Result is Array[Literal[64]]
Note that such names have a purely local scope. That is, the nameK is bound toLiteral[64] only withinmatrix_vector_multiply. To put it anotherway, there’s no relationship between the value ofK in differentsignatures. This is important: it would be inconvenient if every axis namedKwere constrained to have the same value throughout the entire program.
The disadvantage of this approach is that we have no ability to enforce shape semantics acrossdifferent calls. For example, we can’t address the problem mentioned inMotivation: ifone function returns an array with leading dimensions ‘Time × Batch’, and another functiontakes the same array assuming leading dimensions ‘Batch × Time’, we have no way of detecting this.
The main advantage is that in some cases, axis sizes really are what we care about. This is truefor both simple linear algebra operations such as the matrix manipulations above, but also in morecomplicated transformations such as convolutional layers in neural networks, where it would be ofgreat utility to the programmer to be able to inspect the array size after each layer usingstatic analysis. To aid this, in the future we would like to explore possibilities for additionaltype operators that enable arithmetic on array shapes - for example:
defrepeat_each_element(x:Array[N])->Array[Mul[2,N]]:...
Such arithmetic type operators would only make sense if names such asN refer to axis size.
A second approach (the one that most of the examples in this PEP are based around)is to forgo annotation with actual axis size, and instead annotate axistype.
This would enable us to solve the problem of enforcing shape properties across calls.For example:
# lib.pyclassBatch:passclassTime:passdefmake_array()->Array[Batch,Time]:...# user.pyfromlibimportBatch,Time# `Batch` and `Time` have the same identity as in `lib`,# so must take array as produced by `lib.make_array`defuse_array(x:Array[Batch,Time]):...
Note that in this case, names areglobal (to the extent that we use thesameBatch type in different place). However, because names refer onlyto axistypes, this doesn’t constrain thevalue of certain axes to bethe same through (that is, this doesn’t constrain all axes namedHeightto have a value of, say, 480 throughout).
The argumentfor this approach is that in many cases, axistype is the moreimportant thing to verify; we care more about which axis is which than what thespecific size of each axis is.
It also does not preclude cases where we wish to describe shape transformationswithout knowing the type ahead of time. For example, we can still write:
K=TypeVar('K')N=TypeVar('N')defmatrix_vector_multiply(x:Array[K,N],y:Array[N])->Array[K]:...
We can then use this with:
classBatch:passclassValues:passbatch_of_values:Array[Batch,Values]value_weights:Array[Values]matrix_vector_multiply(batch_of_values,value_weights)# Result is Array[Batch]
The disadvantages are the inverse of the advantages from use case 1.In particular, this approach does not lend itself well to arithmeticon axis types:Mul[2,Batch] would be as meaningless as2*int.
Note that use cases 1 and 2 are mutually exclusive in user code. Userscan verify size or semantic type but not both.
As of this PEP, we are agnostic about which approach will provide most benefit.Since the features introduced in this PEP are compatible with both approaches, however,we leave the door open.
Consider the following ‘normal’ code:
deff(x:int):...
Note that we have symbols for both the value of the thing (x) and the type ofthe thing (int). Why can’t we do the same with axes? For example, with an imaginarysyntax, we could write:
deff(array:Array[TimeValue:TimeType]):...
This would allow us to access the axis size (say, 32) through the symbolTimeValueand the type through the symbolTypeType.
This might even be possible using existing syntax, through a second level of parameterisation:
deff(array:array[TimeValue[TimeType]]):..
However, we leave exploration of this approach to the future.
An issue related to those addressed by this PEP concernsaxisselection. For example, if we have an image stored in an array of shape 64×64x3,we might wish to convert to black-and-white by computing the mean over the thirdaxis,mean(image,axis=2). Unfortunately, the simple typoaxis=1 isdifficult to spot and will produce a result that means something completely different(all while likely allowing the program to keep on running, resulting in a bugthat is serious but silent).
In response, some libraries have implemented so-called ‘named tensors’ (in this context,‘tensor’ is synonymous with ‘array’), in which axes are selected not by index but bylabel - e.g.mean(image,axis='channels').
A question we are often asked about this PEP is: why not just use named tensors?The answer is that we consider the named tensors approach insufficient, for two main reasons:
Additionally, there’s the issue ofpoor uptake. At the time of writing, named tensorshave only been implemented in a small number of numerical computing libraries. Possible explanations for thisinclude difficulty of implementation (the whole API must be modified to allow selection by axis nameinstead of index), and lack of usefulness due to the fact that axis ordering conventions are oftenstrong enough that axis names provide little benefit (e.g. when working with images, 3D tensors arebasicallyalways height × width × channels). However, ultimately we are still uncertainwhy this is the case.
Can the named tensors approach be combined with the approach we advocate for inthis PEP? We’re not sure. One area of overlap is that in some contexts, we could do, say:
Image:Array[Height,Width,Channels]im:Imagemean(im,axis=Image.axes.index(Channels)
Ideally, we might write something likeim:Array[Height=64,Width=64,Channels=3] -but this won’t be possible in the short term, due to the rejection ofPEP 637.In any case, our attitude towards this is mostly “Wait and see what happens beforetaking any further steps”.
ndarray;in TensorFlow, theTensor; and so on.videos_batch[0][1] would select the second frame of the first video. If theshape begins with ‘time × batch’, thenvideos_batch[1][0] would select thesame frame.Variadic generics have a wide range of uses. For the fraction of that rangeinvolving numerical computing, how likely is it that relevant libraries willactually make use of the features proposed in this PEP?
We reached out to a number of people with this question, and received thefollowing endorsements.
FromStephan Hoyer, member of the NumPy Steering Council:[14]
I just wanted to thank Matthew & Pradeep for writing this PEP and forclarifications to the broader context ofPEP 646 for array typing inhttps://github.com/python/peps/pull/1904.As someone who is heavily involved in the Python numerical computingcommunity (e.g., NumPy, JAX, Xarray), but who is not so familiar with thedetails of Python’s type system, it is reassuring to see that a broad rangeof use-cases related to type checking of named axes & shapes have beenconsidered, and could build upon the infrastructure in this PEP.
Type checking for shapes is something the NumPy community is veryinterested in – there are more thumbs up on the relevant issue on NumPy’sGitHub than any others (https://github.com/numpy/numpy/issues/7370) and werecently added a “typing” module that is under active development.
It will certainly require experimentation to figure out the best ways touse type checking for ndarrays, but this PEP looks like an excellentfoundation for such work.
FromBas van Beek, who has worked on preliminary support forshape-generics in NumPy:
I very much share Stephan’s opinion here and look forward to integrating thenewPEP 646 variadics into numpy.In the context of numpy (and tensor typing general): the typing of arrayshapes is a fairly complicated subject and the introduction of variadicswill likely play in big role in laying its foundation, as it allows for theexpression of both dimensioability as well as basic shape manipulation.
All in all, I’m very interested in where bothPEP 646 and future PEPs willtake us and look forward to further developments.
FromDan Moldovan, a Senior Software Engineer on the TensorFlow Dev Teamand author of the TensorFlow RFC,TensorFlow Canonical Type System:[15]
I’d be interested in using this the mechanisms defined in this PEP to definerank-generic Tensor types in TensorFlow, which are important in specifyingtf.functionsignatures in a Pythonic way, using type annotations (ratherthan the custominput_signaturemechanism we have today - see thisissue:https://github.com/tensorflow/tensorflow/issues/31579). Variadicgenerics are among the last few missing pieces to create an elegant set oftype definitions for tensors and shapes.
(For the sake of transparency - we also reached out to folks from a third popularnumerical computing library, PyTorch, but didnot receive a statement ofendorsement from them. Our understanding is that although they are interestedin some of the same issues - e.g. static shape inference - they are currentlyfocusing on enabling this through a DSL rather than the Python type system.)
Thank you toAlfonso Castaño,Antoine Pitrou,Bas v.B.,David Foster,Dimitris Vardoulakis,Eric Traut,Guido van Rossum,Jia Chen,Lucio Fernandez-Arjona,Nikita Sobolev,Peilonrayz,Rebecca Chen,Sergei Lebedev, andVladimir Mikulik for helpful feedback and suggestions ondrafts of this PEP.
Thank you especially toLucio for suggesting the star syntax (which has made multiple aspects of this proposal much more concise and intuitive), and toStephan Hoyer andDan Moldovan for their endorsements.
Discussions on variadic generics in Python started in 2016 with Issue 193on the python/typing GitHub repository[4].
Inspired by this discussion,Ivan Levkivskyi made a concrete proposalat PyCon 2019, summarised in notes on ‘Type system improvements’[5]and ‘Static typing of Python numeric stack’[6].
Expanding on these ideas,Mark Mendoza andVincent Siles gave a presentation on‘Variadic Type Variables for Decorators and Tensors’[8] at the 2019 PythonTyping Summit.
Discussion over how type substitution in generic aliases should behavetook place incpython#91162.
This document is placed in the public domain or under theCC0-1.0-Universal license, whichever is more permissive.
Source:https://github.com/python/peps/blob/main/peps/pep-0646.rst
Last modified:2025-02-01 08:55:40 GMT