Important
This PEP is a historical document: seeParamSpec andtyping.ParamSpec for up-to-date specs and documentation. Canonical typing specs are maintained at thetyping specs site; runtime typing behaviour is described in the CPython documentation.
×
See thetyping specification update process for how to propose changes to the typing spec.
There currently are two ways to specify the type of a callable, theCallable[[int,str],bool] syntax defined inPEP 484,and callback protocols fromPEP544. Neither ofthese support forwarding the parameter types of one callable over to anothercallable, making it difficult to annotate function decorators. This PEP proposestyping.ParamSpec andtyping.Concatenate tosupport expressing these kinds of relationships.
The existing standards for annotating higher order functions don’t give us thetools to annotate the following common decorator pattern satisfactorily:
fromtypingimportAwaitable,Callable,TypeVarR=TypeVar("R")defadd_logging(f:Callable[...,R])->Callable[...,Awaitable[R]]:asyncdefinner(*args:object,**kwargs:object)->R:awaitlog_to_database()returnf(*args,**kwargs)returninner@add_loggingdeftakes_int_str(x:int,y:str)->int:returnx+7awaittakes_int_str(1,"A")awaittakes_int_str("B",2)# fails at runtime
add_logging, a decorator which logs before each entry into the decoratedfunction, is an instance of the Python idiom of one function passing allarguments given to it over to another function. This is done through thecombination of the*args and**kwargs features in both parameters and inarguments. When one defines a function (likeinner) that takes(*args,**kwargs) and goes on to call another function with(*args,**kwargs),the wrapping function can only be safely called in all of the ways that thewrapped function could be safely called. To type this decorator, we’d like to beable to place a dependency between the parameters of the callablef and theparameters of the returned function.PEP 484supports dependencies betweensingle types, as indefappend(l:typing.List[T],e:T)->typing.List[T]:..., but there is no existing way to do so with a complicated entity likethe parameters of a function.
Due to the limitations of the status quo, theadd_logging example will typecheck but will fail at runtime.inner will pass the string “B” intotakes_int_str, which will try to add 7 to it, triggering a type error.This was not caught by the type checker because the decoratedtakes_int_strwas given the typeCallable[...,Awaitable[int]] (an ellipsis in place ofparameter types is specified to mean that we do no validation on arguments).
Without the ability to define dependencies between the parameters of differentcallable types, there is no way, at present, to makeadd_logging compatiblewith all functions, while still preserving the enforcement of the parameters ofthe decorated function.
With the addition of theParamSpec variables proposed by thisPEP, we can rewrite the previous example in a way that keeps the flexibility ofthe decorator and the parameter enforcement of the decorated function.
fromtypingimportAwaitable,Callable,ParamSpec,TypeVarP=ParamSpec("P")R=TypeVar("R")defadd_logging(f:Callable[P,R])->Callable[P,Awaitable[R]]:asyncdefinner(*args:P.args,**kwargs:P.kwargs)->R:awaitlog_to_database()returnf(*args,**kwargs)returninner@add_loggingdeftakes_int_str(x:int,y:str)->int:returnx+7awaittakes_int_str(1,"A")# Acceptedawaittakes_int_str("B",2)# Correctly rejected by the type checker
Another common decorator pattern that has previously been impossible to type isthe practice of adding or removing arguments from the decorated function. Forexample:
classRequest:...defwith_request(f:Callable[...,R])->Callable[...,R]:definner(*args:object,**kwargs:object)->R:returnf(Request(),*args,**kwargs)returninner@with_requestdeftakes_int_str(request:Request,x:int,y:str)->int:# use requestreturnx+7takes_int_str(1,"A")takes_int_str("B",2)# fails at runtime
With the addition of theConcatenate operator from this PEP, we can eventype this more complex decorator.
fromtypingimportConcatenatedefwith_request(f:Callable[Concatenate[Request,P],R])->Callable[P,R]:definner(*args:P.args,**kwargs:P.kwargs)->R:returnf(Request(),*args,**kwargs)returninner@with_requestdeftakes_int_str(request:Request,x:int,y:str)->int:# use requestreturnx+7takes_int_str(1,"A")# Acceptedtakes_int_str("B",2)# Correctly rejected by the type checker
ParamSpec VariablesA parameter specification variable is defined in a similar manner to how anormal type variable is defined withtyping.TypeVar.
fromtypingimportParamSpecP=ParamSpec("P")# AcceptedP=ParamSpec("WrongName")# Rejected because P =/= WrongName
The runtime should acceptbounds andcovariant andcontravariantarguments in the declaration just astyping.TypeVar does, but for now wewill defer the standardization of the semantics of those options to a later PEP.
Previously only a list of parameter arguments ([A,B,C]) or an ellipsis(signifying “undefined parameters”) were acceptable as the first “argument” totyping.Callable . We now augment that with two new options: a parameterspecification variable (Callable[P,int]) or a concatenation on aparameter specification variable (Callable[Concatenate[int,P],int]).
callable::=Callable"["parameters_expression,type_expression"]"parameters_expression::=|"..."|"["[type_expression(","type_expression)*]"]"|parameter_specification_variable|concatenate"["type_expression(","type_expression)*","parameter_specification_variable"]"
whereparameter_specification_variable is atyping.ParamSpec variable,declared in the manner as defined above, andconcatenate istyping.Concatenate.
As before,parameters_expressions by themselves are not acceptable inplaces where a type is expected
deffoo(x:P)->P:...# Rejecteddeffoo(x:Concatenate[int,P])->int:...# Rejecteddeffoo(x:typing.List[P])->None:...# Rejecteddeffoo(x:Callable[[int,str],P])->None:...# Rejected
Just as defining a class as inheriting fromGeneric[T] makes a class genericfor a single parameter (whenT is aTypeVar), defining a class asinheriting fromGeneric[P] makes a class generic onparameters_expressions (whenP is aParamSpec).
T=TypeVar("T")P_2=ParamSpec("P_2")classX(Generic[T,P]):f:Callable[P,int]x:Tdeff(x:X[int,P_2])->str:...# Accepteddeff(x:X[int,Concatenate[int,P_2]])->str:...# Accepteddeff(x:X[int,[int,bool]])->str:...# Accepteddeff(x:X[int,...])->str:...# Accepteddeff(x:X[int,int])->str:...# Rejected
By the rules defined above, spelling a concrete instance of a class genericwith respect to only a singleParamSpec would require unsightly doublebrackets. For aesthetic purposes we allow these to be omitted.
classZ(Generic[P]):f:Callable[P,int]deff(x:Z[[int,str,bool]])->str:...# Accepteddeff(x:Z[int,str,bool])->str:...# Equivalent# Both Z[[int, str, bool]] and Z[int, str, bool] express this:classZ_instantiated:f:Callable[[int,str,bool],int]
The inference rules for the return type of a function invocation whose signaturecontains aParamSpec variable are analogous to those aroundevaluating ones withTypeVars.
defchanges_return_type_to_str(x:Callable[P,int])->Callable[P,str]:...defreturns_int(a:str,b:bool)->int:...f=changes_return_type_to_str(returns_int)# f should have the type:# (a: str, b: bool) -> strf("A",True)# Acceptedf(a="A",b=True)# Acceptedf("A","A")# Rejectedexpects_str(f("A",True))# Acceptedexpects_int(f("A",True))# Rejected
Just as with traditionalTypeVars, a user may include the sameParamSpec multiple times in the arguments of the same function,to indicate a dependency between multiple arguments. In these cases a typechecker may choose to solve to a common behavioral supertype (i.e. a set ofparameters for which all of the valid calls are valid in both of the subtypes),but is not obligated to do so.
P=ParamSpec("P")deffoo(x:Callable[P,int],y:Callable[P,int])->Callable[P,bool]:...defx_y(x:int,y:str)->int:...defy_x(y:int,x:str)->int:...foo(x_y,x_y)# Should return (x: int, y: str) -> boolfoo(x_y,y_x)# Could return (__a: int, __b: str) -> bool# This works because both callables have types that are# behavioral subtypes of Callable[[int, str], int]defkeyword_only_x(*,x:int)->int:...defkeyword_only_y(*,y:int)->int:...foo(keyword_only_x,keyword_only_y)# Rejected
The constructors of user-defined classes generic onParamSpecs should beevaluated in the same way.
U=TypeVar("U")classY(Generic[U,P]):f:Callable[P,str]prop:Udef__init__(self,f:Callable[P,str],prop:U)->None:self.f=fself.prop=propdefa(q:int)->str:...Y(a,1)# Should resolve to Y[(q: int), int]Y(a,1).f# Should resolve to (q: int) -> str
The semantics ofConcatenate[X,Y,P] are that it represents the parametersrepresented byP with two positional-only parameters prepended. This meansthat we can use it to represent higher order functions that add, remove ortransform a finite number of parameters of a callable.
defbar(x:int,*args:bool)->int:...defadd(x:Callable[P,int])->Callable[Concatenate[str,P],bool]:...add(bar)# Should return (__a: str, x: int, *args: bool) -> booldefremove(x:Callable[Concatenate[int,P],int])->Callable[P,bool]:...remove(bar)# Should return (*args: bool) -> booldeftransform(x:Callable[Concatenate[int,P],int])->Callable[Concatenate[str,P],bool]:...transform(bar)# Should return (__a: str, *args: bool) -> bool
This also means that while any function that returns anR can satisfytyping.Callable[P,R], only functions that can be called positionally intheir first position with aX can satisfytyping.Callable[Concatenate[X,P],R].
defexpects_int_first(x:Callable[Concatenate[int,P],int])->None:...@expects_int_first# Rejecteddefone(x:str)->int:...@expects_int_first# Rejecteddeftwo(*,x:int)->int:...@expects_int_first# Rejecteddefthree(**kwargs:int)->int:...@expects_int_first# Accepteddeffour(*args:int)->int:...
There are still some classes of decorators still not supported with thesefeatures:
functools.partial will remain untypable even after this PEP)ParamSpecAParamSpec captures both positional and keyword accessibleparameters, but there unfortunately is no object in the runtime that capturesboth of these together. Instead, we are forced to separate them into*argsand**kwargs, respectively. This means we need to be able to split aparta singleParamSpec into these two components, and then bringthem back together into a call. To do this, we introduceP.args torepresent the tuple of positional arguments in a given call andP.kwargs to represent the correspondingMapping of keywords tovalues.
These “properties” can only be used as the annotated types for*args and**kwargs, accessed from a ParamSpec already in scope.
defputs_p_into_scope(f:Callable[P,int])->None:definner(*args:P.args,**kwargs:P.kwargs)->None:# Acceptedpassdefmixed_up(*args:P.kwargs,**kwargs:P.args)->None:# Rejectedpassdefmisplaced(x:P.args)->None:# Rejectedpassdefout_of_scope(*args:P.args,**kwargs:P.kwargs)->None:# Rejectedpass
Furthermore, because the default kind of parameter in Python ((x:int))may be addressed both positionally and through its name, two valid invocationsof a(*args:P.args,**kwargs:P.kwargs) function may give differentpartitions of the same set of parameters. Therefore, we need to make sure thatthese special types are only brought into the world together, and are usedtogether, so that our usage is valid for all possible partitions.
defputs_p_into_scope(f:Callable[P,int])->None:stored_args:P.args# Rejectedstored_kwargs:P.kwargs# Rejecteddefjust_args(*args:P.args)->None:# Rejectedpassdefjust_kwargs(**kwargs:P.kwargs)->None:# Rejectedpass
With those requirements met, we can now take advantage of the unique propertiesafforded to us by this set up:
args has the typeP.args, notTuple[P.args,...] as would be with a normal annotation(and likewise with the**kwargs)args/kwargs of a given call, which cannot be expressedby an indefinite tuple/dictionary type.Callable[P,R] can be called with(*args,**kwargs)if and only ifargs has the typeP.args andkwargs has the typeP.kwargs, and that those types both originated from the same functiondeclaration.definner(*args:P.args,**kwargs:P.kwargs)->Xhas typeCallable[P,X].With these three properties, we now have the ability to fully type checkparameter preserving decorators.
defdecorator(f:Callable[P,int])->Callable[P,None]:deffoo(*args:P.args,**kwargs:P.kwargs)->None:f(*args,**kwargs)# Accepted, should resolve to intf(*kwargs,**args)# Rejectedf(1,*args,**kwargs)# Rejectedreturnfoo# Accepted
To extend this to includeConcatenate, we declare the following properties:
Callable[Concatenate[A,B,P],R] can only becalled with(a,b,*args,**kwargs) whenargs andkwargs are therespective components ofP,a is of typeA andb is oftypeB.definner(a:A,b:B,*args:P.args,**kwargs:P.kwargs)->Rhas typeCallable[Concatenate[A,B,P],R]. Placing keyword-onlyparameters between the*args and**kwargs is forbidden.defadd(f:Callable[P,int])->Callable[Concatenate[str,P],None]:deffoo(s:str,*args:P.args,**kwargs:P.kwargs)->None:# Acceptedpassdefbar(*args:P.args,s:str,**kwargs:P.kwargs)->None:# Rejectedpassreturnfoo# Accepteddefremove(f:Callable[Concatenate[int,P],int])->Callable[P,None]:deffoo(*args:P.args,**kwargs:P.kwargs)->None:f(1,*args,**kwargs)# Acceptedf(*args,1,**kwargs)# Rejectedf(*args,**kwargs)# Rejectedreturnfoo
Note that the names of the parameters preceding theParamSpeccomponents are not mentioned in the resultingConcatenate. This means thatthese parameters can not be addressed via a named argument:
defouter(f:Callable[P,None])->Callable[P,None]:deffoo(x:int,*args:P.args,**kwargs:P.kwargs)->None:f(*args,**kwargs)defbar(*args:P.args,**kwargs:P.kwargs)->None:foo(1,*args,**kwargs)# Acceptedfoo(x=1,*args,**kwargs)# Rejectedreturnbar
This is not an implementation convenience, but a soundness requirement. If wewere to allow that second calling style, then the following snippet would beproblematic.
@outerdefproblem(*,x:object)->None:passproblem(x="uh-oh")
Inside ofbar, we would getTypeError:foo()gotmultiplevaluesforargument'x'. Requiring theseconcatenated arguments to be addressed positionally avoids this kind of problem,and simplifies the syntax for spelling these types. Note that this also why wehave to reject signatures of the form(*args:P.args,s:str,**kwargs:P.kwargs) (SeeConcatenating Keyword Parameters for more details).
If one of these prepended positional parameters contains a freeParamSpec,we consider that variable in scope for the purposes of extracting the componentsof thatParamSpec. That allows us to spell things like this:
deftwice(f:Callable[P,int],*args:P.args,**kwargs:P.kwargs)->int:returnf(*args,**kwargs)+f(*args,**kwargs)
The type oftwice in the above example isCallable[Concatenate[Callable[P,int],P],int], whereP is bound by theouterCallable. This has the following semantics:
defa_int_b_str(a:int,b:str)->int:passtwice(a_int_b_str,1,"A")# Acceptedtwice(a_int_b_str,b="A",a=1)# Acceptedtwice(a_int_b_str,"A",1)# Rejected
The only changes necessary to existing features intyping is allowing theseParamSpec andConcatenate objects to be the first parameter toCallable and to be a parameter toGeneric. CurrentlyCallableexpects a list of types there andGeneric expects single types, so they arecurrently mutually exclusive. Otherwise, existing code that doesn’t referencethe new interfaces will be unaffected.
ThePyre type checker supports all of the behaviordescribed above. A reference implementation of the runtime components neededfor those uses is provided in thepyre_extensions module. A referenceimplementation for CPython can be foundhere.
We considered just trying to make something like this with a callback protocolwhich was parameterized on a list-type variadic, and a map-type variadic likeso:
R = typing.TypeVar(“R”)Tpositionals = ...Tkeywords = ...class BetterCallable(typing.Protocol[Tpositionals, Tkeywords, R]): def __call__(*args: Tpositionals, **kwargs: Tkeywords) -> R: ...
However, there are some problems with trying to come up with a consistentsolution for those type variables for a given callable. This problem comes upwith even the simplest of callables:
def simple(x: int) -> None: ...simple <: BetterCallable[[int], [], None]simple <: BetterCallable[[], {“x”: int}, None]BetterCallable[[int], [], None] </: BetterCallable[[], {“x”: int}, None]Any time where a type can implement a protocol in more than one way that aren’tmutually compatible, we can run into situations where we lose information. If wewere to make a decorator using this protocol, we would have to pick one callingconvention to prefer.
defdecorator(f:BetterCallable[[Ts],[Tmap],int],)->BetterCallable[[Ts],[Tmap],str]:defdecorated(*args:Ts,**kwargs:Tmap)->str:x=f(*args,**kwargs)returnint_to_str(x)returndecorated@decoratordeffoo(x:int)->int:returnxreveal_type(foo)# Option A: BetterCallable[[int], {}, str]# Option B: BetterCallable[[], {x: int}, str]foo(7)# fails under option Bfoo(x=7)# fails under option A
The core problem here is that, by default, parameters in Python can either becalled positionally or as a keyword argument. This means we really havethree categories (positional-only, positional-or-keyword, keyword-only) we’retrying to jam into two categories. This is the same problem that we brieflymentioned when discussing.args and.kwargs. Fundamentally, in order tocapture two categories when there are some things that can be in eithercategory, we need a higher level primitive (ParamSpec) tocapture all three, and then split them out afterward.
Another proposal we considered was definingParametersOf andReturnTypeoperators which would operate on a domain of a newly definedFunction type.Function would be callable with, and only withParametersOf[F].ParametersOf andReturnType would only operate on type variables withprecisely this bound. The combination of these three features could expresseverything that we can express withParamSpecs.
F=TypeVar("F",bound=Function)defno_change(f:F)->F:definner(*args:ParametersOf[F].args,**kwargs:ParametersOf[F].kwargs)->ReturnType[F]:returnf(*args,**kwargs)returninnerdefwrapping(f:F)->Callable[ParametersOf[F],List[ReturnType[F]]]:definner(*args:ParametersOf[F].args,**kwargs:ParametersOf[F].kwargs)->List[ReturnType[F]]:return[f(*args,**kwargs)]returninnerdefunwrapping(f:Callable[ParametersOf[F],List[R]])->Callable[ParametersOf[F],R]:definner(*args:ParametersOf[F].args,**kwargs:ParametersOf[F].kwargs)->R:returnf(*args,**kwargs)[0]returninner
We decided to go withParamSpecs over this approach for several reasons:
ParamSpec just introduces a new variable.ParamSpec based signatures look much more like existing Python.unwrapping is odd to read becauseF is not actually referring to anycallable. It’s just being used as a container for the parameters we wish topropagate. It would read better if we could define an operatorRemoveList[List[X]]=X and thenunwrapping could takeF andreturnCallable[ParametersOf[F],RemoveList[ReturnType[F]]]. Withoutthat, we unfortunately get into a situation where we have to use aFunction-variable as an improvisedParamSpec, in that we neveractually bind the return type.In summary, between these two equivalently powerful syntaxes,ParamSpec fitsmuch more naturally into the status quo.
In principle the idea of concatenation as a means to modify a finite number ofpositional parameters could be expanded to include keyword parameters.
defadd_n(f:Callable[P,R])->Callable[Concatenate[("n",int),P],R]:definner(*args:P.args,n:int,**kwargs:P.kwargs)->R:# use nreturnf(*args,**kwargs)returninner
However, the key distinction is that while prepending positional-only parametersto a valid callable type always yields another valid callable type, the samecannot be said for adding keyword-only parameters. As alluded toabove , theissue is name collisions. The parametersConcatenate[("n",int),P] areonly valid whenP itself does not already have a parameter namedn.
definnocent_wrapper(f:Callable[P,R])->Callable[P,R]:definner(*args:P.args,**kwargs:P.kwargs)->R:added=add_n(f)returnadded(*args,n=1,**kwargs)returninner@innocent_wrapperdefproblem(n:int)->None:pass
Callingproblem(2) works fine, but callingproblem(n=2) leads to aTypeError:problem()gotmultiplevaluesforargument'n' from the call toadded inside ofinnocent_wrapper.
This kind of situation could be avoided, and this kind of decorator could betyped if we could reify the constraint that a set of parametersnot containa certain name, with something like:
P_without_n=ParamSpec("P_without_n",banned_names=["n"])defadd_n(f:Callable[P_without_n,R])->Callable[Concatenate[("n",int),P_without_n],R]:...
The call toadd_n inside ofinnocent_wrapper could then be rejectedsince the callable was not guaranteed not to already have a parameter namedn.
However, enforcing these constraints would require enough additionalimplementation work that we judged this extension to be out of scope of thisPEP. Fortunately the design ofParamSpecs are such that we can return tothis idea later if there is sufficient demand.
ParameterSpecificationWe decided that ParameterSpecification was a little too long-winded for usehere, and that this style of abbreviated name made it look more like TypeVar.
ArgSpecWe think that calling this a ParamSpec is more correct thanreferring to it as an ArgSpec, since callables have parameters,which are distinct from the arguments which are passed to them in a given callsite. A given binding for a ParamSpec is a set of functionparameters, not a call-site’s arguments.
Thanks to all of the members of the Pyre team for their comments on early draftsof this PEP, and for their help with the reference implementation.
Thanks are also due to the whole Python typing community for their earlyfeedback on this idea at a Python typing meetup, leading directly to the muchmore compact.args/.kwargs syntax.
This document is placed in the public domain or under the CC0-1.0-Universallicense, whichever is more permissive.
Source:https://github.com/python/peps/blob/main/peps/pep-0612.rst
Last modified:2024-06-11 22:12:09 GMT