Important
This PEP is a historical document: seerequired-notrequired,typing.Required andtyping.NotRequired for up-to-date specs and documentation. Canonical typing specs are maintained at thetyping specs site; runtime typing behaviour is described in the CPython documentation.
×
See thetyping specification update process for how to propose changes to the typing spec.
PEP 589 defines notationfor declaring a TypedDict with all required keys and notation for defininga TypedDict withall potentially-missing keys, however itdoes not provide a mechanism to declare some keys as required and othersas potentially-missing. This PEP introduces two new notations:Required[], which can be used on individual items of aTypedDict to mark them as required, andNotRequired[], which can be used on individual itemsto mark them as potentially-missing.
This PEP makes no Python grammar changes. Correct usageof required and potentially-missing keys of TypedDicts is intended to beenforced only by static type checkers and need not be enforced byPython itself at runtime.
It is not uncommon to want to define a TypedDict with some keys that arerequired and others that are potentially-missing. Currently the only wayto define such a TypedDict is to declare one TypedDict with one valuefortotal and then inherit it from another TypedDict with adifferent value fortotal:
class_MovieBase(TypedDict):# implicitly total=Truetitle:strclassMovie(_MovieBase,total=False):year:int
Having to declare two different TypedDict types for this purpose iscumbersome.
This PEP introduces two new type qualifiers,typing.Required andtyping.NotRequired, which allow defining asingle TypedDict witha mix of both required and potentially-missing keys:
classMovie(TypedDict):title:stryear:NotRequired[int]
This PEP also makes it possible to define TypedDicts in thealternative functional syntaxwith a mix of required and potentially-missing keys,which is not currently possible at all because the alternative syntax doesnot support inheritance:
Actor=TypedDict('Actor',{'name':str,# "in" is a keyword, so the functional syntax is necessary'in':NotRequired[List[str]],})
One might think it unusual to propose notation that prioritizes markingrequired keys rather thanpotentially-missing keys, as iscustomary in other languages like TypeScript:
interfaceMovie{title:string;year?:number;// ? marks potentially-missing keys}
The difficulty is that the best word for marking a potentially-missingkey,Optional[], is already used in Python for a completelydifferent purpose: marking values that could be either of a particulartype orNone. In particular the following does not work:
classMovie(TypedDict):...year:Optional[int]# means int|None, not potentially-missing!
Attempting to use any synonym of “optional” to mark potentially-missingkeys (likeMissing[]) would be too similar toOptional[]and be easy to confuse with it.
Thus it was decided to focus on positive-form phrasing for required keysinstead, which is straightforward to spell asRequired[].
Nevertheless it is common for folks wanting to extend a regular(total=True) TypedDict to only want to add a small number ofpotentially-missing keys, which necessitates a way to mark keys that arenot required and potentially-missing, and so we also allow theNotRequired[] form for that case.
Thetyping.Required type qualifier is used to indicate that avariable declared in a TypedDict definition is a required key:
classMovie(TypedDict,total=False):title:Required[str]year:int
Additionally thetyping.NotRequired type qualifier is used toindicate that a variable declared in a TypedDict definition is apotentially-missing key:
classMovie(TypedDict):# implicitly total=Truetitle:stryear:NotRequired[int]
It is an error to useRequired[] orNotRequired[] in anylocation that is not an item of a TypedDict.Type checkers must enforce this restriction.
It is valid to useRequired[] andNotRequired[] even foritems where it is redundant, to enable additional explicitness if desired:
classMovie(TypedDict):title:Required[str]# redundantyear:NotRequired[int]
It is an error to use bothRequired[] andNotRequired[] at thesame time:
classMovie(TypedDict):title:stryear:NotRequired[Required[int]]# ERROR
Type checkers must enforce this restriction.The runtime implementations ofRequired[] andNotRequired[]may also enforce this restriction.
Thealternative functional syntaxfor TypedDict also supportsRequired[] andNotRequired[]:
Movie=TypedDict('Movie',{'name':str,'year':NotRequired[int]})
total=FalseAnyPEP 589-style TypedDict declared withtotal=False is equivalentto a TypedDict with an implicittotal=True definition with all of itskeys marked asNotRequired[].
Therefore:
class_MovieBase(TypedDict):# implicitly total=Truetitle:strclassMovie(_MovieBase,total=False):year:int
is equivalent to:
class_MovieBase(TypedDict):title:strclassMovie(_MovieBase):year:NotRequired[int]
Annotated[]Required[] andNotRequired[] can be used withAnnotated[],in any nesting order:
classMovie(TypedDict):title:stryear:NotRequired[Annotated[int,ValueRange(-9999,9999)]]# ok
classMovie(TypedDict):title:stryear:Annotated[NotRequired[int],ValueRange(-9999,9999)]# ok
In particular allowingAnnotated[] to be the outermost annotationfor an item allows better interoperability with non-typing uses ofannotations, which may always wantAnnotated[] as the outermost annotation.[3]
get_type_hints()typing.get_type_hints(...) applied to a TypedDict will by defaultstrip out anyRequired[] orNotRequired[] type qualifiers,since these qualifiers are expected to be inconvenient for codecasually introspecting type annotations.
typing.get_type_hints(...,include_extras=True) howeverwill retainRequired[] andNotRequired[] type qualifiers,for advanced code introspecting type annotations thatwishes to preserveall annotations in the original source:
classMovie(TypedDict):title:stryear:NotRequired[int]assertget_type_hints(Movie)== \{'title':str,'year':int}assertget_type_hints(Movie,include_extras=True)== \{'title':str,'year':NotRequired[int]}
get_origin() andget_args()typing.get_origin() andtyping.get_args() will be updated torecognizeRequired[] andNotRequired[]:
assertget_origin(Required[int])isRequiredassertget_args(Required[int])==(int,)assertget_origin(NotRequired[int])isNotRequiredassertget_args(NotRequired[int])==(int,)
__required_keys__ and__optional_keys__An item marked withRequired[] will always appearin the__required_keys__ for its enclosing TypedDict. Similarly an itemmarked withNotRequired[] will always appear in__optional_keys__.
assertMovie.__required_keys__==frozenset({'title'})assertMovie.__optional_keys__==frozenset({'year'})
No backward incompatible changes are made by this PEP.
To define a TypedDict where most keys are required and some arepotentially-missing, define a single TypedDict as normal(without thetotal keyword)and mark those few keys that are potentially-missing withNotRequired[].
To define a TypedDict where most keys are potentially-missing and a few arerequired, define atotal=False TypedDictand mark those few keys that are required withRequired[].
If some items acceptNone in addition to a regular value, it isrecommended that theTYPE|None notation be preferred overOptional[TYPE] for marking such item values, to avoid usingRequired[] orNotRequired[] alongsideOptional[]within the same TypedDict definition:
Yes:
from__future__importannotations# for Python 3.7-3.9classDog(TypedDict):name:strowner:NotRequired[str|None]
Okay (required for Python 3.5.3-3.6):
classDog(TypedDict):name:strowner:'NotRequired[str|None]'
No:
classDog(TypedDict):name:str# ick; avoid using both Optional and NotRequiredowner:NotRequired[Optional[str]]
If your code supports Python <3.11 and wishes to useRequired[] orNotRequired[] then it should usetyping_extensions.TypedDict ratherthantyping.TypedDict because the latter will not understand(Not)Required[]. In particular__required_keys__ and__optional_keys__ on the resulting TypedDict type will not be correct:
Yes (Python 3.11+ only):
fromtypingimportNotRequired,TypedDictclassDog(TypedDict):name:strowner:NotRequired[str|None]
Yes (Python <3.11 and 3.11+):
from__future__importannotations# for Python 3.7-3.9fromtyping_extensionsimportNotRequired,TypedDict# for Python <3.11 with (Not)RequiredclassDog(TypedDict):name:strowner:NotRequired[str|None]
No (Python <3.11 and 3.11+):
fromtypingimportTypedDict# oops: should import from typing_extensions insteadfromtyping_extensionsimportNotRequiredclassMovie(TypedDict):title:stryear:NotRequired[int]assertMovie.__required_keys__==frozenset({'title','year'})# yikesassertMovie.__optional_keys__==frozenset()# yikes
Themypy0.930,pyright1.1.117,andpyanalyze0.4.0type checkers supportRequired andNotRequired.
A reference implementation of the runtime component is provided in thetyping_extensionsmodule.
class MyThing(TypedDict): opt1?: str # may not exist, but if exists, value is string opt2: Optional[str] # always exists, but may have None value
This notation would require Python grammar changes and it is notbelieved that marking TypedDict items as required or potentially-missingwould meet the high bar required to make such grammar changes.
classMyThing(TypedDict):Optional[opt1]:str# may not exist, but if exists, value is stringopt2:Optional[str]# always exists, but may have None value
This notation causesOptional[] to take on different meanings dependingon where it is positioned, which is inconsistent and confusing.
Also, “let’s just not put funny syntax before the colon.”[1]
We could use unary+ as shorthand to mark a required key, unary- to mark a potentially-missing key, or unary~ to mark a keywith opposite-of-normal totality:
classMyThing(TypedDict,total=False):req1:+int# + means a required key, or Required[]opt1:strreq2:+floatclassMyThing(TypedDict):req1:intopt1:-str# - means a potentially-missing key, or NotRequired[]req2:floatclassMyThing(TypedDict):req1:intopt1:~str# ~ means a opposite-of-normal-totality keyreq2:float
Such operators could be implemented ontype via the__pos__,__neg__ and__invert__ special methods without modifying thegrammar.
It was decided that it would be prudent to introduce long-form notation(i.e.Required[] andNotRequired[]) before introducingany short-form notation. Future PEPs may reconsider introducing thisor other short-form notation options.
Note when reconsidering introducing this short-form notation that+,-, and~ already have existing meanings in the Pythontyping world: covariant, contravariant, and invariant:
>>>fromtypingimportTypeVar>>>(TypeVar('T',covariant=True),TypeVar('U',contravariant=True),TypeVar('V'))(+T, -U, ~V)
We could introduce a new type-level constant which signals the absenceof a value when used as a union member, similar to JavaScript’sundefined type, perhaps calledMissing:
classMyThing(TypedDict):req1:intopt1:str|Missingreq2:float
Such aMissing constant could also be used for other scenarios suchas the type of a variable which is only conditionally defined:
classMyClass:attr:int|Missingdef__init__(self,set_attr:bool)->None:ifset_attr:self.attr=10
deffoo(set_attr:bool)->None:ifset_attr:attr=10reveal_type(attr)# int|Missing
However this use of...|Missing, equivalent toUnion[...,Missing], doesn’t align well with what a union normallymeans:Union[...] always describes the type of avalue that ispresent. By contrast missingness or non-totality is a property of avariable instead. Current precedent for marking properties of avariable includeFinal[...] andClassVar[...], which theproposal forRequired[...] is aligned with.
Furthermore the use ofUnion[...,Missing] doesn’t align with theusual ways that union values are broken down: Normally you can eliminatecomponents of a union type usingisinstance checks:
classPacket:data:Union[str,bytes]defsend_data(packet:Packet)->None:ifisinstance(packet.data,str):reveal_type(packet.data)# strpacket_bytes=packet.data.encode('utf-8')else:reveal_type(packet.data)# bytespacket_bytes=packet.datasocket.send(packet_bytes)
However if we were to allowUnion[...,Missing] you’d either have toeliminate theMissing case withhasattr for object attributes:
classPacket:data:Union[str,Missing]defsend_data(packet:Packet)->None:ifhasattr(packet,'data'):reveal_type(packet.data)# strpacket_bytes=packet.data.encode('utf-8')else:reveal_type(packet.data)# Missing? error?packet_bytes=b''socket.send(packet_bytes)
or a check againstlocals() for local variables:
defsend_data(packet_data:Optional[str])->None:packet_bytes:Union[str,Missing]ifpacket_dataisnotNone:packet_bytes=packet.data.encode('utf-8')if'packet_bytes'inlocals():reveal_type(packet_bytes)# bytessocket.send(packet_bytes)else:reveal_type(packet_bytes)# Missing? error?
or a check via other means, such as againstglobals() for globalvariables:
warning:Union[str,Missing]importsysifsys.version_info<(3,6):warning='Your version of Python is unsupported!'if'warning'inglobals():reveal_type(warning)# strprint(warning)else:reveal_type(warning)# Missing? error?
Weird and inconsistent.Missing is not really a value at all; it’san absence of definition and such an absence should be treatedspecially.
Eric Traut from the Pyright type checker team has stated thatimplementing aUnion[...,Missing]-style notation would bedifficult.[2]
Defining a newMissing type-level constant would be very close tointroducing a newMissing value-level constant at runtime, creatinga second null-like runtime value in addition toNone. Having twodifferent null-like constants in Python (None andMissing) wouldbe confusing. Many newcomers to JavaScript already have difficultydistinguishing between its analogous constantsnull andundefined.
Optional[] is too ubiquitous to deprecate, although use of itmay fade over time in favor of theT|None notation specified byPEP 604.
Consider the use of a special flag on a TypedDict definition to alterthe interpretation ofOptional inside the TypedDict to mean“optional item” rather than its usual meaning of “nullable”:
classMyThing(TypedDict,optional_as_missing=True):req1:intopt1:Optional[str]
or:
classMyThing(TypedDict,optional_as_nullable=False):req1:intopt1:Optional[str]
This would add more confusion for users because it would mean that insome contexts the meaning ofOptional[] is different than inother contexts, and it would be easy to overlook the flag.
Drop, which means somethingdifferentThis document is placed in the public domain or under theCC0-1.0-Universal license, whichever is more permissive.
Source:https://github.com/python/peps/blob/main/peps/pep-0655.rst
Last modified:2024-06-16 22:42:44 GMT