Combining runtime and static uses of annotations

There’s an emerging trend of libraries leveraging the typing annotations atruntime (e.g.: dataclasses); having the ability to extend the typing annotationswith external data would be a great boon for those libraries.

Here’s an example of how a hypothetical module could leverage annotations toread c structs:

UnsignedShort=Annotated[int,struct2.ctype('H')]SignedChar=Annotated[int,struct2.ctype('b')]classStudent(struct2.Packed):# mypy typechecks 'name' field as 'str'name:Annotated[str,struct2.ctype("<10s")]serialnum:UnsignedShortschool:SignedChar# 'unpack' only uses the metadata within the type annotationsStudent.unpack(record)# Student(name=b'raymond   ', serialnum=4658, school=264)

Lowering barriers to developing new typing constructs

Typically when adding a new type, a developer need to upstream that type to thetyping module and change mypy,PyCharm, Pyre,pytype,etc…This is particularly important when working on open-source code thatmakes use of these types, seeing as the code would not be immediatelytransportable to other developers’ tools without additional logic. As a result,there is a high cost to developing and trying out new types in a codebase.Ideally, authors should be able to introduce new types in a manner that allowsfor graceful degradation (e.g.: when clients do not have a custommypy plugin), which would lower the barrier to development and ensure somedegree of backward compatibility.

For example, suppose that an author wanted to add support fortagged unions to Python. One way to accomplish would be toannotateTypedDict in Python such that only one field is allowedto be set:

Currency=Annotated[TypedDict('Currency',{'dollars':float,'pounds':float},total=False),TaggedUnion,]

This is a somewhat cumbersome syntax but it allows us to iterate on thisproof-of-concept and have people with type checkers (or other tools) that don’tyet support this feature work in a codebase with tagged unions. The author couldeasily test this proposal and iron out the kinks before trying to upstream taggedunion totyping, mypy, etc. Moreover, tools that do not have support forparsing theTaggedUnion annotation would still be able to treatCurrencyas aTypedDict, which is still a close approximation (slightly less strict).

Syntax

Annotated is parameterized with a type and an arbitrary list ofPython values that represent the annotations. Here are the specificdetails of the syntax:

• The first argument toAnnotated must be a valid type
• Multiple type annotations are supported (Annotated supports variadicarguments):

  Annotated[int,ValueRange(3,10),ctype("char")]

• Annotated must be called with at least two arguments (Annotated[int] is not valid)
• The order of the annotations is preserved and matters for equalitychecks:

  Annotated[int,ValueRange(3,10),ctype("char")]!=Annotated[int,ctype("char"),ValueRange(3,10)]

• NestedAnnotated types are flattened, with metadata orderedstarting with the innermost annotation:

  Annotated[Annotated[int,ValueRange(3,10)],ctype("char")]==Annotated[int,ValueRange(3,10),ctype("char")]

• Duplicated annotations are not removed:

  Annotated[int,ValueRange(3,10)]!=Annotated[int,ValueRange(3,10),ValueRange(3,10)]

• Annotated can be used with nested and generic aliases:

  TypevarT=...Vec=Annotated[List[Tuple[T,T]],MaxLen(10)]V=Vec[int]V==Annotated[List[Tuple[int,int]],MaxLen(10)]

Consuming annotations

Ultimately, the responsibility of how to interpret the annotations (ifat all) is the responsibility of the tool or library encountering theAnnotated type. A tool or library encountering anAnnotated typecan scan through the annotations to determine if they are of interest(e.g., usingisinstance()).

Unknown annotations: When a tool or a library does not supportannotations or encounters an unknown annotation it should just ignore itand treat annotated type as the underlying type. For example, when encounteringan annotation that is not an instance ofstruct2.ctype to the annotationsfor name (e.g.,Annotated[str,'foo',struct2.ctype("<10s")]), the unpackmethod should ignore it.

Namespacing annotations: Namespaces are not needed for annotations sincethe class used by the annotations acts as a namespace.

Multiple annotations: It’s up to the tool consuming the annotationsto decide whether the client is allowed to have several annotations onone type and how to merge those annotations.

Since theAnnotated type allows you to put several annotations ofthe same (or different) type(s) on any node, the tools or librariesconsuming those annotations are in charge of dealing with potentialduplicates. For example, if you are doing value range analysis you mightallow this:

T1=Annotated[int,ValueRange(-10,5)]T2=Annotated[T1,ValueRange(-20,3)]

Flattening nested annotations, this translates to:

T2=Annotated[int,ValueRange(-10,5),ValueRange(-20,3)]

Interaction withget_type_hints()

typing.get_type_hints() will take a new argumentinclude_extras thatdefaults toFalse to preserve backward compatibility. Wheninclude_extras isFalse, the extra annotations will be strippedout of the returned value. Otherwise, the annotations will be returnedunchanged:

@struct2.packedclassStudent(NamedTuple):name:Annotated[str,struct.ctype("<10s")]get_type_hints(Student)=={'name':str}get_type_hints(Student,include_extras=False)=={'name':str}get_type_hints(Student,include_extras=True)=={'name':Annotated[str,struct.ctype("<10s")]}

Aliases & Concerns over verbosity

Writingtyping.Annotated everywhere can be quite verbose;fortunately, the ability to alias annotations means that in practice wedon’t expect clients to have to write lots of boilerplate code:

T=TypeVar('T')Const=Annotated[T,my_annotations.CONST]classC:defconst_method(self:Const[List[int]])->int:...