Non-goals

Just like inPEP 484 andPEP 526, it should be emphasized thatPythonwill remain a dynamically typed language, and the authors have no desireto ever make type hints mandatory, even by convention.

This PEP is meant to solve the problem of forward references in typeannotations. There are still cases outside of annotations whereforward references will require usage of string literals. Those arelisted in a later section of this document.

Annotations without forced evaluation enable opportunities to improvethe syntax of type hints. This idea will require its own separate PEPand is not discussed further in this document.

Non-typing usage of annotations

While annotations are still available for arbitrary use besides typechecking, it is worth mentioning that the design of this PEP, as wellas its precursors (PEP 484 andPEP 526), is predominantly motivated bythe type hinting use case.

In Python 3.8PEP 484 will graduate from provisional status. Otherenhancements to the Python programming language likePEP 544,PEP 557,orPEP 560, are already being built on this basis as they depend ontype annotations and thetyping module as defined byPEP 484.In fact, the reasonPEP 484 is staying provisional in Python 3.7 is toenable rapid evolution for another release cycle that some of theaforementioned enhancements require.

With this in mind, uses for annotations incompatible with theaforementioned PEPs should be considered deprecated.

Enabling the future behavior in Python 3.7

The functionality described above can be enabled starting from Python3.7 using the following special import:

from__future__importannotations

A reference implementation of this functionality is availableon GitHub.

Runtime annotation resolution and class decorators

Metaclasses and class decorators that need to resolve annotations forthe current class will fail for annotations that use the name of thecurrent class. Example:

defclass_decorator(cls):annotations=get_type_hints(cls)# raises NameError on 'C'print(f'Annotations for{cls}:{annotations}')returncls@class_decoratorclassC:singleton:'C'=None

This was already true before this PEP. The class decorator acts onthe class before it’s assigned a name in the current definition scope.

Runtime annotation resolution andTYPE_CHECKING

Sometimes there’s code that must be seen by a type checker but shouldnot be executed. For such situations thetyping module defines aconstant,TYPE_CHECKING, that is consideredTrue during typechecking butFalse at runtime. Example:

importtypingiftyping.TYPE_CHECKING:importexpensive_moddefa_func(arg:expensive_mod.SomeClass)->None:a_var:expensive_mod.SomeClass=arg...

This approach is also useful when handling import cycles.

Trying to resolve annotations ofa_func at runtime usingtyping.get_type_hints() will fail since the nameexpensive_modis not defined (TYPE_CHECKING variable beingFalse at runtime).This was already true before this PEP.

Deprecation policy

Starting with Python 3.7, a__future__ import is required to use thedescribed functionality. No warnings are raised.

NOTE: Whether this will eventually become the default behavior is currently unclearpending decision onPEP 649. In any case, use of annotations that depend upontheir eager evaluation is incompatible with both proposals and is no longersupported.

Keeping the ability to use function local state when defining annotations

With postponed evaluation, this would require keeping a reference tothe frame in which an annotation got created. This could be achievedfor example by storing all annotations as lambdas instead of strings.

This would be prohibitively expensive for highly annotated code as theframes would keep all their objects alive. That includes predominantlyobjects that won’t ever be accessed again.

To be able to address class-level scope, the lambda approach wouldrequire a new kind of cell in the interpreter. This would proliferatethe number of types that can appear in__annotations__, as well aswouldn’t be as introspectable as strings.

Note that in the case of nested classes, the functionality to get theeffective “globals” and “locals” at definition time is provided bytyping.get_type_hints().

If a function generates a class or a function with annotations thathave to use local variables, it can populate the given generatedobject’s__annotations__ dictionary directly, without relying onthe compiler.

Disallowing local state usage for classes, too

This PEP originally proposed limiting names within annotations to onlyallow names from the model-level scope, including for classes. Theauthor argued this makes name resolution unambiguous, including in casesof conflicts between local names and module-level names.

This idea was ultimately rejected in case of classes. Instead,typing.get_type_hints() got modified to populate the local namespacecorrectly if class-level annotations are needed.

The reasons for rejecting the idea were that it goes against theintuition of how scoping works in Python, and would break enoughexisting type annotations to make the transition cumbersome. Finally,local scope access is required for class decorators to be able toevaluate type annotations. This is because class decorators are appliedbefore the class receives its name in the outer scope.

Introducing a new dictionary for the string literal form instead

Yury Selivanov shared the following idea:

1. Add a new special attribute to functions:__annotations_text__.
2. Make__annotations__ a lazy dynamic mapping, evaluatingexpressions from the corresponding key in__annotations_text__just-in-time.

This idea is supposed to solve the backwards compatibility issue,removing the need for a new__future__ import. Sadly, this is notenough. Postponed evaluation changes which state the annotation hasaccess to. While postponed evaluation fixes the forward referenceproblem, it also makes it impossible to access function-level localsanymore. This alone is a source of backwards incompatibility whichjustifies a deprecation period.

A__future__ import is an obvious and explicit indicator of optingin for the new functionality. It also makes it trivial for externaltools to recognize the difference between a Python files using the oldor the new approach. In the former case, that tool would recognize thatlocal state access is allowed, whereas in the latter case it wouldrecognize that forward references are allowed.

Finally, just-in-time evaluation in__annotations__ is anunnecessary step ifget_type_hints() is used later.

Dropping annotations with -O

There are two reasons this is not satisfying for the purpose of thisPEP.

First, this only addresses runtime cost, not forward references, thosestill cannot be safely used in source code. A library maintainer wouldnever be able to use forward references since that would force thelibrary users to use this new hypothetical -O switch.

Second, this throws the baby out with the bath water. Nowno runtimeannotation use can be performed.PEP 557 is one example of a recentdevelopment where evaluating type annotations at runtime is useful.

All that being said, a granular -O option to drop annotations isa possibility in the future, as it’s conceptually compatible withexisting -O behavior (dropping docstrings and assert statements). ThisPEP does not invalidate the idea.

Passing string literals in annotations verbatim to__annotations__

This PEP originally suggested directly storing the contents of a stringliteral under its respective key in__annotations__. This wasmeant to simplify support for runtime type checkers.

Mark Shannon pointed out this idea was flawed since it wasn’t handlingsituations where strings are only part of a type annotation.

The inconsistency of it was always apparent but given that it doesn’tfully prevent cases of double-wrapping strings anyway, it is not worthit.

Making the name of the future import more verbose

Instead of requiring the following import:

from__future__importannotations

the PEP could call the feature more explicitly, for examplestring_annotations,stringify_annotations,annotation_strings,annotations_as_strings,lazy_annotations,static_annotations, etc.

The problem with those names is that they are very verbose. Each ofthem besideslazy_annotations would constitute the longest futurefeature name in Python. They are long to type and harder to rememberthan the single-word form.

There is precedence of a future import name that sounds overly genericbut in practice was obvious to users as to what it does:

from__future__importdivision

In PEP 484

The forward reference problem was discussed whenPEP 484 was originallydrafted, leading to the following statement in the document:

A compromise is possible where a__future__ import could enableturningall annotations in a given module into string literals, asfollows:

from__future__importannotationsclassImSet:defadd(self,a:ImSet)->List[ImSet]:...assertImSet.add.__annotations__=={'a':'ImSet','return':'List[ImSet]'}

Such a__future__ import statement may be proposed in a separatePEP.

python/typing#400

The problem was discussed at length on the typing module’s GitHubproject, underIssue 400.The problem statement there includes critique of generic types requiringimports fromtyping. This tends to be confusing tobeginners:

Why this:

fromtypingimportList,Setdefdir(o:object=...)->List[str]:...defadd_friends(friends:Set[Friend])->None:...

But not this:

defdir(o:object=...)->list[str]:...defadd_friends(friends:set[Friend])->None...

Why this:

up_to_ten=list(range(10))friends=set()

But not this:

fromtypingimportList,Setup_to_ten=List[int](range(10))friends=Set[Friend]()

While typing usability is an interesting problem, it is out of scopeof this PEP. Specifically, any extensions of the typing syntaxstandardized inPEP 484 will require their own respective PEPs andapproval.

Issue 400 ultimately suggests postponing evaluation of annotations andkeeping them as strings in__annotations__, just like this PEPspecifies. This idea was received well. Ivan Levkivskyi supportedusing the__future__ import and suggested unparsing the AST incompile.c. Jukka Lehtosalo pointed out that there are some casesof forward references where types are used outside of annotations andpostponed evaluation will not help those. For those cases using thestring literal notation would still be required. Those cases arediscussed briefly in the “Forward References” section of this PEP.

The biggest controversy on the issue was Guido van Rossum’s concernthat untokenizing annotation expressions back to their string form hasno precedent in the Python programming language and feels like a hackyworkaround. He said:

One thing that comes to mind is that it’s a very random change tothe language. It might be useful to have a more compact way toindicate deferred execution of expressions (using less syntax thanlambda:). But why would the use case of type annotations be soall-important to change the language to do it there first (ratherthan proposing a more general solution), given that there’s alreadya solution for this particular use case that requires very minimalsyntax?

Eventually, Ethan Smith and schollii voiced that feedback gatheredduring PyCon US suggests that the state of forward references needsfixing. Guido van Rossum suggested coming back to the__future__idea, pointing out that to prevent abuse, it’s important for theannotations to be kept both syntactically valid and evaluating correctlyat runtime.

First draft discussion on python-ideas

Discussion happened largely in two threads,the original announcementand a follow-up calledPEP 563 and expensive backwards compatibility.

The PEP received rather warm feedback (4 strongly in favor,2 in favor with concerns, 2 against). The biggest voice of concern onthe former thread being Steven D’Aprano’s review stating that theproblem definition of the PEP doesn’t justify breaking backwardscompatibility. In this response Steven seemed mostly concerned aboutPython no longer supporting evaluation of annotations that depended onlocal function/class state.

A few people voiced concerns that there are libraries using annotationsfor non-typing purposes. However, none of the named libraries would beinvalidated by this PEP. They do require adapting to the newrequirement to calleval() on the annotation with the correctglobals andlocals set.

This detail aboutglobals andlocals having to be correct waspicked up by a number of commenters. Alyssa (Nick) Coghlan benchmarked turningannotations into lambdas instead of strings, sadly this proved to bemuch slower at runtime than the current situation.

The latter thread was started by Jim J. Jewett who stressed thatthe ability to properly evaluate annotations is an important requirementand backwards compatibility in that regard is valuable. After somediscussion he admitted that side effects in annotations are a code smelland modal support to either perform or not perform evaluation isa messy solution. His biggest concern remained loss of functionalitystemming from the evaluation restrictions on global and local scope.

Alyssa Coghlan pointed out that some of those evaluation restrictions fromthe PEP could be lifted by a clever implementation of an evaluationhelper, which could solve self-referencing classes even in the form of aclass decorator. She suggested the PEP should provide this helperfunction in the standard library.

Second draft discussion on python-dev

Discussion happened mainly in theannouncement thread,followed by a brief discussion underMark Shannon’s post.

Steven D’Aprano was concerned whether it’s acceptable for typos to beallowed in annotations after the change proposed by the PEP. BrettCannon responded that type checkers and other static analyzers (likelinters or programming text editors) will catch this type of error.Jukka Lehtosalo added that this situation is analogous to how names infunction bodies are not resolved until the function is called.

A major topic of discussion was Alyssa Coghlan’s suggestion to storeannotations in “thunk form”, in other words as a specialized lambdawhich would be able to access class-level scope (and allow for scopecustomization at call time). He presented a possible design for it(indirect attribute cells).This was later seen as equivalent to “special forms” in Lisp. Guido vanRossum expressed worry that this sort of feature cannot be safelyimplemented in twelve weeks (i.e. in time before the Python 3.7 betafreeze).

After a while it became clear that the point of division betweensupporters of the string form vs. supporters of the thunk form isactually about whether annotations should be perceived as a generalsyntactic element vs. something tied to the type checking use case.

Finally, Guido van Rossum declared he’s rejecting the thunk ideabased on the fact that it would require a new building block in theinterpreter. This block would be exposed in annotations, multiplyingpossible types of values stored in__annotations__ (arbitraryobjects, strings, and now thunks). Moreover, thunks aren’t asintrospectable as strings. Most importantly, Guido van Rossumexplicitly stated interest in gradually restricting the use ofannotations to static typing (with an optional runtime component).

Alyssa Coghlan got convinced toPEP 563, too, promptly beginningthe mandatory bike shedding session on the name of the__future__import. Many debaters agreed thatannotations seems likean overly broad name for the feature name. Guido van Rossum brieflydecided to call itstring_annotations but then changed his mind,arguing thatdivision is a precedent of a broad name with a clearmeaning.

The final improvement to the PEP suggested in the discussion by MarkShannon was the rejection of the temptation to pass string literalsthrough to__annotations__ verbatim.

A side-thread of discussion started around the runtime penalty ofstatic typing, with topic like the import time of thetypingmodule (which is comparable tore without dependencies, andthree times as heavy asre when counting dependencies).