This PEP proposes the addition of a new@in decorator clause that makesit possible to override the name binding step of a function or classdefinition.
The new clause accepts a single simple statement that can make a forwardreference to decorated function or class definition.
This new clause is designed to be used whenever a “one-shot” function orclass is needed, and placing the function or class definition before thestatement that uses it actually makes the code harder to read. It alsoavoids any name shadowing concerns by making sure the new name is visibleonly to the statement in the@in clause.
This PEP is based heavily on many of the ideas inPEP 3150 (Statement LocalNamespaces) so some elements of the rationale will be familiar to readers ofthat PEP. Both PEPs remain deferred for the time being, primarily due to thelack of compelling real world use cases in either PEP.
Before diving into the long history of this problem and the detailedrationale for this specific proposed solution, here are a few simpleexamples of the kind of code it is designed to simplify.
As a trivial example, a weakref callback could be defined as follows:
@inx=weakref.ref(target,report_destruction)defreport_destruction(obj):print("{} is being destroyed".format(obj))
This contrasts with the current (conceptually) “out of order” syntax forthis operation:
defreport_destruction(obj):print("{} is being destroyed".format(obj))x=weakref.ref(target,report_destruction)
That structure is OK when you’re using the callable multiple times, butit’s irritating to be forced into it for one-off operations.
If the repetition of the name seems especially annoying, then a throwawayname likef can be used instead:
@inx=weakref.ref(target,f)deff(obj):print("{} is being destroyed".format(obj))
Similarly, a sorted operation on a particularly poorly defined type couldnow be defined as:
@insorted_list=sorted(original,key=f)deff(item):try:returnitem.calc_sort_order()exceptNotSortableError:returnfloat('inf')
Rather than:
defforce_sort(item):try:returnitem.calc_sort_order()exceptNotSortableError:returnfloat('inf')sorted_list=sorted(original,key=force_sort)
And early binding semantics in a list comprehension could be attained via:
@infuncs=[adder(i)foriinrange(10)]defadder(i):returnlambdax:x+i
This PEP proposes the addition of a new@in clause that is a variantof the existing class and function decorator syntax.
The new@in clause precedes the decorator lines, and allows forwardreferences to the trailing function or class definition.
The trailing function or class definition is always named - the name ofthe trailing definition is then used to make the forward reference from the@in clause.
The@in clause is allowed to contain any simple statement (includingthose that don’t make any sense in that context, such aspass - whilesuch code would be legal, there wouldn’t be any point in writing it). Thispermissive structure is easier to define and easier to explain, but a morerestrictive approach that only permits operations that “make sense” wouldalso be possible (seePEP 3150 for a list of possible candidates).
The@in clause will not create a new scope - all name bindingoperations aside from the trailing function or class definition will affectthe containing scope.
The name used in the trailing function or class definition is only visiblefrom the associated@in clause, and behaves as if it was an ordinaryvariable defined in that scope. If any nested scopes are created in eitherthe@in clause or the trailing function or class definition, those scopeswill see the trailing function or class definition rather than any otherbindings for that name in the containing scope.
In a very real sense, this proposal is about making it possible to overridethe implicit “name = <defined function or class>” name binding operationthat is part of every function or class definition, specifically in thosecases where the local name binding isn’t actually needed.
Under this PEP, an ordinary class or function definition:
@deco2@deco1defname():...
can be explained as being roughly equivalent to:
@inname=deco2(deco1(name))defname():...
Syntactically, only one new grammar rule is needed:
in_stmt:'@in'simple_stmtdecorated
Grammar:http://hg.python.org/cpython/file/default/Grammar/Grammar
The question of “multi-line lambdas” has been a vexing one for manyPython users for a very long time, and it took an exploration of Ruby’sblock functionality for me to finally understand why this bugs peopleso much: Python’s demand that the function be named and introducedbefore the operation that needs it breaks the developer’s flow of thought.They get to a point where they go “I need a one-shot operation that does<X>”, and instead of being able to justsay that directly, they insteadhave to back up, name a function to do <X>, then call that function fromthe operation they actually wanted to do in the first place. Lambdaexpressions can help sometimes, but they’re no substitute for being able touse a full suite.
Ruby’s block syntax also heavily inspired the style of the solution in thisPEP, by making it clear that even when limited toone anonymous function perstatement, anonymous functions could still be incredibly useful. Consider howmany constructs Python has where one expression is responsible for the bulk ofthe heavy lifting:
However, adopting Ruby’s block syntax directly won’t work for Python, sincethe effectiveness of Ruby’s blocks relies heavily on various conventions inthe way functions aredefined (specifically, using Ruby’syield syntaxto call blocks directly and the&arg mechanism to accept a block as afunction’s final argument).
Since Python has relied on named functions for so long, the signatures ofAPIs that accept callbacks are far more diverse, thus requiring a solutionthat allows one-shot functions to be slotted in at the appropriate location.
The approach taken in this PEP is to retain the requirement to name thefunction explicitly, but allow the relative order of the definition and thestatement that references it to be changed to match the developer’s flow ofthought. The rationale is essentially the same as that used when introducingdecorators, but covering a broader set of applications.
PEP 3150 (Statement Local Namespaces) describes its primary motivationas being to elevate ordinary assignment statements to be on par withclassanddef statements where the name of the item to be defined is presentedto the reader in advance of the details of how the value of that item iscalculated. This PEP achieves the same goal in a different way, by allowingthe simple name binding of a standard function definition to be replacedwith something else (like assigning the result of the function to a value).
Despite having the same author, the two PEPs are in direct competition witheach other.PEP 403 represents a minimalist approach that attempts to achieveuseful functionality with a minimum of change from the status quo. This PEPinstead aims for a more flexible standalone statement design, which requiresa larger degree of change to the language.
Note that wherePEP 403 is better suited to explaining the behaviour ofgenerator expressions correctly, this PEP is better able to explain thebehaviour of decorator clauses in general. Both PEPs support adequateexplanations for the semantics of container comprehensions.
The proposal definitely requiressome kind of prefix to avoid parsingambiguity and backwards compatibility problems with existing constructs.It also needs to be clearly highlighted to readers, since it declares thatthe following piece of code is going to be executed only after the trailingfunction or class definition has been executed.
Thein keyword was chosen as an existing keyword that can be used todenote the concept of a forward reference.
The@ prefix was included in order to exploit the fact that Pythonprogrammers are already used to decorator syntax as an indication ofout of order execution, where the function or class is actually definedfirst and then decorators are applied in reverse order.
For functions, the construct is intended to be read as “in <this statementthat references NAME> define NAME as a function that does <operation>”.
The mapping to English prose isn’t as obvious for the class definition case,but the concept remains the same.
One of the objections to widespread use of lambda expressions is that theyhave a negative effect on traceback intelligibility and other aspects ofintrospection. Similar objections are raised regarding constructs thatpromote short, cryptic function names (including this one, which requiresthat the name of the trailing definition be supplied at least twice,encouraging the use of shorthand placeholder names likef).
However, the introduction of qualified names inPEP 3155 means that evenanonymous classes and functions will now have different representations ifthey occur in different scopes. For example:
>>>deff():...returnlambda:y...>>>f()<function f.<locals>.<lambda> at 0x7f6f46faeae0>
Anonymous functions (or functions that share a name) within thesame scopewill still share representations (aside from the object ID), but this isstill a major improvement over the historical situation where everythingexcept the object ID was identical.
This proposal has at least one titanic advantage overPEP 3150:implementation should be relatively straightforward.
The@in clause will be included in the AST for the associated function orclass definition and the statement that references it. When the@inclause is present, it will be emitted in place of the local name bindingoperation normally implied by a function or class definition.
The one potentially tricky part is changing the meaning of the references tothe statement local function or namespace while within the scope of thein statement, but that shouldn’t be too hard to address by maintainingsome additional state within the compiler (it’s much easier to handle thisfor a single name than it is for an unknown number of names in a fullnested suite).
One interesting feature of the proposed construct is that it can be used asa primitive to explain the scoping and execution order semantics ofboth generator expressions and container comprehensions:
seq2=[xforxinyifq(x)foryinseqifp(y)]# would be equivalent to@inseq2=f(seq):deff(seq)result=[]foryinseq:ifp(y):forxiny:ifq(x):result.append(x)returnresult
The important point in this expansion is that it explains why comprehensionsappear to misbehave at class scope: only the outermost iterator is evaluatedat class scope, while all predicates, nested iterators and value expressionsare evaluated inside a nested scope.
An equivalent expansion is possible for generator expressions:
gen=(xforxinyifq(x)foryinseqifp(y))# would be equivalent to@ingen=g(seq):defg(seq)foryinseq:ifp(y):forxiny:ifq(x):yieldx
Calculating attributes without polluting the local namespace (from os.py):
# Current Python (manual namespace cleanup)def_createenviron():...# 27 line functionenviron=_createenviron()del_createenviron# Becomes:@inenviron=_createenviron()def_createenviron():...# 27 line function
Loop early binding:
# Current Python (default argument hack)funcs=[(lambdax,i=i:x+i)foriinrange(10)]# Becomes:@infuncs=[adder(i)foriinrange(10)]defadder(i):returnlambdax:x+i# Or even:@infuncs=[adder(i)foriinrange(10)]defadder(i):@inreturnincrdefincr(x):returnx+i
A trailing class can be used as a statement local namespace:
# Evaluate subexpressions only once@inc=math.sqrt(x.a*x.a+x.b*x.b)classx:a=calculate_a()b=calculate_b()
A function can be bound directly to a location which isn’t a valididentifier:
@indispatch[MyClass]=fdeff():...
Constructs that verge on decorator abuse can be eliminated:
# Current Python@calldeff():...# Becomes:@inf()deff():...
None as yet.
Huge thanks to Gary Bernhardt for being blunt in pointing out that I had noidea what I was talking about in criticising Ruby’s blocks, kicking off arather enlightening process of investigation.
To avoid retreading previously covered ground, some rejected alternativesare documented in this section.
Earlier versions of this proposal omitted the@ prefix. However, withoutthat prefix, the barein keyword didn’t associate the clause stronglyenough with the subsequent function or class definition. Reusing thedecorator prefix and explicitly characterising the new construct as a kindof decorator clause is intended to help users link the two concepts andsee them as two variants of the same idea.
A previous incarnation of this PEP (see[1]) proposed a syntax where thenew clause was introduced with: and the forward reference was writtenusing@. Feedback on this variant was almost universallynegative, as it was considered both ugly and excessively magical:
:x=weakref.ref(target,@)defreport_destruction(obj):print("{} is being destroyed".format(obj))
A more recent variant always used... for forward references, alongwith genuinely anonymous function and class definitions. However, thisdegenerated quickly into a mass of unintelligible dots in more complexcases:
in funcs = [...(i) for i in range(10)]def ...(i): in return ... def ...(x): return x + iin c = math.sqrt(....a*....a + ....b*....b)class ...: a = calculate_a() b = calculate_b()
The problems with using a full nested suite are best described inPEP 3150. It’s comparatively difficult to implement properly, the scopingsemantics are harder to explain and it creates quite a few situations wherethere are two ways to do it without clear guidelines for choosing betweenthem (as almost any construct that can be expressed with ordinary imperativecode could instead be expressed using a given statement). While the PEP doespropose some newPEP 8 guidelines to help address that last problem, thedifficulties in implementation are not so easily dealt with.
By contrast, the decorator inspired syntax in this PEP explicitly limits thenew feature to cases where it should actually improve readability, ratherthan harming it. As in the case of the original introduction of decorators,the idea of this new syntax is that if itcan be used (i.e. the local namebinding of the function is completely unnecessary) then it probablyshouldbe used.
Another possible variant of this idea is to keep the decorator basedsemantics of this PEP, while adopting the prettier syntax fromPEP 3150:
x=weakref.ref(target,report_destruction)given:defreport_destruction(obj):print("{} is being destroyed".format(obj))
There are a couple of problems with this approach. The main issue is thatthis syntax variant uses something that looks like a suite, but really isn’tone. A secondary concern is that it’s not clear how the compiler will knowwhich name(s) in the leading expression are forward references (althoughthat could potentially be addressed through a suitable definition of thesuite-that-is-not-a-suite in the language grammar).
However, a nested suite has not yet been ruled out completely. The latestversion ofPEP 3150 uses explicit forward reference and name bindingschemes that greatly simplify the semantics of the statement, and itdoes offer the advantage of allowing the definition of arbitrarysubexpressions rather than being restricted to a single function orclass definition.
This document has been placed in the public domain.
Source:https://github.com/python/peps/blob/main/peps/pep-0403.rst
Last modified:2025-02-01 08:59:27 GMT