Introduction

This PEP proposes a new type of compound statement which can beused for resource management purposes. The new statement typeis provisionally called the block-statement because the keywordto be used has not yet been chosen.

This PEP competes with several other PEPs:PEP 288 (GeneratorsAttributes and Exceptions; only the second part),PEP 310(Reliable Acquisition/Release Pairs), andPEP 325(Resource-Release Support for Generators).

I should clarify that using a generator to “drive” a blockstatement is really a separable proposal; with just the definitionof the block statement from the PEP you could implement all theexamples using a class (similar to example 6, which is easilyturned into a template). But the key idea is using a generator todrive a block statement; the rest is elaboration, so I’d like tokeep these two parts together.

(PEP 342, Enhanced Iterators, was originally a part of this PEP;but the two proposals are really independent and with StevenBethard’s help I have moved it to a separate PEP.)

Rejection Notice

I am rejecting this PEP in favor ofPEP 343. See the motivationalsection in that PEP for the reasoning behind this rejection. GvR.

Motivation and Summary

(Thanks to Shane Hathaway – Hi Shane!)

Good programmers move commonly used code into reusable functions.Sometimes, however, patterns arise in the structure of thefunctions rather than the actual sequence of statements. Forexample, many functions acquire a lock, execute some code specificto that function, and unconditionally release the lock. Repeatingthe locking code in every function that uses it is error prone andmakes refactoring difficult.

Block statements provide a mechanism for encapsulating patterns ofstructure. Code inside the block statement runs under the controlof an object called a block iterator. Simple block iteratorsexecute code before and after the code inside the block statement.Block iterators also have the opportunity to execute thecontrolled code more than once (or not at all), catch exceptions,or receive data from the body of the block statement.

A convenient way to write block iterators is to write a generator(PEP 255). A generator looks a lot like a Python function, butinstead of returning a value immediately, generators pause theirexecution at “yield” statements. When a generator is used as ablock iterator, the yield statement tells the Python interpreterto suspend the block iterator, execute the block statement body,and resume the block iterator when the body has executed.

The Python interpreter behaves as follows when it encounters ablock statement based on a generator. First, the interpreterinstantiates the generator and begins executing it. The generatordoes setup work appropriate to the pattern it encapsulates, suchas acquiring a lock, opening a file, starting a databasetransaction, or starting a loop. Then the generator yieldsexecution to the body of the block statement using a yieldstatement. When the block statement body completes, raises anuncaught exception, or sends data back to the generator using acontinue statement, the generator resumes. At this point, thegenerator can either clean up and stop or yield again, causing theblock statement body to execute again. When the generatorfinishes, the interpreter leaves the block statement.

Use Cases

See the Examples section near the end.

Specification: the __exit__() Method

An optional new method for iterators is proposed, called__exit__(). It takes up to three arguments which correspond tothe three “arguments” to the raise-statement: type, value, andtraceback. If all three arguments areNone,sys.exc_info() may beconsulted to provide suitable default values.

Specification: the Anonymous Block Statement

A new statement is proposed with the syntax:

blockEXPR1asVAR1:BLOCK1

Here, ‘block’ and ‘as’ are new keywords;EXPR1 is an arbitraryexpression (but not an expression-list) andVAR1 is an arbitraryassignment target (which may be a comma-separated list).

The “as VAR1” part is optional; if omitted, the assignments toVAR1 in the translation below are omitted (but the expressionsassigned are still evaluated!).

The choice of the ‘block’ keyword is contentious; manyalternatives have been proposed, including not to use a keyword atall (which I actually like).PEP 310 uses ‘with’ for similarsemantics, but I would like to reserve that for a with-statementsimilar to the one found in Pascal and VB. (Though I just foundthat the C# designers don’t like ‘with’[2], and I have to agreewith their reasoning.) To sidestep this issue momentarily I’musing ‘block’ until we can agree on the right keyword, if any.

Note that the ‘as’ keyword is not contentious (it will finally beelevated to proper keyword status).

Note that it is up to the iterator to decide whether ablock-statement represents a loop with multiple iterations; in themost common use caseBLOCK1 is executed exactly once. To theparser, however, it is always a loop; break and continue returntransfer to the block’s iterator (see below for details).

The translation is subtly different from a for-loop:iter() isnot called, soEXPR1 should already be an iterator (not just aniterable); and the iterator is guaranteed to be notified whenthe block-statement is left, regardless if this is due to abreak, return or exception:

itr=EXPR1# The iteratorret=False# True if a return statement is activeval=None# Return value, if ret == Trueexc=None# sys.exc_info() tuple if an exception is activewhileTrue:try:ifexc:ext=getattr(itr,"__exit__",None)ifextisnotNone:VAR1=ext(*exc)# May re-raise *excelse:raiseexc[0],exc[1],exc[2]else:VAR1=itr.next()# May raise StopIterationexceptStopIteration:ifret:returnvalbreaktry:ret=Falseval=exc=NoneBLOCK1except:exc=sys.exc_info()

(However, the variables ‘itr’ etc. are not user-visible and thebuilt-in names used cannot be overridden by the user.)

InsideBLOCK1, the following special translations apply:

• “break” is always legal; it is translated into:

  exc=(StopIteration,None,None)continue

• “return EXPR3” is only legal when the block-statement iscontained in a function definition; it is translated into:

  exc=(StopIteration,None,None)ret=Trueval=EXPR3continue

The net effect is that break and return behave much the same asif the block-statement were a for-loop, except that the iteratorgets a chance at resource cleanup before the block-statement isleft, through the optional__exit__() method. The iterator alsogets a chance if the block-statement is left through raising anexception. If the iterator doesn’t have an__exit__() method,there is no difference with a for-loop (except that a for-loopcallsiter() onEXPR1).

Note that a yield-statement in a block-statement is not treateddifferently. It suspends the function containing the blockwithout notifying the block’s iterator. The block’s iterator isentirely unaware of this yield, since the local control flowdoesn’t actually leave the block. In other words, it isnotlike a break or return statement. When the loop that was resumedby the yield callsnext(), the block is resumed right after theyield. (See example 7 below.) The generator finalizationsemantics described below guarantee (within the limitations of allfinalization semantics) that the block will be resumed eventually.

Unlike the for-loop, the block-statement does not have anelse-clause. I think it would be confusing, and emphasize the“loopiness” of the block-statement, while I want to emphasize itsdifference from a for-loop. In addition, there are severalpossible semantics for an else-clause, and only a very weak usecase.

Specification: Generator Exit Handling

Generators will implement the new__exit__() method API.

Generators will be allowed to have ayield statement inside atry-finally statement.

The expression argument to the yield-statement will becomeoptional (defaulting to None).

When__exit__() is called, the generator is resumed but at thepoint of the yield-statement the exception represented by the__exit__ argument(s) is raised. The generator may re-raise thisexception, raise another exception, or yield another value,except that if the exception passed in to__exit__() wasStopIteration, it ought to raise StopIteration (otherwise theeffect would be that a break is turned into continue, which isunexpected at least). When theinitial call resuming thegenerator is an__exit__() call instead of anext() call, thegenerator’s execution is aborted and the exception is re-raisedwithout passing control to the generator’s body.

When a generator that has not yet terminated is garbage-collected(either through reference counting or by the cyclical garbagecollector), its__exit__() method is called once withStopIteration as its first argument. Together with therequirement that a generator ought to raise StopIteration when__exit__() is called with StopIteration, this guarantees theeventual activation of any finally-clauses that were active whenthe generator was last suspended. Of course, under certaincircumstances the generator may never be garbage-collected. Thisis no different than the guarantees that are made about finalizers(__del__() methods) of other objects.

Alternatives Considered and Rejected

• Many alternatives have been proposed for ‘block’. I haven’tseen a proposal for another keyword that I like better than‘block’ yet. Alas, ‘block’ is also not a good choice; it is arather popular name for variables, arguments and methods.Perhaps ‘with’ is the best choice after all?
• Instead of trying to pick the ideal keyword, the block-statementcould simply have the form:

  EXPR1asVAR1:BLOCK1

  This is at first attractive because, together with a good choiceof function names (like those in the Examples section below)used inEXPR1, it reads well, and feels like a “user-definedstatement”. And yet, it makes me (and many others)uncomfortable; without a keyword the syntax is very “bland”,difficult to look up in a manual (remember that ‘as’ isoptional), and it makes the meaning of break and continue in theblock-statement even more confusing.

• Phillip Eby has proposed to have the block-statement usean entirely different API than the for-loop, to differentiatebetween the two. A generator would have to be wrapped in adecorator to make it support the block API. IMO this adds morecomplexity with very little benefit; and we can’t really denythat the block-statement is conceptually a loop – it supportsbreak and continue, after all.
• This keeps getting proposed: “block VAR1 = EXPR1” instead of“block EXPR1 as VAR1”. That would be very misleading, sinceVAR1 doesnot get assigned the value of EXPR1; EXPR1 resultsin a generator which is assigned to an internal variable, andVAR1 is the value returned by successive calls to the__next__()method of that iterator.
• Why not change the translation to applyiter(EXPR1)? All theexamples would continue to work. But this makes theblock-statementmore like a for-loop, while the emphasis oughtto be on thedifference between the two. Not callingiter()catches a bunch of misunderstandings, like using a sequence asEXPR1.

Comparison to Thunks

Alternative semantics proposed for the block-statement turn theblock into a thunk (an anonymous function that blends into thecontaining scope).

The main advantage of thunks that I can see is that you can savethe thunk for later, like a callback for a button widget (thethunk then becomes a closure). You can’t use a yield-based blockfor that (except in Ruby, which uses yield syntax with athunk-based implementation). But I have to say that I almost seethis as an advantage: I think I’d be slightly uncomfortable seeinga block and not knowing whether it will be executed in the normalcontrol flow or later. Defining an explicit nested function forthat purpose doesn’t have this problem for me, because I alreadyknow that the ‘def’ keyword means its body is executed later.

The other problem with thunks is that once we think of them as theanonymous functions they are, we’re pretty much forced to say thata return statement in a thunk returns from the thunk rather thanfrom the containing function. Doing it any other way would causemajor weirdness when the thunk were to survive its containingfunction as a closure (perhaps continuations would help, but I’mnot about to go there :-).

But then an IMO important use case for the resource cleanuptemplate pattern is lost. I routinely write code like this:

deffindSomething(self,key,default=None):self.lock.acquire()try:foriteminself.elements:ifitem.matches(key):returnitemreturndefaultfinally:self.lock.release()

and I’d be bummed if I couldn’t write this as:

deffindSomething(self,key,default=None):blocklocking(self.lock):foriteminself.elements:ifitem.matches(key):returnitemreturndefault

This particular example can be rewritten using a break:

deffindSomething(self,key,default=None):blocklocking(self.lock):foriteminself.elements:ifitem.matches(key):breakelse:item=defaultreturnitem

but it looks forced and the transformation isn’t always that easy;you’d be forced to rewrite your code in a single-return stylewhich feels too restrictive.

Also note the semantic conundrum of a yield in a thunk – the onlyreasonable interpretation is that this turns the thunk into agenerator!

Greg Ewing believes that thunks “would be a lot simpler, doingjust what is required without any jiggery pokery with exceptionsand break/continue/return statements. It would be easy to explainwhat it does and why it’s useful.”

But in order to obtain the required local variable sharing betweenthe thunk and the containing function, every local variable usedor set in the thunk would have to become a ‘cell’ (our mechanismfor sharing variables between nested scopes). Cells slow downaccess compared to regular local variables: access involves anextra C function call (PyCell_Get() orPyCell_Set()).

Perhaps not entirely coincidentally, the last example above(findSomething() rewritten to avoid a return inside the block)shows that, unlike for regular nested functions, we’ll wantvariablesassigned to by the thunk also to be shared with thecontaining function, even if they are not assigned to outside thethunk.

Greg Ewing again: “generators have turned out to be more powerful,because you can have more than one of them on the go at once. Isthere a use for that capability here?”

I believe there are definitely uses for this; several people havealready shown how to do asynchronous light-weight threads usinggenerators (e.g. David Mertz quoted inPEP 288, and FredrikLundh[3]).

And finally, Greg says: “a thunk implementation has the potentialto easily handle multiple block arguments, if a suitable syntaxcould ever be devised. It’s hard to see how that could be done ina general way with the generator implementation.”

However, the use cases for multiple blocks seem elusive.

(Proposals have since been made to change the implementation ofthunks to remove most of these objections, but the resultingsemantics are fairly complex to explain and to implement, so IMOthat defeats the purpose of using thunks in the first place.)

Examples

(Several of these examples contain “yield None”. IfPEP 342 isaccepted, these can be changed to just “yield” of course.)

1. A template for ensuring that a lock, acquired at the start of ablock, is released when the block is left:

   deflocking(lock):lock.acquire()try:yieldNonefinally:lock.release()

   Used as follows:

   blocklocking(myLock):# Code here executes with myLock held.  The lock is# guaranteed to be released when the block is left (even# if via return or by an uncaught exception).

2. A template for opening a file that ensures the file is closedwhen the block is left:

   defopening(filename,mode="r"):f=open(filename,mode)try:yieldffinally:f.close()

   Used as follows:

   blockopening("/etc/passwd")asf:forlineinf:printline.rstrip()

3. A template for committing or rolling back a databasetransaction:

   deftransactional(db):try:yieldNoneexcept:db.rollback()raiseelse:db.commit()

4. A template that tries something up to n times:

   defauto_retry(n=3,exc=Exception):foriinrange(n):try:yieldNonereturnexceptexc,err:# perhaps log exception herecontinueraise# re-raise the exception we caught earlier

   Used as follows:

   blockauto_retry(3,IOError):f=urllib.urlopen("https://www.example.com/")printf.read()

5. It is possible to nest blocks and combine templates:

   deflocking_opening(lock,filename,mode="r"):blocklocking(lock):blockopening(filename)asf:yieldf

   Used as follows:

   blocklocking_opening(myLock,"/etc/passwd")asf:forlineinf:printline.rstrip()

   (If this example confuses you, consider that it is equivalentto using a for-loop with a yield in its body in a regulargenerator which is invoking another iterator or generatorrecursively; see for example the source code foros.walk().)

6. It is possible to write a regular iterator with thesemantics of example 1:

   classlocking:def__init__(self,lock):self.lock=lockself.state=0def__next__(self,arg=None):# ignores argifself.state:assertself.state==1self.lock.release()self.state+=1raiseStopIterationelse:self.lock.acquire()self.state+=1returnNonedef__exit__(self,type,value=None,traceback=None):assertself.statein(0,1,2)ifself.state==1:self.lock.release()raisetype,value,traceback

   (This example is easily modified to implement the otherexamples; it shows how much simpler generators are for the samepurpose.)

7. Redirect stdout temporarily:

   defredirecting_stdout(new_stdout):save_stdout=sys.stdouttry:sys.stdout=new_stdoutyieldNonefinally:sys.stdout=save_stdout

   Used as follows:

   blockopening(filename,"w")asf:blockredirecting_stdout(f):print"Hello world"

8. A variant onopening() that also returns an error condition:

   defopening_w_error(filename,mode="r"):try:f=open(filename,mode)exceptIOError,err:yieldNone,errelse:try:yieldf,Nonefinally:f.close()

   Used as follows:

   blockopening_w_error("/etc/passwd","a")asf,err:iferr:print"IOError:",errelse:f.write("guido::0:0::/:/bin/sh\n")

Acknowledgements

In no useful order: Alex Martelli, Barry Warsaw, Bob Ippolito,Brett Cannon, Brian Sabbey, Chris Ryland, Doug Landauer, DuncanBooth, Fredrik Lundh, Greg Ewing, Holger Krekel, Jason Diamond,Jim Jewett, Josiah Carlson, Ka-Ping Yee, Michael Chermside,Michael Hudson, Neil Schemenauer, Alyssa Coghlan, Paul Moore,Phillip Eby, Raymond Hettinger, Georg Brandl, SamuelePedroni, Shannon Behrens, Skip Montanaro, Steven Bethard, TerryReedy, Tim Delaney, Aahz, and others. Thanks all for the valuablecontributions!

References

[1]https://mail.python.org/pipermail/python-dev/2005-April/052821.html

Copyright

This document has been placed in the public domain.