Abstract

Generators allow for natural coding and abstraction of traversalover data. Currently if external resources needing proper timelyrelease are involved, generators are unfortunately not adequate.The typical idiom for timely release is not supported, a yieldstatement is not allowed in the try clause of a try-finallystatement inside a generator. The finally clause execution can beneither guaranteed nor enforced.

This PEP proposes that the built-in generator type implement aclose method and destruction semantics, such that the restrictionon yield placement can be lifted, expanding the applicability ofgenerators.

Pronouncement

Rejected in favor ofPEP 342 which includes substantially all ofthe requested behavior in a more refined form.

Rationale

Python generators allow for natural coding of many data traversalscenarios. Their instantiation produces iterators,i.e. first-class objects abstracting traversal (with all theadvantages of first- classness). In this respect they match inpower and offer some advantages over the approach using iteratormethods taking a (smalltalkish) block. On the other hand, givencurrent limitations (no yield allowed in a try clause of atry-finally inside a generator) the latter approach seems bettersuited to encapsulating not only traversal but also exceptionhandling and proper resource acquisition and release.

Let’s consider an example (for simplicity, files in read-mode areused):

defall_lines(index_path):forpathinfile(index_path,"r"):forlineinfile(path.strip(),"r"):yieldline

this is short and to the point, but the try-finally for timelyclosing of the files cannot be added. (While instead of a path, afile, whose closing then would be responsibility of the caller,could be passed in as argument, the same is not applicable for thefiles opened depending on the contents of the index).

If we want timely release, we have to sacrifice the simplicity anddirectness of the generator-only approach: (e.g.)

classAllLines:def__init__(self,index_path):self.index_path=index_pathself.index=Noneself.document=Nonedef__iter__(self):self.index=file(self.index_path,"r")forpathinself.index:self.document=file(path.strip(),"r")forlineinself.document:yieldlineself.document.close()self.document=Nonedefclose(self):ifself.index:self.index.close()ifself.document:self.document.close()

to be used as:

all_lines=AllLines("index.txt")try:forlineinall_lines:...finally:all_lines.close()

The more convoluted solution implementing timely release, seemsto offer a precious hint. What we have done is encapsulate ourtraversal in an object (iterator) with a close method.

This PEP proposes that generators should grow such a close methodwith such semantics that the example could be rewritten as:

# Today this is not valid Python: yield is not allowed between# try and finally, and generator type instances support no# close method.defall_lines(index_path):index=file(index_path,"r")try:forpathinindex:document=file(path.strip(),"r")try:forlineindocument:yieldlinefinally:document.close()finally:index.close()all=all_lines("index.txt")try:forlineinall:...finally:all.close()# close on generator

CurrentlyPEP 255 disallows yield inside a try clause of atry-finally statement, because the execution of the finally clausecannot be guaranteed as required by try-finally semantics.

The semantics of the proposed close method should be such thatwhile the finally clause execution still cannot be guaranteed, itcan be enforced when required. Specifically, the close methodbehavior should trigger the execution of the finally clausesinside the generator, either by forcing a return in the generatorframe or by throwing an exception in it. In situations requiringtimely resource release, close could then be explicitly invoked.

The semantics of generator destruction on the other hand should beextended in order to implement a best-effort policy for thegeneral case. Specifically, destruction should invokeclose().The best-effort limitation comes from the fact that thedestructor’s execution is not guaranteed in the first place.

This seems to be a reasonable compromise, the resulting globalbehavior being similar to that of files and closing.

Possible Semantics

The built-in generator type should have a close methodimplemented, which can then be invoked as:

gen.close()

wheregen is an instance of the built-in generator type.Generator destruction should also invoke close method behavior.

If a generator is already terminated, close should be a no-op.

Otherwise, there are two alternative solutions, Return orException Semantics:

A - Return Semantics: The generator should be resumed, generatorexecution should continue as if the instruction at the re-entrypoint is a return. Consequently, finally clauses surrounding there-entry point would be executed, in the case of a then allowedtry-yield-finally pattern.

Issues: is it important to be able to distinguish forcedtermination by close, normal termination, exception propagationfrom generator or generator-called code? In the normal case itseems not, finally clauses should be there to work the same in allthese cases, still this semantics could make such a distinctionhard.

Except-clauses, like by a normal return, are not executed, suchclauses in legacy generators expect to be executed for exceptionsraised by the generator or by code called from it. Not executingthem in the close case seems correct.

B - Exception Semantics: The generator should be resumed andexecution should continue as if a special-purpose exception(e.g. CloseGenerator) has been raised at re-entry point. Closeimplementation should consume and not propagate further thisexception.

Issues: shouldStopIteration be reused for this purpose? Probablynot. We would like close to be a harmless operation for legacygenerators, which could contain code catchingStopIteration todeal with other generators/iterators.

In general, with exception semantics, it is unclear what to do ifthe generator does not terminate or we do not receive the specialexception propagated back. Other different exceptions shouldprobably be propagated, but consider this possible legacygenerator code:

try:...yield......except:# or except Exception:, etcraiseException("boom")

If close is invoked with the generator suspended after the yield,the except clause would catch our special purpose exception, so wewould get a different exception propagated back, which in thiscase ought to be reasonably consumed and ignored but in generalshould be propagated, but separating these scenarios seems hard.

The exception approach has the advantage to let the generatordistinguish between termination cases and have more control. Onthe other hand, clear-cut semantics seem harder to define.

Remarks

If this proposal is accepted, it should become common practice todocument whether a generator acquires resources, so that its closemethod ought to be called. If a generator is no longer used,calling close should be harmless.

On the other hand, in the typical scenario the code thatinstantiated the generator should call close if required by it.Generic code dealing with iterators/generators instantiatedelsewhere should typically not be littered with close calls.

The rare case of code that has acquired ownership of and need toproperly deal with all of iterators, generators and generatorsacquiring resources that need timely release, is easily solved:

ifhasattr(iterator,'close'):iterator.close()

Open Issues

Definitive semantics ought to be chosen. Currently Guido favorsException Semantics. If the generator yields a value instead ofterminating, or propagating back the special exception, a specialexception should be raised again on the generator side.

It is still unclear whether spuriously converted specialexceptions (as discussed in Possible Semantics) are a problem andwhat to do about them.

Implementation issues should be explored.

Alternative Ideas

The idea that the yield placement limitation should be removed andthat generator destruction should trigger execution of finallyclauses has been proposed more than once. Alone it cannotguarantee that timely release of resources acquired by a generatorcan be enforced.

PEP 288 proposes a more general solution, allowing customexception passing to generators. The proposal in this PEPaddresses more directly the problem of resource release. WerePEP 288 implemented, Exceptions Semantics for close could be layeredon top of it, on the other handPEP 288 should make a separatecase for the more general functionality.

Copyright

This document has been placed in the public domain.