A syntax is proposed for a generator to delegate part of itsoperations to another generator. This allows a section of codecontaining ‘yield’ to be factored out and placed in another generator.Additionally, the subgenerator is allowed to return with a value, andthe value is made available to the delegating generator.
The new syntax also opens up some opportunities for optimisation whenone generator re-yields values produced by another.
Guido officiallyaccepted the PEP on 26th June, 2011.
A Python generator is a form of coroutine, but has the limitation thatit can only yield to its immediate caller. This means that a piece ofcode containing ayield cannot be factored out and put into aseparate function in the same way as other code. Performing such afactoring causes the called function to itself become a generator, andit is necessary to explicitly iterate over this second generator andre-yield any values that it produces.
If yielding of values is the only concern, this can be performedwithout much difficulty using a loop such as
forving:yieldv
However, if the subgenerator is to interact properly with the callerin the case of calls tosend(),throw() andclose(),things become considerably more difficult. As will be seen later, thenecessary code is very complicated, and it is tricky to handle all thecorner cases correctly.
A new syntax will be proposed to address this issue. In the simplestuse cases, it will be equivalent to the above for-loop, but it willalso handle the full range of generator behaviour, and allow generatorcode to be refactored in a simple and straightforward way.
The following new expression syntax will be allowed in the body of agenerator:
yield from<expr>
where <expr> is an expression evaluating to an iterable, from which aniterator is extracted. The iterator is run to exhaustion, during whichtime it yields and receives values directly to or from the caller ofthe generator containing theyieldfrom expression (the“delegating generator”).
Furthermore, when the iterator is another generator, the subgeneratoris allowed to execute areturn statement with a value, and thatvalue becomes the value of theyieldfrom expression.
The full semantics of theyieldfrom expression can be describedin terms of the generator protocol as follows:
send() arepassed directly to the iterator. If the sent value is None, theiterator’s__next__() method is called. If the sent valueis not None, the iterator’ssend() method is called. If thecall raises StopIteration, the delegating generator is resumed.Any other exception is propagated to the delegating generator.throw() method of the iterator.If the call raises StopIteration, the delegating generator isresumed. Any other exception is propagated to the delegatinggenerator.close() method of the delegating generatoris called, then theclose() method of the iterator is calledif it has one. If this call results in an exception, it ispropagated to the delegating generator. Otherwise,GeneratorExit is raised in the delegating generator.yieldfrom expression is the first argumentto theStopIteration exception raised by the iterator whenit terminates.returnexpr in a generator causesStopIteration(expr) tobe raised upon exit from the generator.For convenience, theStopIteration exception will be given avalue attribute that holds its first argument, or None if thereare no arguments.
Python 3 syntax is used in this section.
RESULT=yield fromEXPR
is semantically equivalent to
_i=iter(EXPR)try:_y=next(_i)exceptStopIterationas_e:_r=_e.valueelse:while1:try:_s=yield_yexceptGeneratorExitas_e:try:_m=_i.closeexceptAttributeError:passelse:_m()raise_eexceptBaseExceptionas_e:_x=sys.exc_info()try:_m=_i.throwexceptAttributeError:raise_eelse:try:_y=_m(*_x)exceptStopIterationas_e:_r=_e.valuebreakelse:try:if_sisNone:_y=next(_i)else:_y=_i.send(_s)exceptStopIterationas_e:_r=_e.valuebreakRESULT=_r
returnvalue
is semantically equivalent to
raiseStopIteration(value)
except that, as currently, the exception cannot be caught byexcept clauses within the returning generator.
classStopIteration(Exception):def__init__(self,*args):iflen(args)>0:self.value=args[0]else:self.value=NoneException.__init__(self,*args)
The rationale behind most of the semantics presented above stems fromthe desire to be able to refactor generator code. It should bepossible to take a section of code containing one or moreyieldexpressions, move it into a separate function (using the usualtechniques to deal with references to variables in the surroundingscope, etc.), and call the new function using ayieldfromexpression.
The behaviour of the resulting compound generator should be, as far asreasonably practicable, the same as the original unfactored generatorin all situations, including calls to__next__(),send(),throw() andclose().
The semantics in cases of subiterators other than generators has beenchosen as a reasonable generalization of the generator case.
The proposed semantics have the following limitations with regard torefactoring:
With use cases for these being rare to non-existent, it was notconsidered worth the extra complexity required to support them.
There was some debate as to whether explicitly finalizing thedelegating generator by calling itsclose() method while it issuspended at ayieldfrom should also finalize the subiterator.An argument against doing so is that it would result in prematurefinalization of the subiterator if references to it exist elsewhere.
Consideration of non-refcounting Python implementations led to thedecision that this explicit finalization should be performed, so thatexplicitly closing a factored generator has the same effect as doingso to an unfactored one in all Python implementations.
The assumption made is that, in the majority of use cases, thesubiterator will not be shared. The rare case of a shared subiteratorcan be accommodated by means of a wrapper that blocksthrow() andclose() calls, or by using a means other thanyieldfrom tocall the subiterator.
A motivation for generators being able to return values concerns theuse of generators to implement lightweight threads. When usinggenerators in that way, it is reasonable to want to spread thecomputation performed by the lightweight thread over many functions.One would like to be able to call a subgenerator as though it were anordinary function, passing it parameters and receiving a returnedvalue.
Using the proposed syntax, a statement such as
y=f(x)
where f is an ordinary function, can be transformed into a delegationcall
y=yield fromg(x)
where g is a generator. One can reason about the behaviour of theresulting code by thinking of g as an ordinary function that can besuspended using ayield statement.
When using generators as threads in this way, typically one is notinterested in the values being passed in or out of the yields.However, there are use cases for this as well, where the thread isseen as a producer or consumer of items. Theyieldfromexpression allows the logic of the thread to be spread over as manyfunctions as desired, with the production or consumption of itemsoccurring in any subfunction, and the items are automatically routed toor from their ultimate source or destination.
Concerningthrow() andclose(), it is reasonable to expectthat if an exception is thrown into the thread from outside, it shouldfirst be raised in the innermost generator where the thread issuspended, and propagate outwards from there; and that if the threadis terminated from outside by callingclose(), the chain of activegenerators should be finalised from the innermost outwards.
The particular syntax proposed has been chosen as suggestive of itsmeaning, while not introducing any new keywords and clearly standingout as being different from a plainyield.
Using a specialised syntax opens up possibilities for optimisationwhen there is a long chain of generators. Such chains can arise, forinstance, when recursively traversing a tree structure. The overheadof passing__next__() calls and yielded values down and up thechain can cause what ought to be an O(n) operation to become, in theworst case, O(n**2).
A possible strategy is to add a slot to generator objects to hold agenerator being delegated to. When a__next__() orsend()call is made on the generator, this slot is checked first, and if itis nonempty, the generator that it references is resumed instead. Ifit raises StopIteration, the slot is cleared and the main generator isresumed.
This would reduce the delegation overhead to a chain of C functioncalls involving no Python code execution. A possible enhancementwould be to traverse the whole chain of generators in a loop anddirectly resume the one at the end, although the handling ofStopIteration is more complicated then.
There are a variety of ways that the return value from the generatorcould be passed back. Some alternatives include storing it as anattribute of the generator-iterator object, or returning it as thevalue of theclose() call to the subgenerator. However, theproposed mechanism is attractive for a couple of reasons:
Some ideas were discussed but rejected.
Suggestion: There should be some way to prevent the initial call to__next__(), or substitute it with a send() call with a specifiedvalue, the intention being to support the use of generators wrapped sothat the initial __next__() is performed automatically.
Resolution: Outside the scope of the proposal. Such generators shouldnot be used withyieldfrom.
Suggestion: If closing a subiterator raises StopIteration with avalue, return that value from theclose() call to the delegatinggenerator.
The motivation for this feature is so that the end of a stream ofvalues being sent to a generator can be signalled by closing thegenerator. The generator would catch GeneratorExit, finish itscomputation and return a result, which would then become the returnvalue of the close() call.
Resolution: This usage of close() and GeneratorExit would beincompatible with their current role as a bail-out and clean-upmechanism. It would require that when closing a delegating generator,after the subgenerator is closed, the delegating generator be resumedinstead of re-raising GeneratorExit. But this is not acceptable,because it would fail to ensure that the delegating generator isfinalised properly in the case where close() is being called forcleanup purposes.
Signalling the end of values to a consumer is better addressed byother means, such as sending in a sentinel value or throwing in anexception agreed upon by the producer and consumer. The consumer canthen detect the sentinel or exception and respond by finishing itscomputation and returning normally. Such a scheme behaves correctlyin the presence of delegation.
Suggestion: Ifclose() is not to return a value, then raise anexception if StopIteration with a non-None value occurs.
Resolution: No clear reason to do so. Ignoring a return value is notconsidered an error anywhere else in Python.
Under this proposal, the value of ayieldfrom expression would bederived in a very different way from that of an ordinaryyieldexpression. This suggests that some other syntax not containing thewordyield might be more appropriate, but no acceptablealternative has so far been proposed. Rejected alternatives includecall,delegate andgcall.
It has been suggested that some mechanism other thanreturn in thesubgenerator should be used to establish the value returned by theyieldfrom expression. However, this would interfere with thegoal of being able to think of the subgenerator as a suspendablefunction, since it would not be able to return values in the same wayas other functions.
The use of an exception to pass the return value has been criticisedas an “abuse of exceptions”, without any concrete justification ofthis claim. In any case, this is only one suggested implementation;another mechanism could be used without losing any essential featuresof the proposal.
It has been suggested that a different exception, such asGeneratorReturn, should be used instead of StopIteration to return avalue. However, no convincing practical reason for this has been putforward, and the addition of avalue attribute to StopIterationmitigates any difficulties in extracting a return value from aStopIteration exception that may or may not have one. Also, using adifferent exception would mean that, unlike ordinary functions,‘return’ without a value in a generator would not be equivalent to‘return None’.
Proposals along similar lines have been made before, some using thesyntaxyield* instead ofyieldfrom. Whileyield* ismore concise, it could be argued that it looks too similar to anordinaryyield and the difference might be overlooked when readingcode.
To the author’s knowledge, previous proposals have focused only onyielding values, and thereby suffered from the criticism that thetwo-line for-loop they replace is not sufficiently tiresome to writeto justify a new syntax. By dealing with the full generator protocol,this proposal provides considerably more benefit.
Some examples of the use of the proposed syntax are available, andalso a prototype implementation based on the first optimisationoutlined above.
A version of the implementation updated for Python 3.3 is available fromtrackerissue #11682
This document has been placed in the public domain.
Source:https://github.com/python/peps/blob/main/peps/pep-0380.rst
Last modified:2025-02-01 08:59:27 GMT