Introduction

This PEP proposes some enhancements to the API and syntax of generators, tomake them usable as simple coroutines. It is basically a combination of ideasfrom these two PEPs, which may be considered redundant if this PEP isaccepted:

• PEP 288, Generators Attributes and Exceptions. The current PEP covers itssecond half, generator exceptions (in fact thethrow() method name wastaken fromPEP 288).PEP 342 replaces generator attributes, however, with aconcept from an earlier revision ofPEP 288, theyield expression.
• PEP 325, Resource-Release Support for Generators.PEP 342 ties up a fewloose ends in thePEP 325 spec, to make it suitable for actualimplementation.

Motivation

Coroutines are a natural way of expressing many algorithms, such assimulations, games, asynchronous I/O, and other forms of event-drivenprogramming or co-operative multitasking. Python’s generator functions arealmost coroutines – but not quite – in that they allow pausing execution toproduce a value, but do not provide for values or exceptions to be passed inwhen execution resumes. They also do not allow execution to be paused withinthetry portion oftry/finally blocks, and therefore make it difficultfor an aborted coroutine to clean up after itself.

Also, generators cannot yield control while other functions are executing,unless those functions are themselves expressed as generators, and the outergenerator is written to yield in response to values yielded by the innergenerator. This complicates the implementation of even relatively simple usecases like asynchronous communications, because calling any functions eitherrequires the generator toblock (i.e. be unable to yield control), or else alot of boilerplate looping code must be added around every needed functioncall.

However, if it were possible to pass values or exceptionsinto a generator atthe point where it was suspended, a simple co-routine scheduler ortrampolinefunction would let coroutinescall each other without blocking – atremendous boon for asynchronous applications. Such applications could thenwrite co-routines to do non-blocking socket I/O by yielding control to an I/Oscheduler until data has been sent or becomes available. Meanwhile, code thatperforms the I/O would simply do something like this:

data=(yieldnonblocking_read(my_socket,nbytes))

in order to pause execution until thenonblocking_read() coroutine produceda value.

In other words, with a few relatively minor enhancements to the language and tothe implementation of the generator-iterator type, Python will be able tosupport performing asynchronous operations without needing to write the entireapplication as a series of callbacks, and without requiring the use ofresource-intensive threads for programs that need hundreds or even thousands ofco-operatively multitasking pseudothreads. Thus, these enhancements will givestandard Python many of the benefits of the Stackless Python fork, withoutrequiring any significant modification to the CPython core or its APIs. Inaddition, these enhancements should be readily implementable by any Pythonimplementation (such as Jython) that already supports generators.

Specification Summary

By adding a few simple methods to the generator-iterator type, and with twominor syntax adjustments, Python developers will be able to use generatorfunctions to implement co-routines and other forms of co-operativemultitasking. These methods and adjustments are:

1. Redefineyield to be an expression, rather than a statement. The currentyield statement would become a yield expression whose value is thrown away.A yield expression’s value isNone whenever the generator is resumed bya normalnext() call.
2. Add a newsend() method for generator-iterators, which resumes thegenerator andsends a value that becomes the result of the currentyield-expression. Thesend() method returns the next value yielded bythe generator, or raisesStopIteration if the generator exits withoutyielding another value.
3. Add a newthrow() method for generator-iterators, which raises anexception at the point where the generator was paused, and which returns thenext value yielded by the generator, raisingStopIteration if thegenerator exits without yielding another value. (If the generator does notcatch the passed-in exception, or raises a different exception, then thatexception propagates to the caller.)
4. Add aclose() method for generator-iterators, which raisesGeneratorExit at the point where the generator was paused. If thegenerator then raisesStopIteration (by exiting normally, or due toalready being closed) orGeneratorExit (by not catching the exception),close() returns to its caller. If the generator yields a value, aRuntimeError is raised. If the generator raises any other exception, itis propagated to the caller.close() does nothing if the generator hasalready exited due to an exception or normal exit.
5. Add support to ensure thatclose() is called when a generator iteratoris garbage-collected.
6. Allowyield to be used intry/finally blocks, since garbagecollection or an explicitclose() call would now allow thefinallyclause to execute.

A prototype patch implementing all of these changes against the current PythonCVS HEAD is available as SourceForge patch #1223381(https://bugs.python.org/issue1223381).

Specification: Sending Values into Generators

x=yield42x=yieldx=12+(yield42)x=12+(yield)foo(yield42)foo(yield)

x=12+yield42x=12+yieldfoo(yield42,12)foo(yield,12)

Specification: Exceptions and Cleanup

Let a generator object be the iterator produced by calling a generatorfunction. Below,g always refers to a generator object.

raisetype,value,traceback

defclose(self):try:self.throw(GeneratorExit)except(GeneratorExit,StopIteration):passelse:raiseRuntimeError("generator ignored GeneratorExit")# Other exceptions are not caught

Optional Extensions

Open Issues

Discussion on python-dev has revealed some open issues. I list them here, withmy preferred resolution and its motivation. The PEP as currently writtenreflects this preferred resolution.

1. What exception should be raised byclose() when the generator yieldsanother value as a response to theGeneratorExit exception?

   I originally choseTypeError because it represents gross misbehavior ofthe generator function, which should be fixed by changing the code. But thewith_template decorator class inPEP 343 usesRuntimeError forsimilar offenses. Arguably they should all use the same exception. I’drather not introduce a new exception class just for this purpose, since it’snot an exception that I want people to catch: I want it to turn into atraceback which is seen by the programmer who then fixes the code. So now Ibelieve they should both raiseRuntimeError. There are some precedentsfor that: it’s raised by the core Python code in situations where endlessrecursion is detected, and for uninitialized objects (and for a variety ofmiscellaneous conditions).

2. Oren Tirosh has proposed renaming thesend() method tofeed(), forcompatibility with theconsumer interface (seehttp://effbot.org/zone/consumer.htm for the specification.)

   However, looking more closely at the consumer interface, it seems that thedesired semantics forfeed() are different than forsend(), becausesend() can’t be meaningfully called on a just-started generator. Also,the consumer interface as currently defined doesn’t include handling forStopIteration.

   Therefore, it seems like it would probably be more useful to create a simpledecorator that wraps a generator function to make it conform to the consumerinterface. For example, it couldwarm up the generator with an initialnext() call, trap StopIteration, and perhaps even providereset() byre-invoking the generator function.

Examples

1. A simpleconsumer decorator that makes a generator function automaticallyadvance to its first yield point when initially called:

   defconsumer(func):defwrapper(*args,**kw):gen=func(*args,**kw)gen.next()returngenwrapper.__name__=func.__name__wrapper.__dict__=func.__dict__wrapper.__doc__=func.__doc__returnwrapper

2. An example of using theconsumer decorator to create areverse generatorthat receives images and creates thumbnail pages, sending them on to anotherconsumer. Functions like this can be chained together to form efficientprocessing pipelines ofconsumers that each can have complex internalstate:

   @consumerdefthumbnail_pager(pagesize,thumbsize,destination):whileTrue:page=new_image(pagesize)rows,columns=pagesize/thumbsizepending=Falsetry:forrowinxrange(rows):forcolumninxrange(columns):thumb=create_thumbnail((yield),thumbsize)page.write(thumb,col*thumbsize.x,row*thumbsize.y)pending=TrueexceptGeneratorExit:# close() was called, so flush any pending outputifpending:destination.send(page)# then close the downstream consumer, and exitdestination.close()returnelse:# we finished a page full of thumbnails, so send it# downstream and keep on loopingdestination.send(page)@consumerdefjpeg_writer(dirname):fileno=1whileTrue:filename=os.path.join(dirname,"page%04d.jpg"%fileno)write_jpeg((yield),filename)fileno+=1# Put them together to make a function that makes thumbnail# pages from a list of images and other parameters.#defwrite_thumbnails(pagesize,thumbsize,images,output_dir):pipeline=thumbnail_pager(pagesize,thumbsize,jpeg_writer(output_dir))forimageinimages:pipeline.send(image)pipeline.close()

3. A simple co-routine scheduler ortrampoline that lets coroutinescallother coroutines by yielding the coroutine they wish to invoke. Anynon-generator value yielded by a coroutine is returned to the coroutine thatcalled the one yielding the value. Similarly, if a coroutine raises anexception, the exception is propagated to itscaller. In effect, thisexample emulates simple tasklets as are used in Stackless Python, as long asyou use a yield expression to invoke routines that would otherwiseblock.This is only a very simple example, and far more sophisticated schedulersare possible. (For example, the existing GTasklet framework for Python(http://www.gnome.org/~gjc/gtasklet/gtasklets.html) and the peak.eventsframework (http://peak.telecommunity.com/) already implement similarscheduling capabilities, but must currently use awkward workarounds for theinability to pass values or exceptions into generators.)

   importcollectionsclassTrampoline:"""Manage communications between coroutines"""running=Falsedef__init__(self):self.queue=collections.deque()defadd(self,coroutine):"""Request that a coroutine be executed"""self.schedule(coroutine)defrun(self):result=Noneself.running=Truetry:whileself.runningandself.queue:func=self.queue.popleft()result=func()returnresultfinally:self.running=Falsedefstop(self):self.running=Falsedefschedule(self,coroutine,stack=(),val=None,*exc):defresume():value=valtry:ifexc:value=coroutine.throw(value,*exc)else:value=coroutine.send(value)except:ifstack:# send the error back to the "caller"self.schedule(stack[0],stack[1],*sys.exc_info())else:# Nothing left in this pseudothread to# handle it, let it propagate to the# run loopraiseifisinstance(value,types.GeneratorType):# Yielded to a specific coroutine, push the# current one on the stack, and call the new# one with no argsself.schedule(value,(coroutine,stack))elifstack:# Yielded a result, pop the stack and send the# value to the callerself.schedule(stack[0],stack[1],value)# else: this pseudothread has endedself.queue.append(resume)

4. A simpleecho server, and code to run it using a trampoline (presumes theexistence ofnonblocking_read,nonblocking_write, and other I/Ocoroutines, that e.g. raiseConnectionLost if the connection isclosed):

   # coroutine function that echos data back on a connected# socket#defecho_handler(sock):whileTrue:try:data=yieldnonblocking_read(sock)yieldnonblocking_write(sock,data)exceptConnectionLost:pass# exit normally if connection lost# coroutine function that listens for connections on a# socket, and then launches a service "handler" coroutine# to service the connection#deflisten_on(trampoline,sock,handler):whileTrue:# get the next incoming connectionconnected_socket=yieldnonblocking_accept(sock)# start another coroutine to handle the connectiontrampoline.add(handler(connected_socket))# Create a scheduler to manage all our coroutinest=Trampoline()# Create a coroutine instance to run the echo_handler on# incoming connections#server=listen_on(t,listening_socket("localhost","echo"),echo_handler)# Add the coroutine to the schedulert.add(server)# loop forever, accepting connections and servicing them# "in parallel"#t.run()

Reference Implementation

A prototype patch implementing all of the features described in this PEP isavailable as SourceForge patch #1223381 (https://bugs.python.org/issue1223381).

This patch was committed to CVS 01-02 August 2005.

Acknowledgements

Raymond Hettinger (PEP 288) and Samuele Pedroni (PEP 325) first formallyproposed the ideas of communicating values or exceptions into generators, andthe ability toclose generators. Timothy Delaney suggested the title of thisPEP, and Steven Bethard helped edit a previous version. See also theAcknowledgements section ofPEP 340.

References

TBD.

Copyright

This document has been placed in the public domain.