Abstract

This PEP adds a new statement “with” to the Python language to makeit possible to factor out standard uses oftry/finally statements.

In this PEP, context managers provide__enter__() and__exit__()methods that are invoked on entry to and exit from the body of thewith statement.

Author’s Note

This PEP was originally written in first person by Guido, andsubsequently updated by Alyssa (Nick) Coghlan to reflect later discussionon python-dev. Any first person references are from Guido’soriginal.

Python’s alpha release cycle revealed terminology problems in thisPEP and in the associated documentation and implementation[13].The PEP stabilised around the time of the first Python 2.5 betarelease.

Yes, the verb tense is messed up in a few places. We’ve beenworking on this PEP for over a year now, so things that wereoriginally in the future are now in the past :)

Introduction

After a lot of discussion aboutPEP 340 and alternatives, Idecided to withdrawPEP 340 and proposed a slight variant on PEP310. After more discussion, I have added back a mechanism forraising an exception in a suspended generator using athrow()method, and aclose() method which throws a new GeneratorExitexception; these additions were first proposed on python-dev in[2] and universally approved of. I’m also changing the keyword to‘with’.

After acceptance of this PEP, the following PEPs were rejected dueto overlap:

• PEP 310, Reliable Acquisition/Release Pairs. This is theoriginal with-statement proposal.
• PEP 319, Python Synchronize/Asynchronize Block. Its use casescan be covered by the current PEP by providing suitablewith-statement controllers: for ‘synchronize’ we can use the“locking” template from example 1; for ‘asynchronize’ we can usea similar “unlocking” template. I don’t think having an“anonymous” lock associated with a code block is all thatimportant; in fact it may be better to always be explicit aboutthe mutex being used.

PEP 340 andPEP 346 also overlapped with this PEP, but werevoluntarily withdrawn when this PEP was submitted.

Some discussion of earlier incarnations of this PEP took place onthe Python Wiki[3].

Motivation and Summary

PEP 340, Anonymous Block Statements, combined many powerful ideas:using generators as block templates, adding exception handling andfinalization to generators, and more. Besides praise it receiveda lot of opposition from people who didn’t like the fact that itwas, under the covers, a (potential) looping construct. Thismeant that break and continue in a block-statement would break orcontinue the block-statement, even if it was used as a non-loopingresource management tool.

But the final blow came when I read Raymond Chen’s rant aboutflow-control macros[1]. Raymond argues convincingly that hidingflow control in macros makes your code inscrutable, and I findthat his argument applies to Python as well as to C. I realizedthatPEP 340 templates can hide all sorts of control flow; forexample, its example 4 (auto_retry()) catches exceptions andrepeats the block up to three times.

However, the with-statement ofPEP 310 doesnot hide controlflow, in my view: while a finally-suite temporarily suspends thecontrol flow, in the end, the control flow resumes as if thefinally-suite wasn’t there at all.

Remember,PEP 310 proposes roughly this syntax (the “VAR =” part isoptional):

withVAR=EXPR:BLOCK

which roughly translates into this:

VAR=EXPRVAR.__enter__()try:BLOCKfinally:VAR.__exit__()

Now consider this example:

withf=open("/etc/passwd"):BLOCK1BLOCK2

Here, just as if the first line was “if True” instead, we knowthat ifBLOCK1 completes without an exception, BLOCK2 will bereached; and ifBLOCK1 raises an exception or executes a non-localgoto (a break, continue or return),BLOCK2 isnot reached. Themagic added by the with-statement at the end doesn’t affect this.

(You may ask, what if a bug in the__exit__() method causes anexception? Then all is lost – but this is no worse than withother exceptions; the nature of exceptions is that they can happenanywhere, and you just have to live with that. Even if youwrite bug-free code, a KeyboardInterrupt exception can still causeit to exit between any two virtual machine opcodes.)

This argument almost led me to endorsePEP 310, but I had one idealeft from thePEP 340 euphoria that I wasn’t ready to drop: usinggenerators as “templates” for abstractions like acquiring andreleasing a lock or opening and closing a file is a powerful idea,as can be seen by looking at the examples in that PEP.

Inspired by a counter-proposal toPEP 340 by Phillip Eby I triedto create a decorator that would turn a suitable generator into anobject with the necessary__enter__() and__exit__() methods.Here I ran into a snag: while it wasn’t too hard for the lockingexample, it was impossible to do this for the opening example.The idea was to define the template like this:

@contextmanagerdefopening(filename):f=open(filename)try:yieldffinally:f.close()

and used it like this:

withf=opening(filename):...readdatafromf...

The problem is that inPEP 310, the result of callingEXPR isassigned directly toVAR, and thenVAR’s__exit__() method iscalled upon exit fromBLOCK1. But here,VAR clearly needs toreceive the opened file, and that would mean that__exit__() wouldhave to be a method on the file.

While this can be solved using a proxy class, this is awkward andmade me realize that a slightly different translation would makewriting the desired decorator a piece of cake: letVAR receive theresult from calling the__enter__() method, and save the value ofEXPR to call its__exit__() method later. Then the decorator canreturn an instance of a wrapper class whose__enter__() methodcalls the generator’snext() method and returns whatevernext()returns; the wrapper instance’s__exit__() method callsnext()again but expects it to raise StopIteration. (Details below inthe section Optional Generator Decorator.)

So now the final hurdle was that thePEP 310 syntax:

withVAR=EXPR:BLOCK1

would be deceptive, sinceVAR doesnot receive the value ofEXPR. Borrowing fromPEP 340, it was an easy step to:

withEXPRasVAR:BLOCK1

Additional discussion showed that people really liked being ableto “see” the exception in the generator, even if it was only tolog it; the generator is not allowed to yield another value, sincethe with-statement should not be usable as a loop (raising adifferent exception is marginally acceptable). To enable this, anewthrow() method for generators is proposed, which takes one tothree arguments representing an exception in the usual fashion(type, value, traceback) and raises it at the point where thegenerator is suspended.

Once we have this, it is a small step to proposing anothergenerator method,close(), which callsthrow() with a specialexception,GeneratorExit. This tells the generator to exit, andfrom there it’s another small step to proposing thatclose() becalled automatically when the generator is garbage-collected.

Then, finally, we can allow a yield-statement inside a try-finallystatement, since we can now guarantee that the finally-clause will(eventually) be executed. The usual cautions about finalizationapply – the process may be terminated abruptly without finalizingany objects, and objects may be kept alive forever by cycles ormemory leaks in the application (as opposed to cycles or leaks inthe Python implementation, which are taken care of by GC).

Note that we’re not guaranteeing that the finally-clause isexecuted immediately after the generator object becomes unused,even though this is how it will work in CPython. This is similarto auto-closing files: while a reference-counting implementationlike CPython deallocates an object as soon as the last referenceto it goes away, implementations that use other GC algorithms donot make the same guarantee. This applies to Jython, IronPython,and probably to Python running on Parrot.

(The details of the changes made to generators can now be found inPEP 342 rather than in the current PEP)

Use Cases

See the Examples section near the end.

Specification: The ‘with’ Statement

A new statement is proposed with the syntax:

withEXPRasVAR:BLOCK

Here, ‘with’ and ‘as’ are new keywords;EXPR is an arbitraryexpression (but not an expression-list) andVAR is a singleassignment target. It cannot be a comma-separated sequence ofvariables, but itcan be aparenthesized comma-separatedsequence of variables. (This restriction makes a future extensionpossible of the syntax to have multiple comma-separated resources,each with its own optional as-clause.)

The “as VAR” part is optional.

The translation of the above statement is:

mgr=(EXPR)exit=type(mgr).__exit__# Not calling it yetvalue=type(mgr).__enter__(mgr)exc=Truetry:try:VAR=value# Only if "as VAR" is presentBLOCKexcept:# The exceptional case is handled hereexc=Falseifnotexit(mgr,*sys.exc_info()):raise# The exception is swallowed if exit() returns truefinally:# The normal and non-local-goto cases are handled hereifexc:exit(mgr,None,None,None)

Here, the lowercase variables (mgr, exit, value, exc) are internalvariables and not accessible to the user; they will most likely beimplemented as special registers or stack positions.

The details of the above translation are intended to prescribe theexact semantics. If either of the relevant methods are not foundas expected, the interpreter will raiseAttributeError, in theorder that they are tried (__exit__,__enter__).Similarly, if any of the calls raises an exception, the effect isexactly as it would be in the above code. Finally, ifBLOCKcontains a break, continue or return statement, the__exit__()method is called with three None arguments just as ifBLOCKcompleted normally. (I.e. these “pseudo-exceptions” are not seenas exceptions by__exit__().)

If the “as VAR” part of the syntax is omitted, the “VAR =” part ofthe translation is omitted (butmgr.__enter__() is still called).

The calling convention formgr.__exit__() is as follows. If thefinally-suite was reached through normal completion ofBLOCK orthrough a non-local goto (a break, continue or return statement inBLOCK),mgr.__exit__() is called with threeNone arguments. Ifthe finally-suite was reached through an exception raised inBLOCK,mgr.__exit__() is called with three arguments representingthe exception type, value, and traceback.

IMPORTANT: ifmgr.__exit__() returns a “true” value, the exceptionis “swallowed”. That is, if it returns “true”, executioncontinues at the next statement after the with-statement, even ifan exception happened inside the with-statement. However, if thewith-statement was left via a non-local goto (break, continue orreturn), this non-local return is resumed whenmgr.__exit__()returns regardless of the return value. The motivation for thisdetail is to make it possible formgr.__exit__() to swallowexceptions, without making it too easy (since the default returnvalue,None, is false and this causes the exception to bere-raised). The main use case for swallowing exceptions is tomake it possible to write the@contextmanager decorator sothat a try/except block in a decorated generator behaves exactlyas if the body of the generator were expanded in-line at the placeof the with-statement.

The motivation for passing the exception details to__exit__(), asopposed to the argument-less__exit__() fromPEP 310, was given bythetransactional() use case, example 3 below. The template inthat example must commit or roll back the transaction depending onwhether an exception occurred or not. Rather than just having aboolean flag indicating whether an exception occurred, we pass thecomplete exception information, for the benefit of anexception-logging facility for example. Relying onsys.exc_info()to get at the exception information was rejected;sys.exc_info()has very complex semantics and it is perfectly possible that itreturns the exception information for an exception that was caughtages ago. It was also proposed to add an additional boolean todistinguish between reaching the end ofBLOCK and a non-localgoto. This was rejected as too complex and unnecessary; anon-local goto should be considered unexceptional for the purposesof a database transaction roll-back decision.

To facilitate chaining of contexts in Python code that directlymanipulates context managers,__exit__() methods shouldnotre-raise the error that is passed in to them. It is always theresponsibility of thecaller of the__exit__() method to do anyreraising in that case.

That way, if the caller needs to tell whether the__exit__()invocationfailed (as opposed to successfully cleaning up beforepropagating the original error), it can do so.

If__exit__() returns without an error, this can then beinterpreted as success of the__exit__() method itself (regardlessof whether or not the original error is to be propagated orsuppressed).

However, if__exit__() propagates an exception to its caller, thismeans that__exit__()itself has failed. Thus,__exit__()methods should avoid raising errors unless they have actuallyfailed. (And allowing the original error to proceed isn’t afailure.)

Transition Plan

In Python 2.5, the new syntax will only be recognized if a futurestatement is present:

from__future__importwith_statement

This will make both ‘with’ and ‘as’ keywords. Without the futurestatement, using ‘with’ or ‘as’ as an identifier will cause aWarning to be issued to stderr.

In Python 2.6, the new syntax will always be recognized; ‘with’and ‘as’ are always keywords.

Generator Decorator

WithPEP 342 accepted, it is possible to write a decoratorthat makes it possible to use a generator that yields exactly onceto control a with-statement. Here’s a sketch of such a decorator:

classGeneratorContextManager(object):def__init__(self,gen):self.gen=gendef__enter__(self):try:returnself.gen.next()exceptStopIteration:raiseRuntimeError("generator didn't yield")def__exit__(self,type,value,traceback):iftypeisNone:try:self.gen.next()exceptStopIteration:returnelse:raiseRuntimeError("generator didn't stop")else:try:self.gen.throw(type,value,traceback)raiseRuntimeError("generator didn't stop after throw()")exceptStopIteration:returnTrueexcept:# only re-raise if it's *not* the exception that was# passed to throw(), because __exit__() must not raise# an exception unless __exit__() itself failed.  But# throw() has to raise the exception to signal# propagation, so this fixes the impedance mismatch# between the throw() protocol and the __exit__()# protocol.#ifsys.exc_info()[1]isnotvalue:raisedefcontextmanager(func):defhelper(*args,**kwds):returnGeneratorContextManager(func(*args,**kwds))returnhelper

This decorator could be used as follows:

@contextmanagerdefopening(filename):f=open(filename)# IOError is untouched by GeneratorContexttry:yieldffinally:f.close()# Ditto for errors here (however unlikely)

A robust implementation of this decorator will be madepart of the standard library.

Context Managers in the Standard Library

It would be possible to endow certain objects, like files,sockets, and locks, with__enter__() and__exit__() methods sothat instead of writing:

withlocking(myLock):BLOCK

one could write simply:

withmyLock:BLOCK

I think we should be careful with this; it could lead to mistakeslike:

f=open(filename)withf:BLOCK1withf:BLOCK2

which does not do what one might think (f is closed beforeBLOCK2is entered).

OTOH such mistakes are easily diagnosed; for example, thegenerator context decorator above raisesRuntimeError when asecond with-statement callsf.__enter__() again. A similar errorcan be raised if__enter__ is invoked on a closed file object.

For Python 2.5, the following types have been identified ascontext managers:

-file-thread.LockType-threading.Lock-threading.RLock-threading.Condition-threading.Semaphore-threading.BoundedSemaphore

A context manager will also be added to the decimal module tosupport using a local decimal arithmetic context within the bodyof a with statement, automatically restoring the original contextwhen the with statement is exited.

Standard Terminology

This PEP proposes that the protocol consisting of the__enter__()and__exit__() methods be known as the “context management protocol”,and that objects that implement that protocol be known as “contextmanagers”.[4]

The expression immediately following the with keyword in thestatement is a “context expression” as that expression provides themain clue as to the runtime environment the context managerestablishes for the duration of the statement body.

The code in the body of the with statement and the variable name(or names) after the as keyword don’t really have special terms atthis point in time. The general terms “statement body” and “targetlist” can be used, prefixing with “with” or “with statement” if theterms would otherwise be unclear.

Given the existence of objects such as the decimal module’sarithmetic context, the term “context” is unfortunately ambiguous.If necessary, it can be made more specific by using the terms“context manager” for the concrete object created by the contextexpression and “runtime context” or (preferably) “runtimeenvironment” for the actual state modifications made by the contextmanager. When simply discussing use of the with statement, theambiguity shouldn’t matter too much as the context expression fullydefines the changes made to the runtime environment.The distinction is more important when discussing the mechanics ofthe with statement itself and how to go about actually implementingcontext managers.

Caching Context Managers

Many context managers (such as files and generator-based contexts)will be single-use objects. Once the__exit__() method has beencalled, the context manager will no longer be in a usable state(e.g. the file has been closed, or the underlying generator hasfinished execution).

Requiring a fresh manager object for each with statement is theeasiest way to avoid problems with multi-threaded code and nestedwith statements trying to use the same context manager. It isn’tcoincidental that all of the standard library context managersthat support reuse come from the threading module - they’re allalready designed to deal with the problems created by threadedand nested usage.

This means that in order to save a context manager with particularinitialisation arguments to be used in multiple with statements, itwill typically be necessary to store it in a zero-argument callablethat is then called in the context expression of each statementrather than caching the context manager directly.

When this restriction does not apply, the documentation of theaffected context manager should make that clear.

Resolved Issues

The following issues were resolved by BDFL approval (and a lackof any major objections on python-dev).

1. What exception shouldGeneratorContextManager raise when theunderlying generator-iterator misbehaves? The following quote isthe reason behind Guido’s choice ofRuntimeError for both thisand for the generatorclose() method inPEP 342 (from[8]):

   “I’d rather not introduce a new exception class just for thispurpose, since it’s not an exception that I want people to catch:I want it to turn into a traceback which is seen by theprogrammer who then fixes the code. So now I believe theyshould both raiseRuntimeError.There are some precedents for that: it’s raised by the corePython code in situations where endless recursion is detected,and for uninitialized objects (and for a variety ofmiscellaneous conditions).”

2. It is fine to raiseAttributeError instead ofTypeError if therelevant methods aren’t present on a class involved in a withstatement. The fact that the abstract object C API raisesTypeError rather thanAttributeError is an accident of history,rather than a deliberate design decision[11].
3. Objects with__enter__/__exit__ methods are called “contextmanagers” and the decorator to convert a generator functioninto a context manager factory iscontextlib.contextmanager.There were some other suggestions[15] during the 2.5 releasecycle but no compelling arguments for switching away from theterms that had been used in the PEP implementation were made.

Rejected Options

For several months, the PEP prohibited suppression of exceptionsin order to avoid hidden flow control. Implementationrevealed this to be a right royal pain, so Guido restored theability[12].

Another aspect of the PEP that caused no end of questions andterminology debates was providing a__context__() method thatwas analogous to an iterable’s__iter__() method[5][7][9].The ongoing problems[10][12] with explaining what it was and whyit was and how it was meant to work eventually lead to Guidokilling the concept outright[14] (and there was much rejoicing!).

The notion of using thePEP 342 generator API directly to definethe with statement was also briefly entertained[6], but quicklydismissed as making it too difficult to write non-generatorbased context managers.

Examples

The generator based examples rely onPEP 342. Also, some of theexamples are unnecessary in practice, as the appropriate objects,such asthreading.RLock, are able to be used directly in withstatements.

The tense used in the names of the example contexts is notarbitrary. Past tense (“-ed”) is used when the name refers to anaction which is done in the__enter__ method and undone in the__exit__ method. Progressive tense (“-ing”) is used when the namerefers to an action which is to be done in the__exit__ method.

1. A template for ensuring that a lock, acquired at the start of ablock, is released when the block is left:

   @contextmanagerdeflocked(lock):lock.acquire()try:yieldfinally:lock.release()

   Used as follows:

   withlocked(myLock):# Code here executes with myLock held.  The lock is# guaranteed to be released when the block is left (even# if via return or by an uncaught exception).

2. A template for opening a file that ensures the file is closedwhen the block is left:

   @contextmanagerdefopened(filename,mode="r"):f=open(filename,mode)try:yieldffinally:f.close()

   Used as follows:

   withopened("/etc/passwd")asf:forlineinf:printline.rstrip()

3. A template for committing or rolling back a databasetransaction:

   @contextmanagerdeftransaction(db):db.begin()try:yieldNoneexcept:db.rollback()raiseelse:db.commit()

4. Example 1 rewritten without a generator:

   classlocked:def__init__(self,lock):self.lock=lockdef__enter__(self):self.lock.acquire()def__exit__(self,type,value,tb):self.lock.release()

   (This example is easily modified to implement the otherrelatively stateless examples; it shows that it is easy to avoidthe need for a generator if no special state needs to bepreserved.)

5. Redirect stdout temporarily:

   @contextmanagerdefstdout_redirected(new_stdout):save_stdout=sys.stdoutsys.stdout=new_stdouttry:yieldNonefinally:sys.stdout=save_stdout

   Used as follows:

   withopened(filename,"w")asf:withstdout_redirected(f):print"Hello world"

   This isn’t thread-safe, of course, but neither is doing thissame dance manually. In single-threaded programs (for example,in scripts) it is a popular way of doing things.

6. A variant onopened() that also returns an error condition:

   @contextmanagerdefopened_w_error(filename,mode="r"):try:f=open(filename,mode)exceptIOError,err:yieldNone,errelse:try:yieldf,Nonefinally:f.close()

   Used as follows:

   withopened_w_error("/etc/passwd","a")as(f,err):iferr:print"IOError:",errelse:f.write("guido::0:0::/:/bin/sh\n")

7. Another useful example would be an operation that blockssignals. The use could be like this:

   importsignalwithsignal.blocked():# code executed without worrying about signals

   An optional argument might be a list of signals to be blocked;by default all signals are blocked. The implementation is leftas an exercise to the reader.

8. Another use for this feature is the Decimal context. Here’s asimple example, after one posted by Michael Chermside:

   importdecimal@contextmanagerdefextra_precision(places=2):c=decimal.getcontext()saved_prec=c.precc.prec+=placestry:yieldNonefinally:c.prec=saved_prec

   Sample usage (adapted from the Python Library Reference):

   defsin(x):"Return the sine of x as measured in radians."withextra_precision():i,lasts,s,fact,num,sign=1,0,x,1,x,1whiles!=lasts:lasts=si+=2fact*=i*(i-1)num*=x*xsign*=-1s+=num/fact*sign# The "+s" rounds back to the original precision,# so this must be outside the with-statement:return+s

9. Here’s a simple context manager for the decimal module:

   @contextmanagerdeflocalcontext(ctx=None):"""Set a new local decimal context for the block"""# Default to using the current contextifctxisNone:ctx=getcontext()# We set the thread context to a copy of this context# to ensure that changes within the block are kept# local to the block.newctx=ctx.copy()oldctx=decimal.getcontext()decimal.setcontext(newctx)try:yieldnewctxfinally:# Always restore the original contextdecimal.setcontext(oldctx)

   Sample usage:

   fromdecimalimportlocalcontext,ExtendedContextdefsin(x):withlocalcontext()asctx:ctx.prec+=2# Rest of sin calculation algorithm# uses a precision 2 greater than normalreturn+s# Convert result to normal precisiondefsin(x):withlocalcontext(ExtendedContext):# Rest of sin calculation algorithm# uses the Extended Context from the# General Decimal Arithmetic Specificationreturn+s# Convert result to normal context

10. A generic “object-closing” context manager:

    classclosing(object):def__init__(self,obj):self.obj=objdef__enter__(self):returnself.objdef__exit__(self,*exc_info):try:close_it=self.obj.closeexceptAttributeError:passelse:close_it()

    This can be used to deterministically close anything with aclose method, be it file, generator, or something else. Itcan even be used when the object isn’t guaranteed to requireclosing (e.g., a function that accepts an arbitraryiterable):

    # emulate opening():withclosing(open("argument.txt"))ascontradiction:forlineincontradiction:printline# deterministically finalize an iterator:withclosing(iter(data_source))asdata:fordatumindata:process(datum)

    (Python 2.5’s contextlib module contains a versionof this context manager)

11. PEP 319 gives a use case for also having areleased()context to temporarily release a previously acquired lock;this can be written very similarly to the locked contextmanager above by swapping theacquire() andrelease() calls:

    classreleased:def__init__(self,lock):self.lock=lockdef__enter__(self):self.lock.release()def__exit__(self,type,value,tb):self.lock.acquire()

    Sample usage:

    withmy_lock:# Operations with the lock heldwithreleased(my_lock):# Operations without the lock# e.g. blocking I/O# Lock is held again here

12. A “nested” context manager that automatically nests thesupplied contexts from left-to-right to avoid excessiveindentation:

    @contextmanagerdefnested(*contexts):exits=[]vars=[]try:try:forcontextincontexts:exit=context.__exit__enter=context.__enter__vars.append(enter())exits.append(exit)yieldvarsexcept:exc=sys.exc_info()else:exc=(None,None,None)finally:whileexits:exit=exits.pop()try:exit(*exc)except:exc=sys.exc_info()else:exc=(None,None,None)ifexc!=(None,None,None):# sys.exc_info() may have been# changed by one of the exit methods# so provide explicit exception inforaiseexc[0],exc[1],exc[2]

    Sample usage:

    withnested(a,b,c)as(x,y,z):# Perform operation

    Is equivalent to:

    withaasx:withbasy:withcasz:# Perform operation

    (Python 2.5’s contextlib module contains a versionof this context manager)

Reference Implementation

This PEP was first accepted by Guido at his EuroPythonkeynote, 27 June 2005.It was accepted again later, withthe__context__ method added.The PEP was implemented in Subversion for Python 2.5a1The__context__() method was removed in Python 2.5b1

Acknowledgements

Many people contributed to the ideas and concepts in this PEP,including all those mentioned in the acknowledgements forPEP 340andPEP 346.

Additional thanks goes to (in no meaningful order): Paul Moore,Phillip J. Eby, Greg Ewing, Jason Orendorff, Michael Hudson,Raymond Hettinger, Walter Dörwald, Aahz, Georg Brandl, Terry Reedy,A.M. Kuchling, Brett Cannon, and all those that participated in thediscussions on python-dev.

References

Copyright

This document has been placed in the public domain.