A basic issue in programming isresource management: aresource is anything in limited supply, notably file handles,network sockets, locks, etc., and a key problem is making sure these arereleased after they areacquired. If they are not released, you have aresource leak, and the system may slow down or crash. More generally, you may want cleanup actions to always be done, other than simply releasing resources.
Python provides special syntax for this inthewith statement, which automatically manages resources encapsulated withincontext manager types, or more generally performs startup and cleanup actions around a block of code. You shouldalways use awith statement for resource management. There are many built-in context manager types, including the basic example ofFile, and it is easy to write your own. The code is not hard, but the concepts are slightly subtle, and it is easy to make mistakes.
Basic resource management uses an explicit pair ofopen()...close() functions, as in basic file opening and closing.Don't do this, for the reasons we are about to explain:
f=open(filename)# ...f.close()
The key problem with this simple code is that it fails if there is an early return, either due to areturn statement or an exception, possibly raised by called code. To fix this, ensuring that the cleanup code is called when the block is exited, one uses atry...finally clause:
f=open(filename)try:# ...finally:f.close()
However, this still requires manually releasing the resource, which might be forgotten, and the release code is distant from the acquisition code. The release can be done automatically by instead usingwith, which works becauseFile is a context manager type:
withopen(filename)asf:# ...
This assigns the value ofopen(filename) tof (this point is subtle and varies between context managers), and then automatically releases the resource, in this case callingf.close(), when the block exits.
Newer objects arecontext managers (formallycontext manager types: subtypes, as they implement the context manager interface, which consists of__enter__(),__exit__()), and thus can be used inwith statements easily (seeWith Statement Context Managers).
For older file-like objects that have aclose method but not__exit__(), you can use the@contextlib.closing decorator. If you need to roll your own, this is very easy, particularly using the@contextlib.contextmanager decorator.[1]
Context managers work by calling__enter__() when thewith context is entered, binding the return value to the target ofas, and calling__exit__() when the context is exited. There's some subtlety about handling exceptions during exit, but you can ignore it for simple use.
More subtly,__init__() is called when an object is created, but__enter__() is called when awith context is entered.
The__init__()/__enter__() distinction is important to distinguish betweensingle use, reusable and reentrant context managers. It's not a meaningful distinction for the common use case of instantiating an object in thewith clause, as follows:
withA()asa:...
...in which case any single use context manager is fine.
However, in general it is a difference, notably when distinguishing a reusable context manager from the resource it is managing, as in here:
a_cm=A()witha_cmasa:...
Putting resource acquisition in__enter__() instead of__init__() gives areusable context manager.
Notably,File() objects do the initialization in__init__() and then just returns itself when entering a context, as indef __enter__(): return self. This is fine if you want the target of theas to be bound to anobject (and allows you to use factories likeopen as the source of thewith clause), but if you want it to be bound to something else, notably ahandle (file name or file handle/file descriptor), you want to wrap the actual object in a separate context manager. For example:
@contextmanagerdefFileName(*args,**kwargs):withFile(*args,**kwargs)asf:yieldf.name
For simple uses you don't need to do any__init__() code, and only need to pair__enter__()/__exit__(). For more complicated uses you can havereentrant context managers, but that's not necessary for simple use.
try...finallyNote that atry...finally clauseis necessary with@contextlib.contextmanager, as this does not catch any exceptions raised after theyield, but isnot necessary in__exit__(), which is called even if an exception is raised.
The termcontext manager is carefully chosen, particularly in contrast to “scope”. Local variables in Python have function scope, and thus the target of awith statement, if any, is still visible after the block has exited, though__exit__() has already been called on the context manager (the argument of thewith statement), and thus is often not useful or valid. This is a technical point, but it's worth distinguishing thewith statement context from the overall function scope.
Generators that hold or use resources are a bit tricky.
Beware that creating generators within awith statement and then using them outside the block doesnot work, because generators have deferred evaluation, and thus when they are evaluated, the resource has already been released. This is most easily seen using a file, as in this generator expression to convert a file to a list of lines, stripping the end-of-line character:
withopen(filename)asf:lines=(line.rstrip('\n')forlineinf)
Whenlines is then used – evaluation can be forced withlist(lines) – this fails withValueError: I/O operation on closed file. This is because the file is closed at the end of thewith statement, but the lines are not read until the generator is evaluated.
The simplest solution is to avoid generators, and instead use lists, such as list comprehensions. This is generally appropriate in this case (reading a file) since one wishes to minimize system calls and just read the file all at once (unless the file is very large):
withopen(filename)asf:lines=[line.rstrip('\n')forlineinf]
In case that one does wish to use a resource in a generator, the resource must be held within the generator, as in this generator function:
defstripped_lines(filename):withopen(filename)asf:forlineinf:yieldline.rstrip('\n')
As the nesting makes clear, the file is kept open while iterating through it.
To release the resource, the generator must be explicitly closed, usinggenerator.close(), just as with other objects that hold resources (this is thedispose pattern). This can in turn be automated by making the generator into a context manager, using@contextlib.closing, as:
fromcontextlibimportclosingwithclosing(stripped_lines(filename))aslines:# ...
Resource Acquisition Is Initialization is an alternative form of resource management, particularly used in C++. In RAII, resources are acquired during object construction, and released during object destruction. In Python the analogous functions are__init__() and__del__() (finalizer), but RAII doesnot work in Python, and releasing resources in__del__() doesnot work. This is because there is no guarantee that__del__() will be called: it's just for memory manager use, not for resource handling.
In more detail, Python object construction is two-phase, consisting of (memory) allocation in__new__() and (attribute) initialization in__init__(). Python is garbage-collected via reference counting, with objects being finalized (not destructed) by__del__(). However, finalization is non-deterministic (objects have non-deterministic lifetimes), and the finalizer may be called much later or not at all, particularly if the program crashes. Thus using__del__() for resource management will generally leak resources.
It is possible to use finalizers for resource management, but the resulting code is implementation-dependent (generally working in CPython but not other implementations, such as PyPy) and fragile to version changes. Even if this is done, it requires great care to ensure references drop to zero in all circumstances, including: exceptions, which contain references in tracebacks if caught or if running interactively; and references in global variables, which last until program termination. Prior to Python 3.4, finalizers on objects in cycles were also a serious problem, but this is no longer a problem; however, finalization of objects in cycles is not done in a deterministic order.