Thetry …finally Approach

Working with files is probably the most common example of resource management in programming. In Python, you can use atry …finally statement to handle opening and closing files properly:

# Safely open the filefile=open("hello.txt","w")try:file.write("Hello, World!")finally:# Make sure to close the file after using itfile.close()

In this example, you need to safely open the filehello.txt, which you can do by wrapping the call toopen() in atry …except statement. Later, when you try to write tofile, thefinally clause will guarantee thatfile is properly closed, even if an exception occurs during the call to.write() in thetry clause. You can use this pattern to handle setup and teardown logic when you’re managing external resources in Python.

Thetry block in the above example can potentially raise exceptions, such asAttributeError orNameError. You can handle those exceptions in anexcept clause like this:

# Safely open the filefile=open("hello.txt","w")try:file.write("Hello, World!")exceptExceptionase:print(f"An error occurred while writing to the file:{e}")finally:# Make sure to close the file after using itfile.close()

In this example, you catch any potential exceptions that can occur while writing to the file. In real-life situations, you should use a specificexception type instead of the generalException to prevent unknown errors from passing silently.

Thewith Statement Approach

The Pythonwith statement creates aruntime context that allows you to run a group of statements under the control of acontext manager.PEP 343 added thewith statement to make it possible to factor out standard use cases of thetry …finally statement.

Compared to traditionaltry …finally constructs, thewith statement can make your code clearer, safer, and reusable. Many classes in thestandard library support thewith statement. A classic example of this isopen(), which allows you to work withfile objects usingwith.

To write awith statement, you need to use the following general syntax:

withexpressionastarget_var:do_something(target_var)

The context manager object results from evaluating theexpression afterwith. In other words,expression must return an object that implements thecontext management protocol. This protocol consists of twospecial methods:

1. .__enter__() is called by thewith statement to enter the runtime context.
2. .__exit__() is called when the execution leaves thewith code block.

Theas specifier is optional. If you provide atarget_var withas, then thereturn value of calling.__enter__() on the context manager object is bound to that variable.

Here’s how thewith statement proceeds when Python runs into it:

1. Callexpression to obtain a context manager.
2. Store the context manager’s.__enter__() and.__exit__() methods for later use.
3. Call.__enter__() on the context manager and bind its return value totarget_var if provided.
4. Execute thewith code block.
5. Call.__exit__() on the context manager when thewith code block finishes.

In this case,.__enter__(), typically provides the setup code. Thewith statement is acompound statement that starts a code block, like aconditional statement or afor loop. Inside this code block, you can run several statements. Typically, you use thewith code block to manipulatetarget_var if applicable.

Once thewith code block finishes,.__exit__() gets called. This method typically provides the teardown logic or cleanup code, such as calling.close() on an open file object. That’s why thewith statement is so useful. It makes properly acquiring and releasing resources a breeze.

Here’s how to open yourhello.txt file for writing using thewith statement:

withopen("hello.txt",mode="w")asfile:file.write("Hello, World!")

When you run thiswith statement,open() returns anio.TextIOBase object. This object is also a context manager, so thewith statement calls.__enter__() and assigns its return value tofile. Then you can manipulate the file inside thewith code block. When the block ends,.__exit__() automatically gets called and closes the file for you, even if an exception is raised inside thewith block.

Thiswith construct is shorter than itstry …finally alternative, but it’s also less general, as you already saw. You can only use thewith statement with objects that support the context management protocol, whereastry …finally allows you to perform cleanup actions for arbitrary objects without the need for supporting the context management protocol.

In Python 3.1 and later, thewith statementsupports multiple context managers. You can supply any number of context managers separated by commas:

withA()asa,B()asb:pass

This works like nestedwith statements but without nesting. This might be useful when you need to open two files at a time, the first for reading and the second for writing:

withopen("input.txt")asin_file,open("output.txt","w")asout_file:# Read content from input.txt# Transform the content# Write the transformed content to output.txtpass

In this example, you can add code for reading and transforming the content ofinput.txt. Then you write the final result tooutput.txt in the same code block.

Using multiple context managers in a singlewith has a drawback, though. If you use this feature, then you’ll probably break your line length limit. To work around this, you need to use backslashes (\) for line continuation, so you might end up with an ugly final result.

Thewith statement can make the code that deals with system resources more readable, reusable, and concise, not to mention safer. It helps avoid bugs and leaks by making it almost impossible to forget cleaning up, closing, and releasing a resource after you’re done with it.

Usingwith allows you to abstract away most of the resource handling logic. Instead of having to write an explicittry …finally statement with setup and teardown code each time,with takes care of that for you and avoids repetition.

Working With Files

So far, you’ve usedopen() to provide a context manager and manipulate files in awith construct. Opening files using thewith statement is generally recommended because it ensures that openfile descriptors are automatically closed after the flow of execution leaves thewith code block.

As you saw before, the most common way to open a file usingwith is through the built-inopen():

withopen("hello.txt",mode="w")asfile:file.write("Hello, World!")

In this case, since the context manager closes the file after leaving thewith code block, a common mistake might be the following:

>>>file=open("hello.txt",mode="w")>>>withfile:...file.write("Hello, World!")...13>>>withfile:...file.write("Welcome to Real Python!")...Traceback (most recent call last):  File"<stdin>", line1, in<module>ValueError:I/O operation on closed file.

The firstwith successfully writes"Hello, World!" intohello.txt. Note that.write() returns the number of bytes written into the file,13. When you try to run a secondwith, however, you get aValueError because yourfile is already closed.

Another way to use thewith statement to open and manage files is by usingpathlib.Path.open():

>>>importpathlib>>>file_path=pathlib.Path("hello.txt")>>>withfile_path.open("w")asfile:...file.write("Hello, World!")...13

Path is a class that represents concrete paths to physical files in your computer. Calling.open() on aPath object that points to a physical file opens it just likeopen() would do. So,Path.open() works similarly toopen(), but the file path is automatically provided by thePath object you call the method on.

Sincepathlib provides an elegant, straightforward, andPythonic way to manipulate file system paths, you should consider usingPath.open() in yourwith statements as a best practice in Python.

Finally, whenever you load an external file, your program should check for possible issues, such as a missing file, writing and reading access, and so on. Here’s a general pattern that you should consider using when you’re working with files:

importpathlibimportloggingfile_path=pathlib.Path("hello.txt")try:withfile_path.open(mode="w")asfile:file.write("Hello, World!")exceptOSErroraserror:logging.error("Writing to file%s failed due to:%s",file_path,error)

In this example, you wrap thewith statement in atry …except statement. If anOSError occurs during the execution ofwith, then you uselogging to log the error with a user-friendly and descriptive message.

Traversing Directories

Theos module provides a function calledscandir(), which returns an iterator overos.DirEntry objects corresponding to the entries in a given directory. This function is specially designed to provide optimal performance when you’re traversing a directory structure.

A call toscandir() with the path to a given directory as an argument returns an iterator that supports the context management protocol:

>>>importos>>>withos.scandir(".")asentries:...forentryinentries:...print(entry.name,"->",entry.stat().st_size,"bytes")...Documents -> 4096 bytesVideos -> 12288 bytesDesktop -> 4096 bytesDevSpace -> 4096 bytes.profile -> 807 bytesTemplates -> 4096 bytesPictures -> 12288 bytesPublic -> 4096 bytesDownloads -> 4096 bytes

In this example, you write awith statement withos.scandir() as the context manager supplier. Then you iterate over the entries in the selected directory (".") andprint their name and size on the screen. In this case,.__exit__() callsscandir.close() to close the iterator and release the acquired resources. Note that if you run this on your machine, you’ll get a different output depending on the content of your current directory.

Performing High-Precision Calculations

Unlike built-infloating-point numbers, thedecimal module provides a way to adjust the precision to use in a given calculation that involvesDecimal numbers. The precision defaults to28 places, but you can change it to meet your problem requirements. A quick way to perform calculations with a custom precision is usinglocalcontext() fromdecimal:

>>>fromdecimalimportDecimal,localcontext>>>withlocalcontext()asctx:...ctx.prec=42...Decimal("1")/Decimal("42")...Decimal('0.0238095238095238095238095238095238095238095')>>>Decimal("1")/Decimal("42")Decimal('0.02380952380952380952380952381')

Here,localcontext() provides a context manager that creates a local decimal context and allows you to perform calculations using a custom precision. In thewith code block, you need to set.prec to the new precision you want to use, which is42 places in the example above. When thewith code block finishes, the precision is reset back to its default value,28 places.

Handling Locks in Multithreaded Programs

Another good example of using thewith statement effectively in the Python standard library isthreading.Lock. This class provides a primitive lock to prevent multiple threads from modifying a shared resource at the same time in amultithreaded application.

You can use aLock object as the context manager in awith statement to automatically acquire and release a given lock. For example, say you need to protect the balance of a bank account:

importthreadingbalance_lock=threading.Lock()# Use the try ... finally patternbalance_lock.acquire()try:# Update the account balance here ...finally:balance_lock.release()# Use the with patternwithbalance_lock:# Update the account balance here ...

Thewith statement in the second example automatically acquires and releases a lock when the flow of execution enters and leaves the statement. This way, you can focus on what really matters in your code and forget about those repetitive operations.

In this example, the lock in thewith statement creates a protected region known as thecritical section, which prevents concurrent access to the account balance.

Testing for Exceptions With pytest

So far, you’ve coded a few examples using context managers that are available in the Python standard library. However, several third-party libraries include objects that support the context management protocol.

Say you’retesting your code withpytest. Some of your functions and code blocks raise exceptions under certain situations, and you want to test those cases. To do that, you can usepytest.raises(). This function allows you to assert that a code block or a function call raises a given exception.

Sincepytest.raises() provides a context manager, you can use it in awith statement like this:

>>>importpytest>>>1/0Traceback (most recent call last):  File"<stdin>", line1, in<module>ZeroDivisionError:division by zero>>>withpytest.raises(ZeroDivisionError):...1/0...>>>favorites={"fruit":"apple","pet":"dog"}>>>favorites["car"]Traceback (most recent call last):  File"<stdin>", line1, in<module>KeyError:'car'>>>withpytest.raises(KeyError):...favorites["car"]...

In the first example, you usepytest.raises() to capture theZeroDivisionError that the expression1 / 0 raises. The second example uses the function to capture theKeyError that is raised when you access a key that doesn’t exist in a given dictionary.

If your function or code block doesn’t raise the expected exception, thenpytest.raises() raises a failure exception:

>>>importpytest>>>withpytest.raises(ZeroDivisionError):...4/2...2.0Traceback (most recent call last):...Failed:DID NOT RAISE <class 'ZeroDivisionError'>

Another cool feature ofpytest.raises() is that you can specify a target variable to inspect the raised exception. For example, if you want to verify the error message, then you can do something like this:

>>>withpytest.raises(ZeroDivisionError)asexc:...1/0...>>>assertstr(exc.value)=="division by zero"

You can use all thesepytest.raises() features to capture the exceptions you raise from yourfunctions and code block. This is a cool and useful tool that you can incorporate into your current testing strategy.

Writing a Sample Class-Based Context Manager

Here’s a sample class-based context manager that implements both methods,.__enter__() and.__exit__(). It also shows how Python calls them in awith construct:

>>>classHelloContextManager:...def__enter__(self):...print("Entering the context...")...return"Hello, World!"...def__exit__(self,exc_type,exc_value,exc_tb):...print("Leaving the context...")...print(exc_type,exc_value,exc_tb,sep="\n")...>>>withHelloContextManager()ashello:...print(hello)...Entering the context...Hello, World!Leaving the context...NoneNoneNone

HelloContextManager implements both.__enter__() and.__exit__(). In.__enter__(), you first print a message to signal that the flow of execution is entering a new context. Then you return the"Hello, World!" string. In.__exit__(), you print a message to signal that the flow of execution is leaving the context. You also print the content of its three arguments.

When thewith statement runs, Python creates a new instance ofHelloContextManager and calls its.__enter__() method. You know this because you getEntering the context... printed on the screen.

>>>withHelloContextManagerashello:...print(hello)...Traceback (most recent call last):  File"<stdin>", line1, in<module>AttributeError:__enter__

Then Python runs thewith code block, which printshello to the screen. Note thathello holds the return value of.__enter__().

When the flow of execution exits thewith code block, Python calls.__exit__(). You know that because you getLeaving the context... printed on your screen. The final line in the output confirms that the three arguments to.__exit__() are set toNone.

Now, what happens if an exception occurs during the execution of thewith block? Go ahead and write the followingwith statement:

>>>withHelloContextManager()ashello:...print(hello)...hello[100]...Entering the context...Hello, World!Leaving the context...<class 'IndexError'>string index out of range<traceback object at 0x7f0cebcdd080>Traceback (most recent call last):  File"<stdin>", line3, in<module>IndexError:string index out of range

In this case, you try to retrieve the value at index100 in thestring"Hello, World!". This raises anIndexError, and the arguments to.__exit__() are set to the following:

• exc_type is the exception class,IndexError.
• exc_value is the exception instance.
• exc_tb is the traceback object.

This behavior is quite useful when you want to encapsulate the exception handling in your context managers.

Handling Exceptions in a Context Manager

As an example of encapsulating exception handling in a context manager, say you expectIndexError to be the most common exception when you’re working withHelloContextManager. You might want to handle that exception in the context manager so you don’t have to repeat the exception-handling code in everywith code block. In that case, you can do something like this:

# exc_handling.pyclassHelloContextManager:def__enter__(self):print("Entering the context...")return"Hello, World!"def__exit__(self,exc_type,exc_value,exc_tb):print("Leaving the context...")ifisinstance(exc_value,IndexError):# Handle IndexError here...print(f"An exception occurred in your with block:{exc_type}")print(f"Exception message:{exc_value}")returnTruewithHelloContextManager()ashello:print(hello)hello[100]print("Continue normally from here...")

In.__exit__(), you check ifexc_value is an instance ofIndexError. If so, then you print a couple of informative messages and finally return withTrue. Returning atruthy value makes it possible to swallow the exception and continue the normal execution after thewith code block.

In this example, if noIndexError occurs, then the method returnsNone and the exception propagates out. However, if you want to be more explicit, then you can returnFalse from outside theif block.

If you runexc_handling.py from your command line, then you get the following output:

$pythonexc_handling.pyEntering the context...Hello, World!Leaving the context...An exception occurred in your with block: <class 'IndexError'>Exception message: string index out of rangeContinue normally from here...

HelloContextManager is now able to handleIndexError exceptions that occur in thewith code block. Since you returnTrue when anIndexError occurs, the flow of execution continues in the next line, right after exiting thewith code block.

Opening Files for Writing: First Version

Now that you know how to implement the context management protocol, you can get a sense of what this would look like by coding a practical example. Here’s how you can take advantage ofopen() to create a context manager that opens files for writing:

# writable.pyclassWritableFile:def__init__(self,file_path):self.file_path=file_pathdef__enter__(self):self.file_obj=open(self.file_path,mode="w")returnself.file_objdef__exit__(self,exc_type,exc_val,exc_tb):ifself.file_obj:self.file_obj.close()

WritableFile implements the context management protocol and supports thewith statement, just like the originalopen() does, but it always opens the file for writing using the"w" mode. Here’s how you can use your new context manager:

>>>fromwritableimportWritableFile>>>withWritableFile("hello.txt")asfile:...file.write("Hello, World!")...

After running this code, yourhello.txt file contains the"Hello, World!" string. As an exercise, you can write a complementary context manager that opens files for reading, but usingpathlib functionalities. Go ahead and give it a shot!

Redirecting the Standard Output

A subtle detail to consider when you’re writing your own context managers is that sometimes you don’t have a useful object to return from.__enter__() and therefore to assign to thewith target variable. In those cases, you can returnNone explicitly or you can just rely on Python’simplicit return value, which isNone as well.

For example, say you need to temporarily redirect the standard output,sys.stdout, to a given file on your disk. To do this, you can create a context manager like this:

# redirect.pyimportsysclassRedirectedStdout:def__init__(self,new_output):self.new_output=new_outputdef__enter__(self):self.saved_output=sys.stdoutsys.stdout=self.new_outputdef__exit__(self,exc_type,exc_val,exc_tb):sys.stdout=self.saved_output

This context manager takes a file object through its constructor. In.__enter__(), you reassign the standard output,sys.stdout, to an instance attribute to avoid losing the reference to it. Then you reassign the standard output to point to the file on your disk. In.__exit__(), you just restore the standard output to its original value.

To useRedirectedStdout, you can do something like this:

>>>fromredirectimportRedirectedStdout>>>withopen("hello.txt","w")asfile:...withRedirectedStdout(file):...print("Hello, World!")...print("Back to the standard output...")...Back to the standard output...

The outerwith statement in this example provides the file object that you’re going to use as your new output,hello.txt. The innerwith temporarily redirects the standard output tohello.txt, so the first call toprint() writes directly to that file instead of printing"Hello, World!" on your screen. Note that when you leave the innerwith code block, the standard output goes back to its original value.

RedirectedStdout is a quick example of a context manager that doesn’t have a useful value to return from.__enter__(). However, if you’re only redirecting theprint() output, you can get the same functionality without the need for coding a context manager. You just need to provide afile argument toprint() like this:

>>>withopen("hello.txt","w")asfile:...print("Hello, World!",file=file)...

In this examples,print() takes yourhello.txt file as an argument. This causesprint() to write directly into the physical file on your disk instead of printing"Hello, World!" to your screen.

Measuring Execution Time

Just like every other class, a context manager can encapsulate some internalstate. The following example shows how to create astateful context manager to measure the execution time of a given code block or function:

# timing.pyfromtimeimportperf_counterclassTimer:def__enter__(self):self.start=perf_counter()self.end=0.0returnlambda:self.end-self.startdef__exit__(self,*args):self.end=perf_counter()

When you useTimer in awith statement,.__enter__() gets called. This method usestime.perf_counter() to get the time at the beginning of thewith code block and stores it in.start. It also initializes.end and returns alambda function that computes a time delta. In this case,.start holds the initial state or time measurement.

Once thewith block ends,.__exit__() gets called. The method gets the time at the end of the block and updates the value of.end so that thelambda function can compute the time required to run thewith code block.

Here’s how you can use this context manager in your code:

>>>fromtimeimportsleep>>>fromtimingimportTimer>>>withTimer()astimer:...# Time-consuming code goes here......sleep(0.5)...>>>timer()0.5005456680000862

WithTimer, you can measure the execution time of any piece of code. In this example,timer holds an instance of thelambda function that computes the time delta, so you need to calltimer() to get the final result.

Opening Files for Writing: Second Version

You can use the@contextmanager to reimplement yourWritableFile context manager. Here’s what rewriting it with this technique looks like:

>>>fromcontextlibimportcontextmanager>>>@contextmanager...defwritable_file(file_path):...file=open(file_path,mode="w")...try:...yieldfile...finally:...file.close()...>>>withwritable_file("hello.txt")asfile:...file.write("Hello, World!")...

In this case,writable_file() is a generator function that opensfile for writing. Then it temporarily suspends its own execution andyields the resource sowith can bind it to its target variable. When the flow of execution leaves thewith code block, the function continues to execute and closesfile correctly.

Mocking the Time

As a final example of how to create custom context managers with@contextmanager, say you’re testing a piece of code that works with time measurements. The code usestime.time() to get the current time measurement and do some further computations. Since time measurements vary, you decide to mocktime.time() so you can test your code.

Here’s a function-based context manager that can help you do that:

>>>fromcontextlibimportcontextmanager>>>fromtimeimporttime>>>@contextmanager...defmock_time():...globaltime...saved_time=time...time=lambda:42...yield...time=saved_time...>>>withmock_time():...print(f"Mocked time:{time()}")...Mocked time: 42>>># Back to normal time>>>time()1616075222.4410584

Insidemock_time(), you use aglobal statement to signal that you’re going to modify the global nametime. Then you save the originaltime() function object insaved_time so you can safely restore it later. The next step is tomonkey patchtime() using alambda function that always returns the same value,42.

The bareyield statement specifies that this context manager doesn’t have a useful object to send back to thewith target variable for later use. Afteryield, you reset the globaltime to its original content.

When the execution enters thewith block, any calls totime() return42. Once you leave thewith code block, calls totime() return the expected current time. That’s it! Now you can test your time-related code.