Thetry …finally Construct

Working with files is probably the most common example of resource management in programming. In Python, you can use atry …finally construct to handle opening and closing files properly:

file=open("hello.txt","w")try:file.write("Hello, World!")finally:file.close()

In this example, you open thehello.txt file usingopen(). To write some text into the file, you wrap the call to.write() in atry statement with afinally clause. This clause guarantees that the file is properly closed by calling.close(), even if an exception occurs during the call to.write() in thetry clause. Remember that thefinally clause always runs.

When managing external resources in Python, you can use the construct in the previous example to handle setup and teardown logic. The setup logic might include opening the file and writing content to it, while the teardown logic might consist of closing the file to release the acquired resources.

Thetry block in the example above can potentially raise exceptions, such asAttributeError orNameError. You can handle these exceptions with anexcept clause, as shown in the following example:

file=open("hello.txt","w")try:file.write("Hello, World!")exceptExceptionase:print(f"An error occurred while writing to the file:{e}")finally:file.close()

In this example, you use theException class to catch exceptions that can occur while writing to the file. Then, you print an error message to inform the user. Thefinally clause always runs so that the file is closed correctly.

Even though thetry …finally construct works correctly, Python has a more compact and elegant solution for you: thewith statement.

Thewith Statement

The Pythonwith statement creates aruntime context that allows you to execute a code block under the control of acontext manager.PEP 343 added thewith statement to make it possible to factor out common use cases of thetry …finally construct.

Compared to the traditionaltry …finally construct, thewith statement makes your code clearer, safer, and more readable. Many classes and objects in thestandard library support thewith statement by implementing the context managerprotocol (special methods). For example, when you callopen(), you get afile object that supports this protocol and, therefore, thewith statement.

To write awith statement, you need to use the following general syntax:

withexpression[asobj]:<block>

The context manager object results from evaluating theexpression after thewith keyword. In other words,expression must return an object that implements the context management protocol. This protocol consists of twospecial methods:

1. .__enter__() is called by thewith statement to enter the runtime context.
2. .__exit__() is called when the execution leaves thewith code block.

Theas specifier is optional. If you provide anobj variable withas, then thereturn value of calling.__enter__() on the context manager object is bound to that variable.

Here’s how thewith statement works internally:

1. Executeexpression to obtain a context manager object.
2. Call.__enter__() on the context manager and bind its return value toobj if provided.
3. Execute thewith code block.
4. Call.__exit__() on the context manager when thewith code block finishes.

The.__enter__() method typically provides thesetup logic. Thewith statement is acompound statement that starts a code block, like other compound statements, such asconditional statements andfor loops.

Inside this code block, you can run one or more statements. Typically, you use thewith code block to manipulateobj if applicable.

Once thewith code block finishes, Python automatically calls.__exit__(). This method typically provides theteardown logic or cleanup code, such as calling.close() on an open file object. That’s why thewith statement is so useful. It makes properly acquiring and releasing resources a breeze.

Here’s how to open yourhello.txt file for writing using thewith statement:

withopen("hello.txt",mode="w",encoding="utf-8")asfile:file.write("Hello, World!")

When you run thiswith statement,open() returns anio.TextIOBase object. This object supports the context manager protocol, so thewith statement calls.__enter__() and assigns its return value tofile. Then, you can manipulate the file inside thewith code block. When the block ends, Python automatically calls.__exit__(), which closes the file for you, even if an exception occurs in thewith code block.

Thiswith construct is shorter than itstry …finally equivalent construct, but it’s also less generic because you can only use thewith statement with objects that support the context management protocol. In contrast,try …finally allows you to perform cleanup actions for any object, even if it doesn’t implement the context management protocol.

Thewith statement also supportsmultiple context managers. You can supply any number of context managers separated by commas:

withA()asa,B()asb:<block>

This works like nestedwith statements, but without nesting. It might be useful when you need to open two files at a time—the first for reading and the second for writing:

withopen("hello.txt")asin_file,open("output.txt","w")asout_file:forlineinin_file:out_file.write(line.upper())

In this example, you read and transform the content ofhello.txt line by line. Then, you write each processed line tooutput.txt within the same code block.

Thewith statement can make the code that deals with system resources more readable, concise, and safer. It helps avoid resource leaks by making it almost impossible to forget to clean up, close, and release resources after you’re done with them.

Usingwith allows you to abstract away most of the resource handling logic. Instead of using explicittry …finally constructs with setup and teardown logic, you can pack this logic into a context manager and handle it using thewith statement to avoid repetition.

Working With Files

So far, you’ve usedopen() to manipulate files in awith construct. Managing files this way is generally recommended because it ensures that openedfile descriptors are automatically closed after the flow of execution leaves thewith code block.

Again, a common way to open a file in awith statement is through theopen() function:

withopen("hello.txt",mode="w",encoding="utf-8")asfile:file.write("Hello, World!")

In this example, the context manager closes the file after leaving thewith code block. A common mistake you might encounter is shown below:

>>>file=open("hello.txt",mode="w",encoding="utf-8")>>>withfile:...file.write("Hello, World!")...13>>>withfile:...file.write("Welcome to Real Python!")...Traceback (most recent call last):...ValueError:I/O operation on closed file.

The firstwith successfully writes"Hello, World!" intohello.txt. Note that.write() returns the number of bytes written into the file—13 bytes in this example. When you try to run a secondwith, you get aValueError because the file is already closed.

Another way to use thewith statement to open and manage files is by using the.open() method on aPath object:

>>>importpathlib>>>file_path=pathlib.Path("hello.txt")>>>withfile_path.open("w",encoding="utf-8")asfile:...file.write("Hello, World!")...13

Path is a class that represents concrete paths to physical files on your computer. Calling.open() on aPath object that points to a physical file opens it just likeopen() would. So,Path.open() works similarly toopen(), but the file path is automatically provided by thePath object you call the method on.

Thepathlib package provides an elegant, straightforward, andPythonic way to manipulate file system paths. You should consider usingPath.open() in yourwith statements as a best practice in Python.

Finally, whenever you load an external file, your program should check for possible issues, such as a missing file, writing and reading access, and so on. Here’s a pattern that you should consider using when working with files:

>>>importpathlib>>>importlogging>>>file_path=pathlib.Path("/hello.txt")>>>try:...withfile_path.open(mode="w")asfile:...file.write("Hello, World!")...exceptOSErroraserror:...logging.error("Writing to file%s failed due to:%s",file_path,error)...ERROR:root:Writing to file /hello.txt failed due to: [Errno 13]⮑ Permission denied: '/hello.txt'

In this example, you wrap thewith statement in atry …except block. If anOSError occurs during the execution ofwith, then you use thelogging module to log the error with a user-friendly and descriptive message.

Traversing Directories

Theos module provides a function calledscandir() that returns aniterator overos.DirEntry objects corresponding to the entries in a given directory. This function is specially designed to provide optimal performance when you’re traversing a directory structure.

A call toscandir() with a directory path as an argument returns an iterator that supports the context management protocol:

>>>importos>>>withos.scandir(".")asentries:...forentryinentries:...print(entry.name,"->",entry.stat().st_size,"bytes")...Documents -> 4096 bytesVideos -> 12288 bytesDesktop -> 4096 bytesDevSpace -> 4096 bytes.profile -> 807 bytesTemplates -> 4096 bytesPictures -> 12288 bytesPublic -> 4096 bytesDownloads -> 4096 bytes

In this example, you write awith statement withos.scandir() as the context manager supplier. Then, you iterate over the entries in theworking directory represented by"." andprint their names and sizes on the screen. In this case,.__exit__() callsscandir.close() to close the iterator and release the acquired resources.

Note that if you run this example on your machine, you’ll get a different output depending on the content of your target directory.

Performing High-Precision Calculations

Thedecimal module provides a handy way to tweak the precision to use in a given calculation that involvesdecimal numbers. The precision defaults to28 places, but you can change it to meet your specific requirements.

A quick way to perform calculations with a custom precision is usinglocalcontext() fromdecimal:

>>>fromdecimalimportDecimal,localcontext>>>withlocalcontext(prec=42):...Decimal("1")/Decimal("42")...Decimal('0.0238095238095238095238095238095238095238095')>>>Decimal("1")/Decimal("42")Decimal('0.02380952380952380952380952381')

Here,localcontext() returns a context manager that creates a context allowing you to perform calculations using a custom precision. In thewith code block, you have a precision of42 places. When thewith code block finishes, the precision is back to its default value of28 places.

Handling Locks in Multithreaded Programs

Thethreading.Lock class in the Python standard library also supports thewith statement. This class provides a primitive lock to prevent multiple threads from modifying a shared resource at the same time in amultithreaded application.

You can use aLock object in awith statement to automatically acquire and release a given lock. For example, say you need to protect the balance of a bank account:

importthreadingbalance_lock=threading.Lock()# Use the try ... finally patternbalance_lock.acquire()try:# Update the account balance here ...finally:balance_lock.release()# Use the with patternwithbalance_lock:# Update the account balance here ...

Thewith statement in the second example automatically acquires and releases a lock when the flow of execution enters and leaves the statement. This way, you can focus on what really matters in your code and forget about those repetitive operations.

In this example, the lock in thewith statement creates a protected region known as thecritical section, which prevents concurrent access to the account balance.

Testing for Exceptions With pytest

So far, you’ve coded a few examples using context managers that are available in the Python standard library. However, several third-party libraries include objects that also support the context management protocol.

Say you’retesting your code withpytest. Some of your functions and code blocks raise exceptions under certain situations, and you want to test those cases. To do that, you can usepytest.raises(), which allows you to assert that a block of code or a function call raises a specific exception.

Thepytest.raises() function returns a context manager, so you can use that object in awith statement:

>>>importpytest>>>1/0Traceback (most recent call last):...ZeroDivisionError:division by zero>>>withpytest.raises(ZeroDivisionError):...1/0...>>>favorites={"fruit":"apple","pet":"dog"}>>>favorites["car"]Traceback (most recent call last):...KeyError:'car'>>>withpytest.raises(KeyError):...favorites["car"]...

In the first example, you usepytest.raises() to capture theZeroDivisionError exception that the expression1 / 0 raises. The second example uses this function to capture theKeyError that’s raised when you access a key that doesn’t exist in a given dictionary.

If your function or code block doesn’t raise the expected exception, thenpytest.raises() raises a failure exception:

>>>withpytest.raises(ZeroDivisionError):...4/2...2.0Traceback (most recent call last):...Failed:DID NOT RAISE <class 'ZeroDivisionError'>

Another cool feature ofpytest.raises() is that you can specify a target variable to inspect the raised exception. For example, if you want to verify the error message, then you can do something like this:

>>>withpytest.raises(ZeroDivisionError)asexc:...1/0...>>>assertstr(exc.value)=="division by zero"

You can use all thesepytest.raises() features to capture the exceptions you raise from your functions and code blocks. This is a cool and useful tool that you can incorporate into your current testing strategy.

Writing a Demo Class-Based Context Manager

When thewith statement executes, Python calls.__enter__() on the context manager object to set up the new runtime context. If you provide a target variable with theas specifier, then the return value of.__enter__() is assigned to that variable. When the flow of execution leaves the context,.__exit__() is called.

Below is a demo class-based context manager that shows how Python calls the.__enter__() and.__exit__() methods in awith construct:

>>>classHelloContextManager:...def__enter__(self):...print("Entering the context...")...return"Hello, World!"......def__exit__(self,exc_type,exc_value,exc_tb):...print("Leaving the context...")...print(f"{exc_type= }")...print(f"{exc_value= }")...print(f"{exc_tb= }")...>>>withHelloContextManager()ashello:...print(hello)...Entering the context...Hello, World!Leaving the context...exc_type  = Noneexc_value = Noneexc_tb    = None

In.__enter__(), you print a message to indicate that the flow of execution is entering a new context. Then, you return the"Hello, World!" string. In.__exit__(), you print a message to signal that the execution flow is leaving the context. You also print the content of the three arguments.

When thewith statement runs, Python creates a new instance ofHelloContextManager and calls its.__enter__() method. You know this because you getEntering the context... printed on the screen. Next, Python runs thewith code block, which printshello to the screen. Note thathello holds the return value of.__enter__(), which is"Hello, World!" in this example.

When the execution flow exits thewith code block, Python calls.__exit__(). You know that because you getLeaving the context... printed on your screen. If no exception occurs in thewith code block, then the three arguments to.__exit__() are set toNone. You confirm this behavior in the final output lines.

Now, what if an exception occurs during the execution of thewith block? To find out, go ahead and run the followingwith statement:

>>>withHelloContextManager()ashello:...print(hello)...hello[100]...Entering the context...Hello, World!Leaving the context...exc_type  = <class 'IndexError'>exc_value = IndexError('string index out of range')exc_tb    = <traceback object at 0x7f0cebcdd080>Traceback (most recent call last):...IndexError:string index out of range

In this example, you try to retrieve the value at index100 in thestring"Hello, World!". This action raises anIndexError exception because the string doesn’t have 100 characters. Therefore, the arguments to.__exit__() are set as follows:

• exc_type: the exception class,IndexError.
• exc_value: the concrete exception object with a descriptive message.
• exc_tb: the traceback object,<traceback object at 0x7f0cebcdd080>.

This behavior is especially useful when you want your context manager to handle exceptions internally.

Handling Exceptions Within Context Managers

If the.__exit__() method returnsTrue, then any exception that occurs in thewith block is swallowed and the execution continues at the next statement afterwith. If.__exit__() returnsFalse, then exceptions are propagated out of the context. This is also the default behavior when the method doesn’t return anything explicitly.

You can take advantage of these behaviors to encapsulate exception handling inside the context manager or to propagate the exceptions as needed.

As an example of encapsulating exception handling in a context manager, say you expectIndexError to be a possible exception when you’re working withHelloContextManager. You might want to handle that exception in the context manager so you don’t have to repeat the exception-handling code in everywith code block.

In that situation, you can do something like the following:

>>>classHelloContextManager:...def__enter__(self):...print("Entering the context...")...return"Hello, World!"......def__exit__(self,exc_type,exc_value,exc_tb):...print("Leaving the context...")...ifisinstance(exc_value,IndexError):...# Handle IndexError here......print(f"An exception occurred in your with block:{exc_type}")...print(f"Exception message:{exc_value}")...returnTrue...>>>withHelloContextManager()ashello:...print(hello)...hello[100]...Entering the context...Hello, World!Leaving the context...An exception occurred in your with block: <class 'IndexError'>Exception message: string index out of range

In.__exit__(), you check ifexc_value is an instance ofIndexError using the built-inisinstance() function. If so, you print a couple of informative messages and finally return withTrue.

Again, returning aTrue makes it possible to swallow the exception and continue the normal execution after thewith code block.

In this example, if noIndexError occurs, then the method implicitly returnsNone, and the exception propagates out of the context. This is the desired behavior because it avoids hiding unexpected exceptions. If you want to be more explicit, then you can returnFalse outside theif block.

YourHelloContextManager class is now able to handleIndexError exceptions that occur in thewith code block. Since you returnTrue when anIndexError occurs, the execution flow continues in the next line, right after exiting thewith code block.

Opening Files for Writing

Now that you know how to implement the context management protocol, you can code a more realistic example. For instance, below is how you can create a context manager that opens files for writing directly:

>>>classWritableFile:...def__init__(self,file_path):...self.file_path=file_path......def__enter__(self):...self.file_obj=open(self.file_path,mode="w")...returnself.file_obj......def__exit__(self,*_):...ifself.file_obj:...self.file_obj.close()...

WritableFile allows you to open the file for writing using the"w" mode ofopen(). You do this in the.__enter__() method. In the.__exit__() method, you close the file to release the acquired resources. Note how you use the*args syntax to maintain compatibility with the context management protocol. Since you don’t need these arguments, you use asingle underscore to suppress a potentiallinter warning about unused variables.

Here’s how you can use your context manager:

>>>withWritableFile("hello.txt")asfile:...file.write("Hello, World!")...13

After running this code, you’ll have ahello.txt file containing the"Hello, World!" string. As an exercise, you can write a complementary context manager that opens files for reading, but usingpathlib functionalities. Go ahead and give it a try!

Redirecting the Standard Output to a File

A subtle detail to consider when you’re implementing the context manager protocol is that sometimes you don’t have a useful object to return from.__enter__()—and therefore nothing meaningful to assign to the target variable in thewith header. In those cases, you can explicitly returnNone, or just rely on Python’simplicit return value, which is alsoNone.

For example, say you need to temporarily redirect the standard output,sys.stdout, to a given file on your disk. To do this, you can create a context manager like the following:

>>>importsys>>>classStandardOutputRedirector:...def__init__(self,new_output):...self.new_output=new_output......def__enter__(self):...self.std_output=sys.stdout...sys.stdout=self.new_output......def__exit__(self,*_):...sys.stdout=self.std_output...

This context manager takes a file object as an argument. In.__enter__(), you reassign the standard output,sys.stdout, to an instance attribute to avoid losing the reference to it. Then, you reassign the standard output to point to the file on your disk. In.__exit__(), you restore the standard output to its original value.

To useStandardOutputRedirector, you can do something like this:

>>>withopen("hello.txt","w")asfile:...withStandardOutputRedirector(file):...print("Hello, World!")...print("Back to the standard output...")...Back to the standard output...

The outerwith statement provides the file object that you’ll use as your new output,hello.txt. The innerwith temporarily redirects the standard output tohello.txt, so the first call toprint() writes directly to that file instead of printing"Hello, World!" to your screen. When you leave the innerwith code block, the standard output is set back to its original value.

>>>withopen("hello.txt","w")asfile:...print("Hello, World!",file=file)...

StandardOutputRedirector is a quick example of a context manager that doesn’t have a useful value to return from.__enter__(). In this example, you rely on Python’s implicit behavior of returningNone.

Measuring Execution Time

Just like any other class, a context manager can encapsulate some internalstate. The following example shows how to create astateful context manager to measure the execution time of a given piece of code:

>>>fromtimeimportperf_counter,sleep>>>classTimer:...def__enter__(self):...self.start=perf_counter()......def__exit__(self,*_):...end=perf_counter()...print(f"Elapsed time:{end-self.start:.4f} seconds")...>>>withTimer():...# The code to measure goes here......sleep(0.5)...Elapsed time: 0.5051 seconds

In the.__enter__() method, you calltime.perf_counter() to get the current timestamp at the beginning of awith code block and store it in the.start attribute. This value represents the context manager’s initial state.

Once thewith block ends,.__exit__() gets called. The method gets the time at the end of the block and stores the value in a local variableend. Finally, it prints a message to inform you of the elapsed time.

Indenting Text

What if the resource you wanted to manage is thetext indentation level in some kind of report generator application? For example, say that you want to write code that generatesHTML output like the following:

<div><p>    Hello, World!</p></div>

This could be a great exercise to help you understand how context managers work. So, before you check out the implementation, you might take some time and try to produce this output with the help of a context manager that you can use as shown below:

withIndenter()asindenter:indenter.print("<div>")withindenter:indenter.print("<p>")withindenter:indenter.print("Hello, World!")indenter.print("</p>")indenter.print("</div>")

Notice how this code enters and leaves the same context manager multiple times to switch between different indentation levels.

Ready? Here’s how you can implement this functionality using a class-based context manager:

classIndenter:def__init__(self,width=2):self.indentation=" "*widthself.level=-1def__enter__(self):self.level+=1returnselfdef__exit__(self,*_):self.level-=1defprint(self,text):print(self.indentation*self.level+text)

Here,.__enter__() increments.level by1 every time the flow of execution enters the context. The method also returns the current instance,self. In.__exit__(), you decrease.level, so the printed text steps back one indentation level each time you exit the context.

The key point here is that returningself from.__enter__() allows you to reuse the same context manager across several nestedwith statements. This changes the text indentation level every time you enter and exit a given context.

A good exercise for you at this point would be to write a function-based version of this context manager. Function-based context managers is what you’ll be learning about next.