Cool New Features

Some of the currently announced highlights of Python 3.11 include:

• Exception groups, which will allow programs to raise and handle multiple exceptions at the same time
• Task groups, to improve how you run asynchronous code
• Enhanced error messages, which will help you more effectively debug your code
• Optimizations, promising to make Python 3.11 significantly faster than previous versions
• Static typing improvements, which will let youannotate your code more precisely
• TOML support, which allows you to parse TOML documents using the standard library

There’s a lot to look forward to in Python 3.11! For a comprehensive overview, check outPython 3.11: Cool New Features for You to Try. You can also dive deeper into some of the features listed above in the other articles in this series:

• Python 3.11 Preview: Even Better Error Messages
• Python 3.11 Preview: TOML andtomllib

In this tutorial, you’ll focus on how exception groups can handle multiple unrelated exceptions at once and how this feature paves the way for task groups, which make concurrent programming in Python more convenient. You’ll also get a peek at some of the other, smaller features that’ll be shipping with Python 3.11.

Installation

To play with the code examples in this tutorial, you’ll need to install a version of Python 3.11 onto your system. In this subsection, you’ll learn about a few different ways to do this: usingDocker, usingpyenv, or installing fromsource. Pick the one that works best for you and your system.

If you have access toDocker on your system, then you can download the latest version of Python 3.11 by pulling and running thepython:3.11-rc-slimDocker image:

$dockerpullpython:3.11-rc-slimUnable to find image 'python:3.11-rc-slim' locallylatest: Pulling from library/python[...]$dockerrun-it--rmpython:3.11-rc-slim

This drops you into a Python 3.11 REPL. Check outRun Python Versions in Docker for more information about working with Python through Docker, including how to run scripts.

Thepyenv tool is great for managing different versions of Python on your system, and you can use it to install Python 3.11 alpha if you like. It comes with two different versions, one for Windows and one for Linux and macOS. Choose your platform with the switcher below:

• Windows
• Linux + macOS

PS>pyenvupdate:: [Info] ::  Mirror: https://www.python.org/ftp/python[...]

$pyenvupdateUpdating /home/realpython/.pyenv...[...]

Usepyenv install --list to check which versions of Python 3.11 are available. Then, install the latest one:

$pyenvinstall3.11.0a7Downloading Python-3.11.0a7.tar.xz...[...]

The installation may take a few minutes. Once your new alpha version is installed, then you can create avirtual environment where you can play with it:

• Windows
• Linux + macOS

PS>pyenvlocal3.11.0a7PS>python--versionPython 3.11.0a7PS>python-mvenvvenvPS>venv\Scripts\activate

$pyenvvirtualenv3.11.0a7311_preview$pyenvactivate311_preview(311_preview)$python--versionPython 3.11.0a7

You can also install Python from one of pre-release versions available onpython.org. Choose thelatest pre-release and scroll down to theFiles section at the bottom of the page. Download and install the file corresponding to your system. SeePython 3 Installation & Setup Guide for more information.

Many of the examples in this tutorial will work on older versions of Python, but in general, you should run them with your Python 3.11 executable. Exactly how you run the executable depends on how you installed it. If you need help, see the relevant tutorial onDocker,pyenv,virtual environments, orinstalling from source.

Handle Regular Exceptions Withexcept

Before you explore exception groups, you’ll review how regular exception handling works in Python. If you’re already comfortable handling errors in Python, you won’t learn anything new in this subsection. However, this review will serve as a contrast to what you’ll learn about exception groups later. Everything you’ll see in this subsection of the tutorial works in all versions of Python 3, including Python 3.10.

Exceptions break the normal flow of a program. If an exception is raised, then Python drops everything else and looks for code that handles the error. If there are no such handlers, then the program stops, regardless of what the program was doing.

You can raise an error yourself using theraise keyword:

>>>raiseValueError(654)Traceback (most recent call last):...ValueError:654

Here, you explicitly raise aValueError with the description654. You can see that Python provides atraceback, which tells you that there’s an unhandled error.

Sometimes, you raise errors like this in your code to signal that something has gone wrong. However, it’s more common to encounter errors raised by Python itself or some library that you’re using. For example, Python doesn’t let you add a string and an integer, and raises aTypeError if you attempt this:

>>>"3"+11Traceback (most recent call last):...TypeError:can only concatenate str (not "int") to str

Most exceptions come with a description that can help you figure out what went wrong. In this case, it tells you that your second term should also be a string.

You usetry…except blocks to handle errors. Sometimes, you use these to just log the error and continue running. Other times, you manage to recover from the error or calculate some alternative value instead.

A shorttry…except block may look as follows:

>>>try:...raiseValueError(654)...exceptValueErroraserr:...print(f"Got a bad value:{err}")...Got a bad value: 654

You handleValueError exceptions by printing a message to your console. Note that because you handled the error, there’s no traceback in this example. However, other types of errors aren’t handled:

>>>try:..."3"+11...exceptValueErroraserr:...print(f"Got a bad value:{err}")...Traceback (most recent call last):...TypeError:can only concatenate str (not "int") to str

Even though the error happens within atry…except block, it’s not handled because there’s noexcept clause that matches aTypeError. You can handle several kinds of errors in one block:

>>>try:..."3"+11...exceptValueErroraserr:...print(f"Got a bad value:{err}")...exceptTypeErroraserr:...print(f"Got bad types:{err}")...Got bad types: can only concatenate str (not "int") to str

This example will handle bothValueError andTypeError exceptions.

Exceptions are defined in ahierarchy. For example, aModuleNotFoundError is a kind ofImportError, which is a kind ofException.

The firstexcept clause that matches the error will trigger the exception handling:

>>>try:...importno_such_module...exceptImportErroraserr:...print(f"ImportError:{err.__class__}")...exceptModuleNotFoundErroraserr:...print(f"ModuleNotFoundError:{err.__class__}")...ImportError: <class 'ModuleNotFoundError'>

When you try to import a module that doesn’t exist, Python raises aModuleNotFoundError. However, sinceModuleNotFoundError is a kind ofImportError, your error handling triggers theexcept ImportError clause. Note that:

• At most oneexcept clause will trigger
• The firstexcept clause that matches will trigger

If you’ve worked with exceptions before, this may seem intuitive. However, you’ll see later that exception groups behave differently.

While at most one exception is active at a time, it’s possible to chain related exceptions. This chaining was introduced byPEP 3134 for Python 3.0. As an example, observe what happens if you raise a new exception while handling an error:

>>>try:..."3"+11...exceptTypeError:...raiseValueError(654)...Traceback (most recent call last):...TypeError:can only concatenate str (not "int") to strDuring handling of the above exception, another exception occurred:Traceback (most recent call last):...ValueError:654

Note the lineDuring handling of the above exception, another exception occurred. There’s one traceback before this line, representing the originalTypeError caused by your code. Then, there’s another traceback below the line, representing the newValueError that you raised while handling theTypeError.

This behavior is particularly useful if you happen to have an issue in your error handling code, because you then get information about both your original error and the bug in your error handler.

You can also explicitly chain exceptions together yourself using araise…from statement. While you can use chained exceptions to raise several exceptions at once, note that the mechanism is meant for exceptions that are related, specificially where one exception happens during the handling of another.

This is different from the use case that exception groups are designed to handle. Exception groups will group together exceptions that are unrelated, in the sense that they happen independently of each other. When handling chained exceptions, you’re only able to catch and handle the last error in the chain. As you’ll learn soon, you can catch all the exceptions in an exception group.

Group Exceptions WithExceptionGroup

In this subsection, you’ll explore the newExceptionGroup class that’s available in Python 3.11. First, note that anExceptionGroup is also a kind ofException:

>>>issubclass(ExceptionGroup,Exception)True

AsExceptionGroup is a subclass ofException, you can use Python’s regular exception handling to work with it. You can raise anExceptionGroup withraise, although you probably won’t do that very often unless you’re implementing some low-level library. It’s also possible to catch anExceptionGroup withexcept ExceptionGroup. However, as you’ll learn in thenext subsection, you’re usually better off using the newexcept* syntax.

In contrast to most other exceptions, exception groups take two arguments when they’re initialized:

1. The usual description
2. A sequence of sub-exceptions

The sequence of sub-exceptions can include other exception groups, but it can’t be empty:

>>>ExceptionGroup("one error",[ValueError(654)])ExceptionGroup('one error', [ValueError(654)])>>>ExceptionGroup("two errors",[ValueError(654),TypeError("int")])ExceptionGroup('two errors', [ValueError(654), TypeError('int')])>>>ExceptionGroup("nested",...[...ValueError(654),...ExceptionGroup("imports",...[...ImportError("no_such_module"),...ModuleNotFoundError("another_module"),...]...),...]...)ExceptionGroup('nested', [ValueError(654), ExceptionGroup('imports',  [ImportError('no_such_module'), ModuleNotFoundError('another_module')])])>>>ExceptionGroup("no errors",[])Traceback (most recent call last):...ValueError:second argument (exceptions) must be a non-empty sequence

In this example, you’reinstantiating a few different exception groups that show that exception groups can contain one exception, several exceptions, and even other exception groups. Exception groups aren’t allowed to be empty, though.

Your first encounter with an exception group is likely to be its traceback. Exception group tracebacks are formatted to clearly show you the structure within the group. You’ll see a traceback when you raise an exception group:

>>>raiseExceptionGroup("nested",...[...ValueError(654),...ExceptionGroup("imports",...[...ImportError("no_such_module"),...ModuleNotFoundError("another_module"),...]...),...TypeError("int"),...]...)  + Exception Group Traceback (most recent call last):  |   ...  | ExceptionGroup: nested (3 sub-exceptions)  +-+---------------- 1 ----------------    | ValueError: 654    +---------------- 2 ----------------    | ExceptionGroup: imports (2 sub-exceptions)    +-+---------------- 1 ----------------      | ImportError: no_such_module      +---------------- 2 ----------------      | ModuleNotFoundError: another_module      +------------------------------------    +---------------- 3 ----------------    | TypeError: int    +------------------------------------

The traceback lists all exception that are part of an exception group. Additionally, the nested tree structure of exceptions within the group is indicated, both graphically and by listing how many sub-exceptions there are in each group.

You learned earlier thatExceptionGroup doubles as a regular Python exception. This means that you can catch exception groups with regularexcept blocks:

>>>try:...raiseExceptionGroup("group",[ValueError(654)])...exceptExceptionGroup:...print("Handling ExceptionGroup")...Handling ExceptionGroup

This usually isn’t very helpful, because you’re more interested in the errors that are nested inside the exception group. Note that you’re not able to directly handle those:

>>>try:...raiseExceptionGroup("group",[ValueError(654)])...exceptValueError:...print("Handling ValueError")...  + Exception Group Traceback (most recent call last):  |   ...  | ExceptionGroup: group (1 sub-exception)  +-+---------------- 1 ----------------    | ValueError: 654    +------------------------------------

Even though the exception group contains aValueError, you’re not able to handle it withexcept ValueError. Instead, you should use a newexcept* syntax to handle exception groups. You’ll learn how that works in the next section.

Filter Exceptions Withexcept*

There have been attempts at handling multiple errors in earlier versions of Python. For example, the popularTrio library includes aMultiError exception that can wrap other exceptions. However, because Python is primed toward handling one error at a time, dealing withMultiError exceptions isless than ideal.

The newexcept* syntax in Python 3.11 makes it more convenient to gracefully deal with several errors at the same time. Exception groups have a few attributes and methods that regular exceptions don’t have. In particular, you can access.exceptions to obtain a tuple of all sub-exceptions in the group. You could, for example, rewrite the last example in the previous subsection as follows:

>>>try:...raiseExceptionGroup("group",[ValueError(654)])...exceptExceptionGroupaseg:...forerrineg.exceptions:...ifisinstance(err,ValueError):...print("Handling ValueError")...elifisinstance(err,TypeError):...print("Handling TypeError")...Handling ValueError

Once you catch anExceptionGroup, you loop over all the sub-exceptions and handle them based on their type. While this is possible, it quickly gets cumbersome. Also note that the code above doesn’t handle nested exception groups.

Instead, you should useexcept* to handle exception groups. You can rewrite the example once more:

>>>try:...raiseExceptionGroup("group",[ValueError(654)])...except*ValueError:...print("Handling ValueError")...except*TypeError:...print("Handling TypeError")...Handling ValueError

Eachexcept* clause handles an exception group that’s a subgroup of the original exception group, containing all exceptions matching the given type of error. Consider the slightly more involved example:

>>>try:...raiseExceptionGroup(..."group",[TypeError("str"),ValueError(654),TypeError("int")]...)...except*ValueErroraseg:...print(f"Handling ValueError:{eg.exceptions}")...except*TypeErroraseg:...print(f"Handling TypeError:{eg.exceptions}")...Handling ValueError: (ValueError(654),)Handling TypeError: (TypeError('str'), TypeError('int'))

Note that in this example, bothexcept* clauses trigger. This is different from regularexcept clauses, where at most one clause triggers at a time.

First theValueError is filtered from the original exception group and handled. TheTypeError exceptions remain unhandled until they’re caught byexcept* TypeError. Each clause is only triggered once, even if there are more exceptions of that type. Your handling code must therefore deal with exception groups.

You may end up only partially handling an exception group. For example, you could handle onlyValueError from the previous example:

>>>try:...raiseExceptionGroup(..."group",[TypeError("str"),ValueError(654),TypeError("int")]...)...except*ValueErroraseg:...print(f"Handling ValueError:{eg.exceptions}")...Handling ValueError: (ValueError(654),)  + Exception Group Traceback (most recent call last):  |   ...  | ExceptionGroup: group (2 sub-exceptions)  +-+---------------- 1 ----------------    | TypeError: str    +---------------- 2 ----------------    | TypeError: int    +------------------------------------

In this case, theValueError is handled. But that leaves two unhandled errors in the exception group. Those errors then bubble out and create a traceback. Note that theValueError is not part of the traceback because it’s already been handled. You can see thatexcept* behaves differently fromexcept:

• Severalexcept* clauses may trigger.
• except* clauses that match an error remove that error from the exception group.

This is a clear change from how plainexcept works, and may feel a bit unintuitive at first. However, the changes make it more convenient to deal with multiple concurrent errors.

You can—although you probably don’t need to—split exception groups manually:

>>>eg=ExceptionGroup(..."group",[TypeError("str"),ValueError(654),TypeError("int")]...)>>>egExceptionGroup('group', [TypeError('str'), ValueError(654), TypeError('int')])>>>value_errors,eg=eg.split(ValueError)>>>value_errorsExceptionGroup('group', [ValueError(654)])>>>egExceptionGroup('group', [TypeError('str'), TypeError('int')])>>>import_errors,eg=eg.split(ImportError)>>>print(import_errors)None>>>egExceptionGroup('group', [TypeError('str'), TypeError('int')])>>>type_errors,eg=eg.split(TypeError)>>>type_errorsExceptionGroup('group', [TypeError('str'), TypeError('int')])>>>print(eg)None

You can use.split() on exception groups to split them into two new exception groups. The first group consists of errors that match a given error, while the second group consists of those errors that are left over. If any of the groups end up empty, then they’re replaced byNone. SeePEP 654 and thedocumentation for more information if you want to manually manipulate exception groups.

Exception groups won’t replace regular exceptions! Instead, they’re designed to handle the specific use case where it’s useful to deal with several exceptions at the same time. Libraries should clearly differentiate between functions that can raise regular exceptions and functions that can raise exception groups.

The authors of PEP 654 recommend that changing a function from raising an exception to raising an exception group should be considered a breaking change because anyone using that library needs to update how they handle errors. In the next section, you’ll learn about task groups. They’re new in Python 3.11 and are the first part of the standard library to raise exception groups.

You’ve seen that it’s possible, but cumbersome, to deal with exception groups within regularexcept blocks. It’s also possible to do the opposite.except* can handle regular exceptions:

>>>try:...raiseValueError(654)...except*ValueErroraseg:...print(type(eg),eg.exceptions)...<class 'ExceptionGroup'> (ValueError(654),)

Even though you raise a singleValueError exception, theexcept* mechanism wraps the exception in an exception group before handling it. In theory, this means that you can replace all yourexcept blocks withexcept*. In practice, that would be a bad idea. Exception groups are designed to handle multiple exceptions. Don’t use them unless you need to!

Exception groups are new in Python 3.11. However, if you’re using an older version of Python, then you can use theexceptiongroup backport to access the same functionality. Instead ofexcept*, the backport uses anexceptiongroup.catch()context manager to handle multiple errors. You can learn more about catching multiple exceptions inHow to Catch Multiple Exceptions in Python.

Analyze Files Sequentially

In this subsection, you’ll implement a tool that can count the number of lines in several files. The output will be animated so that you get a nice visual representation of the distribution of file sizes. The final result will look something like this:

You’ll expand this program to explore some features of asynchronous programming. While this tool isn’t necessarily useful on its own, it’s explicit so that you can clearly see what’s happening, and it’s flexible so that you can introduce several exceptions and work toward handling them with exception groups.

Colorama is a library that gives you more control of output in your terminal. You’ll use it to create an animation as your program counts the number of lines in the different files. First, install it withpip:

$python-mpipinstallcolorama

As the name suggests, Colorama’s primary use case is adding color to your terminal. However, you can also use it to print text at specific locations. Write the following code into a file namedcount.py:

# count.pyimportsysimporttimeimportcoloramafromcoloramaimportCursorcolorama.init()defprint_at(row,text):print(Cursor.POS(1,1+row)+str(text))time.sleep(0.03)defcount_lines_in_file(file_num,file_name):counter_text=f"{file_name[:20]:<20} "withopen(file_name,mode="rt",encoding="utf-8")asfile:forline_num,_inenumerate(file,start=1):counter_text+="□"print_at(file_num,counter_text)print_at(file_num,f"{counter_text} ({line_num})")defcount_all_files(file_names):forfile_num,file_nameinenumerate(file_names,start=1):count_lines_in_file(file_num,file_name)if__name__=="__main__":count_all_files(sys.argv[1:])

Theprint_at() function is at the heart of the animation. It uses Colorama’sCursor.POS() to print some text at a particular row or line in your terminal. Next, itsleeps for a short while to create the animation effect.

You usecount_lines_in_file() to analyze and animate one file. The function opens a file and iterates through it, one line at a time. For each line, it adds a box (□) to a string and usesprint_at() to continually print the string on the same row. This creates the animation. At the end, the total number of lines is printed.

# count.py# ...if__name__=="__main__":print(colorama.ansi.clear_screen())count_all_files(sys.argv[1:])print(Cursor.POS(1,1+len(sys.argv)))

Your program’s entry point iscount_all_files(). This loops over all filenames that you provide ascommand-line arguments and callscount_lines_in_file() on them.

Try out your line counter! You run the program by providing files that should be analyzed on the command line. For example, you can count the number of lines in your source code as follows:

$pythoncount.pycount.pycount.py             □□□□□□□□□□□□□□□□□□□□□□□□□□□□ (28)

This counts the number of lines incount.py. You should create a few other files that you’ll use to explore your line counter. Some of these files will expose that you’re not doing any exception handling at the moment. You can create a few new files with the following code:

>>>importpathlib>>>importstring>>>chars=string.ascii_uppercase>>>data=[c1+c2forc1,c2inzip(chars[:13],chars[13:])]>>>pathlib.Path("rot13.txt").write_text("\n".join(data))38>>>pathlib.Path("empty_file.txt").touch()>>>bbj=[98,108,229,98,230,114,115,121,108,116,101,116,248,121]>>>pathlib.Path("not_utf8.txt").write_bytes(bytes(bbj))14

You’ve created three files:rot13.txt,empty_file.txt, andnot_utf8.txt. The first file contains the letters that map to each other in theROT13 cipher. The second file is a completely empty file, while the third file contains some data that’s notUTF-8 encoded. As you’ll see soon, the last two files will create problems for your program.

To count the number of lines in two files, you provide both their names on the command line:

$pythoncount.pycount.pyrot13.txtcount.py             □□□□□□□□□□□□□□□□□□□□□□□□□□□□ (28)rot13.txt            □□□□□□□□□□□□□ (13)

You callcount_all_files() with all the arguments provided at the command line. The function then loops over each file name.

If you provide the name of a file that doesn’t exist, then your program will raise an exception that tells you so:

$pythoncount.pywrong_name.txtTraceback (most recent call last):  ...FileNotFoundError: [Errno 2] No such file or directory: 'wrong_name.txt'

Something similar will happen if you try to analyzeempty_file.txt ornot_utf8.txt:

$pythoncount.pyempty_file.txtTraceback (most recent call last):  ...UnboundLocalError: cannot access local variable 'line_num' where it is                   not associated with a value$pythoncount.pynot_utf8.txtTraceback (most recent call last):  ...UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe5 in position 2:                    invalid continuation byte

Both cases raise errors. Forempty_file.txt, the issue is thatline_num gets defined by iterating over the lines of the file. If there are no lines in the file, thenline_num isn’t defined, and you get an error when you try to access it. The problem withnot_utf8.txt is that you try to UTF-8-decode something that isn’t UTF-8 encoded.

In the next subsections, you’ll use these errors to explore how exception groups can help you improve your error handling. For now, observe what happens if you try to analyze two files that both raise an error:

$pythoncount.pynot_utf8.txtempty_file.txtTraceback (most recent call last):  ...UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe5 in position 2:                    invalid continuation byte

Note that only the first error, corresponding tonot_utf8.txt, is raised. This is natural, because the files are analyzed sequentially. That error happens long beforeempty_file.txt is opened.

Analyze Files Concurrently

In this subsection, you’ll rewrite your program to run asynchronously. This means that the analysis of all the files happens concurrently instead of sequentially. It’s instructive to see your updated program run:

The animation shows that lines are counted in all files at the same time, instead of in one file at the time like before.

You achieve this concurrency by rewriting your functions into asynchronous coroutines using theasync andawait keywords. Note that this new version still uses old async practices, and this code is runnable inPython 3.7 and later. In the next subsection, you’ll take the final step and use the new task groups.

Create a new file namedcount_gather.py with the following code:

# count_gather.pyimportasyncioimportsysimportcoloramafromcoloramaimportCursorcolorama.init()asyncdefprint_at(row,text):print(Cursor.POS(1,1+row)+str(text))awaitasyncio.sleep(0.03)asyncdefcount_lines_in_file(file_num,file_name):counter_text=f"{file_name[:20]:<20} "withopen(file_name,mode="rt",encoding="utf-8")asfile:forline_num,_inenumerate(file,start=1):counter_text+="□"awaitprint_at(file_num,counter_text)awaitprint_at(file_num,f"{counter_text} ({line_num})")asyncdefcount_all_files(file_names):tasks=[asyncio.create_task(count_lines_in_file(file_num,file_name))forfile_num,file_nameinenumerate(file_names,start=1)]awaitasyncio.gather(*tasks)if__name__=="__main__":asyncio.run(count_all_files(sys.argv[1:]))

If you compare this code tocount.py from the previous subsection, then you’ll note that most changes only addasync to function definitions orawait to function calls. Theasync andawaitkeywords constitute Python’s API for doingasynchronous programming.

Next, note thatcount_all_files() has changed significantly. Instead of sequentially callingcount_lines_in_file(), you create one task for each file name. Each task preparescount_lines_in_file() with the relevant arguments. All tasks are collected in a list and passed toasyncio.gather(). Finally,count_all_files() is initiated by callingasyncio.run().

What happens here is thatasyncio.run() creates anevent loop. The tasks are executed by the event loop. In the animation, it looks like all the files are analyzed at the same time. However, while the lines are counted concurrently, they’re not counted in parallel. There’s only one thread in your program, but the thread continously switches which task it’s working on.

Asynchronous programming is sometimes calledcooperative multitasking because each task voluntarily gives up control to let other tasks run. Think ofawait as a marker in your code where you decide that it’s okay to switch tasks. In the example, that’s mainly when the code sleeps before the next animation step.

Run your new code on a few different files and observe how they’re all analyzed in parallel:

$pythoncount_gather.pycount.pyrot13.txtcount_gather.pycount.py             □□□□□□□□□□□□□□□□□□□□□□□□□□□□ (28)rot13.txt            □□□□□□□□□□□□□ (13)count_gather.py      □□□□□□□□□□□□□□□□□□□□□□□□□□□□□□□ (31)

As your files animate in your console, you’ll see thatrot13.txt finishes before the other tasks. Next, try to analyze a few of the troublesome files that you created earlier:

$pythoncount_gather.pynot_utf8.txtempty_file.txtTraceback (most recent call last):  ...UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe5 in position 2:                    invalid continuation byte

Even thoughnot_utf8.txt andempty_file.txt are now analyzed concurrently, you only see the error raised for one of them. As you learned earlier, regular Python exceptions are handled one by one, andasyncio.gather() is limited by this.

Third-party libraries like Trio and Curio do some special error handling that’s able to deal with multiple exceptions. For example, Trio’sMultiError wraps two or more exceptions and provides a context manager that handles them.

More convenient handling of multiple errors is exactly one of the use cases that exception groups were designed to handle. In your counter application, you’d want to see a group containing one exception per file that fails to be analyzed, and have a simple way of handling them. It’s time to give the new Python 3.11TaskGroup a spin!

Control Concurrent Processing With Task Groups

Task groups have been a planned feature forasyncio for a long time. Yuri Selivanov mentions them as a possible enhancement forPython 3.8 inasyncio: What’s Next, a presentation he gave at PyBay 2018. Similar features have been available in other libraries, includingTrio’s nurseries,Curio’s task groups, andQuattro’s task groups.

The main reason the implementation has taken so much time is that task groups require properly dealing with several exceptions at once. The new exception group feature in Python 3.11 has paved the way for including asynchronous task groups as well. They were finallyimplemented byYury Selivanov andGuido van Rossum and made available in Python 3.11.0a6.

In this subsection, you’ll reimplement your counter application to useasyncio.TaskGroup instead ofasyncio.gather(). In the next subsection, you’ll useexcept* to conveniently handle the different exceptions that your application can raise.

Put the following code in a file namedcount_taskgroup.py:

# count_taskgroup.pyimportasyncioimportsysimportcoloramafromcoloramaimportCursorcolorama.init()asyncdefprint_at(row,text):print(Cursor.POS(1,1+row)+str(text))awaitasyncio.sleep(0.03)asyncdefcount_lines_in_file(file_num,file_name):counter_text=f"{file_name[:20]:<20} "withopen(file_name,mode="rt",encoding="utf-8")asfile:forline_num,_inenumerate(file,start=1):counter_text+="□"awaitprint_at(file_num,counter_text)awaitprint_at(file_num,f"{counter_text} ({line_num})")asyncdefcount_all_files(file_names):asyncwithasyncio.TaskGroup()astg:forfile_num,file_nameinenumerate(file_names,start=1):tg.create_task(count_lines_in_file(file_num,file_name))if__name__=="__main__":asyncio.run(count_all_files(sys.argv[1:]))

Compare this tocount_gather.py. You’ll note that the only change is how tasks are created incount_all_files(). Here, you create the task group with a context manager. After that, your code is remarkably similar to the original synchronous implementation incount.py:

defcount_all_files(file_names):forfile_num,file_nameinenumerate(file_names,start=1):count_lines_in_file(file_num,file_name)

Tasks that are created inside aTaskGroup are run concurrently, similar to tasks run byasyncio.gather(). Counting files should work identically to before, as long as you’re using Python 3.11:

$pythoncount_taskgroup.pycount.pyrot13.txtcount_taskgroup.pycount.py             □□□□□□□□□□□□□□□□□□□□□□□□□□□□ (28)rot13.txt            □□□□□□□□□□□□□ (13)count_taskgroup.py   □□□□□□□□□□□□□□□□□□□□□□□□□□□□□ (29)

One great improvement, though, is how errors are handled. Provoke your new code by analyzing some of your troublesome files:

$pythoncount_taskgroup.pynot_utf8.txtempty_file.txt  + Exception Group Traceback (most recent call last):  |   ...  | ExceptionGroup: unhandled errors in a TaskGroup (2 sub-exceptions)  +-+---------------- 1 ----------------    | Traceback (most recent call last):    |   File "count_taskgroup.py", line 18, in count_lines_in_file    |     for line_num, _ in enumerate(file, start=1):    |     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^    | UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe5 in position 2:    |                     invalid continuation byte    +---------------- 2 ----------------    | Traceback (most recent call last):    |   File "count_taskgroup.py", line 21, in count_lines_in_file    |     await print_at(file_num, f"{counter_text} ({line_num})")    |                                                 ^^^^^^^^    | UnboundLocalError: cannot access local variable 'line_num' where it is    |                    not associated with a value    +------------------------------------

Note that you get anException Group Traceback with two sub-exceptions, one for each file that fails to be analyzed. This is already an improvement overasyncio.gather(). In the next subsection, you’ll learn how you can handle these kinds of errors in your code.

Yuri Selivanov points out that the new task groups offer a better API than the oldasyncio.gather(), as task groups are “composable, predictable, and safe.” Additionally, he notes that task groups:

1. Run a set of nested tasks. If one fails, all other tasks that are still running would be canceled.
2. Allow to execute code (incl. awaits) between scheduling nested tasks.
3. Thanks to ExceptionGroups, all errors are propagated and can be handled/reported.

—Yuri Selivanov (Source)

In the next subsection, you’ll experiment with handling and reporting errors in your concurrent code.

Handle Concurrent Errors

You’ve written some concurrent code that sometimes raises errors. How can you handle those exceptions properly? You’ll see examples of error handling soon. First, though, you’ll add one more way that your code can fail.

The problems in your code that you’ve seen so far all raise before the analysis of the file begins. To simulate an error that may happen during the analysis, say that your tool suffers fromtriskaidekaphobia, meaning that it’s irrationally afraid of the number thirteen. Add two lines tocount_lines_in_file():

# count_taskgroup.py# ...asyncdefcount_lines_in_file(file_num,file_name):counter_text=f"{file_name[:20]:<20} "withopen(file_name,mode="rt",encoding="utf-8")asfile:forline_num,_inenumerate(file,start=1):counter_text+="□"awaitprint_at(file_num,counter_text)awaitprint_at(file_num,f"{counter_text} ({line_num})")ifline_num==13:raiseRuntimeError("Files with thirteen lines are too scary!")# ...

If a file has exactly thirteen lines, then aRuntimeError is raised at the end of the analysis. You can see the effect of this by analyzingrot13.txt:

$pythoncount_taskgroup.pyrot13.txtrot13.txt            □□□□□□□□□□□□□ (13)  + Exception Group Traceback (most recent call last):  |   ...  | ExceptionGroup: unhandled errors in a TaskGroup (1 sub-exception)  +-+---------------- 1 ----------------    | Traceback (most recent call last):    |   File "count_taskgroup.py", line 23, in count_lines_in_file    |     raise RuntimeError("Files with thirteen lines are too scary!")    |     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^    | RuntimeError: Files with thirteen lines are too scary!    +------------------------------------

As expected, your new triskaidekaphobic code balks at the thirteen lines inrot13.py. Next combine this with one of the errors you saw earlier:

$pythoncount_taskgroup.pyrot13.txtnot_utf8.txtrot13.txt            □  + Exception Group Traceback (most recent call last):  |   ...  | ExceptionGroup: unhandled errors in a TaskGroup (1 sub-exception)  +-+---------------- 1 ----------------    | Traceback (most recent call last):    |   File "count_taskgroup.py", line 18, in count_lines_in_file    |     for line_num, _ in enumerate(file, start=1):    |     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^    | UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe5 in position 2:    |                     invalid continuation byte    +------------------------------------

This time around, only one error is reported even though you know both files should raise an exception. The reason you get only one error is that the two issues would be raised at different times. One feature of task groups is that they implement acancel scope. Once some tasks fail, other tasks in the same task group are canceled by the event loop.

In general, there are two approaches that you can take to handle errors inside your asynchronous tasks:

1. Use regulartry…except blocks inside your coroutines to handle issues.
2. Use the newtry…except* blocks outside your task groups to handle issues.

In the first case, errors in one task will typically not affect other running tasks. In the second case, however, an error in one task will cancel all other running tasks.

Try this out for yourself! First, addsafe_count_lines_in_file() which uses regular exception handling inside your coroutines:

# count_taskgroup.py# ...asyncdefsafe_count_lines_in_file(file_num,file_name):try:awaitcount_lines_in_file(file_num,file_name)exceptRuntimeErroraserr:awaitprint_at(file_num,err)asyncdefcount_all_files(file_names):asyncwithasyncio.TaskGroup()astg:forfile_num,file_nameinenumerate(file_names,start=1):tg.create_task(safe_count_lines_in_file(file_num,file_name))# ...

You also changecount_all_files() to call the newsafe_count_lines_in_file() instead ofcount_lines_in_file(). In this implementation, you only deal with theRuntimeError raised whenever a file has thirteen lines.

Analyzerot13.txt and some other files to confirm that the error no longer cancels the other tasks:

$pythoncount_taskgroup.pycount.pyrot13.txtcount_taskgroup.pycount.py             □□□□□□□□□□□□□□□□□□□□□□□□□□□□ (28)Files with thirteen lines are too scary!count_taskgroup.py   □□□□□□□□□□□□□□□□□□□□□□□□□□□□□□□□□□□□□ (37)

Errors that are handled don’t bubble up and affect other tasks. In this example,count.py andcount_taskgroup.py were properly analyzed even though the analysis ofrot13.txt failed.

Next, try to useexcept* to handle errors after the fact. You can, for example, wrap your event loop in atry…except* block:

# count_taskgroup.py# ...if__name__=="__main__":try:asyncio.run(count_all_files(sys.argv[1:]))except*UnicodeDecodeErroraseg:print("Bad encoding:",*[str(e)[:50]foreineg.exceptions])

Recall thatexcept* works with exception groups. In this case, you loop through theUnicodeDecodeError exceptions in the group and print their first fifty characters to the console to log them.

Analyzenot_utf8.txt together with some other files to see the effect:

$pythoncount_taskgroup.pyrot13.txtnot_utf8.txtcount.pyrot13.txt            □count.py             □Bad encoding: 'utf-8' codec can't decode byte 0xe5 in position 2

In contrast to the previous example, the other tasks are canceled even though you handle theUnicodeDecodeError. Note that only one line is counted in bothrot13.txt andcount.py.

$pythoncount_taskgroup.pynot_utf8.txtcount_taskgroup.pyempty.txtcount_taskgroup.py   □Bad text: ["'utf-8' codec can't decode byte 0xe5 in position 2"]Empty file: ["cannot access local variable 'line_num' where it i"]

If you don’t handle all exceptions that are raised, then the unhandled exceptions will still cause your program to crash with a traceback. To see this, switchcount.py toempty_file.txt in your analysis:

$pythoncount_taskgroup.pyrot13.txtnot_utf8.txtempty_file.txtrot13.txt            □Bad encoding: 'utf-8' codec can't decode byte 0xe5 in position 2  + Exception Group Traceback (most recent call last):  |   ...  | ExceptionGroup: unhandled errors in a TaskGroup (1 sub-exception)  +-+---------------- 1 ----------------    | Traceback (most recent call last):    |   File "count_taskgroup.py", line 21, in count_lines_in_file    |     await print_at(file_num, f"{counter_text} ({line_num})")    |                                                 ^^^^^^^^    | UnboundLocalError: cannot access local variable 'line_num' where it is    |                    not associated with a value    +------------------------------------

You get the familiarUnboundLocalError. Note that part of the error message points out that there’s one unhandled sub-exception. There’s no record in the traceback of theUnicodeDecodeError sub-exception that you did handle.

You’ve now seen an example of using task groups in order to improve the error handling of your asynchronous application, and in particular being able to comfortably handle several errors happening at the same time. The combination of exception groups and task groups makes Python a very capable language for doing asynchronous programming.

Annotate Exceptions With Custom Notes

You can now add custom notes to an exception. This is yet another improvement to how exceptions are handled in Python. Exception notes were suggested byZac Hatfield-Dodds inPEP 678: Enriching Exceptions with Notes. The PEP has beenaccepted, and an early version of the proposal was implemented for Python 3.11.0a3 to Python 3.11.0a7.

In those alpha versions, you can assign strings to a.__note__ attribute on an exception, and that information will be made available if the error isn’t handled. Here’s a basic example:

>>>try:...raiseValueError(678)...exceptValueErroraserr:...err.__note__="Enriching Exceptions with Notes"...raise...Traceback (most recent call last):...ValueError:678Enriching Exceptions with Notes

You’re adding a note to theValueError before reraising it. Your note is then displayed together with the regular error message at the end of your traceback.

Duringdiscussions of the PEP.__note__ was changed to.__notes__ which can contain several notes. A list of notes can be useful in certain use cases where keeping track of individual notes is important. One example of this is internationalization andtranslation of notes.

There is a also new dedicated method,.add_note(), that can be used to add these notes. Thefull implementation of PEP 678 is available in thefirst beta version of Python 3.11 and later.

Going forward, you should write the previous example as follows:

>>>try:...raiseValueError(678)...exceptValueErroraserr:...err.add_note("Enriching Exceptions with Notes")...raise...Traceback (most recent call last):...ValueError:678Enriching Exceptions with Notes

You can add several notes with repeated calls to.add_note() and recover them by looping over.__notes__. All notes will be printed below the traceback when the exception is raised:

>>>err=ValueError(678)>>>err.add_note("Enriching Exceptions with Notes")>>>err.add_note("Python 3.11")>>>err.__notes__['Enriching Exceptions with Notes', 'Python 3.11']>>>fornoteinerr.__notes__:...print(note)...Enriching Exceptions with NotesPython 3.11>>>raiseerrTraceback (most recent call last):...ValueError:678Enriching Exceptions with NotesPython 3.11

The new exception notes are also compatible with the exception groups.

Reference the Active Exception Withsys.exception()

Internally, Python has represented an exception as a tuple with information about the type of the exception, the exception itself, and the traceback of the exception. Thischanges in Python 3.11. Now, Python will internally store only the exception itself. Both the type and the traceback can be derived from the exception object.

In general, you won’t need to think about this change, as it’s all under the hood. However, if you need to access an active exception, you can now use the newexception() function in thesys module:

>>>importsys>>>try:...raiseValueError("bpo-46328")...exceptValueError:...print(f"Handling{sys.exception()}")...Handling bpo-46328

Note that you usually won’t useexception() in normal error handling like above. Instead, it’s sometimes handy to use in wrapper libraries that are used in error handling but don’t have direct access to active exceptions. In normal error handling, you should name your errors in theexcept clause:

>>>try:...raiseValueError("bpo-46328")...exceptValueErroraserr:...print(f"Handling{err}")...Handling bpo-46328

In versions prior to Python 3.11, you can get the same information fromsys.exc_info():

>>>try:...raiseValueError("bpo-46328")...exceptValueError:...sys.exception()issys.exc_info()[1]...True

Indeed,sys.exception() is identical tosys.exc_info()[1]. The new function was added inbpo-46328 by Irit Katriel, although the idea was originally floated inPEP 3134, all the way back in 2005.

Reference the Active Traceback Consistently

As noted in the previous subsection, older versions of Python represent exceptions as tuples. You can access traceback information in two different ways:

>>>importsys>>>try:...raiseValueError("bpo-45711")...exceptValueError:...exc_type,exc_value,exc_tb=sys.exc_info()...exc_value.__traceback__isexc_tb...True

Note that accessing the traceback throughexc_value andexc_tb returns the exact same object. In general, this is what you want. However, it turns out that there has been a subtle bug hiding around for some time. You can update the traceback onexc_value without updatingexc_tb.

To demonstrate this, code up the following program, which changes the traceback during handling of an exception:

 1# traceback_demo.py 2 3importsys 4importtraceback 5 6deftb_last(tb): 7frame,*_=traceback.extract_tb(tb,limit=1) 8returnf"{frame.name}:{frame.lineno}" 910defbad_calculation():11return1/01213defmain():14try:15bad_calculation()16exceptZeroDivisionErroraserr:17err_tb=err.__traceback__18err=err.with_traceback(err_tb.tb_next)1920exc_type,exc_value,exc_tb=sys.exc_info()21print(f"{tb_last(exc_value.__traceback__)= }")22print(f"{tb_last(exc_tb)= }")2324if__name__=="__main__":25main()

You change the traceback of the active exception on line 18. As you’ll soon see, this wouldn’t update the traceback part of the exception tuple in Python 3.10 and earlier. To show this, lines 20 to 22 compare the last frame of the tracebacks referenced by the active exception and the traceback object.

Run this with Python 3.10 or an earlier version:

$pythontraceback_demo.pytb_last(exc_value.__traceback__) = 'bad_calculation:11'tb_last(exc_tb)                  = 'main:15'

The important thing to note here is that the two line references are different. The active exception points to the updated location, line 11 insidebad_calculation(), while the traceback points to the old location insidemain().

In Python 3.11, the traceback part of the exception tuple is always read from the exception itself. Therefore, the inconsistency is gone:

$python3.11traceback_demo.pytb_last(exc_value.__traceback__) = 'bad_calculation:11'tb_last(exc_tb)                  = 'bad_calculation:11'

Now, both ways of accessing the traceback give the same result. This fixes a bug that has been present in Python for some time. Still, it’s important to note that the inconsistency was mostly academic. Yes, the old way was wrong, but it’s unlikely that it caused issues in actual code.

This bug fix is interesting because it lifts the curtain on something bigger. As you learned in the previous subsection, Python’s internal representation of exceptions changes in version 3.11. This bug fix is an immediate consequence of that change.

Restructuring Python’s exceptions is part of an evenbigger effort to optimize many different parts of Python.Mark Shannon has initiated thefaster-cpython project. Streamlining exceptions is only one of the ideas coming out of that initiative.

The smaller improvements that you’ve learned about in this section examplify all the work that goes into maintaining and developing a programming language, beyond the fewitems stealing most of the headlines. The features that you’ve learned about here are all related to Python’s exception handling. However, there are many other small changes happening as well.What’s New In Python 3.11 keeps track of all of them.