ExceptionGroupsAsPEP 654ExceptionGroup has come into widespread use across thePython community, some common but awkward patterns have emerged. We thereforepropose adding two new methods to exception objects:
BaseExceptionGroup.leaf_exceptions(), returning the ‘leaf’ exceptions asa list, with each traceback composited from any intermediate groups.BaseException.preserve_context(), a context manager whichsaves and restores theself.__context__ attribute ofself,so that re-raising the exception within another handler does not overwritethe existing context.We expect this to enable more concise expression of error handling logic inmany medium-complexity cases. Without them, exception-group handlers willcontinue to discard intermediate tracebacks and mis-handle__context__exceptions, to the detriment of anyone debugging async code.
As exception groups come into widespread use, library authors and end usersoften write code to process or respond to individual leaf exceptions, forexample when implementing middleware, error logging, or response handlers ina web framework.
Searching GitHub found four implementations ofleaf_exceptions() byvarious names in the first sixty hits, of which none handletracebacks.[1] The same search found thirteen cases where.leaf_exceptions() could be used. We therefore believe that providinga method on theBaseException type with proper traceback preservationwill improve error-handling and debugging experiences across the ecosystem.
The rise of exception groups has also made re-raising exceptions caught by anearlier handler much more common: for example, web-server middleware mightunwrapHTTPException if that is the sole leaf of a group:
except*HTTPExceptionasgroup:first,*rest=group.leaf_exceptions()# get the whole traceback :-)ifnotrest:raisefirstraise
However, this innocent-seeming code has a problem:raisefirst will dofirst.__context__=group as a side effect. This discards the originalcontext of the error, which may contain crucial information to understand whythe exception was raised. In many production apps it also causes tracebacksto balloon from hundreds of lines, to tens or evenhundreds of thousands oflines - a volume which makes understanding errors far more difficult thanit should be.
A newBaseException.preserve_context() method would be a discoverable,readable, and easy-to-use solution for these cases.
A new methodleaf_exceptions() will be added toBaseExceptionGroup, with thefollowing signature:
defleaf_exceptions(self,*,fix_tracebacks=True)->list[BaseException]:""" Return a flat list of all 'leaf' exceptions in the group. If fix_tracebacks is True, each leaf will have the traceback replaced with a composite so that frames attached to intermediate groups are still visible when debugging. Pass fix_tracebacks=False to disable this modification, e.g. if you expect to raise the group unchanged. """
A new methodpreserve_context() will be added toBaseException, with thefollowing signature:
defpreserve_context(self)->contextlib.AbstractContextManager[Self]:""" Context manager that preserves the exception's __context__ attribute. When entering the context, the current values of __context__ is saved. When exiting, the saved value is restored, which allows raising an exception inside an except block without changing its context chain. """
Usage example:
# We're an async web framework, where user code can raise an HTTPException# to return a particular HTTP error code to the client. However, it may# (or may not) be raised inside a TaskGroup, so we need to use `except*`;# and if there are *multiple* such exceptions we'll treat that as a bug.try:user_code_here()except*HTTPExceptionasgroup:first,*rest=group.leaf_exceptions()ifrest:raise# handled by internal-server-error middleware...# logging, cache updates, etc.withfirst.preserve_context():raisefirst
Without.preserve_context(), this code would have to either:
except* block,making code difficult to follow in nontrivial cases, or__context__ of thefirst exception, replacingit with anExceptionGroup which is simply an implementation detail, ortry/except instead ofexcept*, handling the possibility that thegroup doesn’t contain anHTTPException at all,[2] or.preserve_context() inline; while this is notliterally unheard-of, it remains very rare.Adding new methods to built-in classes, especially those as widely used asBaseException, can have substantial impacts. However, GitHub search showsno collisions for these method names (zero hits[3] andthree unrelated hits respectively). If user-defined methods with thesenames exist in private code they will shadow those proposed in the PEP,without changing runtime behavior.
Working with exception groups is an intermediate-to-advanced topic, unlikelyto arise for beginning programmers. We therefore suggest teaching this topicvia documentation, and via just-in-time feedback from static analysis tools.In intermediate classes, we recommend teaching.leaf_exceptions() togetherwith the.split() and.subgroup() methods, and mentioning.preserve_context() as an advanced option to address specific pain points.
Both the API reference and the existingExceptionGroup tutorialshould be updated to demonstrate and explain the new methods. The tutorialshould include examples of common patterns where.leaf_exceptions() and.preserve_context() help simplify error handling logic. Downstreamlibraries which often use exception groups could include similar docs.
We have also designed lint rules for inclusion inflake8-async which willsuggest using.leaf_exceptions() when iterating overgroup.exceptionsor re-raising a leaf exception, and suggest using.preserve_context() whenre-raising a leaf exception inside anexcept* block would override anyexisting context.
While the methods on built-in exceptions will be implemented in C if this PEPis accepted, we hope that the following Python implementation will be usefulon older versions of Python, and can demonstrate the intended semantics.
We have found these helper functions quite useful when working withExceptionGroups in a large production codebase.
leaf_exceptions() helper functionimportcopyimporttypesfromtypesimportTracebackTypedefleaf_exceptions(self:BaseExceptionGroup,*,fix_traceback:bool=True)->list[BaseException]:""" Return a flat list of all 'leaf' exceptions. If fix_tracebacks is True, each leaf will have the traceback replaced with a composite so that frames attached to intermediate groups are still visible when debugging. Pass fix_tracebacks=False to disable this modification, e.g. if you expect to raise the group unchanged. """def_flatten(group:BaseExceptionGroup,parent_tb:TracebackType|None=None):group_tb=group.__traceback__combined_tb=_combine_tracebacks(parent_tb,group_tb)result=[]forexcingroup.exceptions:ifisinstance(exc,BaseExceptionGroup):result.extend(_flatten(exc,combined_tb))eliffix_tracebacks:tb=_combine_tracebacks(combined_tb,exc.__traceback__)result.append(exc.with_traceback(tb))else:result.append(exc)returnresultreturn_flatten(self)def_combine_tracebacks(tb1:TracebackType|None,tb2:TracebackType|None,)->TracebackType|None:""" Combine two tracebacks, putting tb1 frames before tb2 frames. If either is None, return the other. """iftb1isNone:returntb2iftb2isNone:returntb1# Convert tb1 to a list of framesframes=[]current=tb1whilecurrentisnotNone:frames.append((current.tb_frame,current.tb_lasti,current.tb_lineno))current=current.tb_next# Create a new traceback starting with tb2new_tb=tb2# Add frames from tb1 to the beginning (in reverse order)forframe,lasti,linenoinreversed(frames):new_tb=types.TracebackType(tb_next=new_tb,tb_frame=frame,tb_lasti=lasti,tb_lineno=lineno)returnnew_tb
preserve_context() context managerclasspreserve_context:def__init__(self,exc:BaseException):self.__exc=excself.__context=exc.__context__def__enter__(self):returnself.__excdef__exit__(self,exc_type,exc_value,traceback):assertexc_valueisself.__exc,f"did not raise the expected exception{self.__exc!r}"exc_value.__context__=self.__contextdelself.__context# break gc cycle
Rather than adding methods to exceptions, we could provide utility functionslike the reference implementations above.There are however several reasons to prefer methods: there’s no obvious placewhere helper functions should live, they take exactly one argument which mustbe an instance ofBaseException, and methods are both more convenient andmore discoverable.
BaseException.as_group() (or group methods)Our survey ofExceptionGroup-related error handling code also observedmany cases of duplicated logic to handle both a bare exception, and the samekind of exception inside a group (often incorrectly, motivating.leaf_exceptions()).
We brieflyproposedadding.split(...) and.subgroup(...) methods too all exceptions,before considering.leaf_exceptions() made us feel this was too clumsy.As a cleaner alternative, we sketched out an.as_group() method:
defas_group(self):ifnotisinstance(self,BaseExceptionGroup):returnBaseExceptionGroup("",[self])returnself
However, applying this method to refactor existing code was a negligibleimprovement over writing the trivial inline version. We also hope that manycurrent uses for such a method will be addressed byexcept* as olderPython versions reach end-of-life.
We recommend documenting a “convert to group” recipe for de-duplicated errorhandling, instead of adding group-related methods toBaseException.
e.raise_with_preserved_context() instead of a context managerWe prefer the context-manager form because it allowsraise...from...if the user wishes to (re)set the__cause__, and is overall somewhatless magical and tempting to use in cases where it would not be appropriate.We could be argued around though, if others prefer this form.
We decided against preserving the__cause__ and__suppress_context__attributes, because they are not changed by re-raising the exception, and weprefer to supportraiseexcfromNone orraiseexcfromcause_exctogether withwithexc.preserve_context():.
Similarly, we considered preserving the__traceback__ attribute, anddecided against because the additionalraise... statement may be animportant clue when understanding some error. If end users wish to pop aframe from the traceback, they can do with a separate context manager.
for\w+in[eg]\w*\.exceptions:, we find:leaf_exceptions() semantics, none ofwhich preserve tracebacks:(one,two,three,four)leaf_exceptions(). We were surprised to note that onlyone of these cases could straightforwardly be replaced by use of anexcept* clause or.subgroup() method.(one,two,three,four,five,six,seven)indicating that more than a quarter ofall hits for this fairly generalsearch would benefit from the methods proposed in this PEP.
exceptFooError: andexceptExceptionGroup: #containingFooErrorclauses rather than using something like theas_group() trick.We expect thatexcept* will be widely used in such cases by the timethat the methods proposed by this PEP are widely available.leaf_exceptions() wasfirst proposed in an earlyprecursor toPEP 654. If the prototype had matchedexcept*in wrapping bare exceptions in a group, we might even have includeda.leaf_exceptions() method in that earlier PEP!This document is placed in the public domain or under the CC0-1.0-Universal license,whichever is more permissive.
Source:https://github.com/python/peps/blob/main/peps/pep-0785.rst
Last modified:2025-04-17 17:42:59 GMT