Syntax

The grammar should be changed to allow the expression in list/setcomprehensions and generator expressions to be preceded by a*, andallowing an alternative form of dictionary comprehension in which adouble-starred expression can be used in place of akey:value pair.

This can be accomplished by updating thelistcomp andsetcomp rules tousestar_named_expression instead ofnamed_expression:

listcomp[expr_ty]:    | '[' a=star_named_expression b=for_if_clauses ']'setcomp[expr_ty]:    | '{' a=star_named_expression b=for_if_clauses '}'

The rule forgenexp would similarly need to be modified to allow astarred_expression:

genexp[expr_ty]:    | '(' a=(assignment_expression | expression !':=' | starred_expression) b=for_if_clauses ')'

The rule for dictionary comprehensions would need to be adjusted as well, toallow for this new form:

dictcomp[expr_ty]:    | '{' a=double_starred_kvpair b=for_if_clauses '}'

No change should be made to the way that argument unpacking is handled infunction calls, i.e., the general rule that generator expressions provided asthe sole argument to functions do not require additional redundant parenthesesshould be retained. Note that this implies that, for example,f(*xforxinit) is equivalent tof((*xforxinit)) (seeStarred Generators as Function Argumentsfor more discussion).

* and** should only be allowed at the top-most level of the expressionin the comprehension (seeFurther Generalizing Unpacking Operators for more discussion).

Semantics: List/Set/Dict Comprehensions

The meaning of a starred expression in a list comprehension[*exprforxinit] is to treat each expression as an iterable, and concatenate them, in thesame way as if they were explicitly listed via[*expr1,*expr2,...].Similarly,{*exprforxinit} forms a set union, as if the expressionswere explicitly listed via{*expr1,*expr2,...}; and{**exprforxinit} combines dictionaries, as if the expressions were explicitly listed via{**expr1,**expr2,...}. These operations should retain all of theequivalent semantics for combining collections in this way (including, forexample, later values replacing earlier ones in the case of a duplicated keywhen combining dictionaries).

Said another way, the objects created by the following comprehensions:

new_list=[*exprforxinits]new_set={*exprforxinits}new_dict={**exprfordindicts}

should be equivalent to the objects created by the following pieces of code,respectively:

new_list=[]forxinits:new_list.extend(expr)new_set=set()forxinits:new_set.update(expr)new_dict={}forxindicts:new_dict.update(expr)

Semantics: Generator Expressions

Generator expressions using the unpacking syntax should form new generatorsproducing values from the concatenation of the iterables given by theexpressions. Specifically, the behavior is defined to be equivalent to thefollowing (though without defining or referencing the looping variablei):

# equivalent to g = (*expr for x in it)defgenerator():forxinit:foriinexpr:yieldig=generator()

# equivalent to g = (*expr async for x in ait())asyncdefgenerator():asyncforxinait():foriinexpr:yieldig=generator()

SeeAlternative Generator Expression Semantics for more discussion of these semantics.

defgenerator():foriin(0,2,4):forjin(y:=[i,i+1]):yieldj

>>>g=(*(y:=[i,i+1])foriin(0,2,4))>>>yTraceback (most recent call last):  File"<python-input-1>", line1, in<module>yNameError:name 'y' is not defined>>>next(g)0>>>y[0, 1]>>>next(g)1>>>y[0, 1]>>>next(g)2>>>y[2, 3]

Error Reporting

Currently, the proposed syntax generates aSyntaxError. Allowing theseforms to be recognized as syntactically valid requires adjusting the grammarrules forinvalid_comprehension andinvalid_dict_comprehension to allowthe use of* and**, respectively.

Additional specific error messages should be provided in at least the followingcases:

• Attempting to use** in a list comprehension or generator expressionshould report that dictionary unpacking cannot be used in those structures,for example:

  >>>[**xforxiny]  File"<stdin>", line1[**xforxiny]^^^SyntaxError:cannot use dict unpacking in list comprehension>>>(**xforxiny)  File"<stdin>", line1(**xforxiny)^^^SyntaxError:cannot use dict unpacking in generator expression

• The existing error message for attempting to use* in a dictionarykey/value should be retained, but similar messages should be reportedwhen attempting to use** unpacking on a dictionary key or value, forexample:

  >>>{*k:vfork,vinitems}  File"<stdin>", line1{*k:vfork,vinitems}^^SyntaxError:cannot use a starred expression in a dictionary key>>>{k:*vfork,vinitems}  File"<stdin>", line1{k:*vfork,vinitems}^^SyntaxError:cannot use a starred expression in a dictionary value>>>{**k:vfork,vinitems}  File"<stdin>", line1{**k:vfork,vinitems}^^^SyntaxError:cannot use dict unpacking in a dictionary key>>>{k:**vfork,vinitems}  File"<stdin>", line1{k:**vfork,vinitems}^^^SyntaxError:cannot use dict unpacking in a dictionary value

• The phrasing of some other existing error messages should similarly beadjusted to account for the presence of the new syntax, and/or to clarifyambiguous or confusing cases relating to unpacking more generally(particularly the cases mentioned inFurther Generalizing Unpacking Operators), forexample:

  >>>[*xifxelsey]  File"<stdin>", line1[*xifxelsey]^^^^^^^^^^^^^^SyntaxError:invalid starred expression. Did you forget to wrap the conditional expression in parentheses? >>> {**x if x else y}  File"<stdin>", line1{**xifxelsey}^^^^^^^^^^^^^^^SyntaxError:invalid double starred expression. Did you forget to wrap the conditional expression in parentheses?>>>[xifxelse*y]  File"<stdin>", line1[xifxelse*y]^SyntaxError:cannot unpack only part of a conditional expression>>>{xifxelse**y}  File"<stdin>", line1{xifxelse**y}^^SyntaxError:cannot use dict unpacking on only part of a conditional expression

Replacing Explicit Loops

Replacing explicit loops compresses multiple lines into one, and avoids theneed for defining and referencing an auxiliary variable.

• Fromemail/_header_value_parser.py:

  # current:comments=[]fortokeninself:comments.extend(token.comments)returncomments# proposed:return[*token.commentsfortokeninself]

• Fromshutil.py:

  # current:ignored_names=[]forpatterninpatterns:ignored_names.extend(fnmatch.filter(names,pattern))returnset(ignored_names)# proposed:return{*fnmatch.filter(names,pattern)forpatterninpatterns}

• Fromhttp/cookiejar.py:

  # current:cookies=[]fordomaininself._cookies.keys():cookies.extend(self._cookies_for_domain(domain,request))returncookies# proposed:return[*self._cookies_for_domain(domain,request)fordomaininself._cookies.keys()]

Replacing from_iterable and Friends

While not always the right choice, replacingitertools.chain.from_iterableandmap can avoid an extra level of redirection, resulting in code thatfollows conventional wisdom that comprehensions are generally more readablethan map/filter.

• Fromdataclasses.py:

  # current:inherited_slots=set(itertools.chain.from_iterable(map(_get_slots,cls.__mro__[1:-1])))# proposed:inherited_slots={*_get_slots(c)forcincls.__mro__[1:-1]}

• Fromimportlib/metadata/__init__.py:

  # current:returnitertools.chain.from_iterable(path.search(prepared)forpathinmap(FastPath,paths))# proposed:return(*FastPath(path).search(prepared)forpathinpaths)

• Fromcollections/__init__.py (Counter class):

  # current:return_chain.from_iterable(_starmap(_repeat,self.items()))# proposed:return(*_repeat(elt,num)forelt,numinself.items())

• Fromzipfile/_path/__init__.py:

  # current:parents=itertools.chain.from_iterable(map(_parents,names))# proposed:parents=(*_parents(name)fornameinnames)

• From_pyrepl/_module_completer.py:

  # current:search_locations=set(chain.from_iterable(getattr(spec,'submodule_search_locations',[])forspecinspecsifspec))# proposed:search_locations={*getattr(spec,'submodule_search_locations',[])forspecinspecsifspec}

Replacing Double Loops in Comprehensions

Replacing double loops in comprehensions avoids the need for defining andreferencing an auxiliary variable.

• Fromimportlib/resources/readers.py:

  # current:children=(childforpathinself._pathsforchildinpath.iterdir())# proposed:children=(*path.iterdir()forpathinself._paths)

• Fromasyncio/base_events.py:

  # current:exceptions=[excforsubinexceptionsforexcinsub]# proposed:exceptions=[*subforsubinexceptions]

• From_weakrefset.py:

  # current:returnself.__class__(eforsin(self,other)foreins)# proposed:returnself.__class__(*sforsin(self,other))

Starred Generators as Function Arguments

One common concern that has arisen multiple times (not only in the discussionthreads linked above but also in previous discussions around this same idea) isa possible syntactical ambiguity when passing a starred generator as the soleargument tof(*xforxiny). In the originalPEP 448, this ambiguitywas cited as a reason for not including a similar generalization as part of theproposal.

This proposal suggests thatf(*xforxiny) should be interpreted asf((*xforxiny)) and should not attempt further unpacking of theresulting generator, but several alternatives were suggested in our discussion(and/or have been suggested in the past), including:

• interpretingf(*xforxiny) asf(*(xforxiny),
• interpretingf(*xforxiny) asf(*(*xforxiny)), or
• continuing to raise aSyntaxError forf(*xforxiny) even if theother aspects of this proposal are accepted.

The reason to prefer this proposal over these alternatives is the preservationof existing conventions for punctuation around generator expressions.Currently, the general rule is that generator expressions must be wrapped inparentheses except when provided as the sole argument to a function, and thisproposal suggests maintaining that rule even as we allow more kinds ofgenerator expressions. This option maintains a full symmetry betweencomprehensions and generator expressions that use unpacking and those thatdon’t.

Currently, we have the following conventions:

f([xforxiny])# pass in a single listf({xforxiny})# pass in a single setf(xforxiny)# pass in a single generator (no additional parentheses required around genexp)f(*[xforxiny])# pass in elements from the list separatelyf(*{xforxiny})# pass in elements from the set separatelyf(*(xforxiny))# pass in elements from the generator separately (parentheses required)

This proposal opts to maintain those conventions even when the comprehensionsmake use of unpacking:

f([*xforxiny])# pass in a single listf({*xforxiny})# pass in a single setf(*xforxiny)# pass in a single generator (no additional parentheses required around genexp)f(*[*xforxiny])# pass in elements from the list separatelyf(*{*xforxiny})# pass in elements from the set separatelyf(*(*xforxiny))# pass in elements from the generator separately (parentheses required)

Further Generalizing Unpacking Operators

Another suggestion that came out of the discussion involved furthergeneralizing the* beyond simply allowing it to be used to unpack theexpression in a comprehension. Two main flavors of this extension wereconsidered:

• making* and** true unary operators that create a new kind ofUnpackable object (or similar), which comprehensions could treat byunpacking it but which could also be used in other contexts; or
• continuing to allow* and** only in the places they are allowedelsewhere in this proposal (expression lists, comprehensions, generatorexpressions, and argument lists), but also allow them to be used insubexpressions within a comprehension, allowing, for example, the followingas a way to flatten a list that contains some iterables but some non-iterableobjects:

  [*xifisinstance(x,Iterable)elsexforxin[[1,2,3],4]]

These variants were considered substantially more complex (both to understandand to implement) and of only marginal utility, so neither is included in thisPEP. As such, these forms should continue to raise aSyntaxError, but witha new error message as described above, though it should not be ruled out as aconsideration for future proposals.

Alternative Generator Expression Semantics

Another point of discussion centered around the semantics of unpacking ingenerator expressions, particularly the relationship between the semantics ofsynchronous and asynchronous generator expressions given that async generatorsdo not supportyieldfrom (see the section ofPEP 525 on Asynchronousyieldfrom).

The core question centered around whether sync and async generator expressionsshould useyieldfrom (or an equivalent) when unpacking, as opposed to anexplicit loop. The main difference between these options is whether theresulting generator delegates to the objects being unpacked, which would affectthe behavior of these generator expressions when used with.send()/.asend(),.throw()/.athrow(), and.close()/.aclose() in thecase where the objects being unpacked are themselves generators. Thedifferences between these options are summarized inAppendix: Semantics of Generator Delegation.

Several reasonable options were considered, none of which was a clear winner inapoll in the Discourse thread.Beyond the proposal outlined above, the following were also considered:

1. Usingyieldfrom for unpacking in synchronous generator expressions butusing an explicit loop in asynchronous generator expressions (as proposed inthe original draft of this PEP).

   This strategy would have allowed unpacking in generator expressions toclosely mimic a popular way of writing generators that perform thisoperation (usingyieldfrom), but it would also have created anasymmetry between synchronous and asynchronous versions, and also betweenthis new syntax anditertools.chain and the double-loop version.

2. Usingyieldfrom for unpacking in synchronous generator expressions andmimicking the behavior ofyieldfrom for unpacking in async generatorexpressions.

   This strategy would also make unpacking in synchronous and asynchronousgenerators behave symmetrically, but it would also be more complex, enoughso that the cost may not be worth the benefit, particularly in the absenceof a compelling use case for delegation.

3. Usingyieldfrom for unpacking in synchronous generator expressions, anddisallowing unpacking in asynchronous generator expressions until theysupportyieldfrom.

   This strategy could possibly reduce friction if asynchronous generatorexpressions do gain support foryieldfrom in the future by making surethat any decision made at that point would be fully backwards-compatible,but in the meantime, it would result in an even bigger discrepancy betweensynchronous and asynchronous generator expressions than option 1.

4. Disallowing unpacking in all generator expressions.

   This would retain symmetry between the two cases, but with the downside oflosing an expressive form and reducing symmetry between list/setcomprehensions and generator expressions.

Each of these options (including the one presented in this PEP) has itsbenefits and drawbacks, with no option being clearly superior on all fronts.The semantics proposed inSemantics: Generator Expressions above represent areasonable compromise by allowing exactly the same kind of unpacking insynchronous and asynchronous generator expressions and retaining an existingproperty of generator expressions (that they do not delegate to subgenerators).

This decision should be revisited in the event that asynchronous generatorsreceive support foryieldfrom in the future, in which case adjusting thesemantics of unpacking in generator expressions to useyieldfrom should beconsidered.

Basic Behavior

For simple iteration over values, which we expect to be by far the most-commonuse of unpacking in generator expressions, both approaches produce identicalresults:

defyield_from(iterables):foriterableiniterables:yield fromiterabledefexplicit_loop(iterables):foriterableiniterables:foriteminiterable:yielditem# Both produce the same sequence of valuesx=list(yield_from([[1,2],[3,4]]))y=list(explicit_loop([[1,2],[3,4]]))print(x==y)# prints True

Advanced Generator Protocol Differences

The differences become apparent when using the advanced generator protocolmethods.send(),.throw(), and.close(), and when the sub-iterablesare themselves generators rather than simple sequences. In these cases, theyieldfrom version results in the associated signal reaching thesubgenerator, but the version with the explicit loop does not.

defsub_generator():x=yield"first"yieldf"received:{x}"yield"last"defyield_from():yield fromsub_generator()defexplicit_loop():foriteminsub_generator():yielditem# With yield from, values are passed through to sub-generatorgen1=yield_from()print(next(gen1))# prints "first"print(gen1.send("hello"))# prints "received: hello"print(next(gen1))# prints "last"# With explicit loop, .send() affects the outer generator; values don't reach the sub-generatorgen2=explicit_loop()print(next(gen2))# prints "first"print(gen2.send("hello"))# prints "received: None" (sub-generator receives None instead of "hello")print(next(gen2))# prints "last"

defsub_generator_with_exception_handling():try:yield"first"yield"second"exceptValueErrorase:yieldf"caught:{e}"defyield_from():yield fromsub_generator_with_exception_handling()defexplicit_loop():foriteminsub_generator_with_exception_handling():yielditem# With yield from, exceptions are passed to sub-generatorgen1=yield_from()print(next(gen1))# prints "first"print(gen1.throw(ValueError("test")))# prints "caught: test"# With explicit loop, exceptions affect the outer generator onlygen2=explicit_loop()print(next(gen2))# prints "first"print(gen2.throw(ValueError("test")))# ValueError is raised; sub-generator doesn't see it

# hold references to sub-generators so GC doesn't close the explicit loop versionreferences=[]defsub_generator_with_cleanup():try:yield"first"yield"second"finally:print("sub-generator received GeneratorExit")defyield_from():try:g=sub_generator_with_cleanup()references.append(g)yield fromgfinally:print("outer generator received GeneratorExit")defexplicit_loop():try:g=sub_generator_with_cleanup()references.append(g)foriteming:yielditemfinally:print("outer generator received GeneratorExit")# With yield from, GeneratorExit is passed through to sub-generatorgen1=yield_from()print(next(gen1))# prints "first"gen1.close()# closes sub-generator and then outer generator# With explicit loop, GeneratorExit goes to outer generator onlygen2=explicit_loop()print(next(gen2))# prints "first"gen2.close()# only closes outer generatorprint('program finished; GC will close the explicit loop subgenerator')# second inner generator closes when GC closes it at the end