Uh oh!
There was an error while loading.Please reload this page.
- Notifications
You must be signed in to change notification settings - Fork33.7k
gh-97933: inline list/dict/set comprehensions#101441
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to ourterms of service andprivacy statement. We’ll occasionally send you account related emails.
Already on GitHub?Sign in to your account
Merged
Uh oh!
There was an error while loading.Please reload this page.
Merged
Changes from1 commit
Commits
Show all changes
56 commits Select commitHold shift + click to select a range
72afa83 gh-97933: inline sync list/dict/set comprehensions
carljm8988234 simplify cell handling code slightly
carljm22c4a86 clarify comments
carljmc1b54f0 enable inlining async comprehensions also
carljm43db9b8 fix typo
carljmed3209b Merge branch 'main' into inlinecomp2
carljmaceb6c7 fix outer-cell, inner-local case
carljme57c354 fix restoring NULL (unbound outer name) followed by load
carljm795d854 emit 1 x SWAP N+1 instead of N x SWAP 2
carljm686221a remove stray dis.dis() call
carljmac99697 fix compiler warning about Py_ssize_t -> int conversion
carljmdb208d5 Merge branch 'main' into inlinecomp2
carljm8b76051 Merge branch 'main' into inlinecomp2
carljm4620856 add a couple more tests
carljm8773653 clear comp locals on entry, eval iter expr first
carljmbe3becc Merge branch 'main' into inlinecomp2
carljmf0c051e fix double decref in error case
carljm142859a adjust to RETURN_CONST
carljm568a470 fix up refcounting
carljmadd772e Merge branch 'main' into inlinecomp2
carljm17d5d84 improve importlib comment
carljmb87d209 mark STORE_FAST_MAYBE_NULL as possibly NULLing a local
carljm36b2917 Merge branch 'main' into inlinecomp2
carljmae0bd02 Merge branch 'main' into inlinecomp2
carljm9f0fc5b Merge branch 'main' into inlinecomp2
carljm463c740 add test for NameError/UnboundLocalError
carljm67f50ba fix case where iter var is free in outer scope
carljm73dc0ed Merge branch 'main' into inlinecomp2
carljmecb313c Merge branch 'main' into inlinecomp2
carljm4109baa add inlining of non-function-scope comprehensions
carljmd8802a1 simplify scope handling
carljm1c019a7 Merge branch 'main' into inlinecomp2
carljm24a9d9f Merge branch 'main' into inlinecomp2
carljmb6a025b add tests for comprehensions in class scope
carljm90b34de run all listcomp scope tests in module, class, and func scope
carljm06db319 Merge branch 'main' into inlinecomp2
carljmb52046b handle frame locals materialization in class/module scope
carljm6c5f269 Merge branch 'main' into inlinecomp2
carljm1a8f4a0 Merge branch 'main' into inlinecomp2
carljm1274e2b Merge branch 'main' into inlinecomp2
carljm0727d6f Merge branch 'main' into inlinecomp2
carljm51a1294 update comment
carljm8a78a36 Merge branch 'main' into inlinecomp2
carljm43722b4 review comments
carljmbf9e1f1 fix single backticks
carljmca636a5 better nested test
carljm46c7a4f fix u_fasthidden in nested case
carljmfb9f89e fix refleak
carljmbaacf5f Merge branch 'main' into inlinecomp2
carljm5914d77 remove assumption that class scopes can't have cellvars
carljmffae4e6 Apply suggestions from code review
carljm76077cd Merge branch 'main' into inlinecomp2
carljma8425a6 review comments
carljm656e46b Apply suggestions from code review
carljm1402e7a Apply suggestions from code review
carljm95401fe Merge branch 'main' into inlinecomp2
carljmFile filter
Filter by extension
Conversations
Failed to load comments.
Loading
Uh oh!
There was an error while loading.Please reload this page.
Jump to
Jump to file
Failed to load files.
Loading
Uh oh!
There was an error while loading.Please reload this page.
Diff view
Diff view
clear comp locals on entry, eval iter expr first
- Loading branch information
Uh oh!
There was an error while loading.Please reload this page.
commit8773653b4c078591d6c1ea28c1285b0922a775f6
There are no files selected for viewing
8 changes: 3 additions & 5 deletionsDoc/library/dis.rst
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.Learn more about bidirectional Unicode characters
4 changes: 2 additions & 2 deletionsInclude/internal/pycore_opcode.h
Some generated files are not rendered by default. Learn more abouthow customized files appear on GitHub.
Oops, something went wrong.
Uh oh!
There was an error while loading.Please reload this page.
2 changes: 1 addition & 1 deletionInclude/opcode.h
Some generated files are not rendered by default. Learn more abouthow customized files appear on GitHub.
Oops, something went wrong.
Uh oh!
There was an error while loading.Please reload this page.
2 changes: 1 addition & 1 deletionLib/opcode.py
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -550,14 +550,14 @@ static int compiler_sync_comprehension_generator( | ||
| asdl_comprehension_seq *generators, int gen_index, | ||
| int depth, | ||
| expr_ty elt, expr_ty val, int type, | ||
| intiter_on_stack); | ||
| static int compiler_async_comprehension_generator( | ||
| struct compiler *c, location loc, | ||
| asdl_comprehension_seq *generators, int gen_index, | ||
| int depth, | ||
| expr_ty elt, expr_ty val, int type, | ||
| intiter_on_stack); | ||
| static int compiler_pattern(struct compiler *, pattern_ty, pattern_context *); | ||
| static int compiler_match(struct compiler *, stmt_ty); | ||
| @@ -1229,7 +1229,7 @@ stack_effect(int opcode, int oparg, int jump) | ||
| case LOAD_FAST: | ||
| case LOAD_FAST_CHECK: | ||
| caseLOAD_FAST_AND_CLEAR: | ||
| return 1; | ||
| case STORE_FAST: | ||
| case STORE_FAST_MAYBE_NULL: | ||
| @@ -5150,18 +5150,18 @@ compiler_comprehension_generator(struct compiler *c, location loc, | ||
| asdl_comprehension_seq *generators, int gen_index, | ||
| int depth, | ||
| expr_ty elt, expr_ty val, int type, | ||
| intiter_on_stack) | ||
| { | ||
| comprehension_ty gen; | ||
| gen = (comprehension_ty)asdl_seq_GET(generators, gen_index); | ||
| if (gen->is_async) { | ||
| return compiler_async_comprehension_generator( | ||
| c, loc, generators, gen_index, depth, elt, val, type, | ||
| iter_on_stack); | ||
| } else { | ||
| return compiler_sync_comprehension_generator( | ||
| c, loc, generators, gen_index, depth, elt, val, type, | ||
| iter_on_stack); | ||
| } | ||
| } | ||
| @@ -5170,7 +5170,7 @@ compiler_sync_comprehension_generator(struct compiler *c, location loc, | ||
| asdl_comprehension_seq *generators, | ||
| int gen_index, int depth, | ||
| expr_ty elt, expr_ty val, int type, | ||
| intiter_on_stack) | ||
| { | ||
| /* generate code for the iterator, then each of the ifs, | ||
| and then write to the element */ | ||
| @@ -5182,37 +5182,39 @@ compiler_sync_comprehension_generator(struct compiler *c, location loc, | ||
| comprehension_ty gen = (comprehension_ty)asdl_seq_GET(generators, | ||
| gen_index); | ||
| if (!iter_on_stack) { | ||
| if (gen_index == 0) { | ||
| /* Receive outermost iter as an implicit argument */ | ||
| c->u->u_argcount = 1; | ||
| ADDOP_I(c, loc, LOAD_FAST, 0); | ||
| } | ||
| else { | ||
| /* Sub-iter - calculate on the fly */ | ||
| /* Fast path for the temporary variable assignment idiom: | ||
| for y in [f(x)] | ||
| */ | ||
| asdl_expr_seq *elts; | ||
| switch (gen->iter->kind) { | ||
| case List_kind: | ||
| elts = gen->iter->v.List.elts; | ||
| break; | ||
| case Tuple_kind: | ||
| elts = gen->iter->v.Tuple.elts; | ||
| break; | ||
| default: | ||
| elts = NULL; | ||
| } | ||
| if (asdl_seq_LEN(elts) == 1) { | ||
| expr_ty elt = asdl_seq_GET(elts, 0); | ||
| if (elt->kind != Starred_kind) { | ||
| VISIT(c, expr, elt); | ||
| start = NO_LABEL; | ||
| } | ||
| } | ||
| if (IS_LABEL(start)) { | ||
| VISIT(c, expr, gen->iter); | ||
| ADDOP(c, loc, GET_ITER); | ||
| } | ||
| } | ||
| } | ||
| if (IS_LABEL(start)) { | ||
| @@ -5287,7 +5289,7 @@ compiler_async_comprehension_generator(struct compiler *c, location loc, | ||
| asdl_comprehension_seq *generators, | ||
| int gen_index, int depth, | ||
| expr_ty elt, expr_ty val, int type, | ||
| intiter_on_stack) | ||
| { | ||
| NEW_JUMP_TARGET_LABEL(c, start); | ||
| NEW_JUMP_TARGET_LABEL(c, except); | ||
| @@ -5296,15 +5298,17 @@ compiler_async_comprehension_generator(struct compiler *c, location loc, | ||
| comprehension_ty gen = (comprehension_ty)asdl_seq_GET(generators, | ||
| gen_index); | ||
| if (!iter_on_stack) { | ||
| if (gen_index == 0) { | ||
| /* Receive outermost iter as an implicit argument */ | ||
| c->u->u_argcount = 1; | ||
| ADDOP_I(c, loc, LOAD_FAST, 0); | ||
| } | ||
| else { | ||
| /* Sub-iter - calculate on the fly */ | ||
| VISIT(c, expr, gen->iter); | ||
| ADDOP(c, loc, GET_AITER); | ||
| } | ||
| } | ||
| USE_LABEL(c, start); | ||
| @@ -5449,7 +5453,7 @@ push_inlined_comprehension_state(struct compiler *c, location loc, | ||
| // in the case of a cell, this will actually push the cell | ||
| // itself to the stack, then we'll create a new one for the | ||
| // comprehension and restore the original one after | ||
| ADDOP_NAME(c, loc,LOAD_FAST_AND_CLEAR, k, varnames); | ||
| if (scope == CELL) { | ||
| ADDOP_NAME(c, loc, MAKE_CELL, k, cellvars); | ||
| } | ||
| @@ -5459,6 +5463,14 @@ push_inlined_comprehension_state(struct compiler *c, location loc, | ||
| } | ||
| } | ||
| } | ||
| if (state->pushed_locals) { | ||
| // Outermost iterable expression was already evaluated and is on the | ||
| // stack, we need to swap it back to TOS. This also rotates the order of | ||
| // `pushed_locals` on the stack, but this will be reversed when we swap | ||
| // out the comprehension result in pop_inlined_comprehension_state | ||
| ADDOP_I(c, loc, SWAP, PyList_GET_SIZE(state->pushed_locals) + 1); | ||
| } | ||
| return SUCCESS; | ||
| } | ||
| @@ -5479,11 +5491,13 @@ pop_inlined_comprehension_state(struct compiler *c, location loc, | ||
| if (state.pushed_locals) { | ||
| // pop names we pushed to stack earlier | ||
| Py_ssize_t npops = PyList_GET_SIZE(state.pushed_locals); | ||
| // Preserve the list/dict/set result of the comprehension as TOS. This | ||
| // reverses the SWAP we did in push_inlined_comprehension_state to get | ||
| // the outermost iterable to TOS, so we can still just iterate | ||
| // pushed_locals in simple reverse order | ||
| ADDOP_I(c, loc, SWAP, npops + 1); | ||
| for (Py_ssize_t i = npops - 1; i >= 0; --i) { | ||
| k = PyList_GetItem(state.pushed_locals, i); | ||
| if (k == NULL) { | ||
| return ERROR; | ||
| } | ||
| @@ -5493,6 +5507,19 @@ pop_inlined_comprehension_state(struct compiler *c, location loc, | ||
| return SUCCESS; | ||
| } | ||
| static inline int | ||
| compiler_comprehension_iter(struct compiler *c, location loc, | ||
| comprehension_ty comp) | ||
| { | ||
| VISIT(c, expr, comp->iter); | ||
| if (comp->is_async) { | ||
| ADDOP(c, loc, GET_AITER); | ||
| } else { | ||
carljm marked this conversation as resolved. OutdatedShow resolvedHide resolvedUh oh!There was an error while loading.Please reload this page. | ||
| ADDOP(c, loc, GET_ITER); | ||
| } | ||
| return SUCCESS; | ||
| } | ||
| static int | ||
| compiler_comprehension(struct compiler *c, expr_ty e, int type, | ||
| identifier name, asdl_comprehension_seq *generators, expr_ty elt, | ||
| @@ -5516,6 +5543,9 @@ compiler_comprehension(struct compiler *c, expr_ty e, int type, | ||
| outermost = (comprehension_ty) asdl_seq_GET(generators, 0); | ||
| if (is_inlined) { | ||
| if (compiler_comprehension_iter(c, loc, outermost)) { | ||
| goto error; | ||
| } | ||
| if (push_inlined_comprehension_state(c, loc, entry, &inline_state)) { | ||
| goto error; | ||
| } | ||
| @@ -5557,10 +5587,13 @@ compiler_comprehension(struct compiler *c, expr_ty e, int type, | ||
| } | ||
| ADDOP_I(c, loc, op, 0); | ||
| if (is_inlined) { | ||
| ADDOP_I(c, loc, SWAP, 2); | ||
| } | ||
| } | ||
| if (compiler_comprehension_generator(c, loc, generators, 0, 0, | ||
| elt, val, type, is_inlined) < 0) { | ||
| goto error_in_scope; | ||
| } | ||
| @@ -5595,13 +5628,8 @@ compiler_comprehension(struct compiler *c, expr_ty e, int type, | ||
| } | ||
| Py_DECREF(co); | ||
| if (compiler_comprehension_iter(c, loc, outermost)) { | ||
| goto error; | ||
| } | ||
| ADDOP_I(c, loc, CALL, 0); | ||
| @@ -8141,6 +8169,7 @@ scan_block_for_locals(basicblock *b, basicblock ***sp) | ||
| uint64_t bit = (uint64_t)1 << instr->i_oparg; | ||
| switch (instr->i_opcode) { | ||
| case DELETE_FAST: | ||
| case LOAD_FAST_AND_CLEAR: | ||
| unsafe_mask |= bit; | ||
| break; | ||
| case STORE_FAST: | ||
| @@ -8194,6 +8223,7 @@ fast_scan_many_locals(basicblock *entryblock, int nlocals) | ||
| assert(arg >= 0); | ||
| switch (instr->i_opcode) { | ||
| case DELETE_FAST: | ||
| case LOAD_FAST_AND_CLEAR: | ||
| states[arg - 64] = blocknum - 1; | ||
| break; | ||
| case STORE_FAST: | ||
5 changes: 3 additions & 2 deletionsPython/generated_cases.c.h
Some generated files are not rendered by default. Learn more abouthow customized files appear on GitHub.
Oops, something went wrong.
Uh oh!
There was an error while loading.Please reload this page.
6 changes: 3 additions & 3 deletionsPython/opcode_metadata.h
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.Learn more about bidirectional Unicode characters
2 changes: 1 addition & 1 deletionPython/opcode_targets.h
Some generated files are not rendered by default. Learn more abouthow customized files appear on GitHub.
Oops, something went wrong.
Uh oh!
There was an error while loading.Please reload this page.
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.