Movatterモバイル変換


[0]ホーム

URL:


Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

Sign up
Appearance settings

gh-97933: inline list/dict/set comprehensions#101441

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to ourterms of service andprivacy statement. We’ll occasionally send you account related emails.

Already on GitHub?Sign in to your account

Merged
carljm merged 56 commits intopython:mainfromcarljm:inlinecomp2
May 9, 2023
Merged
Show file tree
Hide file tree
Changes fromall commits
Commits
Show all changes
56 commits
Select commitHold shift + click to select a range
72afa83
gh-97933: inline sync list/dict/set comprehensions
carljmJan 30, 2023
8988234
simplify cell handling code slightly
carljmJan 31, 2023
22c4a86
clarify comments
carljmJan 31, 2023
c1b54f0
enable inlining async comprehensions also
carljmJan 31, 2023
43db9b8
fix typo
carljmJan 31, 2023
ed3209b
Merge branch 'main' into inlinecomp2
carljmJan 31, 2023
aceb6c7
fix outer-cell, inner-local case
carljmJan 31, 2023
e57c354
fix restoring NULL (unbound outer name) followed by load
carljmJan 31, 2023
795d854
emit 1 x SWAP N+1 instead of N x SWAP 2
carljmJan 31, 2023
686221a
remove stray dis.dis() call
carljmJan 31, 2023
ac99697
fix compiler warning about Py_ssize_t -> int conversion
carljmJan 31, 2023
db208d5
Merge branch 'main' into inlinecomp2
carljmJan 31, 2023
8b76051
Merge branch 'main' into inlinecomp2
carljmFeb 1, 2023
4620856
add a couple more tests
carljmFeb 1, 2023
8773653
clear comp locals on entry, eval iter expr first
carljmFeb 1, 2023
be3becc
Merge branch 'main' into inlinecomp2
carljmFeb 9, 2023
f0c051e
fix double decref in error case
carljmFeb 9, 2023
142859a
adjust to RETURN_CONST
carljmFeb 9, 2023
568a470
fix up refcounting
carljmFeb 10, 2023
add772e
Merge branch 'main' into inlinecomp2
carljmFeb 10, 2023
17d5d84
improve importlib comment
carljmFeb 10, 2023
b87d209
mark STORE_FAST_MAYBE_NULL as possibly NULLing a local
carljmFeb 10, 2023
36b2917
Merge branch 'main' into inlinecomp2
carljmFeb 10, 2023
ae0bd02
Merge branch 'main' into inlinecomp2
carljmFeb 13, 2023
9f0fc5b
Merge branch 'main' into inlinecomp2
carljmFeb 20, 2023
463c740
add test for NameError/UnboundLocalError
carljmFeb 28, 2023
67f50ba
fix case where iter var is free in outer scope
carljmFeb 28, 2023
73dc0ed
Merge branch 'main' into inlinecomp2
carljmFeb 28, 2023
ecb313c
Merge branch 'main' into inlinecomp2
carljmMar 6, 2023
4109baa
add inlining of non-function-scope comprehensions
carljmMar 7, 2023
d8802a1
simplify scope handling
carljmMar 7, 2023
1c019a7
Merge branch 'main' into inlinecomp2
carljmMar 7, 2023
24a9d9f
Merge branch 'main' into inlinecomp2
carljmMar 8, 2023
b6a025b
add tests for comprehensions in class scope
carljmMar 8, 2023
90b34de
run all listcomp scope tests in module, class, and func scope
carljmMar 8, 2023
06db319
Merge branch 'main' into inlinecomp2
carljmMar 8, 2023
b52046b
handle frame locals materialization in class/module scope
carljmMar 14, 2023
6c5f269
Merge branch 'main' into inlinecomp2
carljmMar 14, 2023
1a8f4a0
Merge branch 'main' into inlinecomp2
carljmMar 17, 2023
1274e2b
Merge branch 'main' into inlinecomp2
carljmMay 1, 2023
0727d6f
Merge branch 'main' into inlinecomp2
carljmMay 2, 2023
51a1294
update comment
carljmMay 2, 2023
8a78a36
Merge branch 'main' into inlinecomp2
carljmMay 5, 2023
43722b4
review comments
carljmMay 5, 2023
bf9e1f1
fix single backticks
carljmMay 5, 2023
ca636a5
better nested test
carljmMay 5, 2023
46c7a4f
fix u_fasthidden in nested case
carljmMay 5, 2023
fb9f89e
fix refleak
carljmMay 5, 2023
baacf5f
Merge branch 'main' into inlinecomp2
carljmMay 5, 2023
5914d77
remove assumption that class scopes can't have cellvars
carljmMay 5, 2023
ffae4e6
Apply suggestions from code review
carljmMay 9, 2023
76077cd
Merge branch 'main' into inlinecomp2
carljmMay 9, 2023
a8425a6
review comments
carljmMay 9, 2023
656e46b
Apply suggestions from code review
carljmMay 9, 2023
1402e7a
Apply suggestions from code review
carljmMay 9, 2023
95401fe
Merge branch 'main' into inlinecomp2
carljmMay 9, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletionsDoc/library/dis.rst
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -1196,6 +1196,14 @@ iterations of the loop.

.. versionadded:: 3.12

.. opcode:: LOAD_FAST_AND_CLEAR (var_num)

Pushes a reference to the local ``co_varnames[var_num]`` onto the stack (or
pushes ``NULL`` onto the stack if the local variable has not been
initialized) and sets ``co_varnames[var_num]`` to ``NULL``.

.. versionadded:: 3.12

.. opcode:: STORE_FAST (var_num)

Stores ``STACK.pop()`` into the local ``co_varnames[var_num]``.
Expand Down
24 changes: 24 additions & 0 deletionsDoc/whatsnew/3.12.rst
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -153,6 +153,30 @@ New Features
In Python 3.14, the default will switch to ``'data'``.
(Contributed by Petr Viktorin in :pep:`706`.)

.. _whatsnew312-pep709:

PEP 709: Comprehension inlining
-------------------------------

Dictionary, list, and set comprehensions are now inlined, rather than creating a
new single-use function object for each execution of the comprehension. This
speeds up execution of a comprehension by up to 2x.

Comprehension iteration variables remain isolated; they don't overwrite a
variable of the same name in the outer scope, nor are they visible after the
comprehension. This isolation is now maintained via stack/locals manipulation,
not via separate function scope.

Inlining does result in a few visible behavior changes:

* There is no longer a separate frame for the comprehension in tracebacks,
and tracing/profiling no longer shows the comprehension as a function call.
* Calling :func:`locals` inside a comprehension now includes variables
from outside the comprehension, and no longer includes the synthetic ``.0``
variable for the comprehension "argument".

Contributed by Carl Meyer and Vladimir Matveev in :pep:`709`.

PEP 688: Making the buffer protocol accessible in Python
--------------------------------------------------------

Expand Down
1 change: 1 addition & 0 deletionsInclude/internal/pycore_code.h
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -131,6 +131,7 @@ struct callable_cache {
// Note that these all fit within a byte, as do combinations.
// Later, we will use the smaller numbers to differentiate the different
// kinds of locals (e.g. pos-only arg, varkwargs, local-only).
#define CO_FAST_HIDDEN 0x10
#define CO_FAST_LOCAL 0x20
#define CO_FAST_CELL 0x40
#define CO_FAST_FREE 0x80
Expand Down
3 changes: 3 additions & 0 deletionsInclude/internal/pycore_compile.h
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -70,6 +70,9 @@ typedef struct {
PyObject *u_varnames; /* local variables */
PyObject *u_cellvars; /* cell variables */
PyObject *u_freevars; /* free variables */
PyObject *u_fasthidden; /* dict; keys are names that are fast-locals only
temporarily within an inlined comprehension. When
value is True, treat as fast-local. */

Py_ssize_t u_argcount; /* number of arguments for block */
Py_ssize_t u_posonlyargcount; /* number of positional only arguments for block */
Expand Down
2 changes: 1 addition & 1 deletionInclude/internal/pycore_flowgraph.h
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -94,7 +94,7 @@ _PyCfgInstruction* _PyCfg_BasicblockLastInstr(const _PyCfgBasicblock *b);
int _PyCfg_OptimizeCodeUnit(_PyCfgBuilder *g, PyObject *consts, PyObject *const_cache,
int code_flags, int nlocals, int nparams, int firstlineno);
int _PyCfg_Stackdepth(_PyCfgBasicblock *entryblock, int code_flags);
void_PyCfg_ConvertExceptionHandlersToNops(_PyCfgBasicblock *entryblock);
void_PyCfg_ConvertPseudoOps(_PyCfgBasicblock *entryblock);
int _PyCfg_ResolveJumps(_PyCfgBuilder *g);


Expand Down
13 changes: 7 additions & 6 deletionsInclude/internal/pycore_opcode.h
View file
Open in desktop

Some generated files are not rendered by default. Learn more abouthow customized files appear on GitHub.

1 change: 1 addition & 0 deletionsInclude/internal/pycore_symtable.h
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -64,6 +64,7 @@ typedef struct _symtable_entry {
unsigned ste_needs_class_closure : 1; /* for class scopes, true if a
closure over __class__
should be created */
unsigned ste_comp_inlined : 1; /* true if this comprehension is inlined */
unsigned ste_comp_iter_target : 1; /* true if visiting comprehension target */
int ste_comp_iter_expr; /* non-zero if visiting a comprehension range expression */
int ste_lineno; /* first line of block */
Expand Down
21 changes: 12 additions & 9 deletionsInclude/opcode.h
View file
Open in desktop

Some generated files are not rendered by default. Learn more abouthow customized files appear on GitHub.

3 changes: 2 additions & 1 deletionLib/importlib/_bootstrap_external.py
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -442,6 +442,7 @@ def _write_atomic(path, data, mode=0o666):
# Python 3.12b1 3526 (Add instrumentation support)
# Python 3.12b1 3527 (Add LOAD_SUPER_ATTR)
# Python 3.12b1 3528 (Add LOAD_SUPER_ATTR_METHOD specialization)
# Python 3.12b1 3529 (Inline list/dict/set comprehensions)

# Python 3.13 will start with 3550

Expand All@@ -458,7 +459,7 @@ def _write_atomic(path, data, mode=0o666):
# Whenever MAGIC_NUMBER is changed, the ranges in the magic_values array
# in PC/launcher.c must also be updated.

MAGIC_NUMBER = (3528).to_bytes(2, 'little') + b'\r\n'
MAGIC_NUMBER = (3529).to_bytes(2, 'little') + b'\r\n'

_RAW_MAGIC_NUMBER = int.from_bytes(MAGIC_NUMBER, 'little') # For import.c

Expand Down
4 changes: 4 additions & 0 deletionsLib/opcode.py
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -198,6 +198,8 @@ def pseudo_op(name, op, real_ops):
jrel_op('JUMP_BACKWARD', 140) # Number of words to skip (backwards)
name_op('LOAD_SUPER_ATTR', 141)
def_op('CALL_FUNCTION_EX', 142) # Flags
def_op('LOAD_FAST_AND_CLEAR', 143) # Local variable number
haslocal.append(143)

def_op('EXTENDED_ARG', 144)
EXTENDED_ARG = 144
Expand DownExpand Up@@ -268,6 +270,8 @@ def pseudo_op(name, op, real_ops):
pseudo_op('LOAD_ZERO_SUPER_METHOD', 264, ['LOAD_SUPER_ATTR'])
pseudo_op('LOAD_ZERO_SUPER_ATTR', 265, ['LOAD_SUPER_ATTR'])

pseudo_op('STORE_FAST_MAYBE_NULL', 266, ['STORE_FAST'])

MAX_PSEUDO_OPCODE = MIN_PSEUDO_OPCODE + len(_pseudo_ops) - 1

del def_op, name_op, jrel_op, jabs_op, pseudo_op
Expand Down
24 changes: 6 additions & 18 deletionsLib/test/test_compile.py
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -1352,14 +1352,11 @@ def test_multiline_list_comprehension(self):
and x != 50)]
""")
compiled_code, _ = self.check_positions_against_ast(snippet)
compiled_code = compiled_code.co_consts[0]
self.assertIsInstance(compiled_code, types.CodeType)
self.assertOpcodeSourcePositionIs(compiled_code, 'LIST_APPEND',
line=1, end_line=2, column=1, end_column=8, occurrence=1)
self.assertOpcodeSourcePositionIs(compiled_code, 'JUMP_BACKWARD',
line=1, end_line=2, column=1, end_column=8, occurrence=1)
self.assertOpcodeSourcePositionIs(compiled_code, 'RETURN_VALUE',
line=1, end_line=6, column=0, end_column=32, occurrence=1)

def test_multiline_async_list_comprehension(self):
snippet = textwrap.dedent("""\
Expand All@@ -1374,13 +1371,13 @@ async def f():
compiled_code, _ = self.check_positions_against_ast(snippet)
g = {}
eval(compiled_code, g)
compiled_code = g['f'].__code__.co_consts[1]
compiled_code = g['f'].__code__
self.assertIsInstance(compiled_code, types.CodeType)
self.assertOpcodeSourcePositionIs(compiled_code, 'LIST_APPEND',
line=2, end_line=3, column=5, end_column=12, occurrence=1)
self.assertOpcodeSourcePositionIs(compiled_code, 'JUMP_BACKWARD',
line=2, end_line=3, column=5, end_column=12, occurrence=1)
self.assertOpcodeSourcePositionIs(compiled_code, 'RETURN_VALUE',
self.assertOpcodeSourcePositionIs(compiled_code, 'RETURN_CONST',
line=2, end_line=7, column=4, end_column=36, occurrence=1)

def test_multiline_set_comprehension(self):
Expand All@@ -1393,14 +1390,11 @@ def test_multiline_set_comprehension(self):
and x != 50)}
""")
compiled_code, _ = self.check_positions_against_ast(snippet)
compiled_code = compiled_code.co_consts[0]
self.assertIsInstance(compiled_code, types.CodeType)
self.assertOpcodeSourcePositionIs(compiled_code, 'SET_ADD',
line=1, end_line=2, column=1, end_column=8, occurrence=1)
self.assertOpcodeSourcePositionIs(compiled_code, 'JUMP_BACKWARD',
line=1, end_line=2, column=1, end_column=8, occurrence=1)
self.assertOpcodeSourcePositionIs(compiled_code, 'RETURN_VALUE',
line=1, end_line=6, column=0, end_column=32, occurrence=1)

def test_multiline_async_set_comprehension(self):
snippet = textwrap.dedent("""\
Expand All@@ -1415,13 +1409,13 @@ async def f():
compiled_code, _ = self.check_positions_against_ast(snippet)
g = {}
eval(compiled_code, g)
compiled_code = g['f'].__code__.co_consts[1]
compiled_code = g['f'].__code__
self.assertIsInstance(compiled_code, types.CodeType)
self.assertOpcodeSourcePositionIs(compiled_code, 'SET_ADD',
line=2, end_line=3, column=5, end_column=12, occurrence=1)
self.assertOpcodeSourcePositionIs(compiled_code, 'JUMP_BACKWARD',
line=2, end_line=3, column=5, end_column=12, occurrence=1)
self.assertOpcodeSourcePositionIs(compiled_code, 'RETURN_VALUE',
self.assertOpcodeSourcePositionIs(compiled_code, 'RETURN_CONST',
line=2, end_line=7, column=4, end_column=36, occurrence=1)

def test_multiline_dict_comprehension(self):
Expand All@@ -1434,14 +1428,11 @@ def test_multiline_dict_comprehension(self):
and x != 50)}
""")
compiled_code, _ = self.check_positions_against_ast(snippet)
compiled_code = compiled_code.co_consts[0]
self.assertIsInstance(compiled_code, types.CodeType)
self.assertOpcodeSourcePositionIs(compiled_code, 'MAP_ADD',
line=1, end_line=2, column=1, end_column=7, occurrence=1)
self.assertOpcodeSourcePositionIs(compiled_code, 'JUMP_BACKWARD',
line=1, end_line=2, column=1, end_column=7, occurrence=1)
self.assertOpcodeSourcePositionIs(compiled_code, 'RETURN_VALUE',
line=1, end_line=6, column=0, end_column=32, occurrence=1)

def test_multiline_async_dict_comprehension(self):
snippet = textwrap.dedent("""\
Expand All@@ -1456,13 +1447,13 @@ async def f():
compiled_code, _ = self.check_positions_against_ast(snippet)
g = {}
eval(compiled_code, g)
compiled_code = g['f'].__code__.co_consts[1]
compiled_code = g['f'].__code__
self.assertIsInstance(compiled_code, types.CodeType)
self.assertOpcodeSourcePositionIs(compiled_code, 'MAP_ADD',
line=2, end_line=3, column=5, end_column=11, occurrence=1)
self.assertOpcodeSourcePositionIs(compiled_code, 'JUMP_BACKWARD',
line=2, end_line=3, column=5, end_column=11, occurrence=1)
self.assertOpcodeSourcePositionIs(compiled_code, 'RETURN_VALUE',
self.assertOpcodeSourcePositionIs(compiled_code, 'RETURN_CONST',
line=2, end_line=7, column=4, end_column=36, occurrence=1)

def test_matchcase_sequence(self):
Expand DownExpand Up@@ -1711,9 +1702,6 @@ def test_column_offset_deduplication(self):
for source in [
"lambda: a",
"(a for b in c)",
"[a for b in c]",
"{a for b in c}",
"{a: b for c in d}",
]:
with self.subTest(source):
code = compile(f"{source}, {source}", "<test>", "eval")
Expand Down
5 changes: 4 additions & 1 deletionLib/test/test_compiler_assemble.py
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -16,7 +16,7 @@ def complete_metadata(self, metadata, filename="myfile.py"):
metadata.setdefault(key, key)
for key in ['consts']:
metadata.setdefault(key, [])
for key in ['names', 'varnames', 'cellvars', 'freevars']:
for key in ['names', 'varnames', 'cellvars', 'freevars', 'fasthidden']:
metadata.setdefault(key, {})
for key in ['argcount', 'posonlyargcount', 'kwonlyargcount']:
metadata.setdefault(key, 0)
Expand All@@ -33,6 +33,9 @@ def assemble_test(self, insts, metadata, expected):

expected_metadata = {}
for key, value in metadata.items():
if key == "fasthidden":
# not exposed on code object
continue
if isinstance(value, list):
expected_metadata[key] = tuple(value)
elif isinstance(value, dict):
Expand Down
Loading

[8]ページ先頭

©2009-2025 Movatter.jp