Movatterモバイル変換


[0]ホーム

URL:


Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

Sign up
Appearance settings

gh-104584: Baby steps towards generating and executing traces#105924

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to ourterms of service andprivacy statement. We’ll occasionally send you account related emails.

Already on GitHub?Sign in to your account

Merged
gvanrossum merged 68 commits intopython:mainfromgvanrossum:optim-exec
Jun 27, 2023

Conversation

gvanrossum
Copy link
Member

@gvanrossumgvanrossum commentedJun 19, 2023
edited by bedevere-bot
Loading

This is mostly so we can benchmark how bad the overhead is.

Eclips4 reacted with thumbs up emoji
@gvanrossum
Copy link
MemberAuthor

gvanrossum commentedJun 19, 2023
edited
Loading

Idon't now understand the compilation failure:

Error: ../cpython-ro-srcdir/Python/optimizer.c:304:21: error: variably modified ‘trace’ at file scope  304 |     uop_instruction trace[MAX_TRACE_LENGTH];  // TODO: variable length      |                     ^~~~~

@gvanrossum
Copy link
MemberAuthor

@markshannon
Copy link
Member

Theoparg in normal bytecode is an unsigned 32 bit integer, we should useuint32_t but we useint. In practice it is fine as
0 <= oparg < (2**31) is always true.
The 64 bit caches hold a pointer, so an explicit cast fromuint64_t touintptr_t should keep MSVC happy.

gvanrossum reacted with thumbs up emoji

Now test_mmap on Windows passes,and hopefully so does test_threading on UNIX.
Instead of `int oparg` we now have `uint64_t operand`.In the Tier 2 interpreter, we add `int oparg` and set itto a truncated version of `operand` so instructions canreference it.
@gvanrossum
Copy link
MemberAuthor

gvanrossum commentedJun 26, 2023
edited
Loading

The latest change indeed fixestest_mmap. It does nothing fortest_threading, that was a typo.

I had hoped it would also fixtest_memoryview, and while it reduces the failure rate there from 12 to 1 failing test, there is still that one failing test...

======================================================================ERROR: test_contextmanager (test.test_memoryview.ArrayMemorySliceSliceTest.test_contextmanager)----------------------------------------------------------------------Traceback (most recent call last):  File "/Users/guido/cpython/Lib/test/test_memoryview.py", line 288, in test_contextmanager    self._check_released(m, tp)  File "/Users/guido/cpython/Lib/test/test_memoryview.py", line 273, in _check_released    with m:         ^ValueError: operation forbidden on released memoryview object----------------------------------------------------------------------

I will continue to debug this for a bit, but I'll probably choose to just disable the offending bytecode again.


UPDATE: I gave up. I have a hunch it's related to a trace that containstwoBEFORE_WITH uops where the second one fails, and then possibly the exception table might be off? But that ought to be driven by theSET_IP logic, which I believe is correct. I don't want to dive into this, so I just chickened out by skipping those opcodes.

Apparently the fix using SET_LOCALS_FROM_FRAME() isn't enough(it fixes some but not all failures). Possibly the problem is dueto having *two* BEFORE_WITH uops in one trace?Unclear what's wrong with that opcode.
@gvanrossumgvanrossum added the 🔨 test-with-refleak-buildbotsTest PR w/ refleak buildbots; report in status section labelJun 26, 2023
@bedevere-bot
Copy link

🤖 New build scheduled with the buildbot fleet by@gvanrossum for commit4bd7bb6 🤖

If you want to schedule another build, you need to add the🔨 test-with-refleak-buildbots label again.

@bedevere-botbedevere-bot removed the 🔨 test-with-refleak-buildbotsTest PR w/ refleak buildbots; report in status section labelJun 26, 2023
@gvanrossum
Copy link
MemberAuthor

gvanrossum commentedJun 26, 2023
edited
Loading

Regarding new files, I don't think it will stand in the way of progress if this were merged with the superblock creation code in Python/optimizer.c and the trace interpreter in Python/ceval.c. If at some point in the future that becomes a problem we can refactor it then. I'd like this PR to remain a spike, not become a marathon.

Last bits of work:

  • Run with refleaks bot
  • pylifecycle.c work
  • Add Misc/NEWS.d entry

The uops optimizer is enabled in the main interpreter when either:- -Xuops is passed; or- PYTHONUOPS=1 is set.The env var for uops debugging is now called PYTHONUOPSDEBUG.It only has an effect when configured --with-pydebug *and* the uopsoptimizer is enabled (see above).Also fixed a compiler warning about %ld with an int64 arg.
Those tests now only fail when run with -Xuops.We should probably fix the uop executor instead.Also, test_embed now passes even with -Xuops.This reverts commit91dd86c.
@gvanrossumgvanrossum added the 🔨 test-with-refleak-buildbotsTest PR w/ refleak buildbots; report in status section labelJun 26, 2023
@bedevere-bot
Copy link

🤖 New build scheduled with the buildbot fleet by@gvanrossum for commit6606c5d 🤖

If you want to schedule another build, you need to add the🔨 test-with-refleak-buildbots label again.

@bedevere-botbedevere-bot removed the 🔨 test-with-refleak-buildbotsTest PR w/ refleak buildbots; report in status section labelJun 26, 2023
@gvanrossum
Copy link
MemberAuthor

@markshannon I have a niggling suspicion that if the veryfirst uop in a trace ends with a failing DEOPT_IF (or ERROR_IF?), it will do the wrong thing, sincenext_instr andframe->prev_instr are still set where the ENTER_EXECUTOR found it. But I don't think any tests have triggered such a situation. If I'm right, the fix would be to start the trace with aSET_IP uops. Does that make sense to you?

@gvanrossum
Copy link
MemberAuthor

The refleaks run failed, but that was an existing issue (#105273). They didn't test there were no refleaks left, because I also pushed the change that requires-Xuops to enable the new optimizer. But the leak that@kumaraditya303 found was pretty glaring, and I ran some tests that were previously leaking locally, and those no longer leaked, so I am okay with this aspect. I will now add a news blurb and merge. We can reiterate after that.

@bedevere-bot
Copy link

⚠️⚠️⚠️ Buildbot failure⚠️⚠️⚠️

Hi! The buildbotARM Raspbian 3.x has failed when building commit51fc725.

What do you need to do:

  1. Don't panic.
  2. Checkthe buildbot page in the devguide if you don't know what the buildbots are or how they work.
  3. Go to the page of the buildbot that failed (https://buildbot.python.org/all/#builders/424/builds/4281) and take a look at the build logs.
  4. Check if the failure is related to this commit (51fc725) or if it is a false positive.
  5. If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/all/#builders/424/builds/4281

Summary of the results of the build (if available):

== Tests result: ENV CHANGED ==

426 tests OK.

10 slowest tests:

  • test_venv: 10 min 2 sec
  • test_source_encoding: 9 min 11 sec
  • test_largefile: 8 min 43 sec
  • test_multiprocessing_spawn: 6 min 58 sec
  • test_math: 4 min 25 sec
  • test_peg_generator: 4 min 19 sec
  • test_cppext: 4 min 9 sec
  • test_codecs: 2 min 58 sec
  • test_concurrent_futures: 2 min 58 sec
  • test_argparse: 2 min 33 sec

1 test altered the execution environment:
test_multiprocessing_spawn

19 tests skipped:
test.test_asyncio.test_windows_events
test.test_asyncio.test_windows_utils test_devpoll test_idle
test_ioctl test_kqueue test_launcher test_perf_profiler
test_startfile test_tcl test_tkinter test_ttk test_ttk_textonly
test_turtle test_winconsoleio test_winreg test_winsound test_wmi
test_zipfile64

Total duration: 39 min 6 sec

Click to see traceback logs
remote:Enumerating objects: 36, done.remote:Counting objects:   2% (1/35)remote:Counting objects:   5% (2/35)remote:Counting objects:   8% (3/35)remote:Counting objects:  11% (4/35)remote:Counting objects:  14% (5/35)remote:Counting objects:  17% (6/35)remote:Counting objects:  20% (7/35)remote:Counting objects:  22% (8/35)remote:Counting objects:  25% (9/35)remote:Counting objects:  28% (10/35)remote:Counting objects:  31% (11/35)remote:Counting objects:  34% (12/35)remote:Counting objects:  37% (13/35)remote:Counting objects:  40% (14/35)remote:Counting objects:  42% (15/35)remote:Counting objects:  45% (16/35)remote:Counting objects:  48% (17/35)remote:Counting objects:  51% (18/35)remote:Counting objects:  54% (19/35)remote:Counting objects:  57% (20/35)remote:Counting objects:  60% (21/35)remote:Counting objects:  62% (22/35)remote:Counting objects:  65% (23/35)remote:Counting objects:  68% (24/35)remote:Counting objects:  71% (25/35)remote:Counting objects:  74% (26/35)remote:Counting objects:  77% (27/35)remote:Counting objects:  80% (28/35)remote:Counting objects:  82% (29/35)remote:Counting objects:  85% (30/35)remote:Counting objects:  88% (31/35)remote:Counting objects:  91% (32/35)remote:Counting objects:  94% (33/35)remote:Counting objects:  97% (34/35)remote:Counting objects: 100% (35/35)remote:Counting objects: 100% (35/35), done.remote:Compressing objects:   3% (1/26)remote:Compressing objects:   7% (2/26)remote:Compressing objects:  11% (3/26)remote:Compressing objects:  15% (4/26)remote:Compressing objects:  19% (5/26)remote:Compressing objects:  23% (6/26)remote:Compressing objects:  26% (7/26)remote:Compressing objects:  30% (8/26)remote:Compressing objects:  34% (9/26)remote:Compressing objects:  38% (10/26)remote:Compressing objects:  42% (11/26)remote:Compressing objects:  46% (12/26)remote:Compressing objects:  50% (13/26)remote:Compressing objects:  53% (14/26)remote:Compressing objects:  57% (15/26)remote:Compressing objects:  61% (16/26)remote:Compressing objects:  65% (17/26)remote:Compressing objects:  69% (18/26)remote:Compressing objects:  73% (19/26)remote:Compressing objects:  76% (20/26)remote:Compressing objects:  80% (21/26)remote:Compressing objects:  84% (22/26)remote:Compressing objects:  88% (23/26)remote:Compressing objects:  92% (24/26)remote:Compressing objects:  96% (25/26)remote:Compressing objects: 100% (26/26)remote:Compressing objects: 100% (26/26), done.remote:Total 36 (delta 10), reused 10 (delta 9), pack-reused 1From https://github.com/python/cpython * branch                  main       -> FETCH_HEADNote:switching to '51fc72511733353de15bc633a3d7b6da366842e4'.You are in 'detached HEAD' state. You can look around, make experimentalchanges and commit them, and you can discard any commits you make in thisstate without impacting any branches by switching back to a branch.If you want to create a new branch to retain commits you create, you maydo so (now or later) by using -c with the switch command. Example:  git switch -c <new-branch-name>Or undo this operation with:  git switch -Turn off this advice by setting config variable advice.detachedHead to falseHEAD is now at 51fc725117 gh-104584: Baby steps towards generating and executing traces (#105924)Switched to and reset branch 'main'In file included from ./Include/Python.h:35,from Python/ceval.c:5:Python/ceval.c:In function ‘_PyUopExecute’:Python/ceval_macros.h:271:31: warning: array subscript 310 is above array bounds of ‘const uint8_t[256]’ {aka ‘const unsigned char[256]’} [-Warray-bounds]  271 |         assert(_PyOpcode_Deopt[opcode] == (INSTNAME));      \|~~~~~~~~~~~~~~~^~~~~~~~Python/executor_cases.c.h:210:13: note: in expansion of macro ‘DEOPT_IF’  210 |             DEOPT_IF(!PyUnicode_CheckExact(right), BINARY_OP);|^~~~~~~~In file included from Python/ceval.c:16:./Include/internal/pycore_opcode.h:17:22: note: while referencing ‘_PyOpcode_Deopt’   17 | extern const uint8_t _PyOpcode_Deopt[256];|^~~~~~~~~~~~~~~In file included from ./Include/Python.h:35,from Python/ceval.c:5:Python/ceval_macros.h:271:31: warning: array subscript 310 is above array bounds of ‘const uint8_t[256]’ {aka ‘const unsigned char[256]’} [-Warray-bounds]  271 |         assert(_PyOpcode_Deopt[opcode] == (INSTNAME));      \|~~~~~~~~~~~~~~~^~~~~~~~Python/executor_cases.c.h:209:13: note: in expansion of macro ‘DEOPT_IF’  209 |             DEOPT_IF(!PyUnicode_CheckExact(left), BINARY_OP);|^~~~~~~~In file included from Python/ceval.c:16:./Include/internal/pycore_opcode.h:17:22: note: while referencing ‘_PyOpcode_Deopt’   17 | extern const uint8_t _PyOpcode_Deopt[256];|^~~~~~~~~~~~~~~In file included from ./Include/Python.h:35,from Python/ceval.c:5:Python/ceval_macros.h:271:31: warning: array subscript 306 is above array bounds of ‘const uint8_t[256]’ {aka ‘const unsigned char[256]’} [-Warray-bounds]  271 |         assert(_PyOpcode_Deopt[opcode] == (INSTNAME));      \|~~~~~~~~~~~~~~~^~~~~~~~Python/executor_cases.c.h:160:13: note: in expansion of macro ‘DEOPT_IF’  160 |             DEOPT_IF(!PyFloat_CheckExact(right), BINARY_OP);|^~~~~~~~In file included from Python/ceval.c:16:./Include/internal/pycore_opcode.h:17:22: note: while referencing ‘_PyOpcode_Deopt’   17 | extern const uint8_t _PyOpcode_Deopt[256];|^~~~~~~~~~~~~~~In file included from ./Include/Python.h:35,from Python/ceval.c:5:Python/ceval_macros.h:271:31: warning: array subscript 306 is above array bounds of ‘const uint8_t[256]’ {aka ‘const unsigned char[256]’} [-Warray-bounds]  271 |         assert(_PyOpcode_Deopt[opcode] == (INSTNAME));      \|~~~~~~~~~~~~~~~^~~~~~~~Python/executor_cases.c.h:159:13: note: in expansion of macro ‘DEOPT_IF’  159 |             DEOPT_IF(!PyFloat_CheckExact(left), BINARY_OP);|^~~~~~~~In file included from Python/ceval.c:16:./Include/internal/pycore_opcode.h:17:22: note: while referencing ‘_PyOpcode_Deopt’   17 | extern const uint8_t _PyOpcode_Deopt[256];|^~~~~~~~~~~~~~~In file included from ./Include/Python.h:35,from Python/ceval.c:5:Python/ceval_macros.h:271:31: warning: array subscript 302 is above array bounds of ‘const uint8_t[256]’ {aka ‘const unsigned char[256]’} [-Warray-bounds]  271 |         assert(_PyOpcode_Deopt[opcode] == (INSTNAME));      \|~~~~~~~~~~~~~~~^~~~~~~~Python/executor_cases.c.h:110:13: note: in expansion of macro ‘DEOPT_IF’  110 |             DEOPT_IF(!PyLong_CheckExact(right), BINARY_OP);|^~~~~~~~In file included from Python/ceval.c:16:./Include/internal/pycore_opcode.h:17:22: note: while referencing ‘_PyOpcode_Deopt’   17 | extern const uint8_t _PyOpcode_Deopt[256];|^~~~~~~~~~~~~~~In file included from ./Include/Python.h:35,from Python/ceval.c:5:Python/ceval_macros.h:271:31: warning: array subscript 302 is above array bounds of ‘const uint8_t[256]’ {aka ‘const unsigned char[256]’} [-Warray-bounds]  271 |         assert(_PyOpcode_Deopt[opcode] == (INSTNAME));      \|~~~~~~~~~~~~~~~^~~~~~~~Python/executor_cases.c.h:109:13: note: in expansion of macro ‘DEOPT_IF’  109 |             DEOPT_IF(!PyLong_CheckExact(left), BINARY_OP);|^~~~~~~~In file included from Python/ceval.c:16:./Include/internal/pycore_opcode.h:17:22: note: while referencing ‘_PyOpcode_Deopt’   17 | extern const uint8_t _PyOpcode_Deopt[256];|^~~~~~~~~~~~~~~./Modules/_testcapi/heaptype_relative.c:In function ‘make_sized_heaptypes’:./Modules/_testcapi/heaptype_relative.c:62:28: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast]   62 |                            (unsigned long long)data_ptr,|^make:*** [Makefile:2002: buildbottest] Error 3

Sign up for freeto join this conversation on GitHub. Already have an account?Sign in to comment
Reviewers

@iritkatrieliritkatrieliritkatriel left review comments

@markshannonmarkshannonmarkshannon left review comments

@kumaraditya303kumaraditya303kumaraditya303 left review comments

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

6 participants
@gvanrossum@mdboom@bedevere-bot@markshannon@iritkatriel@kumaraditya303

[8]ページ先頭

©2009-2025 Movatter.jp