Movatterモバイル変換


[0]ホーム

URL:


Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

Sign up
Appearance settings

gh-82054: allow test runner to split test_asyncio to execute in parallel by sharding.#103927

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to ourterms of service andprivacy statement. We’ll occasionally send you account related emails.

Already on GitHub?Sign in to your account

Merged
gpshead merged 5 commits intopython:mainfromzitterbewegung:gh-82054
Apr 30, 2023

Conversation

@zitterbewegung
Copy link
Contributor

@zitterbewegungzitterbewegung commentedApr 27, 2023
edited by gpshead
Loading

Summary:

This runs test_asyncio sub-tests in parallel using sharding from Cinder. This suite is typically the longest-pole in runs because it is a test package with a lot of further sub-tests otherwise run serially. By breaking out the sub-tests as independent modules we can run a lot more in parallel.

After porting we can see the direct impact on a multicore system.

  • Without this change:
    • Running make test is 5 min 26 seconds
  • With this change:
    • Running make test takes 3 min 39 seconds

The drawbacks are that this implementation is hacky and due to the sorting of the tests it obscures when the asyncio tests occur and involves changing CPython test infrastructure but, the wall time saved it is worth it, especially in low-core count CI runs as it pulls a long tail. The win for productivity and reserved CI resource usage is significant.

Future tests that deserve to be refactored into split up suites to benefit from aretest_concurrent_futures and the way the_test_multiprocessing suite gets run for all start methods. As exposed by passing the-o flag topython -m test to get a list of the 10 longest running tests.

@ghost
Copy link

ghost commentedApr 27, 2023
edited by ghost
Loading

All commit authors signed the Contributor License Agreement.
CLA signed

@arhadthedev
Copy link
Member

@zitterbewegung The commits are created using the wrong e-mail. Could yougit config user.email "...@...", recreate the commits and force-push, please?

…asyncio in parallel.By sharding the individual tests in test_multiprocessing and test_asyncio we are able to reduce considerably the time it takes to run the whole test suite.
@zitterbewegungzitterbewegung marked this pull request as ready for reviewApril 27, 2023 20:04
@zitterbewegung
Copy link
ContributorAuthor

zitterbewegung commentedApr 27, 2023
edited
Loading

@zitterbewegung The commits are created using the wrong e-mail. Could yougit config user.email "...@...", recreate the commits and force-push, please?

@arhadthedev Okay recreated and force pushed.

arhadthedev reacted with thumbs up emoji

@carljmcarljm requested a review fromgpsheadApril 29, 2023 00:27
Copy link
Member

@gpsheadgpshead left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

I think we've still got a while before this meaningfully reduces most CI system (presumed 2-4 thread) or developer (presumed 8-32 thread) whole test suite latency. With this PR on a fast system:

10 slowest tests:- test_concurrent_futures: 2 min 14 sec- test_multiprocessing_spawn: 1 min 25 sec- test_multiprocessing_forkserver: 1 min 9 sec- test_signal: 1 min 7 sec- test_tools: 1 min 3 sec- test_multiprocessing_fork: 1 min 3 sec- test_socket: 37.6 sec- test_imaplib: 30.6 sec- test_io: 29.7 sec- test.test_asyncio.test_tasks: 28.8 sec

UntilLib/test_concurrent_futures.py is broken up into a package of parallelizable items we've still got a pretty long tail.

The same really should be done withLib/test/_test_multiprocessing.py as well, but it's a more complicated beast as it is a single test "template" that gets invoked with three different start method settings configured by theLib/test/test_multiprocessing_{fork,forkserver,spawn}.py files.

Regardless this PR is still a win even with our longest test, test_asyncio, which would otherwise be 3 minutes above, split up. That reduces the total run time latency 10% on a CI-like-j 4 run.

zitterbewegung reacted with heart emoji
@gpsheadgpshead self-assigned thisApr 29, 2023
@gpsheadgpshead added the testsTests in the Lib/test dir labelApr 29, 2023
@zitterbewegungzitterbewegung changed the titlegh-82054: allow test runner to split test_asyncio and test_multiprocessing to execute in parallel by sharding.gh-82054: allow test runner to split test_asyncio to execute in parallel by sharding.Apr 29, 2023
zitterbewegungand others added2 commitsApril 29, 2023 08:37
…ode cleanup.For the new test arguments they were made keyword only and also test_multiprocessing was removed from the SPLITTESTDIRS. The argument splittestdirs was renamed to split_test_dirs for findtests.
@gpsheadgpshead merged commit9e011e7 intopython:mainApr 30, 2023
carljm added a commit to carljm/cpython that referenced this pull requestMay 1, 2023
* main: (26 commits)pythongh-104028: Reduce object creation while calling callback function from gc (pythongh-104030)pythongh-104036: Fix direct invocation of test_typing (python#104037)pythongh-102213: Optimize the performance of `__getattr__` (pythonGH-103761)pythongh-103895: Improve how invalid `Exception.__notes__` are displayed (python#103897)  Adjust expression from `==` to `!=` in alignment with the meaning of the paragraph. (pythonGH-104021)pythongh-88496: Fix IDLE test hang on macOS (python#104025)  Improve int test coverage (python#104024)pythongh-88773: Added teleport method to Turtle library (python#103974)pythongh-104015: Fix direct invocation of `test_dataclasses` (python#104017)pythongh-104012: Ensure test_calendar.CalendarTestCase.test_deprecation_warning consistently passes (python#104014)pythongh-103977: compile re expressions in platform.py only if required (python#103981)pythongh-98003: Inline call frames for CALL_FUNCTION_EX (pythonGH-98004)  Replace Netlify with Read the Docs build previews (python#103843)  Update name in acknowledgements and add mailmap (python#103696)pythongh-82054: allow test runner to split test_asyncio to execute in parallel by sharding. (python#103927)  Remove non-existing tools from Sundry skiplist (python#103991)pythongh-103793: Defer formatting task name (python#103767)pythongh-87092: change assembler to use instruction sequence instead of CFG (python#103933)pythongh-103636: issue warning for deprecated calendar constants (python#103833)  Various small fixes to dis docs (python#103923)  ...
vstinner pushed a commit to vstinner/cpython that referenced this pull requestSep 2, 2023
… parallel by sharding. (python#103927)This runs test_asyncio sub-tests in parallel using sharding from Cinder. This suite is typically the longest-pole in runs because it is a test package with a lot of further sub-tests otherwise run serially. By breaking out the sub-tests as independent modules we can run a lot more in parallel.After porting we can see the direct impact on a multicore system.Without this change:  Running make test is 5 min 26 secondsWith this change:  Running make test takes 3 min 39 secondsThat'll vary based on system and parallelism. On a `-j 4` run similar to what CI and buildbot systems often do, it reduced the overall test suite completion latency by 10%.The drawbacks are that this implementation is hacky and due to the sorting of the tests it obscures when the asyncio tests occur and involves changing CPython test infrastructure but, the wall time saved it is worth it, especially in low-core count CI runs as it pulls a long tail. The win for productivity and reserved CI resource usage is significant.Future tests that deserve to be refactored into split up suites to benefit from are test_concurrent_futures and the way the _test_multiprocessing suite gets run for all start methods. As exposed by passing the -o flag to python -m test to get a list of the 10 longest running tests.---------Co-authored-by: Carl Meyer <carl@oddbird.net>Co-authored-by: Gregory P. Smith <greg@krypto.org> [Google, LLC](cherry picked from commit9e011e7)
vstinner added a commit that referenced this pull requestSep 3, 2023
…108820)* Revert "[3.11]gh-101634: regrtest reports decoding error as failed test (#106169) (#106175)"This reverts commitd5418e9.* Revert "[3.11]bpo-46523: fix tests rerun when `setUp[Class|Module]` fails (GH-30895) (GH-103342)"This reverts commitecb09a8.* Revert "gh-95027: Fix regrtest stdout encoding on Windows (GH-98492)"This reverts commitb2aa28e.* Revert "[3.11]gh-94026: Buffer regrtest worker stdout in temporary file (GH-94253) (GH-94408)"This reverts commit0122ab2.* Revert "Run Tools/scripts/reindent.py (GH-94225)"This reverts commitf0f3a42.* Revert "gh-94052: Don't re-run failed tests with --python option (GH-94054)"This reverts commit1347607.* Revert "[3.11]gh-84461: Fix Emscripten umask and permission issues (GH-94002) (GH-94006)"This reverts commit1073184.*gh-93353: regrtest checks for leaked temporary files (#93776)When running tests with -jN, create a temporary directory per processand mark a test as "environment changed" if a test leaks a temporaryfile or directory.(cherry picked from commite566ce5)*gh-93353: Fix regrtest for -jN with N >= 2 (GH-93813)(cherry picked from commit36934a1)*gh-93353: regrtest supports checking tmp files with -j2 (#93909)regrtest now also implements checking for leaked temporary files anddirectories when using -jN for N >= 2. Use tempfile.mkdtemp() tocreate the temporary directory. Skip this check on WASI.(cherry picked from commit4f85cec)*gh-84461: Fix Emscripten umask and permission issues (GH-94002)- Emscripten's default umask is too strict, seeemscripten-core/emscripten#17269- getuid/getgid and geteuid/getegid are stubs that always return 0  (root). Disable effective uid/gid syscalls and fix tests that use  chmod() current user.- Cannot drop X bit from directory.(cherry picked from commit2702e40)*gh-94052: Don't re-run failed tests with --python option (#94054)(cherry picked from commit0ff7b99)* Run Tools/scripts/reindent.py (#94225)Reindent files which were not properly formatted (PEP 8: 4 spaces).Remove also some trailing spaces.(cherry picked from commite87ada4)*gh-94026: Buffer regrtest worker stdout in temporary file (GH-94253)Co-authored-by: Victor Stinner <vstinner@python.org>(cherry picked from commit199ba23)*gh-96465: Clear fractions hash lru_cache under refleak testing (GH-96689)Automerge-Triggered-By: GH:zware(cherry picked from commit9c8f379)*gh-95027: Fix regrtest stdout encoding on Windows (#98492)On Windows, when the Python test suite is run with the -jN option,the ANSI code page is now used as the encoding for the stdouttemporary file, rather than using UTF-8 which can lead to decodingerrors.(cherry picked from commitec1f6f5)*gh-98903: Test suite fails with exit code 4 if no tests ran (#98904)The Python test suite now fails wit exit code 4 if no tests ran. Itshould help detecting typos in test names and test methods.* Add "EXITCODE_" constants to Lib/test/libregrtest/main.py.* Fix a typo: "NO TEST RUN" becomes "NO TESTS RAN"(cherry picked from commitc76db37)*gh-100086: Add build info to test.libregrtest (#100093)The Python test runner (libregrtest) now logs Python build information like"debug" vs "release" build, or LTO and PGO optimizations.(cherry picked from commit3c89202)*bpo-46523: fix tests rerun when `setUp[Class|Module]` fails (#30895)Co-authored-by: Jelle Zijlstra <jelle.zijlstra@gmail.com>Co-authored-by: Łukasz Langa <lukasz@langa.pl>(cherry picked from commit9953860)*gh-82054: allow test runner to split test_asyncio to execute in parallel by sharding. (#103927)This runs test_asyncio sub-tests in parallel using sharding from Cinder. This suite is typically the longest-pole in runs because it is a test package with a lot of further sub-tests otherwise run serially. By breaking out the sub-tests as independent modules we can run a lot more in parallel.After porting we can see the direct impact on a multicore system.Without this change:  Running make test is 5 min 26 secondsWith this change:  Running make test takes 3 min 39 secondsThat'll vary based on system and parallelism. On a `-j 4` run similar to what CI and buildbot systems often do, it reduced the overall test suite completion latency by 10%.The drawbacks are that this implementation is hacky and due to the sorting of the tests it obscures when the asyncio tests occur and involves changing CPython test infrastructure but, the wall time saved it is worth it, especially in low-core count CI runs as it pulls a long tail. The win for productivity and reserved CI resource usage is significant.Future tests that deserve to be refactored into split up suites to benefit from are test_concurrent_futures and the way the _test_multiprocessing suite gets run for all start methods. As exposed by passing the -o flag to python -m test to get a list of the 10 longest running tests.---------Co-authored-by: Carl Meyer <carl@oddbird.net>Co-authored-by: Gregory P. Smith <greg@krypto.org> [Google, LLC](cherry picked from commit9e011e7)* Display the sanitizer config in the regrtest header. (#105301)Display the sanitizers present in libregrtest.Having this in the CI output for tests with the relevant environmentvariable displayed will help make it easier to do what we need tocreate an equivalent local test run.(cherry picked from commit852348a)*gh-101634: regrtest reports decoding error as failed test (#106169)When running the Python test suite with -jN option, if a worker stdoutcannot be decoded from the locale encoding report a failed testn so theexitcode is non-zero.(cherry picked from commit2ac3eec)*gh-108223: test.pythoninfo and libregrtest log Py_NOGIL (#108238)Enable with --disable-gil --without-pydebug:    $ make pythoninfo|grep NOGIL    sysconfig[Py_NOGIL]: 1    $ ./python -m test    ...    == Python build: nogil debug    ...(cherry picked from commit5afe0c1)*gh-90791: test.pythoninfo logs ASAN_OPTIONS env var (#108289)* Cleanup libregrtest code logging ASAN_OPTIONS.* Fix a typo on "ASAN_OPTIONS" vs "MSAN_OPTIONS".(cherry picked from commit3a1ac87)*gh-108388: regrtest splits test_asyncio package (#108393)Currently, test_asyncio package is only splitted into sub-tests whenusing command "./python -m test". With this change, it's alsosplitted when passing it on the command line:"./python -m test test_asyncio".Remove the concept of "STDTESTS". Python is now mature enough to nothave to bother with that anymore. Removing STDTESTS simplify thecode.(cherry picked from commit174e9da)* regrtest computes statistics (#108793)test_netrc, test_pep646_syntax and test_xml_etree now return resultsin the test_main() function.Changes:* Rewrite TestResult as a dataclass with a new State class.* Add test.support.TestStats class and Regrtest.stats_dict attribute.* libregrtest.runtest functions now modify a TestResult instance  in-place.* libregrtest summary lists the number of run tests and skipped  tests, and denied resources.* Add TestResult.has_meaningful_duration() method.* Compute TestResult duration in the upper function.* Use time.perf_counter() instead of time.monotonic().* Regrtest: rename 'resource_denieds' attribute to 'resource_denied'.* Rename CHILD_ERROR to MULTIPROCESSING_ERROR.* Use match/case syntadx to have different code depending on the  test state.Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>(cherry picked from commitd4e534c)*gh-108822: Add Changelog entry for regrtest statistics (#108821)---------Co-authored-by: Christian Heimes <christian@python.org>Co-authored-by: Zachary Ware <zach@python.org>Co-authored-by: Nikita Sobolev <mail@sobolevn.me>Co-authored-by: Joshua Herman <zitterbewegung@gmail.com>Co-authored-by: Gregory P. Smith <greg@krypto.org>
Sign up for freeto join this conversation on GitHub. Already have an account?Sign in to comment

Reviewers

@gpsheadgpsheadgpshead approved these changes

Assignees

@gpsheadgpshead

Labels

skip newstestsTests in the Lib/test dir

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@zitterbewegung@arhadthedev@gpshead@carljm@bedevere-bot

[8]ページ先頭

©2009-2025 Movatter.jp