Movatterモバイル変換

[0]ホーム
URL:

Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

 Sign up
Appearance settings

gh-97696: asyncio eager tasks factory#102853

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to ourterms of service andprivacy statement. We’ll occasionally send you account related emails.

Already on GitHub?Sign in to your account

Merged
carljm merged 38 commits intopython:mainfromitamaro:eager-tasks-factory
May 1, 2023

Conversation

@itamaro
Copy link
Contributor

@itamaroitamaro commentedMar 20, 2023•
edited
Loading

fixesgh-97696

pyperformance results:

  • async tree benchmarks up to ~50% faster when using the eager tasks factory (depending on how many leaf nodes actually suspend)
  • geometric mean for all benchmark 1% faster

pyperformance run performed with local patch that installs the eager tasks factory by default (modifying the benchmark itself to use it requirespsf/pyperf#157)

full comparison inthis gist

@itamaro
Copy link
ContributorAuthor

I'm still chasing a refleak, but otherwise this should be ready for review

@itamaroitamaro marked this pull request as ready for reviewMarch 21, 2023 03:37
@gvanrossum
Copy link
Member

gvanrossum commentedMar 22, 2023•
edited
Loading

It looks like@graingert'sobservation is a serious problem. How can we fix this?

itamaroand others added8 commitsApril 20, 2023 13:42
Always create a task object to resolve the design flaw with structured concurrency.Push the eager step execution into the task implementation, so the task itself knows to eagerly execute the first step if created via the eager task factory.Co-authored-by: Jacob Bower <jbower@meta.com>
Co-authored-by: Jacob Bower <jbower@meta.com>
@itamaro
Copy link
ContributorAuthor

latest version of this PR resolves the issues raised on the issue. tests pass, and rerunning the benchmarks results in pretty much the same results (differences from the original impl within noise margins).

benchmarks ran using pyperformance withthis patch.

jbower-fb reacted with thumbs up emoji

@willingc
Copy link
Contributor

@gvanrossum@carljm Could you please take a quick look at this and make sure it resolved your comments? If so, we can merge. Thanks!

Copy link
Member

@gvanrossumgvanrossum left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

LGTM (assuming the refcount leak buildbots are happy). Go ahead without waiting for me with further changes!

PS. Next time please resist the urge to rebase -- it's harder to pick up a review after a rebase has happened. We squash commits when we merge them into main anyways.

itamaro reacted with thumbs up emoji
Copy link
Member

@carljmcarljm left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

LGTM barring a couple doc nits and one question.

@carljmcarljm added the 🔨 test-with-refleak-buildbotsTest PR w/ refleak buildbots; report in status section labelMay 1, 2023
@bedevere-bot
Copy link

🤖 New build scheduled with the buildbot fleet by@carljm for commitb17f605 🤖

If you want to schedule another build, you need to add the🔨 test-with-refleak-buildbots label again.

@bedevere-botbedevere-bot removed the 🔨 test-with-refleak-buildbotsTest PR w/ refleak buildbots; report in status section labelMay 1, 2023
@carljmcarljm merged commita474e04 intopython:mainMay 1, 2023
@JelleZijlstraJelleZijlstra mentioned this pull requestMay 1, 2023
carljm added a commit to carljm/cpython that referenced this pull requestMay 1, 2023
* main: (463 commits)pythongh-104057: Fix direct invocation of test_super (python#104064)pythongh-87092: Expose assembler to unit tests (python#103988)pythongh-97696: asyncio eager tasks factory (python#102853)pythongh-84436: Immortalize in _PyStructSequence_InitBuiltinWithFlags() (pythongh-104054)pythongh-104057: Fix direct invocation of test_module (pythonGH-104059)pythongh-100458: Clarify Enum.__format__() change of mixed-in types in the whatsnew/3.11.rst (pythonGH-100387)pythongh-104018: disallow "z" format specifier in %-format of byte strings (pythonGH-104033)pythongh-104016: Fixed off by 1 error in f string tokenizer (python#104047)pythonGH-103629: Update Unpack's repr in compliance with PEP 692 (python#104048)pythongh-102799: replace sys.exc_info by sys.exception in inspect and traceback modules (python#104032)  Fix typo in "expected" word in few source files (python#104034)pythongh-103824: fix use-after-free error in Parser/tokenizer.c (python#103993)pythongh-104035: Do not ignore user-defined `__{get,set}state__` in slotted frozen dataclasses (python#104041)pythongh-104028: Reduce object creation while calling callback function from gc (pythongh-104030)pythongh-104036: Fix direct invocation of test_typing (python#104037)pythongh-102213: Optimize the performance of `__getattr__` (pythonGH-103761)pythongh-103895: Improve how invalid `Exception.__notes__` are displayed (python#103897)  Adjust expression from `==` to `!=` in alignment with the meaning of the paragraph. (pythonGH-104021)pythongh-88496: Fix IDLE test hang on macOS (python#104025)  Improve int test coverage (python#104024)  ...
carljm added a commit to carljm/cpython that referenced this pull requestMay 2, 2023
* main: (760 commits)pythonGH-104102: Optimize `pathlib.Path.glob()` handling of `../` pattern segments (pythonGH-104103)pythonGH-104104: Optimize `pathlib.Path.glob()` by avoiding repeated calls to `os.path.normcase()` (pythonGH-104105)pythongh-103822: [Calendar] change return value to enum for day and month APIs (pythonGH-103827)pythongh-65022: Fix description of tuple return value in copyreg (python#103892)pythonGH-103525: Improve exception message from `pathlib.PurePath()` (pythonGH-103526)pythongh-84436: Add integration C API tests for immortal objects (pythongh-103962)pythongh-103743: Add PyUnstable_Object_GC_NewWithExtraData (pythonGH-103744)pythongh-102997: Update Windows installer to SQLite 3.41.2. (python#102999)pythonGH-103484: Fix redirected permanently URLs (python#104001)  Improve assert_type phrasing (python#104081)pythongh-102997: Update macOS installer to SQLite 3.41.2. (pythonGH-102998)pythonGH-103472: close response in HTTPConnection._tunnel (python#103473)pythongh-88496: IDLE - fix another test on macOS (python#104075)pythongh-94673: Hide Objects in PyTypeObject Behind Accessors (pythongh-104074)pythongh-94673: Properly Initialize and Finalize Static Builtin Types for Each Interpreter (pythongh-104072)pythongh-104016: Skip test for deeply neste f-strings on wasi (python#104071)pythongh-104057: Fix direct invocation of test_super (python#104064)pythongh-87092: Expose assembler to unit tests (python#103988)pythongh-97696: asyncio eager tasks factory (python#102853)pythongh-84436: Immortalize in _PyStructSequence_InitBuiltinWithFlags() (pythongh-104054)  ...
}
}else {
// This seems to really help performance on pyperformance benchmarks
Py_CLEAR(task->task_coro);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

Is this documented anywhere? I can't find it

Copy link
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

See my comments on#97696.

Copy link
Contributor

@jbower-fbjbower-fbMay 5, 2023•
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

Also#104304 as possible documentation.

itamaro added a commit to itamaro/cpython that referenced this pull requestMay 7, 2023
@itamaroitamaro deleted the eager-tasks-factory branchMay 7, 2023 22:16
@willingc
Copy link
Contributor

Great to see this land@itamaro. Thanks for being so collaborative on working through the PR. 🎉

itamaro reacted with heart emoji

@Dreamsorcerer
Copy link
Contributor

The proposal at the start of this thread suggested a new method or parameter for eager start. Is there a link to any discussion on why this changed? Are there plans to do something like that in future (or just to make it the default, and remove the old behaviour)?

I'm thinking about this from a library side. Changing the global task factory is inappropriate for a library to do, but we still have certain tasks that would strongly benefit from eager execution (I also wonder if a user enabling this globally could end up breaking a library that wasn't expecting eager tasks...).

So, some way to execute individual tasks eagerly would be useful (unless it simply becomes the default behaviour). We currently instantiateTask(eager_start=True) directly, though the docs seem to strongly discourage this. Adding aneager_start toasyncio.create_task(), or maybe some way to temporarily change task factories (with eager_task_factory:) would be ideal.

Sign up for freeto join this conversation on GitHub. Already have an account?Sign in to comment

Reviewers

@carljmcarljmcarljm left review comments

@gvanrossumgvanrossumgvanrossum approved these changes

@kumaraditya303kumaraditya303kumaraditya303 left review comments

@willingcwillingcwillingc approved these changes

@1st11st1Awaiting requested review from 1st11st1 is a code owner

@asvetlovasvetlovAwaiting requested review from asvetlovasvetlov is a code owner

+1 more reviewer

@jbower-fbjbower-fbjbower-fb left review comments

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Add eager task creation API to asyncio

9 participants

@itamaro@gvanrossum@willingc@bedevere-bot@Dreamsorcerer@carljm@jbower-fb@kumaraditya303@AlexWaygood
[8]ページ先頭
©2009-2025 Movatter.jp