Uh oh!
There was an error while loading.Please reload this page.
- Notifications
You must be signed in to change notification settings - Fork32k
gh-77714: Implement as_completed as an asynchronous generator#10251
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to ourterms of service andprivacy statement. We’ll occasionally send you account related emails.
Already on GitHub?Sign in to your account
Uh oh!
There was an error while loading.Please reload this page.
Conversation
Uh oh!
There was an error while loading.Please reload this page.
Lib/asyncio/tasks.py Outdated
exceptions!) of the original Futures (or coroutines), in the order | ||
in which and as soon as they complete. | ||
This differs from PEP 3148; results are yielded from asynchronous | ||
iteration rather than futures. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
The new implementation should also return futures, just like the one fromconcurrent.futures
does. The difference is that with async iteration we can return theoriginal futures, which are guaranteed to have completed.
Changingyield future.result()
toyield future
in__aiter__
makes it correct, and compatible with PEP 3148.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
The existing implementation ofas_completed
already yields results. I also think yielding results rather than futures is more natural withasync for
.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
The existing implementation yields futures which you mustawait
.
The idea is to yield futures which are guaranteed to have completed, as they complete. This is howconcurrent.futures.as_completed
works, but the benefit goes beyond consistency withconcurrent.futures
. It allows code to be robust in the face of exceptions raised by futures, and to associate side data with the futures and retrieve it fromas_completed
. For example, when yielding futures,this usage example immediately becomes usable withasyncio.as_completed
, just changingfor
toasync for
.
The sync version doesn't support that usage because it gives you adifferent future than any of those passed toas_completed
. Among other things, that's what prompted the BPO issue.
As far as I can tell, your implementation can easily support that, just by removing.result()
at theyield
point.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
You're right, I misread what was going on before. Thanks for the explanation!
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
Hi@mivade, Thanks for the PR. Since this PR has been open for a while, would you prefer to close the PR or resolve the CI errors? Thanks.
It has indeed been quite some time! I should be able to take a look at resolving conflicts later this week if there is not a competing PR that would be ready to merge. |
@kumaraditya303 Can you review this? I just don't have the time, and you expressed interest in the GH issue before. |
bvd0 commentedDec 16, 2023
What needs to happen for this to get merged? |
I’ve asked@serhiy-storchaka for help. |
# This is *not* a @coroutine! It is just an iterator (yielding Futures). | ||
def as_completed(fs, *, timeout=None): | ||
"""Return an iterator whose values are coroutines. | ||
class as_completed(object): |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
classas_completed(object): | |
classas_completed: |
Inheriting from object isn't necessary
Thank you for your PR, but#22491 is more complete. It returns an iterator (not simply an iterable) and contains tests. So we focused on it. |
Glad to see there's movement on addressing this issue! |
Uh oh!
There was an error while loading.Please reload this page.
This implements
asyncio.as_completed
withasync for
support while keeping the old iteration method for backwards compatibility.This obviously needs new tests. I was also hoping for some additional feedback regarding the implementation.
https://bugs.python.org/issue33533