Uh oh!
There was an error while loading.Please reload this page.
- Notifications
You must be signed in to change notification settings - Fork33.3k
bpo-29302: Implement contextlib.AsyncExitStack.#4790
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to ourterms of service andprivacy statement. We’ll occasionally send you account related emails.
Already on GitHub?Sign in to your account
Uh oh!
There was an error while loading.Please reload this page.
Conversation
thehesiod commentedDec 11, 2017
thanks@Kentzo ! |
4374de3 to60000cdComparencoghlan commentedDec 14, 2017 via email
@1st1 I'd like to take a further look at this one before we merge it, butwon't have time until Jan 6th or so (just before the final alpha). Doesthat timeline work for you? |
ncoghlan commentedDec 14, 2017 via email
The alternative plan would be to merge this version based on the existingreviews, and if I have any subsequent changes I want to make, I can do thatany time before the first beta. That's probably a better approach, sinceit's more resilient against my being otherwise occupied in the first weekof January. |
1st1 commentedDec 14, 2017
Sure, I think this is worth another look so let's wait. Let's just make sure we don't forget about this before the feature freeze. |
ncoghlan left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
I like the patch overall, but there's one design concern I have, as well as a backwards compatibility concern:
- for the backwards compatibility issue: calls using keyword arguments mean we can't arbitrarily change parameter names in public APIs
- for the design question,implicitly supporting synchronous context managers in the asynchronous exit stack seems questionable to me.
The design question is one@1st1 and I talked about on the issue tracker (seehttps://bugs.python.org/issue29302#msg288735 and the next couple of comments), and after seeing the code in PR form, Idefinitely want to go with the more explicit API:
- add separate
*_syncvariants of all the callback registration APIs in the async exit stack variant rather than trying to guess the user's intent based on the type of the object they passed in (if guessing proves desirable, we can add it later, but if we start with it, we're likely stuck with it even if it proves confusing) - avoid checking for awaitables when unwinding the stack in the async case (either add an awaitable wrapper to the synchronous callbacks as I suggest in this review, or else store a 2-tuple as@1st1 suggested on the tracker)
Lib/contextlib.py Outdated
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
Folks may be callingstack.push(exit=obj), so we shouldn't gratuitously change parameter names.
Lib/contextlib.py Outdated
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
This introduces a dependency I'd prefer to avoid -inspect pulls in alot of other modules, andcontextlib is used frequently enough that we should aim to minimise its dependency tree.
Lib/contextlib.py Outdated
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
This comment is oddly placed, and doesn't seem especially accurate - the check is being made against__aexit__, not__aenter__.
More generally, implicitly interleaving synchronous with asynchronous context managers seems dubious to me - while it might be a good idea to allow it, doing it automagically rather than explicitly seems like a recipe for confusion when reading and debugging code.
Lib/contextlib.py Outdated
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
If we do decide to allow synchronous context managers in asynchronous exit stacks, I'd suggest wrapping their exit callbacks in an awaitable, so this check can be avoided.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
inspect is also used by_create_exit_wrapper and_create_cb_wrapper.
bedevere-bot commentedJan 20, 2018
A Python core developer has requested some changes be made to your pull request before we can consider merging it. If you could please address their requests along with any other requests in other reviews from core developers that would be appreciated. Once you have made the requested changes, please leave a comment on this pull request containing the phrase |
1st1 commentedJan 20, 2018 • edited
Loading Uh oh!
There was an error while loading.Please reload this page.
edited
Uh oh!
There was an error while loading.Please reload this page.
Couple of thoughts:
withFoo:asyncwithBar:withHam: ...
|
1st1 commentedJan 20, 2018
Also, we need to get this merged before Jan 29.@Kentzo if you don't have time to work on this in the next few days I can handle this PR myself. |
ncoghlan commentedJan 20, 2018 • edited
Loading Uh oh!
There was an error while loading.Please reload this page.
edited
Uh oh!
There was an error while loading.Please reload this page.
@1st1 So the differences between AsyncExitStack and ExitStack would be:
I like that, since it means registering synchronous context managers and callbacks stays the same regardless of which kind of stack you're using. A note regarding the method names: the synchronous method names are just |
1st1 commentedJan 20, 2018
Right.
Yeah, the API consistency for synchronous CMs is great. To be honest I haven't thought about it until you highlighted it, but now I see it as a requirement!
+1. |
njsmith commentedJan 20, 2018
Please name the close method Regarding the bikeshed, I think my preference FWIW is to makeboth sync and async explicitly marked, so it's e.g. |
asvetlov commentedJan 20, 2018
@njsmith following this way we should add |
njsmith commentedJan 20, 2018 via email
The semantics around closing things are much more general and subtle thansending/receiving. Obviously send/recv are inherently semantically blockingoperations, but close() is non-blocking on a regular socket and blocking onan SSL socket. Plus in general there are just a lot more kinds of thingsthat get closed so it's useful to have general cross-type conventions here.You can't use contextlib.closing with an object whose close is async (orrather, you can if it calls the method 'close', and it will claim tosucceed without doing anything!). You *can* use async_generator.aclosing onany object with an async close, but it assumes that the method is calledaclose(), because it's not specific to async generators but that is one ofthe original use cases. In some cases you might want to provide both asynchronous close() and an async aclose() on the same object, e.g. if theobject's native close semantics are synchronous, but the way it's used alsoneeds to match some generic ABC. (Probably it's *better* it you can handlethis via composition – in trio, trio.socket.socket has a synchronousclose() while trio.SocketStream and trio.SocketListener have async aclose()methods – but sometimes stuff happens.) So personally at least I find itvery useful to have a conventional protocol for async close that doesn'toverlap with the existing convention for sync close. …On Jan 20, 2018 04:42, "Andrew Svetlov" ***@***.***> wrote:@njsmith <https://github.com/njsmith> following this way we should add a prefix to all async functions everywhere: aclose(), asend(), arecv() etc. The logic is weird, isn't it? I don't see how context manager is related to generator. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#4790 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAlOaDSFF_kyr6I6me86pr7c6e5W8fUjks5tMd8zgaJpZM4Q85Ru> . |
ncoghlan commentedJan 20, 2018 • edited
Loading Uh oh!
There was an error while loading.Please reload this page.
edited
Uh oh!
There was an error while loading.Please reload this page.
@njsmith Good point regarding close() vs aclose() - I'd forgotten that part of PEP 525. Given that, I agree the method on AsyncExitStack should aclose(), and the synchronous close API should just be missing entirely (as with the other synchronous context management methods). I'm not especially worried about folks having to sprinkle extra "async" mentions through their code to get async context managers to work as expected - that's the default language wide. For example, even though you've written By contrast, I'm definitelynot keen on having the correctness of code like This does mean that |
1st1 commentedJan 20, 2018 • edited
Loading Uh oh!
There was an error while loading.Please reload this page.
edited
Uh oh!
There was an error while loading.Please reload this page.
I agree with everything Nick said here. |
njsmith commentedJan 21, 2018
👍 |
ncoghlan left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
This version looks good to me, but leaving to@1st1 to merge after his review :)
ncoghlan commentedJan 23, 2018
The Appveyor failures look unrelated to me - closing & reopening to check if they're transient errors. |
1st1 commentedJan 23, 2018
I'll take a look today/tomorrow, thanks Nick! |
Kentzo commentedJan 23, 2018
@ncoghlan What do you think about adding |
ncoghlan commentedJan 23, 2018
@Kentzo Worth considering as a separate issue for 3.8, but I wouldn't rush into it for 3.7. |
1st1 left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
There're multiple formatting issues with the PR. Please fix the code to follow PEP 8. Reformat docstrings to have a short first line ended with a period, optionally followed by an empty line with more paragraphs.
Most importantly, I don't like the design of_shutdown_loop. I think we shouldn't use generator.send/throw here, I'd prefer to copy/paste the code which will make the code easier to read and maintain (generators are complicated!)
/cc@ncoghlan
Doc/library/contextlib.rst Outdated
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
Please trim the lines in your to be max 79 chars wide.
Doc/library/contextlib.rst Outdated
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
I'd rewrite to:
An :ref:`asynchronous context manager <async-context-managers>`, similarto :class:`ExitStack`, that supports combining both synchronous and asynchronous context managers, as well as having coroutines for cleanup logic.Doc/library/contextlib.rst Outdated
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
Same as -> Similar to
Doc/library/contextlib.rst Outdated
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
"or a coroutine function" -> "or a coroutine".
Doc/library/contextlib.rst Outdated
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
ditto
Lib/contextlib.py Outdated
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
First line of the docstring should not be empty. Reformat to
"""A base class for ExitStack and AsyncExitStack."""Lib/contextlib.py Outdated
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
Please keep and empty line after the first line of the docstring. I also recommend you to readhttps://www.python.org/dev/peps/pep-0257/.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
@1st1 In that PEP there is no empty line after a doctstring insidedef. Should I add this extra empty line for all functions in my PR?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
Ilya, I mean this:
deffoo():"""First line A paragraph """code
should be formatted as this
deffoo():"""First line. A paragraph. """code
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
The first line should be a complete sentence.
I'm not making these rules up -- there're tons of tools out there that expect this specific code style to extract/parse docstrings.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
Ah I see.
Lib/contextlib.py Outdated
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
I don't like using generator.send/throw for this. I'd better copy/paste the code. Generators have a pretty complicated semantics (especially around closing) and sometimes have issues with keeping exceptions' tracebacks in tact.
Lib/contextlib.py Outdated
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
Put periods. at the end of all your sentences. Throughout the patch.
Lib/contextlib.py Outdated
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
Tabulation is off and many lines are longer than 79. Please make sure that your PR follows PEP 8.
bedevere-bot commentedJan 23, 2018
A Python core developer has requested some changes be made to your pull request before we can consider merging it. If you could please address their requests along with any other requests in other reviews from core developers that would be appreciated. Once you have made the requested changes, please leave a comment on this pull request containing the phrase |
Kentzo commentedJan 24, 2018
I have made the requested changes; please review again |
bedevere-bot commentedJan 24, 2018
1st1 commentedJan 25, 2018
Merged. Thanks so much Ilya! |
Uh oh!
There was an error while loading.Please reload this page.
This change is entirely based on the work of@thehesiod discussed athttps://bugs.python.org/issue29302
I only changed implementation of
__aexit__:to:
and added tests.
https://bugs.python.org/issue29302