Movatterモバイル変換


[0]ホーム

URL:


Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

Sign up
Appearance settings

bpo-29302: Implement contextlib.AsyncExitStack.#4790

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to ourterms of service andprivacy statement. We’ll occasionally send you account related emails.

Already on GitHub?Sign in to your account

Merged
1st1 merged 1 commit intopython:masterfromGreatFruitOmsk:bpo-29302
Jan 25, 2018

Conversation

@Kentzo
Copy link
Contributor

@KentzoKentzo commentedDec 11, 2017
edited by bedevere-bot
Loading

This change is entirely based on the work of@thehesiod discussed athttps://bugs.python.org/issue29302

I only changed implementation of__aexit__:

exceptBaseExceptionase:gen.throw(e)

to:

except:gen.throw(*sys.exc_info())

and added tests.

https://bugs.python.org/issue29302

thehesiod reacted with thumbs up emoji
@thehesiod
Copy link
Contributor

thanks@Kentzo !

@asvetlovasvetlov requested a review from1st1December 11, 2017 07:16
@KentzoKentzo changed the titlebpo-29302: Implement contextlib.AsyncContextManager.bpo-29302: Implement contextlib.AsyncExitStack.Dec 11, 2017
@KentzoKentzoforce-pushed thebpo-29302 branch 2 times, most recently from4374de3 to60000cdCompareDecember 11, 2017 23:17
@1st11st1 requested a review fromncoghlanDecember 11, 2017 23:35
@ncoghlan
Copy link
Contributor

ncoghlan commentedDec 14, 2017 via email

@1st1 I'd like to take a further look at this one before we merge it, butwon't have time until Jan 6th or so (just before the final alpha). Doesthat timeline work for you?

@ncoghlan
Copy link
Contributor

ncoghlan commentedDec 14, 2017 via email

The alternative plan would be to merge this version based on the existingreviews, and if I have any subsequent changes I want to make, I can do thatany time before the first beta. That's probably a better approach, sinceit's more resilient against my being otherwise occupied in the first weekof January.

@1st1
Copy link
Member

@1st1 I'd like to take a further look at this one before we merge it, but
won't have time until Jan 6th or so (just before the final alpha). Does
that timeline work for you?

Sure, I think this is worth another look so let's wait. Let's just make sure we don't forget about this before the feature freeze.

Copy link
Contributor

@ncoghlanncoghlan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

I like the patch overall, but there's one design concern I have, as well as a backwards compatibility concern:

  • for the backwards compatibility issue: calls using keyword arguments mean we can't arbitrarily change parameter names in public APIs
  • for the design question,implicitly supporting synchronous context managers in the asynchronous exit stack seems questionable to me.

The design question is one@1st1 and I talked about on the issue tracker (seehttps://bugs.python.org/issue29302#msg288735 and the next couple of comments), and after seeing the code in PR form, Idefinitely want to go with the more explicit API:

  • add separate*_sync variants of all the callback registration APIs in the async exit stack variant rather than trying to guess the user's intent based on the type of the object they passed in (if guessing proves desirable, we can add it later, but if we start with it, we're likely stuck with it even if it proves confusing)
  • avoid checking for awaitables when unwinding the stack in the async case (either add an awaitable wrapper to the synchronous callbacks as I suggest in this review, or else store a 2-tuple as@1st1 suggested on the tracker)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

Folks may be callingstack.push(exit=obj), so we shouldn't gratuitously change parameter names.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

This introduces a dependency I'd prefer to avoid -inspect pulls in alot of other modules, andcontextlib is used frequently enough that we should aim to minimise its dependency tree.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

This comment is oddly placed, and doesn't seem especially accurate - the check is being made against__aexit__, not__aenter__.

More generally, implicitly interleaving synchronous with asynchronous context managers seems dubious to me - while it might be a good idea to allow it, doing it automagically rather than explicitly seems like a recipe for confusion when reading and debugging code.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

If we do decide to allow synchronous context managers in asynchronous exit stacks, I'd suggest wrapping their exit callbacks in an awaitable, so this check can be avoided.

Copy link
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

inspect is also used by_create_exit_wrapper and_create_cb_wrapper.

@bedevere-bot
Copy link

A Python core developer has requested some changes be made to your pull request before we can consider merging it. If you could please address their requests along with any other requests in other reviews from core developers that would be appreciated.

Once you have made the requested changes, please leave a comment on this pull request containing the phraseI have made the requested changes; please review again. I will then notify any core developers who have left a review that you're ready for them to take another look at this pull request.

@1st1
Copy link
Member

1st1 commentedJan 20, 2018
edited
Loading

Couple of thoughts:

  1. I think thatAsyncExitStack must support both sync and async context managers, otherwise it would be impossible to express code like below:
withFoo:asyncwithBar:withHam:            ...
  1. I'd it like to haveenter_context methods for sync context managers, andenter_async_context for async context managers. This is explicit and verbose, yes, but it handles the weird case when an object implementsboth sync- and async- context managers. Python allows to do that, so we should handle this case instead of guessing.

  2. +1 once for removinginspect.isawaitable call. Instead, we should push tuples like(is_sync, callback) to_exit_callbacks. Otherwise if someone submits a callable that for whatever reason implements__await__, usingisawaitable would result in a wrong guess and the callable would be awaited instead of being called.

@1st1
Copy link
Member

Also, we need to get this merged before Jan 29.@Kentzo if you don't have time to work on this in the next few days I can handle this PR myself.

@ncoghlan
Copy link
Contributor

ncoghlan commentedJan 20, 2018
edited
Loading

@1st1 So the differences between AsyncExitStack and ExitStack would be:

  1. close() becomes a coroutine instead of a synchronous method
  2. it defines__aenter__ and__aexit__ instead of__enter__ and__exit__
  3. it has 3 new methods for callback registration (one coroutine,enter_async_context, and two synchronous methods,push_async_exit andpush_async_callback)

I like that, since it means registering synchronous context managers and callbacks stays the same regardless of which kind of stack you're using.

A note regarding the method names: the synchronous method names are justpush andcallback. I think those are slightly problematic, since they're bothpush operations (making the first one ambiguous), and the latter isn't a verb (which is dubious for a method name). Sincepush_async andcallback_async aren't clear on whether the operation itself is asynchronous, I see value in switching to the more explicit method names before adding theasync qualifier:push_async_exit,push_async_callback.

@1st1
Copy link
Member

@1st1 So the differences between AsyncExitStack and ExitStack would be: [..]

Right.

I like that, since it means registering synchronous context managers and callbacks stays the same regardless of which kind of stack you're using.

Yeah, the API consistency for synchronous CMs is great. To be honest I haven't thought about it until you highlighted it, but now I see it as a requirement!

Since push_async and callback_async aren't clear on whether the operation itself is asynchronous, I see value in switching to the more explicit method names before adding the async qualifier: push_async_exit, push_async_callback.

+1.

@njsmith
Copy link
Contributor

Please name the close methodaclose, for consistency with async generators. (This will also makeAsyncExitStack atrio.abc.AsyncResource so long as all the items in it are themselvesAsyncResources. But that's just becauseAsyncResource was designed to be consistent with async generators :-).)

Regarding the bikeshed, I think my preference FWIW is to makeboth sync and async explicitly marked, so it's e.g.enter_sync_context andenter_async_context, and there is noenter_context. This adds a small amount of hassle in the case where you have an existingExitStack and then realize you want to mix in some async contexts as well, because you have to update existing calls. But it makes things much less confusing when you start out thinking "hmm, I have some async contexts that I need to put in a stack", so you make anAsyncExitStack and then are confused thatAsyncExitStack.enter_context wants a synchronous context.

@asvetlov
Copy link
Contributor

@njsmith following this way we should adda prefix to all async functions everywhere:aclose(),asend(),arecv() etc.
The logic is weird, isn't it? I don't see how context manager is related to generator.

@njsmith
Copy link
Contributor

njsmith commentedJan 20, 2018 via email

The semantics around closing things are much more general and subtle thansending/receiving. Obviously send/recv are inherently semantically blockingoperations, but close() is non-blocking on a regular socket and blocking onan SSL socket. Plus in general there are just a lot more kinds of thingsthat get closed so it's useful to have general cross-type conventions here.You can't use contextlib.closing with an object whose close is async (orrather, you can if it calls the method 'close', and it will claim tosucceed without doing anything!). You *can* use async_generator.aclosing onany object with an async close, but it assumes that the method is calledaclose(), because it's not specific to async generators but that is one ofthe original use cases. In some cases you might want to provide both asynchronous close() and an async aclose() on the same object, e.g. if theobject's native close semantics are synchronous, but the way it's used alsoneeds to match some generic ABC. (Probably it's *better* it you can handlethis via composition – in trio, trio.socket.socket has a synchronousclose() while trio.SocketStream and trio.SocketListener have async aclose()methods – but sometimes stuff happens.) So personally at least I find itvery useful to have a conventional protocol for async close that doesn'toverlap with the existing convention for sync close.
On Jan 20, 2018 04:42, "Andrew Svetlov" ***@***.***> wrote:@njsmith <https://github.com/njsmith> following this way we should add a prefix to all async functions everywhere: aclose(), asend(), arecv() etc. The logic is weird, isn't it? I don't see how context manager is related to generator. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#4790 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAlOaDSFF_kyr6I6me86pr7c6e5W8fUjks5tMd8zgaJpZM4Q85Ru> .

@ncoghlan
Copy link
Contributor

ncoghlan commentedJan 20, 2018
edited
Loading

@njsmith Good point regarding close() vs aclose() - I'd forgotten that part of PEP 525. Given that, I agree the method on AsyncExitStack should aclose(), and the synchronous close API should just be missing entirely (as with the other synchronous context management methods).

I'm not especially worried about folks having to sprinkle extra "async" mentions through their code to get async context managers to work as expected - that's the default language wide. For example, even though you've writtenstack = contextlib.AsyncExitStack(), youalso have to writeasync with stack: - you can't just writewith stack: the way you can with the synchronous version and have Python automatically figure out you must have wanted to use the async variant of the with statement.

By contrast, I'm definitelynot keen on having the correctness of code likef = stack.enter_context(open(file_of_interest)) depend on whether or not "stack" is a regular exit stack or an asynchronous one.

This does mean thatstack.push(async_cm_or_exit) andstack.callback(async_callback) may run into the "coroutine never awaited" scenario, but that's why we have the other issue about detecting unawaited coroutines. I'd also be amenable to having a check (potentially__debug__-only) inAsyncContextManager.__aexit__ itself for nominally synchronous callbacks returning awaitables, as that scenario is likely to inadvertently suppress exceptions as well (since a lot of awaitables are likely to be "True" by default).

@1st1
Copy link
Member

1st1 commentedJan 20, 2018
edited
Loading

I agree with everything Nick said here.

@njsmith
Copy link
Contributor

👍

Copy link
Contributor

@ncoghlanncoghlan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

This version looks good to me, but leaving to@1st1 to merge after his review :)

@ncoghlan
Copy link
Contributor

The Appveyor failures look unrelated to me - closing & reopening to check if they're transient errors.

@1st1
Copy link
Member

I'll take a look today/tomorrow, thanks Nick!

@Kentzo
Copy link
ContributorAuthor

@ncoghlan What do you think about addingpush_exit andpush_callback aliases for API symmetry?

@ncoghlan
Copy link
Contributor

@Kentzo Worth considering as a separate issue for 3.8, but I wouldn't rush into it for 3.7.

Copy link
Member

@1st11st1 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

There're multiple formatting issues with the PR. Please fix the code to follow PEP 8. Reformat docstrings to have a short first line ended with a period, optionally followed by an empty line with more paragraphs.

Most importantly, I don't like the design of_shutdown_loop. I think we shouldn't use generator.send/throw here, I'd prefer to copy/paste the code which will make the code easier to read and maintain (generators are complicated!)

/cc@ncoghlan

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

Please trim the lines in your to be max 79 chars wide.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

I'd rewrite to:

An :ref:`asynchronous context manager <async-context-managers>`, similarto :class:`ExitStack`, that supports combining both synchronous and asynchronous context managers, as well as having coroutines for cleanup logic.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

Same as -> Similar to

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

"or a coroutine function" -> "or a coroutine".

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

ditto

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

First line of the docstring should not be empty. Reformat to

"""A base class for ExitStack and AsyncExitStack."""

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

Please keep and empty line after the first line of the docstring. I also recommend you to readhttps://www.python.org/dev/peps/pep-0257/.

Copy link
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

@1st1 In that PEP there is no empty line after a doctstring insidedef. Should I add this extra empty line for all functions in my PR?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

Ilya, I mean this:

deffoo():"""First line    A paragraph    """code

should be formatted as this

deffoo():"""First line.    A paragraph.    """code

Copy link
Member

@1st11st1Jan 24, 2018
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

The first line should be a complete sentence.

I'm not making these rules up -- there're tons of tools out there that expect this specific code style to extract/parse docstrings.

Copy link
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

Ah I see.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

I don't like using generator.send/throw for this. I'd better copy/paste the code. Generators have a pretty complicated semantics (especially around closing) and sometimes have issues with keeping exceptions' tracebacks in tact.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

Put periods. at the end of all your sentences. Throughout the patch.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

Tabulation is off and many lines are longer than 79. Please make sure that your PR follows PEP 8.

@bedevere-bot
Copy link

A Python core developer has requested some changes be made to your pull request before we can consider merging it. If you could please address their requests along with any other requests in other reviews from core developers that would be appreciated.

Once you have made the requested changes, please leave a comment on this pull request containing the phraseI have made the requested changes; please review again. I will then notify any core developers who have left a review that you're ready for them to take another look at this pull request.

@Kentzo
Copy link
ContributorAuthor

I have made the requested changes; please review again

@bedevere-bot
Copy link

Thanks for making the requested changes!

@1st1,@ncoghlan: please review the changes made to this pull request.

@1st1
Copy link
Member

Merged. Thanks so much Ilya!

Sign up for freeto join this conversation on GitHub. Already have an account?Sign in to comment

Reviewers

@1st11st11st1 approved these changes

@ncoghlanncoghlanncoghlan approved these changes

Assignees

No one assigned

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

8 participants

@Kentzo@thehesiod@ncoghlan@1st1@bedevere-bot@njsmith@asvetlov@the-knights-who-say-ni

[8]ページ先頭

©2009-2025 Movatter.jp