If my voice counts, I am definitely +1 on havingrun in 3.6

+1, definitely!

Aboutasyncio.forever(), wouldn't it be simpler to have a wrapper aroundloop.add_signal_handler(SIGINT, ...)? For instance:

asyncdefwait_for_interrupt():loop=asyncio.get_event_loop()future=loop.create_future()loop.add_signal_handler(signal.SIGINT,future.set_result,None)try:awaitfuturefinally:loop.remove_signal_handler(signal.SIGINT)

+1, LGTM
Do we need tests? I pretty sure we do.

About asyncio.forever(), wouldn't it be simpler to have a wrapper around loop.add_signal_handler(SIGINT, ...)? For instance:

This would work to support signals. I wantasyncio.forever() to supporttry..finally blocks regardless of what stopped the loop.

try:awaitasyncio.forever()finally:# important cleanup code code

TBH I don't want to distract ourselves withasyncio.forever() design. That stuff will require aseparate discussion.

Do we need tests? I pretty sure we do.

For sure. As I explained in the first message, if Guido is in favour of the idea I'll add tests/docs/etc.

Sure, add this. I don't have time for a review though.

I've experimented a little bit, and it turns out that it's not that hard to implement theasyncio.forever() coroutine. The idea is to add a Future object thatloop.run_forever() creates and sets the result to before returning.

With the latest commit it's possible to write coroutines like this:

asyncdeffoo():print('hi')try:awaitasyncio.forever()exceptKeyboardInterrupt:awaitasyncio.sleep(1)print('bye')asyncio.run(foo())

The change modifiesTask._wakeup to handleBaseException (that is a safe and backwards compatible thing to do), and adds a new low-level API method to the loop:get_forever_future().

So forever is just sleep infinity? Why do we need that?

It's similar but not the same. It's designed to replace uses ofloop.run_forever(), moving the program cleanup logic into the main coroutine.

It allows to safely transform this:

loop=asyncio.get_event_loop()server=loop.run_until_complete(asyncio.start_server(handle_echo,'127.0.0.1',8888,loop=loop))try:loop.run_forever()exceptKeyboardInterrupt:pass# Close the serverserver.close()loop.run_until_complete(server.wait_closed())loop.close()

into this:

asyncdefmain():server=awaitasyncio.start_server(handle_echo,'127.0.0.1',8888)try:awaitasyncio.forever()exceptKeyboardInterrupt:passserver.close()awaitserver.wait_closed()asyncio.run(main())

The former example that usesrun_forever andrun_until_complete is incomplete, btw. It doesn't usetry..finally to guarantee theloop.close() call, and it doesn't shutdown generators. The latter example that usesasyncio.run() does all of that.

The key difference from 'sleep(inf)is that you can usetry..except..finallyaroundawait forever()to properly cleanup whenKeyboardInterrupt` occurs or loop is stopped.

Having all program bootstrap logic defined in one coroutine is easier than usingloop.run_until_complete(), passing the loop around, callingloop.run_forever, and later cleaning up the loop.

asyncio.run() would be useful even withoutasyncio.forever(). If you only open connections or spawn subrocesses, you can use it. But if you have a server, or a number of other coroutines that run in "background", you'd need something to await on until the program is done.

If you don't see any pitfalls withasyncio.forever() then I think we have a huge win here. Essentially, event loop then becomes a low-level API that will be recommended for "expert" users. We won't even need to bring it up in asyncio tutorials/examples.

A few comments aboutasyncio.run():

• Isn't restrictingasyncio.run() to the main thread a bit extreme? I guess it might help avoiding some issues though, e.g subprocess handling on unix systems.

• About the clean-up, would it make sense to run the loop once, to get rid of possible pending callbacks? I remember I had to do that a few times to avoid some warnings.

• Shouldn't the current loop be saved and restored in thefinally clause? The current implementation can lead to surprising results:

  loop = asyncio.get_event_loop() # Fine!asyncio.run(some_coro())        # Run coroutineloop = asyncio.get_event_loop() # Raise a RuntimeError

@1st1
Like you said earlier,asyncio.forever() shouldn't distract us from gettingasyncio.run merged, so I'll happily move my comments if you happen to make another PR/issue specially forasyncio.forever(). Anyway, here are my two cents:

I'm not sure it makes sense to have an asynchronous equivalent toloop.run_forever.

• loop.run_forever: run until the loop is stopped
• await asyncio.forever: wait until the loop is stopped

But if the loop is stopped, how can the coroutine keep running? In the current implementation, it is restored by running the loop a second time. Soasyncio.forever() actually means "wait until the loop isrestarted", which is a bit confusing in my opinion. That also meansloop.stop() might be used to notifyasyncio.forever(), which is also quite weird.

I would argue that await_for_interrupt() coroutine is actually enough to cover most of theloop.run_forever() use case. Whenasyncio.run() is used, it should be safe to assume that the loop doesn't stop until the main coroutine is completed.

@gvanrossum

-1. Something that's only meant to be used at the top level doesn't deserve
to be a coroutine IMO.

Normally yes, I agree. Although I don't think this argument fully applies to this particular use case.

The idea is to add APIs to make it possible to move the application bootstrap logic (the "main" function") into a coroutine. The design ofasyncio.start_server() andloop.create_server() API requires you to useloop.run_forever() function to properly cleanup.

asyncio.run() solves a lot of cases, but without something likeasyncio.forever() it cannot help you when your application starts a server. And servers might be the majority of asyncio programs out there. Please take a look at the code example a couple of paragraphs below.

And even if we have forever() I think you should be able to get the same
behavior (relative to KeyboardInterrupt) with sleep(100000).

I see your point. Unfortunately it's not possible to implement this behaviour inasyncio.sleep, so we'd definitely need a better name forforever().

What if we renameforever() tointerrupted():

asyncdefmain():server=awaitasyncio.start_server(handle_echo,'127.0.0.1',8888)try:awaitasyncio.interrupted()finally:server.close()awaitserver.wait_closed()asyncio.run(main())

@vxgmichel

• Isn't restricting asyncio.run() to the main thread a bit extreme? I guess it might help avoiding some issues though, e.g subprocess handling on unix systems.

The function is supposed be used to launch your main program coroutine and we recommend to use asyncio in the main thread. And yes, subprocesses don't properly work when the loop isn't running in the main thread. I don't think that lifting this restriction would help anyone to be honest.

• About the clean-up, would it make sense to run the loop once, to get rid of possible pending callbacks? I remember I had to do that a few times to avoid some warnings.

Well, we runloop.shutdown_asyncgens() before closing, isn't that enough? I don't want to add another props for that, as it's just masking bugs and lack of some APIs in asyncio.

• Shouldn't the current loop be saved and restored in the finally clause? The current implementation can lead to surprising results:

Can't do that.asyncio.get_event_loop behaves in weird ways with the default policy:

• It will be raising aRuntimeError(There is no current event loop in thread) after first call toasyncio.set_event_loop(None).
• I have no idea if the event loop thatasyncio.get_event_loop returns is new or it was used before. Restoring a loop that was just created doesn't make any sense.

I'm -1 on all three.

I'm not sure it makes sense to have an asynchronous equivalent to loop.run_forever.

• loop.run_forever: run until the loop is stopped
• await asyncio.forever: wait until the loop is stopped
  But if the loop is stopped, how can the coroutine keep running?

It's up to the documentation -- the idea is thatforever() wakes up when the loop is interrupted by aBaseException or was stopped byloop.stop(). It then allows you to cleanup your resources.wait_for_interrupt() doesn't cover that -- it only coversKeyboardInterrupt.

Please study my example code in this comment and in#465 (comment).

+1 for keeping.run() only for main thread.
Also I would like to have.interrupt().@1st1 has described the need very well I believe.
Yep,forever was ambiguous name butinterrupt() sounds very clean for me.

That's also ambiguous (is it a verb or a noun?). And the issue of whether

I think I came up with a solution, please see below.

it would just be a shorthand for sleep(inf) is still unsettled.

We can't fixsleep() to support this. We can have N coroutines awaiting onsleep(inf) and the user presses ^C: which coroutine should we dispatch theKeyboardInterrupt to? Or maybe we have a server, and ^C happens in one of its socket read callbacks -- we can't dispatch that to sleep coroutines either.

Anyways, I think we can modifyasyncio.run() to accept asynchronous generators (in addition to regular coroutines). The closest analogy is@contextlib.contextmanager decorator.

With that we can have this:

asyncdefmain():server=awaitasyncio.start_server(handle_echo,'127.0.0.1',8888)try:yieldfinally:server.close()awaitserver.wait_closed()asyncio.run(main())

I think this is a great idea because:

1. No need forforever() or some other obscure coroutine.
2. We don't need to modifyloop.run_forever() or to add any new APIs to the loop.
3. Users are already familiar with the concept because of contextlib.

Theyield in this context means literally "yield control to the loop" from this "main" coroutine. If an exception happens or loop is stopped, please execute my cleanup code.

That's also ambiguous (is it a verb or a noun?)

I think it should rather beinterrupted(), as in Yury's server example above.

No need for forever() or some other obscure coroutine.

I like this, disregard my previous comment.

The latest patch is really great.
Using regular coroutine for client code and something likecontextlib.contextmanager for server ones fits pretty clean into my mind.

IIRC people are currently using a bare yield as the equivalent of sleep(0)
-- just bounce to the loop, run all callbacks, and then continue. How can
italso mean wait until the loop exits?

I really am very concerned that we're going to break things at the very
last moment in a panic response to the critique from Nathaniel Smith.

Also frankly run_in_executor() is a pretty clumsy API due to the initial parameter that is usually None. Is it really important enough to have this? (Again, I worry that this is a panic response rather than something well thought out.)

IIRC people are currently using a bare yield as the equivalent of sleep(0)
-- just bounce to the loop, run all callbacks, and then continue. How can
italso mean wait until the loop exits?

But they use it in old style generator-based coroutines:

@coroutinedeffoo():yield# this is a NOP yield

asyncdefmain():try:yield# <- this is a yield from an async gen, something new in 3.6

Asynchronous generators can do any kind of yields they want -- there is no backwards compatibility issue here.

In fact, I'm thinking about makingasyncio.run() function Python 3.6 only, and clearly documenting both use cases.

I really am very concerned that we're going to break things at the very
last moment in a panic response to the critique from Nathaniel Smith.

I wouldn't say that it's Nathaniel's post that caused this PR. I've been unhappy about the loop for a long time. I think I first proposed to fixget_event_loop 2 years ago.

Also frankly run_in_executor() is a pretty clumsy API due to the initial parameter that is usually None. Is it really important enough to have this? (Again, I worry that this is a panic response rather than something well thought out.)

Yes, I've been thinking aboutrun_in_executor too. I'll just drop it from this PR.

Oh, it's async generators. Too subtle.

Wedid fix get_event_loop(), and we're all happy with that. Can we just stop now please?

@1st1

I'm not sure we need to push more features to asyncio.run. So far I see two options:

I would really prefer option 2: two separate functions --run for coroutines andrun_forever for asyncgen's. I believe this could avoid confusion with old style coroutines mentioned by Guido.

I've modified this PR to add just two functions:

• asyncio.run() to run a coroutine:

  asyncdefmain():awaitasyncio.sleep(1)print('hello')asyncio.run(main())

• asyncio.run_forever() to run asyncio servers etc:

  asyncdefmain():server=awaitasyncio.start_server(...)try:yield# <- run the loop foreverfinally:server.close()awaitserver.wait_closed()asyncio.run_forever(main())

This PR also adds unittests. FWIW I used a custom asyncio policy to control the loop that the new functions use during tests, and it worked great.

Thank you Yury, it looks great! I left few small comments concerning error messages.

Also I think maybe it is worth adding a note somewhere in docs for people who are already familiar withloop.run_until_complete andloop.run_forever thatrun is a wrapper aroundloop.run_until_complete that takes care of set-up and clean-up; andrun_forever is a wrapper aroundloop.run_forever that adds the same set-up and clean-up logic around the logic provided by user in context-manager-like asyncgen?

(This is obvious if one looks at the code, but otherwise it might be not clear why such design was chosen.)

Also I think maybe it is worth adding a note somewhere in docs [..]

Sure, we'll update the docs!

We have very long discussion here.
As I see we have a preliminary agreement that.run() is good addition.
After that@1st1 has remembered about server side code which usually usesrun_forever.
The first proposal for accepting a coroutine for client code and async context manager was working but not perfect solution.
Splitting it intorun andrun_forever make the API clean and obvious.
I'm +1 for the patch.

3.6b4 is scheduled for tomorrow, if there is a chance this goes into 3.6, then it probably makes sense to merge this before that time.

Can this PR be moved topython/cpython?

Can this PR be moved to python/cpython?

IIUC, Yury is working on a PEP now that will cover the features in this PR.

@ilevkivskyi on your example above on the exception I would actually name the special function differently.

instead of :

defserve(mgr):# Set-up looploop.run_until_complete(mgr.__aenter__())try:loop.run_forever()except:result=loop.run_until_complete(mgr.__aexit__(*sys.exc_info()))else:result=loop.run_until_complete(mgr.__aexit__(None,None,None))# Careful clean-up

I would do:

defserve(mgr):# Set-up looploop.run_until_complete(mgr.__aenter__())try:loop.run_forever()except:result=loop.run_until_complete(mgr.__on_error__(*sys.exc_info()))else:result=loop.run_until_complete(mgr.__aexit__(None,None,None))# Careful clean-up

So that way if they do not have or define an__aexit__ that can take in exception messages.

Or another way is to have some sort of decorator that would register an function with asyncio that would handle the exception message so that way asyncio would not error again if they do not have__on_error__ in the type or object.