This is a proposal for asynchronous I/O in Python 3, starting atPython 3.3. Consider this the concrete proposal that is missing fromPEP 3153. The proposal includes a pluggable event loop, transport andprotocol abstractions similar to those in Twisted, and a higher-levelscheduler based onyieldfrom (PEP 380). The proposed packagename isasyncio.
A reference implementation exists under the code name Tulip. TheTulip repo is linked from the References section at the end. Packagesbased on this repo will be provided on PyPI (see References) to enableusing theasyncio package with Python 3.3 installations.
As of October 20th 2013, theasyncio package has been checked intothe Python 3.4 repository and released with Python 3.4-alpha-4, with“provisional” API status. This is an expression of confidence andintended to increase early feedback on the API, and not intended toforce acceptance of the PEP. The expectation is that the package willkeep provisional status in Python 3.4 and progress to final status inPython 3.5. Development continues to occur primarily in the Tuliprepo, with changes occasionally merged into the CPython repo.
Python 3.3 is required for many of the proposed features. Thereference implementation (Tulip) requires no new language or standardlibrary features beyond Python 3.3, no third-party modules orpackages, and no C code, except for the (optional) IOCP support onWindows.
The specification here lives in a new top-level package,asyncio.Different components live in separate submodules of the package. Thepackage will import common APIs from their respective submodules andmake them available as package attributes (similar to the way theemail package works). For such common APIs, the name of the submodulethat actually defines them is not part of the specification. Lesscommon APIs may have to explicitly be imported from their respectivesubmodule, and in this case the submodule name is part of thespecification.
Classes and functions defined without a submodule name are assumed tolive in the namespace of the top-level package. (But do not confusethese with methods of various classes, which for brevity are also usedwithout a namespace prefix in certain contexts.)
The event loop is the place where most interoperability occurs. Itshould be easy for (Python 3.3 ports of) frameworks like Twisted,Tornado, or even gevents to either adapt the default event loopimplementation to their needs using a lightweight adapter or proxy, orto replace the default event loop implementation with an adaptation oftheir own event loop implementation. (Some frameworks, like Twisted,have multiple event loop implementations. This should not be aproblem since these all have the same interface.)
In most cases it should be possible for two different third-partyframeworks to interoperate, either by sharing the default event loopimplementation (each using its own adapter), or by sharing the eventloop implementation of either framework. In the latter case twolevels of adaptation would occur (from framework A’s event loop to thestandard event loop interface, and from there to framework B’s eventloop). Which event loop implementation is used should be undercontrol of the main program (though a default policy for event loopselection is provided).
For this interoperability to be effective, the preferred direction ofadaptation in third party frameworks is to keep the default event loopand adapt it to the framework’s API. Ideally all third partyframeworks would give up their own event loop implementation in favorof the standard implementation. But not all frameworks may besatisfied with the functionality provided by the standardimplementation.
In order to support both directions of adaptation, two separate APIsare specified:
An event loop implementation may provide additional methods andguarantees, as long as these are called out in the documentation asnon-standard. An event loop implementation may also leave certainmethods unimplemented if they cannot be implemented in the givenenvironment; however, such deviations from the standard API should beconsidered only as a last resort, and only if the platform orenvironment forces the issue. (An example would be a platform wherethere is a system event loop that cannot be started or stopped; see“Embedded Event Loops” below.)
The event loop API does not depend onawait/yieldfrom. Rather, it usesa combination of callbacks, additional interfaces (transports andprotocols), and Futures. The latter are similar to those defined inPEP 3148, but have a different implementation and are not tied tothreads. In particular, theresult() method raises an exceptioninstead of blocking when a result is not yet ready; the user isexpected to use callbacks (orawait/yieldfrom) to wait for the result.
All event loop methods specified as returning a coroutine are allowedto return either a Future or a coroutine, at the implementation’schoice (the standard implementation always returns coroutines). Allevent loop methods documented as accepting coroutine argumentsmustaccept both Futures and coroutines for such arguments. (A conveniencefunction,ensure_future(), exists to convert an argument that is either acoroutine or a Future into a Future.)
For users (like myself) who don’t like using callbacks, a scheduler isprovided for writing asynchronous I/O code as coroutines using the PEP380yieldfrom orPEP 492await expressions.The scheduler is not pluggable;pluggability occurs at the event loop level, and the standardscheduler implementation should work with any conforming event loopimplementation. (In fact this is an important litmus test forconforming implementations.)
For interoperability between code written using coroutines and otherasync frameworks, the scheduler defines a Task class that behaves like aFuture. A framework that interoperates at the event loop level canwait for a Future to complete by adding a callback to the Future.Likewise, the scheduler offers an operation to suspend a coroutineuntil a callback is called.
If such a framework cannot use the Future and Task classes as-is, itmay reimplement theloop.create_future() andloop.create_task() methods. These should return objectsimplementing (a superset of) the Future/Task interfaces.
A less ambitious framework may just call theloop.set_task_factory() to replace the Task class withoutimplementing its own event loop.
The event loop API provides limited interoperability with threads:there is an API to submit a function to an executor (seePEP 3148)which returns a Future that is compatible with the event loop, andthere is a method to schedule a callback with an event loop fromanother thread in a thread-safe manner.
For those not familiar with Twisted, a quick explanation of therelationship between transports and protocols is in order. At thehighest level, the transport is concerned withhow bytes aretransmitted, while the protocol determineswhich bytes to transmit(and to some extent when).
A different way of saying the same thing: a transport is anabstraction for a socket (or similar I/O endpoint) while a protocol isan abstraction for an application, from the transport’s point of view.
Yet another view is simply that the transport and protocol interfacestogether define an abstract interface for using network I/O andinterprocess I/O.
There is almost always a 1:1 relationship between transport andprotocol objects: the protocol calls transport methods to send data,while the transport calls protocol methods to pass it data that hasbeen received. Neither transport nor protocol methods “block” – theyset events into motion and then return.
The most common type of transport is a bidirectional stream transport.It represents a pair of buffered streams (one in each direction) thateach transmit a sequence of bytes. The most common example of abidirectional stream transport is probably a TCP connection. Anothercommon example is an SSL/TLS connection. But there are some other thingsthat can be viewed this way, for example an SSH session or a pair ofUNIX pipes. Typically there aren’t many different transportimplementations, and most of them come with the event loopimplementation. However, there is no requirement that all transportsmust be created by calling an event loop method: a third party modulemay well implement a new transport and provide a constructor orfactory function for it that simply takes an event loop as an argumentor callsget_event_loop().
Note that transports don’t need to use sockets, not even if they useTCP – sockets are a platform-specific implementation detail.
A bidirectional stream transport has two “ends”: one end talks tothe network (or another process, or whatever low-level interface itwraps), and the other end talks to the protocol. The former useswhatever API is necessary to implement the transport; but theinterface between transport and protocol is standardized by this PEP.
A protocol can represent some kind of “application-level” protocolsuch as HTTP or SMTP; it can also implement an abstraction shared bymultiple protocols, or a whole application. A protocol’s primaryinterface is with the transport. While some popular protocols (andother abstractions) may have standard implementations, oftenapplications implement custom protocols. It also makes sense to havelibraries of useful third party protocol implementations that can bedownloaded and installed from PyPI.
There general notion of transport and protocol includes otherinterfaces, where the transport wraps some other communicationabstraction. Examples include interfaces for sending and receivingdatagrams (e.g. UDP), or a subprocess manager. The separation ofconcerns is the same as for bidirectional stream transports andprotocols, but the specific interface between transport and protocolis different in each case.
Details of the interfaces defined by the various standard types oftransports and protocols are given later.
Event loop management is controlled by an event loop policy, which isa global (per-process) object. There is a default policy, and an APIto change the policy. A policy defines the notion of context; apolicy manages a separate event loop per context. The defaultpolicy’s notion of context is defined as the current thread.
Certain platforms or programming frameworks may change the defaultpolicy to something more suitable to the expectations of the users ofthat platform or framework. Such platforms or frameworks mustdocument their policy and at what point during their initializationsequence the policy is set, in order to avoid undefined behavior whenmultiple active frameworks want to override the default policy.(See also “Embedded Event Loops” below.)
To get the event loop for current context, useget_event_loop().This returns an event loop object implementing the interface specifiedbelow, or raises an exception in case no event loop has been set forthe current context and the current policy does not specify to createone. It should never returnNone.
To set the event loop for the current context, useset_event_loop(event_loop), whereevent_loop is an event loopobject, i.e. an instance ofAbstractEventLoop, orNone.It is okay to set the current event loop toNone, inwhich case subsequent calls toget_event_loop() will raise anexception. This is useful for testing code that should not depend onthe existence of a default event loop.
It is expected thatget_event_loop() returns a different eventloop object depending on the context (in fact, this is the definitionof context). It may create a new event loop object if none is set andcreation is allowed by the policy. The default policy will create anew event loop only in the main thread (as defined by threading.py,which uses a special subclass for the main thread), and only ifget_event_loop() is called beforeset_event_loop() is evercalled. (To reset this state, reset the policy.) In other threads anevent loop must be explicitly set. Other policies may behavedifferently. Event loop by the default policy creation is lazy;i.e. the first call toget_event_loop() creates an event loopinstance if necessary and specified by the current policy.
For the benefit of unit tests and other special cases there’s a thirdpolicy function:new_event_loop(), which creates and returns a newevent loop object according to the policy’s default rules. To makethis the current event loop, you must callset_event_loop() withit.
To change the event loop policy, callset_event_loop_policy(policy), wherepolicy is an event looppolicy object orNone. If notNone, the policy object must bean instance ofAbstractEventLoopPolicy that defines methodsget_event_loop(),set_event_loop(loop) andnew_event_loop(), all behaving like the functions described above.
Passing a policy value ofNone restores the default event looppolicy (overriding the alternate default set by the platform orframework). The default event loop policy is an instance of the classDefaultEventLoopPolicy. The current event loop policy object canbe retrieved by callingget_event_loop_policy().
TBD: describe child watchers and UNIX quirks for subprocess processing.
It is possible to write code that uses an event loop without relyingon a global or per-thread default event loop. For this purpose, allAPIs that need access to the current event loop (and aren’t methods onan event class) take an optional keyword argument namedloop. Ifthis argument isNone or unspecified, such APIs will callget_event_loop() to get the default event loop, but if theloop keyword argument is set to an event loop object, they willuse that event loop, and pass it along to any other such APIs theycall. For example,Future(loop=my_loop) will create a Future tiedto the event loopmy_loop. When the default current event isNone, theloop keyword argument is effectively mandatory.
Note that an explicitly passed event loop must still belong to thecurrent thread; theloop keyword argument does not magicallychange the constraints on how an event loop can be used.
As usual in Python, all timeouts, intervals and delays are measured inseconds, and may be ints or floats. However, absolute times are notspecified as POSIX timestamps. The accuracy, precision and epoch ofthe clock are up to the implementation.
The default implementation usestime.monotonic(). Books could bewritten about the implications of this choice. Better read the docsfor the standard librarytime module.
On some platforms an event loop is provided by the system. Such aloop may already be running when the user code starts, and there maybe no way to stop or close it without exiting from the program. Inthis case, the methods for starting, stopping and closing the eventloop may not be implementable, andis_running() may always returnTrue.
There is no actual class namedEventLoop. There is anAbstractEventLoop class which defines all the methods withoutimplementations, and serves primarily as documentation. The followingconcrete classes are defined:
SelectorEventLoop is a concrete implementation of the full APIbased on theselectors module (new in Python 3.4). Theconstructor takes one optional argument, aselectors.Selectorobject. By default an instance ofselectors.DefaultSelector iscreated and used.ProactorEventLoop is a concrete implementation of the API exceptfor the I/O event handling and signal handling methods. It is onlydefined on Windows (or on other platforms which support a similarAPI for “overlapped I/O”). The constructor takes one optionalargument, aProactor object. By default an instance ofIocpProactor is created and used. (TheIocpProactor classis not specified by this PEP; it is just an implementationdetail of theProactorEventLoop class.)The methods of a conforming event loop are grouped into severalcategories. The first set of categories must be supported by allconforming event loop implementations, with the exception thatembedded event loops may not implement the methods for starting,stopping and closing. (However, a partially-conforming event loop isstill better than nothing. :-)
run_forever(),run_until_complete(),stop(),is_running(),close(),is_closed().call_soon(),call_later(),call_at(),time().call_soon_threadsafe(),run_in_executor(),set_default_executor().getaddrinfo(),getnameinfo().create_connection(),create_server(),create_datagram_endpoint().sock_recv(),sock_sendall(),sock_connect(),sock_accept().create_future(),create_task(),set_task_factory(),get_task_factory().get_exception_handler(),set_exception_handler(),default_exception_handler(),call_exception_handler().get_debug(),set_debug().The second set of categoriesmay be supported by conforming eventloop implementations. If not supported, they will raiseNotImplementedError. (In the default implementation,SelectorEventLoop on UNIX systems supports all of these;SelectorEventLoop on Windows supports the I/O event handlingcategory;ProactorEventLoop on Windows supports the pipes andsubprocess category.)
add_reader(),remove_reader(),add_writer(),remove_writer().connect_read_pipe(),connect_write_pipe(),subprocess_shell(),subprocess_exec().add_signal_handler(),remove_signal_handler().An (unclosed) event loop can be in one of two states: running orstopped. These methods deal with starting and stopping an event loop:
run_forever(). Runs the event loop untilstop() is called.This cannot be called when the event loop is already running. (Thishas a long name in part to avoid confusion with earlier versions ofthis PEP, whererun() had different behavior, in part becausethere are already too many APIs that have a method namedrun(),and in part because there shouldn’t be many places where this iscalled anyway.)run_until_complete(future). Runs the event loop until theFuture is done. If the Future is done, its result is returned, orits exception is raised. This cannot be called when the event loopis already running.The method creates a newTask object if theparameter is a coroutine.stop(). Stops the event loop as soon as it is convenient. Itis fine to restart the loop withrun_forever() orrun_until_complete() subsequently; no scheduled callbacks willbe lost if this is done. Note:stop() returns normally and thecurrent callback is allowed to continue. How soon after this pointthe event loop stops is up to the implementation, but the intentionis to stop short of polling for I/O, and not to run any callbacksscheduled in the future; the major freedom an implementation has ishow much of the “ready queue” (callbacks already scheduled withcall_soon()) it processes before stopping.is_running(). ReturnsTrue if the event loop is currentlyrunning,False if it is stopped.close(). Closes the event loop, releasing any resources it mayhold, such as the file descriptor used byepoll() orkqueue(), and the default executor. This should not be calledwhile the event loop is running. After it has been called the eventloop should not be used again. It may be called multiple times;subsequent calls are no-ops.is_closed(). ReturnsTrue if the event loop is closed,False otherwise. (Primarily intended for error reporting;please don’t implement functionality based on this method.)Callbacks associated with the same event loop are strictly serialized:one callback must finish before the next one will be called. This isan important guarantee: when two or more callbacks use or modifyshared state, each callback is guaranteed that while it is running, theshared state isn’t changed by another callback.
call_soon(callback,*args). This schedules a callback to becalled as soon as possible. Returns aHandle (see below)representing the callback, whosecancel() method can be used tocancel the callback. It guarantees that callbacks are called in theorder in which they were scheduled.call_later(delay,callback,*args). Arrange forcallback(*args) to be called approximatelydelay seconds inthe future, once, unless cancelled. Returns aHandle representingthe callback, whosecancel() method can be used to cancel thecallback. Callbacks scheduled in the past or at exactly the sametime will be called in an undefined order.call_at(when,callback,*args). This is likecall_later(),but the time is expressed as an absolute time. Returns a similarHandle. There is a simple equivalency:loop.call_later(delay,callback,*args) is the same asloop.call_at(loop.time()+delay,callback,*args).time(). Returns the current time according to the event loop’sclock. This may betime.time() ortime.monotonic() or someother system-specific clock, but it must return a float expressingthe time in units of approximately one second since some epoch.(No clock is perfect – seePEP 418.)Note: A previous version of this PEP defined a method namedcall_repeatedly(), which promised to call a callback at regularintervals. This has been withdrawn because the design of such afunction is overspecified. On the one hand, a simple timer loop caneasily be emulated using a callback that reschedules itself usingcall_later(); it is also easy to write coroutine containing a loopand asleep() call (a toplevel function in the module, see below).On the other hand, due to the complexities of accurate timekeepingthere are many traps and pitfalls here for the unaware (seePEP 418),and different use cases require different behavior in edge cases. Itis impossible to offer an API for this purpose that is bullet-proof inall cases, so it is deemed better to let application designers decidefor themselves what kind of timer loop to implement.
call_soon_threadsafe(callback,*args). Likecall_soon(callback,*args), but when called from another threadwhile the event loop is blocked waiting for I/O, unblocks the eventloop. Returns aHandle. This is theonly method that is safeto call from another thread. (To schedule a callback for a latertime in a threadsafe manner, you can useloop.call_soon_threadsafe(loop.call_later,when,callback,*args).) Note: this is not safe to call from a signal handler(since it may use locks). In fact, no API is signal-safe; if youwant to handle signals, useadd_signal_handler() describedbelow.run_in_executor(executor,callback,*args). Arrange to callcallback(*args) in an executor (seePEP 3148). Returns anasyncio.Future instance whose result on success is the returnvalue of that call. This is equivalent towrap_future(executor.submit(callback,*args)). IfexecutorisNone, the default executor set byset_default_executor()is used. If no default executor has been set yet, aThreadPoolExecutor with a default number of threads is createdand set as the default executor. (The default implementation uses5 threads in this case.)set_default_executor(executor). Set the default executor usedbyrun_in_executor(). The argument must be aPEP 3148Executor instance orNone, in order to reset the defaultexecutor.See also thewrap_future() function described in the section aboutFutures.
These methods are useful if you want to connect or bind a socket to anaddress without the risk of blocking for the name lookup. They areusually called implicitly bycreate_connection(),create_server() orcreate_datagram_endpoint().
getaddrinfo(host,port,family=0,type=0,proto=0,flags=0).Similar to thesocket.getaddrinfo() function but returns aFuture. The Future’s result on success will be a list of the sameformat as returned bysocket.getaddrinfo(), i.e. a list of(address_family,socket_type,socket_protocol,canonical_name,address) whereaddress is a 2-tuple(ipv4_address,port)for IPv4 addresses and a 4-tuple(ipv6_address,port,flow_info,scope_id) for IPv6 addresses. If thefamily argument is zeroor unspecified, the list returned may contain a mixture of IPv4 andIPv6 addresses; otherwise the addresses returned are constrained bythefamily value (similar forproto andflags). Thedefault implementation callssocket.getaddrinfo() usingrun_in_executor(), but other implementations may choose toimplement their own DNS lookup. The optional argumentsmust bespecified as keyword arguments.Note: implementations are allowed to implement a subset of the fullsocket.getaddrinfo() interface; e.g. they may not support symbolicport names, or they may ignore or incompletely implement thetype,proto andflags arguments. However, iftypeandproto are ignored, the argument values passed in should becopied unchanged into the return tuples’socket_type andsocket_protocol elements. (You can’t ignorefamily, sinceIPv4 and IPv6 addresses must be looked up differently. The onlypermissible values forfamily aresocket.AF_UNSPEC (0),socket.AF_INET andsocket.AF_INET6, and the latter only ifit is defined by the platform.)
getnameinfo(sockaddr,flags=0). Similar tosocket.getnameinfo() but returns a Future. The Future’s resulton success will be a tuple(host,port). Same implementationremarks as forgetaddrinfo().These are the high-level interfaces for managing internet connections.Their use is recommended over the corresponding lower-level interfacesbecause they abstract away the differences between selector-basedand proactor-based event loops.
Note that the client and server side of stream connections use thesame transport and protocol interface. However, datagram endpointsuse a different transport and protocol interface.
create_connection(protocol_factory,host,port,<options>).Creates a stream connection to a given internet host and port. Thisis a task that is typically called from the client side of theconnection. It creates an implementation-dependent bidirectionalstream Transport to represent the connection, then callsprotocol_factory() to instantiate (or retrieve) the user’sProtocol implementation, and finally ties the two together. (Seebelow for the definitions of Transport and Protocol.) The user’sProtocol implementation is created or retrieved by callingprotocol_factory() without arguments(*). The coroutine’s resulton success is the(transport,protocol) pair; if a failureprevents the creation of a successful connection, an appropriateexception will be raised. Note that when the coroutine completes,the protocol’sconnection_made() method has not yet been called;that will happen when the connection handshake is complete.(*) There is no requirement thatprotocol_factory is a class.If your protocol class needs to have specific arguments passed toits constructor, you can uselambda.You can also pass a triviallambda that returns a previouslyconstructed Protocol instance.
The <options> are all specified using optional keyword arguments:
ssl: PassTrue to create an SSL/TLS transport (by defaulta plain TCP transport is created). Or pass anssl.SSLContextobject to override the default SSL context object to be used. Ifa default context is created it is up to the implementation toconfigure reasonable defaults. The reference implementationcurrently usesPROTOCOL_SSLv23 and sets theOP_NO_SSLv2option, callsset_default_verify_paths() and setsverify_modetoCERT_REQUIRED. In addition, whenever the context (defaultor otherwise) specifies averify_mode ofCERT_REQUIRED orCERT_OPTIONAL, if a hostname is given, immediately after asuccessful handshakessl.match_hostname(peercert,hostname) iscalled, and if this raises an exception the connection is closed.(To avoid this behavior, pass in an SSL context that hasverify_mode set toCERT_NONE. But this means you are notsecure, and vulnerable to for example man-in-the-middle attacks.)family,proto,flags: Address family, protocol andflags to be passed through togetaddrinfo(). These alldefault to0, which means “not specified”. (The socket typeis alwaysSOCK_STREAM.) If any of these values are notspecified, thegetaddrinfo() method will choose appropriatevalues. Note:proto has nothing to do with the high-levelProtocol concept or theprotocol_factory argument.sock: An optional socket to be used instead of using thehost,port,family,proto andflagsarguments. If this is given,host andport must beexplicitly set toNone.local_addr: If given, a(host,port) tuple used to bindthe socket to locally. This is rarely needed but on multi-homedservers you occasionally need to force a connection to come from aspecific address. This is how you would do that. The host andport are looked up usinggetaddrinfo().server_hostname: This is only relevant when using SSL/TLS; itshould not be used whenssl is not set. Whenssl is set,this sets or overrides the hostname that will be verified. Bydefault the value of thehost argument is used. Ifhostis empty, there is no default and you must pass a value forserver_hostname. To disable hostname verification (which is aserious security risk) you must pass an empty string here and passanssl.SSLContext object whoseverify_mode is set tossl.CERT_NONE as thessl argument.create_server(protocol_factory,host,port,<options>).Enters a serving loop that accepts connections.This is a coroutine that completes once the serving loop is set upto serve. The return value is aServer object which can be usedto stop the serving loop in a controlled fashion (see below).Multiple sockets may be bound if the specified address allowsboth IPv4 and IPv6 connections.Each time a connection is accepted,protocol_factory is called without arguments(**) to create aProtocol, a bidirectional stream Transport is created to representthe network side of the connection, and the two are tied together bycallingprotocol.connection_made(transport).
(**) See previous footnote forcreate_connection(). However, sinceprotocol_factory() is called once for each new incomingconnection, it should return a new Protocol object each time it iscalled.
The <options> are all specified using optional keyword arguments:
ssl: Pass anssl.SSLContext object (or an object with thesame interface) to override the default SSL context object to beused. (Unlike forcreate_connection(), passingTrue doesnot make sense here – theSSLContext object is needed tospecify the certificate and key.)backlog: Backlog value to be passed to thelisten() call.The default is implementation-dependent; in the defaultimplementation the default value is100.reuse_address: Whether to set theSO_REUSEADDR option onthe socket. The default isTrue on UNIX,False onWindows.family,flags: Address family and flags to be passedthrough togetaddrinfo(). The family defaults toAF_UNSPEC; the flags default toAI_PASSIVE. (The sockettype is alwaysSOCK_STREAM; the socket protocol always set to0, to letgetaddrinfo() choose.)sock: An optional socket to be used instead of using thehost,port,family andflags arguments. If thisis given,host andport must be explicitly set toNone.create_datagram_endpoint(protocol_factory,local_addr=None,remote_addr=None,<options>). Creates an endpoint for sending andreceiving datagrams (typically UDP packets). Because of the natureof datagram traffic, there are no separate calls to set up clientand server side, since usually a single endpoint acts as both clientand server. This is a coroutine that returns a(transport,protocol) pair on success, or raises an exception on failure. Ifthe coroutine returns successfully, the transport will callcallbacks on the protocol whenever a datagram is received or thesocket is closed; it is up to the protocol to call methods on theprotocol to send datagrams. The transport returned is aDatagramTransport. The protocol returned is aDatagramProtocol. These are described later.Mandatory positional argument:
protocol_factory: A class or factory function that will becalled exactly once, without arguments, to construct the protocolobject to be returned. The interface between datagram transportand protocol is described below.Optional arguments that may be specified positionally or as keywordarguments:
local_addr: An optional tuple indicating the address to whichthe socket will be bound. If given this must be a(host,port) pair. It will be passed togetaddrinfo() to beresolved and the result will be passed to thebind() method ofthe socket created. Ifgetaddrinfo() returns more than oneaddress, they will be tried in turn. If omitted, nobind()call will be made.remote_addr: An optional tuple indicating the address to whichthe socket will be “connected”. (Since there is no such thing asa datagram connection, this just specifies a default value for thedestination address of outgoing datagrams.) If given this must bea(host,port) pair. It will be passed togetaddrinfo()to be resolved and the result will be passed tosock_connect()together with the socket created. Ifgetaddrinfo() returnsmore than one address, they will be tried in turn. If omitted,nosock_connect() call will be made.The <options> are all specified using optional keyword arguments:
family,proto,flags: Address family, protocol andflags to be passed through togetaddrinfo(). These alldefault to0, which means “not specified”. (The socket typeis alwaysSOCK_DGRAM.) If any of these values are notspecified, thegetaddrinfo() method will choose appropriatevalues.Note that if bothlocal_addr andremote_addr are present,all combinations of local and remote addresses with matching addressfamily will be tried.
The following methods for doing async I/O on sockets are not forgeneral use. They are primarily meant for transport implementationsworking with IOCP through theProactorEventLoop class. However,they are easily implementable for other event loop types, so there isno reason not to require them. The socket argument has to be anon-blocking socket.
sock_recv(sock,n). Receive up ton bytes from socketsock. Returns a Future whose result on success will be abytes object.sock_sendall(sock,data). Send bytesdata to socketsock. Returns a Future whose result on success will beNone. Note: the name usessendall instead ofsend, toreflect that the semantics and signature of this method echo thoseof the standard library socket methodsendall() rather thansend().sock_connect(sock,address). Connect to the given address.Returns a Future whose result on success will beNone.sock_accept(sock). Accept a connection from a socket. Thesocket must be in listening mode and bound to an address. Returns aFuture whose result on success will be a tuple(conn,peer)whereconn is a connected non-blocking socket andpeer isthe peer address.These methods are primarily meant for transport implementationsworking with a selector. They are implemented bySelectorEventLoop but not byProactorEventLoop. Custom eventloop implementations may or may not implement them.
Thefd arguments below may be integer file descriptors, or“file-like” objects with afileno() method that wrap integer filedescriptors. Not all file-like objects or file descriptors areacceptable. Sockets (and socket file descriptors) are alwaysaccepted. On Windows no other types are supported. On UNIX, pipesand possibly tty devices are also supported, but disk files are not.Exactly which special file types are supported may vary by platformand per selector implementation. (Experimentally, there is at leastone kind of pseudo-tty on OS X that is supported byselect andpoll but not bykqueue: it is used by Emacs shell windows.)
add_reader(fd,callback,*args). Arrange forcallback(*args) to be called whenever file descriptorfd isdeemed ready for reading. Callingadd_reader() again for thesame file descriptor implies a call toremove_reader() for thesame file descriptor.add_writer(fd,callback,*args). Likeadd_reader(),but registers the callback for writing instead of for reading.remove_reader(fd). Cancels the current read callback for filedescriptorfd, if one is set. If no callback is currently setfor the file descriptor, this is a no-op and returnsFalse.Otherwise, it removes the callback arrangement and returnsTrue.remove_writer(fd). This is toadd_writer() asremove_reader() is toadd_reader().These methods are supported bySelectorEventLoop on UNIX andProactorEventLoop on Windows.
The transports and protocols used with pipes and subprocesses differfrom those used with regular stream connections. These are describedlater.
Each of the methods below has aprotocol_factory argument, similartocreate_connection(); this will be called exactly once, withoutarguments, to construct the protocol object to be returned.
Each method is a coroutine that returns a(transport,protocol)pair on success, or raises an exception on failure.
connect_read_pipe(protocol_factory,pipe): Create aunidrectional stream connection from a file-like object wrapping theread end of a UNIX pipe, which must be in non-blocking mode. Thetransport returned is aReadTransport.connect_write_pipe(protocol_factory,pipe): Create aunidrectional stream connection from a file-like object wrapping thewrite end of a UNIX pipe, which must be in non-blocking mode. Thetransport returned is aWriteTransport; it does not have anyread-related methods. The protocol returned is aBaseProtocol.subprocess_shell(protocol_factory,cmd,<options>): Create asubprocess fromcmd, which is a string using the platform’s“shell” syntax. This is similar to the standard librarysubprocess.Popen() class called withshell=True. Theremaining arguments and return value are described below.subprocess_exec(protocol_factory,*args,<options>): Create asubprocess from one or more string arguments, where the first stringspecifies the program to execute, and the remaining strings specifythe program’s arguments. (Thus, together the string arguments formthesys.argv value of the program, assuming it is a Pythonscript.) This is similar to the standard librarysubprocess.Popen() class called withshell=False and thelist of strings passed as the first argument; however, wherePopen() takes a single argument which is list of strings,subprocess_exec() takes multiple string arguments. Theremaining arguments and return value are described below.Apart from the way the program to execute is specified, the twosubprocess_*() methods behave the same. The transport returned isaSubprocessTransport which has a different interface than thecommon bidirectional stream transport. The protocol returned is aSubprocessProtocol which also has a custom interface.
The <options> are all specified using optional keyword arguments:
stdin: Either a file-like object representing the pipe to beconnected to the subprocess’s standard input stream usingconnect_write_pipe(), or the constantsubprocess.PIPE (thedefault). By default a new pipe will be created and connected.stdout: Either a file-like object representing the pipe to beconnected to the subprocess’s standard output stream usingconnect_read_pipe(), or the constantsubprocess.PIPE (thedefault). By default a new pipe will be created and connected.stderr: Either a file-like object representing the pipe to beconnected to the subprocess’s standard error stream usingconnect_read_pipe(), or one of the constantssubprocess.PIPE(the default) orsubprocess.STDOUT. By default a new pipe willbe created and connected. Whensubprocess.STDOUT is specified,the subprocess’s standard error stream will be connected to the samepipe as the standard output stream.bufsize: The buffer size to be used when creating a pipe; thisis passed tosubprocess.Popen(). In the default implementationthis defaults to zero, and on Windows it must be zero; thesedefaults deviate fromsubprocess.Popen().executable,preexec_fn,close_fds,cwd,env,startupinfo,creationflags,restore_signals,start_new_session,pass_fds: These optional arguments arepassed tosubprocess.Popen() without interpretation.These methods are only supported on UNIX.
add_signal_handler(sig,callback,*args). Whenever signalsig is received, arrange forcallback(*args) to be called.Specifying another callback for the same signal replaces theprevious handler (only one handler can be active per signal). Thesig must be a valid signal number defined in thesignalmodule. If the signal cannot be handled this raises an exception:ValueError if it is not a valid signal or if it is anuncatchable signal (e.g.SIGKILL),RuntimeError if thisparticular event loop instance cannot handle signals (since signalsare global per process, only an event loop associated with the mainthread can handle signals).remove_signal_handler(sig). Removes the handler for signalsig, if one is set. Raises the same exceptions asadd_signal_handler() (except that it may returnFalseinstead raisingRuntimeError for uncatchable signals). ReturnsTrue if a handler was removed successfully,False if nohandler was set.Note: If these methods are statically known to be unsupported, theymay raiseNotImplementedError instead ofRuntimeError.
An event loop should enforce mutual exclusion of callbacks, i.e. itshould never start a callback while a previously callback is stillrunning. This should apply across all types of callbacks, regardlessof whether they are scheduled usingcall_soon(),call_later(),call_at(),call_soon_threadsafe(),add_reader(),add_writer(), oradd_signal_handler().
There are two categories of exceptions in Python: those that derivefrom theException class and those that derive fromBaseException. Exceptions deriving fromException willgenerally be caught and handled appropriately; for example, they willbe passed through by Futures, and they will be logged and ignored whenthey occur in a callback.
However, exceptions deriving only fromBaseException are typicallynot caught, and will usually cause the program to terminate with atraceback. In some cases they are caught and re-raised. (Examples ofthis category includeKeyboardInterrupt andSystemExit; it isusually unwise to treat these the same as most other exceptions.)
The event loop passes the latter category into itsexceptionhandler. This is a callback which accepts acontext dict as aparameter:
defexception_handler(context):...
context may have many different keys but several of them are verywidely used:
'message': error message.'exception': exception instance;None if there is noexception.'source_traceback': a list of strings representing stack at thepoint the object involved in the error was created.'handle_traceback': a list of strings representing the stack atthe moment the handle involved in the error was created.The loop has the following methods related to exception handling:
get_exception_handler() returns the current exception handlerregistered for the loop.set_exception_handler(handler) sets the exception handler.default_exception_handler(context) thedefault exceptionhandler for this loop implementation.call_exception_handler(context) passescontext into theregistered exception handler. This allows handling uncaughtexceptions uniformly by third-party libraries.The loop usesdefault_exception_handler() if the default was notoverridden by explicitset_exception_handler() call.
By default the loop operates inrelease mode. Applications mayenabledebug mode better error reporting at the cost of someperformance.
In debug mode many additional checks are enabled, for example:
Theloop.slow_callback_duration attribute controls the maximumexecution time allowed between twoyield points before a slowcallback is reported. The default value is 0.1 seconds; it may bechanged by assigning to it.
There are two methods related to debug mode:
get_debug() returnsTrue ifdebug mode is enabled,False otherwise.set_debug(enabled) enablesdebug mode if the argument isTrue.Debug mode is automatically enabled if thePYTHONASYNCIODEBUGenvironment variable is defined and not empty.
The various methods for registering one-off callbacks(call_soon(),call_later(),call_at() andcall_soon_threadsafe()) all return an object representing theregistration that can be used to cancel the callback. This object iscalled aHandle. Handles are opaque and have only one publicmethod:
cancel(): Cancel the callback.Note thatadd_reader(),add_writer() andadd_signal_handler() do not return Handles.
Thecreate_server() method returns aServer instance, whichwraps the sockets (or other network objects) used to accept requests.This class has two public methods:
close(): Close the service. This stops accepting new requestsbut does not cancel requests that have already been accepted and arecurrently being handled.wait_closed(): A coroutine that blocks until the service isclosed and all accepted requests have been handled.Theasyncio.Future class here is intentionally similar to theconcurrent.futures.Future class specified byPEP 3148, but thereare slight differences. Whenever this PEP talks about Futures orfutures this should be understood to refer toasyncio.Future unlessconcurrent.futures.Future is explicitly mentioned. The supportedpublic API is as follows, indicating the differences withPEP 3148:
cancel(). If the Future is already done (or cancelled), donothing and returnFalse. Otherwise, this attempts to cancelthe Future and returnsTrue. If the cancellation attempt issuccessful, eventually the Future’s state will change to cancelled(so thatcancelled() will returnTrue)and the callbacks will be scheduled. For regular Futures,cancellation will always succeed immediately; but for Tasks (seebelow) the task may ignore or delay the cancellation attempt.cancelled(). ReturnsTrue if the Future was successfullycancelled.done(). ReturnsTrue if the Future is done. Note that acancelled Future is considered done too (here and everywhere).result(). Returns the result set withset_result(), orraises the exception set withset_exception(). RaisesCancelledError if cancelled. Difference withPEP 3148: This hasno timeout argument and doesnot wait; if the future is not yetdone, it raises an exception.exception(). Returns the exception if set withset_exception(), orNone if a result was set withset_result(). RaisesCancelledError if cancelled.Difference withPEP 3148: This has no timeout argument and doesnot wait; if the future is not yet done, it raises an exception.add_done_callback(fn). Add a callback to be run when the Futurebecomes done (or is cancelled). If the Future is already done (orcancelled), schedules the callback to usingcall_soon().Difference withPEP 3148: The callback is never called immediately,and always in the context of the caller – typically this is athread. You can think of this as calling the callback throughcall_soon(). Note that in order to matchPEP 3148, the callback(unlike all other callbacks defined in this PEP, and ignoring theconvention from the section “Callback Style” below) is always calledwith a single argument, the Future object. (The motivation forstrictly serializing callbacks scheduled withcall_soon()applies here too.)remove_done_callback(fn). Remove the argument from the list ofcallbacks. This method is not defined byPEP 3148. The argumentmust be equal (using==) to the argument passed toadd_done_callback(). Returns the number of times the callbackwas removed.set_result(result). The Future must not be done (nor cancelled)already. This makes the Future done and schedules the callbacks.Difference withPEP 3148: This is a public API.set_exception(exception). The Future must not be done (norcancelled) already. This makes the Future done and schedules thecallbacks. Difference withPEP 3148: This is a public API.The internal methodset_running_or_notify_cancel() is notsupported; there is no way to set the running state. Likewise,the methodrunning() is not supported.
The following exceptions are defined:
InvalidStateError. Raised whenever the Future is not in a stateacceptable to the method being called (e.g. callingset_result()on a Future that is already done, or callingresult() on a Futurethat is not yet done).InvalidTimeoutError. Raised byresult() andexception()when a nonzerotimeout argument is given.CancelledError. An alias forconcurrent.futures.CancelledError. Raised whenresult() orexception() is called on a Future that is cancelled.TimeoutError. An alias forconcurrent.futures.TimeoutError.May be raised byrun_until_complete().A Future is associated with an event loop when it is created.
Aasyncio.Future object is not acceptable to thewait() andas_completed() functions in theconcurrent.futures package.However, there are similar APIsasyncio.wait() andasyncio.as_completed(), described below.
Aasyncio.Future object is acceptable to ayieldfrom expressionwhen used in a coroutine. This is implemented through the__iter__() interface on the Future. See the section “Coroutinesand the Scheduler” below.
When a Future is garbage-collected, if it has an associated exceptionbut neitherresult() norexception() has ever been called, theexception is logged. (When a coroutine usesyieldfrom to waitfor a Future, that Future’sresult() method is called once thecoroutine is resumed.)
In the future (pun intended) we may unifyasyncio.Future andconcurrent.futures.Future, e.g. by adding an__iter__() methodto the latter that works withyieldfrom. To prevent accidentallyblocking the event loop by calling e.g.result() on a Futurethat’s not done yet, the blocking operation may detect that an eventloop is active in the current thread and raise an exception instead.However the current PEP strives to have no dependencies beyond Python3.3, so changes toconcurrent.futures.Future are off the table fornow.
There are some public functions related to Futures:
asyncio.async(arg). This takes an argument that is either acoroutine object or a Future (i.e., anything you can use withyieldfrom) and returns a Future. If the argument is a Future,it is returned unchanged; if it is a coroutine object, it wraps itin a Task (remember thatTask is a subclass ofFuture).asyncio.wrap_future(future). This takes aPEP 3148 Future(i.e., an instance ofconcurrent.futures.Future) and returns aFuture compatible with the event loop (i.e., aasyncio.Futureinstance).Transports and protocols are strongly influenced by Twisted and PEP3153. Users rarely implement or instantiate transports – rather,event loops offer utility methods to set up transports.
Transports work in conjunction with protocols. Protocols aretypically written without knowing or caring about the exact type oftransport used, and transports can be used with a wide variety ofprotocols. For example, an HTTP client protocol implementation may beused with either a plain socket transport or an SSL/TLS transport.The plain socket transport can be used with many different protocolsbesides HTTP (e.g. SMTP, IMAP, POP, FTP, IRC, SPDY).
The most common type of transport is a bidirectional stream transport.There are also unidirectional stream transports (used for pipes) anddatagram transports (used by thecreate_datagram_endpoint()method).
get_extra_info(name,default=None). This is a catch-all methodthat returns implementation-specific information about a transport.The first argument is the name of the extra field to be retrieved.The optional second argument is a default value to be returned.Consult the implementation documentation to find out the supportedextra field names. For an unsupported name, the default is alwaysreturned.A bidirectional stream transport is an abstraction on top of a socketor something similar (for example, a pair of UNIX pipes or an SSL/TLSconnection).
Most connections have an asymmetric nature: the client and serverusually have very different roles and behaviors. Hence, the interfacebetween transport and protocol is also asymmetric. From theprotocol’s point of view,writing data is done by calling thewrite() method on the transport object; this buffers the data andreturns immediately. However, the transport takes a more active roleinreading data: whenever some data is read from the socket (orother data source), the transport calls the protocol’sdata_received() method.
Nevertheless, the interface between transport and protocol used bybidirectional streams is the same for clients as it is for servers,since the connection between a client and a server is essentially apair of streams, one in each direction.
Bidirectional stream transports have the following public methods:
write(data). Write some bytes. The argument must be a bytesobject. ReturnsNone. The transport is free to buffer thebytes, but it must eventually cause the bytes to be transferred tothe entity at the other end, and it must maintain stream behavior.That is,t.write(b'abc');t.write(b'def') is equivalent tot.write(b'abcdef'), as well as to:t.write(b'a')t.write(b'b')t.write(b'c')t.write(b'd')t.write(b'e')t.write(b'f')
writelines(iterable). Equivalent to:fordatainiterable:self.write(data)
write_eof(). Close the writing end of the connection.Subsequent calls towrite() are not allowed. Once all buffereddata is transferred, the transport signals to the other end that nomore data will be received. Some protocols don’t support thisoperation; in that case, callingwrite_eof() will raise anexception. (Note: This used to be calledhalf_close(), butunless you already know what it is for, that name doesn’t indicatewhich end is closed.)can_write_eof(). ReturnTrue if the protocol supportswrite_eof(),False if it does not. (This method typicallyreturns a fixed value that depends only on the specific Transportclass, not on the state of the Transport object. It is neededbecause some protocols need to change their behavior whenwrite_eof() is unavailable. For example, in HTTP, to send datawhose size is not known ahead of time, the end of the data istypically indicated usingwrite_eof(); however, SSL/TLS does notsupport this, and an HTTP protocol implementation would have to usethe “chunked” transfer encoding in this case. But if the data sizeis known ahead of time, the best approach in both cases is to usethe Content-Length header.)get_write_buffer_size(). Return the current size of thetransport’s write buffer in bytes. This only knows about the writebuffer managed explicitly by the transport; buffering in otherlayers of the network stack or elsewhere of the network is notreported.set_write_buffer_limits(high=None,low=None). Set the high- andlow-water limits for flow control.These two values control when to call the protocol’spause_writing() andresume_writing() methods. If specified,the low-water limit must be less than or equal to the high-waterlimit. Neither value can be negative.
The defaults are implementation-specific. If only the high-waterlimit is given, the low-water limit defaults to animplementation-specific value less than or equal to the high-waterlimit. Setting high to zero forces low to zero as well, and causespause_writing() to be called whenever the buffer becomesnon-empty. Setting low to zero causesresume_writing() to becalled only once the buffer is empty. Use of zero for either limitis generally sub-optimal as it reduces opportunities for doing I/Oand computation concurrently.
pause_reading(). Suspend delivery of data to the protocol untila subsequentresume_reading() call. Betweenpause_reading()andresume_reading(), the protocol’sdata_received() methodwill not be called.resume_reading(). Restart delivery of data to the protocol viadata_received(). Note that “paused” is a binary state –pause_reading() should only be called when the transport is notpaused, whileresume_reading() should only be called when thetransport is paused.close(). Sever the connection with the entity at the other end.Any data buffered bywrite() will (eventually) be transferredbefore the connection is actually closed. The protocol’sdata_received() method will not be called again. Once allbuffered data has been flushed, the protocol’sconnection_lost()method will be called withNone as the argument. Note thatthis method does not wait for all that to happen.abort(). Immediately sever the connection. Any data stillbuffered by the transport is thrown away. Soon, the protocol’sconnection_lost() method will be called withNone asargument.A writing stream transport supports thewrite(),writelines(),write_eof(),can_write_eof(),close() andabort() methodsdescribed for bidirectional stream transports.
A reading stream transport supports thepause_reading(),resume_reading() andclose() methods described forbidirectional stream transports.
A writing stream transport calls onlyconnection_made() andconnection_lost() on its associated protocol.
A reading stream transport can call all protocol methods specified inthe Protocols section below (i.e., the previous two plusdata_received() andeof_received()).
Datagram transports have these methods:
sendto(data,addr=None). Sends a datagram (a bytes object).The optional second argument is the destination address. Ifomitted,remote_addr must have been specified in thecreate_datagram_endpoint() call that created this transport. Ifpresent, andremote_addr was specified, they must match. The(data, addr) pair may be sent immediately or buffered. The returnvalue isNone.abort(). Immediately close the transport. Buffered data willbe discarded.close(). Close the transport. Buffered data will betransmitted asynchronously.Datagram transports call the following methods on the associatedprotocol object:connection_made(),connection_lost(),error_received() anddatagram_received(). (“Connection”in these method names is a slight misnomer, but the concepts stillexist:connection_made() means the transport representing theendpoint has been created, andconnection_lost() means thetransport is closed.)
Subprocess transports have the following methods:
get_pid(). Return the process ID of the subprocess.get_returncode(). Return the process return code, if theprocess has exited; otherwiseNone.get_pipe_transport(fd). Return the pipe transport (aunidirectional stream transport) corresponding to the argument,which should be 0, 1 or 2 representing stdin, stdout or stderr (ofthe subprocess). If there is no such pipe transport, returnNone. For stdin, this is a writing transport; for stdout andstderr this is a reading transport. You must use this method to geta transport you can use to write to the subprocess’s stdin.send_signal(signal). Send a signal to the subprocess.terminate(). Terminate the subprocess.kill(). Kill the subprocess. On Windows this is an alias forterminate().close(). This is an alias forterminate().Note thatsend_signal(),terminate() andkill() wrap thecorresponding methods in the standard librarysubprocess module.
Protocols are always used in conjunction with transports. While a fewcommon protocols are provided (e.g. decent though not necessarilyexcellent HTTP client and server implementations), most protocols willbe implemented by user code or third-party libraries.
Like for transports, we distinguish between stream protocols, datagramprotocols, and perhaps other custom protocols. The most common typeof protocol is a bidirectional stream protocol. (There are nounidirectional protocols.)
A (bidirectional) stream protocol must implement the followingmethods, which will be called by the transport. Think of these ascallbacks that are always called by the event loop in the rightcontext. (See the “Context” section way above.)
connection_made(transport). Indicates that the transport isready and connected to the entity at the other end. The protocolshould probably save the transport reference as an instance variable(so it can call itswrite() and other methods later), and maywrite an initial greeting or request at this point.data_received(data). The transport has read some bytes from theconnection. The argument is always a non-empty bytes object. Thereare no guarantees about the minimum or maximum size of the datapassed along this way.p.data_received(b'abcdef') should betreated exactly equivalent to:p.data_received(b'abc')p.data_received(b'def')
eof_received(). This is called when the other end calledwrite_eof() (or something equivalent). If this returns a falsevalue (includingNone), the transport will close itself. If itreturns a true value, closing the transport is up to the protocol.However, for SSL/TLS connections this is ignored, because the TLSstandard requires that no more data is sent and the connection isclosed as soon as a “closure alert” is received.The default implementation returnsNone.
pause_writing(). Asks that the protocol temporarily stopwriting data to the transport. Heeding the request is optional, butthe transport’s buffer may grow without bounds if you keep writing.The buffer size at which this is called can be controlled throughthe transport’sset_write_buffer_limits() method.resume_writing(). Tells the protocol that it is safe to startwriting data to the transport again. Note that this may be calleddirectly by the transport’swrite() method (as opposed to beingcalled indirectly usingcall_soon()), so that the protocol maybe aware of its paused state immediately afterwrite() returns.connection_lost(exc). The transport has been closed or aborted,has detected that the other end has closed the connection cleanly,or has encountered an unexpected error. In the first three casesthe argument isNone; for an unexpected error, the argument isthe exception that caused the transport to give up.Here is a table indicating the order and multiplicity of the basiccalls:
connection_made() – exactly oncedata_received() – zero or more timeseof_received() – at most onceconnection_lost() – exactly onceCalls topause_writing() andresume_writing() occur in pairsand only between #1 and #4. These pairs will not be nested. Thefinalresume_writing() call may be omitted; i.e. a pausedconnection may be lost and never be resumed.
Datagram protocols haveconnection_made() andconnection_lost() methods with the same signatures as streamprotocols. (As explained in the section about datagram transports, weprefer the slightly odd nomenclature over defining different methodnames to indicating the opening and closing of the socket.)
In addition, they have the following methods:
datagram_received(data,addr). Indicates that a datagramdata (a bytes objects) was received from remote addressaddr(an IPv4 2-tuple or an IPv6 4-tuple).error_received(exc). Indicates that a send or receive operationraised anOSError exception. Since datagram errors may betransient, it is up to the protocol to call the transport’sclose() method if it wants to close the endpoint.Here is a chart indicating the order and multiplicity of calls:
connection_made() – exactly oncedatagram_received(),error_received() – zero or more timesconnection_lost() – exactly onceSubprocess protocols haveconnection_made(),connection_lost(),pause_writing() andresume_writing() methods with the samesignatures as stream protocols. In addition, they have the followingmethods:
pipe_data_received(fd,data). Called when the subprocess writesdata to its stdout or stderr.fd is the file descriptor (1 forstdout, 2 for stderr).data is abytes object.pipe_connection_lost(fd,exc). Called when the subprocesscloses its stdin, stdout or stderr.fd is the file descriptor.exc is an exception orNone.process_exited(). Called when the subprocess has exited. Toretrieve the exit status, use the transport’sget_returncode()method.Note that depending on the behavior of the subprocess it is possiblethatprocess_exited() is called either before or afterpipe_connection_lost(). For example, if the subprocess creates asub-subprocess that shares its stdin/stdout/stderr and then itselfexits,process_exited() may be called while all the pipes arestill open. On the other hand, when the subprocess closes itsstdin/stdout/stderr but does not exit,pipe_connection_lost() maybe called for all three pipes withoutprocess_exited() beingcalled. If (as is the more common case) the subprocess exits andthereby implicitly closes all pipes, the calling order is undefined.
Most interfaces taking a callback also take positional arguments. Forinstance, to arrange forfoo("abc",42) to be called soon, youcallloop.call_soon(foo,"abc",42). To schedule the callfoo(), useloop.call_soon(foo). This convention greatlyreduces the number of small lambdas required in typical callbackprogramming.
This convention specifically doesnot support keyword arguments.Keyword arguments are used to pass optional extra information aboutthe callback. This allows graceful evolution of the API withouthaving to worry about whether a keyword might be significant to acallee somewhere. If you have a callback thatmust be called with akeyword argument, you can use a lambda. For example:
loop.call_soon(lambda:foo('abc',repeat=42))
This is a separate toplevel section because its status is differentfrom the event loop interface. Usage of coroutines is optional, andit is perfectly fine to write code using callbacks only. On the otherhand, there is only one implementation of the scheduler/coroutine API,and if you’re using coroutines, that’s the one you’re using.
A coroutine is a generator that follows certain conventions. Fordocumentation purposes, all coroutines should be decorated with@asyncio.coroutine, but this cannot be strictly enforced.
Coroutines use theyieldfrom syntax introduced inPEP 380,instead of the originalyield syntax.
The word “coroutine”, like the word “generator”, is used for twodifferent (though related) concepts:
asyncio.coroutine). If disambiguation is neededwe will call this acoroutine function.Things a coroutine can do:
result=yieldfromfuture – suspends the coroutine until thefuture is done, then returns the future’s result, or raises anexception, which will be propagated. (If the future is cancelled,it will raise aCancelledError exception.) Note that tasks arefutures, and everything said about futures also applies to tasks.result=yieldfromcoroutine – wait for another coroutine toproduce a result (or raise an exception, which will be propagated).Thecoroutine expression must be acall to another coroutine.returnexpression – produce a result to the coroutine that iswaiting for this one usingyieldfrom.raiseexception – raise an exception in the coroutine that iswaiting for this one usingyieldfrom.Calling a coroutine does not start its code running – it is just agenerator, and the coroutine object returned by the call is really agenerator object, which doesn’t do anything until you iterate over it.In the case of a coroutine object, there are two basic ways to startit running: callyieldfromcoroutine from another coroutine(assuming the other coroutine is already running!), or convert it to aTask (see below).
Coroutines (and tasks) can only run when the event loop is running.
To wait for multiple coroutines or Futures, two APIs similar to thewait() andas_completed() APIs in theconcurrent.futurespackage are provided:
asyncio.wait(fs,timeout=None,return_when=ALL_COMPLETED). Thisis a coroutine that waits for the Futures or coroutines given byfs to complete. Coroutine arguments will be wrapped in Tasks(see below). This returns a Future whose result on success is atuple of two sets of Futures,(done,pending), wheredone isthe set of original Futures (or wrapped coroutines) that are done(or cancelled), andpending is the rest, i.e. those that arestill not done (nor cancelled). Note that with the defaults fortimeout andreturn_when,done will always be an emptylist. Optional argumentstimeout andreturn_when have thesame meaning and defaults as forconcurrent.futures.wait():timeout, if notNone, specifies a timeout for the overalloperation;return_when, specifies when to stop. The constantsFIRST_COMPLETED,FIRST_EXCEPTION,ALL_COMPLETED aredefined with the same values and the same meanings as inPEP 3148:ALL_COMPLETED (default): Wait until all Futures are done (oruntil the timeout occurs).FIRST_COMPLETED: Wait until at least one Future is done (oruntil the timeout occurs).FIRST_EXCEPTION: Wait until at least one Future is done butnot cancelled with an exception set. (The exclusion of cancelledFutures from the condition is surprising, butPEP 3148 does itthis way.)asyncio.as_completed(fs,timeout=None). Returns an iterator whosevalues are Futures or coroutines; waiting for successive valueswaits until the next Future or coroutine from the setfscompletes, and returns its result (or raises its exception). Theoptional argumenttimeout has the same meaning and default as itdoes forconcurrent.futures.wait(): when the timeout occurs, thenext Future returned by the iterator will raiseTimeoutErrorwhen waited for. Example of use:forfinas_completed(fs):result=yield fromf# May raise an exception.# Use result.
Note: if you do not wait for the values produced by the iterator,yourfor loop may not make progress (since you are not allowingother tasks to run).
asyncio.wait_for(f,timeout). This is a convenience to wait fora single coroutine or Future with a timeout. When a timeout occurs,it cancels the task and raises TimeoutError. To avoid the taskcancellation, wrap it inshield().asyncio.gather(f1,f2,...). Returns a Future which waits untilall arguments (Futures or coroutines) are done and return a list oftheir corresponding results. If one or more of the arguments iscancelled or raises an exception, the returned Future is cancelledor has its exception set (matching what happened to the firstargument), and the remaining arguments are left running in thebackground. Cancelling the returned Future does not affect thearguments. Note that coroutine arguments are converted to Futuresusingasyncio.async().asyncio.shield(f). Wait for a Future, shielding it fromcancellation. This returns a Future whose result or exceptionis exactly the same as the argument; however, if the returnedFuture is cancelled, the argument Future is unaffected.A use case for this function would be a coroutine that caches aquery result for a coroutine that handles a request in an HTTPserver. When the request is cancelled by the client, we could(arguably) want the query-caching coroutine to continue to run, sothat when the client reconnects, the query result is (hopefully)cached. This could be written e.g. as follows:
@asyncio.coroutinedefhandle_request(self,request):...cached_query=self.get_cache(...)ifcached_queryisNone:cached_query=yield fromasyncio.shield(self.fill_cache(...))...
The coroutineasyncio.sleep(delay) returns after a given time delay.
A Task is an object that manages an independently running coroutine.The Task interface is the same as the Future interface, and in factTask is a subclass ofFuture. The task becomes done when itscoroutine returns or raises an exception; if it returns a result, thatbecomes the task’s result, if it raises an exception, that becomes thetask’s exception.
Cancelling a task that’s not done yet throws anasyncio.CancelledError exception into the coroutine. If thecoroutine doesn’t catch this (or if it re-raises it) the task will bemarked as cancelled (i.e.,cancelled() will returnTrue); butif the coroutine somehow catches and ignores the exception it maycontinue to execute (andcancelled() will returnFalse).
Tasks are also useful for interoperating between coroutines andcallback-based frameworks like Twisted. After converting a coroutineinto a Task, callbacks can be added to the Task.
To convert a coroutine into a task, call the coroutine function andpass the resulting coroutine object to theloop.create_task()method. You may also useasyncio.ensure_future() for this purpose.
You may ask, why not automatically convert all coroutines to Tasks?The@asyncio.coroutine decorator could do this. However, this wouldslow things down considerably in the case where one coroutine callsanother (and so on), as switching to a “bare” coroutine has much lessoverhead than switching to a Task.
TheTask class is derived fromFuture adding new methods:
current_task(loop=None). Aclass method returning thecurrently running task in an event loop. Ifloop isNone themethod returns the current task for the default loop. Everycoroutine is executed inside atask context, either aTaskcreated usingensure_future() orloop.create_task(), or bybeing called from another coroutine usingyieldfrom orawait. This method returnsNone when calledoutside acoroutine, e.g. in a callback scheduled usingloop.call_later().all_tasks(loop=None). Aclass method returning a set of allactive tasks for the loop. This uses the default loop ifloop isNone.The scheduler has no public interface. You interact with it by usingyieldfromfuture andyieldfromtask. In fact, there is nosingle object representing the scheduler – its behavior isimplemented by theTask andFuture classes using only thepublic interface of the event loop, so it will work with third-partyevent loop implementations, too.
A few functions and classes are provided to simplify the writing ofbasic stream-based clients and servers, such as FTP or HTTP. Theseare:
asyncio.open_connection(host,port): A wrapper forEventLoop.create_connection() that does not require you toprovide aProtocol factory or class. This is a coroutine thatreturns a(reader,writer) pair, wherereader is an instanceofStreamReader andwriter is an instance ofStreamWriter (both described below).asyncio.start_server(client_connected_cb,host,port): A wrapperforEventLoop.create_server() that takes a simple callbackfunction rather than aProtocol factory or class. This is acoroutine that returns aServer object just ascreate_server() does. Each time a client connection isaccepted,client_connected_cb(reader,writer) is called, wherereader is an instance ofStreamReader andwriter is aninstance ofStreamWriter (both described below). If the resultreturned byclient_connected_cb() is a coroutine, it isautomatically wrapped in aTask.StreamReader: A class offering an interface not unlike that of aread-only binary stream, except that the various reading methods arecoroutines. It is normally driven by aStreamReaderProtocolinstance. Note that there should be only one reader. The interfacefor the reader is:readline(): A coroutine that reads a string of bytesrepresenting a line of text ending in'\n', or until the endof the stream, whichever comes first.read(n): A coroutine that reads up ton bytes. Ifnis omitted or negative, it reads until the end of the stream.readexactly(n): A coroutine that reads exactlyn bytes, oruntil the end of the stream, whichever comes first.exception(): Return the exception that has been set on thestream usingset_exception(), or None if no exception is set.The interface for the driver is:
feed_data(data): Appenddata (abytes object) to theinternal buffer. This unblocks a blocked reading coroutine if itprovides sufficient data to fulfill the reader’s contract.feed_eof(): Signal the end of the buffer. This unblocks ablocked reading coroutine. No more data should be fed to thereader after this call.set_exception(exc): Set an exception on the stream. Allsubsequent reading methods will raise this exception. No moredata should be fed to the reader after this call.StreamWriter: A class offering an interface not unlike that of awrite-only binary stream. It wraps a transport. The interface isan extended subset of the transport interface: the following methodsbehave the same as the corresponding transport methods:write(),writelines(),write_eof(),can_write_eof(),get_extra_info(),close(). Note that the writing methodsare _not_ coroutines (this is the same as for transports, butdifferent from theStreamReader class). The following method isin addition to the transport interface:drain(): This should be called withyieldfrom afterwriting significant data, for the purpose of flow control. Theintended use is like this:writer.write(data)yield fromwriter.drain()
Note that this is not technically a coroutine: it returns either aFuture or an empty tuple (both can be passed toyieldfrom).Use of this method is optional. However, when it is not used, theinternal buffer of the transport underlying theStreamWritermay fill up with all data that was ever written to the writer. Ifan app does not have a strict limit on how much data it writes, it_should_ callyieldfromdrain() occasionally to avoid fillingup the transport buffer.
StreamReaderProtocol: A protocol implementation used as anadapter between the bidirectional stream transport/protocolinterface and theStreamReader andStreamWriter classes. Itacts as a driver for a specificStreamReader instance, callingits methodsfeed_data(),feed_eof(), andset_exception()in response to various protocol callbacks. It also controls thebehavior of thedrain() method of theStreamWriter instance.Locks, events, conditions and semaphores modeled after those in thethreading module are implemented and can be accessed by importingtheasyncio.locks submodule. Queues modeled after those in thequeue module are implemented and can be accessed by importing theasyncio.queues submodule.
In general these have a close correspondence to their threadedcounterparts, however, blocking methods (e.g.acquire() on locks,put() andget() on queues) are coroutines, and timeoutparameters are not provided (you can useasyncio.wait_for() to adda timeout to a blocking call, however).
The docstrings in the modules provide more complete documentation.
The following classes are provided byasyncio.locks. For allthese exceptEvent, thewith statement may be used incombination withyieldfrom to acquire the lock and ensure thatthe lock is released regardless of how thewith block is left, asfollows:
with(yield frommy_lock):...
Lock: a basic mutex, with methodsacquire() (a coroutine),locked(), andrelease().Event: an event variable, with methodswait() (a coroutine),set(),clear(), andis_set().Condition: a condition variable, with methodsacquire(),wait(),wait_for(predicate) (all three coroutines),locked(),release(),notify(), andnotify_all().Semaphore: a semaphore, with methodsacquire() (acoroutine),locked(), andrelease(). The constructorargument is the initial value (default1).BoundedSemaphore: a bounded semaphore; this is similar toSemaphore but the initial value is also the maximum value.The following classes and exceptions are provided byasyncio.queues.
Queue: a standard queue, with methodsget(),put() (bothcoroutines),get_nowait(),put_nowait(),empty(),full(),qsize(), andmaxsize().PriorityQueue: a subclass ofQueue that retrieves entriesin priority order (lowest first).LifoQueue: a subclass ofQueue that retrieves the mostrecently added entries first.JoinableQueue: a subclass ofQueue withtask_done() andjoin() methods (the latter a coroutine).Empty,Full: exceptions raised whenget_nowait() orput_nowait() is called on a queue that is empty or full,respectively.All logging performed by theasyncio package uses a singlelogging.Logger object,asyncio.logger. To customize loggingyou can use the standardLogger API on this object. (Do notreplace the object though.)
SIGCHLD handling on UNIXEfficient implementation of theprocess_exited() method onsubprocess protocols requires aSIGCHLD signal handler. However,signal handlers can only be set on the event loop associated with themain thread. In order to support spawning subprocesses from eventloops running in other threads, a mechanism exists to allow sharing aSIGCHLD handler between multiple event loops. There are twoadditional functions,asyncio.get_child_watcher() andasyncio.set_child_watcher(), and corresponding methods on theevent loop policy.
There are two child watcher implementation classes,FastChildWatcher andSafeChildWatcher. Both useSIGCHLD.TheSafeChildWatcher class is used by default; it is inefficientwhen many subprocesses exist simultaneously. TheFastChildWatcherclass is efficient, but it may interfere with other code (either Ccode or Python code) that spawns subprocesses without using anasyncio event loop. If you are sure you are not using other codethat spawns subprocesses, to use the fast implementation, run thefollowing in your main thread:
watcher=asyncio.FastChildWatcher()asyncio.set_child_watcher(watcher)
(There is agreement that these features are desirable, but noimplementation was available when Python 3.4 beta 1 was released, andthe feature freeze for the rest of the Python 3.4 release cycleprohibits adding them in this late stage. However, they willhopefully be added in Python 3.5, and perhaps earlier in the PyPIdistribution.)
Former wish list items that have since been implemented (but aren’tspecified by the PEP):
(Note that these have been resolved de facto in favor of the statusquo by the acceptance of the PEP. However, the PEP’s provisionalstatus allows revising these decisions for Python 3.5.)
create_connection() andcreate_datagram_endpoint()have aproto argument but notcreate_server()? And why arethe family, flag, proto arguments forgetaddrinfo() sometimeszero and sometimes named constants (whose value is also zero)?sock_sendto() andsock_recvfrom(), and perhaps others likepipe_read()?I guess users can write their own (it’s not rocket science).timeout keyword arguments to some methods,and other timeouts can probably be implemented by clever use ofcall_later() andTask.cancel(). But it’s possible that someoperations need default timeouts, and we may want to change thedefault for a specific operation globally (i.e., per event loop).async/await.yieldfrom.yieldfrom tutorials:http://www.cosc.canterbury.ac.nz/greg.ewing/python/yield-from/yield_from.htmlconcurrent.futures.Future.while,for andwith:http://python-notes.boredomandlaziness.org/en/latest/pep_ideas/async_programming.htmlApart fromPEP 3153, influences includePEP 380 and Greg Ewing’stutorial foryieldfrom, Twisted, Tornado, ZeroMQ, pyftpdlib, andwattle (Steve Dower’s counter-proposal). My previous work onasynchronous support in the NDB library for Google App Engine providedan important starting point.
I am grateful for the numerous discussions on python-ideas fromSeptember through December 2012, and many more on python-tulip sincethen; a Skype session with Steve Dower and Dino Viehland; emailexchanges with and a visit by Ben Darnell; an audience with NielsProvos (original author of libevent); and in-person meetings (as wellas frequent email exchanges) with several Twisted developers,including Glyph, Brian Warner, David Reid, and Duncan McGreggor.
Contributors to the implementation includeEli Bendersky,Gustavo Carneiro (Gambit Research),Saúl Ibarra Corretgé,Geert Jansen,A. Jesse Jiryu Davis,Nikolay Kim,Charles-François Natali,Richard Oudkerk,Antoine Pitrou,Giampaolo Rodolá,Andrew Svetlov,and many others who submitted bugs and/or fixes.
I thank Antoine Pitrou for his feedback in his role of official PEPBDFL.
This document has been placed in the public domain.
Source:https://github.com/python/peps/blob/main/peps/pep-3156.rst
Last modified:2025-02-01 08:59:27 GMT