Subprocesses¶
Source code:Lib/asyncio/subprocess.py,Lib/asyncio/base_subprocess.py
This section describes high-level async/await asyncio APIs tocreate and manage subprocesses.
Here’s an example of how asyncio can run a shell command andobtain its result:
importasyncioasyncdefrun(cmd):proc=awaitasyncio.create_subprocess_shell(cmd,stdout=asyncio.subprocess.PIPE,stderr=asyncio.subprocess.PIPE)stdout,stderr=awaitproc.communicate()print(f'[{cmd!r} exited with{proc.returncode}]')ifstdout:print(f'[stdout]\n{stdout.decode()}')ifstderr:print(f'[stderr]\n{stderr.decode()}')asyncio.run(run('ls /zzz'))
will print:
['ls /zzz'exitedwith1][stderr]ls:/zzz:Nosuchfileordirectory
Because all asyncio subprocess functions are asynchronous and asyncioprovides many tools to work with such functions, it is easy to executeand monitor multiple subprocesses in parallel. It is indeed trivialto modify the above example to run several commands simultaneously:
asyncdefmain():awaitasyncio.gather(run('ls /zzz'),run('sleep 1; echo "hello"'))asyncio.run(main())
See also theExamples subsection.
Creating Subprocesses¶
- asyncasyncio.create_subprocess_exec(program,*args,stdin=None,stdout=None,stderr=None,limit=None,**kwds)¶
Create a subprocess.
Thelimit argument sets the buffer limit for
StreamReader
wrappers forstdout
andstderr
(ifsubprocess.PIPE
is passed tostdout andstderr arguments).Return a
Process
instance.See the documentation of
loop.subprocess_exec()
for otherparameters.Changed in version 3.10:Removed theloop parameter.
- asyncasyncio.create_subprocess_shell(cmd,stdin=None,stdout=None,stderr=None,limit=None,**kwds)¶
Run thecmd shell command.
Thelimit argument sets the buffer limit for
StreamReader
wrappers forstdout
andstderr
(ifsubprocess.PIPE
is passed tostdout andstderr arguments).Return a
Process
instance.See the documentation of
loop.subprocess_shell()
for otherparameters.Important
It is the application’s responsibility to ensure that all whitespace andspecial characters are quoted appropriately to avoidshell injectionvulnerabilities. The
shlex.quote()
function can be used to properlyescape whitespace and special shell characters in strings that are goingto be used to construct shell commands.Changed in version 3.10:Removed theloop parameter.
Note
Subprocesses are available for Windows if aProactorEventLoop
isused. SeeSubprocess Support on Windowsfor details.
See also
asyncio also has the followinglow-level APIs to work with subprocesses:loop.subprocess_exec()
,loop.subprocess_shell()
,loop.connect_read_pipe()
,loop.connect_write_pipe()
,as well as theSubprocess TransportsandSubprocess Protocols.
Constants¶
- asyncio.subprocess.PIPE¶
Can be passed to thestdin,stdout orstderr parameters.
IfPIPE is passed tostdin argument, the
Process.stdin
attributewill point to aStreamWriter
instance.IfPIPE is passed tostdout orstderr arguments, the
Process.stdout
andProcess.stderr
attributes will point toStreamReader
instances.
- asyncio.subprocess.STDOUT¶
Special value that can be used as thestderr argument and indicatesthat standard error should be redirected into standard output.
- asyncio.subprocess.DEVNULL¶
Special value that can be used as thestdin,stdout orstderr argumentto process creation functions. It indicates that the special file
os.devnull
will be used for the corresponding subprocess stream.
Interacting with Subprocesses¶
Bothcreate_subprocess_exec()
andcreate_subprocess_shell()
functions return instances of theProcess class.Process is a high-levelwrapper that allows communicating with subprocesses and watching fortheir completion.
- classasyncio.subprocess.Process¶
An object that wraps OS processes created by the
create_subprocess_exec()
andcreate_subprocess_shell()
functions.This class is designed to have a similar API to the
subprocess.Popen
class, but there are somenotable differences:unlike Popen, Process instances do not have an equivalent tothe
poll()
method;the
communicate()
andwait()
methods don’t have atimeout parameter: use thewait_for()
function;the
Process.wait()
methodis asynchronous, whereassubprocess.Popen.wait()
methodis implemented as a blocking busy loop;theuniversal_newlines parameter is not supported.
This class isnot thread safe.
See also theSubprocess and Threadssection.
- asyncwait()¶
Wait for the child process to terminate.
Set and return the
returncode
attribute.Note
This method can deadlock when using
stdout=PIPE
orstderr=PIPE
and the child process generates so much outputthat it blocks waiting for the OS pipe buffer to acceptmore data. Use thecommunicate()
method when using pipesto avoid this condition.
- asynccommunicate(input=None)¶
Interact with process:
send data tostdin (ifinput is not
None
);closesstdin;
read data fromstdout andstderr, until EOF is reached;
wait for process to terminate.
The optionalinput argument is the data (
bytes
object)that will be sent to the child process.Return a tuple
(stdout_data,stderr_data)
.If either
BrokenPipeError
orConnectionResetError
exception is raised when writinginput intostdin, theexception is ignored. This condition occurs when the processexits before all data are written intostdin.If it is desired to send data to the process’stdin,the process needs to be created with
stdin=PIPE
. Similarly,to get anything other thanNone
in the result tuple, theprocess has to be created withstdout=PIPE
and/orstderr=PIPE
arguments.Note, that the data read is buffered in memory, so do not usethis method if the data size is large or unlimited.
Changed in version 3.12:stdin gets closed when
input=None
too.
- send_signal(signal)¶
Sends the signalsignal to the child process.
Note
On Windows,
SIGTERM
is an alias forterminate()
.CTRL_C_EVENT
andCTRL_BREAK_EVENT
can be sent to processesstarted with acreationflags parameter which includesCREATE_NEW_PROCESS_GROUP
.
- terminate()¶
Stop the child process.
On POSIX systems this method sends
SIGTERM
to thechild process.On Windows the Win32 API function
TerminateProcess()
iscalled to stop the child process.
- kill()¶
Kill the child process.
On POSIX systems this method sends
SIGKILL
to the childprocess.On Windows this method is an alias for
terminate()
.
- stdin¶
Standard input stream (
StreamWriter
) orNone
if the process was created withstdin=None
.
- stdout¶
Standard output stream (
StreamReader
) orNone
if the process was created withstdout=None
.
- stderr¶
Standard error stream (
StreamReader
) orNone
if the process was created withstderr=None
.
Warning
Use the
communicate()
method rather thanprocess.stdin.write()
,awaitprocess.stdout.read()
orawaitprocess.stderr.read()
.This avoids deadlocks due to streams pausing reading or writingand blocking the child process.- pid¶
Process identification number (PID).
Note that for processes created by the
create_subprocess_shell()
function, this attribute is the PID of the spawned shell.
- returncode¶
Return code of the process when it exits.
A
None
value indicates that the process has not terminated yet.A negative value
-N
indicates that the child was terminatedby signalN
(POSIX only).
Subprocess and Threads¶
Standard asyncio event loop supports running subprocesses from different threads bydefault.
On Windows subprocesses are provided byProactorEventLoop
only (default),SelectorEventLoop
has no subprocess support.
On UNIXchild watchers are used for subprocess finish waiting, seeProcess Watchers for more info.
Changed in version 3.8:UNIX switched to useThreadedChildWatcher
for spawning subprocesses fromdifferent threads without any limitation.
Spawning a subprocess withinactive current child watcher raisesRuntimeError
.
Note that alternative event loop implementations might have own limitations;please refer to their documentation.
See also
TheConcurrency and multithreading in asyncio section.
Examples¶
An example using theProcess
class tocontrol a subprocess and theStreamReader
class to read fromits standard output.
The subprocess is created by thecreate_subprocess_exec()
function:
importasyncioimportsysasyncdefget_date():code='import datetime; print(datetime.datetime.now())'# Create the subprocess; redirect the standard output# into a pipe.proc=awaitasyncio.create_subprocess_exec(sys.executable,'-c',code,stdout=asyncio.subprocess.PIPE)# Read one line of output.data=awaitproc.stdout.readline()line=data.decode('ascii').rstrip()# Wait for the subprocess exit.awaitproc.wait()returnlinedate=asyncio.run(get_date())print(f"Current date:{date}")
See also thesame examplewritten using low-level APIs.