Subprocesses

Source code:Lib/asyncio/subprocess.py,Lib/asyncio/base_subprocess.py


This section describes high-level async/await asyncio APIs tocreate and manage subprocesses.

Here’s an example of how asyncio can run a shell command andobtain its result:

importasyncioasyncdefrun(cmd):proc=awaitasyncio.create_subprocess_shell(cmd,stdout=asyncio.subprocess.PIPE,stderr=asyncio.subprocess.PIPE)stdout,stderr=awaitproc.communicate()print(f'[{cmd!r} exited with{proc.returncode}]')ifstdout:print(f'[stdout]\n{stdout.decode()}')ifstderr:print(f'[stderr]\n{stderr.decode()}')asyncio.run(run('ls /zzz'))

will print:

['ls /zzz'exitedwith1][stderr]ls:/zzz:Nosuchfileordirectory

Because all asyncio subprocess functions are asynchronous and asyncioprovides many tools to work with such functions, it is easy to executeand monitor multiple subprocesses in parallel. It is indeed trivialto modify the above example to run several commands simultaneously:

asyncdefmain():awaitasyncio.gather(run('ls /zzz'),run('sleep 1; echo "hello"'))asyncio.run(main())

See also theExamples subsection.

Creating Subprocesses

asyncasyncio.create_subprocess_exec(program,*args,stdin=None,stdout=None,stderr=None,limit=None,**kwds)

Create a subprocess.

Thelimit argument sets the buffer limit forStreamReaderwrappers forstdout andstderr(ifsubprocess.PIPE is passed tostdout andstderr arguments).

Return aProcess instance.

See the documentation ofloop.subprocess_exec() for otherparameters.

Changed in version 3.10:Removed theloop parameter.

asyncasyncio.create_subprocess_shell(cmd,stdin=None,stdout=None,stderr=None,limit=None,**kwds)

Run thecmd shell command.

Thelimit argument sets the buffer limit forStreamReaderwrappers forstdout andstderr(ifsubprocess.PIPE is passed tostdout andstderr arguments).

Return aProcess instance.

See the documentation ofloop.subprocess_shell() for otherparameters.

Important

It is the application’s responsibility to ensure that all whitespace andspecial characters are quoted appropriately to avoidshell injectionvulnerabilities. Theshlex.quote() function can be used to properlyescape whitespace and special shell characters in strings that are goingto be used to construct shell commands.

Changed in version 3.10:Removed theloop parameter.

Note

Subprocesses are available for Windows if aProactorEventLoop isused. SeeSubprocess Support on Windowsfor details.

See also

asyncio also has the followinglow-level APIs to work with subprocesses:loop.subprocess_exec(),loop.subprocess_shell(),loop.connect_read_pipe(),loop.connect_write_pipe(),as well as theSubprocess TransportsandSubprocess Protocols.

Constants

asyncio.subprocess.PIPE

Can be passed to thestdin,stdout orstderr parameters.

IfPIPE is passed tostdin argument, theProcess.stdin attributewill point to aStreamWriter instance.

IfPIPE is passed tostdout orstderr arguments, theProcess.stdout andProcess.stderrattributes will point toStreamReader instances.

asyncio.subprocess.STDOUT

Special value that can be used as thestderr argument and indicatesthat standard error should be redirected into standard output.

asyncio.subprocess.DEVNULL

Special value that can be used as thestdin,stdout orstderr argumentto process creation functions. It indicates that the special fileos.devnull will be used for the corresponding subprocess stream.

Interacting with Subprocesses

Bothcreate_subprocess_exec() andcreate_subprocess_shell()functions return instances of theProcess class.Process is a high-levelwrapper that allows communicating with subprocesses and watching fortheir completion.

classasyncio.subprocess.Process

An object that wraps OS processes created by thecreate_subprocess_exec() andcreate_subprocess_shell()functions.

This class is designed to have a similar API to thesubprocess.Popen class, but there are somenotable differences:

This class isnot thread safe.

See also theSubprocess and Threadssection.

asyncwait()

Wait for the child process to terminate.

Set and return thereturncode attribute.

Note

This method can deadlock when usingstdout=PIPE orstderr=PIPE and the child process generates so much outputthat it blocks waiting for the OS pipe buffer to acceptmore data. Use thecommunicate() method when using pipesto avoid this condition.

asynccommunicate(input=None)

Interact with process:

  1. send data tostdin (ifinput is notNone);

  2. closesstdin;

  3. read data fromstdout andstderr, until EOF is reached;

  4. wait for process to terminate.

The optionalinput argument is the data (bytes object)that will be sent to the child process.

Return a tuple(stdout_data,stderr_data).

If eitherBrokenPipeError orConnectionResetErrorexception is raised when writinginput intostdin, theexception is ignored. This condition occurs when the processexits before all data are written intostdin.

If it is desired to send data to the process’stdin,the process needs to be created withstdin=PIPE. Similarly,to get anything other thanNone in the result tuple, theprocess has to be created withstdout=PIPE and/orstderr=PIPE arguments.

Note, that the data read is buffered in memory, so do not usethis method if the data size is large or unlimited.

Changed in version 3.12:stdin gets closed wheninput=None too.

send_signal(signal)

Sends the signalsignal to the child process.

Note

On Windows,SIGTERM is an alias forterminate().CTRL_C_EVENT andCTRL_BREAK_EVENT can be sent to processesstarted with acreationflags parameter which includesCREATE_NEW_PROCESS_GROUP.

terminate()

Stop the child process.

On POSIX systems this method sendsSIGTERM to thechild process.

On Windows the Win32 API functionTerminateProcess() iscalled to stop the child process.

kill()

Kill the child process.

On POSIX systems this method sendsSIGKILL to the childprocess.

On Windows this method is an alias forterminate().

stdin

Standard input stream (StreamWriter) orNoneif the process was created withstdin=None.

stdout

Standard output stream (StreamReader) orNoneif the process was created withstdout=None.

stderr

Standard error stream (StreamReader) orNoneif the process was created withstderr=None.

Warning

Use thecommunicate() method rather thanprocess.stdin.write(),awaitprocess.stdout.read() orawaitprocess.stderr.read().This avoids deadlocks due to streams pausing reading or writingand blocking the child process.

pid

Process identification number (PID).

Note that for processes created by thecreate_subprocess_shell()function, this attribute is the PID of the spawned shell.

returncode

Return code of the process when it exits.

ANone value indicates that the process has not terminated yet.

A negative value-N indicates that the child was terminatedby signalN (POSIX only).

Subprocess and Threads

Standard asyncio event loop supports running subprocesses from different threads bydefault.

On Windows subprocesses are provided byProactorEventLoop only (default),SelectorEventLoop has no subprocess support.

On UNIXchild watchers are used for subprocess finish waiting, seeProcess Watchers for more info.

Changed in version 3.8:UNIX switched to useThreadedChildWatcher for spawning subprocesses fromdifferent threads without any limitation.

Spawning a subprocess withinactive current child watcher raisesRuntimeError.

Note that alternative event loop implementations might have own limitations;please refer to their documentation.

Examples

An example using theProcess class tocontrol a subprocess and theStreamReader class to read fromits standard output.

The subprocess is created by thecreate_subprocess_exec()function:

importasyncioimportsysasyncdefget_date():code='import datetime; print(datetime.datetime.now())'# Create the subprocess; redirect the standard output# into a pipe.proc=awaitasyncio.create_subprocess_exec(sys.executable,'-c',code,stdout=asyncio.subprocess.PIPE)# Read one line of output.data=awaitproc.stdout.readline()line=data.decode('ascii').rstrip()# Wait for the subprocess exit.awaitproc.wait()returnlinedate=asyncio.run(get_date())print(f"Current date:{date}")

See also thesame examplewritten using low-level APIs.