Python Enhancement Proposals

Python »
PEP Index »
PEP 324

PEP 324 – subprocess - New process module

Author:: Peter Astrand <astrand at lysator.liu.se>
Status:

Abstract

This PEP describes a new module for starting and communicatingwith processes.

Motivation

Starting new processes is a common task in any programminglanguage, and very common in a high-level language like Python.Good support for this task is needed, because:

Inappropriate functions for starting processes could mean asecurity risk: If the program is started through the shell, andthe arguments contain shell meta characters, the result can bedisastrous.[1]
It makes Python an even better replacement language forover-complicated shell scripts.

Currently, Python has a large number of different functions forprocess creation. This makes it hard for developers to choose.

The subprocess module provides the following enhancements overprevious functions:

One “unified” module provides all functionality from previousfunctions.
Cross-process exceptions: Exceptions happening in the childbefore the new process has started to execute are re-raised inthe parent. This means that it’s easy to handleexec()failures, for example. With popen2, for example, it’simpossible to detect if the execution failed.
A hook for executing custom code between fork and exec. Thiscan be used for, for example, changing uid.
No implicit call of /bin/sh. This means that there is no needfor escaping dangerous shell meta characters.
All combinations of file descriptor redirection is possible.For example, the “python-dialog”[2] needs to spawn a processand redirect stderr, but not stdout. This is not possible withcurrent functions, without using temporary files.
With the subprocess module, it’s possible to control if all openfile descriptors should be closed before the new program isexecuted.
Support for connecting several subprocesses (shell “pipe”).
Universal newline support.
Acommunicate() method, which makes it easy to send stdin dataand read stdout and stderr data, without risking deadlocks.Most people are aware of the flow control issues involved withchild process communication, but not all have the patience orskills to write a fully correct and deadlock-free select loop.This means that many Python applications contain raceconditions. Acommunicate() method in the standard librarysolves this problem.

Rationale

The following points summarizes the design:

subprocess was based on popen2, which is tried-and-tested.
The factory functions in popen2 have been removed, because Iconsider the class constructor equally easy to work with.
popen2 contains several factory functions and classes fordifferent combinations of redirection. subprocess, however,contains one single class. Since the subprocess module supports12 different combinations of redirection, providing a class orfunction for each of them would be cumbersome and not veryintuitive. Even with popen2, this is a readability problem.For example, many people cannot tell the difference betweenpopen2.popen2 and popen2.popen4 without using the documentation.
One small utility function is provided:subprocess.call(). Itaims to be an enhancement overos.system(), while still veryeasy to use:
- It does not use the Standard C function system(), which haslimitations.
- It does not call the shell implicitly.
- No need for quoting; using an argument list.
- The return value is easier to work with.
Thecall() utility function accepts an ‘args’ argument, justlike thePopen class constructor. It waits for the command tocomplete, then returns thereturncode attribute. Theimplementation is very simple:
```
defcall(*args,**kwargs):returnPopen(*args,**kwargs).wait()
```
The motivation behind thecall() function is simple: Starting aprocess and wait for it to finish is a common task.
WhilePopen supports a wide range of options, many users havesimple needs. Many people are usingos.system() today, mainlybecause it provides a simple interface. Consider this example:
```
os.system("stty sane -F "+device)
```
Withsubprocess.call(), this would look like:
```
subprocess.call(["stty","sane","-F",device])
```
or, if executing through the shell:
```
subprocess.call("stty sane -F "+device,shell=True)
```
The “preexec” functionality makes it possible to run arbitrarycode between fork and exec. One might ask why there are specialarguments for setting the environment and current directory, butnot for, for example, setting the uid. The answer is:
- Changing environment and working directory is consideredfairly common.
- Old functions likespawn() has support for an“env”-argument.
- env and cwd are considered quite cross-platform: They makesense even on Windows.
On POSIX platforms, no extension module is required: the moduleusesos.fork(),os.execvp() etc.
On Windows platforms, the module requires either Mark Hammond’sWindows extensions[5], or a small extension module called_subprocess.

Specification

This module defines one class called Popen:

classPopen(args,bufsize=0,executable=None,stdin=None,stdout=None,stderr=None,preexec_fn=None,close_fds=False,shell=False,cwd=None,env=None,universal_newlines=False,startupinfo=None,creationflags=0):

Arguments are:

args should be a string, or a sequence of program arguments.The program to execute is normally the first item in the argssequence or string, but can be explicitly set by using theexecutable argument.
On UNIX, withshell=False (default): In this case, thePopenclass usesos.execvp() to execute the child program.argsshould normally be a sequence. A string will be treated as asequence with the string as the only item (the program toexecute).
On UNIX, withshell=True: Ifargs is a string, it specifies thecommand string to execute through the shell. Ifargs is asequence, the first item specifies the command string, and anyadditional items will be treated as additional shell arguments.
On Windows: thePopen class usesCreateProcess() to execute thechild program, which operates on strings. Ifargs is asequence, it will be converted to a string using thelist2cmdline method. Please note that not all MS Windowsapplications interpret the command line the same way: Thelist2cmdline is designed for applications using the same rulesas the MS C runtime.
bufsize, if given, has the same meaning as the correspondingargument to the built-inopen() function: 0 means unbuffered, 1means line buffered, any other positive value means use a bufferof (approximately) that size. A negativebufsize means to usethe system default, which usually means fully buffered. Thedefault value forbufsize is 0 (unbuffered).
stdin,stdout andstderr specify the executed programs’ standardinput, standard output and standard error file handles,respectively. Valid values arePIPE, an existing filedescriptor (a positive integer), an existing file object, andNone.PIPE indicates that a new pipe to the child should becreated. WithNone, no redirection will occur; the child’s filehandles will be inherited from the parent. Additionally,stderrcan be STDOUT, which indicates that the stderr data from theapplications should be captured into the same file handle as forstdout.
Ifpreexec_fn is set to a callable object, this object will becalled in the child process just before the child is executed.
Ifclose_fds is true, all file descriptors except 0, 1 and 2will be closed before the child process is executed.
Ifshell is true, the specified command will be executed throughthe shell.
Ifcwd is notNone, the current directory will be changed to cwdbefore the child is executed.
Ifenv is notNone, it defines the environment variables for thenew process.
Ifuniversal_newlines is true, the file objects stdout andstderr are opened as a text file, but lines may be terminatedby any of\n, the Unix end-of-line convention,\r, theMacintosh convention or\r\n, the Windows convention. All ofthese external representations are seen as\n by the Pythonprogram. Note: This feature is only available if Python isbuilt with universal newline support (the default). Also, thenewlines attribute of the file objects stdout, stdin and stderrare not updated by thecommunicate() method.
Thestartupinfo andcreationflags, if given, will be passed tothe underlyingCreateProcess() function. They can specifythings such as appearance of the main window and priority forthe new process. (Windows only)

This module also defines two shortcut functions:

call(*args,**kwargs):
Run command with arguments. Wait for command to complete,then return thereturncode attribute.
The arguments are the same as for the Popen constructor.Example:
retcode=call(["ls","-l"])

Exceptions raised in the child process, before the new program hasstarted to execute, will be re-raised in the parent.Additionally, the exception object will have one extra attributecalled ‘child_traceback’, which is a string containing tracebackinformation from the child’s point of view.

The most common exception raised isOSError. This occurs, forexample, when trying to execute a non-existent file. Applicationsshould prepare forOSErrors.

AValueError will be raised if Popen is called with invalidarguments.

Security

Unlike some other popen functions, this implementation will nevercall /bin/sh implicitly. This means that all characters,including shell meta-characters, can safely be passed to childprocesses.

Popen objects

Instances of the Popen class have the following methods:

poll()

Check if child process has terminated. Returnsreturncodeattribute.

wait()

Wait for child process to terminate. Returnsreturncodeattribute.

communicate(input=None)

Interact with process: Send data to stdin. Read data fromstdout and stderr, until end-of-file is reached. Wait forprocess to terminate. The optional stdin argument should be astring to be sent to the child process, orNone, if no datashould be sent to the child.

communicate() returns a tuple(stdout,stderr).

Note: The data read is buffered in memory, so do not use thismethod if the data size is large or unlimited.

The following attributes are also available:

stdin: If thestdin argument isPIPE, this attribute is a file objectthat provides input to the child process. Otherwise, it isNone.
stdout: If thestdout argument isPIPE, this attribute is a fileobject that provides output from the child process.Otherwise, it isNone.
stderr: If thestderr argument isPIPE, this attribute is file objectthat provides error output from the child process. Otherwise,it isNone.
pid: The process ID of the child process.
returncode: The child return code. ANone value indicates that theprocess hasn’t terminated yet. A negative value -N indicatesthat the child was terminated by signal N (UNIX only).

Replacing older functions with the subprocess module

In this section, “a ==> b” means that b can be used as areplacement for a.

Note: All functions in this section fail (more or less) silentlyif the executed program cannot be found; this module raises anOSError exception.

In the following examples, we assume that the subprocess module isimported withfromsubprocessimport*.

Replacing /bin/sh shell backquote

output=`mycmd myarg`==>output = Popen(["mycmd", "myarg"], stdout=PIPE).communicate()[0]

Replacing shell pipe line

output=`dmesg | grep hda`==>p1 = Popen(["dmesg"], stdout=PIPE)p2 = Popen(["grep", "hda"], stdin=p1.stdout, stdout=PIPE)output = p2.communicate()[0]

Replacing`os.system()`

sts=os.system("mycmd"+" myarg")==>p=Popen("mycmd"+" myarg",shell=True)sts=os.waitpid(p.pid,0)

Note:

Calling the program through the shell is usually not required.
It’s easier to look at the returncode attribute than theexit status.

A more real-world example would look like this:

try:retcode=call("mycmd"+" myarg",shell=True)ifretcode<0:print>>sys.stderr,"Child was terminated by signal",-retcodeelse:print>>sys.stderr,"Child returned",retcodeexceptOSError,e:print>>sys.stderr,"Execution failed:",e

Replacing`os.spawn*`

P_NOWAIT example:

pid=os.spawnlp(os.P_NOWAIT,"/bin/mycmd","mycmd","myarg")==>pid=Popen(["/bin/mycmd","myarg"]).pid

P_WAIT example:

retcode=os.spawnlp(os.P_WAIT,"/bin/mycmd","mycmd","myarg")==>retcode=call(["/bin/mycmd","myarg"])

Vector example:

os.spawnvp(os.P_NOWAIT,path,args)==>Popen([path]+args[1:])

Environment example:

os.spawnlpe(os.P_NOWAIT,"/bin/mycmd","mycmd","myarg",env)==>Popen(["/bin/mycmd","myarg"],env={"PATH":"/usr/bin"})

Replacing`os.popen*`

pipe=os.popen(cmd,mode='r',bufsize)==>pipe=Popen(cmd,shell=True,bufsize=bufsize,stdout=PIPE).stdoutpipe=os.popen(cmd,mode='w',bufsize)==>pipe=Popen(cmd,shell=True,bufsize=bufsize,stdin=PIPE).stdin(child_stdin,child_stdout)=os.popen2(cmd,mode,bufsize)==>p=Popen(cmd,shell=True,bufsize=bufsize,stdin=PIPE,stdout=PIPE,close_fds=True)(child_stdin,child_stdout)=(p.stdin,p.stdout)(child_stdin,child_stdout,child_stderr)=os.popen3(cmd,mode,bufsize)==>p=Popen(cmd,shell=True,bufsize=bufsize,stdin=PIPE,stdout=PIPE,stderr=PIPE,close_fds=True)(child_stdin,child_stdout,child_stderr)=(p.stdin,p.stdout,p.stderr)(child_stdin,child_stdout_and_stderr)=os.popen4(cmd,mode,bufsize)==>p=Popen(cmd,shell=True,bufsize=bufsize,stdin=PIPE,stdout=PIPE,stderr=STDOUT,close_fds=True)(child_stdin,child_stdout_and_stderr)=(p.stdin,p.stdout)

Replacing`popen2.*`

Note: If the cmd argument topopen2 functions is a string, thecommand is executed through /bin/sh. If it is a list, the commandis directly executed.

(child_stdout,child_stdin)=popen2.popen2("somestring",bufsize,mode)==>p=Popen(["somestring"],shell=True,bufsize=bufsizestdin=PIPE,stdout=PIPE,close_fds=True)(child_stdout,child_stdin)=(p.stdout,p.stdin)(child_stdout,child_stdin)=popen2.popen2(["mycmd","myarg"],bufsize,mode)==>p=Popen(["mycmd","myarg"],bufsize=bufsize,stdin=PIPE,stdout=PIPE,close_fds=True)(child_stdout,child_stdin)=(p.stdout,p.stdin)

Thepopen2.Popen3 andpopen3.Popen4 basically works assubprocess.Popen, except that:

subprocess.Popen raises an exception if the execution fails
thecapturestderr argument is replaced with the stderr argument.
stdin=PIPE andstdout=PIPE must be specified.
popen2 closes all file descriptors by default, but you have tospecifyclose_fds=True withsubprocess.Popen.

Open Issues

Some features have been requested but is not yet implemented.This includes:

Support for managing a whole flock of subprocesses
Support for managing “daemon” processes
Built-in method for killing subprocesses

While these are useful features, it’s expected that these can beadded later without problems.

expect-like functionality, including pty support.

pty support is highly platform-dependent, which is aproblem. Also, there are already other modules that provide thiskind of functionality[6].

Backwards Compatibility

Since this is a new module, no major backward compatible issuesare expected. The module name “subprocess” might collide withother, previous modules[3] with the same name, but the name“subprocess” seems to be the best suggested name so far. Thefirst name of this module was “popen5”, but this name wasconsidered too unintuitive. For a while, the module was called“process”, but this name is already used by Trent Mick’smodule[4].

The functions and modules that this new module is trying toreplace (os.system,os.spawn*,os.popen*,popen2.*,commands.*) are expected to be available in future Python versionsfor a long time, to preserve backwards compatibility.