os.systemos.spawn*os.popen*popen2.*commands.*

17.1.1.1.Frequently Used Arguments¶

To support a wide variety of use cases, thePopen constructor (andthe convenience functions) accept a large number of optional arguments. Formost typical use cases, many of these arguments can be safely left at theirdefault values. The arguments that are most commonly needed are:

args is required for all calls and should be a string, or a sequence ofprogram arguments. Providing a sequence of arguments is generallypreferred, as it allows the module to take care of any required escapingand quoting of arguments (e.g. to permit spaces in file names). If passinga single string, eithershell must beTrue (see below) or elsethe string must simply name the program to be executed without specifyingany arguments.

stdin,stdout andstderr specify the executed program’s standard input,standard output and standard error file handles, respectively. Valid valuesarePIPE, an existing file descriptor (a positive integer), anexisting file object, andNone.PIPE indicates that a new pipeto the child should be created. With the default settings ofNone, noredirection will occur; the child’s file handles will be inherited from theparent. Additionally,stderr can beSTDOUT, which indicates thatthe stderr data from the child process should be captured into the same filehandle as for stdout.

Whenstdout orstderr are pipes anduniversal_newlines isTrue then all line endings will be converted to'\n' as describedfor theuniversal newlines'U' mode argument toopen().

Ifshell isTrue, the specified command will be executed throughthe shell. This can be useful if you are using Python primarily for theenhanced control flow it offers over most system shells and still wantconvenient access to other shell features such as shell pipes, filenamewildcards, environment variable expansion, and expansion of~ to auser’s home directory. However, note that Python itself offersimplementations of many shell-like features (in particular,glob,fnmatch,os.walk(),os.path.expandvars(),os.path.expanduser(), andshutil).

Warning

Executing shell commands that incorporate unsanitized input from anuntrusted source makes a program vulnerable toshell injection,a serious security flaw which can result in arbitrary command execution.For this reason, the use ofshell=True isstrongly discouragedin cases where the command string is constructed from external input:

>>>fromsubprocessimportcall>>>filename=input("What file would you like to display?\n")What file would you like to display?non_existent; rm -rf / #>>>call("cat "+filename,shell=True)# Uh-oh. This will end badly...

shell=False disables all shell based features, but does not sufferfrom this vulnerability; see the Note in thePopen constructordocumentation for helpful hints in gettingshell=False to work.

When usingshell=True,pipes.quote() can be used to properlyescape whitespace and shell metacharacters in strings that are going tobe used to construct shell commands.

These options, along with all of the other options, are described in moredetail in thePopen constructor documentation.

17.1.1.2.Popen Constructor¶

The underlying process creation and management in this module is handled bythePopen class. It offers a lot of flexibility so that developersare able to handle the less common cases not covered by the conveniencefunctions.

classsubprocess.Popen(args,bufsize=0,executable=None,stdin=None,stdout=None,stderr=None,preexec_fn=None,close_fds=False,shell=False,cwd=None,env=None,universal_newlines=False,startupinfo=None,creationflags=0)¶

Execute a child program in a new process. On Unix, the class usesos.execvp()-like behavior to execute the child program. On Windows,the class uses the WindowsCreateProcess() function. The arguments toPopen are as follows.

args should be a sequence of program arguments or else a single string.By default, the program to execute is the first item inargs ifargs isa sequence. Ifargs is a string, the interpretation isplatform-dependent and described below. See theshell andexecutablearguments for additional differences from the default behavior. Unlessotherwise stated, it is recommended to passargs as a sequence.

On Unix, ifargs is a string, the string is interpreted as the name orpath of the program to execute. However, this can only be done if notpassing arguments to the program.

Note

shlex.split() can be useful when determining the correcttokenization forargs, especially in complex cases:

>>>importshlex,subprocess>>>command_line=raw_input()/bin/vikings -input eggs.txt -output "spam spam.txt" -cmd "echo '$MONEY'">>>args=shlex.split(command_line)>>>printargs['/bin/vikings', '-input', 'eggs.txt', '-output', 'spam spam.txt', '-cmd', "echo '$MONEY'"]>>>p=subprocess.Popen(args)# Success!

Note in particular that options (such as-input) and arguments (suchaseggs.txt) that are separated by whitespace in the shell go in separatelist elements, while arguments that need quoting or backslash escaping whenused in the shell (such as filenames containing spaces or theecho commandshown above) are single list elements.

On Windows, ifargs is a sequence, it will be converted to a string in amanner described inConverting an argument sequence to a string on Windows. This is becausethe underlyingCreateProcess() operates on strings.

Theshell argument (which defaults toFalse) specifies whether to usethe shell as the program to execute. Ifshell isTrue, it isrecommended to passargs as a string rather than as a sequence.

On Unix withshell=True, the shell defaults to/bin/sh. Ifargs is a string, the string specifies the commandto execute through the shell. This means that the string must beformatted exactly as it would be when typed at the shell prompt. Thisincludes, for example, quoting or backslash escaping filenames with spaces inthem. Ifargs is a sequence, the first item specifies the command string, andany additional items will be treated as additional arguments to the shellitself. That is to say,Popen does the equivalent of:

Popen(['/bin/sh','-c',args[0],args[1],...])

On Windows withshell=True, theCOMSPEC environment variablespecifies the default shell. The only time you need to specifyshell=True on Windows is when the command you wish to execute is builtinto the shell (e.g.dir orcopy). You do not needshell=True to run a batch file or console-based executable.

Warning

Passingshell=True can be a security hazard if combined withuntrusted input. See the warning underFrequently Used Argumentsfor details.

bufsize, if given, has the same meaning as the corresponding argument to thebuilt-in open() function:0 means unbuffered,1 means linebuffered, any other positive value means use a buffer of (approximately) thatsize. A negativebufsize means to use the system default, which usually meansfully buffered. The default value forbufsize is0 (unbuffered).

Note

If you experience performance issues, it is recommended that you try toenable buffering by settingbufsize to either -1 or a large enoughpositive value (such as 4096).

Theexecutable argument specifies a replacement program to execute. Itis very seldom needed. Whenshell=False,executable replaces theprogram to execute specified byargs. However, the originalargs isstill passed to the program. Most programs treat the program specifiedbyargs as the command name, which can then be different from the programactually executed. On Unix, theargs namebecomes the display name for the executable in utilities such asps. Ifshell=True, on Unix theexecutable argumentspecifies a replacement shell for the default/bin/sh.

stdin,stdout andstderr specify the executed program’s standard input,standard output and standard error file handles, respectively. Valid valuesarePIPE, an existing file descriptor (a positive integer), anexisting file object, andNone.PIPE indicates that a new pipeto the child should be created. With the default settings ofNone, noredirection will occur; the child’s file handles will be inherited from theparent. Additionally,stderr can beSTDOUT, which indicates thatthe stderr data from the child process should be captured into the same filehandle as for stdout.

Ifpreexec_fn is set to a callable object, this object will be called in thechild process just before the child is executed. (Unix only)

Ifclose_fds is true, all file descriptors except0,1 and2 will be closed before the child process is executed. (Unix only).Or, on Windows, ifclose_fds is true then no handles will be inherited by thechild process. Note that on Windows, you cannot setclose_fds to true andalso redirect the standard handles by settingstdin,stdout orstderr.

Ifcwd is notNone, the child’s current directory will be changed tocwdbefore it is executed. Note that this directory is not considered whensearching the executable, so you can’t specify the program’s path relative tocwd.

Ifenv is notNone, it must be a mapping that defines the environmentvariables for the new process; these are used instead of inheriting the currentprocess’ environment, which is the default behavior.

Note

If specified,env must provide any variables requiredfor the program to execute. On Windows, in order to run aside-by-side assembly the specifiedenvmust include a validSystemRoot.

Ifuniversal_newlines isTrue, the file objectsstdout andstderrare opened as text files inuniversal newlines mode. Lines may beterminated by any of'\n', the Unix end-of-line convention,'\r',the old Macintosh convention or'\r\n', the Windows convention. All ofthese external representations are seen as'\n' by the Python program.

Note

This feature is only available if Python is built with universal newlinesupport (the default). Also, the newlines attribute of the file objectsstdout,stdin andstderr are not updated by thecommunicate() method.

If given,startupinfo will be aSTARTUPINFO object, which ispassed to the underlyingCreateProcess function.creationflags, if given, can beCREATE_NEW_CONSOLE orCREATE_NEW_PROCESS_GROUP. (Windows only)

17.1.1.3.Exceptions¶

Exceptions raised in the child process, before the new program has started toexecute, will be re-raised in the parent. Additionally, the exception objectwill have one extra attribute calledchild_traceback, which is a stringcontaining traceback information from the child’s point of view.

The most common exception raised isOSError. This occurs, for example,when trying to execute a non-existent file. Applications should prepare forOSError exceptions.

AValueError will be raised ifPopen is called with invalidarguments.

check_call() andcheck_output() will raiseCalledProcessError if the called process returns a non-zero returncode.

17.1.1.4.Security¶

Unlike some other popen functions, this implementation will never call asystem shell implicitly. This means that all characters, including shellmetacharacters, can safely be passed to child processes. Obviously, if theshell is invoked explicitly, then it is the application’s responsibility toensure that all whitespace and metacharacters are quoted appropriately.

Warning

Usecommunicate() rather than.stdin.write,.stdout.read or.stderr.read to avoiddeadlocks due to any of the other OS pipe buffers filling up and blocking thechild process.

17.1.3.1.Constants¶

Thesubprocess module exposes the following constants.

subprocess.STD_INPUT_HANDLE¶

The standard input device. Initially, this is the console input buffer,CONIN$.

subprocess.STD_OUTPUT_HANDLE¶

The standard output device. Initially, this is the active console screenbuffer,CONOUT$.

subprocess.STD_ERROR_HANDLE¶

The standard error device. Initially, this is the active console screenbuffer,CONOUT$.

subprocess.SW_HIDE¶

Hides the window. Another window will be activated.

subprocess.STARTF_USESTDHANDLES¶

Specifies that theSTARTUPINFO.hStdInput,STARTUPINFO.hStdOutput, andSTARTUPINFO.hStdError attributescontain additional information.

subprocess.STARTF_USESHOWWINDOW¶

Specifies that theSTARTUPINFO.wShowWindow attribute containsadditional information.

subprocess.CREATE_NEW_CONSOLE¶

The new process has a new console, instead of inheriting its parent’sconsole (the default).

This flag is always set whenPopen is created withshell=True.

subprocess.CREATE_NEW_PROCESS_GROUP¶

APopencreationflags parameter to specify that a new processgroup will be created. This flag is necessary for usingos.kill()on the subprocess.

This flag is ignored ifCREATE_NEW_CONSOLE is specified.

Note

All “a” functions in this section fail (more or less) silently if theexecuted program cannot be found; the “b” replacements raiseOSErrorinstead.

In addition, the replacements usingcheck_output() will fail with aCalledProcessError if the requested operation produces a non-zeroreturn code. The output is still available as theoutput attribute of the raised exception.

17.1.4.1.Replacing /bin/sh shell backquote¶

output=`mycmd myarg`

becomes:

output=check_output(["mycmd","myarg"])

17.1.4.2.Replacing shell pipeline¶

output=`dmesg| grep hda`

becomes:

p1=Popen(["dmesg"],stdout=PIPE)p2=Popen(["grep","hda"],stdin=p1.stdout,stdout=PIPE)p1.stdout.close()# Allow p1 to receive a SIGPIPE if p2 exits.output=p2.communicate()[0]

The p1.stdout.close() call after starting the p2 is important in order for p1to receive a SIGPIPE if p2 exits before p1.

Alternatively, for trusted input, the shell’s own pipeline support may stillbe used directly:

output=`dmesg| grep hda`

becomes:

output=check_output("dmesg | grep hda",shell=True)

17.1.4.3.Replacingos.system()¶

status=os.system("mycmd"+" myarg")# becomesstatus=subprocess.call("mycmd"+" myarg",shell=True)

Notes:

• Calling the program through the shell is usually not required.

A more realistic example would look like this:

try:retcode=call("mycmd"+" myarg",shell=True)ifretcode<0:print>>sys.stderr,"Child was terminated by signal",-retcodeelse:print>>sys.stderr,"Child returned",retcodeexceptOSErrorase:print>>sys.stderr,"Execution failed:",e

17.1.4.4.Replacing theos.spawn family¶

P_NOWAIT example:

pid=os.spawnlp(os.P_NOWAIT,"/bin/mycmd","mycmd","myarg")==>pid=Popen(["/bin/mycmd","myarg"]).pid

P_WAIT example:

retcode=os.spawnlp(os.P_WAIT,"/bin/mycmd","mycmd","myarg")==>retcode=call(["/bin/mycmd","myarg"])

Vector example:

os.spawnvp(os.P_NOWAIT,path,args)==>Popen([path]+args[1:])

Environment example:

os.spawnlpe(os.P_NOWAIT,"/bin/mycmd","mycmd","myarg",env)==>Popen(["/bin/mycmd","myarg"],env={"PATH":"/usr/bin"})

17.1.4.5.Replacingos.popen(),os.popen2(),os.popen3()¶

pipe=os.popen("cmd",'r',bufsize)==>pipe=Popen("cmd",shell=True,bufsize=bufsize,stdout=PIPE).stdout

pipe=os.popen("cmd",'w',bufsize)==>pipe=Popen("cmd",shell=True,bufsize=bufsize,stdin=PIPE).stdin

(child_stdin,child_stdout)=os.popen2("cmd",mode,bufsize)==>p=Popen("cmd",shell=True,bufsize=bufsize,stdin=PIPE,stdout=PIPE,close_fds=True)(child_stdin,child_stdout)=(p.stdin,p.stdout)

(child_stdin,child_stdout,child_stderr)=os.popen3("cmd",mode,bufsize)==>p=Popen("cmd",shell=True,bufsize=bufsize,stdin=PIPE,stdout=PIPE,stderr=PIPE,close_fds=True)(child_stdin,child_stdout,child_stderr)=(p.stdin,p.stdout,p.stderr)

(child_stdin,child_stdout_and_stderr)=os.popen4("cmd",mode,bufsize)==>p=Popen("cmd",shell=True,bufsize=bufsize,stdin=PIPE,stdout=PIPE,stderr=STDOUT,close_fds=True)(child_stdin,child_stdout_and_stderr)=(p.stdin,p.stdout)

On Unix, os.popen2, os.popen3 and os.popen4 also accept a sequence asthe command to execute, in which case arguments will be passeddirectly to the program without shell intervention. This usage can bereplaced as follows:

(child_stdin,child_stdout)=os.popen2(["/bin/ls","-l"],mode,bufsize)==>p=Popen(["/bin/ls","-l"],bufsize=bufsize,stdin=PIPE,stdout=PIPE)(child_stdin,child_stdout)=(p.stdin,p.stdout)

Return code handling translates as follows:

pipe=os.popen("cmd",'w')...rc=pipe.close()ifrcisnotNoneandrc>>8:print"There were some errors"==>process=Popen("cmd",shell=True,stdin=PIPE)...process.stdin.close()ifprocess.wait()!=0:print"There were some errors"

17.1.4.6.Replacing functions from thepopen2 module¶

(child_stdout,child_stdin)=popen2.popen2("somestring",bufsize,mode)==>p=Popen("somestring",shell=True,bufsize=bufsize,stdin=PIPE,stdout=PIPE,close_fds=True)(child_stdout,child_stdin)=(p.stdout,p.stdin)

On Unix, popen2 also accepts a sequence as the command to execute, inwhich case arguments will be passed directly to the program withoutshell intervention. This usage can be replaced as follows:

(child_stdout,child_stdin)=popen2.popen2(["mycmd","myarg"],bufsize,mode)==>p=Popen(["mycmd","myarg"],bufsize=bufsize,stdin=PIPE,stdout=PIPE,close_fds=True)(child_stdout,child_stdin)=(p.stdout,p.stdin)

popen2.Popen3 andpopen2.Popen4 basically work assubprocess.Popen, except that:

• Popen raises an exception if the execution fails.

• thecapturestderr argument is replaced with thestderr argument.

• stdin=PIPE andstdout=PIPE must be specified.

• popen2 closes all file descriptors by default, but you have to specifyclose_fds=True withPopen.

17.1.5.1.Converting an argument sequence to a string on Windows¶

On Windows, anargs sequence is converted to a string that can be parsedusing the following rules (which correspond to the rules used by the MS Cruntime):

1. Arguments are delimited by white space, which is either aspace or a tab.

2. A string surrounded by double quotation marks isinterpreted as a single argument, regardless of white spacecontained within. A quoted string can be embedded in anargument.

3. A double quotation mark preceded by a backslash isinterpreted as a literal double quotation mark.

4. Backslashes are interpreted literally, unless theyimmediately precede a double quotation mark.

5. If backslashes immediately precede a double quotation mark,every pair of backslashes is interpreted as a literalbackslash. If the number of backslashes is odd, the lastbackslash escapes the next double quotation mark asdescribed in rule 3.