os.systemos.spawn*

17.5.1.1. Frequently Used Arguments¶

To support a wide variety of use cases, thePopen constructor (andthe convenience functions) accept a large number of optional arguments. Formost typical use cases, many of these arguments can be safely left at theirdefault values. The arguments that are most commonly needed are:

args is required for all calls and should be a string, or a sequence ofprogram arguments. Providing a sequence of arguments is generallypreferred, as it allows the module to take care of any required escapingand quoting of arguments (e.g. to permit spaces in file names). If passinga single string, eithershell must beTrue (see below) or elsethe string must simply name the program to be executed without specifyingany arguments.

stdin,stdout andstderr specify the executed program’s standard input,standard output and standard error file handles, respectively. Valid valuesarePIPE,DEVNULL, an existing file descriptor (a positiveinteger), an existing file object, andNone.PIPE indicatesthat a new pipe to the child should be created.DEVNULL indicatesthat the special fileos.devnull will be used. With the defaultsettings ofNone, no redirection will occur; the child’s file handleswill be inherited from the parent. Additionally,stderr can beSTDOUT, which indicates that the stderr data from the childprocess should be captured into the same file handle as forstdout.

Ifuniversal_newlines isFalse the file objectsstdin,stdout andstderr will be opened as binary streams, and no line ending conversion isdone.

Ifuniversal_newlines isTrue, these file objectswill be opened as text streams inuniversal newlines modeusing the encoding returned bylocale.getpreferredencoding(False). Forstdin, line ending characters'\n' in the input will be converted to the default line separatoros.linesep. Forstdout andstderr, all line endings in theoutput will be converted to'\n'. For more information see thedocumentation of theio.TextIOWrapper class when thenewlineargument to its constructor isNone.

Note

The newlines attribute of the file objectsPopen.stdin,Popen.stdout andPopen.stderr are not updated bythePopen.communicate() method.

Ifshell isTrue, the specified command will be executed throughthe shell. This can be useful if you are using Python primarily for theenhanced control flow it offers over most system shells and still wantconvenient access to other shell features such as shell pipes, filenamewildcards, environment variable expansion, and expansion of~ to auser’s home directory. However, note that Python itself offersimplementations of many shell-like features (in particular,glob,fnmatch,os.walk(),os.path.expandvars(),os.path.expanduser(), andshutil).

Changed in version 3.3:Whenuniversal_newlines isTrue, the class uses the encodinglocale.getpreferredencoding(False)instead oflocale.getpreferredencoding(). See theio.TextIOWrapper class for more information on this change.

Warning

Executing shell commands that incorporate unsanitized input from anuntrusted source makes a program vulnerable toshell injection,a serious security flaw which can result in arbitrary command execution.For this reason, the use ofshell=True isstrongly discouragedin cases where the command string is constructed from external input:

>>>fromsubprocessimportcall>>>filename=input("What file would you like to display?\n")What file would you like to display?non_existent; rm -rf / #>>>call("cat "+filename,shell=True)# Uh-oh. This will end badly...

shell=False disables all shell based features, but does not sufferfrom this vulnerability; see the Note in thePopen constructordocumentation for helpful hints in gettingshell=False to work.

When usingshell=True,shlex.quote() can be used to properlyescape whitespace and shell metacharacters in strings that are going tobe used to construct shell commands.

These options, along with all of the other options, are described in moredetail in thePopen constructor documentation.

17.5.1.2. Popen Constructor¶

The underlying process creation and management in this module is handled bythePopen class. It offers a lot of flexibility so that developersare able to handle the less common cases not covered by the conveniencefunctions.

classsubprocess.Popen(args,bufsize=-1,executable=None,stdin=None,stdout=None,stderr=None,preexec_fn=None,close_fds=True,shell=False,cwd=None,env=None,universal_newlines=False,startupinfo=None,creationflags=0,restore_signals=True,start_new_session=False,pass_fds=())¶

Execute a child program in a new process. On Unix, the class usesos.execvp()-like behavior to execute the child program. On Windows,the class uses the WindowsCreateProcess() function. The arguments toPopen are as follows.

args should be a sequence of program arguments or else a single string.By default, the program to execute is the first item inargs ifargs isa sequence. Ifargs is a string, the interpretation isplatform-dependent and described below. See theshell andexecutablearguments for additional differences from the default behavior. Unlessotherwise stated, it is recommended to passargs as a sequence.

On Unix, ifargs is a string, the string is interpreted as the name orpath of the program to execute. However, this can only be done if notpassing arguments to the program.

Note

shlex.split() can be useful when determining the correcttokenization forargs, especially in complex cases:

>>>importshlex,subprocess>>>command_line=input()/bin/vikings -input eggs.txt -output "spam spam.txt" -cmd "echo '$MONEY'">>>args=shlex.split(command_line)>>>print(args)['/bin/vikings', '-input', 'eggs.txt', '-output', 'spam spam.txt', '-cmd', "echo '$MONEY'"]>>>p=subprocess.Popen(args)# Success!

Note in particular that options (such as-input) and arguments (suchaseggs.txt) that are separated by whitespace in the shell go in separatelist elements, while arguments that need quoting or backslash escaping whenused in the shell (such as filenames containing spaces or theecho commandshown above) are single list elements.

On Windows, ifargs is a sequence, it will be converted to a string in amanner described inConverting an argument sequence to a string on Windows. This is becausethe underlyingCreateProcess() operates on strings.

Theshell argument (which defaults toFalse) specifies whether to usethe shell as the program to execute. Ifshell isTrue, it isrecommended to passargs as a string rather than as a sequence.

On Unix withshell=True, the shell defaults to/bin/sh. Ifargs is a string, the string specifies the commandto execute through the shell. This means that the string must beformatted exactly as it would be when typed at the shell prompt. Thisincludes, for example, quoting or backslash escaping filenames with spaces inthem. Ifargs is a sequence, the first item specifies the command string, andany additional items will be treated as additional arguments to the shellitself. That is to say,Popen does the equivalent of:

Popen(['/bin/sh','-c',args[0],args[1],...])

On Windows withshell=True, theCOMSPEC environment variablespecifies the default shell. The only time you need to specifyshell=True on Windows is when the command you wish to execute is builtinto the shell (e.g.dir orcopy). You do not needshell=True to run a batch file or console-based executable.

Warning

Passingshell=True can be a security hazard if combined withuntrusted input. See the warning underFrequently Used Argumentsfor details.

bufsize will be supplied as the corresponding argument to theopen()function when creating the stdin/stdout/stderr pipe file objects:0means unbuffered (read and write are one system call and can return short),1 means line buffered, any other positive value means use a bufferof approximately that size. A negative bufsize (the default) means thesystem default of io.DEFAULT_BUFFER_SIZE will be used.

Changed in version 3.3.1:bufsize now defaults to -1 to enable buffering by default to match thebehavior that most code expects. In versions prior to Python 3.2.4 and3.3.1 it incorrectly defaulted to0 which was unbufferedand allowed short reads. This was unintentional and did not match thebehavior of Python 2 as most code expected.

Theexecutable argument specifies a replacement program to execute. Itis very seldom needed. Whenshell=False,executable replaces theprogram to execute specified byargs. However, the originalargs isstill passed to the program. Most programs treat the program specifiedbyargs as the command name, which can then be different from the programactually executed. On Unix, theargs namebecomes the display name for the executable in utilities such asps. Ifshell=True, on Unix theexecutable argumentspecifies a replacement shell for the default/bin/sh.

stdin,stdout andstderr specify the executed program’s standard input,standard output and standard error file handles, respectively. Valid valuesarePIPE,DEVNULL, an existing file descriptor (a positiveinteger), an existingfile object, andNone.PIPEindicates that a new pipe to the child should be created.DEVNULLindicates that the special fileos.devnull will be used. With thedefault settings ofNone, no redirection will occur; the child’s filehandles will be inherited from the parent. Additionally,stderr can beSTDOUT, which indicates that the stderr data from the applicationsshould be captured into the same file handle as for stdout.

Ifpreexec_fn is set to a callable object, this object will be called in thechild process just before the child is executed.(Unix only)

Warning

Thepreexec_fn parameter is not safe to use in the presence of threadsin your application. The child process could deadlock before exec iscalled.If you must use it, keep it trivial! Minimize the number of librariesyou call into.

Note

If you need to modify the environment for the child use theenvparameter rather than doing it in apreexec_fn.Thestart_new_session parameter can take the place of a previouslycommon use ofpreexec_fn to call os.setsid() in the child.

Ifclose_fds is true, all file descriptors except0,1 and2 will be closed before the child process is executed. (Unix only).The default varies by platform: Always true on Unix. On Windows it istrue whenstdin/stdout/stderr areNone, false otherwise.On Windows, ifclose_fds is true then no handles will be inherited by thechild process. Note that on Windows, you cannot setclose_fds to true andalso redirect the standard handles by settingstdin,stdout orstderr.

Changed in version 3.2:The default forclose_fds was changed fromFalse towhat is described above.

pass_fds is an optional sequence of file descriptors to keep openbetween the parent and child. Providing anypass_fds forcesclose_fds to beTrue. (Unix only)

New in version 3.2:Thepass_fds parameter was added.

Ifcwd is notNone, the function changes the working directory tocwd before executing the child. In particular, the function looks forexecutable (or for the first item inargs) relative tocwd if theexecutable path is a relative path.

Ifrestore_signals is true (the default) all signals that Python has set toSIG_IGN are restored to SIG_DFL in the child process before the exec.Currently this includes the SIGPIPE, SIGXFZ and SIGXFSZ signals.(Unix only)

Changed in version 3.2:restore_signals was added.

Ifstart_new_session is true the setsid() system call will be made in thechild process prior to the execution of the subprocess. (Unix only)

Changed in version 3.2:start_new_session was added.

Ifenv is notNone, it must be a mapping that defines the environmentvariables for the new process; these are used instead of the defaultbehavior of inheriting the current process’ environment.

Note

If specified,env must provide any variables required for the program toexecute. On Windows, in order to run aside-by-side assembly thespecifiedenvmust include a validSystemRoot.

Ifuniversal_newlines isTrue, the file objectsstdin,stdoutandstderr are opened as text streams in universal newlines mode, asdescribed above inFrequently Used Arguments, otherwise they areopened as binary streams.

If given,startupinfo will be aSTARTUPINFO object, which ispassed to the underlyingCreateProcess function.creationflags, if given, can beCREATE_NEW_CONSOLE orCREATE_NEW_PROCESS_GROUP. (Windows only)

Popen objects are supported as context managers via thewith statement:on exit, standard file descriptors are closed, and the process is waited for.

withPopen(["ifconfig"],stdout=PIPE)asproc:log.write(proc.stdout.read())

Changed in version 3.2:Added context manager support.

17.5.1.3. Exceptions¶

Exceptions raised in the child process, before the new program has started toexecute, will be re-raised in the parent. Additionally, the exception objectwill have one extra attribute calledchild_traceback, which is a stringcontaining traceback information from the child’s point of view.

The most common exception raised isOSError. This occurs, for example,when trying to execute a non-existent file. Applications should prepare forOSError exceptions.

AValueError will be raised ifPopen is called with invalidarguments.

check_call() andcheck_output() will raiseCalledProcessError if the called process returns a non-zero returncode.

All of the functions and methods that accept atimeout parameter, such ascall() andPopen.communicate() will raiseTimeoutExpired ifthe timeout expires before the process exits.

Exceptions defined in this module all inherit fromSubprocessError.

New in version 3.3:TheSubprocessError base class was added.

17.5.1.4. Security¶

Unlike some other popen functions, this implementation will never call asystem shell implicitly. This means that all characters, including shellmetacharacters, can safely be passed to child processes. Obviously, if theshell is invoked explicitly, then it is the application’s responsibility toensure that all whitespace and metacharacters are quoted appropriately.

Warning

Usecommunicate() rather than.stdin.write,.stdout.read or.stderr.read to avoiddeadlocks due to any of the other OS pipe buffers filling up and blocking thechild process.

17.5.3.1. Constants¶

Thesubprocess module exposes the following constants.

subprocess.STD_INPUT_HANDLE¶

The standard input device. Initially, this is the console input buffer,CONIN$.

subprocess.STD_OUTPUT_HANDLE¶

The standard output device. Initially, this is the active console screenbuffer,CONOUT$.

subprocess.STD_ERROR_HANDLE¶

The standard error device. Initially, this is the active console screenbuffer,CONOUT$.

subprocess.SW_HIDE¶

Hides the window. Another window will be activated.

subprocess.STARTF_USESTDHANDLES¶

Specifies that theSTARTUPINFO.hStdInput,STARTUPINFO.hStdOutput, andSTARTUPINFO.hStdError attributescontain additional information.

subprocess.STARTF_USESHOWWINDOW¶

Specifies that theSTARTUPINFO.wShowWindow attribute containsadditional information.

subprocess.CREATE_NEW_CONSOLE¶

The new process has a new console, instead of inheriting its parent’sconsole (the default).

This flag is always set whenPopen is created withshell=True.

subprocess.CREATE_NEW_PROCESS_GROUP¶

APopencreationflags parameter to specify that a new processgroup will be created. This flag is necessary for usingos.kill()on the subprocess.

This flag is ignored ifCREATE_NEW_CONSOLE is specified.

Note

All “a” functions in this section fail (more or less) silently if theexecuted program cannot be found; the “b” replacements raiseOSErrorinstead.

In addition, the replacements usingcheck_output() will fail with aCalledProcessError if the requested operation produces a non-zeroreturn code. The output is still available as theoutput attribute of the raised exception.

17.5.4.1. Replacing /bin/sh shell backquote¶

output=`mycmd myarg`# becomesoutput = check_output(["mycmd", "myarg"])

17.5.4.2. Replacing shell pipeline¶

output=`dmesg | grep hda`# becomesp1 = Popen(["dmesg"], stdout=PIPE)p2 = Popen(["grep", "hda"], stdin=p1.stdout, stdout=PIPE)p1.stdout.close()  # Allow p1 to receive a SIGPIPE if p2 exits.output = p2.communicate()[0]

The p1.stdout.close() call after starting the p2 is important in order for p1to receive a SIGPIPE if p2 exits before p1.

Alternatively, for trusted input, the shell’s own pipeline support may stillbe used directly:

output=`dmesg | grep hda`# becomesoutput=check_output("dmesg | grep hda", shell=True)

17.5.4.3. Replacingos.system()¶

sts=os.system("mycmd"+" myarg")# becomessts=call("mycmd"+" myarg",shell=True)

Notes:

• Calling the program through the shell is usually not required.

A more realistic example would look like this:

try:retcode=call("mycmd"+" myarg",shell=True)ifretcode<0:print("Child was terminated by signal",-retcode,file=sys.stderr)else:print("Child returned",retcode,file=sys.stderr)exceptOSErrorase:print("Execution failed:",e,file=sys.stderr)

17.5.4.4. Replacing theos.spawn family¶

P_NOWAIT example:

pid=os.spawnlp(os.P_NOWAIT,"/bin/mycmd","mycmd","myarg")==>pid=Popen(["/bin/mycmd","myarg"]).pid

P_WAIT example:

retcode=os.spawnlp(os.P_WAIT,"/bin/mycmd","mycmd","myarg")==>retcode=call(["/bin/mycmd","myarg"])

Vector example:

os.spawnvp(os.P_NOWAIT,path,args)==>Popen([path]+args[1:])

Environment example:

os.spawnlpe(os.P_NOWAIT,"/bin/mycmd","mycmd","myarg",env)==>Popen(["/bin/mycmd","myarg"],env={"PATH":"/usr/bin"})

17.5.4.5. Replacingos.popen(),os.popen2(),os.popen3()¶

(child_stdin,child_stdout)=os.popen2(cmd,mode,bufsize)==>p=Popen(cmd,shell=True,bufsize=bufsize,stdin=PIPE,stdout=PIPE,close_fds=True)(child_stdin,child_stdout)=(p.stdin,p.stdout)

(child_stdin,child_stdout,child_stderr)=os.popen3(cmd,mode,bufsize)==>p=Popen(cmd,shell=True,bufsize=bufsize,stdin=PIPE,stdout=PIPE,stderr=PIPE,close_fds=True)(child_stdin,child_stdout,child_stderr)=(p.stdin,p.stdout,p.stderr)

(child_stdin,child_stdout_and_stderr)=os.popen4(cmd,mode,bufsize)==>p=Popen(cmd,shell=True,bufsize=bufsize,stdin=PIPE,stdout=PIPE,stderr=STDOUT,close_fds=True)(child_stdin,child_stdout_and_stderr)=(p.stdin,p.stdout)

Return code handling translates as follows:

pipe=os.popen(cmd,'w')...rc=pipe.close()ifrcisnotNoneandrc>>8:print("There were some errors")==>process=Popen(cmd,'w',stdin=PIPE)...process.stdin.close()ifprocess.wait()!=0:print("There were some errors")

17.5.4.6. Replacing functions from thepopen2 module¶

(child_stdout,child_stdin)=popen2.popen2("somestring",bufsize,mode)==>p=Popen(["somestring"],shell=True,bufsize=bufsize,stdin=PIPE,stdout=PIPE,close_fds=True)(child_stdout,child_stdin)=(p.stdout,p.stdin)

(child_stdout,child_stdin)=popen2.popen2(["mycmd","myarg"],bufsize,mode)==>p=Popen(["mycmd","myarg"],bufsize=bufsize,stdin=PIPE,stdout=PIPE,close_fds=True)(child_stdout,child_stdin)=(p.stdout,p.stdin)

popen2.Popen3 andpopen2.Popen4 basically work assubprocess.Popen, except that:

• Popen raises an exception if the execution fails.
• thecapturestderr argument is replaced with thestderr argument.
• stdin=PIPE andstdout=PIPE must be specified.
• popen2 closes all file descriptors by default, but you have to specifyclose_fds=True withPopen to guarantee this behavior onall platforms or past Python versions.

17.5.6.1. Converting an argument sequence to a string on Windows¶

On Windows, anargs sequence is converted to a string that can be parsedusing the following rules (which correspond to the rules used by the MS Cruntime):

1. Arguments are delimited by white space, which is either aspace or a tab.
2. A string surrounded by double quotation marks isinterpreted as a single argument, regardless of white spacecontained within. A quoted string can be embedded in anargument.
3. A double quotation mark preceded by a backslash isinterpreted as a literal double quotation mark.
4. Backslashes are interpreted literally, unless theyimmediately precede a double quotation mark.
5. If backslashes immediately precede a double quotation mark,every pair of backslashes is interpreted as a literalbackslash. If the number of backslashes is odd, the lastbackslash escapes the next double quotation mark asdescribed in rule 3.