Inheritance of File Descriptors

Each operating system handles the inheritance of file descriptorsdifferently. Windows creates non-inheritable handles by default, whereasUNIX and the POSIX API on Windows create inheritable file descriptors bydefault. Python prefers the POSIX API over the native Windows API, tohave a single code base and to use the same type for file descriptors,and so it creates inheritable file descriptors.

There is one exception:os.pipe() creates non-inheritable pipes onWindows, whereas it creates inheritable pipes on UNIX. The reason is animplementation artifact:os.pipe() callsCreatePipe() on Windows(native API), whereas it callspipe() on UNIX (POSIX API). The calltoCreatePipe() was added in Python in 1994, before the introductionofpipe() in the POSIX API in Windows 98. Theissue #4708 proposes to changeos.pipe() onWindows to create inheritable pipes.

Inheritance of File Descriptors on Windows

On Windows, the native type of file objects is handles (C typeHANDLE). These handles have aHANDLE_FLAG_INHERIT flag whichdefines if a handle can be inherited in a child process or not. For thePOSIX API, the C runtime (CRT) also provides file descriptors (C typeint). The handle of a file descriptor can be retrieve using thefunction_get_osfhandle(fd). A file descriptor can be created from ahandle using the function_open_osfhandle(handle).

UsingCreateProcess(),handles are only inherited if their inheritable flag(HANDLE_FLAG_INHERIT) is setand thebInheritHandlesparameter ofCreateProcess() isTRUE; all file descriptorsexcept standard streams (0, 1, 2) are closed in the child process, evenifbInheritHandles isTRUE. Using thespawnv() function, allinheritable handles and all inheritable file descriptors are inheritedin the child process. This function uses the undocumented fieldscbReserved2 andlpReserved2 of theSTARTUPINFOstructure to pass an array of file descriptors.

To replace standard streams (stdin, stdout, stderr) usingCreateProcess(), theSTARTF_USESTDHANDLES flag must be set inthedwFlags field of theSTARTUPINFO structure and thebInheritHandles parameter ofCreateProcess() must be set toTRUE. So when at least one standard stream is replaced, allinheritable handles are inherited by the child process.

The default value of theclose_fds parameter ofsubprocess processisTrue (bInheritHandles=FALSE) ifstdin,stdout andstderr parameters areNone,False (bInheritHandles=TRUE)otherwise.

See also:

• Handle Inheritance
• Stackoverflow: Can TCP SOCKET handles be set not inheritable?

Only Inherit Some Handles on Windows

Since Windows Vista,CreateProcess() supports an extension of theSTARTUPINFO structure: theSTARTUPINFOEX structure.Using this new structure, it is possible to specify a list of handles toinherit:PROC_THREAD_ATTRIBUTE_HANDLE_LIST. ReadProgrammaticallycontrolling which handles are inherited by new processes in Win32(Raymond Chen, Dec 2011) for more information.

Before Windows Vista, it is possible to make handles inheritable andcallCreateProcess() withbInheritHandles=TRUE. This optionworks if all other handles are non-inheritable. There is a racecondition: if another thread callsCreateProcess() withbInheritHandles=TRUE, handles will also be inherited in the secondprocess.

Microsoft suggests to use a lock to avoid the race condition: readQ315939: PRB: Child Inherits Unintended Handles During CreateProcessCall (last review:November 2006). ThePython issue #16500 “Add an atfork module” proposes to add such lock, it canbe used to make handles non-inheritable without the race condition. Suchlock only protects against a race condition between Python threads; Cthreads are not protected.

Another option is to duplicate handles that must be inherited, passing thevalues of the duplicated handles to the child process, so the childprocess can steal duplicated handles usingDuplicateHandle()withDUPLICATE_CLOSE_SOURCE. Handle values change between theparent and the child process because the handles are duplicated (twice);the parent and/or the child process must be adapted to handle thischange. If the child program cannot be modified, an intermediate programcan be used to steal handles from the parent process before spawning thefinal child program. The intermediate program has to pass the handle from thechild process to the parent process. The parent may have to closeduplicated handles if all handles were not stolen, for example if theintermediate process fails. If the command line is used to pass thehandle values, the command line must be modified when handles areduplicated, because their values are modified.

This PEP does not include a solution to this problem because there is noperfect solution working on all Windows versions. This point is deferreduntil use cases relying on handle or file descriptor inheritance onWindows are well known, so we can choose the best solution and carefullytest its implementation.

Inheritance of File Descriptors on UNIX

POSIX provides aclose-on-exec flag on file descriptors to automaticallyclose a file descriptor when the C functionexecv() iscalled. File descriptors with theclose-on-exec flag cleared areinherited in the child process, file descriptors with the flag set areclosed in the child process.

The flag can be set in two syscalls (one to get current flags, a secondto set new flags) usingfcntl():

intflags,res;flags=fcntl(fd,F_GETFD);if(flags==-1){/*handletheerror*/}flags|=FD_CLOEXEC;/*or"flags &= ~FD_CLOEXEC;"tocleartheflag*/res=fcntl(fd,F_SETFD,flags);if(res==-1){/*handletheerror*/}

FreeBSD, Linux, Mac OS X, NetBSD, OpenBSD and QNX also support settingthe flag in a single syscall using ioctl():

int res;res = ioctl(fd, FIOCLEX, 0);if (!res) { /* handle the error */ }

NOTE: Theclose-on-exec flag has no effect onfork(): all filedescriptors are inherited by the child process. ThePython issue #16500“Add an atfork module” proposes toadd a newatfork module to execute code at fork, which may be used toautomatically close file descriptors.

Issues with Inheritable File Descriptors

Most of the time, inheritable file descriptors “leaked” to childprocesses are not noticed, because they don’t cause major bugs. It doesnot mean that these bugs must not be fixed.

Two common issues with inherited file descriptors:

• On Windows, a directory cannot be removed before all file handles openin the directory are closed. The same issue can be seen with files,except if the file was created with theFILE_SHARE_DELETE flag(O_TEMPORARY mode foropen()).
• If a listening socket is leaked to a child process, the socket addresscannot be reused before the parent and child processes terminated. Forexample, if a web server spawns a new program to handle a process, andthe server restarts while the program is not done, the server cannotstart because the TCP port is still in use.

Example of issues in open source projects:

• Mozilla (Firefox):open since 2002-05
• dbus library:fixed in 2008-05 (dbus commit),close file descriptors in the child process
• autofs:fixed in 2009-02, set the CLOEXEC flag
• qemu:fixed in 2009-12 (qemu commit),set CLOEXEC flag
• Tor:fixed in 2010-12, set CLOEXEC flag
• OCaml: open since2011-04, “PR#5256: Processes opened using Unix.open_process* inheritall opened file descriptors (including sockets)”
• ØMQ:open since 2012-08
• Squid:open since 2012-07

See also:Excuse me son, but your code is leaking !!! (Dan Walsh, March 2012)for SELinux issues with leaked file descriptors.

Security Vulnerability

Leaking sensitive file handles and file descriptors can lead to securityvulnerabilities. An untrusted child process might read sensitive data likepasswords or take control of the parent process though a leaked filedescriptor. With a leaked listening socket, a child process can acceptnew connections to read sensitive data.

Example of vulnerabilities:

• Hijacking Apache https by mod_php (2003)
  • Apache:Apr should set FD_CLOEXEC if APR_FOPEN_NOCLEANUP is not set:fixed in 2009
  • PHP:system() (and similar) don’t cleanup opened handles of Apache: open since 2006
• CWE-403: Exposure of File Descriptor to Unintended Control Sphere (2008)
• OpenSSH Security Advisory: portable-keysign-rand-helper.adv(2011)

Read also the CERT Secure Coding Standards:FIO42-C. Ensure files are properly closed when they are no longerneeded.

Issues fixed in the subprocess module

Inherited file descriptors caused 4 issues in thesubprocessmodule:

• Issue #2320: Race condition in subprocess using stdin (opened in 2008)
• Issue #3006: subprocess.Popen causes socket to remain open afterclose (opened in 2008)
• Issue #7213: subprocess leaks open file descriptors between Popeninstances causing hangs(opened in 2009)
• Issue #12786: subprocess wait() hangs when stdin is closed (opened in 2011)

These issues were fixed in Python 3.2 by 4 different changes in thesubprocess module:

• Pipes are now non-inheritable;
• The default value of theclose_fds parameter is nowTrue,with one exception on Windows: the default value isFalse ifat least one standard stream is replaced;
• A newpass_fds parameter has been added;
• Creation of a_posixsubprocess module implemented in C.

Atomic Creation of non-inheritable File Descriptors

In a multi-threaded application, an inheritable file descriptor may becreated just before a new program is spawned, before the file descriptoris made non-inheritable. In this case, the file descriptor is leaked tothe child process. This race condition could be avoided if the filedescriptor is created directly non-inheritable.

FreeBSD, Linux, Mac OS X, Windows and many other operating systemssupport creating non-inheritable file descriptors with the inheritableflag cleared atomically at the creation of the file descriptor.

A newWSA_FLAG_NO_HANDLE_INHERIT flag forWSASocket() was addedin Windows 7 SP1 and Windows Server 2008 R2 SP1 to createnon-inheritable sockets. If this flag is used on an older Windowsversion (ex: Windows XP SP3),WSASocket() fails withWSAEPROTOTYPE.

On UNIX, new flags were added for files and sockets:

• O_CLOEXEC: available on Linux (2.6.23), FreeBSD (8.3),Mac OS 10.8, OpenBSD 5.0, Solaris 11, QNX, BeOS, next NetBSD release(6.1?). This flag is part of POSIX.1-2008.
• SOCK_CLOEXEC flag forsocket() andsocketpair(),available on Linux 2.6.27, OpenBSD 5.2, NetBSD 6.0.
• fcntl():F_DUPFD_CLOEXEC flag, available on Linux 2.6.24,OpenBSD 5.0, FreeBSD 9.1, NetBSD 6.0, Solaris 11. This flag is partof POSIX.1-2008.
• fcntl():F_DUP2FD_CLOEXEC flag, available on FreeBSD 9.1and Solaris 11.
• recvmsg():MSG_CMSG_CLOEXEC, available on Linux 2.6.23,NetBSD 6.0.

On Linux older than 2.6.23,O_CLOEXEC flag is simply ignored. Sofcntl() must be called to check if the file descriptor isnon-inheritable:O_CLOEXEC is not supported if theFD_CLOEXECflag is missing. On Linux older than 2.6.27,socket() orsocketpair() fail witherrno set toEINVAL if theSOCK_CLOEXEC flag is set in the socket type.

New functions:

• dup3(): available on Linux 2.6.27 (and glibc 2.9)
• pipe2(): available on Linux 2.6.27 (and glibc 2.9)
• accept4(): available on Linux 2.6.28 (and glibc 2.10)

On Linux older than 2.6.28,accept4() fails witherrno set toENOSYS.

Summary:

Legend:

• “Atomic File”: first version of the operating system supportingcreating atomically a non-inheritable file descriptor usingopen()
• “Atomic Socket”: first version of the operating system supportingcreating atomically a non-inheritable socket
• “X”: not supported yet

See also:

• Secure File Descriptor Handling (Ulrich Drepper,2008)
• Ghosts of Unix past, part 2: Conflated designs (Neil Brown, 2010) explains thehistory ofO_CLOEXEC andO_NONBLOCK flags
• File descriptor handling changes in 2.6.27
• FreeBSD: atomic close on exec

Status of Python 3.3

Python 3.3 creates inheritable file descriptors on all platforms, exceptos.pipe() which creates non-inheritable file descriptors on Windows.

New constants and functions related to the atomic creation ofnon-inheritable file descriptors were added to Python 3.3:os.O_CLOEXEC,os.pipe2() andsocket.SOCK_CLOEXEC.

On UNIX, thesubprocess module closes all file descriptors in thechild process by default, except standard streams (0, 1, 2) and filedescriptors of thepass_fds parameter. If theclose_fds parameter isset toFalse, all inheritable file descriptors are inherited in thechild process.

On Windows, thesubprocess closes all handles and file descriptorsin the child process by default. If at least one standard stream (stdin,stdout or stderr) is replaced (ex: redirected into a pipe), allinheritable handles and file descriptors 0, 1 and 2 are inherited in thechild process.

Using the functions of theos.execv*() andos.spawn*() families,all inheritable handles and all inheritable file descriptors areinherited by the child process.

On UNIX, themultiprocessing module usesos.fork() and so allfile descriptors are inherited by child processes.

On Windows, all inheritable handles and file descriptors 0, 1 and 2 areinherited by the child process using themultiprocessing module, allfile descriptors except standard streams are closed.

Summary:

Legend:

• “all”: allinheritable file descriptors or handles are inherited inthe child process
• “none”: all handles are closed in the child process
• “STD”: only file descriptors 0 (stdin), 1 (stdout) and 2 (stderr) areinherited in the child process
• “pass_fds”: file descriptors of thepass_fds parameter of thesubprocess are inherited
• “not applicable”: on UNIX, the multiprocessing usesfork(),so this case is not affected by this PEP.

Closing All Open File Descriptors

On UNIX, thesubprocess module closes almost all file descriptors inthe child process. This operation requires MAXFD system calls, whereMAXFD is the maximum number of file descriptors, even if there are onlyfew open file descriptors. This maximum can be read using:os.sysconf("SC_OPEN_MAX").

The operation can be slow if MAXFD is large. For example, on a FreeBSDbuildbot withMAXFD=655,000, the operation took 300 ms: seeissue #11284: slow close file descriptors.

On Linux, Python 3.3 gets the list of all open file descriptors from/proc/<PID>/fd/, and so performances depends on the number of openfile descriptors, not on MAXFD.

See also:

• Python issue #1663329:subprocess close_fds perform poor ifSC_OPEN_MAX is high
• Squid Bug #837033:Squid should set CLOEXEC on opened FDs. “32k+ close() calls in eachchild process take a long time ([12-56] seconds) in Xen PV guests.”

Non-inheritable File Descriptors

The following functions are modified to make newly created filedescriptors non-inheritable by default:

• asyncore.dispatcher.create_socket()
• io.FileIO
• io.open()
• open()
• os.dup()
• os.fdopen()
• os.open()
• os.openpty()
• os.pipe()
• select.devpoll()
• select.epoll()
• select.kqueue()
• socket.socket()
• socket.socket.accept()
• socket.socket.dup()
• socket.socket.fromfd()
• socket.socketpair()

os.dup2() still creates inheritable by default, see below.

When available, atomic flags are used to make file descriptorsnon-inheritable. The atomicity is not guaranteed because a fallback isrequired when atomic flags are not available.

New Functions And Methods

New functions available on all platforms:

• os.get_inheritable(fd:int): returnTrue if the filedescriptor can be inherited by child processes,False otherwise.
• os.set_inheritable(fd:int,inheritable:bool): set theinheritable flag of the specified file descriptor.

New functions only available on Windows:

• os.get_handle_inheritable(handle:int): returnTrue if thehandle can be inherited by child processes,False otherwise.
• os.set_handle_inheritable(handle:int,inheritable:bool):set the inheritable flag of the specified handle.

New methods:

• socket.socket.get_inheritable(): returnTrue if thesocket can be inherited by child processes,False otherwise.
• socket.socket.set_inheritable(inheritable:bool):set the inheritable flag of the specified socket.

Other Changes

On UNIX, subprocess makes file descriptors of thepass_fds parameterinheritable. The file descriptor is made inheritable in the childprocess after thefork() and beforeexecv(), so the inheritableflag of file descriptors is unchanged in the parent process.

os.dup2() has a new optionalinheritable parameter:os.dup2(fd,fd2,inheritable=True).fd2 is created inheritable by default, butnon-inheritable ifinheritable isFalse.

os.dup2() behaves differently thanos.dup() because the mostcommon use case ofos.dup2() is to replace the file descriptors ofthe standard streams:stdin (0),stdout (1) andstderr (2). Standard streams are expected to be inherited bychild processes.

Add a new open_noinherit() function

In June 2007, Henning von Bargen proposed on the python-dev mailing listto add a new open_noinherit() function to fix issues of inherited filedescriptors in child processes. At this time, the default value of theclose_fds parameter of the subprocess module wasFalse.

Read the mail thread:[Python-Dev] Proposal for a new function“open_noinherit” to avoid problems with subprocesses and security risks.

PEP 433

PEP 433, “Easier suppression of file descriptor inheritance”,was a previous attempt proposing various other alternatives, but noconsensus could be reached.