Background

When using thesubprocess module on Linux,vfork() rather thanfork() has been preferred when possible for a few Python releases as is generally offers MUCH higher performance, the larger the parent process, the more cpu time it saves on page table copying.

Problem

vfork() - by design - does not return control to parent process until after the child process has successfully performed anexec() system call or has died.

Pythonsubprocess uses_posixsubprocess.fork_exec() extension module implementation to do handle async-signal-safe fork/vfork+exec code path (as that cannot safely be implemented in Python). This function does not release the GIL. Thus the GIL is held across thevfork() call. Meaning the parent process does not resume execution until the exec has succeeded, or failed and returned an error which our child writes to it'serrpipe_write fd before exiting.

This can result inall Python threads hanging (ie: the GIL is held by a blocked thread) when the childexec system call takes a long time. Such as the binary existing on a potentially slow or high latency network filesystem. (We witnessed this on a FUSE filesystem backed by remote (high latency) cloud storage with a large executable... but any slow or high latency filesystem containing an executable or anything interfering in the exec system call can cause the same problem).

Workarounds

Building withoutvfork support in_posixsubprocess or otherwise passing an arg to subprocess API that happens to disable vfork because it is incompatible with the vfork-concept are workarounds.(I won't recommend any of those because none of them are a good idea to pass when not needed for their primary purpose).

Another workaround viable in our specific case is to pre-read the executable before asking subprocess to launch it as that moves the high IO latency from theexec system call into a read without the GIL held beforehand as it pulls the executable into a fast-enough local storage cache. (That being useful at all depends on your particular filesystem latency implementation situation)

Potential Solutions

If we could simply release the GIL around thevfork() call that would seem to be ideal... It's not clear to me that we can actually do that, I'm working out what a PR would look like or what blocks us from doing so safely.

Offering yet another keyword argument to subprocess APIs to disable the use of APIs that could block the entire process uponexec() such asvfork is viable - but doesn't feel great given our existing long list of subprocess args.

Linked PRs

• gh-104372: Unlock the GIL before exec when using vfork(). #104515
• gh-104372: Cleanup _posixsubprocessmake_inheritable for async signal safety #104518
• gh-104372: Use non-Raw malloc for c_fds_to_keep in _posixsubprocess #104697
• gh-104372: Drop the GIL around the vfork() call. #104782
• [3.11] gh-104372: Cleanup _posixsubprocess make_inheritable for async signal safety gh-104518 #104785
• gh-104372: use == -1 before PyErr_Occurred #104831
• [3.12] gh-104372: use == -1 before PyErr_Occurred (GH-104831) #104833
• [3.12] gh-104372: Drop the GIL around the vfork() call. (GH-104782) #104942
• [3.11] gh-104372: Drop the GIL around the vfork() call. (#104782) #104958