Movatterモバイル変換


[0]ホーム

URL:


Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

Sign up
Appearance settings

gh-83069: Use efficient event-drivensubprocess.Popen.wait() on Linux / macOS / BSD#144047

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to ourterms of service andprivacy statement. We’ll occasionally send you account related emails.

Already on GitHub?Sign in to your account

Merged
vstinner merged 68 commits intopython:mainfromgiampaolo:subprocess-fast-wait
Jan 28, 2026
Merged
Show file tree
Hide file tree
Changes fromall commits
Commits
Show all changes
68 commits
Select commitHold shift + click to select a range
288e4db
Use pidfd_open() and save the FD as an attribute
giampaoloJan 18, 2026
6e494ca
Create _busy_wait() and move code in there
giampaoloJan 18, 2026
aaf193c
Move existing code in new _blocking_wait() method
giampaoloJan 18, 2026
7cbb0ad
Use _wait_pidfd()
giampaoloJan 18, 2026
d9760c3
Use pidfd_open() not at class level but at method level
giampaoloJan 18, 2026
5fd8eec
Add kqueue() implementation for macOS and BSD
giampaoloJan 18, 2026
100b111
Add docstrings
giampaoloJan 18, 2026
74ac2f4
Be conservative and check for specific error codes
giampaoloJan 18, 2026
fc0cfd6
Document possible failures
giampaoloJan 18, 2026
f282459
Add missing import
giampaoloJan 18, 2026
6125976
Write test for pidfd_open() failing
giampaoloJan 18, 2026
41dc6c0
Write test case for pidfd_open() / kqueue failing. Assert fallback is…
giampaoloJan 18, 2026
73ba380
Write test case for kqueue failing. Assert fallback is used.
giampaoloJan 18, 2026
3c8e603
Move tests in their own class
giampaoloJan 18, 2026
932ae58
Add test for terminated PID race
giampaoloJan 18, 2026
bb5080a
Add test_kqueue_race()
giampaoloJan 19, 2026
f067073
Add test_kqueue_control_error
giampaoloJan 19, 2026
4d96c2f
Add docstring
giampaoloJan 19, 2026
fe05acc
Guard against possible slow test
giampaoloJan 19, 2026
adb444e
Timeout: use math.ceil to avoid truncation
giampaoloJan 19, 2026
4ec17c1
Remove unused exception var
giampaoloJan 19, 2026
e807ba9
Timeout: use math.ceil to avoid truncation
giampaoloJan 19, 2026
1ddc52b
Replace _can_use_kqueue() -> _CAN_USE_KQUEUE
giampaoloJan 19, 2026
645ef6c
Shorten code
giampaoloJan 19, 2026
6ee771b
Use waitpid() + WNOHANG even if process exited to avoid rare race
giampaoloJan 19, 2026
61c6b99
Revert prev change
giampaoloJan 19, 2026
527646d
Use ceil(timeout) to avoid truncation
giampaoloJan 19, 2026
0a8a1b2
Remove check for timeout < 0 + rm test case which didn't make sense.
giampaoloJan 19, 2026
73b97dc
Don't use _blocking_wait() as it has a while loop
giampaoloJan 19, 2026
2d3c3f7
Add docstring
giampaoloJan 19, 2026
4eac42f
Rm _busy_wait()
giampaoloJan 19, 2026
3c156a9
Add comment
giampaoloJan 19, 2026
dac7d3b
Add assert
giampaoloJan 19, 2026
5c7ec2f
Update comments
giampaoloJan 19, 2026
5c29144
Add test for timeout=0
giampaoloJan 19, 2026
4359b07
Update comment about PID reuse race
giampaoloJan 19, 2026
81275c8
Update comment
giampaoloJan 19, 2026
43b500f
Handle rare case where poll() says we're done, but waitpid() doesn't
giampaoloJan 19, 2026
b64e42b
Update Doc/library/subprocess.rst
giampaoloJan 19, 2026
a101406
Add news entry
giampaoloJan 19, 2026
452f8c4
Add entry in Doc/whatsnew/3.15.rst
giampaoloJan 19, 2026
b0c9890
Fix typo
giampaoloJan 19, 2026
27b7c9f
Re-wording
giampaoloJan 19, 2026
e1da996
Raise on timeout < 0 and re-add test case
giampaoloJan 19, 2026
5c78acc
Check if can really use can_use_pidfd() in unit tests
giampaoloJan 19, 2026
df0538a
Check if can really use kqueue() in unit tests
giampaoloJan 19, 2026
6d8e36c
Pre-emptively check whether to use the fast way methods
giampaoloJan 19, 2026
6ba7465
Add test_fast_path_avoid_busy_loop
giampaoloJan 19, 2026
e3c7977
Update comments
giampaoloJan 19, 2026
97cc3be
Fix missing import on Windows
giampaoloJan 19, 2026
c4342e3
Merge branch 'main' into subprocess-fast-wait
giampaoloJan 19, 2026
5d78d24
Try to fix doc build error
giampaoloJan 19, 2026
a916da3
Try to fix doc build error 2
giampaoloJan 19, 2026
86200bd
Try to fix doc build error 3
giampaoloJan 19, 2026
85c38bc
Try to fix doc build error 4
giampaoloJan 19, 2026
3c92c1d
Try to fix doc build error 5
giampaoloJan 19, 2026
33c8b1f
Minor rewordings
giampaoloJan 19, 2026
525c047
Small refact
giampaoloJan 21, 2026
2b931b8
Address review comments (bare assert, don't use ceil())
giampaoloJan 21, 2026
d1c6e91
Address review comments (use test class VARIABLES)
giampaoloJan 21, 2026
3e9d303
Address review comments (add periods in doc)
giampaoloJan 21, 2026
a44b2d7
Merge branch 'main' into subprocess-fast-wait
giampaoloJan 21, 2026
b4f6020
Remove weird unicode char from doc added by accident
giampaoloJan 21, 2026
4e17856
Don't import select on Windows
giampaoloJan 21, 2026
6f6600d
Make kq.control() invocation clearer by passing kwargs
giampaoloJan 21, 2026
37288c6
Revert previous commit
giampaoloJan 21, 2026
300692e
Remove bare assert statements
giampaoloJan 22, 2026
7bee1e9
Merge branch 'main' into subprocess-fast-wait
giampaoloJan 22, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 18 additions & 3 deletionsDoc/library/subprocess.rst
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -803,14 +803,29 @@ Instances of the :class:`Popen` class have the following methods:

..note::

When the ``timeout`` parameter is not ``None``, then (on POSIX) the
function is implemented using a busy loop (non-blocking call and short
sleeps). Use the:mod:`asyncio` module for an asynchronous wait: see
When ``timeout`` is not ``None`` and the platform supports it, an
efficient event-driven mechanism is used to wait for process termination:

- Linux >= 5.3 uses:func:`os.pidfd_open` +:func:`select.poll`
- macOS and other BSD variants use:func:`select.kqueue` +
``KQ_FILTER_PROC`` + ``KQ_NOTE_EXIT``
- Windows uses ``WaitForSingleObject``

If none of these mechanisms are available, the function falls back to a
busy loop (non-blocking call and short sleeps).

..note::

Use the:mod:`asyncio` module for an asynchronous wait: see
:class:`asyncio.create_subprocess_exec`.

..versionchanged::3.3
*timeout* was added.

..versionchanged::3.15
if *timeout* is not ``None``, use efficient event-driven implementation
on Linux >= 5.3 and macOS / BSD.

..method::Popen.communicate(input=None, timeout=None)

Interact with process: Send data to stdin. Read data from stdout and stderr,
Expand Down
14 changes: 14 additions & 0 deletionsDoc/whatsnew/3.15.rst
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -736,6 +736,20 @@ ssl

(Contributed by Ron Frederick in :gh:`138252`.)

subprocess
----------

* :meth:`subprocess.Popen.wait`: when ``timeout`` is not ``None`` and the
platform supports it, an efficient event-driven mechanism is used to wait for
process termination:

- Linux >= 5.3 uses :func:`os.pidfd_open` + :func:`select.poll`.
- macOS and other BSD variants use :func:`select.kqueue` + ``KQ_FILTER_PROC`` + ``KQ_NOTE_EXIT``.
- Windows keeps using ``WaitForSingleObject`` (unchanged).

If none of these mechanisms are available, the function falls back to the
traditional busy loop (non-blocking call and short sleeps).
(Contributed by Giampaolo Rodola in :gh:`83069`).

sys
---
Expand Down
145 changes: 143 additions & 2 deletionsLib/subprocess.py
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -748,6 +748,60 @@ def _use_posix_spawn():
returnFalse


def_can_use_pidfd_open():
# Availability: Linux >= 5.3
ifnothasattr(os,"pidfd_open"):
returnFalse
try:
pidfd=os.pidfd_open(os.getpid(),0)
exceptOSErroraserr:
iferr.errnoin {errno.EMFILE,errno.ENFILE}:
# transitory 'too many open files'
returnTrue
# likely blocked by security policy like SECCOMP (EPERM,
# EACCES, ENOSYS)
returnFalse
else:
os.close(pidfd)
returnTrue


def_can_use_kqueue():
# Availability: macOS, BSD
names= (
"kqueue",
"KQ_EV_ADD",
"KQ_EV_ONESHOT",
"KQ_FILTER_PROC",
"KQ_NOTE_EXIT",
)
ifnotall(hasattr(select,x)forxinnames):
returnFalse
kq=None
try:
kq=select.kqueue()
kev=select.kevent(
os.getpid(),
filter=select.KQ_FILTER_PROC,
flags=select.KQ_EV_ADD|select.KQ_EV_ONESHOT,
fflags=select.KQ_NOTE_EXIT,
)
kq.control([kev],1,0)
returnTrue
exceptOSErroraserr:
iferr.errnoin {errno.EMFILE,errno.ENFILE}:
# transitory 'too many open files'
returnTrue
returnFalse
finally:
ifkqisnotNone:
kq.close()


_CAN_USE_PIDFD_OPEN=not_mswindowsand_can_use_pidfd_open()
_CAN_USE_KQUEUE=not_mswindowsand_can_use_kqueue()


# These are primarily fail-safe knobs for negatives. A True value does not
# guarantee the given libc/syscall API will be used.
_USE_POSIX_SPAWN=_use_posix_spawn()
Expand DownExpand Up@@ -2046,14 +2100,100 @@ def _try_wait(self, wait_flags):
sts=0
return (pid,sts)

def_wait_pidfd(self,timeout):
"""Wait for PID to terminate using pidfd_open() + poll().
Linux >= 5.3 only.
"""
ifnot_CAN_USE_PIDFD_OPEN:
returnFalse
try:
pidfd=os.pidfd_open(self.pid,0)
exceptOSError:
# May be:
# - ESRCH: no such process
# - EMFILE, ENFILE: too many open files (usually 1024)
# - ENODEV: anonymous inode filesystem not supported
# - EPERM, EACCES, ENOSYS: undocumented; may happen if
# blocked by security policy like SECCOMP
returnFalse

try:
poller=select.poll()
poller.register(pidfd,select.POLLIN)
events=poller.poll(timeout*1000)
ifnotevents:
raiseTimeoutExpired(self.args,timeout)
returnTrue
finally:
os.close(pidfd)

def_wait_kqueue(self,timeout):
"""Wait for PID to terminate using kqueue(). macOS and BSD only."""
ifnot_CAN_USE_KQUEUE:
returnFalse
try:
kq=select.kqueue()
exceptOSError:
# likely EMFILE / ENFILE (too many open files)
returnFalse

try:
kev=select.kevent(
self.pid,
filter=select.KQ_FILTER_PROC,
flags=select.KQ_EV_ADD|select.KQ_EV_ONESHOT,
fflags=select.KQ_NOTE_EXIT,
)
try:
events=kq.control([kev],1,timeout)# wait
exceptOSError:
returnFalse
else:
ifnotevents:
raiseTimeoutExpired(self.args,timeout)
returnTrue
finally:
kq.close()

def_wait(self,timeout):
"""Internal implementation of wait() on POSIX."""
"""Internal implementation of wait() on POSIX.
Uses efficient pidfd_open() + poll() on Linux or kqueue()
on macOS/BSD when available. Falls back to polling
waitpid(WNOHANG) otherwise.
"""
ifself.returncodeisnotNone:
returnself.returncode

iftimeoutisnotNone:
endtime=_time()+timeout
iftimeout<0:
raiseTimeoutExpired(self.args,timeout)
started=_time()
endtime=started+timeout

# Try efficient wait first.
ifself._wait_pidfd(timeout)orself._wait_kqueue(timeout):
# Process is gone. At this point os.waitpid(pid, 0)
# will return immediately, but in very rare races
# the PID may have been reused.
# os.waitpid(pid, WNOHANG) ensures we attempt a
# non-blocking reap without blocking indefinitely.
withself._waitpid_lock:
ifself.returncodeisnotNone:
returnself.returncode# Another thread waited.
(pid,sts)=self._try_wait(os.WNOHANG)
assertpid==self.pidorpid==0
ifpid==self.pid:
self._handle_exitstatus(sts)
returnself.returncode
# os.waitpid(pid, WNOHANG) returned 0 instead
# of our PID, meaning PID has not yet exited,
# even though poll() / kqueue() said so. Very
# rare and mostly theoretical. Fallback to busy
# polling.
elapsed=_time()-started
endtime-=elapsed

# Enter a busy loop if we have a timeout. This busy loop was
# cribbed from Lib/threading.py in Thread.wait() at r71065.
delay=0.0005# 500 us -> initial delay of 1 ms
Expand DownExpand Up@@ -2085,6 +2225,7 @@ def _wait(self, timeout):
# http://bugs.python.org/issue14396.
ifpid==self.pid:
self._handle_exitstatus(sts)

returnself.returncode


Expand Down
119 changes: 119 additions & 0 deletionsLib/test/test_subprocess.py
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -1423,6 +1423,8 @@ def test_wait(self):
deftest_wait_timeout(self):
p=subprocess.Popen([sys.executable,
"-c","import time; time.sleep(0.3)"])
withself.assertRaises(subprocess.TimeoutExpired)asc:
p.wait(timeout=0)
withself.assertRaises(subprocess.TimeoutExpired)asc:
p.wait(timeout=0.0001)
self.assertIn("0.0001",str(c.exception))# For coverage of __str__.
Expand DownExpand Up@@ -4094,5 +4096,122 @@ def test_broken_pipe_cleanup(self):
self.assertTrue(proc.stdin.closed)



classFastWaitTestCase(BaseTestCase):
"""Tests for efficient (pidfd_open() + poll() / kqueue()) process
waiting in subprocess.Popen.wait().
"""
CAN_USE_PIDFD_OPEN=subprocess._CAN_USE_PIDFD_OPEN
CAN_USE_KQUEUE=subprocess._CAN_USE_KQUEUE
COMMAND= [sys.executable,"-c","import time; time.sleep(0.3)"]
WAIT_TIMEOUT=0.0001# 0.1 ms

defassert_fast_waitpid_error(self,patch_point):
# Emulate a case where pidfd_open() or kqueue() fails.
# Busy-poll wait should be used as fallback.
exc=OSError(errno.EMFILE,os.strerror(errno.EMFILE))
withmock.patch(patch_point,side_effect=exc)asm:
p=subprocess.Popen(self.COMMAND)
withself.assertRaises(subprocess.TimeoutExpired):
p.wait(self.WAIT_TIMEOUT)
self.assertEqual(p.wait(timeout=support.SHORT_TIMEOUT),0)
self.assertTrue(m.called)

@unittest.skipIf(notCAN_USE_PIDFD_OPEN,reason="needs pidfd_open()")
deftest_wait_pidfd_open_error(self):
self.assert_fast_waitpid_error("os.pidfd_open")

@unittest.skipIf(notCAN_USE_KQUEUE,reason="needs kqueue() for proc")
deftest_wait_kqueue_error(self):
self.assert_fast_waitpid_error("select.kqueue")

@unittest.skipIf(notCAN_USE_KQUEUE,reason="needs kqueue() for proc")
deftest_kqueue_control_error(self):
# Emulate a case where kqueue.control() fails. Busy-poll wait
# should be used as fallback.
p=subprocess.Popen(self.COMMAND)
kq_mock=mock.Mock()
kq_mock.control.side_effect=OSError(
errno.EPERM,os.strerror(errno.EPERM)
)
kq_mock.close=mock.Mock()

withmock.patch("select.kqueue",return_value=kq_mock)asm:
withself.assertRaises(subprocess.TimeoutExpired):
p.wait(self.WAIT_TIMEOUT)
self.assertEqual(p.wait(timeout=support.SHORT_TIMEOUT),0)
self.assertTrue(m.called)

defassert_wait_race_condition(self,patch_target,real_func):
# Call pidfd_open() / kqueue(), then terminate the process.
# Make sure that the wait call (poll() / kqueue.control())
# still works for a terminated PID.
p=subprocess.Popen(self.COMMAND)

defwrapper(*args,**kwargs):
ret=real_func(*args,**kwargs)
try:
os.kill(p.pid,signal.SIGTERM)
os.waitpid(p.pid,0)
exceptOSError:
pass
returnret

withmock.patch(patch_target,side_effect=wrapper)asm:
status=p.wait(timeout=support.SHORT_TIMEOUT)
self.assertTrue(m.called)
self.assertEqual(status,0)

@unittest.skipIf(notCAN_USE_PIDFD_OPEN,reason="needs pidfd_open()")
deftest_pidfd_open_race(self):
self.assert_wait_race_condition("os.pidfd_open",os.pidfd_open)

@unittest.skipIf(notCAN_USE_KQUEUE,reason="needs kqueue() for proc")
deftest_kqueue_race(self):
self.assert_wait_race_condition("select.kqueue",select.kqueue)

defassert_notification_without_immediate_reap(self,patch_target):
# Verify fallback to busy polling when poll() / kqueue()
# succeeds, but waitpid(pid, WNOHANG) returns (0, 0).
defwaitpid_wrapper(pid,flags):
nonlocalncalls
ncalls+=1
ifncalls==1:
return (0,0)
returnreal_waitpid(pid,flags)

ncalls=0
real_waitpid=os.waitpid
withmock.patch.object(subprocess.Popen,patch_target,return_value=True)asm1:
withmock.patch("os.waitpid",side_effect=waitpid_wrapper)asm2:
p=subprocess.Popen(self.COMMAND)
withself.assertRaises(subprocess.TimeoutExpired):
p.wait(self.WAIT_TIMEOUT)
self.assertEqual(p.wait(timeout=support.SHORT_TIMEOUT),0)
self.assertTrue(m1.called)
self.assertTrue(m2.called)

@unittest.skipIf(notCAN_USE_PIDFD_OPEN,reason="needs pidfd_open()")
deftest_pidfd_open_notification_without_immediate_reap(self):
self.assert_notification_without_immediate_reap("_wait_pidfd")

@unittest.skipIf(notCAN_USE_KQUEUE,reason="needs kqueue() for proc")
deftest_kqueue_notification_without_immediate_reap(self):
self.assert_notification_without_immediate_reap("_wait_kqueue")

@unittest.skipUnless(
CAN_USE_PIDFD_OPENorCAN_USE_KQUEUE,
"fast wait mechanism not available"
)
deftest_fast_path_avoid_busy_loop(self):
# assert that the busy loop is not called as long as the fast
# wait is available
withmock.patch('time.sleep')asm:
p=subprocess.Popen(self.COMMAND)
withself.assertRaises(subprocess.TimeoutExpired):
p.wait(self.WAIT_TIMEOUT)
self.assertEqual(p.wait(timeout=support.LONG_TIMEOUT),0)
self.assertFalse(m.called)

if__name__=="__main__":
unittest.main()
View file
Open in desktop
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
:meth:`subprocess.Popen.wait`: when ``timeout`` is not ``None``, an efficient
event-driven mechanism now waits for process termination, if available. Linux
>= 5.3 uses:func:`os.pidfd_open` +:func:`select.poll`. macOS and other BSD
variants use:func:`select.kqueue` + ``KQ_FILTER_PROC`` + ``KQ_NOTE_EXIT``.
Windows keeps using ``WaitForSingleObject`` (unchanged). If none of these
mechanisms are available, the function falls back to the traditional busy loop
(non-blocking call and short sleeps). Patch by Giampaolo Rodola.
Loading

[8]ページ先頭

©2009-2026 Movatter.jp