NotificationsYou must be signed in to change notification settings
Fork34k
Star71.2k

gh-83069: Use efficient event-driven`subprocess.Popen.wait()` on Linux / macOS / BSD#144047

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to ourterms of service andprivacy statement. We’ll occasionally send you account related emails.

Already on GitHub?Sign in to your account

Jump to bottom

Open

giampaolo wants to merge68 commits intopython:main

base:main

Choose a base branch

fromgiampaolo:subprocess-fast-wait

Open

gh-83069: Use efficient event-driven`subprocess.Popen.wait()` on Linux / macOS / BSD#144047

giampaolo wants to merge68 commits intopython:mainfromgiampaolo:subprocess-fast-wait

+301 −5

Conversation

Copy link

Contributor

giampaolo commentedJan 19, 2026•
edited by github-actionsbot
Loading

This patch replaces the busy-loop mechanism used bysubprocess.Popen.wait() on Linux, BSD, and macOS when atimeout is specified:

Linux ≥= 5.3: usesos.pidfd_open() +select.poll()
macOS, BSD: usesselect.kqueue() +KQ_FILTER_PROC +KQ_NOTE_EXIT

The kernel wakes us up exactly when the PID terminates or the timeout expires. If the fast-wait mechanisms is not available or fails (e.g. "too many open files"), the function falls back to the traditional busy loop. A similar approach was recently implemented in psutil (seegiampaolo/psutil#2706). While working on that change, I noticed that the subprocess module used essentially the same busy-loop strategy, which motivated this proposal.

Issue:Popen should use pidfd_open to implement a non-busy wait #83069

📚 Documentation preview 📚:https://cpython-previews--144047.org.readthedocs.build/

giampaolo added30 commits

January 18, 2026 20:51

Use pidfd_open() and save the FD as an attribute

288e4db

Create _busy_wait() and move code in there

6e494ca

Move existing code in new _blocking_wait() method

aaf193c

Use _wait_pidfd()

7cbb0ad

Use pidfd_open() not at class level but at method level

d9760c3

Add kqueue() implementation for macOS and BSD

5fd8eec

Add docstrings

100b111

Be conservative and check for specific error codes

74ac2f4

Document possible failures

fc0cfd6

Add missing import

f282459

Write test for pidfd_open() failing

6125976

Write test case for pidfd_open() / kqueue failing. Assert fallback is…

41dc6c0

… used.

Write test case for kqueue failing. Assert fallback is used.

73ba380

Move tests in their own class

3c8e603

Add test for terminated PID race

932ae58

Add test_kqueue_race()

bb5080a

Add test_kqueue_control_error

f067073

Add docstring

4d96c2f

Guard against possible slow test

fe05acc

Timeout: use math.ceil to avoid truncation

adb444e

Remove unused exception var

4ec17c1

Timeout: use math.ceil to avoid truncation

e807ba9

Replace _can_use_kqueue() -> _CAN_USE_KQUEUE

1ddc52b

Shorten code

645ef6c

Use waitpid() + WNOHANG even if process exited to avoid rare race

6ee771b

Revert prev change

61c6b99

Use ceil(timeout) to avoid truncation

527646d

Remove check for timeout < 0 + rm test case which didn't make sense.

0a8a1b2

Don't use _blocking_wait() as it has a while loop

73b97dc

Add docstring

2d3c3f7

giampaolo added5 commits

January 19, 2026 20:28

Try to fix doc build error 2

a916da3

Try to fix doc build error 3

86200bd

Try to fix doc build error 4

85c38bc

Try to fix doc build error 5

3c92c1d

Minor rewordings

33c8b1f

giampaolo mentioned this pull request

Jan 20, 2026

Eliminate busy-loop inProcess.wait() on Linux and BSDgiampaolo/psutil#2706

Merged

giampaolo requested review frombenjaminp andvstinner

January 20, 2026 15:39

giampaolo mentioned this pull request

Jan 20, 2026

[Linux, macOS, BSD] speedupProcess.wait()giampaolo/psutil#2705

Closed

vstinner reviewed

Jan 21, 2026

View reviewed changes

Copy link

Member

vstinner left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

Nice change! I like it. Here is a first review.

cc@gpshead

Lib/subprocess.py OutdatedShow resolvedHide resolved

Doc/library/subprocess.rst OutdatedShow resolvedHide resolved

Lib/test/test_subprocess.py OutdatedShow resolvedHide resolved

Lib/test/test_subprocess.py Outdated

Comment on lines 4193 to 4194

		assertm1.called
		assertm2.called

Copy link

Member

vstinnerJan 21, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

Suggested change

	assertm1.called
	assertm2.called
	self.assertTrue(m1.called)
	self.assertTrue(m2.called)

Copy link

Member

vstinnerJan 22, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

You don't want to replaceassert here?

Lib/test/test_subprocess.pyShow resolvedHide resolved

Lib/test/test_subprocess.py Outdated

		withself.assertRaises(subprocess.TimeoutExpired):
		p.wait(timeout=0.0001)
		self.assertEqual(p.wait(timeout=support.LONG_TIMEOUT),0)
		assertnotm.called

Copy link

Member

vstinnerJan 21, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

Suggested change

	assertnotm.called
	self.assertFalse(m.called)

Copy link

Member

vstinnerJan 22, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

You don't want to replaceassert here?

Doc/whatsnew/3.15.rst OutdatedShow resolvedHide resolved

giampaolo added5 commits

January 21, 2026 16:57

Small refact

525c047

Address review comments (bare assert, don't use ceil())

2b931b8

Address review comments (use test class VARIABLES)

d1c6e91

Address review comments (add periods in doc)

3e9d303

Merge branch 'main' into subprocess-fast-wait

a44b2d7

Copy link

ContributorAuthor

giampaolo commentedJan 21, 2026

Thanks for the review@vstinner. I addressed your comments.

vstinner reviewed

Jan 21, 2026

View reviewed changes

Lib/subprocess.py OutdatedShow resolvedHide resolved

giampaolo added4 commits

January 21, 2026 17:59

Remove weird unicode char from doc added by accident

b4f6020

Don't import select on Windows

4e17856

Make kq.control() invocation clearer by passing kwargs

6f6600d

Revert previous commit

37288c6

vstinner approved these changes

Jan 22, 2026

View reviewed changes

Copy link

Member

vstinner left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

LGTM. Thanks for the updates!

Lib/test/test_subprocess.py Outdated

Comment on lines 4193 to 4194

		assertm1.called
		assertm2.called

Copy link

Member

vstinnerJan 22, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

You don't want to replaceassert here?

Lib/test/test_subprocess.py Outdated

		withself.assertRaises(subprocess.TimeoutExpired):
		p.wait(timeout=0.0001)
		self.assertEqual(p.wait(timeout=support.LONG_TIMEOUT),0)
		assertnotm.called

Copy link

Member

vstinnerJan 22, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

You don't want to replaceassert here?

bedevere-appbot added awaiting merge and removed awaiting core review labels

Jan 22, 2026

giampaolo added2 commits

January 22, 2026 12:22

Remove bare assert statements

300692e

Merge branch 'main' into subprocess-fast-wait

7bee1e9

Copy link

ContributorAuthor

giampaolo commentedJan 22, 2026

Whops! I fixed the remaining bareassert statements.

Copy link

Member

vstinner commentedJan 22, 2026

@gpshead: Do you want to double check the change?

giampaolo mentioned this pull request

Jan 24, 2026

Extendselectors module to support platform-specific events#144211

Open

Copy link

ContributorAuthor

giampaolo commentedJan 24, 2026•
edited
Loading

In the hope of adding BSD / macOS support forasyncio in the future, I have opened#144211.

Labels

awaiting merge

Movatterモバイル変換

Uh oh!

gh-83069: Use efficient event-drivensubprocess.Popen.wait() on Linux / macOS / BSD#144047

Are you sure you want to change the base?

gh-83069: Use efficient event-drivensubprocess.Popen.wait() on Linux / macOS / BSD#144047

Conversation

giampaolo commentedJan 19, 2026• edited by github-actionsbotLoading Uh oh!There was an error while loading.Please reload this page.

Uh oh!

Uh oh!

vstinner left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

vstinnerJan 21, 2026

Choose a reason for hiding this comment

Uh oh!

vstinnerJan 22, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

vstinnerJan 21, 2026

Choose a reason for hiding this comment

Uh oh!

vstinnerJan 22, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

giampaolo commentedJan 21, 2026

Uh oh!

Uh oh!

vstinner left a comment

Choose a reason for hiding this comment

Uh oh!

vstinnerJan 22, 2026

Choose a reason for hiding this comment

Uh oh!

vstinnerJan 22, 2026

Choose a reason for hiding this comment

Uh oh!

giampaolo commentedJan 22, 2026

Uh oh!

vstinner commentedJan 22, 2026

Uh oh!

giampaolo commentedJan 24, 2026• editedLoading Uh oh!There was an error while loading.Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

gh-83069: Use efficient event-driven`subprocess.Popen.wait()` on Linux / macOS / BSD#144047

gh-83069: Use efficient event-driven`subprocess.Popen.wait()` on Linux / macOS / BSD#144047

giampaolo commentedJan 19, 2026•
edited by github-actionsbot
Loading

giampaolo commentedJan 24, 2026•
edited
Loading