Movatterモバイル変換


[0]ホーム

URL:


Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

Sign up
Appearance settings

gh-121313: multiprocessing: change connection buffer size to 64KiB#123559

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to ourterms of service andprivacy statement. We’ll occasionally send you account related emails.

Already on GitHub?Sign in to your account

Merged
gpshead merged 9 commits intopython:mainfrommethane:mp-bufsize
Sep 3, 2024

Conversation

@methane
Copy link
Member

@methanemethane commentedSep 1, 2024
edited
Loading

Windows:

Current buffer size is 8KiB since multiprocessing is introduced.
It seems small for recent Python usages.

e711caf#diff-2c54a007d7fe1d9ac5ca008fe2d054394c39a4f521eea2cb580101a284d7b7ecR28

macOS/BSD:

They use 64KiB buffer for pipes. Current 16 pages (256KiB) buffer makes ~10% slowdown compared to 64KiB on M1 mac.

Linux:

I don't have 16k/64k page Linux. But when I change the pipe buffer size via fcntl, 256KiB buffer doesn't make notable performance benefit.

64KiB seems good default buffer size.
If it is not suitable, user can try other size by changingmultiprocessing.connection.BUFSIZE.

@methane
Copy link
MemberAuthor

Previous discussion:
#121315 (comment)

quick bench code:
https://gist.github.com/methane/a6cb799704a11f2bb2f64b16c0b830cc

64k vs 256k quick bench:

$ time python hello.py # 64kreal0m25.999suser0m9.753ssys0m29.687sreal0m25.964suser0m9.553ssys0m29.854s$ time python hello.py # 256kreal0m25.927suser0m9.421ssys0m29.958sreal0m25.933suser0m9.330ssys0m30.108s

How to change the pipe buffer size:

diff --git a/Lib/multiprocessing/connection.py b/Lib/multiprocessing/connection.pyindex ede5596e4fc..9c1bfb93728 100644--- a/Lib/multiprocessing/connection.py+++ b/Lib/multiprocessing/connection.py@@ -544,7 +544,10 @@ def Pipe(duplex=True):             c1 = Connection(s1.detach())             c2 = Connection(s2.detach())         else:+            import fcntl             fd1, fd2 = os.pipe()+            fcntl.fcntl(fd1, fcntl.F_SETPIPE_SZ, 256*1024)+            fcntl.fcntl(fd2, fcntl.F_SETPIPE_SZ, 256*1024)             c1 = Connection(fd1, writable=False)             c2 = Connection(fd2, readable=False)

I confirmed that this setting works by checkingn = len(chunk) size.

So I don't think having Linux pipe only optimization is worth enough.
Static 64KiB size seems good enough.

Copy link
Member

@picnixzpicnixz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

Really small nitpick. But otherwise LGTM.

methaneand others added3 commitsSeptember 2, 2024 19:28
Co-authored-by: Bénédikt Tran <10796600+picnixz@users.noreply.github.com>
methaneand others added2 commitsSeptember 2, 2024 20:32
Co-authored-by: Victor Stinner <vstinner@python.org>
Copy link
Member

@vstinnervstinner left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

LGTM

@methane
Copy link
MemberAuthor

I tested this PR on Windows but no speedup. It is because Windows can read whole remaining data at once via this function.

def_get_more_data(self,ov,maxsize):

Sign up for freeto join this conversation on GitHub. Already have an account?Sign in to comment

Reviewers

@picnixzpicnixzpicnixz approved these changes

@gpsheadgpsheadgpshead approved these changes

@vstinnervstinnervstinner approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@methane@gpshead@vstinner@picnixz

[8]ページ先頭

©2009-2025 Movatter.jp