Uh oh!
There was an error while loading.Please reload this page.
- Notifications
You must be signed in to change notification settings - Fork33.3k
gh-121313: multiprocessing: change connection buffer size to 64KiB#123559
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to ourterms of service andprivacy statement. We’ll occasionally send you account related emails.
Already on GitHub?Sign in to your account
Uh oh!
There was an error while loading.Please reload this page.
Conversation
methane commentedSep 1, 2024
Previous discussion: quick bench code: 64k vs 256k quick bench: How to change the pipe buffer size: I confirmed that this setting works by checking So I don't think having Linux pipe only optimization is worth enough. |
Uh oh!
There was an error while loading.Please reload this page.
picnixz left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
Really small nitpick. But otherwise LGTM.
Uh oh!
There was an error while loading.Please reload this page.
Co-authored-by: Bénédikt Tran <10796600+picnixz@users.noreply.github.com>
Misc/NEWS.d/next/Library/2024-07-03-10-11-53.gh-issue-121313.D7gARW.rst OutdatedShow resolvedHide resolved
Uh oh!
There was an error while loading.Please reload this page.
Uh oh!
There was an error while loading.Please reload this page.
Uh oh!
There was an error while loading.Please reload this page.
Misc/NEWS.d/next/Library/2024-07-03-10-11-53.gh-issue-121313.D7gARW.rst OutdatedShow resolvedHide resolved
Uh oh!
There was an error while loading.Please reload this page.
Co-authored-by: Victor Stinner <vstinner@python.org>
vstinner left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
LGTM
methane commentedSep 3, 2024
I tested this PR on Windows but no speedup. It is because Windows can read whole remaining data at once via this function. cpython/Lib/multiprocessing/connection.py Line 350 in7dc489b
|
Uh oh!
There was an error while loading.Please reload this page.
Windows:
Current buffer size is 8KiB since multiprocessing is introduced.
It seems small for recent Python usages.
e711caf#diff-2c54a007d7fe1d9ac5ca008fe2d054394c39a4f521eea2cb580101a284d7b7ecR28
macOS/BSD:
They use 64KiB buffer for pipes. Current 16 pages (256KiB) buffer makes ~10% slowdown compared to 64KiB on M1 mac.
Linux:
I don't have 16k/64k page Linux. But when I change the pipe buffer size via fcntl, 256KiB buffer doesn't make notable performance benefit.
64KiB seems good default buffer size.
If it is not suitable, user can try other size by changing
multiprocessing.connection.BUFSIZE.