Movatterモバイル変換


[0]ホーム

URL:


Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

Sign up
Appearance settings

gh-117151: IO performance improvement, increase io.DEFAULT_BUFFER_SIZE to 128k#118144

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to ourterms of service andprivacy statement. We’ll occasionally send you account related emails.

Already on GitHub?Sign in to your account

Merged
gpshead merged 19 commits intopython:mainfromman-group:io-buffer-size
Mar 7, 2025

Conversation

@morotti
Copy link
Contributor

@morottimorotti commentedApr 22, 2024
edited
Loading

See discussiongh-117151

This patch adjusts the buffer size. That gives 3 to 5 times I/O performance improvement on modern hardware.

  • increase io.DEFAULT_BUFFER_SIZE to 128k
  • fix open() to use max(st_blksize, io.DEFAULT_BUFFER_SIZE)

edgarrmondragon, Zaczero, erlend-aasland, and guptarohit reacted with thumbs up emojinetwork-shark reacted with rocket emoji
@ghost
Copy link

ghost commentedApr 22, 2024
edited by ghost
Loading

All commit authors signed the Contributor License Agreement.
CLA signed

@bedevere-app
Copy link

Most changes to Pythonrequire a NEWS entry. Add one using theblurb_it web app or theblurb command-line tool.

If this change has little impact on Python users, wait for a maintainer to apply theskip news label instead.

@encukouencukou added the performancePerformance or resource usage labelApr 22, 2024
@bedevere-app
Copy link

Most changes to Pythonrequire a NEWS entry. Add one using theblurb_it web app or theblurb command-line tool.

If this change has little impact on Python users, wait for a maintainer to apply theskip news label instead.

@morotti
Copy link
ContributorAuthor

@serhiy-storchaka hello, you were kind to review my first PR for buffering performance (118037), would you be able to review this PR too?

@eendebakpt
Copy link
Contributor

eendebakpt commentedApr 30, 2024
edited
Loading

@morotti You still need to write a news entry for the PR. For your other PR this is was not required, but due to the performance impact of this PR I think we should. The most convenient way for this is to click on theDetails button next tobedevere/news of the CI checks which will open a tool to add the news entry. For more information also seehttps://devguide.python.org/core-developers/committing/#updating-news-and-what-s-new-in-python

The CLA is not yet working, but it worked for the other PR (which was created after this one), so maybe it will resolve itself automatically in a couple of days

@eendebakpt
Copy link
Contributor

Pinging@benjaminp as expert on io. Could you review this PR? Are there other benchmarks (besides the one in the corresponding issue) we can perform to test this PR?

@morotti
Copy link
ContributorAuthor

Thank you for reviewing,

The CLA check is green now and I added a news entry.

@morotti
Copy link
ContributorAuthor

@eendebakpt@benjaminp could you review?

@eendebakpt
Copy link
Contributor

@eendebakpt@benjaminp could you review?

@morotti The PR looks good from my side, but I am no core dev so I cannot approve. Currently many core devs are at pycon US, so I think we should wait a bit more. If in a few weeks there has been no further response, we can post a message to discourse (seehttps://devguide.python.org/getting-started/pull-request-lifecycle/#reviewing).

@itamaroitamaro added the 🔨 test-with-buildbotsTest PR w/ buildbots; report in status section labelMay 23, 2024
@bedevere-bot
Copy link

🤖 New build scheduled with the buildbot fleet by@itamaro for commit3668d1a 🤖

If you want to schedule another build, you need to add the🔨 test-with-buildbots label again.

@bedevere-botbedevere-bot removed the 🔨 test-with-buildbotsTest PR w/ buildbots; report in status section labelMay 23, 2024
@itamaroitamaro added the 🔨 test-with-buildbotsTest PR w/ buildbots; report in status section labelJun 1, 2024
@bedevere-bot
Copy link

🤖 New build scheduled with the buildbot fleet by@itamaro for commit28e7bb7 🤖

If you want to schedule another build, you need to add the🔨 test-with-buildbots label again.

@bedevere-botbedevere-bot removed the 🔨 test-with-buildbotsTest PR w/ buildbots; report in status section labelJun 1, 2024
@erlend-aasland
Copy link
Contributor

Requesting Serhiy's review, since he reviewd the other linked (and merged) PR#118037.

Copy link
Member

@serhiy-storchakaserhiy-storchaka left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

I was convinced that such a change could be beneficial.

But for the case if it has unexpected effect, please ask onhttps://discuss.python.org/. Update also the C implementation ofopen() which is used by default.

@serhiy-storchaka
Copy link
Member

Please do not use rebase and force-push. It makes reviewing more difficult.

@morotti
Copy link
ContributorAuthor

alright, I've removed the constants _MAXIMUM_BUFFER_SIZE. build green.

I think we're good to merge

eendebakpt reacted with hooray emoji

@erlend-aasland
Copy link
Contributor

(rebasing on master again because CI builds no longer work)

Please usegit merge --no-ff main instead of rebasing. This has already been pointed out in earlier review.

@morotti
Copy link
ContributorAuthor

morotti commentedFeb 11, 2025
edited
Loading

I've updated the branch to main, anything else you want?

@morotti
Copy link
ContributorAuthor

@cmaloney@gpshead@erlend-aasland the PR is approved and passing builds. can this be merged?

@bedevere-app
Copy link

A Python core developer has requested some changes be made to your pull request before we can consider merging it. If you could please address their requests along with any other requests in other reviews from core developers that would be appreciated.

Once you have made the requested changes, please leave a comment on this pull request containing the phraseI have made the requested changes; please review again. I will then notify any core developers who have left a review that you're ready for them to take another look at this pull request.

@morotti
Copy link
ContributorAuthor

morotti commentedFeb 20, 2025
edited
Loading

@gpshead I've made the changes, would you be able to review again?

@morotti
Copy link
ContributorAuthor

hello@gpshead@cmaloney@serhiy-storchaka@eendebakpt would you be able to review and unblock the PR?

gpshead reacted with thumbs up emoji

Copy link
Member

@gpsheadgpshead left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

the change looks good, but this PR was created with the checkbox allowing committers to make direct edits to the PR branch disabled so we can't take care of trivia that may be blocking it ourselves.

can you merge master so that it reruns modern CI?

@morotti
Copy link
ContributorAuthor

@gpshead merged with master.

sorry, I haven't found any checkbox to allow to make direct edits or I would have ticket. I suspect it's because the fork is in my work organization, maybe that comes with extra restrictions.

gpshead reacted with thumbs up emoji

@gpsheadgpshead merged commitb1b4f96 intopython:mainMar 7, 2025
42 checks passed
seehwan pushed a commit to seehwan/cpython that referenced this pull requestApr 16, 2025
…ER_SIZE to 128k (pythonGH-118144)Co-authored-by: rmorotti <romain.morotti@man.com>
Sign up for freeto join this conversation on GitHub. Already have an account?Sign in to comment

Reviewers

@gpsheadgpsheadgpshead approved these changes

@eendebakpteendebakpteendebakpt left review comments

@serhiy-storchakaserhiy-storchakaserhiy-storchaka left review comments

@cmaloneycmaloneycmaloney approved these changes

Assignees

No one assigned

Labels

performancePerformance or resource usage

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

10 participants

@morotti@eendebakpt@bedevere-bot@erlend-aasland@serhiy-storchaka@cmaloney@gpshead@itamaro@encukou@rmmancom

[8]ページ先頭

©2009-2025 Movatter.jp