Movatterモバイル変換

[0]ホーム
URL:

Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

 Sign up
Appearance settings

gh-129005: Remove copies from _pyio using take_bytes#141539

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to ourterms of service andprivacy statement. We’ll occasionally send you account related emails.

Already on GitHub?Sign in to your account

Merged
vstinner merged 2 commits intopython:mainfromcmaloney:gh129005_take_bytes
Nov 18, 2025

Conversation

@cmaloney
Copy link
Contributor

@cmaloneycmaloney commentedNov 13, 2025•
edited
Loading

Memory usage now matches that of _io for large files.

$ ./python -Wd -I -m test test_io.test_largefile -uall,largefile,walltime -M32G -v== CPython 3.15.0a1+ (heads/gh129005_take_bytes-dirty:d8b88498ebf, Nov 13 2025, 15:43:51) [Clang 21.1.5 ]== Linux-6.17.7-arch1-1-x86_64-with-glibc2.42 little-endian== Python build: debug== cwd: <build_dir>/build/test_python_worker_116910æ== CPU count: 32== encodings: locale=UTF-8 FS=utf-8== resources: allUsing random seed: 3652806750:00:00 load avg: 9.60 Run 1 test sequentially in a single process0:00:00 load avg: 9.60 [1/1] test_io.test_largefiletest_large_read (test.test_io.test_largefile.CLargeFileTest.test_large_read) ...  ... expected peak memory use: 2.3G ... process data size: 2.3Goktest_lseek (test.test_io.test_largefile.CLargeFileTest.test_lseek) ... oktest_osstat (test.test_io.test_largefile.CLargeFileTest.test_osstat) ... oktest_seek_read (test.test_io.test_largefile.CLargeFileTest.test_seek_read) ... oktest_seek_readall (test.test_io.test_largefile.CLargeFileTest.test_seek_readall) ...  ... expected peak memory use: 2.3G ... process data size: 2.3G ... process data size: 2.3Goktest_seekable (test.test_io.test_largefile.CLargeFileTest.test_seekable) ... oktest_truncate (test.test_io.test_largefile.CLargeFileTest.test_truncate) ... oktest_large_read (test.test_io.test_largefile.PyLargeFileTest.test_large_read) ...  ... expected peak memory use: 2.3G ... process data size: 2.3Goktest_lseek (test.test_io.test_largefile.PyLargeFileTest.test_lseek) ... oktest_osstat (test.test_io.test_largefile.PyLargeFileTest.test_osstat) ... oktest_seek_read (test.test_io.test_largefile.PyLargeFileTest.test_seek_read) ... oktest_seek_readall (test.test_io.test_largefile.PyLargeFileTest.test_seek_readall) ...  ... expected peak memory use: 2.3G ... process data size: 2.3G ... process data size: 2.3Goktest_seekable (test.test_io.test_largefile.PyLargeFileTest.test_seekable) ... oktest_truncate (test.test_io.test_largefile.PyLargeFileTest.test_truncate) ... oktest_it (test.test_io.test_largefile.TestCopyfile.test_it) ... oktest_it (test.test_io.test_largefile.TestSocketSendfile.test_it) ... ok----------------------------------------------------------------------Ran 16 tests in 6.950sOK0:00:06 load avg: 9.72 [1/1] test_io.test_largefile passed== Tests result: SUCCESS ==1 test OK.Total duration: 7.0 secTotal tests: run=16Total test files: run=1/1Result: SUCCESS

For reference; this test goes from 8.5sec overall duration on my AMD 64 bit Arch Linux box to 7 seconds.

Memory usage now matches that of _io for large files
@cmaloney
Copy link
ContributorAuthor

Optimization enabled bygh-139871 (Trying to keep that one issue referencing all the things which I use it for; planning to expand the What's New with a single "here's all the modules that got faster" list once optimize)

@cmaloneycmaloney changed the titlegh-129005: Remove copy from _pyio using take_bytesgh-129005: Remove copies from _pyio using take_bytesNov 14, 2025
returnNone
ifn<0orn>len(b):
raiseValueError(f"readinto returned{n} outside buffer size{len(b)}")
delb[n:]
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

Is there a difference betweenb.resize(n) anddel b[n:]?

Copy link
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

.resize() could grow while thedel b[n:] never will. There's some code which doesa = bytearray(b'12'); del a[3:] (intentionallydel a slice past the end) as part of its buffer management. Generally I prefer resize because there's a lot less code involved than the slice code (and the slice code eventually calls resize anyways)

vstinner reacted with thumbs up emoji
Copy link
Member

@vstinnervstinner left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

LGTM

@vstinnervstinner merged commit58f3fe0 intopython:mainNov 18, 2025
48 checks passed
@cmaloneycmaloney deleted the gh129005_take_bytes branchNovember 18, 2025 22:10
StanFromIreland pushed a commit to StanFromIreland/cpython that referenced this pull requestDec 6, 2025
Sign up for freeto join this conversation on GitHub. Already have an account?Sign in to comment

Reviewers

@vstinnervstinnervstinner approved these changes

Assignees

No one assigned

Labels

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@cmaloney@vstinner
[8]ページ先頭
©2009-2025 Movatter.jp