Movatterモバイル変換

Copy link

bedevere-bot commentedJan 22, 2023

Most changes to Pythonrequire a NEWS entry.

Please add it using theblurb_it web app or theblurb command-line tool.

CCLDArjun changed the title~~Buffer gzip writes to reduce execution time by ~15%~~gh-89550: Buffer GzipFile.write to reduce execution time by ~15%

bedevere-bot mentioned this pull request

GzipFile.write should be buffered#89550

Closed

AlexWaygood added the performancePerformance or resource usage label

Buffer compression writerspycompression/xopen#78

news blurb

bc4d929

Copy link

Contributor

rhpvorderman commentedJan 24, 2023

Wow@CCLDArjun such a simple and effective solution. My solution was to split out a _GzipWriter (just like _GzipReader) and try to work from there, but I never got that to work easily and it was quite some extra code too. I am so glad you made this. I hope this will get merged soon!

Copy link

ContributorAuthor

CCLDArjun commentedJan 24, 2023

My solution was to split out a _GzipWriter (just like _GzipReader) and try to work from there

Initially that was my idea too, where I subclassed_pyio.BufferedWriter to override its flush methods but that also wasn't working easily and was pretty messy:943ca9c

Copy link

Contributor

rhpvorderman commentedJan 25, 2023

I am glad that you were able to come up with a better idea. I have been hacking at the gzip module pretty thoroughly but I could not come up with this. I am in awe 😀 . I will port this topython-isal and I have recently started on making azlib-ng module too. That way people will not have to wait for 3.12 to ship to get all this fast compression.

rhpvorderman mentioned this pull request

Jan 25, 2023

Merged

Merge branch 'main' into buffered-gzip-writes

ef9dd50

Copy link

Member

orsenthil commentedJan 27, 2023

This looks good to me. A core developer in the expert list will review before merging.

This was referencedFeb 6, 2023

Buffer writing gzipfilepycompression/python-isal#131

Closed

Buffered writing for GzipFilepycompression/python-zlib-ng#8

Closed

Copy link

Contributor

rhpvorderman commentedFeb 8, 2023

@orsenthil This is still labeled "awaiting review" rather then "awaiting core review" does that matter?

Copy link

Member

arhadthedev commentedFeb 8, 2023

@rhpvorderman Only review messages from the Python team count.

@Yhg1s,@gpshead as zipfile experts (not quite the same but still).

arhadthedev added the stdlibPython modules in the Lib dir label

Feb 8, 2023

svisser mentioned this pull request

Feb 17, 2023

Slow performance due to lack of buffering for GzipFile.writepiskvorky/smart_open#760

Open

gpshead self-requested a review

February 17, 2023 23:54

gpshead requested changes

Feb 18, 2023

View reviewed changes

Copy link

Member

gpshead left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

Overall I like this change. It's great how cleanly this can be added. :)

Lib/gzip.py OutdatedShow resolvedHide resolved

Lib/gzip.pyShow resolvedHide resolved

bedevere-bot added awaiting changes and removed awaiting review labels

Feb 18, 2023

Copy link

bedevere-bot commentedFeb 18, 2023

A Python core developer has requested some changes be made to your pull request before we can consider merging it. If you could please address their requests along with any other requests in other reviews from core developers that would be appreciated.

Once you have made the requested changes, please leave a comment on this pull request containing the phraseI have made the requested changes; please review again. I will then notify any core developers who have left a review that you're ready for them to take another look at this pull request.

gpshead self-assigned this

Feb 18, 2023

CCLDArjun added2 commits

February 17, 2023 23:15

added buffer_size for file writes

b3aa1f7

:Merge branch 'buffered-gzip-writes' ofhttps://github.com/ccldarjun/…

ad2dc9b

…cpython into buffered-gzip-writes

Copy link

ContributorAuthor

CCLDArjun commentedFeb 22, 2023

@gpshead tests are passing! Can you re review?

svisser reviewed

Feb 23, 2023

View reviewed changes

Lib/gzip.pyShow resolvedHide resolved

CCLDArjun added2 commits

February 23, 2023 14:54

use constant for write buffer size

fe31588

Merge branch 'main' into buffered-gzip-writes

875ebd8

Copy link

Contributor

rhpvorderman commentedMay 8, 2023

ping

It would be great if this gets into 3.12. The measured execution difference of 15%is at gzip level 9 which is the default. At a more modest compression level such as 1, the difference can be a couple of hundreds of percents.

Merge branch 'main' into buffered-gzip-writes

e33fc19

Copy link

ContributorAuthor

CCLDArjun commentedMay 8, 2023

pinging@gpshead for a final review

improve news wording and ReSTify

5ef4674

gpshead approved these changes

View reviewed changes

bedevere-bot added awaiting merge and removed awaiting change review labels

Merge branch 'main' into buffered-gzip-writes

c0f3d2a

gpsheadenabled auto-merge (squash)

May 8, 2023 17:29

gpshead merged commit9af4854 intopython:main

bedevere-bot removed the awaiting merge label

jbower-fb pushed a commit to jbower-fb/cpython that referenced this pull request

pythongh-89550: Buffer GzipFile.write to reduce execution time by ~15% (

a49c8a8

python#101251)Use `io.BufferedWriter` to buffer gzip writes.---------Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>Co-authored-by: Gregory P. Smith <greg@krypto.org>

carljm added a commit to carljm/cpython that referenced this pull request

Merge branch 'main' into inlinecomp2

76077cd

* main: (47 commits)pythongh-97696 Remove unnecessary check for eager_start kwarg (python#104188)pythonGH-104308: socket.getnameinfo should release the GIL (python#104307)pythongh-104310: Add importlib.util.allowing_all_extensions() (pythongh-104311)pythongh-99113: A Per-Interpreter GIL! (pythongh-104210)pythonGH-104284: Fix documentation gettext build (python#104296)pythongh-89550: Buffer GzipFile.write to reduce execution time by ~15% (python#101251)pythongh-104223: Fix issues with inheriting from buffer classes (python#104227)pythongh-99108: fix typo in Modules/Setup (python#104293)pythonGH-104145: Use fully-qualified cross reference types for the bisect module (python#104172)pythongh-103193: Improve `getattr_static` test coverage (python#104286)  Trim trailing whitespace and test on CI (python#104275)pythongh-102500: Remove mention of bytes shorthand (python#104281)pythongh-97696: Improve and fix documentation for asyncio eager tasks (python#104256)pythongh-99108: Replace SHA3 implementation HACL* version (python#103597)pythongh-104273: Remove redundant len() calls in argparse function (python#104274)pythongh-64660: Don't hardcode Argument Clinic return converter result variable name (python#104200)pythongh-104265 Disallow instantiation of `_csv.Reader` and `_csv.Writer` (python#104266)pythonGH-102613: Improve performance of `pathlib.Path.rglob()` (pythonGH-104244)pythongh-103650: Fix perf maps address format (python#103651)pythonGH-89812: Churn `pathlib.Path` methods (pythonGH-104243)  ...

carljm added a commit to carljm/cpython that referenced this pull request

Merge branch 'main' into fixsendfamily

a614216

* main: (29 commits)pythongh-104276: Make `_struct.unpack_iterator` type use type flag instead of custom constructor (python#104277)pythongh-97696: Move around and update the whatsnew entry for asyncio eager task factory (python#104298)pythongh-103193: Fix refleaks in `test_inspect` and `test_typing` (python#104320)  require-pr-label.yml: Add missing "permissions:" (python#104309)pythongh-90656: Add platform triplets for 64-bit LoongArch (LA64) (python#30939)pythongh-104180: Read SOCKS proxies from macOS System Configuration (python#104181)pythongh-97696 Remove unnecessary check for eager_start kwarg (python#104188)pythonGH-104308: socket.getnameinfo should release the GIL (python#104307)pythongh-104310: Add importlib.util.allowing_all_extensions() (pythongh-104311)pythongh-99113: A Per-Interpreter GIL! (pythongh-104210)pythonGH-104284: Fix documentation gettext build (python#104296)pythongh-89550: Buffer GzipFile.write to reduce execution time by ~15% (python#101251)pythongh-104223: Fix issues with inheriting from buffer classes (python#104227)pythongh-99108: fix typo in Modules/Setup (python#104293)pythonGH-104145: Use fully-qualified cross reference types for the bisect module (python#104172)pythongh-103193: Improve `getattr_static` test coverage (python#104286)  Trim trailing whitespace and test on CI (python#104275)pythongh-102500: Remove mention of bytes shorthand (python#104281)pythongh-97696: Improve and fix documentation for asyncio eager tasks (python#104256)pythongh-99108: Replace SHA3 implementation HACL* version (python#103597)  ...

carljm added a commit to carljm/cpython that referenced this pull request

Merge branch 'main' into supermon

c48eee3

* main: (156 commits)pythongh-97696 Add documentation for get_coro() behavior with eager tasks (python#104304)pythongh-97933: (PEP 709) inline list/dict/set comprehensions (python#101441)pythongh-99889: Fix directory traversal security flaw in uu.decode() (python#104096)pythongh-104184: fix building --with-pydebug --enable-pystats (python#104217)pythongh-104139: Add itms-services to uses_netloc urllib.parse. (python#104312)pythongh-104240: return code unit metadata from codegen (python#104300)pythongh-104276: Make `_struct.unpack_iterator` type use type flag instead of custom constructor (python#104277)pythongh-97696: Move around and update the whatsnew entry for asyncio eager task factory (python#104298)pythongh-103193: Fix refleaks in `test_inspect` and `test_typing` (python#104320)  require-pr-label.yml: Add missing "permissions:" (python#104309)pythongh-90656: Add platform triplets for 64-bit LoongArch (LA64) (python#30939)pythongh-104180: Read SOCKS proxies from macOS System Configuration (python#104181)pythongh-97696 Remove unnecessary check for eager_start kwarg (python#104188)pythonGH-104308: socket.getnameinfo should release the GIL (python#104307)pythongh-104310: Add importlib.util.allowing_all_extensions() (pythongh-104311)pythongh-99113: A Per-Interpreter GIL! (pythongh-104210)pythonGH-104284: Fix documentation gettext build (python#104296)pythongh-89550: Buffer GzipFile.write to reduce execution time by ~15% (python#101251)pythongh-104223: Fix issues with inheriting from buffer classes (python#104227)pythongh-99108: fix typo in Modules/Setup (python#104293)  ...

carljm added a commit to carljm/cpython that referenced this pull request

GzipFile.flush doesn't flush compressor in 3.12 beta#105808

Merge branch 'main' into fixloadsuperspec

cf11908

* main: (35 commits)pythongh-97696 Add documentation for get_coro() behavior with eager tasks (python#104304)pythongh-97933: (PEP 709) inline list/dict/set comprehensions (python#101441)pythongh-99889: Fix directory traversal security flaw in uu.decode() (python#104096)pythongh-104184: fix building --with-pydebug --enable-pystats (python#104217)pythongh-104139: Add itms-services to uses_netloc urllib.parse. (python#104312)pythongh-104240: return code unit metadata from codegen (python#104300)pythongh-104276: Make `_struct.unpack_iterator` type use type flag instead of custom constructor (python#104277)pythongh-97696: Move around and update the whatsnew entry for asyncio eager task factory (python#104298)pythongh-103193: Fix refleaks in `test_inspect` and `test_typing` (python#104320)  require-pr-label.yml: Add missing "permissions:" (python#104309)pythongh-90656: Add platform triplets for 64-bit LoongArch (LA64) (python#30939)pythongh-104180: Read SOCKS proxies from macOS System Configuration (python#104181)pythongh-97696 Remove unnecessary check for eager_start kwarg (python#104188)pythonGH-104308: socket.getnameinfo should release the GIL (python#104307)pythongh-104310: Add importlib.util.allowing_all_extensions() (pythongh-104311)pythongh-99113: A Per-Interpreter GIL! (pythongh-104210)pythonGH-104284: Fix documentation gettext build (python#104296)pythongh-89550: Buffer GzipFile.write to reduce execution time by ~15% (python#101251)pythongh-104223: Fix issues with inheriting from buffer classes (python#104227)pythongh-99108: fix typo in Modules/Setup (python#104293)  ...

bdarnell mentioned this pull request

Jun 15, 2023

Closed

Yhg1s added a commit to Yhg1s/cpython that referenced this pull request

pythonGH-105808: Fix a regression introduced inpythonGH-101251, causing

b7ecf06

GzipFile.flush() to not flush the compressor (nor pass along the zip_modeargument).

Yhg1s added a commit to Yhg1s/cpython that referenced this pull request

Fix a regression introduced inpythonGH-101251, causing GzipFile.flus…

5b3a658

…h() tonot flush the compressor (nor pass along the zip_mode argument).

Yhg1s mentioned this pull request

GH-105808: Fix a regression introduced in GH-101251#105910

Merged

gpshead pushed a commit that referenced this pull request

GH-105808: Fix a regression introduced inGH-101251(#105910)

1858db7

Fix a regression introduced inGH-101251, causing GzipFile.flush() tonot flush the compressor (nor pass along the zip_mode argument).

miss-islington mentioned this pull request

[3.12] GH-105808: Fix a regression introduced in GH-101251 (GH-105910)#105920

Merged

miss-islington pushed a commit to miss-islington/cpython that referenced this pull request

pythonGH-105808: Fix a regression introduced inpythonGH-101251(pyth…

15b4976

…onGH-105910)Fix a regression introduced inpythonGH-101251, causing GzipFile.flush() tonot flush the compressor (nor pass along the zip_mode argument).(cherry picked from commit1858db7)Co-authored-by: T. Wouters <thomas@python.org>

Yhg1s added a commit that referenced this pull request