Movatterモバイル変換


[0]ホーム

URL:


Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

Sign up
Appearance settings

Comments

gh-131492: gh-131461: handle exceptions in GzipFile constructor while owning resources#131462

Merged
vstinner merged 20 commits intopython:mainfrom
graingert:fix-resource-warning-in-gzipfile-ctor
Mar 20, 2025
Merged

gh-131492: gh-131461: handle exceptions in GzipFile constructor while owning resources#131462
vstinner merged 20 commits intopython:mainfrom
graingert:fix-resource-warning-in-gzipfile-ctor

Conversation

@graingert
Copy link
Contributor

@graingertgraingert commentedMar 19, 2025
edited by bedevere-appbot
Loading

@graingertgraingert marked this pull request as ready for reviewMarch 19, 2025 14:28
@vstinner
Copy link
Member

@cmaloney: Would you mind to review this fix?

cmaloney reacted with eyes emoji

Copy link
Member

@vstinnervstinner left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

Would it be possible to write a test with such broken file object?

@graingert
Copy link
ContributorAuthor

Would it be possible to write a test with such broken file object?

There's already a test in test_tarfile: test_open_nonwritable_fileobj the failure will be enforced by#128973

I added check_no_resource_warning to make the assert hold until then

vstinner reacted with thumbs up emoji

graingertand others added2 commitsMarch 19, 2025 17:10
Co-authored-by: Victor Stinner <vstinner@python.org>
Copy link
Member

@vstinnervstinner left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

LGTM

@vstinner
Copy link
Member

The CI fails:

TypeError: check_no_resource_warning() missing 1 required positional argument: 'testcase'

Co-authored-by: Victor Stinner <vstinner@python.org>
@graingert
Copy link
ContributorAuthor

The CI fails:

TypeError: check_no_resource_warning() missing 1 required positional argument: 'testcase'

🤦

self.fileobj = fileobj
try:
if fileobj is None:
fileobj = self.myfileobj = builtins.open(filename, mode or 'rb')
Copy link
ContributorAuthor

@graingertgraingertMar 19, 2025
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

I pushed a more thorough change inspired by

cpython/Lib/ssl.py

Lines 1014 to 1015 ina4832f6

# Now SSLSocket is responsible for closing the file descriptor.
try:

previously it was possible for this self.myfileobj to be left open if any of the constructor failed

cmaloney reacted with thumbs up emoji
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

Rather than the try/except wrapping, why not add a flag which is set toFalse at the beginning of the constructor andTrue at the end, in__del__ (which is what emits the resource warning), check that flag. If it's not fully constructed then don't emit the resource warning?

The I/O stack (GzipFile inherits from IOBase) inIOBase.__del__ always does the close call ifself.closed is False, so shouldn't need to manually add a special case here. GzipFile.closed is:

    @property    def closed(self):        return self.fileobj is None

(so if there is a fileobj set, close should always be called).

Copy link
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

This wouldn't help with myfileobj closing

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

How about usingself.fileobj is not None as the "fully initialized" flag (already set to None always, only set to non-None at end of__init__), and checking aroundif self.filobj is None and self.myfileobj is not None in__del__?

It feels a lot like this is adding more cases / code around closing. Trying to find a way to get this change to be smaller / more minimal to reduce risks. IO stack construction, dealloc warnings, and close + destruction order is already really intricate and hard to understand in full.

Copy link
ContributorAuthor

@graingertgraingertMar 19, 2025
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

We can't rely onself.myfileobj being closed in__del__ and we don't return an instance ofGzipFile for the caller to close soself.myfileobj needs to be closed at the end of__init__ if there's an exception: so we need the try/catch

Copy link
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

Right but this is closing objects that are inaccessible to the caller

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

TheGzipFile /self is equally inaccessible, and cleaned up by its__del__ / finalizertp_finalize (ortp_dealloc). It's guaranteed__del__ is called and able to release any held resources (such asself.myfileobj) seePEP-442.

I'm okay adding a cleanup for the case whereself.fileobj is None and self.myfileobj is not None inside__del__. Matches the behavior of otherio objects and is a lot smaller diff.

Copy link
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

This is not correct, you cannot rely on__del__ to be called when an exception is raised because the exception traceback holds a reference to self

Copy link
Contributor

@cmaloneycmaloneyMar 20, 2025
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

That statement seems to contradictPEP 442 Predictability section to me...

Following this scheme, an object’s finalizer is always called exactly once, even if it was resurrected afterwards.
For CI objects, the order in which finalizers are called (step 2 above) is undefined.

Even if there is an exception traceback that keeps the reference to self, eventually that traceback will go out of scope and file will be closed / not leaked...

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

I'll defer to@vstinner and@serhiy-storchaka for approval. From my perspective, this is a lot more complex of a bugfix than required, didn't match stated description and arguments aroundself.myfileobj. Happy to do quick fixes, can produce a much smaller patch here if desired with less behavior change.

@graingertgraingert changed the titlegh-131461: fix ResourceWarning when writing a unwritable gzipfilegh-131492: gh-131461: fix ResourceWarning when writing a unwritable gzipfileMar 20, 2025
@graingertgraingert added needs backport to 3.12only security fixes needs backport to 3.13bugs and security fixes labelsMar 20, 2025
@graingertgraingert changed the titlegh-131492: gh-131461: fix ResourceWarning when writing a unwritable gzipfilegh-131492: gh-131461: handle exceptions in GzipFile constructor while owning resourcesMar 20, 2025
Copy link
Member

@serhiy-storchakaserhiy-storchaka left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

LGTM.

I would addsupport.gc_collect() in tests.

graingertand others added2 commitsMarch 20, 2025 16:40
…aC2cA.rstCo-authored-by: Victor Stinner <vstinner@python.org>
…aC2cA.rstCo-authored-by: Victor Stinner <vstinner@python.org>
@vstinnervstinnerenabled auto-merge (squash)March 20, 2025 16:49
Copy link
Member

@vstinnervstinner left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

LGTM

@vstinnervstinner merged commitce79274 intopython:mainMar 20, 2025
39 checks passed
@miss-islington-app
Copy link

Thanks@graingert for the PR, and@vstinner for merging it 🌮🎉.. I'm working now to backport this PR to: 3.12, 3.13.
🐍🍒⛏🤖

miss-islington pushed a commit to miss-islington/cpython that referenced this pull requestMar 20, 2025
…ructor while owning resources (pythonGH-131462)(cherry picked from commitce79274)Co-authored-by: Thomas Grainger <tagrain@gmail.com>Co-authored-by: Victor Stinner <vstinner@python.org>
@bedevere-app
Copy link

GH-131518 is a backport of this pull request to the3.13 branch.

miss-islington pushed a commit to miss-islington/cpython that referenced this pull requestMar 20, 2025
…ructor while owning resources (pythonGH-131462)(cherry picked from commitce79274)Co-authored-by: Thomas Grainger <tagrain@gmail.com>Co-authored-by: Victor Stinner <vstinner@python.org>
@bedevere-appbedevere-appbot removed the needs backport to 3.13bugs and security fixes labelMar 20, 2025
@bedevere-app
Copy link

GH-131519 is a backport of this pull request to the3.12 branch.

@bedevere-appbedevere-appbot removed the needs backport to 3.12only security fixes labelMar 20, 2025
@vstinner
Copy link
Member

Hum, the backports mention ResourceWarning, whereas ResourceWarning only exists in Python 3.14. If we want to backport this change to 3.13 and 3.12, maybe references to ResourceWarning should be removed?

What do you think@graingert?

@graingert
Copy link
ContributorAuthor

graingert commentedMar 20, 2025
edited
Loading

Oh yeah nix the 3.14 specific bits of the changelog from the backports

@graingertgraingert deleted the fix-resource-warning-in-gzipfile-ctor branchMarch 20, 2025 17:35
@graingert
Copy link
ContributorAuthor

I've removed the references to a ResourceWarning in the backports

vstinner reacted with thumbs up emoji

vstinner added a commit that referenced this pull requestMar 21, 2025
…r while owning resources (GH-131462) (#131518)(cherry picked from commitce79274)Co-authored-by: Thomas Grainger <tagrain@gmail.com>Co-authored-by: Victor Stinner <vstinner@python.org>
vstinner added a commit that referenced this pull requestMar 21, 2025
…r while owning resources (GH-131462) (#131519)(cherry picked from commitce79274)Co-authored-by: Thomas Grainger <tagrain@gmail.com>Co-authored-by: Victor Stinner <vstinner@python.org>
Sign up for freeto join this conversation on GitHub. Already have an account?Sign in to comment

Reviewers

@cmaloneycmaloneycmaloney left review comments

@vstinnervstinnervstinner approved these changes

@serhiy-storchakaserhiy-storchakaserhiy-storchaka approved these changes

@ethanfurmanethanfurmanAwaiting requested review from ethanfurmanethanfurman is a code owner

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@graingert@vstinner@cmaloney@serhiy-storchaka

[8]ページ先頭

©2009-2026 Movatter.jp