Uh oh!
There was an error while loading.Please reload this page.
- Notifications
You must be signed in to change notification settings - Fork34.1k
Comments
gh-131492: gh-131461: handle exceptions in GzipFile constructor while owning resources#131462
Conversation
Uh oh!
There was an error while loading.Please reload this page.
Uh oh!
There was an error while loading.Please reload this page.
Misc/NEWS.d/next/Library/2025-03-19-14-36-54.gh-issue-131461.uDUmdY.rst OutdatedShow resolvedHide resolved
Uh oh!
There was an error while loading.Please reload this page.
Uh oh!
There was an error while loading.Please reload this page.
…DUmdY.rstCo-authored-by: Victor Stinner <vstinner@python.org>
vstinner commentedMar 19, 2025
@cmaloney: Would you mind to review this fix? |
vstinner left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
Would it be possible to write a test with such broken file object?
graingert commentedMar 19, 2025
There's already a test in test_tarfile: test_open_nonwritable_fileobj the failure will be enforced by#128973 I added check_no_resource_warning to make the assert hold until then |
Uh oh!
There was an error while loading.Please reload this page.
Uh oh!
There was an error while loading.Please reload this page.
Co-authored-by: Victor Stinner <vstinner@python.org>
vstinner left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
LGTM
Uh oh!
There was an error while loading.Please reload this page.
vstinner commentedMar 19, 2025
The CI fails: |
Co-authored-by: Victor Stinner <vstinner@python.org>
graingert commentedMar 19, 2025
🤦 |
| self.fileobj = fileobj | ||
| try: | ||
| if fileobj is None: | ||
| fileobj = self.myfileobj = builtins.open(filename, mode or 'rb') |
graingertMar 19, 2025 • edited
Loading Uh oh!
There was an error while loading.Please reload this page.
edited
Uh oh!
There was an error while loading.Please reload this page.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
I pushed a more thorough change inspired by
Lines 1014 to 1015 ina4832f6
| # Now SSLSocket is responsible for closing the file descriptor. | |
| try: |
previously it was possible for this self.myfileobj to be left open if any of the constructor failed
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
Rather than the try/except wrapping, why not add a flag which is set toFalse at the beginning of the constructor andTrue at the end, in__del__ (which is what emits the resource warning), check that flag. If it's not fully constructed then don't emit the resource warning?
The I/O stack (GzipFile inherits from IOBase) inIOBase.__del__ always does the close call ifself.closed is False, so shouldn't need to manually add a special case here. GzipFile.closed is:
@property def closed(self): return self.fileobj is None(so if there is a fileobj set, close should always be called).
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
This wouldn't help with myfileobj closing
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
How about usingself.fileobj is not None as the "fully initialized" flag (already set to None always, only set to non-None at end of__init__), and checking aroundif self.filobj is None and self.myfileobj is not None in__del__?
It feels a lot like this is adding more cases / code around closing. Trying to find a way to get this change to be smaller / more minimal to reduce risks. IO stack construction, dealloc warnings, and close + destruction order is already really intricate and hard to understand in full.
graingertMar 19, 2025 • edited
Loading Uh oh!
There was an error while loading.Please reload this page.
edited
Uh oh!
There was an error while loading.Please reload this page.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
We can't rely onself.myfileobj being closed in__del__ and we don't return an instance ofGzipFile for the caller to close soself.myfileobj needs to be closed at the end of__init__ if there's an exception: so we need the try/catch
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
Right but this is closing objects that are inaccessible to the caller
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
TheGzipFile /self is equally inaccessible, and cleaned up by its__del__ / finalizertp_finalize (ortp_dealloc). It's guaranteed__del__ is called and able to release any held resources (such asself.myfileobj) seePEP-442.
I'm okay adding a cleanup for the case whereself.fileobj is None and self.myfileobj is not None inside__del__. Matches the behavior of otherio objects and is a lot smaller diff.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
This is not correct, you cannot rely on__del__ to be called when an exception is raised because the exception traceback holds a reference to self
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
That statement seems to contradictPEP 442 Predictability section to me...
Following this scheme, an object’s finalizer is always called exactly once, even if it was resurrected afterwards.
For CI objects, the order in which finalizers are called (step 2 above) is undefined.
Even if there is an exception traceback that keeps the reference to self, eventually that traceback will go out of scope and file will be closed / not leaked...
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
I'll defer to@vstinner and@serhiy-storchaka for approval. From my perspective, this is a lot more complex of a bugfix than required, didn't match stated description and arguments aroundself.myfileobj. Happy to do quick fixes, can produce a much smaller patch here if desired with less behavior change.
serhiy-storchaka left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
LGTM.
I would addsupport.gc_collect() in tests.
Uh oh!
There was an error while loading.Please reload this page.
Misc/NEWS.d/next/Library/2025-03-20-08-32-49.gh-issue-131492.saC2cA.rst OutdatedShow resolvedHide resolved
Uh oh!
There was an error while loading.Please reload this page.
…aC2cA.rstCo-authored-by: Victor Stinner <vstinner@python.org>
…aC2cA.rstCo-authored-by: Victor Stinner <vstinner@python.org>
vstinner left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
LGTM
ce79274 intopython:mainUh oh!
There was an error while loading.Please reload this page.
Thanks@graingert for the PR, and@vstinner for merging it 🌮🎉.. I'm working now to backport this PR to: 3.12, 3.13. |
…ructor while owning resources (pythonGH-131462)(cherry picked from commitce79274)Co-authored-by: Thomas Grainger <tagrain@gmail.com>Co-authored-by: Victor Stinner <vstinner@python.org>
GH-131518 is a backport of this pull request to the3.13 branch. |
…ructor while owning resources (pythonGH-131462)(cherry picked from commitce79274)Co-authored-by: Thomas Grainger <tagrain@gmail.com>Co-authored-by: Victor Stinner <vstinner@python.org>
GH-131519 is a backport of this pull request to the3.12 branch. |
vstinner commentedMar 20, 2025
Hum, the backports mention ResourceWarning, whereas ResourceWarning only exists in Python 3.14. If we want to backport this change to 3.13 and 3.12, maybe references to ResourceWarning should be removed? What do you think@graingert? |
graingert commentedMar 20, 2025 • edited
Loading Uh oh!
There was an error while loading.Please reload this page.
edited
Uh oh!
There was an error while loading.Please reload this page.
Oh yeah nix the 3.14 specific bits of the changelog from the backports |
graingert commentedMar 20, 2025
I've removed the references to a ResourceWarning in the backports |
Uh oh!
There was an error while loading.Please reload this page.