Uh oh!
There was an error while loading.Please reload this page.
- Notifications
You must be signed in to change notification settings - Fork33.7k
bpo-26175: Fix SpooledTemporaryFile IOBase abstract#3249
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to ourterms of service andprivacy statement. We’ll occasionally send you account related emails.
Already on GitHub?Sign in to your account
bpo-26175: Fix SpooledTemporaryFile IOBase abstract#3249
Uh oh!
There was an error while loading.Please reload this page.
Conversation
One would assume that this class implements the IOBase abstract. As theunderlying file-like object is either io.BytesIO, io.StringIO, or a truefile object, this is a reasonable abstract expect and to implement.Regardless, the behaviour of this class does not change much in the caseof the attribute being missing from the underlying file-like; anAttributeError is still raised, albeit from one additional frame on thestack trace.
the-knights-who-say-ni commentedAug 30, 2017
Hello, and thanks for your contribution! I'm a bot set up to make sure that the project can legally accept your contribution by verifying you have signed thePSF contributor agreement (CLA). Unfortunately we couldn't find an account corresponding to your GitHub username onbugs.python.org (b.p.o) to verify you have signed the CLA (this might be simply due to a missing "GitHub Name" entry in your b.p.o account settings). This is necessary for legal reasons before we can look at your contribution. Please followthe steps outlined in the CPython devguide to rectify this issue. Thanks again to your contribution and we look forward to looking at it! |
merwok left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
Patch looks good! Note that the bug contains a stronger motivation than the first mssage here: «This was discovered when seeking a SpooledTemporaryFile-backed lzma file.»
Uh oh!
There was an error while loading.Please reload this page.
bedevere-bot commentedSep 20, 2017
A Python core developer has requested some changes be made to your pull request before we can consider merging it. If you could please address their requests along with any other requests in other reviews from core developers that would be appreciated. Once you have made the requested changes, please leave a comment on this pull request containing the phrase |
GFernie commentedSep 30, 2017
Thanks for the feedback@merwok I've made some changes to more accurately test howSpooledTemporaryFile implementsIOBase. So, there are now two separate tests:
From theIOBase docs:
I didn't expect the Spanish Inquisition! |
bedevere-bot commentedSep 30, 2017
Nobody expects the Spanish Inquisition! @merwok: please review the changes made to this pull request. |
embe commentedFeb 19, 2018
There is at least one more incompatibility with |
| def__iter__(self): | ||
| returnself._file.__iter__() | ||
| def__del__(self): |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
This shouldn't be added: deleting the SpooledTemporaryFile will null out the reference to the underline file, and therefor call its __del__
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
I think if you inherit the defaultIOBase.__del__ implementation, it will callclose and defeat any ResourceWarning that might otherwise be emitted. Perhaps it is better to make__del__ do nothing,or set it to. [Seems that last option doesn’t exist.]object.__del__
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
I agree: the underlying file should not be explicitly deleted as this is not expected behaviour. I can imagine a situation where someone is deliberately holding a reference to the underlying file and they would not expect/want it to be closed until their own reference has fallen out of scope. I've changed the method to do nothing and added a docstring to reflect this.
Thanks for your feedback
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
I agree: the underlying file should not be explicitly deleted as this is not expected behaviour
The doc says: """This function operates exactly as TemporaryFile() does, except [irrelevant differences]."""
And then, about TemporaryFile: """On completion of the context or destruction of the file object the temporary file will be removed from the filesystem."""
So it seems the underlying fileshould be deleted when the file object disappears.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
The point is thatIOBase implements a__del__ which has some side-effects. Those side effects are not desirable here. Any underlying buffers being wrapped, e.g. theTemporaryFile or theBytesIO, will be gc'd and handled as they should.
I think this implementation is correct.
We don't want to delete the underlying file explicitly as the expectedbehaviour is for the file to be deleted *after* it falls out of scope.
GFernie commentedSep 9, 2018 • edited
Loading Uh oh!
There was an error while loading.Please reload this page.
edited
Uh oh!
There was an error while loading.Please reload this page.
You are right,@embe; it's part of the interface: The method now returns the value returned from Thanks for pointing this out |
GFernie commentedSep 9, 2018
Re seek: I've also done the same with This is also part of the file interface: |
GFernie commentedNov 23, 2019
@pitrou I've considered your comments on what we should expect when calling
I've picked the third option, as you suggested. Thanks for your review. I have made the requested changes; please review again. |
bedevere-bot commentedNov 23, 2019
pitrou left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
Thanks for posting this. Generally speaking, calling__del__ directly is incorrect. Only the interpreter should do that. From a user's point of view, what's important is whether an object is still reachable or not.
| ..versionchanged::3.8 | ||
| Added *errors* parameter. | ||
| Implemented:class:`io.IOBase` inteface. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
Since 3.8.0 has been released, this should be moved to a newversionchanged:: 3.9 block.
| returnself._file.__iter__() | ||
| def__del__(self): | ||
| returnself._file.__del__() |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
Hmm, calling the__del__ method directly like that is not correct.__del__ is only to be called when all external references are gone. Instead I would suggest something like the following:
def__del__(self):ifnotself.closed:importwarningswarnings.warn('unclosed file %r'% (self,),ResourceWarning,stacklevel=2,source=self)self.close()
(this is adapted fromFileIO.__del__ inLib/pyio.py)
| # ResourceWarning since the file was not explicitly closed. | ||
| f=self.do_create(max_size=2) | ||
| f.write(b'foo') | ||
| filename=f.name |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
Does this actually work? Here I get something like:
>>>f=tempfile.SpooledTemporaryFile()>>>f.name>>>f.rollover()>>>f.name3
| f.write(b'foo') | ||
| filename=f.name | ||
| self.assertTrue(os.path.exists(filename)) | ||
| withself.assertWarns(ResourceWarning): |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
Do we also expect aResourceWarning to be raised when the file is not rolled over? If yes, there should be a test for it. If not, then the__del__ implementation can be removed.
| filename=f.name | ||
| self.assertTrue(os.path.exists(filename)) | ||
| withself.assertWarns(ResourceWarning): | ||
| f.__del__() |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
It is not correct to call__del__ directly. Instead, usedel f orf = None.
bedevere-bot commentedNov 23, 2019
A Python core developer has requested some changes be made to your pull request before we can consider merging it. If you could please address their requests along with any other requests in other reviews from core developers that would be appreciated. Once you have made the requested changes, please leave a comment on this pull request containing the phrase |
csabella commentedJan 12, 2020
| ifsizeisNone: | ||
| self._file.truncate() | ||
| returnself._file.truncate() | ||
| else: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
this else is not needed, PEP warning, might as well fix
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
It's not necessary to remove the else. It does no harm.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
I never said it was necessary, just bad style as it unnecessarily increased the code depth, take my comment as an FYI
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
Just wanted to confirm@merwok - nothing needs to be changed here for approval?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
Right, I advise to not change theelse line.
danieldjewell commentedMay 13, 2020
merwok commentedMay 13, 2020
This PR needs to merge master and address Antoine’s latest review. |
csabella commentedMay 23, 2020
@danieldjewell, you're right. Would you like to create a new PR based on this one? If so, please follow this section of thedevguide to give GFernie the credit as the original author. Thanks! |
carlos-jenkins commentedOct 31, 2020
Hello, For anyone that got here looking for the error In my case it was trying to pass an In one of your modules: fromtempfileimportSpooledTemporaryFiledef_seekable(self):""" Monkey patched seekable() method for the SpooledTemporaryFile class. Sadly, we cannot send an instance of the SpooledTemporaryFile to the ZipFile object. This is because the SpooledTemporaryFile doesn't inherit / implement the IOBase class. This bug is reported in CPython: https://bugs.python.org/issue26175 And an unmerged PR is located here: https://github.com/python/cpython/pull/3249/files In Python 3.8, attempting to pass the tmpfd fails with: AttributeError: 'SpooledTemporaryFile' object has no attribute 'seekable' So, the workaround to avoid copying the file is to monkey patch the class and add the missing method. Sorry. """returnself._file.seekable()SpooledTemporaryFile.seekable=_seekable |
Molaire commentedJan 18, 2021
To Google people: this also breaks pandas.read_csv() and this monkey patch can be used for property readable writable and seekable to fix it.
|
MikeEcho commentedDec 26, 2021
Python dev here from the future (2021 - save yourselves while you can). Still getting |
pR0Ps commentedDec 31, 2021
This PR is stale because it has been open for 30 days with no activity. |
methane commentedMay 3, 2022
Close this because I merged#29560. |
Uh oh!
There was an error while loading.Please reload this page.
One would assume that this class implements the IOBase abstract. As the
underlying file-like object is either io.BytesIO, io.StringIO, or a true
file object, this is a reasonable abstract expect and to implement.
Regardless, the behaviour of this class does not change much in the case
of the attribute being missing from the underlying file-like; an
AttributeError is still raised, albeit from one additional frame on the
stack trace.
https://bugs.python.org/issue26175