Movatterモバイル変換


[0]ホーム

URL:


Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

Sign up
Appearance settings

bpo-26175: Fix SpooledTemporaryFile IOBase abstract#3249

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to ourterms of service andprivacy statement. We’ll occasionally send you account related emails.

Already on GitHub?Sign in to your account

Closed
GFernie wants to merge13 commits intopython:mainfromGFernie:bpo-26175_fix-SpooledTemporaryFile-IOBase
Closed

bpo-26175: Fix SpooledTemporaryFile IOBase abstract#3249

GFernie wants to merge13 commits intopython:mainfromGFernie:bpo-26175_fix-SpooledTemporaryFile-IOBase

Conversation

@GFernie
Copy link
Contributor

@GFernieGFernie commentedAug 30, 2017
edited by bedevere-bot
Loading

One would assume that this class implements the IOBase abstract. As the
underlying file-like object is either io.BytesIO, io.StringIO, or a true
file object, this is a reasonable abstract expect and to implement.

Regardless, the behaviour of this class does not change much in the case
of the attribute being missing from the underlying file-like; an
AttributeError is still raised, albeit from one additional frame on the
stack trace.

https://bugs.python.org/issue26175

thehesiod, pypae, spengjie, nwh, jonathan-s, NicolaiSoeborg, tc-imba, and farhaanshaikh19 reacted with thumbs up emoji
One would assume that this class implements the IOBase abstract. As theunderlying file-like object is either io.BytesIO, io.StringIO, or a truefile object, this is a reasonable abstract expect and to implement.Regardless, the behaviour of this class does not change much in the caseof the attribute being missing from the underlying file-like; anAttributeError is still raised, albeit from one additional frame on thestack trace.
@the-knights-who-say-ni

Hello, and thanks for your contribution!

I'm a bot set up to make sure that the project can legally accept your contribution by verifying you have signed thePSF contributor agreement (CLA).

Unfortunately we couldn't find an account corresponding to your GitHub username onbugs.python.org (b.p.o) to verify you have signed the CLA (this might be simply due to a missing "GitHub Name" entry in your b.p.o account settings). This is necessary for legal reasons before we can look at your contribution. Please followthe steps outlined in the CPython devguide to rectify this issue.

Thanks again to your contribution and we look forward to looking at it!

Copy link
Member

@merwokmerwok left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

Patch looks good! Note that the bug contains a stronger motivation than the first mssage here: «This was discovered when seeking a SpooledTemporaryFile-backed lzma file.»

@bedevere-bot
Copy link

A Python core developer has requested some changes be made to your pull request before we can consider merging it. If you could please address their requests along with any other requests in other reviews from core developers that would be appreciated.

Once you have made the requested changes, please leave a comment on this pull request containing the phraseI didn't expect the Spanish Inquisition!. I will then notify any core developers who have left a review that you're ready for them to take another look at this pull request.

@GFernie
Copy link
ContributorAuthor

Thanks for the feedback@merwok

I've made some changes to more accurately test howSpooledTemporaryFile implementsIOBase. So, there are now two separate tests:

  • Assert that an instance ofSpooledTemporaryFile is an instance ofIOBase. BecauseIOBase already implements the interface (nearly) this makes it redundant to check for the individual attributes thatSpooledTemporaryFile should implement.
  • Assert that the attributes which not declared inIOBase but should be implemented by subclasses are implemented bySpooledTemporaryFile.

From theIOBase docs:

Even though IOBase does not declare read(), readinto(), or write() because their signatures will vary, implementations and clients should consider those methods part of the interface.

I didn't expect the Spanish Inquisition!

@bedevere-bot
Copy link

Nobody expects the Spanish Inquisition!

@merwok: please review the changes made to this pull request.

@embe
Copy link

There is at least one more incompatibility withIOBase:seek should return the current position instead ofNone.

GFernie reacted with thumbs up emoji

def__iter__(self):
returnself._file.__iter__()

def__del__(self):
Copy link

@ppperyppperyJun 6, 2018
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

This shouldn't be added: deleting the SpooledTemporaryFile will null out the reference to the underline file, and therefor call its __del__

Copy link
Member

@vadmiumvadmiumJun 9, 2018
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

I think if you inherit the defaultIOBase.__del__ implementation, it will callclose and defeat any ResourceWarning that might otherwise be emitted. Perhaps it is better to make__del__ do nothing,or set it toobject.__del__. [Seems that last option doesn’t exist.]

Copy link
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

I agree: the underlying file should not be explicitly deleted as this is not expected behaviour. I can imagine a situation where someone is deliberately holding a reference to the underlying file and they would not expect/want it to be closed until their own reference has fallen out of scope. I've changed the method to do nothing and added a docstring to reflect this.

Thanks for your feedback

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

I agree: the underlying file should not be explicitly deleted as this is not expected behaviour

The doc says: """This function operates exactly as TemporaryFile() does, except [irrelevant differences]."""

And then, about TemporaryFile: """On completion of the context or destruction of the file object the temporary file will be removed from the filesystem."""

So it seems the underlying fileshould be deleted when the file object disappears.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

The point is thatIOBase implements a__del__ which has some side-effects. Those side effects are not desirable here. Any underlying buffers being wrapped, e.g. theTemporaryFile or theBytesIO, will be gc'd and handled as they should.

I think this implementation is correct.

@terryjreedyterryjreedy requested a review fromvadmiumJune 8, 2018 17:51
We don't want to delete the underlying file explicitly as the expectedbehaviour is for the file to be deleted *after* it falls out of scope.
@GFernie
Copy link
ContributorAuthor

GFernie commentedSep 9, 2018
edited
Loading

You are right,@embe; it's part of the interface:
https://docs.python.org/library/io.html#io.IOBase.seek

The method now returns the value returned fromself._file.seek(), delegating the responsibility of returning the current new absolute file position to the underlying file.

Thanks for pointing this out

@GFernie
Copy link
ContributorAuthor

Re seek: I've also done the same withtruncate, as per Martin's recommendations on the Python bug tracker:https://bugs.python.org/issue26175#msg319145

This is also part of the file interface:
https://docs.python.org/library/io.html#io.IOBase.truncate

@GFernie
Copy link
ContributorAuthor

@pitrou I've considered your comments on what we should expect when calling__del__. There's three options which have been discussed:

  • Do nothing. The underlying buffer or temp file will be deleted when the object is finalised anyway. If the object is deleted withdel, this would happen immediately. However, if__del__ is called then this would delay the deletion of the underlying buffer or temp file until the object is deleted withdel or is garbage collected, so possibly never. Here, we are in a situation where we called__del__, expected the resources to be freed, but not even a warning was raised.
  • Inherit theio.IOBase.__del__ implementation. The underlying buffer or temp file will be deleted when__del__ is called, the object is deleted withdel, or the object is garbage collected. This is the expected behaviour, but base implementation will explicitly call close, suppressing theResourceWarning which should be raised when the object is deleted before closing the file.
  • Delegate the__del__ call to the underlying file. If it is a buffer then then the buffer will be freed and if it's rolled over to disk then the temp file will be deleted. If the temp file was not closed before the object was deleted then aResourceWarning will be raised, just likeTemporaryFile. This ensures that resources are freed exactly when we expect them to be and also that we are raising a warning when resources are not being managed properly by the programmer.

I've picked the third option, as you suggested. Thanks for your review.

I have made the requested changes; please review again.

@bedevere-bot
Copy link

Thanks for making the requested changes!

@merwok,@pitrou: please review the changes made to this pull request.

Copy link
Member

@pitroupitrou left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

Thanks for posting this. Generally speaking, calling__del__ directly is incorrect. Only the interpreter should do that. From a user's point of view, what's important is whether an object is still reachable or not.


..versionchanged::3.8
Added *errors* parameter.
Implemented:class:`io.IOBase` inteface.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

Since 3.8.0 has been released, this should be moved to a newversionchanged:: 3.9 block.

returnself._file.__iter__()

def__del__(self):
returnself._file.__del__()
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

Hmm, calling the__del__ method directly like that is not correct.__del__ is only to be called when all external references are gone. Instead I would suggest something like the following:

def__del__(self):ifnotself.closed:importwarningswarnings.warn('unclosed file %r'% (self,),ResourceWarning,stacklevel=2,source=self)self.close()

(this is adapted fromFileIO.__del__ inLib/pyio.py)

# ResourceWarning since the file was not explicitly closed.
f=self.do_create(max_size=2)
f.write(b'foo')
filename=f.name
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

Does this actually work? Here I get something like:

>>>f=tempfile.SpooledTemporaryFile()>>>f.name>>>f.rollover()>>>f.name3

f.write(b'foo')
filename=f.name
self.assertTrue(os.path.exists(filename))
withself.assertWarns(ResourceWarning):
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

Do we also expect aResourceWarning to be raised when the file is not rolled over? If yes, there should be a test for it. If not, then the__del__ implementation can be removed.

filename=f.name
self.assertTrue(os.path.exists(filename))
withself.assertWarns(ResourceWarning):
f.__del__()
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

It is not correct to call__del__ directly. Instead, usedel f orf = None.

@bedevere-bot
Copy link

A Python core developer has requested some changes be made to your pull request before we can consider merging it. If you could please address their requests along with any other requests in other reviews from core developers that would be appreciated.

Once you have made the requested changes, please leave a comment on this pull request containing the phraseI have made the requested changes; please review again. I will then notify any core developers who have left a review that you're ready for them to take another look at this pull request.

@csabella
Copy link
Contributor

@GFernie, thank you for your last comments and for addressing the code review. It looks like@pitrou had some additional notes after that, so it would be great if you could look at this again. Thank you!

ifsizeisNone:
self._file.truncate()
returnself._file.truncate()
else:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

this else is not needed, PEP warning, might as well fix

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

It's not necessary to remove the else. It does no harm.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

I never said it was necessary, just bad style as it unnecessarily increased the code depth, take my comment as an FYI

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

Just wanted to confirm@merwok - nothing needs to be changed here for approval?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

Right, I advise to not change theelse line.

@danieldjewell
Copy link

Not sure if@GFernie is still around -- are the outstanding changes/questions still those raised by@pitrou?@merwok?

@merwok
Copy link
Member

This PR needs to merge master and address Antoine’s latest review.

@csabella
Copy link
Contributor

@danieldjewell, you're right. Would you like to create a new PR based on this one? If so, please follow this section of thedevguide to give GFernie the credit as the original author. Thanks!

@carlos-jenkins
Copy link

Hello,

For anyone that got here looking for the error'SpooledTemporaryFile' object has no attribute 'seekable'

In my case it was trying to pass anSpooledTemporaryFile to anZipFile to decompress it (Starlette's UploadFile class if anyone cares). But there are other cases. Until the issue referenced in this PR is fixed, an option is to monkey patch the class. Sorry, ugly hack follows.

In one of your modules:

fromtempfileimportSpooledTemporaryFiledef_seekable(self):"""    Monkey patched seekable() method for the SpooledTemporaryFile class.    Sadly, we cannot send an instance of the SpooledTemporaryFile to the    ZipFile object.    This is because the SpooledTemporaryFile doesn't inherit / implement the    IOBase class.    This bug is reported in CPython:        https://bugs.python.org/issue26175    And an unmerged PR is located here:        https://github.com/python/cpython/pull/3249/files    In Python 3.8, attempting to pass the tmpfd fails with:        AttributeError:            'SpooledTemporaryFile' object has no attribute 'seekable'    So, the workaround to avoid copying the file is to monkey patch the class    and add the missing method. Sorry.    """returnself._file.seekable()SpooledTemporaryFile.seekable=_seekable
Molaire, mh739025250, MBogda, and fberanizo reacted with hooray emojiMolaire and fberanizo reacted with heart emoji

@Molaire
Copy link

To Google people: this also breaks pandas.read_csv() and this monkey patch can be used for property readable writable and seekable to fix it.

Hello,

For anyone that got here looking for the error'SpooledTemporaryFile' object has no attribute 'seekable'

In my case it was trying to pass anSpooledTemporaryFile to anZipFile to decompress it (Starlette's UploadFile class if anyone cares). But there are other cases. Until the issue referenced in this PR is fixed, an option is to monkey patch the class. Sorry, ugly hack follows.

In one of your modules:

fromtempfileimportSpooledTemporaryFiledef_seekable(self):"""    Monkey patched seekable() method for the SpooledTemporaryFile class.    Sadly, we cannot send an instance of the SpooledTemporaryFile to the    ZipFile object.    This is because the SpooledTemporaryFile doesn't inherit / implement the    IOBase class.    This bug is reported in CPython:        https://bugs.python.org/issue26175    And an unmerged PR is located here:        https://github.com/python/cpython/pull/3249/files    In Python 3.8, attempting to pass the tmpfd fails with:        AttributeError:            'SpooledTemporaryFile' object has no attribute 'seekable'    So, the workaround to avoid copying the file is to monkey patch the class    and add the missing method. Sorry.    """returnself._file.seekable()SpooledTemporaryFile.seekable=_seekable

@MikeEcho
Copy link

Python dev here from the future (2021 - save yourselves while you can). Still gettingSpooledTemporaryFile errors while reading CSVs (via tempfiles) in. Kindly asking if requested changes will be addressed@GFernie ?

@pR0Ps
Copy link
Contributor

@MikeEcho since it seemed like@GFernie had moved on I submitted an updated and rebased PR for this fix in#29560 with all of the code review comments addressed. Just waiting on a code review.

@github-actions
Copy link

This PR is stale because it has been open for 30 days with no activity.

@github-actionsgithub-actionsbot added the staleStale PR or inactive for long period of time. labelFeb 21, 2022
@methane
Copy link
Member

Close this because I merged#29560.

Sign up for freeto join this conversation on GitHub. Already have an account?Sign in to comment

Reviewers

@pitroupitroupitrou requested changes

@vadmiumvadmiumAwaiting requested review from vadmium

@merwokmerwokAwaiting requested review from merwok

+5 more reviewers

@coleifercoleifercoleifer left review comments

@thehesiodthehesiodthehesiod left review comments

@danieldjewelldanieldjewelldanieldjewell left review comments

@ppperyppperypppery left review comments

@nubirsteinnubirsteinnubirstein approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

awaiting changesstaleStale PR or inactive for long period of time.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

18 participants

@GFernie@the-knights-who-say-ni@bedevere-bot@embe@coleifer@csabella@danieldjewell@merwok@carlos-jenkins@Molaire@MikeEcho@pR0Ps@methane@vadmium@pitrou@thehesiod@pppery@nubirstein

[8]ページ先頭

©2009-2025 Movatter.jp