My suggestion is to reuse the code fromTarfile._next() to yield a tarinfo each time instead of caching it. This way it doesn't take up a tonne of memory.

I am looking for feedback on this so I will mark this as a draft.

Edit:

As from suggestions a parameter called stream can be passed to the object which will make sure nothing is cached. I will add a code example later today.

Issue:tarfile's cache balloons in memory when streaming a big tarfile #102120

pythongh-102120: Added iter_no_cache to TarFile

affec4f

bedevere-bot added the awaiting review label

Feb 21, 2023

bedevere-bot mentioned this pull request

Feb 21, 2023

tarfile's cache balloons in memory when streaming a big tarfile#102120

Closed

Copy link

Member

ethanfurman commentedFeb 22, 2023

I would rather add a new keyword to theTarFile signature -- perhapsstream -- and update the relevent locations in the code to not updatemembers.

Copy link

ContributorAuthor

PurityLake commentedFeb 22, 2023

That would probably be better

PurityLakeand others added2 commits

February 24, 2023 21:22

pythongh-102120: Added a stream attribute to TarFile

2741ad5

Merge branch 'main' intopythongh-102120-fix-tar-cache

66b7f7f

arhadthedev added the stdlibStandard Library Python modules in the Lib/ directory label

Feb 25, 2023

Copy link

ContributorAuthor

PurityLake commentedFeb 26, 2023

I'll be adding some tests later today

PurityLakeand others added5 commits

March 8, 2023 19:23

pythongh-102120: Added tests for tarfile stream mode

f901e29

pythongh-102120: Added to tarfile docs

62416fe

📜🤖 Added by blurb_it.

c8f57d4

Merge branch 'pythongh-102120-fix-tar-cache' ofhttps://github.com/Pu…

b74bd2a

…rityLake/cpython intopythongh-102120-fix-tar-cache

pythongh-102120: Fixed documentation error

7e9a5f9

PurityLake marked this pull request as ready for review

March 8, 2023 20:08

PurityLake requested a review fromethanfurman as acode owner

March 8, 2023 20:08

Merge branch 'main' intopythongh-102120-fix-tar-cache

ab10974

Copy link

ContributorAuthor

PurityLake commentedMar 14, 2023

I believe this is working as intended now. I'm open to criticism in the implementation

Merge branch 'main' intopythongh-102120-fix-tar-cache

aae14d4

ethanfurman reviewed

May 23, 2023

View reviewed changes

Doc/library/tarfile.rst OutdatedShow resolvedHide resolved

Update wording on Doc/library/tarfile.rst

3ce2b8a

Co-authored-by: Ethan Furman <ethan@stoneleaf.us>

ethanfurman requested changes

May 23, 2023

View reviewed changes

Copy link

Member

ethanfurman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

Overall, looks good. I missed feature-freeze, though, so this will have to go in 3.13. Update the3.12s to3.13s and I'll get it merged.

bedevere-bot added awaiting changes and removed awaiting review labels

May 23, 2023

Copy link

bedevere-bot commentedMay 23, 2023

A Python core developer has requested some changes be made to your pull request before we can consider merging it. If you could please address their requests along with any other requests in other reviews from core developers that would be appreciated.

Once you have made the requested changes, please leave a comment on this pull request containing the phraseI have made the requested changes; please review again. I will then notify any core developers who have left a review that you're ready for them to take another look at this pull request.