Movatterモバイル変換

[0]ホーム
URL:

Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

 Sign up
Appearance settings

gh-132983: Don't allow trailer data in ZstdFile#133736

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to ourterms of service andprivacy statement. We’ll occasionally send you account related emails.

Already on GitHub?Sign in to your account

Merged
AA-Turner merged 1 commit intopython:mainfromRogdham:zstdfile-trailer-exception
May 10, 2025

Conversation

Rogdham
Copy link
Contributor

@RogdhamRogdham commentedMay 9, 2025•
edited
Loading

We previously made sure that an exception is raised when decompressing trailer data withdecompress:

>>>fromcompression.zstdimportcompress,decompress>>>invalid=compress(b'xxx')+b'yyy'>>>decompress(invalid)Traceback (mostrecentcalllast):File"<python-input-2>",line1,in<module>decompress(invalid)~~~~~~~~~~^^^^^^^^^File"/redacted/Lib/compression/zstd/__init__.py",line157,indecompressresults.append(decomp.decompress(data))~~~~~~~~~~~~~~~~~^^^^^^_zstd.ZstdError:Unabletodecompresszstddata:Unknownframedescriptor

Indeed,the Zstandard specification says “Zstandard compressed data is made of one or more frames”, and it does not say that random data can be added at the end.

However, this is not the case inZstdFile /zstd.open:

>>>fromcompression.zstdimportZstdFile>>>fromioimportBytesIO>>>ZstdFile(BytesIO(invalid)).read()b'xxx'

After this PR, the last call becomes:

>>>ZstdFile(BytesIO(invalid)).read()Traceback (mostrecentcalllast):File"<python-input-5>",line1,in<module>ZstdFile(BytesIO(invalid)).read()~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^File"/redacted/Lib/compression/zstd/_zstdfile.py",line176,inreadreturnself._buffer.read(size)~~~~~~~~~~~~~~~~~^^^^^^File"/redacted/Lib/compression/_common/_streams.py",line118,inreadallwhiledata:=self.read(sys.maxsize):~~~~~~~~~^^^^^^^^^^^^^File"/redacted/Lib/compression/_common/_streams.py",line91,inreaddata=self._decompressor.decompress(rawblock,size)_zstd.ZstdError:Unabletodecompresszstddata:Unknownframedescriptor

@RogdhamRogdham marked this pull request as ready for reviewMay 9, 2025 09:36
@AA-TurnerAA-Turner added the needs backport to 3.14bugs and security fixes labelMay 9, 2025
@emmatyping
Copy link
Member

The current behavior matches LZMA. I think unlikedecompress which is handed what is necessarily a zstd stream of one or more frames, withZstdFile, a user may be parsing a format which has additional information after a zstd stream.

>>>from lzmaimport LZMAFile, compress>>>from ioimport BytesIO>>> invalid= compress(b'foo')+b'bar'>>> LZMAFile(BytesIO(invalid)).read()b'foo'>>>

@Rogdham
Copy link
ContributorAuthor

You are right this is the case forLZMAFile with formatFORMAT_AUTO (which is the default) and also forBZ2File.

However,LZMAFile with formatFORMAT_XZ as well asGzipFile raise an exception in that case.

>>>fromlzmaimportLZMAFile,compress,FORMAT_XZ>>>fromioimportBytesIO>>>invalid=compress(b'foo')+b'bar'>>>LZMAFile(BytesIO(invalid),format=FORMAT_XZ).read()Traceback (mostrecentcalllast):File"<python-input-3>",line1,in<module>LZMAFile(BytesIO(invalid),format=FORMAT_XZ).read()~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^File"/redacted/lzma.py",line208,inreadreturnself._buffer.read(size)~~~~~~~~~~~~~~~~~^^^^^^File"/redacted/_compression.py",line118,inreadallwhiledata:=self.read(sys.maxsize):~~~~~~~~~^^^^^^^^^^^^^File"/redacted/_compression.py",line99,inreadraiseEOFError("Compressed file ended before the ""end-of-stream marker was reached")EOFError:Compressedfileendedbeforetheend-of-streammarkerwasreached
emmatyping reacted with thumbs up emoji

Copy link
Member

@emmatypingemmatyping left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

Okay this looks good then!

@Rogdham
Copy link
ContributorAuthor

Rogdham commentedMay 9, 2025•
edited
Loading

In addition, considerdecompress(compress(b"xxx") + b"yyy"):

  • returnsb"xxx" on:lzma (formatFORMAT_AUTO),bz2
  • raises an exception on:lzma (formatFORMAT_XZ),gzip

Since forzstd we raise an exception on that, I would say to do the same forZstdFile to be consistent.

emmatyping reacted with thumbs up emoji

@AA-TurnerAA-Turner merged commit50b5370 intopython:mainMay 10, 2025
48 checks passed
@miss-islington-app
Copy link

Thanks@Rogdham for the PR, and@AA-Turner for merging it 🌮🎉.. I'm working now to backport this PR to: 3.14.
🐍🍒⛏🤖

miss-islington pushed a commit to miss-islington/cpython that referenced this pull requestMay 10, 2025
(cherry picked from commit50b5370)Co-authored-by: Rogdham <3994389+Rogdham@users.noreply.github.com>
@bedevere-app
Copy link

GH-133799 is a backport of this pull request to the3.14 branch.

@bedevere-appbedevere-appbot removed the needs backport to 3.14bugs and security fixes labelMay 10, 2025
AA-Turner pushed a commit that referenced this pull requestMay 10, 2025
…133799)gh-132983: Don't allow trailer data in ZstdFile (GH-133736)(cherry picked from commit50b5370)Co-authored-by: Rogdham <3994389+Rogdham@users.noreply.github.com>
@RogdhamRogdham deleted the zstdfile-trailer-exception branchMay 10, 2025 06:45
Sign up for freeto join this conversation on GitHub. Already have an account?Sign in to comment
Reviewers

@emmatypingemmatypingemmatyping approved these changes

Assignees
No one assigned
Labels
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

3 participants
@Rogdham@emmatyping@AA-Turner
[8]ページ先頭
©2009-2025 Movatter.jp