Movatterモバイル変換


[0]ホーム

URL:


Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

Sign up
Appearance settings

Allow full unfiltering for partial data (enum-based approach)#664

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to ourterms of service andprivacy statement. We’ll occasionally send you account related emails.

Already on GitHub?Sign in to your account

Open
anforowicz wants to merge3 commits intoimage-rs:master
base:master
Choose a base branch
Loading
fromanforowicz:partial-decode-december-2025

Conversation

@anforowicz
Copy link
Contributor

@anforowiczanforowicz commentedDec 10, 2025
edited
Loading

Hello!

This PRfixes#639. This is an alternative approach to@197g's#640 - the two main differences are:

  • prev_start: usize is replaced withprev_row: PrevRow wherePrevRow is anenum with separateNone,InPlace, andScratch variants. I think this results in easier to read code then imposing a special meaning on certain values of offsets (e.g.self.prev_start == self.curr_start => no previous row; or havingself.prev_start sometimes trailself.curr_start byrowlen-1 and sometimes point to mutable scratch space in the same buffer ascurr_start).
  • There is a test and a fix for consistently detecting errors in the tail portion of zlib stream (see the discussion atAllow full unfiltering for partial data #640 (comment)). This part may conflict withHave UnfilteringBuffer track number of remaining bytes in the frame #662 where@fintelia proposes to always ignore the tail zlib bytes (rather than as in this PR, to always decode them to detect errors). I think I should wait until@fintelia's PR lands, and then rebase on top of it (removing changes inthis PR that won't be necessary afterwards).

PTAL?

The refactoring in this commit is desirable to support shifting by adifferent `discard_size` (i.e. making that independent from`self.prev_start`).This commit also opportunistically adds an `assert!` that ensures thatshifting left won't accidentally clobber the immutable "lookback"window.
The refactoring in this commit is desirable because:* Main reason: To support calling the extracted function from another  place in a follow-up commit.* Secondary reason: To improve readability a little bit, by making  `fn unfilter_curr_row` slightly less noisy / more focused on its core  functionality, which is:    - extracting `prev_row`, and `row`    - calling `unfilter`    - updating `current_start` and `prev_start`
This commit makes the following two changes:* Falls back to unfiltering the current row out-of-place if an  `UnexpectedEof` error is encountered when in-place bytes cannot yet be  mutated.  Thisfixesimage-rs#639.  See the following changes:    - `test_partial_decode_success` in `tests/partial_decode.rs`    - Changes in `fn next_raw_interlaced_row` method of `Reader`      in `src/decoder/mod.rs`    - Changes in `src/decoder/unfiltering_buffer.rs`* Ensures that `finish_compressed_chunks` is always called (even if all  rows have already been successfully decoded).  This is required to  deterministically report errors, regardless of how the input is  buffered.  The other changes above would have resulted in fuzzing  failures without fixing this.  See the following changes:    - `test_partial_decode_with_unterminated_redundant_zlib_tail`      in `tests/partial_decode.rs`    - Changes in `fn finish_decoding` method of `Reader`      in `src/decoder/mod.rs`    - Changes of `src/decoder/read_decoder.rs`
@anforowicz
Copy link
ContributorAuthor

There are someCIFuzz failures, but they seem to be caused by some infrastructure problems (E: Package 'python' has no installation candidate) rather than by this PR? FWIW I've been runningcargo fuzz run buf_independent for the last hour or so on my local machine and it didn't find any issues so far.

@197g
Copy link
Member

Did you use the corpus from oss-fuzz? For what it's worth the fuzz failure on the other PR is also somewhat spurious, it's a highly specific condition and fuzzing locally never found the same fault for me either (even after 64-threads for a day, it's just peanuts).

@anforowicz
Copy link
ContributorAuthor

Did you use the corpus from oss-fuzz?

I've just runcargo fuzz run buf_independent and when that didn't find anything for a while, I've added--jobs 60.

Would you have instructions for using oss-fuzz? (We may want to stash them intofuzz/README.md.)

For what it's worth the fuzz failure on the other PR is also somewhat spurious, it's a highly specific condition and fuzzing locally never found the same fault for me either (even after 64-threads for a day, it's just peanuts).

Without changingsrc/decoder/read_decoder.rs I was able to find the discrepancy betweenbaseline_reader andintermittent_eofs_reader within 5-10 minutes of fuzzing locally.

@197g
Copy link
Member

Would you have instructions for using oss-fuzz? (We may want to stash them into fuzz/README.md.)

Sorry, I've always relied on the API job for this and hoped you'd be closer to the source 😄 . For some concrete failures there are the failures on the mailing list but I would like to know how to retrieve the full corpus myself.

@197g
Copy link
Member

197g commentedDec 11, 2025
edited
Loading

Update: the fuzzer results are now running and look on first site like a genuine detection.

2025-12-11T16:06:58.4416156Z     #0 0x56378293a841 in __sanitizer_print_stack_trace /rustc/llvm/src/llvm-project/compiler-rt/lib/asan/asan_stack.cpp:87:32025-12-11T16:06:58.4430239Z     #1 0x563782a51ba8 in fuzzer::PrintStackTrace() /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerUtil.cpp:210:52025-12-11T16:06:58.4431699Z     #2 0x563782a3566b in fuzzer::Fuzzer::AlarmCallback() /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerLoop.cpp:304:52025-12-11T16:06:58.4432650Z     #3 0x7f9a7bd3d32f  (/lib/x86_64-linux-gnu/libc.so.6+0x4532f) (BuildId: 274eec488d230825a136fa9c4d85370fed7a0a5e)2025-12-11T16:06:58.4433501Z     #4 0x5637829fef63 in fdeflate::decompress::Decompressor::read::h5122298fcea00f39 /rust/registry/src/index.crates.io-6f17d22bba15001f/fdeflate-0.3.7/src/decompress.rs2025-12-11T16:06:58.4434526Z     #5 0x56378297e455 in png::decoder::zlib::UnfilterBuf::decompress::h9f4dbf88ef0f2ad9 /src/image-png/src/decoder/zlib.rs:199:43

The first assert I see for that method is assert!(output_position <= output.len());

Sign up for freeto join this conversation on GitHub. Already have an account?Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Unexpected change of behavior when reading partial data

2 participants

@anforowicz@197g

[8]ページ先頭

©2009-2025 Movatter.jp