Bug report

Bug description:

OSS-Fuzz has found a heap buffer overflow in_PyTokenizer_ensure_utf8.Link to OSS-Fuzz bug report.

The root cause is thatvalid_utf8() inParser/tokenizer/helpers.c checks continuation bytes in reverse order thus readers[expected] befores[1] on these lines:

When a multi-byte UTF-8 sequence is truncated - such as a 3-byte lead\xEA followed immediately by a null terminator - the backward loop reads past the end of the valid data before encountering the null byte that would stop it.

This is not a security-critical issue.

CPython versions tested on:

CPython main branch

Operating systems tested on:

No response

Linked PRs

• gh-144872: fix heap buffer overflow_PyTokenizer_ensure_utf8 #144807
• [3.14] gh-144872: fix heap buffer overflow_PyTokenizer_ensure_utf8 (GH-144807) #145287