Your environment

• CPython versions tested on: 3.12.0 alpha 7
• Operating system and architecture: ubuntu20.04.1,x86_64
• Compiler flags: clang with ASAN and UBSAN instrument

Bug description

The AddressSanitizer (ASAN) tool has detected multiple heap-use-after-free errors and a segmentation fault (SEGV) in the Python interpreter. The heap-use-after-free errors occurred in the ascii_decode and unicode_decode_utf8 functions in the unicodeobject.c file, and the SEGV occurred in the tok_backup function in the tokenizer.c file. Additionally, a memory leak was detected in the pystate.c file.

Steps to reproduce

1. Compile Python with ASAN enabled:./configure && make
2. Run Python with ASAN enabled:./python < poc_file
3. The heap-use-after-free errors and SEGV should be detected and logged by ASAN.

Expected behavior

No heap-use-after-free errors or SEGV should occur.

Actual behavior

ASAN detected multiple heap-use-after-free errors and a SEGV, as well as a memory leak.

Relevant logs and/or screenshots

The ASAN summary output is as follows:

AddressSanitizer: heap-use-after-free /src/cpython/Objects/unicodeobject.c:4474:28 in ascii_decodeAddressSanitizer: heap-use-after-free /src/cpython/Objects/unicodeobject.c:4506:28 in ascii_decodeAddressSanitizer: heap-use-after-free /src/cpython/Objects/unicodeobject.c:4483:32 in ascii_decodeAddressSanitizer: SEGV /src/cpython/Parser/tokenizer.c:1234:33 in tok_backupAddressSanitizer: heap-use-after-free /src/cpython/Objects/unicodeobject.c:4526:37 in unicode_decode_utf8AddressSanitizer: 3824 byte(s) leaked in 4 allocation(s).AddressSanitizer: heap-use-after-free /src/cpython/Python/pystate.c:229:23 in bind_tstateThe full ASAN log can be found in the asan.log file.

asan.log
python_bug_poc.zip

Linked PRs

• gh-103824: fix use-after-free error in Parser/tokenizer.c #103993