Currently most syntax errors raised in the compiler (except these raised in the parser) usePyErr_ProgramTextObject() to get the line of the code. It does not know the encoding of the source file and interpret it as UTF-8 (failing if it contain non-UTF-8 sequences). The parser uses_PyErr_ProgramDecodedTextObject().

There are two ways to solve this issue:

• Pass the source file encoding from the parser to the code generator. This may require changing some data structures. But this is more efficient.
• Detect the encoding inPyErr_ProgramTextObject(). Since the latter is in the public C API, this can also affect the third-party code.

There are other issues withPyErr_ProgramTextObject():

• It leave the BOM in the first line if the source line contains it. This is not consistent with offsets.
• For very long lines, it returns the tail of the line that exceeds 1000 bytes. It can be short, it can start with invalid character, it is not consistent with offsets. If return incomplete line, it is better to return the head.

This all applies toPyErr_ProgramText() as well.

Linked PRs

• gh-124188: Fix PyErr_ProgramTextObject() #124189
• [3.13] gh-124188: Fix PyErr_ProgramTextObject() (GH-124189) #124423
• [3.12] gh-124188: Fix PyErr_ProgramTextObject() (GH-124189) #124426