In Python 3.12, porting the tokenizer to use the C tokenizer underneath to support PEP 701 has now a documented change indocs.python.org/3.12/whatsnew/3.12.html#changes-in-the-python-api:

Some final DEDENT tokens are now emitted within the bounds of the input. This means that for a file containing 3 lines, the old version of the tokenizer returned a DEDENT token in line 4 whilst the new version returns the token in line 3.

Apparently, this affects negatively some formatting tools (seePyCQA/pycodestyle#1142). Let's consider what options do we have and see if we can fix this without adding a lot of maintenance burden to the C tokenizer or slowing down everything.

Linked PRs

• gh-104976: Ensure trailing dedent tokens are emitted as the previous tokenizer #104980
• [3.12] gh-104976: Ensure trailing dedent tokens are emitted as the previous tokenizer (GH-104980) #105000