@lysnikolaou Hummm, can you quickly cover why moving this chunk solves the problem? I think I am missing some key insight here

Sure. There's two different cases to consider here:

1. When we're calling the tokenizer from thetokenize module, it uses the readline tokenizer, which you added in9216e69. In that branch,update_fstring_expr wasn't being calledat all fromtok_underflow_readline, which resulted in thelast_expr_size not being updated. That's why we were passing a negative count toPyUnicode_DecodeUTF8.

cpythonon mainviaCv14.0.3-clangvia 🐍pyenv3.11.3 ❯cattmp/t.pyprint(f'''{3=}''')%cpythonon mainviaCv14.0.3-clangvia 🐍pyenv3.11.3 ❯ ./python.exe-mtokenizetmp/t.pyunexpectederror:NegativesizepassedtoPyUnicode_NewTraceback (mostrecentcalllast):File"/Users/lysnikolaou/repos/python/cpython/Lib/runpy.py",line198,in_run_module_as_mainreturn_run_code(code,main_globals,None,^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^File"/Users/lysnikolaou/repos/python/cpython/Lib/runpy.py",line88,in_run_codeexec(code,run_globals)File"/Users/lysnikolaou/repos/python/cpython/Lib/tokenize.py",line547,in<module>main()File"/Users/lysnikolaou/repos/python/cpython/Lib/tokenize.py",line489,inmaintokens=list(tokenize(f.readline))^^^^^^^^^^^^^^^^^^^^^^^^^^File"/Users/lysnikolaou/repos/python/cpython/Lib/tokenize.py",line447,intokenizeyieldfrom_generate_tokens_from_c_tokenizer(rl_gen.__next__,encoding,extra_tokens=True)File"/Users/lysnikolaou/repos/python/cpython/Lib/tokenize.py",line537,in_generate_tokens_from_c_tokenizerforinfoinit:SystemError:NegativesizepassedtoPyUnicode_New

2. There was a second bug with an f-string being tokenized from a file, only when the closing brace of the f-string expression part is in the last line of the file and there's no newline at the end. In that branch we usetok->cur to calculate the size of the bufferbefore adding an implicit newline and thentok->startafter adding the newline. That resulted in the end buffer being one character too short.

cpythonon mainviaCv14.0.3-clangvia 🐍pyenv3.11.3took15s ❯cattmp/t.pyprint(f'''{3=}''')%cpythonon mainviaCv14.0.3-clangvia 🐍pyenv3.11.3 ❯ ./python.exetmp/t.py33

Do you think I should split this PR in two and open a second issue for the second case above?

Do you think I should split this PR in two and open a second issue for the second case above?

Nah, I think is fine, but we may want to add two NEWS entries

@lysnikolaou If you want add the second NEWS entry and then land it.

Thanks@lysnikolaou for the PR 🌮🎉.. I'm working now to backport this PR to: 3.12.
🐍🍒⛏🤖

GH-105832 is a backport of this pull request to the3.12 branch.