Bug report

Bug description:

importtokenize,iosource_code=r'''f"""{80 * '*'}\n{{test}}{{details}}{{test2}}\n{80 * '*'}"""'''tokens=tokenize.generate_tokens(io.StringIO(source_code).readline)x=tokenize.untokenize((t,s)fort,s,*_intokens)print(x)

Expected:

f"""{80 *'*'}\n{{test}}{{details}}{{test2}}\n{80 *'*'}"""

Got:

f"""{80 *'*'}\n{test}}{{details}}{{test2}}\n{80 *'*'}"""

Note the absence of a second { in the {{ after the \n — but in no other positions.

Unlike some other roundtrip failures of tokenize, some of which are minor infelicities, this one actually creates a syntactically invalid program on roundtrip, which is quite bad. You get aSyntaxError: f-string: single '}' is not allowed when trying to use the results.

CPython versions tested on:

3.12

Operating systems tested on:

Linux, Windows

Linked PRs

• gh-125008: Fixtokenize.untokenize roundtrip for\n{{ #125013
• [3.13] gh-125008: Fixtokenize.untokenize roundtrip for\n{{ (GH-125013) #125020
• [3.12] gh-125008: Fixtokenize.untokenize roundtrip for\n{{ (GH-125013) #125021