Movatterモバイル変換

[0]ホーム
URL:

Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

 Sign up
Appearance settings

gh-139516: Fix lambda colon start format spec in f-string in tokenizer#139657

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to ourterms of service andprivacy statement. We’ll occasionally send you account related emails.

Already on GitHub?Sign in to your account

Merged
pablogsal merged 4 commits intopython:mainfromtom-pytel:fix-issue-139516
Oct 7, 2025

Conversation

@tom-pytel
Copy link
Contributor

@tom-pyteltom-pytel commentedOct 6, 2025•
edited by bedevere-appbot
Loading

A= followed by a: in an f-string expression could cause the tokenizer to erroneously think it was starting a format spec, leading to incorrect internal state and possible decode errors if this results in split unicode characters on copy. This PR fixes this by disallowing= to setin_debug state unless it is encountered at the top level of an f-string expression.

This problem exists back to py 3.13 and this PR can probably be backported easily enough.

@tom-pytel
Copy link
ContributorAuthor

Ping@pablogsal. I added the test totest_tokenize instead oftest_fstring as it seems to fit there better.

@pablogsal
Copy link
Member

pablogsal commentedOct 6, 2025•
edited
Loading

Please add a rest for the f-string test file as well as this will be a semantic test that needs to hold true even if we change the tokenizer of some other implementation doesn't have the same tokenizer

# gh-139516
# The '\n' is explicit to ensure no trailing whitespace which would invalidate the test.
# Must use tokenize instead of compile so that source is parsed by line which exposes the bug.
list(tokenize.tokenize(BytesIO('''f"{f(a=lambda: 'à'\n)}"'''.encode()).readline))
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

I am confused. Isn't it possible to trigger this in anexec oreval call? Or perhaps a file with an encoding?

Copy link
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

See below VVV

@tom-pytel
Copy link
ContributorAuthor

Please add a rest for the f-string test file as well as this will be a semantic test that needs to hold true even if we change the tokenizer of some other implementation doesn't have the same tokenizer

Done. But I had to usetokenize() because of an interesting quirk. The bug shows up withtokenize() or executing a python script directly with the bad source or typing it into the repl. It does not show up withcompile() orast.parse() oreval() orexec() orimport .... The difference seems to be if the source is read line by line or not, in which case if the full string is available on parse then the tail end of the string past the NL is present to offset from on copy and the bug doesn't present.

Let me know if this test is good enough or if you want something else.

@pablogsal
Copy link
Member

pablogsal commentedOct 6, 2025•
edited
Loading

Let me know if this test is good enough or if you want something else.

yes, going via the tokenizer makes no sense here. The pourpose of what I asked is that alternative implementations will still run these tests files to check if they are compliant and we need to provide a way to run a file or exec some code and say "this is what we expect". You are triggering the bug via a specific aspect of CPython but I would prefer if we could trigger it end-to-end via a file. There are more tests executing python over files, check intest_syntax ortest_grammar ortest_compile.

tom-pytel reacted with thumbs up emoji

@tom-pytel
Copy link
ContributorAuthor

Running error as script.

@pablogsal
Copy link
Member

LGTM

Thank you very much@tom-pytel !

tom-pytel reacted with thumbs up emoji

@pablogsalpablogsal added needs backport to 3.13bugs and security fixes needs backport to 3.14bugs and security fixes labelsOct 7, 2025
@pablogsalpablogsal merged commit539461d intopython:mainOct 7, 2025
53 checks passed
@miss-islington-app
Copy link

Thanks@tom-pytel for the PR, and@pablogsal for merging it 🌮🎉.. I'm working now to backport this PR to: 3.13, 3.14.
🐍🍒⛏🤖

miss-islington pushed a commit to miss-islington/cpython that referenced this pull requestOct 7, 2025
…kenizer (pythonGH-139657)(cherry picked from commit539461d)Co-authored-by: Tomasz Pytel <tompytel@gmail.com>
@miss-islington-app
Copy link

Sorry,@tom-pytel and@pablogsal, I could not cleanly backport this to3.13 due to a conflict.
Please backport usingcherry_picker on command line.

cherry_picker 539461d9ec8e5322ead638f7be733fd196aa6c79 3.13

@bedevere-app
Copy link

GH-139701 is a backport of this pull request to the3.14 branch.

@bedevere-appbedevere-appbot removed the needs backport to 3.14bugs and security fixes labelOct 7, 2025
pablogsal pushed a commit that referenced this pull requestOct 7, 2025
…okenizer (GH-139657) (#139701)gh-139516: Fix lambda colon start format spec in f-string in tokenizer (GH-139657)(cherry picked from commit539461d)Co-authored-by: Tomasz Pytel <tompytel@gmail.com>
@pablogsal
Copy link
Member

Sorry,@tom-pytel and@pablogsal, I could not cleanly backport this to3.13 due to a conflict. Please backport usingcherry_picker on command line.

cherry_picker 539461d9ec8e5322ead638f7be733fd196aa6c79 3.13

@tom-pytel can you make the backport following the instructions?

@tom-pytel
Copy link
ContributorAuthor

Sorry,@tom-pytel and@pablogsal, I could not cleanly backport this to3.13 due to a conflict. Please backport usingcherry_picker on command line.

cherry_picker 539461d9ec8e5322ead638f7be733fd196aa6c79 3.13

@tom-pytel can you make the backport following the instructions?

Sure, in a bit.

@bedevere-app
Copy link

GH-139726 is a backport of this pull request to the3.13 branch.

@bedevere-appbedevere-appbot removed the needs backport to 3.13bugs and security fixes labelOct 7, 2025
pablogsal pushed a commit that referenced this pull requestOct 7, 2025
#139726)[3.13]gh-139516: Fix lambda colon start format spec in f-string in tokenizer (GH-139657)(cherry picked from commit539461d)
Sign up for freeto join this conversation on GitHub. Already have an account?Sign in to comment

Reviewers

@pablogsalpablogsalpablogsal approved these changes

@lysnikolaoulysnikolaouAwaiting requested review from lysnikolaoulysnikolaou is a code owner

Assignees

@pablogsalpablogsal

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@tom-pytel@pablogsal
[8]ページ先頭
©2009-2025 Movatter.jp