Movatterモバイル変換


[0]ホーム

URL:


Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

Sign up
Appearance settings

generate_tokens starts to giveSyntaxError #105549

Closed
Assignees
pablogsallysnikolaou
Labels
3.12only security fixes3.13bugs and security fixestype-bugAn unexpected behavior, bug, or error
@sylee957

Description

@sylee957

Bug report

generate_tokens gives error in python 3.12

fromtokenizeimportgenerate_tokensfromioimportStringIOlist(generate_tokens(StringIO('01234').readline))
Traceback (most recent call last):  File "<stdin>", line 1, in <module>  File "/home/sylee957/.pyenv/versions/3.12.0b1/lib/python3.12/tokenize.py", line 451, in _tokenize    for token in _generate_tokens_from_c_tokenizer(source, extra_tokens=True):  File "/home/sylee957/.pyenv/versions/3.12.0b1/lib/python3.12/tokenize.py", line 542, in _generate_tokens_from_c_tokenizer    for info in c_tokenizer.TokenizerIter(source, extra_tokens=extra_tokens):  File "<string>", line 1    01234    ^SyntaxError: leading zeros in decimal integer literals are not permitted; use an 0o prefix for octal integers

While in Python 3.11, it successfully returns some result

[TokenInfo(type=2 (NUMBER), string='0', start=(1, 0), end=(1, 1), line='01234'), TokenInfo(type=2 (NUMBER), string='1234', start=(1, 1), end=(1, 5), line='01234'), TokenInfo(type=4 (NEWLINE), string='', start=(1, 5), end=(1, 6), line=''), TokenInfo(type=0 (ENDMARKER), string='', start=(2, 0), end=(2, 0), line='')]

This is related to some issues in SymPy
sympy/sympy#25185

Your environment

  • CPython versions tested on: 3.12.0b1
  • Operating system and architecture: Ubuntu 22.04.2 LTS

Linked PRs

Metadata

Metadata

Labels

3.12only security fixes3.13bugs and security fixestype-bugAn unexpected behavior, bug, or error

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions


    [8]ページ先頭

    ©2009-2025 Movatter.jp