Movatterモバイル変換

[0]ホーム
URL:

Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

 Sign up
Appearance settings

gh-63161: Fix tokenize.detect_encoding()#139446

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to ourterms of service andprivacy statement. We’ll occasionally send you account related emails.

Already on GitHub?Sign in to your account

Conversation

@serhiy-storchaka
Copy link
Member

@serhiy-storchakaserhiy-storchaka commentedSep 30, 2025•
edited by bedevere-appbot
Loading

  • Support non-UTF-8 shebang and comments if non-UTF-8 encoding is specified.
  • Detect decoding error for non-UTF-8 encoding.
  • Detect null bytes in source code.

* Support non-UTF-8 shebang and comments if non-UTF-8 encoding is specified.* Detect decoding error for non-UTF-8 encoding.* Detect null bytes in source code.
@serhiy-storchaka
Copy link
MemberAuthor

It is a draft until we fix the Python interpreter.

@serhiy-storchakaserhiy-storchaka marked this pull request as ready for reviewOctober 13, 2025 12:36
@serhiy-storchakaserhiy-storchaka merged commit38d4b43 intopython:mainOct 20, 2025
47 checks passed
@serhiy-storchakaserhiy-storchaka deleted the tokenize-detect_encoding branchOctober 20, 2025 17:08
@serhiy-storchakaserhiy-storchaka added the needs backport to 3.14bugs and security fixes labelOct 20, 2025
@miss-islington-app
Copy link

Thanks@serhiy-storchaka for the PR 🌮🎉.. I'm working now to backport this PR to: 3.14.
🐍🍒⛏🤖

miss-islington pushed a commit to miss-islington/cpython that referenced this pull requestOct 20, 2025
* Support non-UTF-8 shebang and comments if non-UTF-8 encoding is specified.* Detect decoding error for non-UTF-8 encoding.* Detect null bytes in source code.(cherry picked from commit38d4b43)Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>
@bedevere-app
Copy link

GH-140378 is a backport of this pull request to the3.14 branch.

@bedevere-appbedevere-appbot removed the needs backport to 3.14bugs and security fixes labelOct 20, 2025
serhiy-storchaka added a commit that referenced this pull requestOct 20, 2025
* Support non-UTF-8 shebang and comments if non-UTF-8 encoding is specified.* Detect decoding error for non-UTF-8 encoding.* Detect null bytes in source code.(cherry picked from commit38d4b43)Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>
@bedevere-bot
Copy link

⚠️⚠️⚠️ Buildbot failure⚠️⚠️⚠️

Hi! The buildbotAMD64 CentOS9 NoGIL Refleaks 3.x (tier-1) has failed when building commit38d4b43.

What do you need to do:

  1. Don't panic.
  2. Checkthe buildbot page in the devguide if you don't know what the buildbots are or how they work.
  3. Go to the page of the buildbot that failed (https://buildbot.python.org/#/builders/1610/builds/2273) and take a look at the build logs.
  4. Check if the failure is related to this commit (38d4b43) or if it is a false positive.
  5. If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/#/builders/1610/builds/2273

Failed tests:

  • test_free_threading

Test leaking resources:

  • test_free_threading: file descriptors
  • test_free_threading: memory blocks

Summary of the results of the build (if available):

==

Click to see traceback logs
remote:Enumerating objects: 11, done.remote:Counting objects:  11% (1/9)remote:Counting objects:  22% (2/9)remote:Counting objects:  33% (3/9)remote:Counting objects:  44% (4/9)remote:Counting objects:  55% (5/9)remote:Counting objects:  66% (6/9)remote:Counting objects:  77% (7/9)remote:Counting objects:  88% (8/9)remote:Counting objects: 100% (9/9)remote:Counting objects: 100% (9/9), done.remote:Compressing objects:  16% (1/6)remote:Compressing objects:  33% (2/6)remote:Compressing objects:  50% (3/6)remote:Compressing objects:  66% (4/6)remote:Compressing objects:  83% (5/6)remote:Compressing objects: 100% (6/6)remote:Compressing objects: 100% (6/6), done.remote:Total 11 (delta 3), reused 3 (delta 3), pack-reused 2 (from 2)From https://github.com/python/cpython * branch                    main       -> FETCH_HEADNote:switching to '38d4b436ca767351db834189b3a5379406cd52a8'.You are in 'detached HEAD' state. You can look around, make experimentalchanges and commit them, and you can discard any commits you make in thisstate without impacting any branches by switching back to a branch.If you want to create a new branch to retain commits you create, you maydo so (now or later) by using -c with the switch command. Example:  git switch -c <new-branch-name>Or undo this operation with:  git switch -Turn off this advice by setting config variable advice.detachedHead to falseHEAD is now at 38d4b436ca7 gh-63161: Fix tokenize.detect_encoding() (GH-139446)Switched to and reset branch 'main'configure:WARNING: no system libmpdec found; falling back to pure-Python version for the decimal modulemake:*** [Makefile:2489: buildbottest] Error 2

@bedevere-bot
Copy link

⚠️⚠️⚠️ Buildbot failure⚠️⚠️⚠️

Hi! The buildbotAMD64 FreeBSD Refleaks 3.x (tier-3) has failed when building commit38d4b43.

What do you need to do:

  1. Don't panic.
  2. Checkthe buildbot page in the devguide if you don't know what the buildbots are or how they work.
  3. Go to the page of the buildbot that failed (https://buildbot.python.org/#/builders/1613/builds/2204) and take a look at the build logs.
  4. Check if the failure is related to this commit (38d4b43) or if it is a false positive.
  5. If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/#/builders/1613/builds/2204

Test leaking resources:

  • test_events: references
  • test_events: memory blocks

Summary of the results of the build (if available):

==

Click to see traceback logs
Traceback (most recent call last):  File"/buildbot/buildarea/3.x.ware-freebsd.refleak/build/Lib/test/support/__init__.py", line847, ingc_collect    gc.collect()~~~~~~~~~~^^ResourceWarning:unclosed <socket.socket fd=9, family=2, type=1, proto=6, laddr=('127.0.0.1', 12865), raddr=('127.0.0.1', 12866)>Task was destroyed but it is pending!task:<Task pending name='Task-4087' coro=<BaseSelectorEventLoop._accept_connection2() done, defined at /buildbot/buildarea/3.x.ware-freebsd.refleak/build/Lib/asyncio/selector_events.py:217> wait_for=<Future pending cb=[Task.task_wakeup()]>>Warning -- Unraisable exceptionException ignored while calling deallocator <function _SelectorTransport.__del__ at 0x83ecdd310>:Traceback (most recent call last):  File"/buildbot/buildarea/3.x.ware-freebsd.refleak/build/Lib/asyncio/selector_events.py", line873, in__del__    _warn(f"unclosed transport{self!r}",ResourceWarning,source=self)~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ResourceWarning:unclosed transport <_SelectorSocketTransport closing fd=9>kTraceback (most recent call last):  File"/buildbot/buildarea/3.x.ware-freebsd.refleak/build/Lib/test/support/__init__.py", line847, ingc_collect    gc.collect()~~~~~~~~~~^^ResourceWarning:unclosed <socket.socket fd=9, family=2, type=1, proto=6, laddr=('127.0.0.1', 20768), raddr=('127.0.0.1', 20769)>Task was destroyed but it is pending!task:<Task pending name='Task-1035' coro=<BaseSelectorEventLoop._accept_connection2() done, defined at /buildbot/buildarea/3.x.ware-freebsd.refleak/build/Lib/asyncio/selector_events.py:217> wait_for=<Future pending cb=[Task.task_wakeup()]>>Warning -- Unraisable exceptionException ignored while calling deallocator <function _SelectorTransport.__del__ at 0x83a6c6810>:Traceback (most recent call last):  File"/buildbot/buildarea/3.x.ware-freebsd.refleak/build/Lib/asyncio/selector_events.py", line873, in__del__    _warn(f"unclosed transport{self!r}",ResourceWarning,source=self)~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ResourceWarning:unclosed transport <_SelectorSocketTransport closing fd=9>k

@asottile
Copy link
Contributor

changing this regex to bytes appears to break code:asottile/pyupgrade#1038

@serhiy-storchaka
Copy link
MemberAuthor

This is not a public API.

Sign up for freeto join this conversation on GitHub. Already have an account?Sign in to comment

Reviewers

@vstinnervstinnerAwaiting requested review from vstinner

@pablogsalpablogsalAwaiting requested review from pablogsalpablogsal is a code owner

@lysnikolaoulysnikolaouAwaiting requested review from lysnikolaoulysnikolaou is a code owner

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@serhiy-storchaka@bedevere-bot@asottile
[8]ページ先頭
©2009-2025 Movatter.jp