Uh oh!
There was an error while loading.Please reload this page.
- Notifications
You must be signed in to change notification settings - Fork32.4k
Description
Bug report
If you copy and paste the following code into arepro.py file and runpython -m tokenize on it on a Windows machine, the output is different on 3.12/3.13 compared to what it was on 3.11 (the file ends with a single newline):
foo='bar'spam='eggs'
On Python 3.11, on Windows, the output is this:
> python -m tokenize cpython/repro.py0,0-0,0: ENCODING 'utf-8'1,0-1,3: NAME 'foo'1,4-1,5: OP '='1,6-1,11: STRING "'bar'"1,11-1,13: NEWLINE '\r\n'2,0-2,4: NAME 'spam'2,5-2,6: OP '='2,7-2,13: STRING "'eggs'"2,13-2,15: NEWLINE '\r\n'3,0-3,0: ENDMARKER ''On Python 3.13 (@6e62eb2) on Windows, however, the output is this:
> python -m tokenize repro.py0,0-0,0: ENCODING 'utf-8'1,0-1,3: NAME 'foo'1,4-1,5: OP '='1,6-1,11: STRING "'bar'"1,11-1,12: NEWLINE '\n'2,0-2,4: NAME 'spam'2,5-2,6: OP '='2,7-2,13: STRING "'eggs'"2,13-2,14: NEWLINE '\n'3,0-3,1: NL '\n'4,0-4,0: ENDMARKER ''There appear to be two changes here:
- All the NEWLINE tokens now have
\nvalues, whereas on Python 3.11 they all had\r\nvalues - There is an additional NL token at the end, immediately before the ENDMARKER token.
As discussed inPyCQA/pycodestyle#1142, this appears to be Windows-specific, andmay be the cause of a large number of spurious W391 errors from the pycodestyle linting tool. (W391 dictates that there should be one, and only one, newline at the end of a file.) The pycodestyle tool is included in flake8, meaning that test failures in pycodestyle can cause test failures for other flake8 plugins. (All tests forthe flake8-pyi plugin, for example, currently fail on Python 3.13 on Windows.)
Your environment
Python 3.13.0a0 (heads/main:6e62eb2e70, May 27 2023, 14:00:13) [MSC v.1932 64 bit (AMD64)] on win32Linked PRs
- gh-105017: Fix including additional NL token when using CRLF #105022
- [3.12] gh-105017: Fix including additional NL token when using CRLF (GH-105022) #105023
- gh-105017: Include CRLF lines in strings and column numbers #105030
- [3.12] gh-105017: Include CRLF lines in strings and column numbers (GH-105030) #105041