tokenize does not handle line joining properly, as the following stringfails the CPython tokenizer but passes the tokenize module.Example 1:>>> s = "if 1:\n  \\\n  #hey\n  print 1">>> exec sTraceback (most recent call last):  File "<stdin>", line 1, in <module>  File "<string>", line 3    #hey       ^SyntaxError: invalid syntax>>> tokenize.tokenize(StringIO(s).readline)1,0-1,2:NAME'if'1,3-1,4:NUMBER'1'1,4-1,5:OP':'1,5-1,6:NEWLINE'\n'2,0-2,2:INDENT'  '3,2-3,6:COMMENT'#hey'3,6-3,7:NEWLINE'\n'4,2-4,7:NAME'print'4,8-4,9:NUMBER'1'5,0-5,0:DEDENT''5,0-5,0:ENDMARKER''

CPython allows \ at EOF, but tokenize does not.>>> s = 'print 1\\\n'>>> exec s1>>> tokenize.tokenize(StringIO(s).readline)1,0-1,5:NAME'print'1,6-1,7:NUMBER'1'Traceback (most recent call last):  File "<stdin>", line 1, in <module>  File"/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/tokenize.py",line 153, in tokenize    tokenize_loop(readline, tokeneater)  File"/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/tokenize.py",line 159, in tokenize_loop    for token_info in generate_tokens(readline):  File"/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/tokenize.py",line 283, in generate_tokens    raise TokenError, ("EOF in multi-line statement", (lnum, 0))tokenize.TokenError: ('EOF in multi-line statement', (2, 0))

Nobody appears to be interested so I'll close this in a couple of weeks unless someone objects, unless a patch is provided.

Mark, please stop closing these based on age.The needs to be a determination whether thisis a valid bug.  If so, then a patch is needed.If not, it can be closed.

That syntax error is coming from the CPython parser and *not* the tokenizer.  Both CPython and the 'tokenizer' modules produce the same tokenization:[meadori@motherbrain cpython]$ cat repro.pyif 1:  \  pass[meadori@motherbrain cpython]$ ./python tokenize.py repro.py 0,0-0,0:        ENCODING        'utf-8'1,0-1,2:        NAME            'if'1,3-1,4:        NUMBER          '1'1,4-1,5:        OP              ':'1,5-1,6:        NEWLINE         '\n'2,0-2,2:        INDENT          '  '3,0-3,1:        NEWLINE         '\n'4,2-4,6:        NAME            'pass'4,6-4,7:        NEWLINE         '\n'5,0-5,0:        DEDENT          ''5,0-5,0:        ENDMARKER       ''[44319 refs][meadori@motherbrain cpython]$ ./python -d repro.py | grep Token | tail -10  File "repro.py", line 3        ^SyntaxError: invalid syntax[44305 refs]Token NEWLINE/'' ... It's a token we knowToken DEDENT/'' ... It's a token we knowToken NEWLINE/'' ... It's a token we knowToken ENDMARKER/'' ... It's a token we knowToken NAME/'if' ... It's a keywordToken NUMBER/'1' ... It's a token we knowToken COLON/':' ... It's a token we knowToken NEWLINE/'' ... It's a token we knowToken INDENT/'' ... It's a token we knowToken NEWLINE/'' ... It's a token we knowThe NEWLINE INDENT NEWLINE tokenization causes the parser to choke because 'suite' nonterminals:suite: simple_stmt | NEWLINE INDENT stmt+ DEDENTare defined as NEWLINE INDENT.It seems appropriate that the NEWLINE after INDENT should be dropped by both tokenizers.  In other words, I think:"""if 1:  \  pass"""should produce the same tokenization as:"""if 1:    pass"""This seems consistent with with how explicit line joining is defined [2].[1]http://hg.python.org/cpython/file/92842e347d98/Grammar/Grammar[2]http://docs.python.org/reference/lexical_analysis.html#explicit-line-joining

Here's an example in the wild which still reproduces with python3.8a3:https://github.com/SecureAuthCorp/impacket/blob/194b22ed2fc85c4f241375fb7ebe4e0d89626c8c/impacket/examples/remcomsvc.py#L1669This was reported as a bug on flake8:https://gitlab.com/pycqa/flake8/issues/532Here's the reproduction with python3.8:$ python3.8 --version --versionPython 3.8.0a3 (default, Mar 27 2019, 03:46:44) [GCC 7.3.0]$ python3.8 impacket/examples/remcomsvc.py $ python3.8 -mtokenize impacket/examples/remcomsvc.py impacket/examples/remcomsvc.py:1670:0: error: EOF in multi-line statement

New changesetabea73bf4a320ff658c9a98fef3d948a142e61a9 by Miss Islington (bot) (Anthony Sottile) in branch 'master':bpo-2180: Treat line continuation at EOF as a `SyntaxError` (GH-13401)https://github.com/python/cpython/commit/abea73bf4a320ff658c9a98fef3d948a142e61a9

Thanks for figuring this one out Anthony! :)