Just fyi this code right here is the one that I mentioned inwe-like-parsers#121 (comment)

Whereby a SyntaxError with an offset relative to the start of the file will end up pointing to the right place. I'm tempted to say we should eventually just remove it since the new parser will always provide line-relative offsets.

Hm... Why would the offset in the SyntaxError object ever end up being file-relative? Do you know of any code that produces such SyntaxErrors?

In the old parser it was this code:

tok->cur is the current read index of the tokenizer andtok->buf is thestart of the file. Also see how it copies the entire file up till the error into theSyntaxError.text field.

Try this out with the old parser as a quick example:

code="""\a =\\\\\\?"""try:compile(code,'<stdin>','exec')exceptSyntaxErrorase:print(e)print(e.lineno,e.offset)

Oh, you mean the opposite of file. :-) It occurs when it's not read from a file.

Also it doesn't occur all the time -- perhaps only when there's a continuation line? E.g. here all is good:

>>> try: compile("def f():\n 1+\n", "", "exec")... except SyntaxError as e: e... SyntaxError('invalid syntax', ('', 2, 4, ' 1+\n'))>>>

FWIW, your example does show up weird in the old parser:

>>> code = """\a = \\    \\    \\?"""... ... ... >>> code'a = \\\n    \\\n    \\?'>>> try: compile(code, "", "exec")... except SyntaxError as e: e... SyntaxError('unexpected character after line continuation character', ('', 3, 19, 'a = \\\n    \\\n    \\?\n'))>>>

The new parser seems to solve the dilemma by suppressing the source text (also the offset is set to zero, meaning unknown):

>>> code = """\a = \\    \\    \\?"""... ... ... >>> >>> try: compile(code, "", "exec")... except Exception as e: e; e.lineno, e.offset, e.text... SyntaxError('unexpected character after line continuation character')(3, 0, None)>>>

Aah yes, it seems like it's just an issue with line continuations. I thought I had another example but it must have been a misunderstanding becausetok->buf gets advanced with newlines.

So I guess this code here inpythonrun.c is just for this one case?

Okay, so I have a serious question here. Is there a regression due to this PR for people using-X oldparser? I have tried to research this and cannot find a regression. The C code in pythonrun.c still skips through newlines. The traceback.py code doesn't, but it never did.

Here is what I did for research. First Python 3.8:

>>> import traceback; traceback.print_exception(None, SyntaxError("msg", ("f.py", 3, 10, "aaa\nbbb\nccc\n")), None)  File "f.py", line 3    aaabbbccc           ^SyntaxError: msg>>> raise SyntaxError("msg", ("f.py", 3, 10, "aaa\nbbb\nccc\n"))Traceback (most recent call last):  File "<stdin>", line 1, in <module>  File "f.py", line 3    ccc     ^SyntaxError: msg>>>

Then the master branch:

>>> import traceback; traceback.print_exception(None, SyntaxError("msg", ("f.py", 3, 10, "aaa\nbbb\nccc\n")), None)  File "f.py", line 3    aaabbbccc           ^SyntaxError: msg>>> raise SyntaxError("msg", ("f.py", 3, 10, "aaa\nbbb\nccc\n"))Traceback (most recent call last):  File "<stdin>", line 1, in <module>  File "f.py", line 3    ccc     ^SyntaxError: msg>>>

To me that looks like in both versions, the C formatter (invoked byraise) does the right thing, while traceback.py messes up the output.