Given an input module such asclass klass(object):    """multi line comment    continued on this line    """    """single line comment"""    """    Another multi    line    comment"""and implementing a custom ast.NodeVisitor such asimport asclass CustomVisitor(ast.NodeVisitor):    def visit_ClassDef(self, node):        for childNode in node.body:            self.visit(childNode)    def visit_Expr(self, node):        print(node.col_offset)        print(node.value.col_offset)and feeding it the compiled ast from the module abovef = open('./module.py')source = f.read()node = ast.parse(source, mode = 'exec')visitor = CustomVisitor()visitor.visit(node)should yield -1/-1 for the docstring that is the firstchild node expression of the classdef body.it will, however, yield the correct col_offset of 4/4 forthe single line docstring following the first one.the multi line docstring following that will againyield a -1/-1 col_offset.It believe that this behaviour is not correct and insteadthe col_offset should be 4 for both the expression nodeand its str value.

Please note that, regardless of the indent level, the col_offset for multi line str expressions will always be -1.

In addition, the reported lineno will be set to the last line of the multi line string instead of the first line where parsing the parse began parsing the string.

Please see the attached patch that will resolve the issue. It also includes a test case in test_ast.py.What the patch does is as follows:- tok_state is extended by two fields, namely first_lineno  and multi_line_start- first_lineno will be set by tok_get as soon as the beginning  of a STRING is detected and it will be set to the current line  tok->lineno.- multi_line_start is the beginning of the first line of a string- in parsetok we now distinguish between STRING nodes and other  nodes. in case of STRING nodes, we will use the values of the  above fields for determining the actual lineno and the col_offset,  otherwise tok->col_offset and tok->lineno will be used when  creating the token.The included test case ensures that the col_offset and lineno ofmulti line strings is calculated correctly.

I have created a patch for Python 2.7.3 that fixes the issue for that release, too.

If this is really an 'enhancement', it will only go in 3.4. If it is a bug/behavior issue, then it should be marked as such and 2.7,3.2,3.3 selected. I have not read the doc and messages well enough to know, so I leave that to you and Benjamin.The patch includes a test. It needs a patch toMisc/ACKS to add Carsten Klein between Reid Kleckner and Bastian Kleineidam

I left comments on Rietveld a few days ago.

Any updates on this? I'm running into this as well (still a problem in 3.4)```$ python3.4Python 3.4.2 (default, Oct 11 2014, 17:59:27) [GCC 4.4.3] on linuxType "help", "copyright", "credits" or "license" for more information.>>> import ast>>> ast.parse("""'''foo\n'''""").body[0].value.col_offset-1```

What's the status on this? Anything preventing it getting fixed? Still the same in 3.6.1:>>> import ast>>> ast.parse("""'''foo\n'''""").body[0].value.col_offset-1

pypy seems to have this right (though I don't know enough about their internals to know if cpython can benefit from their patch)$ venvpypy/bin/pythonPython 2.7.10 (3260adbeba4a, Apr 19 2016, 17:42:20)[PyPy 5.1.0 with GCC 4.8.4] on linux2Type "help", "copyright", "credits" or "license" for more information.>>>> import ast, astpretty>>>> astpretty.pprint(ast.parse('"""\n"""'))Module(    body=[        Expr(            lineno=1,            col_offset=0,            value=Str(lineno=1, col_offset=0, s='\n'),        ),    ],)

Still a problem in 3.7:$ python3.7Python 3.7.0b2 (default, Feb 28 2018, 06:59:18) [GCC 5.4.0 20160609] on linuxType "help", "copyright", "credits" or "license" for more information.>>> import ast>>> ast.parse("""x = '''foo\n'''""").body[-1].value<_ast.Str object at 0x7fcde6898358>>>> ast.parse("""x = '''foo\n'''""").body[-1].value.col_offset-1

Should we backport this to 3.7?AST changes including bugfix affects existing software unexpectedly.

New changeset995d9b92979768125ced4da3a56f755bcdf80f6e by INADA Naoki (Anthony Sottile) in branch 'master':bpo-16806: Fix `lineno` and `col_offset` for multi-line string tokens (GH-10021)https://github.com/python/cpython/commit/995d9b92979768125ced4da3a56f755bcdf80f6e

I agree -- probably safer to not backport to 3.7 in case someone is relying on this behaviour.

commit995d9b92979768125ced4da3a56f755bcdf80f6e introduced a regression:bpo-37603: parsetok(): Assertion `(intptr_t)(int)(a - line_start) == (a - line_start)' failed, when running get-pip.py.

> commit995d9b92979768125ced4da3a56f755bcdf80f6e introduced a regression:bpo-37603: parsetok(): Assertion `(intptr_t)(int)(a - line_start) == (a - line_start)' failed, when running get-pip.py.Fixed inhttps://bugs.python.org/issue39209