Movatterモバイル変換


[0]ホーム

URL:


homepage

Issue25555

This issue trackerhas been migrated toGitHub, and is currentlyread-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title:Fix parser and AST: fill lineno and col_offset when compiling AST from Python objects
Type:Stage:
Components:Versions:Python 3.6, Python 3.5
process
Status:closedResolution:fixed
Dependencies:Superseder:
Assigned To:Nosy List: benjamin.peterson, matrixise, python-dev, vstinner, yselivanov
Priority:normalKeywords:patch

Created on2015-11-05 09:42 byvstinner, last changed2022-04-11 14:58 byadmin. This issue is nowclosed.

Files
File nameUploadedDescriptionEdit
ast.patchvstinner,2015-11-05 09:42review
Messages (7)
msg254093 -(view)Author: STINNER Victor (vstinner)*(Python committer)Date: 2015-11-05 09:42
I wrote an AST optimizer which build AST from source code using ast.parse(), modify AST in place, and then compile the AST to bytecode.The problem is that the lineno and col_offset attributes of the "arg" AST node are uninitialized when AST is rebuild from Python objects: compile(ast_tree, ...). As a consequence, the compilation may fail because lineno or col_offset values are invalid.Attached patch enhancesParser/asdl_c.py to take "arg" attributes in account. It also updates the generated filesInclude/Python-ast.h andPython/Python-ast.c, and fix usage of the arg() function inPython/ast.c.It looks like only the "arg" AST node had attributes which were initialized (other AST nodes are handled with a different code path which already filled attributes).SeeParser/Python.asdl for the definition of the "arg" AST node.Note: Python 2.7 is not affected. In Python 2, function arguments are simple expressions, they don't have a dedicated type with lineno and col_offset attributes. "arg" was introduced in Python 3 with thePEP 3107 (function annotations).
msg254155 -(view)Author: Stéphane Wirtel (matrixise)*(Python committer)Date: 2015-11-06 01:07
Hi Victor,1. Result of the compilation -> success2. I have checked your patch, I think there is no problem.3. How can I test it? currently, everything is fine.For me, this patch is ok and working, but I need the point 3.
msg254202 -(view)Author: Roundup Robot (python-dev)(Python triager)Date: 2015-11-06 16:02
New changeset54d4290f0ec6 by Victor Stinner in branch 'default':Issue#25555: Fix parser and AST: fill lineno and col_offset of "arg" node whenhttps://hg.python.org/cpython/rev/54d4290f0ec6
msg254203 -(view)Author: STINNER Victor (vstinner)*(Python committer)Date: 2015-11-06 16:05
> 3. How can I test it? currently, everything is fine.Sorry, I don't know how to test it.To write my fix, I had to modify the compile() builtin function to be able to compile Python to AST, and then AST to Python. But without this fix I don't know how to test.I tried to revert the AST patch on my FAT Python to try to find which part of the code fails without the fix, but FAT Python doesn't compile anymore. I reverted the revert, and it still doesn't work!?For this bug, sorry, you have to trust me :-(I applied the fix to Python 3.6. I will wait a bit for buildbots before backporting to 3.5.
msg254204 -(view)Author: Stéphane Wirtel (matrixise)*(Python committer)Date: 2015-11-06 16:07
I trust you, I have reviewed the source code and it seems to be right.
msg254403 -(view)Author: Roundup Robot (python-dev)(Python triager)Date: 2015-11-09 20:56
New changeset2d8988f64cea by Victor Stinner in branch '3.5':Issue#25555: Fix parser and AST: fill lineno and col_offset of "arg" node whenhttps://hg.python.org/cpython/rev/2d8988f64cea
msg254409 -(view)Author: STINNER Victor (vstinner)*(Python committer)Date: 2015-11-09 21:14
I backported the fix to Python 3.5. Since I'm not convinced that the bug can be triggered without modifying CPython (the bug only impacts FAT Python which installs a hook on the AST compiler), I don't want to backport it to Python 2.7 to avoid the risk of regression.
History
DateUserActionArgs
2022-04-11 14:58:23adminsetgithub: 69741
2015-11-09 21:14:10vstinnersetstatus: open -> closed
resolution: fixed
messages: +msg254409
2015-11-09 20:56:15python-devsetmessages: +msg254403
2015-11-06 16:07:51matrixisesetmessages: +msg254204
2015-11-06 16:05:58vstinnersetmessages: +msg254203
2015-11-06 16:02:35python-devsetnosy: +python-dev
messages: +msg254202
2015-11-06 01:07:21matrixisesetmessages: +msg254155
2015-11-05 09:42:04vstinnercreate
Supported byThe Python Software Foundation,
Powered byRoundup
Copyright © 1990-2022,Python Software Foundation
Legal Statements

[8]ページ先頭

©2009-2026 Movatter.jp