Test file (test.py):print(set(x for x in range(2),))Python runs it nicely:% python2 test.py set([0, 1])% python3 test.py{0, 1}2to3 parser (on both Python 2.7.11 and 3.5.2) chokes on it though:% /usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/bin/2to3 test.py RefactoringTool: Skipping optional fixer: bufferRefactoringTool: Skipping optional fixer: idiomsRefactoringTool: Skipping optional fixer: set_literalRefactoringTool: Skipping optional fixer: ws_commaRefactoringTool: Can't parse test.py: ParseError: bad input: type=8, value=u')', context=('', (1, 30))RefactoringTool: No files need to be modified.RefactoringTool: There was 1 error:RefactoringTool: Can't parse test.py: ParseError: bad input: type=8, value=u')', context=('', (1, 30))% /usr/local/Cellar/python3/3.5.2/Frameworks/Python.framework/Versions/3.5/bin/2to3 test.py RefactoringTool: Skipping optional fixer: bufferRefactoringTool: Skipping optional fixer: idiomsRefactoringTool: Skipping optional fixer: set_literalRefactoringTool: Skipping optional fixer: ws_commaRefactoringTool: Can't parse test.py: ParseError: bad input: type=8, value=')', context=('', (1, 30))RefactoringTool: No files need to be modified.RefactoringTool: There was 1 error:RefactoringTool: Can't parse test.py: ParseError: bad input: type=8, value=')', context=('', (1, 30))For reference:https://github.com/smarkets/flake8-strict/issues/9 (project using lib2to3 parser)

Adding benjamin.peterson as he is listed as the expert for 2to3.

set(x for x in range(2),) can be interpreted as set(x for x in (range(2),)).Wouldn't be better to forbid such ambiguous syntax? The trailing comma in argument list is supported because it helps to add new arguments (or temporary comment out arguments).    foo(x,        y,        #z,       )But set(x for x in range(2),) is not syntactically valid if add an argument after the comma. Parenthesis around a generator expression can be omitted only if it is the only argument in a function call. I think that it would be better to forbid a trailing comma in this case.

By "forbid" do you mean "forbid in Python" (as in change Python syntax)? I like the idea but that seems like a more serious change and 2to3 arguably needs to handle code targeting older Python versions anyway.

New changesetaf810b35b494ef1d255d4bf340b92a9dad446995 by Benjamin Peterson (Jakub Stasiak) in branch 'master':closesbpo-27494: Fix 2to3 handling of trailing comma after a generator expression (#3771)https://github.com/python/cpython/commit/af810b35b494ef1d255d4bf340b92a9dad446995

Actually this syntax isn't allowed by the Python language specification. Seeissue32012 for fixing the Python parser.

Since this syntax never was valid according to the Python language specification, and it causes SyntaxError in 3.7 (seeissue32012), I think that this change should be reverted.

New changeset4b8a7f51da224d1a0ad8159935f78ba4e6e16037 by Serhiy Storchaka in branch 'master': Revert "closesbpo-27494: Fix 2to3 handling of trailing comma after a generator expression (GH-3771)" (#8241)https://github.com/python/cpython/commit/4b8a7f51da224d1a0ad8159935f78ba4e6e16037

New changeset9ecbe3321f7bb3726017a053e583ca507d4453fc by Miss Islington (bot) in branch '3.7':Revert "closesbpo-27494: Fix 2to3 handling of trailing comma after a generator expression (GH-3771)" (GH-8241)https://github.com/python/cpython/commit/9ecbe3321f7bb3726017a053e583ca507d4453fc

Apologies for only responding now, I've not received any notifications after my original pull request had been merged. I only learned about the change being reverted fromhttps://github.com/python/cpython/pull/8580, so let me leave my two cents here:I don't think the syntax not being valid (formally - since forever, practically - since 3.7) is good enough reason to make (lib)2to3 reject it. 2to3 is supposed to handle old syntax, isn't it? I'd argue that since it is (or was) possible to use this syntax in Python 2.x it should be handled gracefully.

Sorry, I missed that you didn't receive a notification about creating of the reverting PR. I should announce this explicitly.2to3 handles the old syntax, but a comma after a generator expression was not a valid old syntax. With your patch it accepted this syntax and produced invalid Python program.$ ./python -m lib2to3 -w t9.pyRefactoringTool: Skipping optional fixer: bufferRefactoringTool: Skipping optional fixer: idiomsRefactoringTool: Skipping optional fixer: set_literalRefactoringTool: Skipping optional fixer: ws_commaRefactoringTool: Refactored t9.py--- t9.py       (original)+++ t9.py       (refactored)@@ -1 +1 @@-print(set(x for x in range(2),))+print((set(x for x in list(range(2)),)))RefactoringTool: Files that were modified:RefactoringTool: t9.py$ ./python t9.py   File "t9.py", line 1    print((set(x for x in list(range(2)),)))              ^SyntaxError: Generator expression must be parenthesized

I appreciate the example, but I'd claim that's a "missing fixer" issue, not a "parser accepts too much" issue. Considering the syntax wasn't ambiguous (I think) and had been accepted before 3.7 I'll remain not totally convinced here.

This is one of those issues where it's clear that the parser and 2to3 should be separate.