If I understand the reference manual correctly, these should probablybe rejected by the compiler:>>> async def f():...     def g(): pass...     async = 10... >>> async def f():...     def async():...         pass... >>> async def f(): async = 10... >>> async def f():...     def await(): pass... >>> And this should perhaps be accepted:>>> async def f():...     return lambda await: await  File "<stdin>", line 2    return lambda await: await                      ^SyntaxError: invalid syntaxThis, too:>>> async def f():...     async def g(): pass...     await z  File "<stdin>", line 3    await z          ^SyntaxError: invalid syntax

Sorry for not responding earlier, I'm on vacation and don't check email too often.While investigating what can be done in tokenizer.c to fix some of the bugs that Stefan pointed out, I discovered a simpler approach: instead of checking what kind of function we have on top of the stack, I now have a counter of nested async functions.  This aligns nicely with what Nick said on python-dev once: we essentially treat 'async def' as a future import, everything inside it parses differently.In other words, before this patch:   async def foo():       def bar():           async = 1was legal, now it's a syntax error, as 'async' will be parsed as 'ASYNC' token.This simplifies the implementation of tokenizer.c hacks, fixes all of the bugs that Stefan posted here, *and* enables one-line 'async' functions:   async def foo(): await bar() # legal with the patchIt will also help with migrating the code to Python 3.7 when async/await will become proper keywords.And, with this patch we can effectively remove the shortcomings section from thePEP 492 (https://www.python.org/dev/peps/pep-0492/#transition-period-shortcomings).Please review the attached patch, it would be great if we can commit it before 3.5beta4.

Haven't reviewed the patch, but this approach sounds great (in fact I hadassumed you were doing this already, and I was a bit surprised by some ofthe problems you encountered :-).

Good news :) I guess this means we can also remove the sentence I added at <https://docs.python.org/3.5/reference/compound_stmts.html#coroutine-function-definition>.

An updated patch is attached.  I had to implement a little bit more sophisticated tracking of one-line functions to fix parsing of things like    def foo():        async def f(): pass        async def f(): pass        async = 1I hope that test_coroutine.py now covers all possible legal and illegal async/await syntax.> Haven't reviewed the patch, but this approach sounds great (in fact I had> assumed you were doing this already, and I was a bit surprised by some of> the problems you encountered :-).Yes, I'm a bit surprised myself ;)  I guess one of the reasons why I tried to do more in tokenizer is that at the time of me hacking the tokenizer, compile.c wasn't quite ready (specifically, catching async for/async with/await outside of async functions).> Good news :) I guess this means we can also remove the sentence I added at [..]Right!

Patch & test cases look good to me.I'm so used to thinking of the tokenisation phase as a linear token streamthat it never occurred to me to just count the function nesting directly todetermine if the "async def" tokenisation rules are in effect - it's a verynice simplifying improvement.

New changeset9da080ecadb2 by Yury Selivanov in branch '3.5':Issue#24619: New approach for tokenizing async/await.https://hg.python.org/cpython/rev/9da080ecadb2New changeset987b72921a0c by Yury Selivanov in branch 'default':Merge 3.5 (Issue#24619)https://hg.python.org/cpython/rev/987b72921a0c

Thanks Nick!  I've committed the patch with a few more unittests and a couple of additional comments in tokenizer.(c|h).

New changesete4e01488afff by Yury Selivanov in branch '3.5':Issue#24619: More tests; fix nits in compiler.chttps://hg.python.org/cpython/rev/e4e01488afffNew changeset6f4e0c462daf by Yury Selivanov in branch 'default':Merge 3.5 (Issue#24619)https://hg.python.org/cpython/rev/6f4e0c462daf

This is a very nice solution!I'm just curious if the 'ctx' is still needed:  It looks likethe outermost "async def" dominates all further nested scopesw.r.t the tokenizer mode, no matter whether they're "def" or"async def" scopes.IOW, a single indent_level variable that follows all INDENTs/DEDENTsonce the outermost "async def" scope is entered might be sufficient.[This is in no way urgent, please do not feel obliged to respondduring your holiday.]

> I'm just curious if the 'ctx' is still needed:  It looks like> the outermost "async def" dominates all further nested scopes> w.r.t the tokenizer mode, no matter whether they're "def" or> "async def" scopes.This is a wonderful idea, that's the way it should be done inboth tokenizer.c and tokenize.py.Please see the new patch.  I love the simplicity of it, no more stacks or hard to follow code.

Updated patch attached (rebased; minor optimization).  I'll commit this patch in a few hours to make sure it lands in 3.5beta4.

New changesetd03f86e41066 by Yury Selivanov in branch '3.5':Issue#24619: Simplify async/await tokenization.https://hg.python.org/cpython/rev/d03f86e41066New changeset3f8048926690 by Yury Selivanov in branch 'default':Merge 3.5 (Issue#24619)https://hg.python.org/cpython/rev/3f8048926690