CurrentlyLib/token.py is generated fromInclude/token.h. This contradicts common practice when the C code is generated from the Python code (see for example opcode.py and sre_constants.py). In additional the table inParser/tokenizer.c should be manually supported matchingInclude/token.h.GeneratingInclude/token.h andParser/tokenizer.c fromLib/token.py would be simpler and more reliable.

I like the idea.

I can work on it

I already write a patch.

PR 1860 makes following files be generated from token.py:*Include/token.h*Parser/token.c. New file containing the array of token names _PyParser_TokenNames, and functions PyToken_OneChar(), PyToken_TwoChars(), PyToken_ThreeChars(), moved fromParser/tokenizer.c.*Doc/library/token-list.inc. New file containing the list of token.py constants, it is included inDoc/library/token.rst.New Makefile target regen-token regenerates these files.The dict EXACT_TOKEN_TYPES that maps operator strings to token names now is automatically generated and moved from tokenize.py to token.py. Tokens COMMENT, NL and ENCODING used only in tokenize.py now are added in token.py as inissue25324.

I think this covers all the changes from PR#1608. Looks a lot nicer too, building it every time from the make file. You may want to add to the docs that token.py is now the source of the tokens.

The regular expression tokenize.Funny also can be generated. Information is not enough for distinguish between Operator, Bracket and Special, but seems this isn't needed.Some token names can be generated fromGrammar/Grammar. But needed an additional mapping for relations between token strings and names ('+' <-> PLUS, etc).

AlternatePR 10370 generates all files from a single fileGrammar/Tokens using a single scriptTools/scripts/generate_token.py.In addition, the script doesn't write files when the content is not changed. Thus it can be used with read-only sources.

Could anybody please make a review? There are two alternate PRs:PR 1860 andPR 10370. The difference between them is that the former one usesLib/token.py as a source, and the latter one usesGrammar/Tokens as a source and generatesLib/token.py too.

If there are no objections I am going to mergePR 10370 in few days.

> If there are no objections I am going to mergePR 10370 in few days.LGTM. I guess thatPR 9343 should be closed oncePR 10370 is merged.

New changeset8ac658114dec4964479baecfbc439fceb40eaa79 by Serhiy Storchaka in branch 'master':bpo-30455: Generate all token related code and docs fromGrammar/Tokens. (GH-10370)https://github.com/python/cpython/commit/8ac658114dec4964479baecfbc439fceb40eaa79