Note

This PEP has been moved to the Python dev guide[1].

Abstract

There’s more to changing Python’s grammar than editingGrammar/Grammar andPython/compile.c. This PEP aims to be achecklist of places that must also be fixed.

It is probably incomplete. If you see omissions, just add them ifyou can – you are not going to offend the author’s sense ofownership. Otherwise submit a bug or patch and assign it to mwh.

This PEP is not intended to be an instruction manual on Pythongrammar hacking, for several reasons.

Rationale

People are getting this wrong all the time; it took well over ayear before someone noticed[2] that adding the floor divisionoperator (//) broke theparser module.

Checklist

• Grammar/Grammar: OK, you’d probably worked this one out :)
• Parser/Python.asdl may need changes to match theGrammar. Runmake to regenerateInclude/Python-ast.h andPython/Python-ast.c.
• Python/ast.c will need changes to create the AST objectsinvolved with theGrammar change.Lib/compiler/ast.py willneed matching changes to the pure-python AST objects.
• Parser/pgen needs to be rerun to regenerateInclude/graminit.handPython/graminit.c. (make should handle this for you.)
• Python/symbtable.c: This handles the symbol collection passthat happens immediately before the compilation pass.
• Python/compile.c: You will need to create or modify thecompiler_* functions to generate opcodes for your productions.
• You may need to regenerateLib/symbol.py and/orLib/token.pyand/orLib/keyword.py.
• Theparser module. Add some of your new syntax totest_parser,bang onModules/parsermodule.c until it passes.
• Add some usage of your new syntax totest_grammar.py.
• Thecompiler package. A good test is to compile the standardlibrary and test suite with thecompiler package and then checkit runs. Note that this only needs to be done in Python 2.x.
• If you’ve gone so far as to change the token structure ofPython, then theLib/tokenizer.py library module will need tobe changed.
• Certain changes may require tweaks to the library modulepyclbr.
• Documentation must be written!
• After everything’s been checked in, you’re likely to see a newchange toPython/Python-ast.c. This is because this(generated) file contains the SVN version of the source fromwhich it was generated. There’s no way to avoid this; you justhave to submit this file separately.

References

Copyright

This document has been placed in the public domain.