Documentation

The current documentation of Python syntax (the later chapters of thelanguage reference) uses hand-maintainedproduction lists, like this:

A)

compound_stmt ::=  if_stmt                   | while_stmt                   | for_stmt                   | try_stmt                   | with_stmt                   | match_stmt                   | funcdef                   | classdef                   | async_with_stmt                   | async_for_stmt                   | async_funcdefsuite         ::=  stmt_list NEWLINE | NEWLINE INDENT statement+ DEDENTstatement     ::=  stmt_list NEWLINE | compound_stmtstmt_list     ::=  simple_stmt (";" simple_stmt)* [";"]

There is no mechanism to ensure that these are in sync with theactual grammar, and they inevitably do get out of sync.
See some of the“docs” issues mentioning “grammar”.

It's not easy to write an automatic tool to keep them in sync, because wedo want to elide some details -- the parser rules,unnecessary lookaheads, cuts, etc. But, it'spossible to write it, and we wrote aproof of concept, which will need to be rewritten, tuned, and reviewed. Before introducing it, I'd like to go through all the docs, correct the existing documentation, bring it closer to what a tool could generate, and discuss what theideal presentation would look like. That needs to be a manual process, and it will also need to touch theprose that's next to the grammar snippets.

As a first step, I propose an update to the tooling, which brings thepresentation a bit closer to thepython.gram syntax.

From theexisting ReST source, we can get this:

B)

compound_stmt: if_stmt               | while_stmt               | for_stmt               | try_stmt               | with_stmt               | match_stmt               | funcdef               | classdef               | async_with_stmt               | async_for_stmt               | async_funcdefsuite:         stmt_list NEWLINE | NEWLINE INDENT statement+ DEDENTstatement:     stmt_list NEWLINE | compound_stmtstmt_list:     simple_stmt (";" simple_stmt)* [";"]

Since Sphinx hard-codes theproductionlist formatting (the::= symbol and the aligning), we'll need to override theproductionlist directive to achieve this.

Then, by changing the ReST and using a different directive, we can get to something like:

C)

compound_stmt:    | if_stmt    | while_stmt    | for_stmt    | try_stmt    | with_stmt    | match_stmt    | funcdef    | classdef    | async_with_stmt    | async_for_stmt    | async_funcdefsuite:    | stmt_list NEWLINE | NEWLINE INDENT statement+ DEDENTstatement:    | stmt_list NEWLINE | compound_stmtstmt_list:    | simple_stmt (";" simple_stmt)* [";"]

I propose to go from A) to B) at once (by overridingproductionlist), and from B) to C) gradually, while also updating the content (including changing rule names to match the grammar, and adjusting/reorganizing nearby prose).
I think that the B) and C) styles are similar enough that mixing them in a single version of the docs should not be jarring.

By the way, one additional benefit of a custom directive is that we can add syntax highlighting. (Ideally, with support from the theme.) I think that making strings stand out makes the listings more readable:

Linked PRs

• gh-127833: Docs: Add agrammar-snippet directive & replaceproductionlist #127835
• [3.13] gh-127833: Docs: Add agrammar-snippet directive & replaceproductionlist (GH-127835) #129689
• [3.12] gh-127833: Docs: Add agrammar-snippet directive & replaceproductionlist (GH-127835) #129690
• gh-127833: Fix grammar snippet formatting for help() output #129692
• gh-127833: Useproductionlist nodes to implement thegrammar-snippet directive #130376
• gh-127833: Add links to token types to the lexical analysis intro #131468
• gh-127833: lexical analysis: Improve section on Names #131474
• gh-127833: lexical analysis: Add backticks to BOM example #132407
• [3.14] gh-127833: lexical analysis: Add backticks to BOM example (GH-132407) #133632
• [3.13] gh-127833: lexical analysis: Add backticks to BOM example (GH-132407) #133633
• [3.14] gh-127833: Add links to token types to the lexical analysis intro (GH-131468) #133652
• [3.14] gh-127833: lexical analysis: Improve section on Names (GH-131474) #134423
• gh-127833: Reword and expand the Notation section #134443

Other related PRs

• gh-130587: Add hand-written docs for non-OP tokens #130588
• gh-134026: Fix grammar description of for statement #134034