Abstract

This document proposes to lift some of the restrictions originally formulated inPEP 498 and to provide a formalized grammar for f-strings that can beintegrated into the parser directly. The proposed syntactic formalization off-strings will have some small side-effects on how f-strings are parsed andinterpreted, allowing for a considerable number of advantages for end users andlibrary developers, while also dramatically reducing the maintenance cost ofthe code dedicated to parsing f-strings.

Motivation

When f-strings were originally introduced inPEP 498, the specification wasprovided without providing a formal grammar for f-strings. Additionally, thespecification contains several restrictions that are imposed so the parsing off-strings could be implemented into CPython without modifying the existinglexer. These limitations have been recognized previously and previous attemptshave been made to lift them inPEP 536, butnone of this work was ever implemented.Some of these limitations (collected originally byPEP 536) are:

1. It is impossible to use the quote character delimiting the f-stringwithin the expression portion:

   >>>f'Magic wand:{bag['wand']}'                             ^SyntaxError: invalid syntax

2. A previously considered way around it would lead to escape sequencesin executed code and is prohibited in f-strings:

   >>> f'Magic wand { bag[\'wand\'] } string'SyntaxError: f-string expression portion cannot include a backslash

3. Comments are forbidden even in multi-line f-strings:

   >>> f'''A complex trick: {... bag['bag']  # recursive bags!... }'''SyntaxError: f-string expression part cannot include '#'

4. Arbitrary nesting of expressions without expansion of escape sequences isavailable in many other languages that employ a string interpolationmethod that uses expressions instead of just variable names. Some examples:

   # Ruby"#{ "#{1+2}" }"# JavaScript`${`${1+2}`}`# Swift"\("\(1+2)")"# C#$"{$"{1+2}"}"

These limitations serve no purpose from a language user perspective andcan be lifted by giving f-string literals a regular grammar without exceptionsand implementing it using dedicated parse code.

The other issue that f-strings have is that the current implementation inCPython relies on tokenising f-strings asSTRING tokens and a post processing ofthese tokens. This has the following problems:

1. It adds a considerable maintenance cost to the CPython parser. This is becausethe parsing code needs to be written by hand, which has historically led to aconsiderable number of inconsistencies and bugs. Writing and maintaining parsingcode by hand in C has always been considered error prone and dangerous as it needsto deal with a lot of manual memory management over the original lexer buffers.
2. The f-string parsing code is not able to use the new improved error message mechanismsthat the new PEG parser, originally introduced inPEP 617, has allowed. Theimprovements that these error messages brought has been greatly celebrated butunfortunately f-strings cannot benefit from them because they are parsed in aseparate piece of the parsing machinery. This is especially unfortunate, sincethere are several syntactical features of f-strings that can be confusing dueto the different implicit tokenization that happens inside the expressionpart (for instancef"{y:=3}" is not an assignment expression).
3. Other Python implementations have no way to know if they have implementedf-strings correctly because contrary to other language features, they are notpart of theofficial Python grammar.This is important because several prominentalternative implementations are using CPython’s PEG parser,such as PyPy,and/or are basing their grammars on the official PEG grammar. Thefact that f-strings use a separate parser prevents these alternative implementationsfrom leveraging the official grammar and benefiting from improvements in error messages derivedfrom the grammar.

A version of this proposal was originallydiscussed on Python-Dev andpresented at the Python Language Summit 2022 where it was enthusiasticallyreceived.

Rationale

By building on top of the new Python PEG Parser (PEP 617), this PEP proposesto redefine “f-strings”, especially emphasizing the clear separation of thestring component and the expression (or replacement,{...}) component.PEP 498summarizes the syntactical part of “f-strings” as the following:

In Python source code, an f-string is a literal string, prefixed with ‘f’, whichcontains expressions inside braces. The expressions are replaced with their values.

However,PEP 498 also contained a formal list of exclusions on whatcan or cannot be contained inside the expression component (primarily due to thelimitations of the existing parser). By clearly establishing the formal grammar, wenow also have the ability to define the expression component of an f-string as truly “anyapplicable Python expression” (in that particular context) without being boundby the limitations imposed by the details of our implementation.

The formalization effort and the premise above also has a significant benefit forPython programmers due to its ability to simplify and eliminate the obscurelimitations. This reduces the mental burden and the cognitive complexity off-string literals (as well as the Python language in general).

1. The expression component can include any string literal that a normal Python expressioncan include. This opens up the possibility of nesting string literals (formatted ornot) inside the expression component of an f-string with the same quote type (and length):

   >>>f"These are the things:{", ".join(things)}">>>f"{source.removesuffix(".py")}.c: $(srcdir)/{source}">>>f"{f"{f"infinite"}"}"+" "+f"{f"nesting!!!"}"

   This “feature” is not universally agreed to be desirable, and some users find this unreadable.For a discussion on the different views on this, see theconsiderations regarding quote reuse section.

2. Another issue that has felt unintuitive to most is the lack of support for backslasheswithin the expression component of an f-string. One example that keeps coming up is includinga newline character in the expression part for joining containers. For example:

   >>>a=["hello","world"]>>>f"{'\n'.join(a)}"File "<stdin>", line 1    f"{'\n'.join(a)}"                    ^SyntaxError: f-string expression part cannot include a backslash

   A common work-around for this was to either assign the newline to an intermediate variable orpre-create the whole string prior to creating the f-string:

   >>>a=["hello","world"]>>>joined='\n'.join(a)>>>f"{joined}"'hello\nworld'

   It only feels natural to allow backslashes in the expression part now that the new PEG parsercan easily support it.

   >>>a=["hello","world"]>>>f"{'\n'.join(a)}"'hello\nworld'

3. Before the changes proposed in this document, there was no explicit limit inhow f-strings can be nested, but the fact that string quotes cannot be reusedinside the expression component of f-strings made it impossible to nestf-strings arbitrarily. In fact, this is the most nested-fstring that can bewritten:

   >>>f"""{f'''{f'{f"{1+1}"}'}'''}"""'2'

   As this PEP allows placingany valid Python expression inside theexpression component of the f-strings, it is now possible to reuse quotes andtherefore is possible to nest f-strings arbitrarily:

   >>>f"{f"{f"{f"{f"{f"{1+1}"}"}"}"}"}"'2'

   Although this is just a consequence of allowing arbitrary expressions, theauthors of this PEP do not believe that this is a fundamental benefit and wehave decided that the language specification will not explicitly mandate thatthis nesting can be arbitrary. This is because allowing arbitrarily-deepnesting imposes a lot of extra complexity to the lexer implementation(particularly as lexer/parser pipelines need to allow “untokenizing” tosupport the ‘f-string debugging expressions’ and this is especially taxing whenarbitrary nesting is allowed). Implementations are therefore free to impose alimit on the nesting depth if they need to. Note that this is not an uncommonsituation, as the CPython implementation already imposes several limits allover the place, including a limit on the nesting depth of parentheses andbrackets, a limit on the nesting of the blocks, a limit in the number ofbranches inif statements, a limit on the number of expressions instar-unpacking, etc.

Specification

The formal proposed PEG grammar specification for f-strings is (seePEP 617for details on the syntax):

fstring|FSTRING_STARTfstring_middle*FSTRING_ENDfstring_middle|fstring_replacement_field|FSTRING_MIDDLEfstring_replacement_field|'{'(yield_expr|star_expressions)"="?[ "!" NAME][ ':' fstring_format_spec*]'}'fstring_format_spec:|FSTRING_MIDDLE|fstring_replacement_field

The new tokens (FSTRING_START,FSTRING_MIDDLE,FSTRING_END) are definedlater in this document.

This PEP leaves up to the implementation the level of f-string nesting allowed(f-strings within the expression parts of other f-strings) butspecifies alower bound of 5 levels of nesting. This is to ensure that users can have areasonable expectation of being able to nest f-strings with “reasonable” depth.This PEP implies that limiting nesting isnot part of the languagespecification but also the language specificationdoesn’t mandate arbitrarynesting.

Similarly, this PEP leaves up to the implementation the level of expression nestingin format specifiers butspecifies a lower bound of 2 levels of nesting. This meansthat the following should always be valid:

f"{'':*^{1:{1}}}"

but the following can be valid or not depending on the implementation:

f"{'':*^{1:{1:{1}}}}"

The new grammar will preserve the Abstract Syntax Tree (AST) of the currentimplementation. This means that no semantic changes will be introduced by thisPEP on existing code that uses f-strings.

>>>a=1>>>f"{1+1=}"'1+1=2'

f'some words{a+b:.3f} more words{c+d=} final words'

FSTRING_START-"f'"FSTRING_MIDDLE-'some words 'LBRACE-'{'NAME-'a'PLUS-'+'NAME-'b'OP-':'FSTRING_MIDDLE-'.3f'RBRACE-'}'FSTRING_MIDDLE-' more words 'LBRACE-'{'NAME-'c'PLUS-'+'NAME-'d'OP-'='RBRACE-'}'FSTRING_MIDDLE-' final words'FSTRING_END-"'"

FSTRING_START-'f"""'FSTRING_MIDDLE-'some words'FSTRING_END-'"""'

>>>f" something{my_dict["key"]} something else "

Backwards Compatibility

This PEP does not introduce any backwards incompatible syntactic or semantic changesto the Python language. However, thetokenize module (a quasi-public part of the standardlibrary) will need to be updated to support the new f-string tokens (to allow tool authorsto correctly tokenize f-strings). Seechanges to the tokenize module for more details regardinghow the public API oftokenize will be affected.

How to Teach This

As the concept of f-strings is already ubiquitous in the Python community, there isno fundamental need for users to learn anything new. However, as the formalized grammarallows some new possibilities, it is important that the formal grammar is added to thedocumentation and explained in detail, explicitly mentioning what constructs are possiblesince this PEP is aiming to avoid confusion.

It is also beneficial to provide users with a simple framework for understanding what canbe placed inside an f-string expression. In this case the authors think that this work willmake it even simpler to explain this aspect of the language, since it can be summarized as:

You can place any valid Python expression inside an f-string expression.

With the changes in this PEP, there is no need to clarify that string quotes arelimited to be different from the quotes of the enclosing string, because this isnow allowed: as an arbitrary Python string can contain any possible choice ofquotes, so can any f-string expression. Additionally there is no need to clarifythat certain things are not allowed in the expression part because ofimplementation restrictions such as comments, new line characters orbackslashes.

The only “surprising” difference is that as f-strings allow specifying aformat, expressions that allow a: character at the top level still need to beenclosed in parenthesis. This is not new to this work, but it is important toemphasize that this restriction is still in place. This allows for an easiermodification of the summary:

You can place any valid Python expression insidean f-string expression, and everything after a: character at the top level willbe identified as a format specification.

Reference Implementation

A reference implementation can be found in theimplementation fork.

Rejected Ideas

1. Although we think the readability arguments that have been raised againstallowing quote reuse in f-string expressions are valid and very important,we have decided to propose not rejecting quote reuse in f-strings at the parserlevel. The reason is that one of the cornerstones of this PEP is to reduce thecomplexity and maintenance of parsing f-strings in CPython and this will notonly work against that goal, but it may even make the implementation even morecomplex than the current one. We believe that forbidding quote reuse should bedone in linters and code style tools and not in the parser, the same way otherconfusing or hard-to-read constructs in the language are handled today.
2. We have decided not to lift the restriction that some expression portionsneed to wrap':' and'!' in parentheses at the top level, e.g.:

   >>>f'Useless use of lambdas:{lambdax: x*2}'SyntaxError: unexpected EOF while parsing

   The reason is that this will introduce a considerable amount ofcomplexity for no real benefit. This is due to the fact that the: characternormally separates the f-string format specification. This format specificationis currently tokenized as a string. As the tokenizer MUST tokenize what’s on theright of the: as either a string or a stream of tokens, this won’t allow theparser to differentiate between the different semantics as that would require thetokenizer to backtrack and produce a different set of tokens (this is, first tryas a stream of tokens, and if it fails, try as a string for a format specifier).

   As there is no fundamental advantage in being able to allow lambdas and similarexpressions at the top level, we have decided to keep the restriction that these mustbe parenthesized if needed:

   >>>f'Useless use of lambdas:{(lambdax:x*2)}'

3. We have decided to disallow (for the time being) using escaped braces (\{ and\})in addition to the{{ and}} syntax. Although the authors of the PEP believe thatallowing escaped braces is a good idea, we have decided to not include it in this PEP, as it is not strictlynecessary for the formalization of f-strings proposed here, and it can beadded independently in a regular CPython issue.

Open Issues

None yet

Copyright

This document is placed in the public domain or under theCC0-1.0-Universal license, whichever is more permissive.