Abstract

PEP 498 introduced Literal String Interpolation (or “f-strings”).The expression portions of those literals however are subject tocertain restrictions. This PEP proposes a formal grammar liftingthose restrictions, promoting “f-strings” to “f expressions” or f-literals.

This PEP expands upon the f-strings introduced byPEP 498,so this text requires familiarity withPEP 498.

PEP Withdrawal

This PEP has been withdrawn in favour ofPEP 701.PEP 701 addresses all important points of this PEP.

Terminology

This text will refer to the existing grammar as “f-strings”,and the proposed one as “f-literals”.

Furthermore, it will refer to the{}-delimited expressions inf-literals/f-strings as “expression portions” and the static string contentaround them as “string portions”.

Motivation

The current implementation of f-strings in CPython relies on the existingstring parsing machinery and a post processing of its tokens. This results inseveral restrictions to the possible expressions usable within f-strings:

1. It is impossible to use the quote character delimiting the f-stringwithin the expression portion:

   >>>f'Magic wand:{bag['wand']}'                             ^SyntaxError: invalid syntax

2. A previously considered way around it would lead to escape sequencesin executed code and is prohibited in f-strings:

   >>> f'Magic wand { bag[\'wand\'] } string'SyntaxError: f-string expression portion cannot include a backslash

3. Comments are forbidden even in multi-line f-strings:

   >>> f'''A complex trick: {... bag['bag']  # recursive bags!... }'''SyntaxError: f-string expression part cannot include '#'

4. Expression portions need to wrap':' and'!' in braces:

   >>>f'Useless use of lambdas:{lambdax: x*2}'SyntaxError: unexpected EOF while parsing

These limitations serve no purpose from a language user perspective andcan be lifted by giving f-literals a regular grammar without exceptionsand implementing it using dedicated parse code.

Rationale

The restrictions mentioned inMotivation are non-obvious and counter-intuitiveunless the user is familiar with the f-literals’ implementation details.

As mentioned, a previous version ofPEP 498 allowed escape sequencesanywhere in f-strings, including as ways to encode the braces delimitingthe expression portions and in their code. They would be expanded beforethe code is parsed, which would have had several important ramifications:

#. It would not be clear to human readers which portions are Expressionsand which are strings. Great material for an “obfuscated/underhandedPython challenge”#. Syntax highlighters are good in parsing nested grammar, but notin recognizing escape sequences. ECMAScript 2016 (JavaScript) allowsescape sequences in its identifiers[1] and the author knows of nosyntax highlighter able to correctly highlight code making use of this.

As a consequence, the expression portions would be harder to recognizewith and without the aid of syntax highlighting. With the new grammar,it is easy to extend syntax highlighters to correctly parseand display f-literals:

f'Magic wand:{bag['wand']:^10}'

Highlighting expression portions with possible escape sequences wouldmean to create a modified copy of all rules of the complete expressiongrammar, accounting for the possibility of escape sequences in key words,delimiters, and all other language syntax. One such duplication wouldyield one level of escaping depth and have to be repeated for a deeperescaping in a recursive f-literal. This is the case since no highlightingengine known to the author supports expanding escape sequences beforeapplying rules to a certain context. Nesting contexts however is astandard feature of all highlighting engines.

Familiarity also plays a role: Arbitrary nesting of expressionswithout expansion of escape sequences is available in every singleother language employing a string interpolation method that usesexpressions instead of just variable names.[2]

Specification

PEP 498 specified f-strings as the following, but places restrictions on it:

f' <text> { <expression> <optional !s, !r, or !a> <optional : format specifier> } <text> ... '

All restrictions mentioned in the PEP are lifted from f-literals,as explained below:

1. Expression portions may now contain strings delimited with the samekind of quote that is used to delimit the f-literal.
2. Backslashes may now appear within expressions just like anywhere elsein Python code. In case of strings nested within f-literals,escape sequences are expanded when the innermost string is evaluated.
3. Comments, using the'#' character, are possible only in multi-linef-literals, since comments are terminated by the end of the line(which makes closing a single-line f-literal impossible).
4. Expression portions may contain':' or'!' whereversyntactically valid. The first':' or'!' that is not partof an expression has to be followed a valid coercion or format specifier.

A remaining restriction not explicitly mentioned byPEP 498 is line breaksin expression portions. Since strings delimited by single' or"characters are expected to be single line, line breaks remain illegalin expression portions of single line strings.

Backwards Compatibility

f-literals are fully backwards compatible to f-strings,and expands the syntax considered legal.

Reference Implementation

TBD

References

Copyright

This document has been placed in the public domain.