token — Constants used with Python parse trees¶

Source code:Lib/token.py

This module provides constants which represent the numeric values of leaf nodesof the parse tree (terminal tokens). Refer to the fileGrammar/Tokensin the Python distribution for the definitions of the names in the context ofthe language grammar. The specific numeric values which the names map to maychange between Python versions.

The module also provides a mapping from numeric codes to names and somefunctions. The functions mirror definitions in the Python C header files.

Note that a token’s value may depend on tokenizer options. For example, a"+" token may be reported as eitherPLUS orOP, ora"match" token may be eitherNAME orSOFT_KEYWORD.

token.tok_name¶

Dictionary mapping the numeric values of the constants defined in this moduleback to name strings, allowing more human-readable representation of parse treesto be generated.

token.ISTERMINAL(x)¶

ReturnTrue for terminal token values.

token.ISNONTERMINAL(x)¶

ReturnTrue for non-terminal token values.

token.ISEOF(x)¶

ReturnTrue ifx is the marker indicating the end of input.

The token constants are:

token.NAME¶

Token value that indicates anidentifier.Note that keywords are also initially tokenized asNAME tokens.

token.NUMBER¶

Token value that indicates anumeric literal

token.STRING¶

Token value that indicates astring or byte literal,excludingformatted string literals.The token string is not interpreted:it includes the surrounding quotation marks and the prefix (if given);backslashes are included literally, without processing escape sequences.

token.OP¶

A generic token value that indicates anoperator ordelimiter.

CPython implementation detail: This value is only reported by thetokenize module.Internally, the tokenizer usesexact token types instead.

token.COMMENT¶

Token value used to indicate a comment.The parser ignoresCOMMENT tokens.

token.NEWLINE¶

Token value that indicates the end of alogical line.

token.NL¶

Token value used to indicate a non-terminating newline.NL tokens are generated when a logical line of code is continuedover multiple physical lines. The parser ignoresNL tokens.

token.INDENT¶

Token value used at the beginning of alogical lineto indicate the start of anindented block.

token.DEDENT¶

Token value used at the beginning of alogical lineto indicate the end of anindented block.

token.FSTRING_START¶

Token value used to indicate the beginning of anf-string literal.

CPython implementation detail: The token string includes the prefix and the opening quote(s), but noneof the contents of the literal.

token.FSTRING_MIDDLE¶

Token value used for literal text inside anf-string literal,including format specifications.

CPython implementation detail: Replacement fields (that is, the non-literal parts of f-strings) usethe same tokens as other expressions, and are delimited byLBRACE,RBRACE,EXCLAMATION andCOLONtokens.

token.FSTRING_END¶

Token value used to indicate the end of af-string.

CPython implementation detail: The token string contains the closing quote(s).

token.TSTRING_START¶

Token value used to indicate the beginning of a template string literal.

CPython implementation detail: The token string includes the prefix and the opening quote(s), but noneof the contents of the literal.

Added in version 3.14.

token.TSTRING_MIDDLE¶

Token value used for literal text inside a template string literalincluding format specifications.

CPython implementation detail: Replacement fields (that is, the non-literal parts of t-strings) usethe same tokens as other expressions, and are delimited byLBRACE,RBRACE,EXCLAMATION andCOLONtokens.

Added in version 3.14.

token.TSTRING_END¶

Token value used to indicate the end of a template string literal.

CPython implementation detail: The token string contains the closing quote(s).

Added in version 3.14.

token.ENDMARKER¶

Token value that indicates the end of input.Used intop-level grammar rules.

token.ENCODING¶

Token value that indicates the encoding used to decode the source bytesinto text. The first token returned bytokenize.tokenize() willalways be anENCODING token.

CPython implementation detail: This token type isn’t used by the C tokenizer but is needed forthetokenize module.

The following token types are not produced by thetokenize module,and are defined for special uses in the tokenizer or parser:

token.TYPE_IGNORE¶

Token value indicating that atype:ignore comment was recognized.Such tokens are produced instead of regularCOMMENT tokens onlywith thePyCF_TYPE_COMMENTS flag.

token.TYPE_COMMENT¶

Token value indicating that a type comment was recognized.Such tokens are produced instead of regularCOMMENT tokens onlywith thePyCF_TYPE_COMMENTS flag.

token.SOFT_KEYWORD¶

Token value indicating asoft keyword.

The tokenizer never produces this value.To check for a soft keyword, pass aNAME token’s string tokeyword.issoftkeyword().

token.ERRORTOKEN¶

Token value used to indicate wrong input.

Thetokenize module generally indicates errors byraising exceptions instead of emitting this token.It can also emit tokens such asOP orNAME with strings thatare later rejected by the parser.

The remaining tokens represent specificoperators anddelimiters.(Thetokenize module reports these asOP; seeexact_typein thetokenize documentation for details.)

The following non-token constants are provided:

token.N_TOKENS¶

The number of token types defined in this module.

token.EXACT_TOKEN_TYPES¶

A dictionary mapping the string representation of a token to its numeric code.

Added in version 3.8.