Tokenizing Input¶

The primary entry point is agenerator:

tokenize.tokenize(readline)¶

Thetokenize() generator requires one argument,readline, whichmust be a callable object which provides the same interface as theio.IOBase.readline() method of file objects. Each call to thefunction should return one line of input as bytes.

The generator produces 5-tuples with these members: the token type; thetoken string; a 2-tuple(srow,scol) of ints specifying the row andcolumn where the token begins in the source; a 2-tuple(erow,ecol) ofints specifying the row and column where the token ends in the source; andthe line on which the token was found. The line passed (the last tuple item)is thephysical line. The 5 tuple is returned as anamed tuplewith the field names:typestringstartendline.

The returnednamed tuple has an additional property namedexact_type that contains the exact operator type forOP tokens. For all other token typesexact_typeequals the named tupletype field.

Changed in version 3.1:Added support for named tuples.

Changed in version 3.3:Added support forexact_type.

tokenize() determines the source encoding of the file by looking for aUTF-8 BOM or encoding cookie, according toPEP 263.

tokenize.generate_tokens(readline)¶

Tokenize a source reading unicode strings instead of bytes.

Liketokenize(), thereadline argument is a callable returninga single line of input. However,generate_tokens() expectsreadlineto return a str object rather than bytes.

The result is an iterator yielding named tuples, exactly liketokenize(). It does not yield anENCODING token.

All constants from thetoken module are also exported fromtokenize.

Another function is provided to reverse the tokenization process. This isuseful for creating tools that tokenize a script, modify the token stream, andwrite back the modified script.

tokenize.untokenize(iterable)¶

Converts tokens back into Python source code. Theiterable must returnsequences with at least two elements, the token type and the token string.Any additional sequence elements are ignored.

The result is guaranteed to tokenize back to match the input so that theconversion is lossless and round-trips are assured. The guarantee appliesonly to the token type and token string as the spacing between tokens(column positions) may change.

It returns bytes, encoded using theENCODING token, whichis the first token sequence output bytokenize(). If there is noencoding token in the input, it returns a str instead.

tokenize() needs to detect the encoding of source files it tokenizes. Thefunction it uses to do this is available:

tokenize.detect_encoding(readline)¶

Thedetect_encoding() function is used to detect the encoding thatshould be used to decode a Python source file. It requires one argument,readline, in the same way as thetokenize() generator.

It will call readline a maximum of twice, and return the encoding used(as a string) and a list of any lines (not decoded from bytes) it has readin.

It detects the encoding from the presence of a UTF-8 BOM or an encodingcookie as specified inPEP 263. If both a BOM and a cookie are present,but disagree, aSyntaxError will be raised. Note that if the BOM is found,'utf-8-sig' will be returned as an encoding.

If no encoding is specified, then the default of'utf-8' will bereturned.

Useopen() to open Python source files: it usesdetect_encoding() to detect the file encoding.

tokenize.open(filename)¶

Open a file in read only mode using the encoding detected bydetect_encoding().

Added in version 3.2.

exceptiontokenize.TokenError¶

Raised when either a docstring or expression that may be split over severallines is not completed anywhere in the file, for example:

"""Beginning ofdocstring

or:

[1,2,3

Command-Line Usage¶

Thetokenize module can be executed as a script from the command line.It is as simple as:

python-mtokenize[-e][filename.py]

The following options are accepted:

-h,--help¶

show this help message and exit

-e,--exact¶

display token names using the exact type

Iffilename.py is specified its contents are tokenized to stdout.Otherwise, tokenization is performed on stdin.

Examples¶

Example of a script rewriter that transforms float literals into Decimalobjects:

fromtokenizeimporttokenize,untokenize,NUMBER,STRING,NAME,OPfromioimportBytesIOdefdecistmt(s):"""Substitute Decimals for floats in a string of statements.    >>> from decimal import Decimal    >>> s = 'print(+21.3e-5*-.1234/81.7)'    >>> decistmt(s)    "print (+Decimal ('21.3e-5')*-Decimal ('.1234')/Decimal ('81.7'))"    The format of the exponent is inherited from the platform C library.    Known cases are "e-007" (Windows) and "e-07" (not Windows).  Since    we're only showing 12 digits, and the 13th isn't close to 5, the    rest of the output should be platform-independent.    >>> exec(s)  #doctest: +ELLIPSIS    -3.21716034272e-0...7    Output from calculations with Decimal should be identical across all    platforms.    >>> exec(decistmt(s))    -3.217160342717258261933904529E-7    """result=[]g=tokenize(BytesIO(s.encode('utf-8')).readline)# tokenize the stringfortoknum,tokval,_,_,_ing:iftoknum==NUMBERand'.'intokval:# replace NUMBER tokensresult.extend([(NAME,'Decimal'),(OP,'('),(STRING,repr(tokval)),(OP,')')])else:result.append((toknum,tokval))returnuntokenize(result).decode('utf-8')

Example of tokenizing from the command line. The script:

defsay_hello():print("Hello, World!")say_hello()

will be tokenized to the following output where the first column is the rangeof the line/column coordinates where the token is found, the second column isthe name of the token, and the final column is the value of the token (if any)

$python-mtokenizehello.py0,0-0,0:            ENCODING       'utf-8'1,0-1,3:            NAME           'def'1,4-1,13:           NAME           'say_hello'1,13-1,14:          OP             '('1,14-1,15:          OP             ')'1,15-1,16:          OP             ':'1,16-1,17:          NEWLINE        '\n'2,0-2,4:            INDENT         '    '2,4-2,9:            NAME           'print'2,9-2,10:           OP             '('2,10-2,25:          STRING         '"Hello, World!"'2,25-2,26:          OP             ')'2,26-2,27:          NEWLINE        '\n'3,0-3,1:            NL             '\n'4,0-4,0:            DEDENT         ''4,0-4,9:            NAME           'say_hello'4,9-4,10:           OP             '('4,10-4,11:          OP             ')'4,11-4,12:          NEWLINE        '\n'5,0-5,0:            ENDMARKER      ''

The exact token type names can be displayed using the-e option:

$python-mtokenize-ehello.py0,0-0,0:            ENCODING       'utf-8'1,0-1,3:            NAME           'def'1,4-1,13:           NAME           'say_hello'1,13-1,14:          LPAR           '('1,14-1,15:          RPAR           ')'1,15-1,16:          COLON          ':'1,16-1,17:          NEWLINE        '\n'2,0-2,4:            INDENT         '    '2,4-2,9:            NAME           'print'2,9-2,10:           LPAR           '('2,10-2,25:          STRING         '"Hello, World!"'2,25-2,26:          RPAR           ')'2,26-2,27:          NEWLINE        '\n'3,0-3,1:            NL             '\n'4,0-4,0:            DEDENT         ''4,0-4,9:            NAME           'say_hello'4,9-4,10:           LPAR           '('4,10-4,11:          RPAR           ')'4,11-4,12:          NEWLINE        '\n'5,0-5,0:            ENDMARKER      ''

Example of tokenizing a file programmatically, reading unicodestrings instead of bytes withgenerate_tokens():

importtokenizewithtokenize.open('hello.py')asf:tokens=tokenize.generate_tokens(f.readline)fortokenintokens:print(token)

Or reading bytes directly withtokenize():

importtokenizewithopen('hello.py','rb')asf:tokens=tokenize.tokenize(f.readline)fortokenintokens:print(token)