Specification

This PEP proposes to change the default source encoding from ASCII toUTF-8. Support for alternative source encodings (PEP 263) continues toexist; an explicit encoding declaration takes precedence over thedefault.

A Bit of History

In Python 1, the source encoding was unspecified, except that thesource encoding had to be a superset of the system’s basic executioncharacter set (i.e. an ASCII superset, on most systems). The sourceencoding was only relevant for the lexis itself (bytes representingletters for keywords, identifiers, punctuation, line breaks, etc).The contents of a string literal was copied literally from the fileon source.

In Python 2.0, the source encoding changed to Latin-1 as a side effectof introducing Unicode. For Unicode string literals, the characterswere still copied literally from the source file, but widened on acharacter-by-character basis. As Unicode gives a fixed interpretationto code points, this algorithm effectively fixed a source encoding, atleast for files containing non-ASCII characters in Unicode literals.

PEP 263 identified the problem that you can use only those Unicodecharacters in a Unicode literal which are also in Latin-1, andintroduced a syntax for declaring the source encoding. If no sourceencoding was given, the default should be ASCII. For compatibilitywith Python 2.0 and 2.1, files were interpreted as Latin-1 for atransitional period. This transition ended with Python 2.5, whichgives an error if non-ASCII characters are encountered and no sourceencoding is declared.

Rationale

WithPEP 263, using arbitrary non-ASCII characters in a Python file ispossible, but tedious. One has to explicitly add an encodingdeclaration. Even though some editors (like IDLE and Emacs) supportthe declarations ofPEP 263, many editors still do not (and neverwill); users have to explicitly adjust the encoding which the editorassumes on a file-by-file basis.

When the default encoding is changed to UTF-8, adding non-ASCII textto Python files becomes easier and more portable: On some systems,editors will automatically choose UTF-8 when saving text (e.g. on Unixsystems where the locale uses UTF-8). On other systems, editors willguess the encoding when reading the file, and UTF-8 is easy toguess. Yet other editors support associating a default encoding with afile extension, allowing users to associate .py with UTF-8.

For Python 2, an important reason for using non-UTF-8 encodings wasthat byte string literals would be in the source encoding at run-time,allowing then to output them to a file or render them to the useras-is. With Python 3, all strings will be Unicode strings, so theoriginal encoding of the source will have no impact at run-time.

Implementation

The parser needs to be changed to accept bytes > 127 if no sourceencoding is specified; instead of giving an error, it needs to checkthat the bytes are well-formed UTF-8 (decoding is not necessary,as the parser converts all source code to UTF-8, anyway).

IDLE needs to be changed to use UTF-8 as the default encoding.

Copyright

This document has been placed in the public domain.