Using the default encoding is a common mistake

Developers using macOS or Linux may forget that the default encodingis not always UTF-8.

For example, usinglong_description=open("README.md").read() insetup.py is a common mistake. Many Windows users cannot installsuch packages if there is at least one non-ASCII character(e.g. emoji, author names, copyright symbols, and the like)in their UTF-8-encodedREADME.md file.

Of the 4000 most downloaded packages from PyPI, 489 use non-ASCIIcharacters in their README, and 82 fail to install from source onnon-UTF-8 locales due to not specifying an encoding for a non-ASCIIfile.[1]

Another example islogging.basicConfig(filename="log.txt").Some users might expect it to use UTF-8 by default, but the localeencoding is actually what is used.[2]

Even Python experts may assume that the default encoding is UTF-8.This creates bugs that only happen on Windows; see[3],[4],[5],and[6] for example.

Emitting a warning when theencoding argument is omitted will helpfind such mistakes.

Explicit way to use locale-specific encoding

open(filename) isn’t explicit about which encoding is expected:

• If ASCII is assumed, this isn’t a bug, but may result in decreasedperformance on Windows, particularly with non-Latin-1 locale encodings
• If UTF-8 is assumed, this may be a bug or a platform-specific script
• If the locale encoding is assumed, the behavior is as expected(but could change if future versions of Python modify the default)

From this point of view,open(filename) is not readable code.

encoding=locale.getpreferredencoding(False) can be used tospecify the locale encoding explicitly, but it is too long and easyto misuse (e.g. one can forget to passFalse as its argument).

This PEP provides an explicit way to specify the locale encoding.

Prepare to change the default encoding to UTF-8

Since UTF-8 has become the de-facto standard text encoding,we might default to it for opening files in the future.

However, such a change will affect many applications and libraries.If we start emittingDeprecationWarning everywhere theencodingargument is omitted, it will be too noisy and painful.

Although this PEP doesn’t propose changing the default encoding,it will help enable that change by:

• Reducing the number of omittedencoding arguments in librariesbefore we start emitting aDeprecationWarning by default.
• Allowing users to passencoding="locale" to suppressthe current warning and anyDeprecationWarning added in the future,as well as retaining consistent behavior if later Python versionschange the default, ensuring support for any Python version >=3.10.

EncodingWarning

Add a newEncodingWarning warning class as a subclass ofWarning. It is emitted when theencoding argument is omitted andthe default locale-specific encoding is used.

Options to enable the warning

The-Xwarn_default_encoding option and thePYTHONWARNDEFAULTENCODING environment variable are added. Theyare used to enableEncodingWarning.

sys.flags.warn_default_encoding is also added. The flag is true whenEncodingWarning is enabled.

When the flag is set,io.TextIOWrapper(),open() and othermodules using them will emitEncodingWarning when theencodingargument is omitted.

SinceEncodingWarning is a subclass ofWarning, they areshown by default (if thewarn_default_encoding flag is set), unlikeDeprecationWarning.

encoding="locale"

io.TextIOWrapper will accept"locale" as a valid argument toencoding. It has the same meaning as the currentencoding=None,except thatio.TextIOWrapper doesn’t emitEncodingWarning whenencoding="locale" is specified.

io.text_encoding()

io.text_encoding() is a helper for functions with anencoding=None parameter that pass it toio.TextIOWrapper() oropen().

A pure Python implementation will look like this:

deftext_encoding(encoding,stacklevel=1):"""A helper function to choose the text encoding.    When *encoding* is not None, just return it.    Otherwise, return the default text encoding (i.e. "locale").    This function emits an EncodingWarning if *encoding* is None and    sys.flags.warn_default_encoding is true.    This function can be used in APIs with an encoding=None parameter    that pass it to TextIOWrapper or open.    However, please consider using encoding="utf-8" for new APIs.    """ifencodingisNone:ifsys.flags.warn_default_encoding:importwarningswarnings.warn("'encoding' argument not specified.",EncodingWarning,stacklevel+2)encoding="locale"returnencoding

For example,pathlib.Path.read_text() can use it like this:

defread_text(self,encoding=None,errors=None):encoding=io.text_encoding(encoding)withself.open(mode='r',encoding=encoding,errors=errors)asf:returnf.read()

By usingio.text_encoding(),EncodingWarning is emitted forthe caller ofread_text() instead ofread_text() itself.

Affected standard library modules

Many standard library modules will be affected by this change.

Most APIs acceptingencoding=None will useio.text_encoding()as written in the previous section.

Where using the locale encoding as the default encoding is reasonable,encoding="locale" will be used instead. For example,thesubprocess module will use the locale encoding as the defaultfor pipes.

Many tests useopen() withoutencoding specified to readASCII text files. They should be rewritten withencoding="ascii".

Opt-in warning

AlthoughDeprecationWarning is suppressed by default, alwaysemittingDeprecationWarning when theencoding argument isomitted would be too noisy.

Noisy warnings may lead developers to dismiss theDeprecationWarning.

“locale” is not a codec alias

We don’t add “locale” as a codec alias because the locale can bechanged at runtime.

Additionally,TextIOWrapper checksos.device_encoding()whenencoding=None. This behavior cannot be implemented ina codec.

For new users

SinceEncodingWarning is used to write cross-platform code,there is no need to teach it to new users.

We can just recommend using UTF-8 for text files and usingencoding="utf-8" when opening them.

For experienced users

Usingopen(filename) to read text files encoded in UTF-8 is acommon mistake. It may not work on Windows because UTF-8 is not thedefault encoding.

You can use-Xwarn_default_encoding orPYTHONWARNDEFAULTENCODING=1 to find this type of mistake.

Omitting theencoding argument is not a bug when opening text filesencoded in the locale encoding, butencoding="locale" is recommendedin Python 3.10 and later because it is more explicit.