Mar 29, 2021 · Apr 7, 2020 · Apr 12, 2020 · Apr 12, 2020 · Jan 29, 2021 · Jan 29, 2021
diff --git a/Doc/c-api/init_config.rst b/Doc/c-api/init_config.rst
   .. c:member:: int warn_default_encoding

      If non-zero, emit a :exc:`EncodingWarning` warning when :class:`io.TextIOWrapper`
      uses its default encoding. See :pep:`597` fordetail.

      uses its default encoding. See :ref:`io-encoding-warning` fordetails.

      Default: ``0``.

      .. versionadded:: 3.10
diff --git a/Doc/library/exceptions.rst b/Doc/library/exceptions.rst

   Base class for warnings about encodings.

   :class:`io.TextIOWrapper`, :func:`open`, and other functions having
   ``encoding=None`` option and locale specific default encoding may emit this
   warning. See :pep:`597` for more details.
   See :ref:`io-encoding-warning` for details.

   .. versionadded:: 3.10

diff --git a/Doc/library/io.rst b/Doc/library/io.rst
 The raw stream API is described in detail in the docs of :class:`RawIOBase`.


 .. _io-text-encoding:

 Text Encoding
 -------------

 The default encoding of :class:`TextIOWrapper` and :func:`open` is
 locale-specific. (:func:`locale.getpreferredencoding(False) <locale.getpreferredencoding>`)

 But many developers forget to specify encoding when opening text files
 encoded in UTF-8 (e.g. JSON, TOML, Markdown, etc...) since most Unix
 platforms uses UTF-8 locale by default. It cause a bug because locale-specific
 encoding is not UTF-8 for most Windows users. For example::

   # may not work on Windows when non-ASCII characters in the file.
   with open("README.md") as f:
       long_description = f.read()

 Additionally, Python may change the default text encoding to UTF-8 in the
 future, although there is no plan yet.

 So it is highly recommended to specify encoding explicitly when opening text
 files. If you want to use UTF-8, specify ``encoding="utf-8"``. If you need to
 use locale-specific encoding, ``encoding="locale"`` is supported since Python
 3.10.

 When you need to run code using the default encoding to open UTF-8 files on
 Windows, you can enable the UTF-8 mode. See also the
 :ref:`UTF-8 mode on Windows <win-utf8-mode>`

 .. _io-encoding-warning:

 Opt-in EncodingWarning
 ^^^^^^^^^^^^^^^^^^^^^^

 .. versionadded:: 3.10
   See :pep:`597` for more details.

 To find where the default encoding is used, you can use
 a ``-X warn_default_encoding`` command line argument or a
 :envvar:`PYTHONWARNDEFAULTENCODING` environment variable to emit
 an :exc:`EncodingWarning` when the defaut encoding is used.

 If you are providing APIs using :func:`open` or :class:`TextIOWrapper` and
 having ``encoding=None`` parameter, you can use :func:`text_encoding` to emit
 an :exc:`EncodingWarning` to the user too. But please consider using UTF-8
 by default (i.e. ``encoding="utf-8"``).


 High-level Module Interface
 ---------------------------

 .. function:: text_encoding(encoding, stacklevel=1)

   This is a helper function for functions that use :func:`open` or
   :class:`TextIOWrapper` and take ``encoding=None``option.
   :class:`TextIOWrapper` and take ``encoding=None``argument.

   This function returns *encoding* if it is not ``None`` and "locale" if
   *encoding* is ``None``.
   ``read_text()``. If *stacklevel* is greater than 1, more stack frames are
   skipped.

   See :envvar:`PYTHONWARNDEFAULTENCODING` and :pep:`597` for more information.
   See :ref:`io-default-encoding` for more information.

   .. versionadded:: 3.10

   *encoding* gives the name of the encoding that the stream will be decoded or
   encoded with.  It defaults to
   :func:`locale.getpreferredencoding(False) <locale.getpreferredencoding>`.

   If ``sys.flags.warn_default_encoding`` is true and the default encoding
   is used, this function emits an :class:`EncodingWarning`. You can suppress
   the warning by using ``encoding="locale"`` option.
   See :envvar:`PYTHONWARNDEFAULTENCODING` and :pep:`597` for more information.
   ``encoding="locale"`` can be used to specify the locale specific encoding
   explicitly. See :ref:`io-text-encoding` for more information.

   *errors* is an optional string that specifies how encoding and decoding
   errors are to be handled.  Pass ``'strict'`` to raise a :exc:`ValueError`
diff --git a/Doc/using/cmdline.rst b/Doc/using/cmdline.rst
   * ``-X pycache_prefix=PATH`` enables writing ``.pyc`` files to a parallel
     tree rooted at the given directory instead of to the code tree. See also
     :envvar:`PYTHONPYCACHEPREFIX`.
   * ``-X warn_default_encoding`` issues a :class:`EncodingWarning` when
 an ``encoding`` option is omitted and thedefault encoding islocale-specific.
   * ``-X warn_default_encoding`` issues a :class:`EncodingWarning` when the
 locale-specificdefault encoding isused.
     See also :envvar:`PYTHONWARNDEFAULTENCODING`.

   It also allows passing arbitrary values and retrieving them through the
 .. envvar:: PYTHONWARNDEFAULTENCODING

   If this environment variable is set to a non-empty string, issue a
   :class:`EncodingWarning` when an ``encoding`` option is omitted and
   the default encoding is locale-specific.
   :class:`EncodingWarning` when the locale-specific default encoding is used.

   This option can be used to find bugs caused by not passing
   ``encoding="utf8"`` option. For example::

      # This code may cause UnicodeDecodeError on Windows.
      # encoding="utf8" or "b" mode must be used.
      with open(path) as f:
          data = json.load(f)

   ``encoding="locale"`` option can be used to specify locale-specific
   encoding explicitly since Python 3.10. Python won't issue a
   :class:`EncodingWarning` for it.

   See :pep:`597` for detail.
   See :ref:`io-encoding-warning` for details.

   .. versionadded:: 3.10
Original file line number	Diff line number	Diff line change
Expand Up		@@ -586,8 +586,8 @@ PyConfig
		.. c:member:: int warn_default_encoding

		If non-zero, emit a :exc:`EncodingWarning` warning when :class:`io.TextIOWrapper`
		uses its default encoding. See :pep:`597` fordetail.

		uses its default encoding. See :ref:`io-encoding-warning` fordetails.

		Default: ``0``.

		.. versionadded:: 3.10
Expand Down
Original file line number	Diff line number	Diff line change
Expand Up		@@ -692,9 +692,7 @@ The following exceptions are used as warning categories; see the

		Base class for warnings about encodings.
Copy link Member CAM-GerlachMar 22, 2021 Choose a reason for hiding this comment The reason will be displayed to describe this comment to others.Learn more. You might want to be more specific here, considering that (AFAIK) this isnot the base class for`UnicodeWarning`, which also has to do with encodings (yes, Unicode is a character set, not an encoding, but if developers were knowledgeable enough to understand that distinction, they certainly would know to specify their encoding and we wouldn't have this problem ^_^).

		:class:`io.TextIOWrapper`, :func:`open`, and other functions having
		``encoding=None`` option and locale specific default encoding may emit this
		warning. See :pep:`597` for more details.
		See :ref:`io-encoding-warning` for details.

		.. versionadded:: 3.10

Expand Down
Original file line number	Diff line number	Diff line change
Expand Up		@@ -106,6 +106,54 @@ stream by opening a file in binary mode with buffering disabled::
		The raw stream API is described in detail in the docs of :class:`RawIOBase`.


		.. _io-text-encoding:

		Text Encoding
		-------------

		The default encoding of :class:`TextIOWrapper` and :func:`open` is
		locale-specific. (:func:`locale.getpreferredencoding(False) <locale.getpreferredencoding>`)

		But many developers forget to specify encoding when opening text files
		encoded in UTF-8 (e.g. JSON, TOML, Markdown, etc...) since most Unix
		platforms uses UTF-8 locale by default. It cause a bug because locale-specific
		encoding is not UTF-8 for most Windows users. For example::
methane marked this conversation as resolved. OutdatedShow resolvedHide resolved

		# may not work on Windows when non-ASCII characters in the file.
methane marked this conversation as resolved. OutdatedShow resolvedHide resolved
		with open("README.md") as f:
		long_description = f.read()

		Additionally, Python may change the default text encoding to UTF-8 in the
		future, although there is no plan yet.
methane marked this conversation as resolved. OutdatedShow resolvedHide resolved

		So it is highly recommended to specify encoding explicitly when opening text
		files. If you want to use UTF-8, specify ``encoding="utf-8"``. If you need to
		use locale-specific encoding, ``encoding="locale"`` is supported since Python
		3.10.
methane marked this conversation as resolved. OutdatedShow resolvedHide resolved

		When you need to run code using the default encoding to open UTF-8 files on
		Windows, you can enable the UTF-8 mode. See also the
		:ref:`UTF-8 mode on Windows <win-utf8-mode>`
methane marked this conversation as resolved. OutdatedShow resolvedHide resolved

		.. _io-encoding-warning:

		Opt-in EncodingWarning
		^^^^^^^^^^^^^^^^^^^^^^

		.. versionadded:: 3.10
		See :pep:`597` for more details.

		To find where the default encoding is used, you can use
		a ``-X warn_default_encoding`` command line argument or a
		:envvar:`PYTHONWARNDEFAULTENCODING` environment variable to emit
		an :exc:`EncodingWarning` when the defaut encoding is used.
methane marked this conversation as resolved. OutdatedShow resolvedHide resolved

		If you are providing APIs using :func:`open` or :class:`TextIOWrapper` and
		having ``encoding=None`` parameter, you can use :func:`text_encoding` to emit
		an :exc:`EncodingWarning` to the user too. But please consider using UTF-8
		by default (i.e. ``encoding="utf-8"``).
methane marked this conversation as resolved. OutdatedShow resolvedHide resolved


		High-level Module Interface
		---------------------------

Expand DownExpand Up		@@ -146,7 +194,7 @@ High-level Module Interface
		.. function:: text_encoding(encoding, stacklevel=1)

		This is a helper function for functions that use :func:`open` or
		:class:`TextIOWrapper` and take ``encoding=None``option.
		:class:`TextIOWrapper` and take ``encoding=None``argument.

		This function returns encoding if it is not ``None`` and "locale" if
		encoding is ``None``.
Expand All		@@ -164,7 +212,7 @@ High-level Module Interface
		``read_text()``. If stacklevel is greater than 1, more stack frames are
		skipped.

		See :envvar:`PYTHONWARNDEFAULTENCODING` and :pep:`597` for more information.
		See :ref:`io-default-encoding` for more information.

		.. versionadded:: 3.10

Expand DownExpand Up		@@ -905,11 +953,8 @@ Text I/O
		encoding gives the name of the encoding that the stream will be decoded or
		encoded with. It defaults to
		:func:`locale.getpreferredencoding(False) <locale.getpreferredencoding>`.

		If ``sys.flags.warn_default_encoding`` is true and the default encoding
		is used, this function emits an :class:`EncodingWarning`. You can suppress
		the warning by using ``encoding="locale"`` option.
		See :envvar:`PYTHONWARNDEFAULTENCODING` and :pep:`597` for more information.
		``encoding="locale"`` can be used to specify the locale specific encoding
		explicitly. See :ref:`io-text-encoding` for more information.
methane marked this conversation as resolved. OutdatedShow resolvedHide resolved

		errors is an optional string that specifies how encoding and decoding
		errors are to be handled. Pass ``'strict'`` to raise a :exc:`ValueError`
Expand Down
Original file line number	Diff line number	Diff line change
Expand Up		@@ -453,8 +453,8 @@ Miscellaneous options
		* ``-X pycache_prefix=PATH`` enables writing ``.pyc`` files to a parallel
		tree rooted at the given directory instead of to the code tree. See also
		:envvar:`PYTHONPYCACHEPREFIX`.
		* ``-X warn_default_encoding`` issues a :class:`EncodingWarning` when
		an ``encoding`` option is omitted and thedefault encoding islocale-specific.
		* ``-X warn_default_encoding`` issues a :class:`EncodingWarning` when the
		locale-specificdefault encoding isused.
methane marked this conversation as resolved. OutdatedShow resolvedHide resolved
		See also :envvar:`PYTHONWARNDEFAULTENCODING`.

		It also allows passing arbitrary values and retrieving them through the
Expand DownExpand Up		@@ -916,22 +916,9 @@ conflict.
		.. envvar:: PYTHONWARNDEFAULTENCODING

		If this environment variable is set to a non-empty string, issue a
		:class:`EncodingWarning` when an ``encoding`` option is omitted and
		the default encoding is locale-specific.
		:class:`EncodingWarning` when the locale-specific default encoding is used.

		This option can be used to find bugs caused by not passing
		``encoding="utf8"`` option. For example::

		# This code may cause UnicodeDecodeError on Windows.
		# encoding="utf8" or "b" mode must be used.
		with open(path) as f:
		data = json.load(f)

		``encoding="locale"`` option can be used to specify locale-specific
		encoding explicitly since Python 3.10. Python won't issue a
		:class:`EncodingWarning` for it.

		See :pep:`597` for detail.
		See :ref:`io-encoding-warning` for details.

		.. versionadded:: 3.10

Expand Down