NotificationsYou must be signed in to change notification settings
Fork32.1k
Star67.4k

GH-133711: Enable UTF-8 mode by default (PEP 686)#133712

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to ourterms of service andprivacy statement. We’ll occasionally send you account related emails.

Already on GitHub?Sign in to your account

Jump to bottom

Open

AA-Turner wants to merge11 commits intopython:main

base:main

Choose a base branch

fromAA-Turner:utf-8-mode

Open

GH-133711: Enable UTF-8 mode by default (PEP 686)#133712

AA-Turner wants to merge11 commits intopython:mainfromAA-Turner:utf-8-mode

Conversation

Copy link

Member

AA-Turner commentedMay 8, 2025•
edited by github-actionsbot
Loading

Issue:Enable UTF-8 mode by default (implement PEP 686) #133711

📚 Documentation preview 📚:https://cpython-previews--133712.org.readthedocs.build/

Enable UTF-8 mode by default (PEP 686)

fc19177

AA-Turner requested a review frommethane

May 8, 2025 21:21

AA-Turner requested review fromgpshead,FFY00 andericsnowcurrently ascode owners

May 8, 2025 21:21

bedevere-appbot mentioned this pull request

May 8, 2025

Enable UTF-8 mode by default (implement PEP 686)#133711

Open

bedevere-appbot added the awaiting core review label

May 8, 2025

AA-Turner requested a review fromvstinner

May 8, 2025 21:21

This comment was marked as outdated.

This comment was marked as resolved.

This comment was marked as resolved.

vstinner reviewed

May 9, 2025

View reviewed changes

Doc/library/os.rst OutdatedShow resolvedHide resolved

Doc/library/os.rstShow resolvedHide resolved

Doc/library/sys.rst OutdatedShow resolvedHide resolved

zooba reviewed

May 9, 2025

View reviewed changes

Doc/whatsnew/3.15.rst Outdated

		* Python UTF-8 mode is now enabled by default.
		It may be disabled with by setting :envvar:`PYTHONUTF8=0 <PYTHONUTF8>` as
		an environment variable or by using the :option:`-X utf8=0 <-X>` flag.
		See :pep:`686` for further details.

Copy link

Member

zoobaMay 9, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

I feel like we can probably put some more explanation in here, such as that it affects TextIOWrapper and henceopen(). The current description doesn't sound as scary as it needs to, in my opinion.

Along the lines of: "Python UTF-8 mode is now enabled by default. This means that (files/console/etc.) will now use UTF-8 regardless of system settings, unless specifically overridden in code (typically with anencoding= argument)."

Copy link

Member

zoobaMay 9, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

To be clear, it's nothing new. But we shouldn't assume that everyone already knows what UTF-8 mode implies. There are many more people out there who haven't ever thought about it than those who are waiting for it to be the default.

Copy link

Member

vstinnerMay 9, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

Another effect of the UTF-8 Mode is that Python ignores the locale encoding.

Copy link

MemberAuthor

AA-TurnerMay 10, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

I've expanded theWhat's New entry, please seehttps://cpython-previews--133712.org.readthedocs.build/en/133712/whatsnew/3.15.html#other-language-changes

AA-Turner added3 commits

May 10, 2025 01:47

Victor's documentation comments

c706cfa

Fix test_cmd_line

f095ab0

Improve What's New entry

ff066b1

AA-Turner force-pushed theutf-8-mode branch from036555d toff066b1Compare

May 10, 2025 02:09

AA-Turnerand others added2 commits

May 10, 2025 03:20

Add references to UTF-8 and Unicode

abfb864

Merge branch 'main' into utf-8-mode

f40b986

FFY00 removed their request for review

May 10, 2025 02:28

vstinner reviewed

May 10, 2025

View reviewed changes

Doc/using/windows.rstShow resolvedHide resolved

vstinner reviewed

May 10, 2025

View reviewed changes

Doc/whatsnew/3.15.rst

		@@ -75,7 +75,30 @@ New features
		Other language changes
		======================


		* Python now uses UTF-8_ as the default encoding, independent of the system's

Copy link

Member

vstinnerMay 10, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

You might mention the UTF-8 Mode earlier since it has other side effects documented in the UTF-8 Mode section, such as changing sys.stdout error handler and ignoring the locale encoding.

methane reviewed

May 10, 2025

View reviewed changes

Lib/test/test_cmd_line.py OutdatedShow resolvedHide resolved

AA-Turner added2 commits

May 10, 2025 23:50

Check stream buffer type in test_python_legacy_windows_stdio

3226e1d

Merge branch 'main' into utf-8-mode

b83c966

# Conflicts:#Doc/whatsnew/3.15.rst

This comment was marked as resolved.

This comment was marked as resolved.

methane reviewed

May 10, 2025

View reviewed changes

Lib/test/test_cmd_line.py OutdatedShow resolvedHide resolved

methane reviewed

May 11, 2025

View reviewed changes

Lib/test/test_cmd_line.py OutdatedShow resolvedHide resolved

StanFromIreland mentioned this pull request

May 11, 2025

gh-133677: Fix tests when running in non-UTF-8 locale#133865

Merged

This comment was marked as resolved.

This comment was marked as outdated.

This comment was marked as resolved.

AA-Turner added2 commits

May 26, 2025 10:32

Merge branch 'main' into utf-8-mode

b56e01b

# Conflicts:#Lib/test/test_cmd_line.py

line length

3da6bf2

AA-Turner added the 🔨 test-with-buildbotsTest PR w/ buildbots; report in status section label

May 26, 2025

This comment was marked as outdated.

bedevere-bot removed the 🔨 test-with-buildbotsTest PR w/ buildbots; report in status section label

May 26, 2025

This comment was marked as resolved.

Copy link

Member

vstinner commentedMay 28, 2025

test_regrtest failed:

I wrote#134839 to fix test_regrtest when PYTHONUTF8=1 env var is set.

Copy link

Member

vstinner commentedMay 28, 2025

I can also reproduce the [test_readline] issue in the main branch using the fr_FR locale and the command:
PYTHONUTF8=1 LANG=fr_FR ./python -m test -v test_readline -u all

I wrote#134841 to fix test_readline for the Python UTF-8 Mode.

Merge branch 'main' into utf-8-mode

9b3ebf7

AA-Turner added the 🔨 test-with-buildbotsTest PR w/ buildbots; report in status section label

May 28, 2025

This comment was marked as duplicate.

bedevere-bot removed the 🔨 test-with-buildbotsTest PR w/ buildbots; report in status section label

May 28, 2025

Copy link

Member

vstinner commentedMay 28, 2025

buildbot/wasm32-wasi Non-Debug PR — Build done.

Configure host Python fails with:

checking for /dev/ptmx... not setconfigure: error: set ac_cv_file__dev_ptmx to yes/no in your CONFIG_SITE file when cross compiling

It seems to be unrelated to this PR.

Copy link

MemberAuthor

AA-Turner commentedMay 29, 2025

buildbot/wasm32-wasi Non-Debug PR — Build done.
Configure host Python fails with:
checking for /dev/ptmx... not setconfigure: error: set ac_cv_file__dev_ptmx to yes/no in your CONFIG_SITE file when cross compiling
It seems to be unrelated to this PR.

cc@brettcannon, this builder seems to beconsistently failing since ~15/05.

vstinner approved these changes

May 30, 2025

View reviewed changes

Copy link

Member

vstinner left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

LGTM.

Doc/library/os.rst


		Python UTF-8 mode is now enabled by default (:pep:`686`).
		It may be disabled with by setting :envvar:`PYTHONUTF8=0 <PYTHONUTF8>` as
		an environment variable or by using the :option:`-X utf8=0 <-X>` flag.

Copy link

Member

vstinnerMay 30, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

Suggested change

	an environment variable or by using the:option:`-X utf8=0 <-X>`flag.
	an environment variable or by using the:option:`-X utf8=0 <-X>`command line option.

Include/cpython/initconfig.h

Comment on lines +108 to +109

		Enabled by default (equalto 1; PEP 686), or if Py_UTF8Mode=1,
		or if "-X utf8=1" or PYTHONUTF8=1.

Copy link

Member

vstinnerMay 30, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

Suggested change

	Enabledbydefault (equalto1;PEP686),orifPy_UTF8Mode=1,
	orif"-X utf8=1"orPYTHONUTF8=1.
	Enabledbydefault (equalto1;PEP686),orifPy_UTF8Mode=1,
	orif"-X utf8=1"orPYTHONUTF8=1.