This PRcloses#113274. This is done by changing the UCS-2 codepoint 126 (~, TILDE) to 65374 (～, FULLWIDTH TILDE) in the__jisx0212_decmap reference table. Since no character is added or removed, no other changes are needed.

Bug report

Bug description:

Python decodes the bytes 0x8FA2A7 as ~ (TILDE) in EUC-JP.

assertb'\x8f\xa2\xb7'.decode('euc_jp')=='~'

Thisreference document is ambiguous in that it shows a simple ~ (TILDE), but most other software (iconv, Vim, Firefox, Rust's encoding_rs) interpret this as ～ (FULLWIDTH TILDE). Note that EUC-JP already includes US-ASCII, and so:

assert'~'.encode('euc-jp')==b'~'

CPython versions tested on:

3.11, CPython main branch

Operating systems tested on:

Linux

Issue:Python decodes EUC-JP 8FA2A7 as TILDE instead of FULLWIDTH TILDE #113274

qsantos requested a review fromcorona10 as acode owner

December 19, 2023 07:37

bedevere-appbot added the awaiting review label

Dec 19, 2023

bedevere-appbot mentioned this pull request

Dec 19, 2023

Python decodes EUC-JP 8FA2A7 as TILDE instead of FULLWIDTH TILDE#113274

Open

corona10 requested changes

Dec 19, 2023

View reviewed changes

Modules/cjkcodecs/mappings_jp.h Outdated

		@@ -591,7 +591,7 @@ __jisx0208_decmap+6950,33,38},{0,0,0},{0,0,0},{0,0,0},{0,0,0},{0,0,0},{0,0,0},
		};

		static const ucs2_t __jisx0212_decmap[6179] = {
		728,711,184,729,733,175,731,730,126,900,901,U,U,U,U,U,U,U,U,161,166,191,U,U,U,
		728,711,184,729,733,175,731,730,65374,900,901,U,U,U,U,U,U,U,U,161,166,191,U,U,U,

Copy link

Member

corona10Dec 19, 2023•
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

cpython/Modules/cjkcodecs/mappings_jp.h

Line 1 in171ebb2

// AUTO-GENERATED FILE FROM genmap_japanese.py: DO NOT EDIT

Please take a look at the comment on the first line.
I 've not taken a look at the issue deeply yet, but if you want to change the mapping file, you should modify the generator.

Copy link

Author

qsantosDec 19, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

I totally missed that. I will do that.

Copy link

Author

qsantosDec 19, 2023•
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

Python generates the data using this as its source of truth:https://www.unicode.org/Public/MAPPINGS/OBSOLETE/EASTASIA/JIS/JIS0212.TXT.
encoding_rs: also generated, but fromhttps://encoding.spec.whatwg.org/index-jis0212.txt
iconv: seems to be maintained indenpendently, and we can actually seen when they replaced TILDE with FULLWIDTH TILDE in their changelog:https://git.savannah.gnu.org/gitweb/?p=libiconv.git;a=blob;f=ChangeLog;h=0e2f5f00bc6aff84932cd92bd09a7c14f802c44d;hb=refs/heads/master#l4729

I couldn't figure out how Vim and Firefox proceed (although Vim does fallback toiconv for non-internally-supported encodings). Since I do not expect to get Unicode to update the document of an obsolete standard that they only used to adapt, I see two options:

add a hack ingenmap_japanese.py around line 90 after the call toloadmap(jisx0212file) to reassignjisx0212decmap[34][55] = ord('～') (it does fix__jisx0212_decmap as expected and changes nothing else), and comment that properly
switch to WhatWG as the source of truth

1 can be done pretty quickly and with minimal risk of breaking anything. 2 has the advantage that it might help discover other issues in mappings, but is obviously much more involved.

If 1 seems reasonable to you, I will update this PR accordingly. If 2, I will close it and document this in the ticket for later.

Copy link

Author

qsantosDec 21, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

I have updated this MR with the hack ingenmap_japanese.py and regeneratedmappings_jp.h.

Copy link

Member

corona10Dec 24, 2023•
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

I will take a look at which will be the better, If we decide to solve this issue, option 2 will be the better way.
But I need to check the current status related to JIS0212 and side effect.
Also need to consider stability of whatwg spec.

bedevere-appbot added awaiting changes and removed awaiting review labels

Dec 19, 2023

Copy link

bedevere-appbot commentedDec 19, 2023

A Python core developer has requested some changes be made to your pull request before we can consider merging it. If you could please address their requests along with any other requests in other reviews from core developers that would be appreciated.

Once you have made the requested changes, please leave a comment on this pull request containing the phraseI have made the requested changes; please review again. I will then notify any core developers who have left a review that you're ready for them to take another look at this pull request.

pythongh-113274: fix EUC-JP decoding of FULLWIDTH TILDE

1631b56

qsantos force-pushed thefix-issue-113274 branch from171ebb2 to1631b56Compare

December 21, 2023 16:19

Pass FULLWIDTH TILDE in euc_jisx0213

87d6103

Copy link

Author

qsantos commentedDec 24, 2023

I have made the requested changes; please review again.

Copy link

Member

corona10 commentedDec 24, 2023

@qsantos I will take a look the PR by next week. Enjoy your Christmas.

corona10 self-assigned this

Dec 24, 2023

Copy link

Member

corona10 commentedDec 24, 2023•
edited
Loading

Okay, I like this PR, it will guarantee the round trip

a = b'\x8f\xa2\xb7'assert a.decode('euc_jp').encode('euc_jp') == a

But as I said, I will take a look at this PR by next week.

Copy link

Author

qsantos commentedDec 24, 2023

@qsantos I will take a look the PR by next week. Enjoy your Christmas.

Thanks, I was not sure if I had properly triggered the bot. Enjoy your Christmas as well!

corona10 requested changes

Dec 31, 2023

View reviewed changes

Copy link

Member

corona10 left a comment•
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

Okay, here is my temporal conclusion for this change.
(If we decide to adopt the change!)

Let's use uncode.org data as same as now, whatwg is great but not stable. We are not a browser, I am not sure that we can follow every change without side effect.
Let's vendor the modified JIS0212.TXT intohttps://github.com/python/cpython/tree/main/Tools/unicode/python-mappings and store the diff file tohttps://github.com/python/cpython/tree/main/Tools/unicode/python-mappings/diff for tracking changes.

But I need to listen to the opinion of@ezio-melotti, who is a more experienced unicode expert than me.

corona10 assignedezio-melotti

Dec 31, 2023

corona10 added the topic-unicode label

Dec 31, 2023

Copy link

Author

qsantos commentedFeb 1, 2024

@ezio-melotti What is your take on this?

Copy link

Author

qsantos commentedApr 5, 2024

Should we consider this dead?

python deleted a comment

Apr 7, 2025

Labels

awaiting changes topic-unicode

3 participants

Movatterモバイル変換

Uh oh!

gh-113274: fix EUC-JP decoding of FULLWIDTH TILDE#113275

Are you sure you want to change the base?

gh-113274: fix EUC-JP decoding of FULLWIDTH TILDE#113275

Conversation

qsantos commentedDec 19, 2023• edited by bedevere-appbotLoading Uh oh!There was an error while loading.Please reload this page.

Uh oh!

Bug report

Bug description:

CPython versions tested on:

Operating systems tested on:

Uh oh!

corona10Dec 19, 2023• editedLoading Uh oh!There was an error while loading.Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

qsantosDec 19, 2023

Choose a reason for hiding this comment

Uh oh!

qsantosDec 19, 2023• editedLoading Uh oh!There was an error while loading.Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

qsantosDec 21, 2023

Choose a reason for hiding this comment

Uh oh!

corona10Dec 24, 2023• editedLoading Uh oh!There was an error while loading.Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bedevere-appbot commentedDec 19, 2023

Uh oh!

qsantos commentedDec 24, 2023

Uh oh!

corona10 commentedDec 24, 2023

Uh oh!

corona10 commentedDec 24, 2023• editedLoading Uh oh!There was an error while loading.Please reload this page.

Uh oh!

Uh oh!

qsantos commentedDec 24, 2023

Uh oh!

corona10 left a comment• editedLoading Uh oh!There was an error while loading.Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

qsantos commentedFeb 1, 2024

Uh oh!

qsantos commentedApr 5, 2024

Uh oh!

Uh oh!

qsantos commentedDec 19, 2023•
edited by bedevere-appbot
Loading

corona10Dec 19, 2023•
edited
Loading

qsantosDec 19, 2023•
edited
Loading

corona10Dec 24, 2023•
edited
Loading

corona10 commentedDec 24, 2023•
edited
Loading

corona10 left a comment•
edited
Loading