Uh oh!
There was an error while loading.Please reload this page.
- Notifications
You must be signed in to change notification settings - Fork32k
gh-109559: Updateunicodedata
for Unicode 15.1#109560
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to ourterms of service andprivacy statement. We’ll occasionally send you account related emails.
Already on GitHub?Sign in to your account
Uh oh!
There was an error while loading.Please reload this page.
Conversation
('2B740', '2B81D'), # CJK Ideograph Extension D | ||
('2B820', '2CEA1'), # CJK Ideograph Extension E | ||
('2CEB0', '2EBE0'), # CJK Ideograph Extension F | ||
('2EBF0', '2EE5D'), # CJK Ideograph Extension I |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
The range check that occurs later in this file implicitly assumes this list is in sorted order. It seems simpler to have an idiosyncratic order here than to try to introducesorted()
or somesuch.
for char, (propname, *propinfo) in UcdFile(DERIVED_CORE_PROPERTIES, version).expanded(): | ||
if propinfo: | ||
# this is not a binary property, ignore it | ||
continue |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
All the properties defined inDerivedCoreProperties.txt
happened to be binary until the latest update, so this tool was getting away with the assumption that they arealways binary.
As of Unicode 15.1, this file also includes definitions that use theIndict_Conjunct_Break
(InCB
) property, which is an enumeration.
With this change, the loop skips over any non-binary properties, since we have nothing to do with them.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
It seems like it would be safer to explicitly ignoreIndict_Conjunct_Break
rather than throw out everything with a second column.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
Is there a particular failure mode you have in mind? My rationale here was that the current internalized DB only cares about binary properties in this file, but in practice any of theproperty types enumerated by UAX#44 could appear in a future revision.
I'm not strongly opposed to ignoring the specific property that breaks the tool against the current revision, but my rationale was that it seems safer to prevent this class of failure in the future if/when additional non-binary properties are added.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
There are various parts in the documentation which reference the Unicode standard version, which should be updated.
Uh oh!
There was an error while loading.Please reload this page.
for char, (propname, *propinfo) in UcdFile(DERIVED_CORE_PROPERTIES, version).expanded(): | ||
if propinfo: | ||
# this is not a binary property, ignore it | ||
continue |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
It seems like it would be safer to explicitly ignoreIndict_Conjunct_Break
rather than throw out everything with a second column.
Good catch! The only reference I could find to Unicode 15.0.0 was in |
That's the one you shouldn't touch because it's generated. :) See references in |
Whoops! Guess my initial grep was poorly-formed. Thanks for the guidance. |
ghost commentedSep 20, 2023 • edited by ghost
Loading Uh oh!
There was an error while loading.Please reload this page.
edited by ghost
Uh oh!
There was an error while loading.Please reload this page.
ca72e1c
to44f6770
Comparebedevere-bot commentedSep 20, 2023
|
bedevere-bot commentedSep 20, 2023
|
bedevere-bot commentedSep 20, 2023
|
bedevere-bot commentedSep 20, 2023
|
bedevere-bot commentedSep 20, 2023
|
bedevere-bot commentedSep 20, 2023
|
bedevere-bot commentedSep 20, 2023
|
bedevere-bot commentedSep 20, 2023
|
bedevere-bot commentedSep 20, 2023
|
bedevere-bot commentedSep 20, 2023
|
bedevere-bot commentedSep 20, 2023
|
bedevere-bot commentedSep 20, 2023
|
bedevere-bot commentedSep 20, 2023
|
bedevere-bot commentedSep 20, 2023
|
bedevere-bot commentedSep 20, 2023
|
bedevere-bot commentedSep 20, 2023
|
bedevere-bot commentedSep 20, 2023
|
bedevere-bot commentedSep 20, 2023
|
bedevere-bot commentedSep 20, 2023
|
bedevere-bot commentedSep 20, 2023
|
bedevere-bot commentedSep 20, 2023
|
bedevere-bot commentedSep 20, 2023
|
bedevere-bot commentedSep 20, 2023
|
bedevere-bot commentedSep 20, 2023
|
bedevere-bot commentedSep 20, 2023
|
bedevere-bot commentedSep 20, 2023
|
bedevere-bot commentedSep 20, 2023
|
bedevere-bot commentedSep 20, 2023
|
Whoops, sorry about the breakage, I forgot to include |
…nGH-109560)---------Co-authored-by: Benjamin Peterson <benjamin@python.org>
…nGH-109560)---------Co-authored-by: Benjamin Peterson <benjamin@python.org>
Uh oh!
There was an error while loading.Please reload this page.
This changeset implements#109559, adding Unicode 15.1 support to the internal databases that support the
unicodedata
module. The bulk of this Unicode update is the addition ofa new CJK Ideograph Extension, which requires updating the CJK ranges baked into themakeunicodedata.py
tool andunicodedata.c
.I have also taken the liberty of making the documentation of CJK Ideograph Extension ranges consistent between
makeunicodedata.py
andunicodedata.c
(i.e. each range includes a comment naming the range).