Movatterモバイル変換

[0]ホーム
URL:

Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

 Sign up
Appearance settings

gh-109559: Updateunicodedata for Unicode 15.1#109560

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to ourterms of service andprivacy statement. We’ll occasionally send you account related emails.

Already on GitHub?Sign in to your account

Merged
benjaminp merged 12 commits intopython:mainfromSnoopJ:feature/Unicode-15.1
Sep 20, 2023
Merged
Show file tree
Hide file tree
Changes fromall commits
Commits
Show all changes
12 commits
Select commitHold shift + click to select a range
21e297c
Bump UNIDATA_VERSION to 15.1
SnoopJSep 19, 2023
122a732
Label CJK ranges as in unicodedata.c
SnoopJSep 19, 2023
6d5238e
Add CJK Ideograph Extension I to CJK ranges
SnoopJSep 19, 2023
cd9cbf5
Handle non-binary character properties
SnoopJSep 19, 2023
818a36c
Update Unicode DBs for Unicode 15.1
SnoopJSep 19, 2023
24088ca
Add NEWS fragment
SnoopJSep 19, 2023
110c552
Use proper Sphinx syntax for NEWS fragment
SnoopJSep 19, 2023
d8d9f98
Use common in-source ordering for CJK extension blocks
SnoopJSep 19, 2023
27b1c13
Bump Unicode version reference in pydoc_data
SnoopJSep 19, 2023
af730eb
Revert "Bump Unicode version reference in pydoc_data"
SnoopJSep 20, 2023
0db6920
Update UCD version references to 15.1.0
SnoopJSep 20, 2023
44f6770
Update 2023-09-19-01-22-43.gh-issue-109559.ijaycU.rst
benjaminpSep 20, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 4 additions & 4 deletionsDoc/library/stdtypes.rst
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -1641,7 +1641,7 @@ expression support in the :mod:`re` module).

The casefolding algorithm is
`described in section 3.13 'Default Case Folding' of the Unicode Standard
<https://www.unicode.org/versions/Unicode15.0.0/ch03.pdf>`__.
<https://www.unicode.org/versions/Unicode15.1.0/ch03.pdf>`__.

.. versionadded:: 3.3

Expand DownExpand Up@@ -1805,7 +1805,7 @@ expression support in the :mod:`re` module).
property being one of "Lm", "Lt", "Lu", "Ll", or "Lo". Note that this is different
from the `Alphabetic property defined in the section 4.10 'Letters, Alphabetic, and
Ideographic' of the Unicode Standard
<https://www.unicode.org/versions/Unicode15.0.0/ch04.pdf>`_.
<https://www.unicode.org/versions/Unicode15.1.0/ch04.pdf>`_.


.. method:: str.isascii()
Expand DownExpand Up@@ -1941,7 +1941,7 @@ expression support in the :mod:`re` module).

The lowercasing algorithm used is
`described in section 3.13 'Default Case Folding' of the Unicode Standard
<https://www.unicode.org/versions/Unicode15.0.0/ch03.pdf>`__.
<https://www.unicode.org/versions/Unicode15.1.0/ch03.pdf>`__.


.. method:: str.lstrip([chars])
Expand DownExpand Up@@ -2290,7 +2290,7 @@ expression support in the :mod:`re` module).

The uppercasing algorithm used is
`described in section 3.13 'Default Case Folding' of the Unicode Standard
<https://www.unicode.org/versions/Unicode15.0.0/ch03.pdf>`__.
<https://www.unicode.org/versions/Unicode15.1.0/ch03.pdf>`__.


.. method:: str.zfill(width)
Expand Down
8 changes: 4 additions & 4 deletionsDoc/library/unicodedata.rst
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -17,8 +17,8 @@

This module provides access to the Unicode Character Database (UCD) which
defines character properties for all Unicode characters. The data contained in
this database is compiled from the `UCD version 15.0.0
<https://www.unicode.org/Public/15.0.0/ucd>`_.
this database is compiled from the `UCD version 15.1.0
<https://www.unicode.org/Public/15.1.0/ucd>`_.

The module uses the same names and symbols as defined by Unicode
Standard Annex #44, `"Unicode Character Database"
Expand DownExpand Up@@ -175,6 +175,6 @@ Examples:

.. rubric:: Footnotes

.. [#] https://www.unicode.org/Public/15.0.0/ucd/NameAliases.txt
.. [#] https://www.unicode.org/Public/15.1.0/ucd/NameAliases.txt

.. [#] https://www.unicode.org/Public/15.0.0/ucd/NamedSequences.txt
.. [#] https://www.unicode.org/Public/15.1.0/ucd/NamedSequences.txt
8 changes: 4 additions & 4 deletionsDoc/reference/lexical_analysis.rst
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -315,16 +315,16 @@ The Unicode category codes mentioned above stand for:
* *Nd* - decimal numbers
* *Pc* - connector punctuations
* *Other_ID_Start* - explicit list of characters in `PropList.txt
<https://www.unicode.org/Public/15.0.0/ucd/PropList.txt>`_ to support backwards
<https://www.unicode.org/Public/15.1.0/ucd/PropList.txt>`_ to support backwards
compatibility
* *Other_ID_Continue* - likewise

All identifiers are converted into the normal form NFKC while parsing; comparison
of identifiers is based on NFKC.

A non-normative HTML file listing all valid identifier characters for Unicode
15.0.0 can be found at
https://www.unicode.org/Public/15.0.0/ucd/DerivedCoreProperties.txt
15.1.0 can be found at
https://www.unicode.org/Public/15.1.0/ucd/DerivedCoreProperties.txt


.. _keywords:
Expand DownExpand Up@@ -1045,4 +1045,4 @@ occurrence outside string literals and comments is an unconditional error:

.. rubric:: Footnotes

.. [#] https://www.unicode.org/Public/15.0.0/ucd/NameAliases.txt
.. [#] https://www.unicode.org/Public/15.1.0/ucd/NameAliases.txt
View file
Open in desktop
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
Update :mod:`unicodedata` database to Unicode 15.1.0.
1 change: 1 addition & 0 deletionsModules/unicodedata.c
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -1035,6 +1035,7 @@ is_unified_ideograph(Py_UCS4 code)
(0x2B740 <= code && code <= 0x2B81D) || /* CJK Ideograph Extension D */
(0x2B820 <= code && code <= 0x2CEA1) || /* CJK Ideograph Extension E */
(0x2CEB0 <= code && code <= 0x2EBE0) || /* CJK Ideograph Extension F */
(0x2EBF0 <= code && code <= 0x2EE5D) || /* CJK Ideograph Extension I */
(0x30000 <= code && code <= 0x3134A) || /* CJK Ideograph Extension G */
(0x31350 <= code && code <= 0x323AF); /* CJK Ideograph Extension H */
}
Expand Down
2,630 changes: 1,486 additions & 1,144 deletionsModules/unicodedata_db.h
View file
Open in desktop

Large diffs are not rendered by default.

32,946 changes: 16,476 additions & 16,470 deletionsModules/unicodename_db.h
View file
Open in desktop

Large diffs are not rendered by default.

1,963 changes: 1,013 additions & 950 deletionsObjects/unicodetype_db.h
View file
Open in desktop

Large diffs are not rendered by default.

29 changes: 17 additions & 12 deletionsTools/unicode/makeunicodedata.py
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -44,7 +44,7 @@
# * Doc/library/stdtypes.rst, and
# * Doc/library/unicodedata.rst
# * Doc/reference/lexical_analysis.rst (two occurrences)
UNIDATA_VERSION = "15.0.0"
UNIDATA_VERSION = "15.1.0"
UNICODE_DATA = "UnicodeData%s.txt"
COMPOSITION_EXCLUSIONS = "CompositionExclusions%s.txt"
EASTASIAN_WIDTH = "EastAsianWidth%s.txt"
Expand DownExpand Up@@ -101,15 +101,16 @@

# these ranges need to match unicodedata.c:is_unified_ideograph
cjk_ranges = [
('3400', '4DBF'),
('4E00', '9FFF'),
('20000', '2A6DF'),
('2A700', '2B739'),
('2B740', '2B81D'),
('2B820', '2CEA1'),
('2CEB0', '2EBE0'),
('30000', '3134A'),
('31350', '323AF'),
('3400', '4DBF'), # CJK Ideograph Extension A CJK
('4E00', '9FFF'), # CJK Ideograph
('20000', '2A6DF'), # CJK Ideograph Extension B
('2A700', '2B739'), # CJK Ideograph Extension C
('2B740', '2B81D'), # CJK Ideograph Extension D
('2B820', '2CEA1'), # CJK Ideograph Extension E
('2CEB0', '2EBE0'), # CJK Ideograph Extension F
('2EBF0', '2EE5D'), # CJK Ideograph Extension I
Copy link
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

The range check that occurs later in this file implicitly assumes this list is in sorted order. It seems simpler to have an idiosyncratic order here than to try to introducesorted() or somesuch.

('30000', '3134A'), # CJK Ideograph Extension G
('31350', '323AF'), # CJK Ideograph Extension H
]


Expand DownExpand Up@@ -1105,11 +1106,15 @@ def __init__(self, version, cjk_check=True):
table[i].east_asian_width = widths[i]
self.widths = widths

for char, (p,) in UcdFile(DERIVED_CORE_PROPERTIES, version).expanded():
for char, (propname, *propinfo) in UcdFile(DERIVED_CORE_PROPERTIES, version).expanded():
if propinfo:
# this is not a binary property, ignore it
continue
Comment on lines +1109 to +1112
Copy link
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

All the properties defined inDerivedCoreProperties.txt happened to be binary until the latest update, so this tool was getting away with the assumption that they arealways binary.

As of Unicode 15.1, this file also includes definitions that use theIndict_Conjunct_Break (InCB) property, which is an enumeration.

With this change, the loop skips over any non-binary properties, since we have nothing to do with them.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

It seems like it would be safer to explicitly ignoreIndict_Conjunct_Break rather than throw out everything with a second column.

Copy link
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

Is there a particular failure mode you have in mind? My rationale here was that the current internalized DB only cares about binary properties in this file, but in practice any of theproperty types enumerated by UAX#44 could appear in a future revision.

I'm not strongly opposed to ignoring the specific property that breaks the tool against the current revision, but my rationale was that it seems safer to prevent this class of failure in the future if/when additional non-binary properties are added.


if table[char]:
# Some properties (e.g. Default_Ignorable_Code_Point)
# apply to unassigned code points; ignore them
table[char].binary_properties.add(p)
table[char].binary_properties.add(propname)

for char_range, value in UcdFile(LINE_BREAK, version):
if value not in MANDATORY_LINE_BREAKS:
Expand Down
[8]ページ先頭
©2009-2025 Movatter.jp