- Notifications
You must be signed in to change notification settings - Fork61
Support Unicode 15.1#124
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to ourterms of service andprivacy statement. We’ll occasionally send you account related emails.
Already on GitHub?Sign in to your account
Uh oh!
There was an error while loading.Please reload this page.
Conversation
83dcbc1 toa909537Comparesyvb commentedSep 22, 2023 • edited
Loading Uh oh!
There was an error while loading.Please reload this page.
edited
Uh oh!
There was an error while loading.Please reload this page.
I originally described a categorization issue with - turns out the Unicode data files are correct, I was just using outdated ones. Oops. I kept the tests that verify (and the Syriac abbreviation mark) are categorized correctly. |
| run: ./scripts/unicode.py && diff tables.rs src/tables.rs |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
sweet, thanks for adding this. I've been adding this for the other unicode- crates bit by bit
| @@ -50,6 +50,9 @@ fn test_graphemes() { | |||
| ]; | |||
| for &(s, g) in TEST_SAME.iter().chain(EXTRA_SAME) { | |||
| if s.starts_with("क\u{94d}") || s.starts_with("क\u{93c}") { | |||
| continue; // TODO: fix these | |||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
please file an issue for this
Uh oh!
There was an error while loading.Please reload this page.
Adds Unicode 15.1 support.
Updating tests
Turns out
scripts/unicode_gen_breaktests.pywas last run for Unicode 11 - every subsequent updater forgot to run it. I updated the GitHub Action that checksscripts/unicode.pywas run to also check forscripts/unicode_gen_breaktests.pybeing run.Devanagari mis-segmentation
There are a few cases where Devanagari grapheme segmentation fails after updating the test data from Unicode 11 to Unicode 15. I just skipped those failing tests for now.