- Notifications
You must be signed in to change notification settings - Fork5
Closed
Description
This is needed for mixed script detection.
The easy way to do this is just to store a slice of script_extensions for each code point / range, but there's actually a limited set of ways script_extensions mix (taken fromhere):
Adlam (Adlam), Adlam,Arabic,Hanifi_Rohingya,Mandaic,Manichaean,Psalter_Pahlavi,Sogdian,Syriac (Adlam,Arabic,Hanifi_Rohingya,Mandaic,Manichaean,Psalter_Pahlavi,Sogdian,Syriac), Ahom (Ahom), Anatolian_Hieroglyphs (Anatolian_Hieroglyphs), Arabic (Arabic), Arabic,Coptic (Arabic,Coptic), Arabic,Hanifi_Rohingya (Arabic,Hanifi_Rohingya), Arabic,Hanifi_Rohingya,Syriac,Thaana (Arabic,Hanifi_Rohingya,Syriac,Thaana), Arabic,Syriac (Arabic,Syriac), Arabic,Syriac,Thaana (Arabic,Syriac,Thaana), Arabic,Thaana (Arabic,Thaana), Armenian (Armenian), Armenian,Georgian (Armenian,Georgian), Avestan (Avestan), Balinese (Balinese), Bamum (Bamum), Bassa_Vah (Bassa_Vah), Batak (Batak), Bengali (Bengali), Bengali,Chakma,Syloti_Nagri (Bengali,Chakma,Syloti_Nagri), Bengali,Devanagari (Bengali,Devanagari), Bengali,Devanagari,Dogra,Grantha,Gujarati,Gunjala_Gondi,Gurmukhi,Kannada,Khudawadi,Limbu,Mahajani,Malayalam,Masaram_Gondi,Nandinagari,Oriya,Sinhala,Syloti_Nagri,Takri,Tamil,Telugu,Tirhuta (Bengali,Devanagari,Dogra,Grantha,Gujarati,Gunjala_Gondi,Gurmukhi,Kannada,Khudawadi,Limbu,Mahajani,Malayalam,Masaram_Gondi,Nandinagari,Oriya,Sinhala,Syloti_Nagri,Takri,Tamil,Telugu,Tirhuta), Bengali,Devanagari,Dogra,Grantha,Gujarati,Gunjala_Gondi,Gurmukhi,Kannada,Khudawadi,Mahajani,Malayalam,Masaram_Gondi,Nandinagari,Oriya,Sinhala,Syloti_Nagri,Takri,Tamil,Telugu,Tirhuta (Bengali,Devanagari,Dogra,Grantha,Gujarati,Gunjala_Gondi,Gurmukhi,Kannada,Khudawadi,Mahajani,Malayalam,Masaram_Gondi,Nandinagari,Oriya,Sinhala,Syloti_Nagri,Takri,Tamil,Telugu,Tirhuta), Bengali,Devanagari,Grantha,Gujarati,Gurmukhi,Kannada,Latin,Malayalam,Oriya,Sharada,Tamil,Telugu,Tirhuta (Bengali,Devanagari,Grantha,Gujarati,Gurmukhi,Kannada,Latin,Malayalam,Oriya,Sharada,Tamil,Telugu,Tirhuta), Bengali,Devanagari,Grantha,Gujarati,Gurmukhi,Kannada,Latin,Malayalam,Oriya,Tamil,Telugu,Tirhuta (Bengali,Devanagari,Grantha,Gujarati,Gurmukhi,Kannada,Latin,Malayalam,Oriya,Tamil,Telugu,Tirhuta), Bengali,Devanagari,Grantha,Kannada (Bengali,Devanagari,Grantha,Kannada), Bengali,Devanagari,Grantha,Kannada,Nandinagari,Oriya,Telugu,Tirhuta (Bengali,Devanagari,Grantha,Kannada,Nandinagari,Oriya,Telugu,Tirhuta), Bhaiksuki (Bhaiksuki), Bopomofo (Bopomofo), Bopomofo,Han (Bopomofo,Han), Bopomofo,Han,Hangul,Hiragana,Katakana (Bopomofo,Han,Hangul,Hiragana,Katakana), Bopomofo,Han,Hangul,Hiragana,Katakana,Yi (Bopomofo,Han,Hangul,Hiragana,Katakana,Yi), Brahmi (Brahmi), Braille (Braille), Buginese (Buginese), Buginese,Javanese (Buginese,Javanese), Buhid (Buhid), Buhid,Hanunoo,Tagalog,Tagbanwa (Buhid,Hanunoo,Tagalog,Tagbanwa), Canadian_Aboriginal (Canadian_Aboriginal), Carian (Carian), Caucasian_Albanian (Caucasian_Albanian), Chakma (Chakma), Chakma,Myanmar,Tai_Le (Chakma,Myanmar,Tai_Le), Cham (Cham), Cherokee (Cherokee), Common (Common), Coptic (Coptic), Cuneiform (Cuneiform), Cypriot (Cypriot), Cypriot,Linear_A,Linear_B (Cypriot,Linear_A,Linear_B), Cypriot,Linear_B (Cypriot,Linear_B), Cyrillic (Cyrillic), Cyrillic,Glagolitic (Cyrillic,Glagolitic), Cyrillic,Latin (Cyrillic,Latin), Cyrillic,Old_Permic (Cyrillic,Old_Permic), Deseret (Deseret), Devanagari (Devanagari), Devanagari,Dogra,Gujarati,Gurmukhi,Kaithi,Kannada,Khojki,Khudawadi,Mahajani,Malayalam,Modi,Nandinagari,Takri,Tirhuta (Devanagari,Dogra,Gujarati,Gurmukhi,Kaithi,Kannada,Khojki,Khudawadi,Mahajani,Malayalam,Modi,Nandinagari,Takri,Tirhuta), Devanagari,Dogra,Gujarati,Gurmukhi,Kaithi,Kannada,Khojki,Khudawadi,Mahajani,Modi,Nandinagari,Takri,Tirhuta (Devanagari,Dogra,Gujarati,Gurmukhi,Kaithi,Kannada,Khojki,Khudawadi,Mahajani,Modi,Nandinagari,Takri,Tirhuta), Devanagari,Dogra,Gujarati,Gurmukhi,Kaithi,Khojki,Khudawadi,Mahajani,Modi,Takri,Tirhuta (Devanagari,Dogra,Gujarati,Gurmukhi,Kaithi,Khojki,Khudawadi,Mahajani,Modi,Takri,Tirhuta), Devanagari,Dogra,Kaithi,Mahajani (Devanagari,Dogra,Kaithi,Mahajani), Devanagari,Grantha (Devanagari,Grantha), Devanagari,Grantha,Kannada (Devanagari,Grantha,Kannada), Devanagari,Grantha,Latin (Devanagari,Grantha,Latin), Devanagari,Kannada,Malayalam,Oriya,Tamil,Telugu (Devanagari,Kannada,Malayalam,Oriya,Tamil,Telugu), Devanagari,Nandinagari (Devanagari,Nandinagari), Devanagari,Sharada (Devanagari,Sharada), Devanagari,Tamil (Devanagari,Tamil), Dogra (Dogra), Duployan (Duployan), Egyptian_Hieroglyphs (Egyptian_Hieroglyphs), Elbasan (Elbasan), Elymaic (Elymaic), Ethiopic (Ethiopic), Georgian (Georgian), Georgian,Latin (Georgian,Latin), Glagolitic (Glagolitic), Gothic (Gothic), Grantha (Grantha), Grantha,Tamil (Grantha,Tamil), Greek (Greek), Gujarati (Gujarati), Gujarati,Khojki (Gujarati,Khojki), Gunjala_Gondi (Gunjala_Gondi), Gurmukhi (Gurmukhi), Gurmukhi,Multani (Gurmukhi,Multani), Han (Han), Han,Hiragana,Katakana (Han,Hiragana,Katakana), Hangul (Hangul), Hanifi_Rohingya (Hanifi_Rohingya), Hanunoo (Hanunoo), Hatran (Hatran), Hebrew (Hebrew), Hiragana (Hiragana), Hiragana,Katakana (Hiragana,Katakana), Imperial_Aramaic (Imperial_Aramaic), Inherited (Inherited), Inscriptional_Pahlavi (Inscriptional_Pahlavi), Inscriptional_Parthian (Inscriptional_Parthian), Javanese (Javanese), Kaithi (Kaithi), Kannada (Kannada), Kannada,Nandinagari (Kannada,Nandinagari), Katakana (Katakana), Kayah_Li (Kayah_Li), Kayah_Li,Latin,Myanmar (Kayah_Li,Latin,Myanmar), Kharoshthi (Kharoshthi), Khmer (Khmer), Khojki (Khojki), Khudawadi (Khudawadi), Lao (Lao), Latin (Latin), Latin,Mongolian (Latin,Mongolian), Lepcha (Lepcha), Limbu (Limbu), Linear_A (Linear_A), Linear_B (Linear_B), Lisu (Lisu), Lycian (Lycian), Lydian (Lydian), Mahajani (Mahajani), Makasar (Makasar), Malayalam (Malayalam), Mandaic (Mandaic), Manichaean (Manichaean), Marchen (Marchen), Masaram_Gondi (Masaram_Gondi), Medefaidrin (Medefaidrin), Meetei_Mayek (Meetei_Mayek), Mende_Kikakui (Mende_Kikakui), Meroitic_Cursive (Meroitic_Cursive), Meroitic_Hieroglyphs (Meroitic_Hieroglyphs), Miao (Miao), Modi (Modi), Mongolian (Mongolian), Mongolian,Phags_Pa (Mongolian,Phags_Pa), Mro (Mro), Multani (Multani), Myanmar (Myanmar),Nabataean (Nabataean), Nandinagari (Nandinagari), New_Tai_Lue (New_Tai_Lue), Newa (Newa), Nko (Nko), Nushu (Nushu), Nyiakeng_Puachue_Hmong (Nyiakeng_Puachue_Hmong),Ogham (Ogham), Ol_Chiki (Ol_Chiki), Old_Hungarian (Old_Hungarian), Old_Italic (Old_Italic), Old_North_Arabian (Old_North_Arabian), Old_Permic (Old_Permic), Old_Persian (Old_Persian), Old_Sogdian (Old_Sogdian), Old_South_Arabian (Old_South_Arabian), Old_Turkic (Old_Turkic), Oriya (Oriya), Osage (Osage), Osmanya (Osmanya),Pahawh_Hmong (Pahawh_Hmong), Palmyrene (Palmyrene), Pau_Cin_Hau (Pau_Cin_Hau), Phags_Pa (Phags_Pa), Phoenician (Phoenician), Psalter_Pahlavi (Psalter_Pahlavi),Rejang (Rejang), Runic (Runic),Samaritan (Samaritan), Saurashtra (Saurashtra), Sharada (Sharada), Shavian (Shavian), Siddham (Siddham), Sign_Writing (Sign_Writing), Sinhala (Sinhala), Sogdian (Sogdian), Sora_Sompeng (Sora_Sompeng), Soyombo (Soyombo), Sundanese (Sundanese), Syloti_Nagri (Syloti_Nagri), Syriac (Syriac),Tagalog (Tagalog), Tagbanwa (Tagbanwa), Tai_Le (Tai_Le), Tai_Tham (Tai_Tham), Tai_Viet (Tai_Viet), Takri (Takri), Tamil (Tamil), Tangut (Tangut), Telugu (Telugu), Thaana (Thaana), Thai (Thai), Tibetan (Tibetan), Tifinagh (Tifinagh), Tirhuta (Tirhuta),Ugaritic (Ugaritic), Unknown (Unknown),Vai (Vai),Wancho (Wancho),Warang_Citi (Warang_Citi),Yi (Yi),Zanabazar_Square (Zanabazar_Square)We can very easily make a single enum value for each one, and programmatically generate anintersect() function that can calculate intersections. This would be faster.
(For performance it would also probably be worth only running these checks on non-ascii identifiers)
Metadata
Metadata
Assignees
Labels
No labels