NotificationsYou must be signed in to change notification settings
Fork6
Star31

Commitec0a69e

committed

Extend the default rules file for contrib/unaccent with Vietnamese letters.

Improve generate_unaccent_rules.py to handle composed characters whose baseis another composed character rather than a plain letter. The net effectof this is to add a bunch of multi-accented Vietnamese characters tounaccent.rules.Original complaint from Kha Nguyen, diagnosis of the script's shortcomingby Thomas Munro.Dang Minh Huong and Michael PaquierDiscussion:https://postgr.es/m/CALo3sF6EC8cy1F2JUz=GRf5h4LMUJTaG3qpdoiLrNbWEXL-tRg@mail.gmail.com

1 parent2b74303 commitec0a69eCopy full SHA for ec0a69e

File tree

2 files changed

+145

-8

lines changed

contrib/unaccent
- generate_unaccent_rules.py
- unaccent.rules

2 files changed

+145

-8

lines changed

`‎contrib/unaccent/generate_unaccent_rules.py`

Lines changed: 31 additions & 8 deletions

Original file line number	Diff line number	Diff line change
`@@ -48,24 +48,47 @@ def is_mark(codepoint):`
`48`	`48`	`returncodepoint.general_categoryin ("Mn","Me","Mc")`
`49`	`49`
`50`	`50`	`defis_letter_with_marks(codepoint,table):`
`51`		`-"""Returns true forplainletters combined with one or more marks."""`
	`51`	`+"""Returns true for letters combined with one or more marks."""`
`52`	`52`	`# See http://www.unicode.org/reports/tr44/tr44-14.html#General_Category_Values`
`53`		`-returnlen(codepoint.combining_ids)>1and \`
`54`		`-is_plain_letter(table[codepoint.combining_ids[0]])and \`
`55`		`-all(is_mark(table[i])foriincodepoint.combining_ids[1:])`
	`53`	`+`
	`54`	`+# Letter may have no combining characters, in which case it has`
	`55`	`+# no marks.`
	`56`	`+iflen(codepoint.combining_ids)==1:`
	`57`	`+returnFalse`
	`58`	`+`
	`59`	`+# A letter without diacritical marks has none of them.`
	`60`	`+ifany(is_mark(table[i])foriincodepoint.combining_ids[1:])isFalse:`
	`61`	`+returnFalse`
	`62`	`+`
	`63`	`+# Check if the base letter of this letter has marks.`
	`64`	`+codepoint_base=codepoint.combining_ids[0]`
	`65`	`+if (is_plain_letter(table[codepoint_base])isFalseand \`
	`66`	`+is_letter_with_marks(table[codepoint_base],table)isFalse):`
	`67`	`+returnFalse`
	`68`	`+`
	`69`	`+returnTrue`
`56`	`70`
`57`	`71`	`defis_letter(codepoint,table):`
`58`	`72`	`"""Return true for letter with or without diacritical marks."""`
`59`	`73`	`returnis_plain_letter(codepoint)oris_letter_with_marks(codepoint,table)`
`60`	`74`
`61`	`75`	`defget_plain_letter(codepoint,table):`
`62`		`-"""Return the base codepoint without marks."""`
	`76`	`+"""Return the base codepoint without marks. If this codepoint has more`
	`77`	`+ than one combining character, do a recursive lookup on the table to`
	`78`	`+ find out its plain base letter."""`
`63`	`79`	`ifis_letter_with_marks(codepoint,table):`
`64`		`-returntable[codepoint.combining_ids[0]]`
	`80`	`+iflen(table[codepoint.combining_ids[0]].combining_ids)>1:`
	`81`	`+returnget_plain_letter(table[codepoint.combining_ids[0]],table)`
	`82`	`+elifis_plain_letter(table[codepoint.combining_ids[0]]):`
	`83`	`+returntable[codepoint.combining_ids[0]]`
	`84`	`+`
	`85`	`+# Should not come here`
	`86`	`+assert(False)`
`65`	`87`	`elifis_plain_letter(codepoint):`
`66`	`88`	`returncodepoint`
`67`		`-else:`
`68`		`-raise"mu"`
	`89`	`+`
	`90`	`+# Should not come here`
	`91`	`+assert(False)`
`69`	`92`
`70`	`93`	`defis_ligature(codepoint,table):`
`71`	`94`	`"""Return true for letters combined with letters."""`

`‎contrib/unaccent/unaccent.rules`

Lines changed: 114 additions & 0 deletions

Original file line number	Diff line number	Diff line change
`@@ -254,6 +254,18 @@`
`254`	`254`	`ǒo`
`255`	`255`	`ǓU`
`256`	`256`	`ǔu`
	`257`	`+ǕU`
	`258`	`+ǖu`
	`259`	`+ǗU`
	`260`	`+ǘu`
	`261`	`+ǙU`
	`262`	`+ǚu`
	`263`	`+ǛU`
	`264`	`+ǜu`
	`265`	`+ǞA`
	`266`	`+ǟa`
	`267`	`+ǠA`
	`268`	`+ǡa`
`257`	`269`	`ǤG`
`258`	`270`	`ǥg`
`259`	`271`	`ǦG`
`@@ -262,6 +274,8 @@`
`262`	`274`	`ǩk`
`263`	`275`	`ǪO`
`264`	`276`	`ǫo`
	`277`	`+ǬO`
	`278`	`+ǭo`
`265`	`279`	`ǰj`
`266`	`280`	`ǱDZ`
`267`	`281`	`ǲDz`
`@@ -270,6 +284,8 @@`
`270`	`284`	`ǵg`
`271`	`285`	`ǸN`
`272`	`286`	`ǹn`
	`287`	`+ǺA`
	`288`	`+ǻa`
`273`	`289`	`ȀA`
`274`	`290`	`ȁa`
`275`	`291`	`ȂA`
`@@ -307,8 +323,14 @@`
`307`	`323`	`ȧa`
`308`	`324`	`ȨE`
`309`	`325`	`ȩe`
	`326`	`+ȪO`
	`327`	`+ȫo`
	`328`	`+ȬO`
	`329`	`+ȭo`
`310`	`330`	`ȮO`
`311`	`331`	`ȯo`
	`332`	`+ȰO`
	`333`	`+ȱo`
`312`	`334`	`ȲY`
`313`	`335`	`ȳy`
`314`	`336`	`ȴl`
`@@ -441,6 +463,8 @@`
`441`	`463`	`ḅb`
`442`	`464`	`ḆB`
`443`	`465`	`ḇb`
	`466`	`+ḈC`
	`467`	`+ḉc`
`444`	`468`	`ḊD`
`445`	`469`	`ḋd`
`446`	`470`	`ḌD`
`@@ -451,10 +475,16 @@`
`451`	`475`	`ḑd`
`452`	`476`	`ḒD`
`453`	`477`	`ḓd`
	`478`	`+ḔE`
	`479`	`+ḕe`
	`480`	`+ḖE`
	`481`	`+ḗe`
`454`	`482`	`ḘE`
`455`	`483`	`ḙe`
`456`	`484`	`ḚE`
`457`	`485`	`ḛe`
	`486`	`+ḜE`
	`487`	`+ḝe`
`458`	`488`	`ḞF`
`459`	`489`	`ḟf`
`460`	`490`	`ḠG`
`@@ -471,6 +501,8 @@`
`471`	`501`	`ḫh`
`472`	`502`	`ḬI`
`473`	`503`	`ḭi`
	`504`	`+ḮI`
	`505`	`+ḯi`
`474`	`506`	`ḰK`
`475`	`507`	`ḱk`
`476`	`508`	`ḲK`
`@@ -479,6 +511,8 @@`
`479`	`511`	`ḵk`
`480`	`512`	`ḶL`
`481`	`513`	`ḷl`
	`514`	`+ḸL`
	`515`	`+ḹl`
`482`	`516`	`ḺL`
`483`	`517`	`ḻl`
`484`	`518`	`ḼL`
`@@ -497,6 +531,14 @@`
`497`	`531`	`ṉn`
`498`	`532`	`ṊN`
`499`	`533`	`ṋn`
	`534`	`+ṌO`
	`535`	`+ṍo`
	`536`	`+ṎO`
	`537`	`+ṏo`
	`538`	`+ṐO`
	`539`	`+ṑo`
	`540`	`+ṒO`
	`541`	`+ṓo`
`500`	`542`	`ṔP`
`501`	`543`	`ṕp`
`502`	`544`	`ṖP`
`@@ -505,12 +547,20 @@`
`505`	`547`	`ṙr`
`506`	`548`	`ṚR`
`507`	`549`	`ṛr`
	`550`	`+ṜR`
	`551`	`+ṝr`
`508`	`552`	`ṞR`
`509`	`553`	`ṟr`
`510`	`554`	`ṠS`
`511`	`555`	`ṡs`
`512`	`556`	`ṢS`
`513`	`557`	`ṣs`
	`558`	`+ṤS`
	`559`	`+ṥs`
	`560`	`+ṦS`
	`561`	`+ṧs`
	`562`	`+ṨS`
	`563`	`+ṩs`
`514`	`564`	`ṪT`
`515`	`565`	`ṫt`
`516`	`566`	`ṬT`
`@@ -525,6 +575,10 @@`
`525`	`575`	`ṵu`
`526`	`576`	`ṶU`
`527`	`577`	`ṷu`
	`578`	`+ṸU`
	`579`	`+ṹu`
	`580`	`+ṺU`
	`581`	`+ṻu`
`528`	`582`	`ṼV`
`529`	`583`	`ṽv`
`530`	`584`	`ṾV`
`@@ -563,12 +617,42 @@`
`563`	`617`	`ạa`
`564`	`618`	`ẢA`
`565`	`619`	`ảa`
	`620`	`+ẤA`
	`621`	`+ấa`
	`622`	`+ẦA`
	`623`	`+ầa`
	`624`	`+ẨA`
	`625`	`+ẩa`
	`626`	`+ẪA`
	`627`	`+ẫa`
	`628`	`+ẬA`
	`629`	`+ậa`
	`630`	`+ẮA`
	`631`	`+ắa`
	`632`	`+ẰA`
	`633`	`+ằa`
	`634`	`+ẲA`
	`635`	`+ẳa`
	`636`	`+ẴA`
	`637`	`+ẵa`
	`638`	`+ẶA`
	`639`	`+ặa`
`566`	`640`	`ẸE`
`567`	`641`	`ẹe`
`568`	`642`	`ẺE`
`569`	`643`	`ẻe`
`570`	`644`	`ẼE`
`571`	`645`	`ẽe`
	`646`	`+ẾE`
	`647`	`+ếe`
	`648`	`+ỀE`
	`649`	`+ềe`
	`650`	`+ỂE`
	`651`	`+ểe`
	`652`	`+ỄE`
	`653`	`+ễe`
	`654`	`+ỆE`
	`655`	`+ệe`
`572`	`656`	`ỈI`
`573`	`657`	`ỉi`
`574`	`658`	`ỊI`
`@@ -577,10 +661,40 @@`
`577`	`661`	`ọo`
`578`	`662`	`ỎO`
`579`	`663`	`ỏo`
	`664`	`+ỐO`
	`665`	`+ốo`
	`666`	`+ỒO`
	`667`	`+ồo`
	`668`	`+ỔO`
	`669`	`+ổo`
	`670`	`+ỖO`
	`671`	`+ỗo`
	`672`	`+ỘO`
	`673`	`+ộo`
	`674`	`+ỚO`
	`675`	`+ớo`
	`676`	`+ỜO`
	`677`	`+ờo`
	`678`	`+ỞO`
	`679`	`+ởo`
	`680`	`+ỠO`
	`681`	`+ỡo`
	`682`	`+ỢO`
	`683`	`+ợo`
`580`	`684`	`ỤU`
`581`	`685`	`ụu`
`582`	`686`	`ỦU`
`583`	`687`	`ủu`
	`688`	`+ỨU`
	`689`	`+ứu`
	`690`	`+ỪU`
	`691`	`+ừu`
	`692`	`+ỬU`
	`693`	`+ửu`
	`694`	`+ỮU`
	`695`	`+ữu`
	`696`	`+ỰU`
	`697`	`+ựu`
`584`	`698`	`ỲY`
`585`	`699`	`ỳy`
`586`	`700`	`ỴY`

0 commit comments

Comments

(0)

Movatterモバイル変換

Navigation Menu

Search code, repositories, users, issues, pull requests...

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Commitec0a69e

File tree

2 files changed

2 files changed

`‎contrib/unaccent/generate_unaccent_rules.py`

`‎contrib/unaccent/unaccent.rules`

0 commit comments