Repository files navigation

This project is an attempt to create a pronunciation lexicon covering both English and Chinese wordsin a unified phoneset for ASR applications.

P.S. "CiDian" means "lexicon" in Chinese.

typical use cases in Chinese ASR applications:

你手机上都装了什么 APP ?APPLE 的新 MACBOOK PRO 真漂亮上个月 PRADA 出了款新包包手机开了 GPRS 导航世界杯 H 组小组赛

The unified phoneset should be a simple and precise phoneset that covers both languages. Note that the mapping listed below are heavily based on IPA.

English entries are derived from CMUDict 0.7b, hence we need a mapping from ARPA phoneset to target phoneset.

notes: If you find anything that doesn't make sense in the mapping table, please let me know, thanks

Chinese entries are extracted fromDaCiDian project

Here is a PinYin to IPA mapping from educational prospective:https://resources.allsetlearning.com/chinese/pronunciation/Pinyin_chart

With a few mapping modifications and symbolic adaptations, here is the finalPinYin to target phoneset mapping

There are normally 5 tones in Chinese PinYin system ranging from 0 ~ 4.However there is no tone definition in English. In BigCiDian, Chinese tonal information is retained and merged with untoned English, so the resulting phoneset may contain 6 tonal variation(1 from English and 5 from Chinese):

e.g. for phoneme *ai*1. HI -> h ai2. 哎 -> ai_03. 掰 -> b ai_14. 还 -> h ai_25. 凯 -> k ai_36. 外 -> w ai_4

The final unified bi-lingual phoneset details are listed below:

So overall there are 56 phonemes in the unified phoneset(regardless of tones).

Theoretically some phonemes can be split with smaller granularity(eg. au->a u, ɔi->ɔ i, an->a n ...), hence making the phoneset even more compact. But it is a common practice that larger acoustic modeling units are beneficial for Chinese ASR accuracy, and the existence of decision-tree based state-tying, makes base phoneset size less irrelevant to ASR problem.

I may or may not change the unified phoneset in the future, currently it seems to be sufficient for my purpose.

sh run.sh should give you a ready-to-use bi-lingual ASR lexicon (lexicon.txt), and a phoneset list(phones.list) in project root directory.

To extend the final lexicon with entries of your own interest(say "IPHONE", "华为P30"), you can either:

• add those entries into the very bottom sources(CMUDict and DaCiDian)

or:

• maintain a seperate extension-lexicon, and merge it with main lexicon automatically generated above.

InAISHELL-2 Mandarin ASR task, replacing Chinese lexicon(DaCiDian) with multilingual CN-EN lexicon(BigCiDian), details are showed below:

For DaCiDian, system performance:

----- test -----:%WER 44.39 [ 21986 / 49532, 338 ins, 2085 del, 19563 sub ] exp/mono/decode_test/cer_9_0.0%WER 24.25 [ 12011 / 49532, 393 ins, 792 del, 10826 sub ] exp/tri1/decode_test/cer_12_0.0%WER 22.13 [ 10963 / 49532, 396 ins, 644 del, 9923 sub ] exp/tri2/decode_test/cer_12_0.0%WER 19.29 [ 9555 / 49532, 263 ins, 640 del, 8652 sub ] exp/tri3/decode_test/cer_13_0.5%WER 8.33 [ 4125 / 49532, 84 ins, 192 del, 3849 sub ] exp/chain/tdnn_1a/decode_test/cer_8_0.5

For BigCiDian, system performance:

%WER 43.92 [ 21754 / 49532, 405 ins, 1574 del, 19775 sub ] exp/mono/decode_test/cer_7_0.0%WER 22.54 [ 11163 / 49532, 406 ins, 652 del, 10105 sub ] exp/tri1/decode_test/cer_11_0.0%WER 21.09 [ 10445 / 49532, 377 ins, 609 del, 9459 sub ] exp/tri2/decode_test/cer_12_0.0%WER 18.47 [ 9148 / 49532, 265 ins, 621 del, 8262 sub ] exp/tri3/decode_test/cer_13_0.5%WER 8.22 [ 4072 / 49532, 68 ins, 260 del, 3744 sub ] exp/chain/tdnn_1a/decode_test/cer_9_0.5

Conclusion

• It shows that BigCiDian only gives slightly better results than DaCiDian.
• But more importantly, BigCiDian turns a pure Chinese ASR system to multiligual system, which is pretty much the case in nowadays Chinese ASR applications.

THE END