 | This articleneeds additional citations forverification. Please helpimprove this article byadding citations to reliable sources. Unsourced material may be challenged and removed. Find sources: "Unicode equivalence" – news ·newspapers ·books ·scholar ·JSTOR(November 2014) (Learn how and when to remove this message) |
Unicode equivalence is the specification by theUnicodecharacter encoding standard that some sequences ofcode points represent essentially the same character. This feature was introduced in the standard to allow compatibility with pre-existing standardcharacter sets, which often included similar or identical characters.
Unicode provides two such notions,canonical equivalence and compatibility.Code point sequences that are defined ascanonically equivalent are assumed to have the same appearance and meaning when printed or displayed. For example, the code pointU+006EnLATIN SMALL LETTER N followed byU+0303◌̃COMBINING TILDE is defined by Unicode to be canonically equivalent to the single code pointU+00F1ñLATIN SMALL LETTER N WITH TILDE of theSpanish alphabet). Therefore, those sequences should be displayed in the same manner, should be treated in the same way by applications such asalphabetizing names orsearching, and may be substituted for each other. Similarly, eachHangul syllable block that is encoded as a single character may be equivalently encoded as a combination of a leading conjoining jamo, a vowel conjoining jamo, and, if appropriate, a trailing conjoining jamo.
Sequences that are defined ascompatible are assumed to have possibly distinct appearances, but the same meaning in some contexts. Thus, for example, the code point U+FB00 (thetypographic ligature "ff") is defined to be compatible—but not canonically equivalent—to the sequence U+0066 U+0066 (two Latin "f" letters). Compatible sequences may be treated the same way in some applications (such assorting andindexing), but not in others; and may be substituted for each other in some situations, but not in others. Sequences that are canonically equivalent are also compatible, but the opposite is not necessarily true.
The standard also defines atext normalization procedure, calledUnicode normalization, that replaces equivalent sequences of characters so that any two texts that are equivalent will be reduced to the same sequence of code points, called thenormalization form ornormal form of the original text. For each of the two equivalence notions, Unicode defines two normal forms, onefully composed (where multiple code points are replaced by single points whenever possible), and onefully decomposed (where single points are split into multiple ones).
For compatibility or other reasons, Unicode sometimes assigns two different code points to entities that are essentially the same character. For example, the letter "A with aring diacritic above" is encoded asU+00C5ÅLATIN CAPITAL LETTER A WITH RING ABOVE (a letter of thealphabet inSwedish and several otherlanguages) or asU+212BÅANGSTROM SIGN. Yet the symbol forangstrom is defined to be that Swedish letter, and most other symbols that are letters (such as⟨V⟩ forvolt) do not have a separate code point for each usage. In general, the code points of truly identical characters are defined to be canonically equivalent.
For consistency with some older standards, Unicode provides single code points for many characters that could be viewed as modified forms of other characters (such as U+00F1 for "ñ" or U+00C5 for "Å") or as combinations of two or more characters (such as U+FB00 for the ligature "ff" or U+0132 for theDutch letter "ij")
For consistency with other standards, and for greater flexibility, Unicode also provides codes for many elements that are not used on their own, but are meant instead to modify or combine with a precedingbase character. Examples of thesecombining characters areU+0303◌̃COMBINING TILDE and theJapanese diacriticdakuten (U+3099◌゙COMBINING KATAKANA-HIRAGANA VOICED SOUND MARK).
In the context of Unicode,character composition is the process of replacing the code points of a base letter followed by one or more combining characters into a singleprecomposed character; andcharacter decomposition is the opposite process.
In general, precomposed characters are defined to be canonically equivalent to the sequence of their base letter and subsequent combining diacritic marks, in whatever order these may occur.
| NFC character | A | m | é | l | i | e | |
|---|---|---|---|---|---|---|---|
| NFC code point | 0041 | 006d | 00e9 | 006c | 0069 | 0065 | |
| NFD code point | 0041 | 006d | 0065 | 0301 | 006c | 0069 | 0065 |
| NFD character | A | m | e | ◌́ | l | i | e |
Some scripts regularly use multiple combining marks that do not, in general, interact typographically, and do not have precomposed characters for the combinations. Pairs of such non-interacting marks can be stored in either order. These alternative sequences are, in general, canonically equivalent. The rules that define their sequencing in the canonical form also define whether they are considered to interact.
Unicode provides code points for some characters or groups of characters which are modified only for aesthetic reasons (such asligatures, thehalf-width katakana characters, or thefull-width Latin letters for use in Japanese texts), or to add new semantics without losing the original one (such as digits insubscript orsuperscript positions, or the circled digits (such as "①") inherited from some Japanese fonts). Such a sequence is considered compatible with the sequence of original (individual and unmodified) characters, for the benefit of applications where the appearance and added semantics are not relevant. However, the two sequences are not declared canonically equivalent, since the distinction has some semantic value and affects the rendering of the text.
UTF-8 andUTF-16 (and also some other Unicode encodings) do not allow all possible sequences ofcode units. Different software will convert invalid sequences into Unicode characters using varying rules, some of which are very lossy (e.g., turning all invalid sequences into the same character). This can be considered a form of normalization and can lead to the same difficulties as others.
A text processing software implementing the Unicode string search and comparison functionality must take into account the presence of equivalent code points. In the absence of this feature, users searching for a particular code point sequence would be unable to find other visually indistinguishable glyphs that have a different, but canonically equivalent, code point representation.
Unicode provides standard normalization algorithms that produce a unique (normal) code point sequence for all sequences that are equivalent; the equivalence criteria can be either canonical (NF) or compatibility (NFK). Since one can arbitrarily choose therepresentative element of anequivalence class, multiple canonical forms are possible for each equivalence criterion. Unicode provides two normal forms that are semantically meaningful for each of the two compatibility criteria: the composed forms NFC and NFKC, and the decomposed forms NFD and NFKD. Both the composed and decomposed forms impose acanonical ordering on the code point sequence, which is necessary for the normal forms to be unique.
In order to compare or search Unicode strings, software can use either composed or decomposed forms; this choice does not matter as long as it is the same for all strings involved in a search, comparison, etc. On the other hand, the choice of equivalence criteria can affect search results. For instance, sometypographic ligatures like U+FB03 (ffi),Roman numerals like U+2168 (Ⅸ) and evensubscripts and superscripts, e.g. U+2075 (⁵) have their own Unicode code points. Canonical normalization (NF) does not affect any of these, but compatibility normalization (NFK) will decompose the ffi ligature into the constituent letters, so a search for U+0066 (f) as substring would succeed in an NFKC normalization of U+FB03 but not in NFC normalization of U+FB03. Likewise when searching for the Latin letterI (U+0049) in the precomposed Roman numeralⅨ (U+2168). Similarly, the superscript⁵ (U+2075) is transformed to5 (U+0035) by compatibility mapping.
Transforming superscripts into baseline equivalents may not be appropriate, however, forrich text software, because the superscript information is lost in the process. To allow for this distinction, the Unicode character database containscompatibility formatting tags that provide additional details on the compatibility transformation.[1] In the case of typographic ligatures, this tag is simply<compat>, while for the superscript it is<super>. Rich text standards likeHTML take into account the compatibility tags. For instance, HTML uses its own markup to position a U+0035 in a superscript position.[2]
The four Unicode normalization forms and the algorithms (transformations) for obtaining them are listed in the table below.
| NFD Normalization Form Canonical Decomposition | Characters are decomposed by canonical equivalence, and multiple combining characters are arranged in a specific order. |
| NFC Normalization Form Canonical Composition | Characters are decomposed and then recomposed by canonical equivalence. |
| NFKD Normalization Form Compatibility Decomposition | Characters are decomposed by compatibility, and multiple combining characters are arranged in a specific order. |
| NFKC Normalization Form Compatibility Composition | Characters are decomposed by compatibility, then recomposed by canonical equivalence. |
All these algorithms areidempotent transformations, meaning that a string that is already in one of these normalized forms will not be modified if processed again by the same algorithm.
The normal forms are notclosed under stringconcatenation.[3] For defective Unicode strings starting with a Hangul vowel or trailingconjoining jamo, concatenation can break Composition.
However, they are notinjective (they map different original glyphs and sequences to the same normalized sequence) and thus also notbijective (cannot be restored). For example, the distinct Unicode strings "U+212B" (the angstrom sign "Å") and "U+00C5" (the Swedish letter "Å") are both expanded by NFD (or NFKD) into the sequence "U+0041 U+030A" (Latin letter "A" and combiningring above "°") which is then reduced by NFC (or NFKC) to "U+00C5" (the Swedish letter "Å").
A single character (other than a Hangul syllable block) that will get replaced by another under normalization can be identified in the Unicode tables for having a non-empty compatibility field but lacking a compatibility tag.
The canonical ordering is mainly concerned with the ordering of a sequence of combining characters. For the examples in this section we assume these characters to bediacritics, even though in general some diacritics are not combining characters, and some combining characters are not diacritics.
Unicode assigns each character acombining class, which is identified by a numerical value. Non-combining characters have class number 0, while combining characters have a positive combining class value. To obtain the canonical ordering, every substring of characters having non-zero combining class value must be sorted by the combining class value using astable sorting algorithm. Stable sorting is required because combining characters with the same class value are assumed to interact typographically, thus the two possible orders arenot considered equivalent.
For example, the character U+1EBF (ế), used inVietnamese, has both an acute and a circumflex accent. Its canonical decomposition is the three-character sequence U+0065 (e) U+0302 (circumflex accent) U+0301 (acute accent). The combining classes for the two accents are both 230, thus U+1EBF is not equivalent to U+0065 U+0301 U+0302.
Since not all combining sequences have a precomposed equivalent (the last one in the previous example can only be reduced to U+00E9 U+0302), even the normal form NFC is affected by combining characters' behavior.
When two applications share Unicode data, but normalize them differently, errors and data loss can result. In one specific instance,OS X normalized Unicode filenames sent from theNetatalk andSamba file- and printer-sharing software. Netatalk and Samba did not recognize the altered filenames as equivalent to the original, leading to data loss.[4][5] Resolving such an issue is non-trivial, as normalization is not losslessly invertible.
