Unicode® Technical Standard #39

Unicode Security Mechanisms

Summary

Because Unicode contains such a large number of characters andincorporates the varied writing systems of the world, incorrectusage can expose programs or systems to possible security attacks.This document specifies mechanisms that can be used to detectpossible security problems.

Status

This document has been reviewed by Unicode members and other interested parties, and has been approved for publication by the Unicode Consortium. This is a stable document and may be used as reference material or cited as a normative reference by other specifications.

A Unicode Technical Standard (UTS) is an independentspecification. Conformance to the Unicode Standard does not implyconformance to any UTS.

Please submit corrigenda and other comments with the onlinereporting form [Feedback].Related information that is useful in understanding this document isfound in theReferences. For the latestversion of the Unicode Standard, see [Unicode]. For alist of current Unicode Technical Reports, see [Reports]. For moreinformation about versions of the Unicode Standard, see [Versions].

Contents

• 1Introduction
• 2Conformance
• 3Identifier Characters
  • 3.1GeneralSecurity Profile for Identifiers
    • Table 1.Identifier_Status and Identifier_Type
    • 3.1.1Joining Controls
      • 3.1.1.1Limited Contexts for Joining Controls
      • 3.1.1.2Limitations
  • 3.2IDN SecurityProfiles for Identifiers
  • 3.3EmailSecurity Profiles for Identifiers
• 4Confusable Detection
  • 4.1Whole-ScriptConfusables
  • 4.2Mixed-ScriptConfusables
• 5Detection Mechanisms
  • 5.1Mixed-ScriptDetection
    • Table 1a.Mixed Script Examples
  • 5.2Restriction-LevelDetection
  • 5.3Mixed-NumberDetection
  • 5.4Optional Detection
• 6Development Process
  • 6.1Confusables DataCollection
  • 6.2IdentifierModification Data Collection
• 7Data Files
  • Table 2.Data File List
• Migration
  • Table 3.VersionCorrespondence
  • MigratingPersistent Data
  • Version 8.0 Migration
  • Version 7.0 Migration
• Acknowledgments
• References
• Modifications

1Introduction

Unicode Technical Report #36, "Unicode SecurityConsiderations" [UTR36]provides guidelines for detecting and avoiding security problemsconnected with the use of Unicode. This document specifies mechanismsthat are used in that document, and can be used elsewhere. Readersshould be familiar with [UTR36] beforecontinuing. See also the Unicode FAQ onSecurityIssues [FAQSec].

2Conformance

An implementation claiming conformance to this specificationmust do so in conformance to the following clauses:

UTS-39-C1.An implementation claiming to implementtheGeneral Profile for Identifiers shall do so by conforming to eitherUTS-39-C1-1 orUTS-39-C1-2.

UTS-39-C1-1.The Implementation shall be in accordance with the specifications in Section 3.1,General Security Profile for Identifiers, without change.

UTS-39-C1-2.The implementation shall provide a precise list of characters that are added to or removed from the profile, but otherwise be in accordance with the specifications in Section 3.1,General Security Profile for Identifiers.

UTS-39-C1.1.An implementation claiming to implementtheIDN Security Profiles for Identifiers shall do so by conforming to eitherUTS-39-C1.1-1 orUTS-39-C1.1-2.

UTS-39-C1.1-1.The implementation shall be in accordance with the specifications in Section 3.2,IDN Security Profiles for Identifiers for Identifiers, without change.

UTS-39-C1.1-2.The implementation shall provide a precise list of characters that are added to or removed from the profile, but otherwise be in accordance with the specifications in Section 3.2,IDN Security Profiles for Identifiers.

UTS-39-C1.2.An implementation claiming to implement theEmail Security Profiles for Identifiers shall do so by conforming to eitherUTS-39-C1.2-1 orUTS-39-C1.2-2.

UTS-39-C1.2-1.The implementation shall be in accordance with the specifications in Section 3.3,Email Security Profiles for Identifiers, without change.

UTS-39-C1.2-2.The implementation shall provide a precise list of characters that are added to or removed from the profile, but otherwise be in accordance with the specifications in Section 3.3,Email Security Profiles for Identifiers.

UTS-39-C2.An implementation claiming to implement any of the following confusable-detection functions for Identifiers defined in Section 4,Confusable Detection shall do so by conforming to eitherUTS-39-C2-1 orUTS-39-C2-2.

1. X and Y are single-script confusables
2. X and Y are mixed-script confusables
3. X and Y are whole-script confusables
4. X has whole-script confusables in set of scripts S

UTS-39-C2-1.The implementation of the function shall be in accordance with the specifications in Section 4,Confusable Detection, without change.

UTS-39-C2-2.The implementation shall provide a precise list of character mappings that are added to or removed from those provided, but otherwise be in accordance with the specifications in Section 4,Confusable Detection.

UTS-39-C3.An implementation claiming to detect mixed scripts shall do so by conforming to eitherUTS-39-C3-1 orUTS-39-C3-2.

UTS-39-C3-1.The implementation shall be in accordance with the specifications in Section 5.1,Mixed-script Detection, without change.

UTS-39-C3-2.The implementation shall provide a precise description of changes in behavior, but otherwise be in accordance with the specifications in Section 5.1,Mixed-Script Detection.

UTS-39-C4.An implementation claiming to detect Restriction-Levels shall do so by conforming to eitherUTS-39-C4-1 orUTS-39-C4-2.

UTS-39-C4-1.The implementation shall be in accordance with the specifications in Section 5.2,Restriction-Level Detection, without change.

UTS-39-C4-2.The implementation shall provide a precise description of changes in behavior, but otherwise be in accordance with the specifications in Section 5.2,Restriction-Level Detection.

UTS-39-C5.An implementation claiming to detect mixed numbers shall do so by conforming to eitherUTS-39-C5-1 orUTS-39-C5-2.

UTS-39-C5-1.The implementation shall be in accordance with the specifications in Section 5.3,Mixed-Number Detection, without change.

UTS-39-C5-2.The implementation shall provide a precise description of changes in behavior, but otherwise be in accordance with the specifications in Section 5.3,Mixed-Number Detection.

3IdentifierCharacters

Identifiers ("IDs") are strings used in application contextsto refer to specific entities of certain significance in the given application. In agiven application, an identifier will map to at most one specific entity.Many applications have security requirements related to identifiers. A common example is URLs referring to pagesor other resources on the Internet: when a user wishes to access aresource, it is important that the user can be certain what resource theyare interacting with. For example, they need to know that they are interacting with aparticular financial service and not some other entity that is spoofing theintended service for malicious purposes. This illustrates ageneral security concern for identifiers: potential ambiguity of strings.While a machine has no difficulty distinguishing between any two differentcharacter sequences, it could be very difficult for humans torecognize and distinguish identifiers if an application did not limit whichUnicode characters could be in identifiers. The focus of this specification is mitigation of such issues relatedto the security of identifiers.

Deliberately restricting the characters that can be used in identifiers is an important security technique. The exclusion of characters from identifiers does not affect the generaluse of those characters for other purposes, such as for general text in documents. Unicode Standard Annex #31,"Unicode Identifier and Pattern Syntax" [UAX31]provides a recommended method of determining which strings shouldqualify as identifiers. The UAX #31 specification extends the commonpractice of defining identifiers in terms of letters and numbers tothe Unicode repertoire.

That specification also permits other protocols to use that method asa base, and to define a profile that adds or removescharacters. For example, identifiers for specific programminglanguages typically add some characters like "$", andremove others like "-" (because of the use asminus),while IDNA removes "_" (among others)—see UnicodeTechnical Standard #46, "Unicode IDNA CompatibilityProcessing" [UTS46], as well as [IDNA2003], and [IDNA2008].

This document provides for additional identifier profiles forenvironments where security is an issue. These are profiles of theextended identifiers based on properties and specifications of theUnicode Standard [Unicode], including:

• The XID_Start and XID_Continue properties defined in theUnicode Character Database (see [DCore])
• The toCasefold(X) operation defined inChapter3, Conformance of [Unicode]
• The NFKC and NFKD normalizations defined inChapter3, Conformance of [Unicode]

The data files used in defining these profiles follow the UCD FileFormat, which has a semicolon-delimited list of data fieldsassociated with given characters, with each field referenced bynumber. For more details, see [UCDFormat].

3.1General Security Profile forIdentifiers

The files under [idmod] provide data for a profile ofidentifiers in environments where security is at issue. The filescontain a set of characters recommended to be restricted from use.They also contain a small set of characters that are recommended asadditions to the list of characters defined by the XID_Start andXID_Continue properties, because they may be used in identifiers in abroader context than programming identifiers.

The Restricted characters are characters not in common use, andthey can be blocked to further reduce the possibilities for visualconfusion. They include the following:

• characters not in modern use
• characters only used in specialized fields, such asliturgical characters, phonetic letters, and mathematicalletter-like symbols
• characters in limited use by very small communities

The choice of which characters to specify as Restricted starts conservatively, but allows additionsin the future as requirements for characters are refined. For information on handling modificationsover time, see2.10.1, BackwardCompatibility inUnicode Technical Report #36,"Unicode Security Considerations" [UTR36].

An implementation following the General Security Profile does notpermit any characters in \p{Identifier_Status=Restricted}, unless it documents theadditional characters that it does allow. Such documentation can specify characters via properties, such as \p{Identifier_Status=Technical}, or by explicit lists, or by combinations of these. Implementations may also specify that fewer characters are allowed than implied by \p{Identifier_Status=Restricted}; for example, they can restrict characters to only those permitted by [IDNA2008].

Common candidates for suchadditions include characters for scripts listed inTable 7,Limited Use Scripts of [UAX31]. However,characters from these scripts have not been a priority forexamination for confusables or to determine specialized, non-modern,or uncommon-use characters.

Canonical equivalence is applied when testing candidate identifiersfor inclusion ofAllowed characters. For example, supposethe candidate string is the sequence

<u,combining-diaeresis>

The target string would be Allowed ineither of thefollowing 2 situations:

1. u is Allowed and ¨ is Allowed, or
2. ü is Allowed

For details of the format for the [idmod] files, seeSection 7,Data Files.

Table 1.Identifier_Status and Identifier_Type

Note: In Unicode 15.0, the Joiner_Control characters (ZWJ/ZWNJ) have been removed fromIdentifier_Type=Inclusion.They thereby have the propertiesIdentifier_Type=Default_Ignorable andIdentifier_Status=Restricted.Their inclusion in programming language identifier profiles has usability and security implications.

Implementations of the General Profile for Identifiers that wish to retain ZWJ and ZWNJ should declare that they use a modification of the profile perSection 2, Conformance,and should ensure that they implement the restrictions described inSection 3.1.1, Joining Controls.

Identifier_Status and Identifier_Type are properties of characters (code points).SeeUTS #18: Unicode Regular Expressions [UTS18] andUTR #23: The Unicode Character Property Model [UTR23] formore discussion.

For stability considerations, seeMigratingPersistent Data.

There may be multiple reasons for restricting a character; therefore,the Identifier_Type property allows multiple values that correspond withRestricted. For example, some characters have Identifier_Type values ofLimited_Use and Technical. Multiple values are not assigned to characters with strong restrictions: Not_Character, Deprecated, Default_Ignorable, Not_NFKC. For example, if a character is Deprecated, there is little value in also marking it as Uncommon_Use. For the qualifiers on usage, Obsolete, Uncommon_Use and Technical, the distinctions among the Identifier_Type values is not strict andonly one might be given. The important characteristic is the Identifier_Status:whether or not the character is Restricted.

The default Identifier_Type property value should be Uncommon_Use if no other categories apply.

As moreinformation is gathered about characters, this data may change insuccessive versions. That can cause either the Identifier_Statusor Identifier_Type to change for a particular character. Thus users ofthis data should be prepared for changes in successive versions, suchas by having a grandfathering policy in place for previouslysupported characters or registrations. Both Identifier_Statusand Identifier_Type values are to be comparedcase-insensitively and ignoring hyphens and underbars.

Restricted characters should be treated with caution when considering possible use in identifiers,and should be disallowed unless there is good reason to allow them in theenvironment in question. However, the set of Identifier_Status=Allowedcharacters are not typically used as is by implementations. Instead,they are applied as filters to the set of characters C that aresupported by the identifier syntax, generating a new set C′.Typically there are also particular characters or classes ofcharacters from C that are retained asExceptioncharacters.

C′ = (C ∩ {Identifier_Status=Allowed}) ∪Exception

The implementation may simply restrict use of new identifiers to C′,or may apply some other strategy. For example, there might be anappeal process for registrations of ids that contain charactersoutside of C′ (but still inside of C), or in user interfaces forlookup of identifiers, warnings of some kind may be appropriate. Formore information, see [UTR36].

TheException characters would beimplementation-specific. For example, a particular implementationmight extend the default Unicode identifier syntax by addingExceptioncharacters with the Unicode propertyXID_Continue=False,such as “$”, “-”, and “.”. Those characters are specific to thatidentifier syntax, and would be retained even though they are not inthe Identifier_Status=Allowed set. Someimplementations may also wish to add some [CLDR]exemplar characters for particular supported languages that haveunusual characters.

The Identifier_Type=Inclusion characters alreadycontain some characters that are not letters or numbers, but that areused within words in some languages. For example, it is recommendedthat U+00B7 (·) MIDDLE DOT be allowed in identifiers, because it isrequired for Catalan.

The implementation may also apply other restrictions discussedin this document, such as checking for confusable characters or doingmixed-script detection.

3.1.1Joining Controls

Visible distinctionscreated by certain characters excluded by theGeneral Security Profile because their Identifier_Type is Default_Ignorable (particularly theJoin_Controlcharacters) are necessary in certain languages. A blanket exclusionof these characters makes it impossible to create identifiers withthe correct visual appearance for common words or phrases in thoselanguages.

Identifier systems that attempt to provide more naturalrepresentations of terms in "modern, customary usage"should allow these characters in input and display, but limit them tocontexts in which they are necessary. The termmoderncustomary usage includes characters that are in common use innewspapers, journals, lay publications; on street signs; incommercial signage; and as part of common geographic names andcompany names, and so on. It does not include technical or academicusage such as in mathematical expressions, using archaic scripts orwords, or pedagogical use (such as illustration of half-forms orjoining forms in isolation), or liturgical use.

The goals for such a restriction of format characters toparticular contexts are to:

• Allow the use of these characters where required in normaltext
• Exclude as many cases as possible where no visibledistinction results
• Be simple enough to be easily implemented with standardmechanisms such as regular expressions

An implementation following the General Security Profile that allows the additional characters ZWJ and ZWNJ shall only permit them where they satisfy the conditions A1, A2, and B inSection 3.1.1.1,Limited Contexts for Joiner Controls, unless it documents theadditional contexts where it allows them.

More advanced implementations may use script-specific information for more detailed testing. In particular, they can:

1.Disallow joining controls in sequences that meet the conditions of A1, A2, and B, where in common fonts the resulting appearance of the sequence is normally not distinct from appearance in the same sequences with the joining controls removed.

2.Allow joining controls in sequences that don't meet the conditions of A1, A2, and B, where in common fonts the resulting appearance of the sequence is normally distinct from the appearance in the same sequences with the joining controls removed. The following regular expressions describe sequences that typically result in distinct rendering. They use the notation explained below inA1.

/$L ZWNJ $V $L/

/$L ZWJ $V $L/

3.1.1.1Limited Contexts for Joining Controls

An implementation thatattempts to provide more natural representations of terms in "modern, customary usage" should allow thefollowing Join_Control characters in the limited contexts specifiedinA1,A2, andB below.

U+200C ZERO WIDTH NON-JOINER (ZWNJ)
U+200D ZERO WIDTH JOINER(ZWJ)

There are also two global conditions incorporated in each ofA1,A2, andB:

• Script Restriction. In each of the following cases,the specified sequence must only consist of characters from a singlescript (after ignoringCommon andInherited scriptcharacters).
• Normalization.In each of the following cases, thespecified sequence must be in NFC format. (To test an identifierthat is not required to be in NFC, first transform into NFC formatand then test the condition.)

Implementations may also impose tighter restrictions than provided below, in order to eliminate some other circumstances where the characters either have no visual effect or the effect has no semantic importance.

A1. Allow ZWNJ in thefollowing context:

Breaking a cursive connection. That is, in the context basedon the Joining_Type property, consisting of:

• A Left-Joining or Dual-Joining character, followed by zeroor more Transparent characters, followed by a ZWNJ, followed by zeroor more Transparent characters, followed by a Right-Joining orDual-Joining character

This corresponds to the following regular expression (in Perl-stylesyntax):/$LJ $T* ZWNJ $T* $RJ/

Where the character classes like $T could bedefined with Unicode properties(similar to UnicodeSet notation) like this:

$T = \p{Joining_Type=Transparent}
$RJ =[\p{Joining_Type=Dual_Joining}\p{Joining_Type=Right_Joining}]
$LJ = [\p{Joining_Type=Dual_Joining}\p{Joining_Type=Left_Joining}]

For example, consider Farsi <Noon, Alef, Meem, Heh, Alef,Farsi Yeh>. Without a ZWNJ, it translates to "names",as shown in the first row; with a ZWNJ between Heh and Alef, it means"a letter", as shown in the second row ofFigure 1.

Figure 1.Persian Example withZWNJ

Appearance	Code Points	Abbreviated Names
	0646 + 0627 + 0645 + 0647 +0627 + 06CC	NOON + ALEF + MEEM + HEH + ALEF+ FARSI YEH
	0646 + 0627 + 0645 + 0647 +200C + 0627 + 06CC	NOON + ALEF + MEEM + HEH + ZWNJ+ ALEF + FARSI YEH

A2. Allow ZWNJ in thefollowing context:

In a conjunct context. That is, a sequence of the form:

• A Letter, followed by a Virama, followed by a ZWNJ (optionally preceded or followed by certain nonspacing marks), followed by a Letter.

This corresponds to the following regular expression (in Perl-stylesyntax):/$L $M* $V $M₁* ZWNJ $M₁* $L/

Where:

$L = \p{General_Category=Letter}
$V =\p{Canonical_Combining_Class=Virama}
$M = \p{General_Category=Mn}
$M₁ = [\p{General_Category=Mn}&\p{CCC≠0}]

For example, the Malayalam word foreyewitness is shown inFigure2. The form without the ZWNJ in the second row is incorrect in thiscase.

Figure 2.Malayalam Examplewith ZWNJ

Appearance	Code Points	Abbreviated Names
	0D26 + 0D43 + 0D15 + 0D4D +200C + 0D38 + 0D3E + 0D15 + 0D4D + 0D37 + 0D3F	DA + VOWEL SIGN VOCALIC R + KA+ VIRAMA + ZWNJ + SA + VOWEL SIGN AA + KA + VIRAMA + SSA + VOWELSIGN I
	0D26 + 0D43 + 0D15 + 0D4D +0D38 + 0D3E + 0D15 + 0D4D + 0D37 + 0D3F	DA + VOWEL SIGN VOCALIC R + KA+ VIRAMA + SA + VOWEL SIGN AA + KA + VIRAMA + SSA + VOWEL SIGN I

B. Allow ZWJ in thefollowing context:

In a conjunct context.That is, a sequence of the form:

• A Letter, followed by a Virama, followed by a ZWJ (optionally preceded or followed by certain nonspacing marks), and not followed by a character of type Indic_Syllabic_Category=Vowel_Dependent

This corresponds to the following regular expression (in Perl-stylesyntax): /$L $M* $V $M₁* ZWJ (?!$D)/

Where:

$L= \p{General_Category=Letter}
$V =\p{Canonical_Combining_Class=Virama}
$M = \p{General_Category=Mn}
$M₁ = [\p{General_Category=Mn}&\p{CCC≠0}]
$D = \p{Indic_Syllabic_Category=Vowel_Dependent}

For example, the Sinhala word for the country 'Sri Lanka' isshown in the first row ofFigure 3, which uses both a spacecharacter and a ZWJ. Removing the space results in the text shown inthe second row ofFigure 3, which is still legible, butremoving the ZWJ completely modifies the appearance of the'Sri' cluster and results in the unacceptable text appearanceshown in the third row ofFigure 3.

Figure 3.Sinhala Example withZWJ

Appearance	Code Points	Abbreviated Names
	0DC1 + 0DCA + 200D + 0DBB +0DD3 + 0020 + 0DBD + 0D82 + 0D9A + 0DCF	SHA + VIRAMA + ZWJ + RA + VOWELSIGN II + SPACE + LA + ANUSVARA + KA + VOWEL SIGN AA
	0DC1 + 0DCA + 200D + 0DBB +0DD3 + 0DBD + 0D82 + 0D9A + 0DCF	SHA + VIRAMA + ZWJ + RA + VOWELSIGN II + LA + ANUSVARA + KA + VOWEL SIGN AA
	0DC1 + 0DCA + 0DBB + 0DD3 +0020 + 0DBD + 0D82 + 0D9A + 0DCF	SHA + VIRAMA + RA + VOWEL SIGNII + SPACE + LA + ANUSVARA + KA + VOWEL SIGN AA

Note: The restrictions inA1,A2, andBare similar to the CONTEXTJ rules defined inAppendix A, Contextual Rules Registry,inThe Unicode Code Points and Internationalized Domain Names for Applications (IDNA)[IDNA2008].

3.1.1.2Limitations

While the restrictions inA1,A2, andB greatlylimit visual confusability, they do not prevent it. For example,because Tamil only uses a Join_Control character in one specificcase, most of the sequences these rules allow in Tamil are, in fact,visually confusable. Therefore based on their knowledge of the scriptconcerned, implementations may choose to have tighter restrictionsthan specified inSection 3.1.1.2,Limited Contexts for Joining Controls—for example, by explicitly providing for the exceptional sequence, while otherwise disallowing the joiner in context.

There are also cases where a joiner preceding avirama makes a visual distinction in some scripts. It is currentlyunclear whether this distinction is important enough in identifiersto warrant retention of a joiner. For more information, see UTR #36:Unicode Security Considerations [UTR36].

Performance. Parsing identifiers can be aperformance-sensitive task. However, these characters are quite rarein practice, thus the regular expressions (or equivalent processing)only rarely would need to be invoked. Thus these tests should not addany significant performance cost overall.

3.2IDNSecurity Profiles for Identifiers

Version 1 of this document defined operations and data that apply to[IDNA2003], which has been superseded by [IDNA2008] and Unicode TechnicalStandard #46, "Unicode IDNA CompatibilityProcessing" [UTS46]. The identifiermodification data can be applied to whichever specification of IDNAis being used. For more information, see the [IDNFAQ].

However, implementations can claim conformance to other features ofthis document as applied to domain names, such asRestriction Levels.

3.3Email Security Profiles for Identifiers

TheSMTP Extension for Internationalized Email provides for specifications of internationalized email addresses [EAI]. However, it does not provide for testing those addresses for security issues. This section provides an email security profile that may be used for that. It can be applied for different purposes, such as:

1. When an email address is registered, flag anything thatdoes not meet the profile:
   • Either forbid the registration, or
   • Allow for an appeals process.
2. When an email address is detected in linkification of plaintext:
   • Do not linkify if the identifier does not meetthe profile.
3. When an email address is displayed in incoming email:
   • Flag it as suspicious with a wavy underline, if itdoes not meet the profile.
   • Filter characters from the quoted-string-part to preventdisplay problems.

This profile does not exclude characters fromEAI. Instead, it provides a profile that can be used for registration, linkification,and notification. The goal is to flag addresses that are structurally unsound or contain unexpected detritus.

An email address is formed from three main parts. (There are more elements of an email address, but these are the ones for which Unicode security is important.) For example:

"Joey" <joe31834@gmail.com>

• Thedomain-part is "gmail.com"
• Thelocal-part is "joe31834"
• Thequoted-string-part is "Joey"

To meet the requirements of theEmail Security Profiles for Identifiers section of this specification, an identifier must satisfy the following conditions for the specified <restriction level>.

Domain-Part

The domain-part of an email address must satisfySection 3.2,IDNSecurity Profiles for Identifiers, and satisfy the conformanceclauses of [UTS46].

Local-Part

The local-part of an email address must satisfy all the following conditions:

1. It must be in NFKC format
2. It must have level = <restriction level> or less, fromRestriction_Level_Detection
3. It must not have mixed number systems according toMixed_Number_Detection
4. It must satisfydot-atom-text fromRFC 5322 §3.2.3, whereatext is extended as follows:

Where C ≤ U+007F, C is defined as in§3.2.3. (That is, C ∈ [!#-'*+\-/-9=?A-Z\^-~]. This list copies what is already in §3.2.3, and followsHTML5 for ASCII.)

Where C > U+007F, both of the following conditions are true:

1. C has Identifier_Status=Allowed fromGeneral Security Profile
2. If C is the first character, it must be XID_Start fromDefault Identifier_Syntax in [UAX31]

Note that inRFC 5322 §3.2.3:

dot-atom-text   =   1*atext *("." 1*atext)

That is, dots can also occur in the local-part, but not leading, trailing, or two in a row. In more conventional regex syntax, this would be:

dot-atom-text   =   atext+ ("." atext+)*

Note that bidirectional controls and other format characters are specifically disallowed in the local-part, according to the above.

Quoted-String-Part

The quoted-string-part of an email address must satisfy the following conditions:

1. It must be in NFC.
2. It must not contain any stateful bidirectional format characters.
   • That is, no [:bidicontrol:] except for the LRM, RLM, and ALM, since the bidirectional controls could influence the ordering of characters outside the quotes.
3. It must not contain more than four nonspacing marks in a row, and no sequence of two of the same nonspacing marks.
4. It may contain mixed scripts, symbols (including emoji), and so on.

Other Issues

The restrictions above are insufficient toprevent bidirectional-reordering that could intermix the quoted-string-partwith the local-part or the domain-part in display. To prevent that,implementations could use bidirectional isolates (or equivalent) around theeach of these parts in display.

Implementations may also want to use other checks, such as for confusability, or services such as Safe Browsing.

A serious practical issue is that clients do not know what theidentity rules are for any particular email server: that is, when twoemail addresses are considered equivalent. For example, aremark@macchiato.comandMark@macchiato.com treated the same by the server?Unfortunately, there is no way to query a server to seewhat identity rules it follows. One of the techniques used to deal withthis problem is having whitelists of email providers indicating which of them are case-insensitive, dot-insensitive, or both.

4ConfusableDetection

The data in [confusables] provide amechanism for determining when two strings are visually confusable.The data in these files may be refined and extended over time. Forinformation on handling modifications over time, seeSection2.10.1, Backward Compatibility in Unicode Technical Report #36,"Unicode Security Considerations" [UTR36]and theMigration section of this document.

Collection of data for detecting gatekeeper-confusable strings is notcurrently a goal for the confusable detection mechanism in thisdocument. For more information, seeSection 2, VisualSecurity Issues in [UTR36].

The data provides a mapping from source characters to their prototypes. A prototype should be thought of as a sequence of one or more classes of symbols, where each class has an exemplar character. For example, the character U+0153 (œ), LATIN SMALL LIGATURE OE, has a prototype consisting of two symbol classes: the one with exemplar character U+006F (o), and the one with exemplar character U+0065 (e). If an input character does not have a prototype explicitly defined in the data file, the prototype is assumed to consist of the class of symbols with the input character as the exemplar character.

For an input string X, defineskeleton(X) to be the following transformation on the string:

1. Convert X to NFD format, as described in [UAX15].
2. Remove any characters in X that have the property Default_Ignorable_Code_Point.
3. Concatenate the prototypes for each character in X according to the specified data, producing a string of exemplar characters.
4. Reapply NFD.

The strings X and Y are defined to beconfusable if and only if skeleton(X) = skeleton(Y). This is abbreviated as X ≅ Y.

This mechanism imposes transitivity on the data, so if X ≅ Y and Y ≅ Z, then X ≅ Z. It is possible to provide a more sophisticated confusable detection, by providing a metric between given characters, indicating their "closeness." However, that is computationally much more expensive, and requires more sophisticated data, so at this point in time the simpler mechanism has been chosen. That means that in some cases the test may be overly inclusive.

Note: The stringsskeleton(X) andskeleton(Y)arenot intended for display, storage or transmission.They should be thought of as an intermediate processing form,similar to a hashcode. The exemplar characters arenot guaranteed to be identifier characters.

Note: Some implementations of confusable detection outside Unicode use different terminology.In particular, in the ICANN Root Zone Label Generation Rules [RZLGR5], the termvariant of X is used for a property similar toconfusable with X, and the termindex variant is used for the equivalent ofskeleton.

For an input string X and a direction 𝑑 ∈ {RTL, LTR, FS}, define bidiSkeleton(𝑑, X) to be the following transformation on the string:

1. Reorder the code points in X for display by applying the rules of the Unicode Bidirectional Algorithm [UAX9] up to and including L2, treating X in isolation; if 𝑑≠FS, apply protocol HL1 to set the paragraph level to 1 if 𝑑=RTL, and to 0 if 𝑑=LTR; this yields the reordered sequence of characters R.
2. Apply rule L3 of the UBA: move combining marks after their base in R; this yields the sequence R′.
3. Replace any character whose glyph would be mirrored by rule L4 of the UBA by the value of its Bidi_Mirroring_Glyph property, yielding R″.
4. bidiSkeleton(𝑑, X) is then skeleton(R″).

The strings X and Y are defined to be 𝑑-confusable if and only if bidiSkeleton(𝑑, X) = bidiSkeleton(𝑑, Y). This is abbreviated as X ≒ Y (𝑑).

Like confusability, 𝑑-confusability is an equivalence relation; in particular, it is transitive: if X ≒ Y (𝑑) and Y ≒ Z (𝑑), then X ≒ Z (𝑑).

Note: The operationskeleton may change the Bidi_Class of characters, so it does not commute with the reordering and mirroring steps, and needs to be performed after them.

Example: The sequences of code points S₁ and S₂ are LTR-confusable:

S₁ ≔ "A1<שׂ" = (LATIN CAPITAL LETTER A, DIGIT ONE, LESS-THAN SIGN, HEBREW LETTER SHIN, HEBREW POINT SIN DOT)
S₂ ≔ "Αשֺ>1" = (GREEK CAPITAL LETTER ALPHA, HEBREW LETTER SHIN, HEBREW POINT HOLAM HASER FOR VAV, GREATER-THAN SIGN, DIGIT ONE)

Computation of bidiSkeleton(LTR, S₁):

R₁ = (LATIN CAPITAL LETTER A, DIGIT ONE, LESS-THAN SIGN, HEBREW POINT SIN DOT, HEBREW LETTER SHIN)
R′₁ = (LATIN CAPITAL LETTER A, DIGIT ONE, LESS-THAN SIGN, HEBREW LETTER SHIN, HEBREW POINT SIN DOT)
R″₁ = (LATIN CAPITAL LETTER A, DIGIT ONE, LESS-THAN SIGN, HEBREW LETTER SHIN, HEBREW POINT SIN DOT)
bidiskeleton(LTR, S₁) = skeleton(R″₁) = (LATIN CAPITAL LETTER A, LATIN SMALL LETTER L, LESS-THAN SIGN, HEBREW LETTER SHIN, COMBINING DOT ABOVE)

Computation of bidiSkeleton(LTR, S₂):

R₂ = (GREEK CAPITAL LETTER ALPHA, DIGIT ONE, GREATER-THAN SIGN, HEBREW POINT HOLAM HASER FOR VAV, HEBREW LETTER SHIN)
R′₂ = (GREEK CAPITAL LETTER ALPHA, DIGIT ONE, GREATER-THAN SIGN, HEBREW LETTER SHIN, HEBREW POINT HOLAM HASER FOR VAV)
R″₂ = (GREEK CAPITAL LETTER ALPHA, DIGIT ONE, LESS-THAN SIGN, HEBREW LETTER SHIN, HEBREW POINT HOLAM HASER FOR VAV)
bidiskeleton(LTR, S₂) = skeleton(R″₂) = (LATIN CAPITAL LETTER A, LATIN SMALL LETTER L, LESS-THAN SIGN, HEBREW LETTER SHIN, COMBINING DOT ABOVE)

Note that these sequences are not RTL-confusable; indeed in a right-to-left paragraph, the strings look distinct:

S₁ = "A1<שׂ"
S₂ = "Αשֺ>1"

LTR, and RTL, and FS confusability should be used when it is inappropriate to enforce that strings be single-script,or at least single-directionality; this is the case in programming language identifiers.SeeSection 5.1, Confusability Mitigation Diagnostics, inUnicode Technical Standard #55, Unicode Source Code Handling [UTS55].

Bidirectional confusability is costlier to check than confusability, as the bidirectional algorithm must be applied.However, a fast path can be used: if 𝑑=LTR and X has no characters with bidi classes R or AL, bidiSkeleton(X) = skeleton(X).

Further, if the strings are known not to contain explicit directional formatting characters (as is the case for UAX31-R1 Default Identifiersdefined inUnicode Standard Annex #31, Identifiers and Syntax [UAX31]), the algorithm can be drastically simplified,as the X rules are trivial, obviating the need for the directional status stack of the Unicode Bidirectional Algorithm.The highest possible resolved level is then 2; seeTable 5, Resolving Implicit Levels,inUnicode Standard Annex #9, Unicode Bidirectional Algorithm [UAX9].

Note: As is the case for skeleton, the strings bidiSkeleton(d, X) and bidiSkeleton(d, Y) are not intended for display, storage or transmission.

Definitions

Confusables are divided into three classes: single-script confusables, mixed-script confusables, and whole-script confusables, defined below. All confusables are either a single-script confusable or a mixed-script confusable, but not both. All whole-script confusables are also mixed-script confusables.

The definitions of these three classes of confusables depend on the definitions ofresolved script set andsingle-script, which are provided inSection 5,Mixed-ScriptDetection.

X and Y aresingle-script confusables ifand only if they are confusable, and their resolved script sets have at least one element in common.

Examples:“ljeto” and “ljeto” in Latin (the Croatian word for “summer”), where the first word uses only four codepoints, the first of which is U+01C9 (lj) LATIN SMALL LETTER LJ.

X and Y aremixed-script confusables ifand only if they are confusable but their resolved script sets have no elements in common.

Examples: "paypal" and "pаypаl", where thesecond word has the characterU+0430 ( а )CYRILLIC SMALL LETTER A.

X and Y arewhole-script confusables ifand only if they aremixed-script confusables, and each of them is asingle-script string.

Example: "scope" in Latin and "ѕсоре" in Cyrillic.

As noted in Section 5, the resolved script set ignores characters with Script_Extensions {Common} and {Inherited} and augments characters with CJK scripts with their respective writing systems. Characters with the Script_Extension property values COMMON orINHERITED are ignored when testing for differences in script.

Data File Format

Each line in the data file has the following format: Field 1 is the source, Field 2 is the target, and Field 3 is obsolete, always containing the letters “MA” for backwards compatibility. For example:

0441 ; 0063 ; MA # ( с → c ) CYRILLIC SMALL LETTER ES → LATIN SMALLLETTER C #

2CA5 ; 0063 ; MA # ( ⲥ → c ) COPTIC SMALL LETTER SIMA → LATINSMALL LETTER C # →ϲ→

Everything after the # is a comment and is purely informative. Aasterisk after the comment indicates that the character is not an XIDcharacter [UAX31]. The comments provide thecharacter names.

Implementations that use the confusable data do not have torecursively apply the mappings, because the transforms areidempotent. That is,

skeleton(skeleton(X)) = skeleton(X)

If the data was derived via transitivity, there isan extra comment at the end. For instance, in the above example thederivation was:

1. ⲥ (U+2CA5 COPTIC SMALL LETTER SIMA)
2. → ϲ (U+03F2 GREEK LUNATE SIGMA SYMBOL)
3. → c (U+0063 LATIN SMALL LETTER C)

To reduce security risks, it is advised that identifiers usecasefolded forms, thus eliminating uppercase variants where possible.

The data may change between versions. Even where the data is thesame, the order of lines in the files may change between versions.For more information, seeMigration.

Note: Due to production problems, versionsbefore 7.0 did not maintain idempotency in all cases. For moreinformation, seeMigration.

4.1Whole-Script Confusables

For some applications, it is useful to determine if a given input string has any whole-script confusable. For example, the identifier "ѕсоре" using Cyrillic characters would pass the single-script test described inSection 5.2, Restriction-Level Detection, even though it is likely to be a spoof attempt.

It is possible to determine whether a single-script string X has a whole-script confusable:

1. Consider Q, the set of all strings that are confusable with X.
2. Remove all strings from Q whose resolved script set intersects with the resolved script set of X.
3. If Q is nonempty and contains any single-script string, return TRUE.
4. Otherwise, return FALSE.

The logical description above can be used for a reference implementation for testing, but is not particularly efficient. A production implementation can be optimized as long as it produces the same results.

Note that the confusables data include a large number of mappings between Latin and Cyrillic text. For this reason, the above algorithm is likely to flag a large number of legitimate strings written in Latin or Cyrillic as potential whole-script confusables. To effectively use whole-script confusables, it is often useful to determine both whether a string has a whole-script confusable, andwhich scripts those whole-script confusables have.

This information can be used, for example, to distinguish between reasonable versus suspect whole-script confusables. Consider the Latin-script domain-name label “circle”. It would be appropriate to have that in the domain name “circle.com”. It would also be appropriate to have the Cyrillic confusable “сігсӀе” in the Cyrillic domain name “сігсӀе.рф”. However, a browser may want to alert the user to possible spoofs if the Cyrillic “сігсӀе” is used with .com or the Latin “circle” is used with .рф.

The process of determining suspect usage of whole-script confusables is more complicated than simply looking at the scripts of the labels in a domain name. For example, it can be perfectly legitimate to have scripts in a SLD (second level domain) not be the same as scripts in a TLD (top-level domain), such as:

• Cyrillic labels in a domain name with a TLD of .ru or .рф
• Chinese labels in a domain name with a TLD of .com.au or .com
• Cyrillic labelsthat aren’t confusable with Latin with a TLD of .com.au or .com

The following high-level algorithm can be used to determine all scripts that contain a whole-script confusable with a string X:

1. Consider Q, the set of all strings confusable with X.
2. Remove all strings from Q whose resolved script set is ∅ orALL (that is, keep only single-script strings plus those with characters only in Common).
3. Take the union of the resolved script sets of all strings remaining in Q.

As usual, this algorithm is intended only as a definition; implementations should use an optimized routine that produces the same result.

4.2Mixed-Script Confusables

To determine the existence of a mixed-script confusable, a similar process could be used:

1. Consider Q, the set of all strings that are confusable with X.
2. Remove all strings from Q whose resolved script set intersects with the resolved script set of X.
3. If Q is nonempty, return TRUE.
4. Otherwise, return FALSE.

The logical description above can be used for a reference implementation for testing, but is not particularly efficient. A production implementation can be optimized as long as it produces the same results.

Note that due to the number of mappings provided by the confusables data, the above algorithm is likely to flag a large number of legitimate strings as potential mixed-script confusables.

5DetectionMechanisms

5.1Mixed-ScriptDetection

The Unicode Standard supplies information that can be used fordetermining the script of characters and detecting mixed-script text.The determination of script is according to theUAX #24, Unicode Script Property[UAX24], using data from the Unicode Character Database [UCD].

Define a character'saugmented script set to be a character's Script_Extensions with the following two modifications.

1. Entries for the writing systems containing multiple scripts — Hanb (Han with Bopomofo), Jpan (Japanese), and Kore (Korean) — are added according to the following rules.
   1. If Script_Extensions contains Hani (Han), add Hanb, Jpan, and Kore.
   2. If Script_Extensions contains Hira (Hiragana), add Jpan.
   3. If Script_Extensions contains Kana (Katakana), add Jpan.
   4. If Script_Extensions contains Hang (Hangul), add Kore.
   5. If Script_Extensions contains Bopo (Bopomofo), add Hanb.
2. Sets containing Zyyy (Common) or Zinh (Inherited) are treated asALL, the set of all script values.

The Script_Extensions data is from the Unicode Character Database [UCD]. For more information on the Script_Extensions property and Jpan, Kore, and Hanb, seeUAX #24, Unicode Script Property [UAX24].

Define theresolved script set for a string to be the intersection of the augmented script sets over all characters in the string.

A string is defined to bemixed-script if its resolved script set is empty and defined to besingle-script if its resolved script set is nonempty.

Note: The term “single-script string” may be confusing. It means that there isat least one script in the resolved script set, not that there isonly one. For example, the string “〆切” is single-script, because it hasfour scripts {Hani, Hanb, Jpan, Kore} in its resolved script set.

As well as providing an API to detect whether a stringhas mixed-scripts, is also useful to offer an API that returns those scripts. Look at the examples below.

Table 1a.Mixed Script Examples

A set of scripts is defined tocover a string if the intersection of that set with the augmented script sets of all characters in the string is nonempty; in other words, if every character in the string shares at least one script with the cover set. For example, {Latn, Cyrl} covers "Сirсlе", the third example inTable 1a.

A cover set is defined to beminimal if there is no smaller cover set. For example, {Hira, Hani} covers "〆切", the seventh example inTable 1a, but it is not minimal, since {Hira} also covers the string, and {Hira} is smaller than {Hira, Hani}. Note that minimal cover sets are not unique: a string may have different minimal cover sets.

Typically an API that returns the scripts in a string will return one of the minimal cover sets.

For computational efficiency, a set of script sets (SOSS) can be computed, where the augmented script sets for each character in the string map to one entry in the SOSS. For example, { {Latn}, {Cyrl} } would be the SOSS for "Сirсlе". A set of scripts that covers the SOSS also covers the input string. Likewise, the intersection of all entries of the SOSS will be the input string's resolved script set.

5.2Restriction-Level Detection

Restriction Levels 1-5 are defined here for use in implementations.These place restrictions on the use of identifiers according to theappropriateidentifier profile as specified inSection 3,IdentifierCharacters. The lists of Recommended scripts aretaken fromTable5, Recommended Scripts of [UAX31]. Formore information on the use of Restriction Levels, seeSection2.9, Restriction Levels and Alerts in [UTR36].

For each of the Restriction Levels 1-6, the identifier must be well-formed according to whatever general syntactic constraints are in force, such as the Default Identifier Syntax in [UAX31].

In addition, an application may provide anidentifier profile such as theGeneral Security Profile for Identifiers, which restricts the allowed characters further. For each of the Restriction Levels 1-5, characters in the string must also be in theidentifier profile. Where there is no suchidentifier profile, Levels 5 and 6 are identical.

1. ASCII-Only
   • All characters in the string are in the ASCII range.
2. Single Script
   • The string qualifies as ASCII-Only, or
   • The string issingle-script, according to the definition in Section 5.1.
3. Highly Restrictive
   • The string qualifies as Single Script, or
   • The string iscovered by any of the following sets of scripts, according to the definition in Section 5.1:
     • Latin + Han + Hiragana + Katakana; or equivalently: Latn + Jpan
     • Latin + Han + Bopomofo; or equivalently: Latn + Hanb
     • Latin + Han + Hangul; or equivalently: Latn + Kore
4. Moderately Restrictive
   • The string qualifies as Highly Restrictive, or
   • The string iscovered by Latin and any one other Recommended script, except Cyrillic, Greek
5. Minimally Restrictive
   • There are no restrictions on the set of scripts thatcover the string.
   • The only restrictions are the identifier well-formedness criteria andidentifier profile, allowing arbitrary mixtures of scripts such as Ωmega, Teχ,HλLF-LIFE, Toys-Я-Us.
6. Unrestricted
   • There are no restrictions on the script coverage of the string.
   • The only restrictions are the criteria on identifier well-formedness. Characters may be outside of theidentifier profile.
   • This level is primarily for use in detection APIs, providing return value indicating that the string does not match any of the levels 1-5.

Note that in all levels except ASCII-Only, any character having Script_Extensions {Common} or {Inherited} are allowed in the identifier, as long as those characters meet theidentifier profile requirements.

These levels can be detected by reusing some of the mechanismsof Section 5.1. For a given input string, the Restriction Level isdetermined by the following logical process:

1. If the string contains any characters outside of theIdentifier Profile, returnUnrestricted.
2. If no character in the string is above 0x7F, returnASCII-Only.
3. Compute the string's SOSS according to Section 5.1.
4. If the SOSS is empty or the intersection of all entries in the SOSS is nonempty, returnSingle Script.
5. Remove all the entries from the SOSS that contain Latin.
6. If any of the following sets cover SOSS, returnHighlyRestrictive.
   • {Kore}
   • {Hanb}
   • {Japn}
7. If the intersection of all entries in the SOSS contains any singleRecommendedscript exceptCyrillicor Greek, returnModeratelyRestrictive.
8. Otherwise, returnMinimally Restrictive.

The actual implementation of this algorithm can be optimized;as usual, the specification only depends on the results.

5.3Mixed-NumberDetection

There are three different types of numbers in Unicode. Only numberswith General_Category = Decimal_Numbers (Nd) should be allowed inidentifiers. However, characters from different decimal numbersystems can be easily confused. For example,U+0660 ( ٠ )ARABIC-INDIC DIGIT ZERO can be confused withU+06F0 ( ۰ )EXTENDED ARABIC-INDIC DIGIT ZERO, andU+09EA ( ৪ )BENGALI DIGIT FOUR can be confused withU+0038 ( 8 )DIGIT EIGHT. There are other reasons for disallowing mixed number systems in identifiers, just as there are for mixing scripts.

For a given input string which does not contain non-decimalnumbers, the logical process of detecting mixed numbers is thefollowing:

For each character in the string:

1. Find the decimal number value for that character, if any.
2. Map the value to the unique zero character for that numbersystem.

If there is more than one such zero character, then the stringcontains multiple decimal number systems.

The actual implementation of this algorithm can be optimized; asusual, the specification only depends on the results. The followingJava sample using [ICU] shows how this can be done:

    public UnicodeSet getNumberRepresentatives(String identifier) {
        int cp;
        UnicodeSet numerics = new UnicodeSet();
        for (int i = 0; i < identifier.length(); i += Character.charCount(i)) {
            cp = Character.codePointAt(identifier, i);
            // Store a representative character for each kind of decimal digit
            switch (UCharacter.getType(cp)) {
            case UCharacterCategory.DECIMAL_DIGIT_NUMBER:
                // Just store the zero character as a representative for comparison.
                // Unicode guarantees it is cp - value.
                numerics.add(cp - UCharacter.getNumericValue(cp));
                break;
            case UCharacterCategory.OTHER_NUMBER:
            case UCharacterCategory.LETTER_NUMBER:
                throw new IllegalArgumentException("Should not be in identifiers.");
            }
        }
        return numerics;
    }...    UnicodeSet numerics = getMixedNumbers(String identifier);    if (numerics.size() > 1) reject(identifier, numerics);

5.4OptionalDetection

There are additional enhancements that may be useful in spoofdetection, such as:

1. Check to see that all the characters are in the sets ofexemplar characters for at least one language in the Unicode CommonLocale Data Repository [CLDR].
2. Check for unlikely sequences of combining marks:
   1. Forbid sequences of the same nonspacing mark.
   2. Forbid sequences of more than 4 nonspacing marks (gc=Mn or gc=Me).
   3. Forbid sequences of base character + nonspacing mark that look the same as or confusingly similar to the base character alone (because the nonspacing mark overlays a portion of the base character). An example is U+0069 LOWERCASE LETTER I + U+0307 COMBINING DOT ABOVE.
3. Add support for detecting two distinctsequences that have identical representations. The current data files only handle cases where a single code point is confusable with another code point or sequence. It does not handle cases likeshri, as below.

The characters U+0BB6 TAMIL LETTER SHA and U+0BB8 TAMIL LETTER SA are normally quite distinct. However, they can both be used in the representation of the Tamil wordshri. On some very common platforms, the following sequences result in exactly the same visual appearance:

= ஶ்ரீ

= ஸ்ரீ

6DevelopmentProcess

As discussed in Unicode TechnicalReport #36, "Unicode Security Considerations" [UTR36], confusability among characters cannot bean exact science. There are many factors that make confusability amatter of degree:

• Shapes of characters vary greatly among fonts used torepresent them. The Unicode Standard uses representative glyphs inthe code charts, but font designers are free to create their ownglyphs. Because fonts can easily be created using an arbitrary glyphto represent any Unicode code point, character confusability witharbitrary fonts can never be avoided. For example, one could designa font where the ‘a’ looks like a ‘b’ , ‘c’ like a ‘d’, and so on.
• Writing systems using contextual shaping (such as Arabic and many South Asian systems) introduce even more variation in textrendering. Characters do not really have an abstract shape inisolation and are only rendered as part of cluster of charactersmaking words, expressions, and sentences. It is a fairly commonoccurrence to find the same visual text representation correspondingto very different logical words that can only be recognized bycontext, if at all.
• Font style variants such as italics may introduce aconfusability which does not exist in another style. For example, inthe Cyrillic script, theU+0442 ( т )CYRILLIC SMALL LETTER TE looks like a small caps Latin ‘T’ in normalstyle, while it looks like a small Latin ‘m’ in italic style.

In-script confusability is extremely user-dependent. For example, inthe Latin script, characters with accents or appendices may looksimilar to the unadorned characters for some users, especially ifthey are not familiar with their meaning in a particular language.However, most users will have at least a minimum understanding of therange of characters in their own script, and there are separatemechanisms available to deal with other scripts, as discussed in [UTR36].

As described elsewhere, there are cases where the confusable data maybe different than expected. Sometimes this is because two charactersor two strings may only be confusable in some fonts. In other cases,it is because of transitivity. For example, the dotless and dotted Iare considered equivalent (ı ↔ i), because they look the same whenaccents such as anacute are applied to each. However, forpractical implementation usage, transitivity is sufficientlyimportant that some oddities are accepted.

The data may be enhanced in future versions of thisspecification. For information on handling changes in data overtime, see2.10.1, Backward Compatibility of [UTR36].

6.1Confusables Data Collection

The confusability data was created by collecting a number ofprospective confusables, examining those confusables according to aset of common fonts, and processing the result for transitiveclosure.

The primary goal is to include characters that would be Identifier_Status=Allowedas inTable 1,Identifier_Status and Identifier_Type. Other characters, such as NFKCvariants, are not a primary focus for data collection. However, suchvariants may certainly be included in the data, and may be submittedusing the online forms at [Feedback].

The prospective confusables were gathered from a number ofsources. Erik van der Poel contributed a list derived from running aprogram over a large number of fonts to catch characters that sharedidentical glyphs within a font, and Mark Davis did the same morerecently for fonts on Windows and the Macintosh. Volunteers fromGoogle, IBM, Microsoft and other companies gathered other lists ofcharacters. These included native speakers for languages withdifferent writing systems. The Unicode compatibility mappings werealso used as a source. The process of gathering visual confusables isongoing: the Unicode Consortium welcomes submission of additionalmappings. The complex scripts of South and Southeast Asia needspecial attention. The focus is on characters that have Identifier_Status=Allowed, because they are of mostconcern.

The fonts used to assess the confusables included those used bythe major operating systems in user interfaces. In addition, therepresentative glyphs used in the Unicode Standard were alsoconsidered. Fonts used for the user interface in operating systemsare an important source, because they are the ones that will usuallybe seen by users in circumstances where confusability is important,such such as when using IRIS (Internationalized Resource Identifiers)and their sub-elements (such as domain names). These fonts have anumber of other relevant characteristics:

• They rarely changed in updates to operating systems andapplications; changes brought by system upgrades tend to be gradualto avoid usability disruption.
• Because user interface elements need to be legible at lowscreen resolution (implying a low number of pixels per EM), fontsused in these contexts tend to be designed in sans-serif style,which has the tendency to increase the possibility of confusables.There are, however, some languages such as Chinese where a serifstyle is in common use.
• Strict bounding box requirements create even moreconstraints for scripts which use relatively large ascenders anddescenders. This also limits space allocated for accent or tonemarks, and can also create more opportunities for confusability.

Pairs of prospective confusables were removed if they were alwaysvisually distinct at common sizes, both within and across fonts. Thedata was then closed under transitivity, so that if X≅Y and Y≅Z, thenX≅Z. In addition, the data was closed under substring operations, sothat if X≅Y then AXB≅AYB. It was then processed to produce thein-script and cross-script data, so that a single data table can beused to map an input string to a resultingskeleton.

A skeleton is intendedonly for internal use for testingconfusability of strings; the resulting text is not suitable fordisplay to users, because it will appear to be a hodgepodge ofdifferent scripts. In particular, the result of mapping an identifierwill not necessary be an identifier. Thus the confusability mappingscan be used to test whether two identifiers are confusable (if theirskeletons are the same), but should definitely not be used as a"normalization" of identifiers.

6.2IdentifierModification Data Collection

Theidmod data is gathered in the following way. Thebasic assignments are derived based on UCD character properties,information in [UAX31], and a curated list ofexceptions based on information from various sources, including thecore specification of the Unicode Standard, annotations in the codecharts, information regarding CLDR exemplar characters, and externalfeedback.

The first condition that matches in the order of the items from topto bottom inTable 1.Identifier_Status and Identifier_Type is used, with a few exceptions:

1. When a character is inTable 3a,Optional Characters for MedialorTable 3b,Optional Characters for Continue in [UAX31], then it is given the Identifier_Type=Inclusion,regardless of other properties.
2. When the Script_Extensions property value for a charactercontains multiple Script property values, the Script used for thederivation is the first in the following list:
   1. Table 5,Recommended Scripts
   2. Table 7,Limited Use Scripts
   3. Table 4,Excluded Scripts

The script information inTable4,Table5, andTable7 is in machine-readable form in CLDR, as scriptMetadata.txt.

7Data Files

The following files provide data used to implement therecommendations in this document. The data may be refined in futureversions of this specification. For more information,see2.10.1, Backward Compatibility of [UTR36].For illustration, this UTS shows sample data values, but for theactual data for the current version of Unicode always refer to the data files.

The Unicode Consortium welcomes feedbackon additional confusables or identifier restrictions. There areonline forms at [Feedback] where you cansuggest additional characters or corrections.

The files are inhttps://www.unicode.org/Public/security/.The directories there contain data files associated with a givenversion. The directory forthis version is:

https://www.unicode.org/Public/security/15.1.0/

The data files for the latest approved version are also in thedirectory:

https://www.unicode.org/Public/security/latest

The format for IdentifierStatus.txt follows the normal conventions for UCD data files, and is described in the header of that file. All characters not listed in the file default to Identifier_Type=Restricted. Thus the file only lists characters with Identifier_Status=Allowed. For example:

002D..002E ; Allowed # 1.1 HYPHEN-MINUS..FULL STOP

The format for IdentifierType.txt follows the normal conventions for UCD data files, and is described in the header of that file. The value is a set whose elements are delimited by spaces. This format is identical to that used for ScriptExtensions.txt. This differs from prior versions which only listed the strongest reason for exclusion. This new convention allows the values to be used for more nuanced filtering. For example, if an implementation wants to allow an Exclusion script, it could still exclude Obsolete and Deprecated characters in that script. All characters not listed in the file default to Identifier_Type=Recommended. For example:

2460..24EA ; Technical Not_XID Not_NFKC # 1.1 CIRCLED DIGIT ONE..CIRCLED DIGIT ZERO

Both of these files have machine-readable# @missing lines for the default property values, as in many UCD files. For details about this syntax seeSection 4.2.10,@missing Conventions in [UAX44].

Table 2.Data File List

Migration

Beginning with version 6.3.0, the version numbering of thisdocument has been changed to indicate the version of the UCD that thedata is based on. For versions up to and including 6.3.0, thefollowing table shows the correspondence between the versions of thisdocument and UCD versions that they were based on.

Table 3.Version Correspondence

If an update version of this standard is required betweenthe associated UCD versions, the version numbering will include anupdate number in the 3rd field. For example, if a version of thisdocument and its associated data is needed between UCD 6.3.0 and UCD7.0.0, then a version 6.3.1 could be used.

MigratingPersistent Data

Implementations must migrate their persistent data stores (suchas database indexes) whenever those implementations update to use thedata files from a new version of this specification.

Stability is never guaranteed between versions, although it ismaintained where feasible. In particular, an updated version ofconfusable mapping data may use a mapping for a particular characterthat is different from the mapping used for that character in anearlier version. Thus there may be cases where X → Y in Version N,and X → Z in Version N+1, where Z may or may not have mapped to Y inVersion N. Even in cases where the logical data has not changedbetween versions, the order of lines in the data files may have beenchanged.

The Identifier_Status does not have stability guarantees (such as “Once a character is Allowed, it will not become Restricted in future versions”), because the data is changing over time as we find out more about character usage. Certain of the Identifier_Type values, such as Not_XID, are backward compatible but most may change as new data becomes available. The identifier data may also not appear to be completely consistent when just viewed from the perspective of script and general category. For example, it may well be that one character out of a set of nonspacing marks in a script is Restricted, while others are not. But that can be just a reflection of the fact that that character is obsolete and the others are not.

For identifier lookup, the data is aimed more at flagging possibly questionable characters, thus serving as one factor (among perhaps many, like using the "Safe Browsing" service) in determining whether the user should be notified in some way. For registration, flagged characters can result in a "soft no", that is, require the user to appeal a denial with more information.

For dealing with characters whose status changes to Restricted, implementations can use a grandfathering mechanism to maintain backwards compatibility.

Implementations should therefore have a strategy for migratingtheir persistent data stores (such as database indexes) that use anyof the confusable mapping data or other data files.

Version 13.0 Migration

As of Unicode 13.0, the Identifier_Status and Identifier_Type are consistently written with underbars. This may cause parsers to malfunction, those that do not follow Unicode conventions for matching of property names.

Version 10.0 Migration

As of Unicode 10.0, Identifier_Type=Aspirational is now empty; for more information, see [UAX31].

Version9.0 Migration

There is an important data format change between versions 8.0 and 9.0. In particular, the xidmodifications.txt file from Version 8.0 has been split into two files for Version 9.0: IdentifierStatus.txt and IdentifierType.txt.

Multiple values are listed in field 1 of IdentifierType.txt. To convert to the old format of xidmodifications.txt, use thelast value of that field. For example, the following values would correspond:

Version8.0 Migration

In Version 8.0, the following changes were made to theIdentifier_Status and Identifier_Type:

• Changed to the standard UCD formatting. For example,limited-use→Limited_Use.
  • Usually this was simply changing the case and hyphen, butnot-chars changed toNot_Character.
• Aligned the Identifier_Type better with UAX 31 and Unicodeproperties
  • historic
    • → Exclusion, where fromTable 4,Candidate Characters for Exclusion from Identifiers,
    • → Obsolete, otherwise
  • limited-use
    • → Limited_Use, where fromTable 7,Limited Use Scripts,
    • → Aspirational, where fromTable 6,Aspirational Use Scripts (later incorporated into Limited_Use in Version 10.0)
    • → Uncommon-Use, otherwise
  • obsolete
    • → Deprecated, where matching the Unicode property

Version7.0 Migration

Due to production problems, versions of the confusable mappingtables before 7.0 did not maintain idempotency in all cases, soupdating to version 8.0 is strongly advised.

Anyone using the skeleton mappings needs to rebuild anypersistent uses of skeletons, such as in database indexes.

The SL, SA, and ML mappings in 7.0 were significantly changedto address the idempotency problem. However, the tables SL, SA, andML were still problematic, and discouraged from use in 7.0. They werethus removed from version 8.0.

All of the data necessary for an implementation to recreate theremoved tables is available in the remaining data (MA) plus theUnicode Character Database properties (script, casing, etc.). Such arecreation would examine each of the equivalence classes from the MAdata, and filter out instances that did not fit the constraints (ofscript or casing). For the target character, it would choose the mostneutral character, typically a symbol. However, the reasons fordeprecating them still stand, so it is not recommended thatimplementations recreate them.

Note also that as the Script_Extensions data is made more complete,it may cause characters in the whole-script confusables data file tono longer match. For more information, seeSection 4,Confusable Detection.

Acknowledgments

Mark Davis and Michel Suignard authored the bulk of thetext, under direction from the Unicode Technical Committee. StevenLoomis and other people on the ICU team were very helpful indeveloping the original proposal for this technical report. Shane Carr analyzed the algorithms and supplied the source text for the rewrite of Sections 4 and 5 in version 10.

The attendees of the Source Code Working Group meetings assisted with the substantial changes made in Versions 15.0 and 15.1:Peter Constable,Elnar Dakeshov,Mark Davis,Barry Dorrans,Steve Dower,Michael Fanning,Asmus Freytag,Dante Gagne,Rich Gillam,Manish Goregaokar,Tom Honermann,Jan Lahoda,Nathan Lawrence,Robin Leroy,Chris Ries,Markus Scherer,Richard Smith.

Thanksalso to the following people for their feedback or contributions tothis document or earlier versions of it, or to the source data forconfusables or idmod: Julie Allen, Andrew Arnold, Vernon Cole, David Corbett (specal thanks for the many contributions),Douglas Davidson, Rob Dawson, Alex Dejarnatt, Chris Fynn, Martin Dürst, Asmus Freytag, DeborahGoldsmith, Manish Goregaokar, Paul Hoffman, Ned Holbrook, Denis Jacquerye, Cibu Johny, Patrick L.Jones, Peter Karlsson, Robin Leroy, Mike Kaplinskiy, Gervase Markham, Eric Muller,David Patterson, Erik van der Poel, Roozbeh Pournader, Michael van Riper, Marcos Sanz,Alexander Savenkov, Dominikus Scherkl, Manuel Strehl, Chris Weber, Ken Whistler,and Waïl Yahyaoui. Thanks to Peter Peng for his assistance with fontconfusables.

References

Modifications

The following summarizes modifications from the previouspublished version of this document.

Revision 28

• Reissued for Unicode 15.1
• Section 2Conformance
  • Replaced the conformance clauses by clearer requirements for profiles, rather than the former "Alternatively…".
• Section 3.1.1Joining Controls
  • Moved the definition and discussion of the contexts for Joining controls from UAX #31 to this section; they are no longer used in UAX #31.
• Section 4Confusable Detection
  • Added a note with a gloss to RZ-LGR terminology.
  • Changed the definition of confusability to take default ignorable code points into account.
  • Added a new confusability relation suitable for identifiers containing bidirectional text.
• Section 5.1Mixed-Script Detection
  • Fixed typo in code for Katakana script (again).

Modifications for previous versions are listed in those respective versions.

© 2023 Unicode, Inc. All Rights Reserved. The UnicodeConsortium makes no expressed or implied warranty of any kind, andassumes no liability for errors or omissions. No liability is assumedfor incidental and consequential damages in connection with orarising out of the use of the information or programs contained oraccompanying this technical report. The Unicode Terms of Useapply.

Unicode and the Unicode logo are trademarksof Unicode, Inc., and are registered in some jurisdictions.