Movatterモバイル変換


[0]ホーム

URL:


Jump to content
WikipediaThe Free Encyclopedia
Search

UTF-32

From Wikipedia, the free encyclopedia
(Redirected fromUTF-32BE)
Encoding Unicode characters as 4 bytes per code point

UTF-32 (32-bitUnicode Transformation Format), sometimes called UCS-4, is a fixed-lengthencoding used to encode Unicodecode points that uses exactly 32 bits (fourbytes) per code point (but a number of leading bits must be zero as there are far fewer than 232 Unicode code points, needing actually only 21 bits).[1] In contrast, all other Unicode transformation formats are variable-length encodings. Each 32-bit value in UTF-32 represents one Unicode code point and is exactly equal to that code point's numerical value.

The main advantage of UTF-32 is that the Unicode code points are directly indexed. Finding theNth code point in a sequence of code points is aconstant-time operation. In contrast, avariable-length code requireslinear-time to countN code points from the start of the string. This makes UTF-32 a simple replacement in code that usesintegers that are incremented by one to examine each location in astring, as was commonly done forASCII. However, Unicode code points are rarely processed in complete isolation, such ascombining character sequences and for emoji.[2]

The main disadvantage of UTF-32 is that it is space-inefficient, using fourbytes per code point, including 11 bits that are always zero. Characters beyond theBMP are relatively rare in most texts (except, for example, in the case of texts with some popular emojis), and can typically be ignored for sizing estimates. This makes UTF-32 close to twice the size ofUTF-16. It can be up to four times the size ofUTF-8 depending on how many of the characters are in theASCII subset.[2]

History

[edit]

The originalISO/IEC 10646 standard defines a 32-bitencoding form calledUCS-4, in which each code point in theUniversal Character Set (UCS) is represented by a 31-bit value from 0 to 0x7FFFFFFF (the sign bit was unused and zero). In November 2003, Unicode was restricted by RFC 3629 to match the constraints of theUTF-16 encoding: explicitly prohibiting code points greater than U+10FFFF (and also the high and low surrogates U+D800 through U+DFFF). This limited subset defines UTF-32.[3][1] Although the ISO standard had (as of 1998 in Unicode 2.1) "reserved for private use" 0xE00000 to 0xFFFFFF, and 0x60000000 to 0x7FFFFFFF[4] these areas were removed in later versions. Because the Principles and Procedures document ofISO/IEC JTC 1/SC 2 Working Group 2 states that all future assignments of code points will be constrained to the Unicode range, UTF-32 will be able to represent all UCS code points and UTF-32 and UCS-4 are identical.[5]

Utility of fixed width

[edit]

A fixed number of bytes per code point has theoretical advantages, but each of these has problems in reality:

  • Truncation becomes easier, but not significantly so compared toUTF-8 andUTF-16 (both of which can search backwards for the point to truncate by looking at 2–4 code units at most).[a][citation needed]
  • Finding theNth character in a string. For fixed width, this is simply aO(1) problem, while it isO(n) problem in a variable-width encoding. Novice programmers often vastly overestimate how useful this is.[6] Also what a user might call a "character" is still variable-width, for instance thecombining character sequenceá could be 2 code points, the emoji👨‍🦲 is three,[7] and the ligature is one.
  • Quickly knowing the "width" of a string. However even "fixed width" fonts have varying width, oftenCJK ideographs are twice as wide,[6] plus the already-mentioned problems with the number of code points not being equal to the number of characters.

Use

[edit]

The main use of UTF-32 is in internal APIs where the data is single code points orglyphs, rather than strings of characters. For instance, in modern text rendering, it is common[citation needed] that the last step is to build a list of structures each containingcoordinates (x, y), attributes, and a single UTF-32 code point identifying the glyph to draw. Often non-Unicode information is stored in the "unused" 11 bits of each word.[citation needed]

Use of UTF-32 strings on Windows (wherewchar_t is 16 bits) is almost non-existent. On Unix systems, UTF-32 strings are sometimes, but rarely, used internally by applications, due to the typewchar_t being defined as 32-bit.

UTF-32 is also forbidden as an HTML character encoding.[8][9]

Programming languages

[edit]

Python versions up to 3.2 can be compiled to use them[clarification needed] instead ofUTF-16; from version 3.3 onward, Unicode strings are stored in UTF-32 if there is at least 1 non-BMP character in the string, but with leading zero bytes optimized away "depending on the [code point] with the largest Unicode ordinal (1, 2, or 4 bytes)" to make all code points that size.[10]

Seed7[11] andLasso[citation needed] programming languages encode all strings with UTF-32, in the belief that direct indexing is important, whereas theJulia programming language moved away from built-in UTF-32 support with its 1.0 release, simplifying the language to having only UTF-8 strings (with all the other encodings considered legacy and moved out of the standard library to package[12]) following the "UTF-8 Everywhere Manifesto".[13]

C++11 has 2 built-in data types that use UTF-32. Thechar32_t data type stores 1 character in UTF-32. Theu32string data type stores a string of UTF-32-encoded characters. A UTF-32-encoded character or string literal is marked withU before the character or string literal.[14][15]

#include<string>char32_tUTF32_character=U'🔟';// also written as U'\U0001F51F'std::u32stringUTF32_string=U"UTF–32-encoded string";// defined as `const char32_t*´

C# has aUTF32Encoding class which represents Unicode characters as bytes, rather than as a string.[16]

Variants

[edit]

Though technically invalid, the surrogate halves are often encoded and allowed. This allows invalid UTF-16 (such as Windows filenames) to be translated to UTF-32, similar to how theWTF-8 variant of UTF-8 works. Sometimes paired surrogates are encoded instead of non-BMP characters, similar toCESU-8. Due to the large number of unused 32-bit values, it is also possible to preserve invalid UTF-8 by using non-Unicode values to encode UTF-8 errors, though there is no standard for this.

UTF-32 has 2 versions for big-endian and little-endian:UTF-32-BE andUTF-32-LE.

See also

[edit]

Notes

[edit]
  1. ^For UTF-8: Select point to truncate at. If the byte before it is 0-0x7F, or the byte after it is anything other than the continuation bytes 0x80-0xBF, the string can be truncated at that point. Otherwise search up to 3 bytes backwards for such a point and truncate at that. If not found, truncate at the original position. This works even if there are encoding errors in the UTF-8. UTF-16 is trivial and only has to back up one word at most.

References

[edit]
  1. ^abConstable, Peter (2001-06-13)."Mapping codepoints to Unicode encoding forms".Computers and Writing Systems - SIL International. Retrieved2022-10-03.
  2. ^ab"FAQ - UTF-8, UTF-16, UTF-32 & BOM".Unicode. Retrieved2022-09-04.
  3. ^"Publicly Available Standards - ISO/IEC 10646:2020".ISO Standards. Retrieved2021-10-12.Clause 9.4: "Because surrogate code points are not UCS scalar values, UTF-32 code units in the range 0000 D800-0000 DFFF are ill-formed". Clause 4.57: "[UCS codespace] consisting of the integers from 0 to 10 FFFF (hexadecimal)". Clause 4.58: "[UCS scalar value] any UCS code point except high-surrogate and low-surrogate code points".
  4. ^"Annex B - The Universal Character Set (UCS)".DKUUG Standardizing.Archived from the original on Jan 22, 2022. Retrieved2022-10-03.
  5. ^"C.2 Encoding Forms in ISO/IEC 10646"(PDF).The Unicode Standard, version 6.0. Mountain View, CA:Unicode Consortium. February 2011. p. 573.ISBN 978-1-936213-01-6.It [UCS-4] is now treated simply as a synonym for UTF-32, and is considered the canonical form for representation of characters in 10646.
  6. ^abGoregaokar, Manish (January 14, 2017)."Let's Stop Ascribing Meaning to Code Points".In Pursuit of Laziness. Retrieved2020-06-14.Folks start implying that code points mean something, and that O(1) indexing or slicing at code point boundaries is a useful operation.
  7. ^"👨‍🦲 Man: Bald Emoji".Emojipedia. Retrieved2021-10-12.
  8. ^"HTML Standard".html.spec.whatwg.org. Retrieved2024-11-11.
  9. ^"Choisir et appliquer un encodage de caractères".www.w3.org (in French). Retrieved2024-11-11.
  10. ^Löwis, Martin."PEP 393 -- Flexible String Representation".python.org. Python. Retrieved26 October 2014.
  11. ^"Seed7 FAQ".seed7.sourceforge.net.
  12. ^JuliaStrings/LegacyStrings.jl: Legacy Unicode string types, JuliaStrings, 2019-05-17, retrieved2019-10-15
  13. ^"UTF-8 Everywhere".utf8everywhere.org.
  14. ^"u32string".cplusplus.com. Retrieved2024-11-12.
  15. ^"String literal - cppreference.com".en.cppreference.com. Retrieved2024-11-14.
  16. ^dotnet-bot."UTF32Encoding Class (System.Text)".learn.microsoft.com. Retrieved2024-11-27.

External links

[edit]
Unicode
Code points
Characters
Special purpose
Lists
Processing
Algorithms
Comparison of encodings
On pairs of
code points
Usage
Related standards
Related topics
Scripts and symbols in Unicode
Common and
inherited scripts
Modern scripts
Ancient and
historic scripts
Notational scripts
Symbols, emojis
Early telecommunications
ISO/IEC 8859
Bibliographic use
National standards
ISO/IEC 2022
Mac OSCode pages
("scripts")
DOS code pages
IBM AIX code pages
Windows code pages
EBCDIC code pages
DEC terminals (VTx)
Platform specific
Unicode /ISO/IEC 10646
TeX typesetting system
Miscellaneous code pages
Control character
Related topics
Retrieved from "https://en.wikipedia.org/w/index.php?title=UTF-32&oldid=1271513878#UTF-32BE"
Categories:
Hidden categories:

[8]ページ先頭

©2009-2025 Movatter.jp