| Standard | Unicode Standard |
|---|---|
| Classification | Unicode Transformation Format,extended ASCII,variable-length encoding |
| Extends | ASCII |
| Transforms / Encodes | ISO/IEC 10646 (Unicode) |
| Preceded by | UTF-1 |
UTF-8 is acharacter encoding standard used for electronic communication. Defined by theUnicode Standard, the name is derived fromUnicode Transformation Format – 8-bit.[1] As of July 2025, almost every webpage is transmitted as UTF-8.[2]
UTF-8 supports all 1,112,064[3] valid Unicodecode points using avariable-width encoding of one to four one-byte (8-bit) code units.
Code points with lower numerical values, which tend to occur more frequently, are encoded using fewer bytes. It was designed forbackward compatibility withASCII: the first 128 characters of Unicode, which correspond one-to-one with ASCII, are encoded using a single byte with the same binary value as ASCII, so that a UTF-8-encoded file using only those characters is identical to an ASCII file. Most software designed for anyextended ASCII can read and write UTF-8, and this results in fewer internationalization issues than any alternative text encoding.[4][5]
UTF-8 is dominant for all countries/languages on the internet, is used in most standards, often the only allowed encoding, and is supported by all modern operating systems and programming languages.
TheInternational Organization for Standardization (ISO) set out to compose a universal multi-byte character set in 1989. The draft ISO 10646 standard contained a non-requiredannex calledUTF-1 that provided a byte stream encoding of its32-bit code points. This encoding was not satisfactory on performance grounds, among other problems, and the biggest problem was probably that it did not have a clear separation between ASCII and non-ASCII: new UTF-1 tools would be backward compatible with ASCII-encoded text, but UTF-1-encoded text could confuse existing code expecting ASCII (orextended ASCII), because it could contain continuation bytes in the range0x21–0x7E that meant something else in ASCII, e.g.,0x2F for/, theUnixpath directory separator.
In July 1992, theX/Open committee XoJIG was looking for a better encoding. Dave Prosser ofUnix System Laboratories submitted a proposal for one that had faster implementation characteristics and introduced the improvement that 7-bit ASCII characters wouldonly represent themselves; multi-byte sequences would only include bytes with the high bit set. The nameFile System Safe UCS Transformation Format (FSS-UTF)[6] and most of the text of this proposal were later preserved in the final specification.[7][8][9] In August 1992, this proposal was circulated by anIBM X/Open representative to interested parties.
A modification byKen Thompson of thePlan 9 operating system group atBell Labs made itself-synchronizing, letting a reader start anywhere and immediately detect character boundaries, at the cost of being somewhat less bit-efficient than the previous proposal. It also abandoned the use of biases that preventedoverlong encodings.[9][10] Thompson's design was outlined on September 2, 1992, on aplacemat in a New Jersey diner withRob Pike. In the following days, Pike and Thompson implemented it and updatedPlan 9 to use it throughout,[11] and then communicated their success back to X/Open, which accepted it as the specification forFSS-UTF.[9] UTF-8 was first officially presented at theUSENIX conference inSan Diego, from January 25 to 29, 1993.[12] TheInternet Engineering Task Force adopted UTF-8 in its Policy on Character Sets and Languages in RFC 2277 (BCP 18) for future internet standards work in January 1998, replacingSingle Byte Character Sets such asLatin-1 in older RFCs.[13]
In November 2003, UTF-8 was restricted byRFC 3629 to match the constraints of theUTF-16 character encoding: explicitly prohibiting code points corresponding to the high and low surrogate characters removed more than 3% of the three-byte sequences, and ending atU+10FFFF removed more than 48% of the four-byte sequences and all five- and six-byte sequences.[14]
UTF-8 encodes code points in one to four bytes, depending on the value of the code point. In the following table, the charactersu toz are replaced by the bits of the code point, from the positionsU+uvwxyz:
| First code point | Last code point | Byte 1 | Byte 2 | Byte 3 | Byte 4 |
|---|---|---|---|---|---|
| U+0000 | U+007F | 0yyyzzzz | |||
| U+0080 | U+07FF | 110xxxyy | 10yyzzzz | ||
| U+0800 | U+FFFF | 1110wwww | 10xxxxyy | 10yyzzzz | |
| U+010000 | U+10FFFF | 11110uvv | 10vvwwww | 10xxxxyy | 10yyzzzz |
The first 128 code points (ASCII) need 1 byte. The next 1,920 code points need two bytes to encode, which covers the remainder of almost allLatin-script alphabets, and alsoIPA extensions,Greek,Cyrillic,Coptic,Armenian,Hebrew,Arabic,Syriac,Thaana andN'Ko alphabets, as well asCombining Diacritical Marks. Three bytes are needed for the remaining 61,440 codepoints of theBasic Multilingual Plane (BMP), including mostChinese, Japanese and Korean characters. Four bytes are needed for the 1,048,576 non-BMP code points, which includeemoji, less commonCJK characters, and other useful characters.[15]
UTF-8 is aprefix code and it is unnecessary to read past the last byte of a code point to decode it. Unlike many earlier multi-byte text encodings such asShift-JIS, it isself-synchronizing so searches for short strings or characters are possible; and the start of a code point can be found from a random position by backing up at most 3 bytes. The values chosen for the lead bytes means sorting a list of UTF-8 strings puts them in the same order as sortingUTF-32 strings.
Using a row in the above table to encode a code point less than "First code point" (thus using more bytes than necessary) is termed anoverlong encoding. These are a security problem because they allow character sequences such as malicious JavaScript and../ to bypass security validations, which has been reported in numerous high-profile products such as Microsoft'sIIS web server[16] and Apache's Tomcat servlet container.[17] Overlong encodings should therefore be considered an error and never decoded.Modified UTF-8 allows the two-byte overlong encoding0xC0,0x80, forU+0000 instead of just the single-byte0x00.[18]
Not all sequences of bytes are valid UTF-8. A UTF-8 decoder should be prepared for:
Many of the first UTF-8 decoders would decode these, ignoring incorrect bits. Carefully crafted invalid UTF-8 could make them either skip or create ASCII characters such asNUL, slash, or quotes, leading to security vulnerabilities. It is also common to throw an exception or truncate the string at an error[19] but this turns what would otherwise be harmless errors (i.e. "file not found") into adenial of service, for instance early versions of Python 3.0 would exit immediately if the command line orenvironment variables contained invalid UTF-8.[20]
RFC 3629 states "Implementations of the decoding algorithm MUST protect against decoding invalid sequences."[21]The Unicode Standard requires decoders to: "... treat any ill-formed code unit sequence as an error condition. This guarantees that it will neither interpret nor emit an ill-formed code unit sequence." The standard now recommends replacing each error with thereplacement character "�" (U+FFFD) and continue decoding.
Some decoders consider the sequenceE1,A0,20 (a truncated 3-byte code followed by a space) as a single error. This is not a good idea as a search for a space character would find the one hidden in the error. Since Unicode 6 (October 2010)[1] the standard (chapter 3) has recommended a "best practice" where the error is either one continuation byte, or ends at the first byte that is disallowed, soE1,A0,20 is a two-byte error followed by a space. An error is no more than three bytes long, never contains the start of a valid character, and there are21,952 different possible errors. Many decoders instead makeeach byte be an error, in which caseE1,A0,20 istwo errors followed by a space; there are now only 128 different errors which makes it practical to store the errors in the output string,[22] or replace them with characters from a legacy encoding.
Only a small subset of possible byte strings are error-free UTF-8: several bytes cannot appear; a byte with the high bit set cannot be alone; and in a truly random string a byte with a high bit set has only a1⁄15 chance of starting a valid UTF-8 character. This has the consequence of making it easy to detect if a legacy text encoding is accidentally used instead of UTF-8, making conversion of a system to UTF-8 easier and avoiding the need to require aByte Order Mark or any other metadata.
Since RFC 3629 (November 2003), the high and low surrogates used byUTF-16 (U+D800 throughU+DFFF) are not legal Unicode values, and their UTF-8 encodings must be treated as an invalid byte sequence.[21] These encodings all start with0xED followed by0xA0 or higher. This rule is often ignored as surrogates are allowed in Windows filenames and this means there must be a way to store them in a string.[23] UTF-8 that allows these surrogate halves has been (informally) calledWTF-8,[24] while another variation that also encodes all non-BMP characters as two surrogates (6 bytes instead of 4) is calledCESU-8.
The chart below gives the detailed meaning of each byte in a stream encoded in UTF-8.
| 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | A | B | C | D | E | F | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | ␀ | ␁ | ␂ | ␃ | ␄ | ␅ | ␆ | ␇ | ␈ | ␉ | ␊ | ␋ | ␌ | ␍ | ␎ | ␏ |
| 1 | ␐ | ␑ | ␒ | ␓ | ␔ | ␕ | ␖ | ␗ | ␘ | ␙ | ␚ | ␛ | ␜ | ␝ | ␞ | ␟ |
| 2 | ␠ | ! | " | # | $ | % | & | ' | ( | ) | * | + | , | - | . | / |
| 3 | 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | : | ; | < | = | > | ? |
| 4 | @ | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O |
| 5 | P | Q | R | S | T | U | V | W | X | Y | Z | [ | \ | ] | ^ | _ |
| 6 | ` | a | b | c | d | e | f | g | h | i | j | k | l | m | n | o |
| 7 | p | q | r | s | t | u | v | w | x | y | z | { | | | } | ~ | ␡ |
| 8 | ||||||||||||||||
| 9 | ||||||||||||||||
| A | ||||||||||||||||
| B | ||||||||||||||||
| C | 2 | 2 | 2 | 2 | 2 | 2 | 2 | 2 | 2 | 2 | 2 | 2 | 2 | 2 | 2 | 2 |
| D | 2 | 2 | 2 | 2 | 2 | 2 | 2 | 2 | 2 | 2 | 2 | 2 | 2 | 2 | 2 | 2 |
| E | 3 | 3 | 3 | 3 | 3 | 3 | 3 | 3 | 3 | 3 | 3 | 3 | 3 | 3 | 3 | 3 |
| F | 4 | 4 | 4 | 4 | 4 | 4 | 4 | 4 | 5 | 5 | 5 | 5 | 6 | 6 |
| ASCII control character | |
| ASCII character | |
| Continuation byte | |
| First byte of a N-byte code unit sequence | |
| Not all continuation bytes are allowed | |
| Unused |
If the Unicodebyte-order markU+FEFF is at the start of a UTF-8 file, the first three bytes will be0xEF,0xBB,0xBF.
The Unicode Standard neither requires nor recommends the use of the BOM for UTF-8, but warns that it may be encountered at the start of a file trans-coded from another encoding.[25] While ASCII text encoded using UTF-8 is backward compatible with ASCII, this is not true when Unicode Standard recommendations are ignored and a BOM is added. A BOM can confuse software that isn't prepared for it but can otherwise accept UTF-8, e.g. programming languages that permit non-ASCII bytes instring literals but not at the start of the file. Nevertheless, there was and still is software that always inserts a BOM when writing UTF-8, and refuses to correctly interpret UTF-8 unless the first character is a BOM (or the file only contains ASCII).[26]
 | This sectiondoes notcite anysources. Please helpimprove this section byadding citations to reliable sources. Unsourced material may be challenged andremoved. Find sources: "UTF-8" comparison to UTF-16 – news ·newspapers ·books ·scholar ·JSTOR(December 2024) (Learn how and when to remove this message) |
For a long time there was considerable argument as to whether it was better to process text inUTF-16 or in UTF-8. The primary advantage of UTF-16 is that theWindows API required it for access to all Unicode characters (UTF-8 was not fully supported in Windows until May 2019). This caused several libraries such asQt to also use UTF-16 strings which propagates this requirement to non-Windows platforms. In the early days of Unicode there were no characters greater thanU+FFFF andcombining characters were rarely used, so the 16-bit encoding was effectively fixed-size. Some believed fixed-size encoding could make processing more efficient, but any such advantages were lost as soon as UTF-16 became variable width as well. The code pointsU+0800–U+FFFF take 3 bytes in UTF-8 but only 2 in UTF-16. This led to the idea that text in Chinese and other languages would take more space in UTF-8. However, text is only larger if there are more of these code points than 1-byte ASCII code points, and this rarely happens in real-world documents due to spaces, newlines, digits, punctuation, English words, andmarkup[27]. UTF-8 has the advantages of being trivial to retrofit to any system that could handle anextended ASCII, not having byte-order problems, and taking about half the space for any language using mostlyLatin letters.
UTF-8 has been the most common encoding for theWorld Wide Web since 2008.[29] As of July 2025[update], UTF-8 is used by 98.8% of surveyed web sites.[2] Although many pages only use ASCII characters to display content, very few websites now declare their encoding to only be ASCII instead of UTF-8.[30] Virtually all countries and languages have 95% or more use of UTF-8 encodings on the web.
Many standards only support UTF-8, e.g.JSON exchange requires it (without a byte-order mark (BOM)).[31] UTF-8 is also the recommendation from theWHATWG for HTML andDOM specifications, and stating "UTF-8 encoding is the most appropriate encoding for interchange ofUnicode"[5] and theInternet Mail Consortium recommends that all e‑mail programs be able to display and create mail using UTF-8.[32][33] TheWorld Wide Web Consortium recommends UTF-8 as the default encoding in XML and HTML (and not just using UTF-8, also declaring it in metadata), "even when all characters are in the ASCII range ... Using non-UTF-8 encodings can have unexpected results".[34]
Many software programs have the ability to read/write UTF-8. It may require the user to change options from the normal settings, or may require a BOM (byte-order mark) as the first character to read the file. Examples of software supporting UTF-8 includeMicrosoft Word,[35][36][37]Microsoft Excel (Office 2003 and later),[38][39]Google Drive,LibreOffice,[40] and most databases.
Software that "defaults" to UTF-8 (meaning it writes it without the user changing settings, and it reads it without a BOM) has become more common since 2010.[41]Windows Notepad, in all currently supported versions of Windows, defaults to writing UTF-8 without a BOM (a change fromWindows 7Notepad), bringing it into line with most other text editors.[42] Some system files onWindows 11 require UTF-8[43] with no requirement for a BOM, and almost all files on macOS and Linux are required to be UTF-8 without a BOM.[citation needed] Programming languages that default to UTF-8 forI/O includeRuby 3.0,[44][45]R 4.2.2,[46]Raku andJava 18.[47] Although the current version ofPython requires an option toopen() to read/write UTF-8,[48] plans exist to make UTF-8 I/O the default in Python 3.15.[49]C++23 adopts UTF-8 as the only portable source code file format.[50]
Backwards compatibility is a serious impediment to changing code and APIs usingUTF-16 to use UTF-8, but this is happening. As of May 2019[update], Microsoftadded the capability for an application to set UTF-8 as the "code page" for the Windows API, removing the need to use UTF-16; and more recently has recommended programmers use UTF-8,[51] and even states "UTF-16 [...] is a unique burden that Windows places on code that targets multiple platforms".[4]The default string primitive inGo,[52]Julia,Rust,Swift (since version 5),[53] andPyPy[54] uses UTF-8 internally in all cases. Python (since version 3.3) uses UTF-8 internally for Python C API extensions[55][56] and sometimes for strings[55][57] and a future version of Python is planned to store strings as UTF-8 by default.[58][59] Modern versions ofMicrosoft Visual Studio use UTF-8 internally.[60] Microsoft's SQL Server 2019 added support for UTF-8, and using it results in a 35% speed increase, and "nearly 50% reduction in storage requirements."[61]
Java internally uses UTF-16 for thechar data type and, consequentially, theCharacter,String, and theStringBuffer classes,[62] but for I/O usesModified UTF-8 (MUTF-8), in which thenull characterU+0000 uses the two-byte overlong encoding0xC0, 0x80, instead of just0x00.[18] Modified UTF-8 strings never contain any actual null bytes but can contain all Unicode code points includingU+0000,[63] which allows such strings (with a null byte appended) to be processed by traditionalnull-terminated string functions. Java reads and writes normal UTF-8 to files and streams,[64] but it uses Modified UTF-8 for objectserialization,[65][66] for theJava Native Interface,[67] and for embedding constant strings inJava class files.[63] The dex format defined byDalvik also uses the same modified UTF-8 to represent string values.[68]Tcl also uses the same modified UTF-8[69] as Java for internal representation of Unicode data, but uses strict CESU-8 for external data. All known Modified UTF-8 implementations also treat the surrogate pairs as inCESU-8.
TheRaku programming language (formerly Perl 6) usesutf-8 encoding by default for I/O (Perl 5 also supports it); though that choice in Raku also implies "normalization into UnicodeNFC (normalization form canonical). In some cases the user will want to ensure no normalization is done; for thisutf8-c8" can be used.[70] ThatUTF-8 Clean-8 variant, implemented by Raku, is an encoder/decoder that preserves bytes as is (even illegal UTF-8 sequences) and allows for Normal Form Grapheme synthetics.[71]
Version 3 of thePython programming language treats each byte of an invalid UTF-8 bytestream as an error (see also changes with new UTF-8 mode in Python 3.7[72]); this gives 128 different possible errors. Extensions have been created to allow any byte sequence that is assumed to be UTF-8 to be losslessly transformed to UTF-16 or UTF-32, by translating the 128 possible error bytes to 128 reserved code points, and transforming those code points back to error bytes to output UTF-8. The most common approach is to translate the codes toU+DC80...U+DCFF which are low (trailing) surrogate values and thus "invalid" UTF-16, as used byPython'sPEP 383 (or "surrogateescape") approach.[22]NumPy version 2.0, and its file formats, support UTF-8 (adding StringDType for it).[73] Another encoding calledMirBSD OPTU-8/16 converts them toU+EF80...U+EFFF in aPrivate Use Area.[74] In either approach, the byte value is encoded in the low eight bits of the output code point. These encodings are needed if invalid UTF-8 is to survive translation to and then back from the UTF-16 used internally by Python, and as Unix filenames can contain invalid UTF-8 it is necessary for this to work.[75]
Linux and macOS filesystems support UTF-8:ext4[76] and Apple'sAPFS.[77] Apple's olderHFS Plus usesUTF-16 for file names, but uses UTF-8 insymbolic links.[78] Windows' filesystem,NTFS, uses UTF-16 for file names.
The official name for the encoding isUTF-8, the spelling used in all Unicode Consortium documents. Thehyphen-minus is required and no spaces are allowed. Some other names used are:
utf-8 is often used.[citation needed]utf8 and many other aliases.[79]csUTF8 as the only alias,[80] which is rarely used.UTF-8N means UTF-8without abyte-order mark (BOM), and in this caseUTF-8may imply thereis a BOM.[81][82]65001[83] with the symbolic nameCP_UTF8 in source code.utf8mb4,[84] whileutf8 andutf8mb3 refer to the obsoleteCESU-8 variant.[85]AL32UTF8 means UTF-8 (since version 9.0), whileUTF8 means CESU-8 (since 8.0),[86] andOracle's UTF8 encoding should not be used since it's only a subset of the full (UTF8/AL32UTF8; missing support for Japanese and Chinese, only supporting Unicode 3.0, not 4-byte form;AL32UTFFSS is another incomplete subset for older Oracle databases).[87]18N.[88]There are several current definitions of UTF-8 in various standards documents:
They supersede the definitions given in the following obsolete works:
They are all the same in their general mechanics, with the main differences being on issues such as allowed range of code point values and safe handling of invalid input.
Each encoding form maps the Unicode code points U+0000..U+D7FF and U+E000..U+10FFFF
DOC andDOCX files?".Stack Overflow. Retrieved2021-11-01..txt file fromWord".support.3playmedia.com. 14 March 2023.XLSX files UTF-8 encoded, by definition?".Stack Overflow. Excel. Retrieved2021-11-01.CSV file inExcel without mis-conversion of characters in Japanese and Chinese language for both Mac and Windows?".Microsoft Support Community. Retrieved2021-11-01.... in reality, you usually just assume UTF-8 since that is by far the most common encoding.
Microsoft is now defaulting to saving new text files as UTF-8 without BOM, as shown below.
Make sure your LayoutModification.json uses UTF-8 encoding.
UTF-8 representation is created on demand and cached in the Unicode object.
The deprecatedwstrandwstr_lengthmembers of the C implementation of unicode objects were removed, per PEP 623.
Visual Studio uses UTF-8 as the internal character encoding during conversion between the source character set and the execution character set.
InputStreamReader andOutputStreamWriterDataInput andDataOutputPreviously under XP (and, unverified, but probably Vista, too) for loops simply did not work while codepage 65001 was active
| Early telecommunications | |
|---|---|
| ISO/IEC 8859 |
|
| Bibliographic use | |
| National standards | |
| ISO/IEC 2022 | |
| Mac OSCode pages ("scripts") | |
| DOS code pages | |
| IBM AIX code pages | |
| Windows code pages | |
| EBCDIC code pages | |
| DEC terminals (VTx) | |
| Platform specific |
|
| Unicode /ISO/IEC 10646 | |
| TeX typesetting system | |
| Miscellaneous code pages | |
| Control character | |
| Related topics | |
