Historical Note

This document was first written by Marc-Andre in the pre-PEP days,and was originally distributed as Misc/unicode.txt in Pythondistributions up to and included Python 2.1. The last revision ofthe proposal in that location was labeled version 1.7 (CVSrevision 3.10). Because the document clearly serves the purposeof an informational PEP in the post-PEP era, it has been movedhere and reformatted to comply with PEP guidelines. Futurerevisions will be made to this document, while Misc/unicode.txtwill contain a pointer to this PEP.

-Barry Warsaw, PEP editor

Introduction

The idea of this proposal is to add native Unicode 3.0 support toPython in a way that makes use of Unicode strings as simple aspossible without introducing too many pitfalls along the way.

Since this goal is not easy to achieve – strings being one of themost fundamental objects in Python – we expect this proposal toundergo some significant refinements.

Note that the current version of this proposal is still a bitunsorted due to the many different aspects of the Unicode-Pythonintegration.

The latest version of this document is always available at:http://starship.python.net/~lemburg/unicode-proposal.txt

Older versions are available as:http://starship.python.net/~lemburg/unicode-proposal-X.X.txt

[ed. note: new revisions should be made to this PEP document,while the historical record previous to version 1.7 should beretrieved from MAL’s url, or Misc/unicode.txt]

Conventions

• In examples we use u = Unicode object and s = Python string
• ‘XXX’ markings indicate points of discussion (PODs)

General Remarks

• Unicode encoding names should be lower case on output andcase-insensitive on input (they will be converted to lower caseby all APIs taking an encoding name as input).
• Encoding names should follow the name conventions as used by theUnicode Consortium: spaces are converted to hyphens, e.g. ‘utf16’ is written as ‘utf-16’.
• Codec modules should use the same names, but with hyphensconverted to underscores, e.g.utf_8,utf_16,iso_8859_1.

Unicode Default Encoding

The Unicode implementation has to make some assumption about theencoding of 8-bit strings passed to it for coercion and about theencoding to as default for conversion of Unicode to strings whenno specific encoding is given. This encoding is called <defaultencoding> throughout this text.

For this, the implementation maintains a global which can be setin the site.py Python startup script. Subsequent changes are notpossible. The <default encoding> can be set and queried using thetwo sys module APIs:

sys.setdefaultencoding(encoding)

Sets the <default encoding> used by the Unicode implementation.encoding has to be an encoding which is supported by thePython installation, otherwise, a LookupError is raised.

Note: This API is only available in site.py! It isremoved from the sys module by site.py after usage.

sys.getdefaultencoding()

Returns the current <default encoding>.

If not otherwise defined or set, the <default encoding> defaultsto ‘ascii’. This encoding is also the startup default of Python(and in effect before site.py is executed).

Note that the default site.py startup module contains disabledoptional code which can set the <default encoding> according tothe encoding defined by the current locale. The locale module isused to extract the encoding from the locale default settingsdefined by the OS environment (see locale.py). If the encodingcannot be determined, is unknown or unsupported, the code defaultsto setting the <default encoding> to ‘ascii’. To enable thiscode, edit the site.py file or place the appropriate code into thesitecustomize.py module of your Python installation.

Unicode Constructors

Python should provide a built-in constructor for Unicode stringswhich is available through__builtins__:

u=unicode(encoded_string[,encoding=<defaultencoding>][,errors="strict"])u=u'<unicode-escape encoded Python string>'u=ur'<raw-unicode-escape encoded Python string>'

With the ‘unicode-escape’ encoding being defined as:

• all non-escape characters represent themselves as Unicodeordinal (e.g. ‘a’ -> U+0061).
• all existing defined Python escape sequences are interpreted asUnicode ordinals; note that\xXXXX can represent all Unicodeordinals, and\OOO (octal) can represent Unicode ordinals up toU+01FF.
• a new escape sequence,\uXXXX, represents U+XXXX; it is a syntaxerror to have fewer than 4 digits after\u.

For an explanation of possible values for errors see the Codecsection below.

Examples:

u'abc'->U+0061U+0062U+0063u'\u1234'->U+1234u'abc\u1234\n'->U+0061U+0062U+0063U+1234U+005c

The ‘raw-unicode-escape’ encoding is defined as follows:

• \uXXXX sequence represent the U+XXXX Unicode character if andonly if the number of leading backslashes is odd
• all other characters represent themselves as Unicode ordinal(e.g. ‘b’ -> U+0062)

Note that you should provide some hint to the encoding you used towrite your programs as pragma line in one the first few commentlines of the source file (e.g. ‘# source file encoding: latin-1’).If you only use 7-bit ASCII then everything is fine and no suchnotice is needed, but if you include Latin-1 characters notdefined in ASCII, it may well be worthwhile including a hint sincepeople in other countries will want to be able to read your sourcestrings too.

Unicode Type Object

Unicode objects should have the type UnicodeType with type name‘unicode’, made available through the standard types module.

Unicode Output

Unicode objects have a method .encode([encoding=<default encoding>])which returns a Python string encoding the Unicode string using thegiven scheme (see Codecs).

printu:=printu.encode()# using the <default encoding>str(u):=u.encode()# using the <default encoding>repr(u):="u%s"%repr(u.encode('unicode-escape'))

Also see Internal Argument Parsing and Buffer Interface fordetails on how other APIs written in C will treat Unicode objects.

Unicode Ordinals

Since Unicode 3.0 has a 32-bit ordinal character set, theimplementation should provide 32-bit aware ordinal conversionAPIs:

ord(u[:1])(thisisthestandardord()extendedtoworkwithUnicodeobjects)-->Unicodeordinalnumber(32-bit)unichr(i)-->Unicodeobjectforcharacteri(provideditis32-bit);ValueErrorotherwise

Both APIs should go into__builtins__ just like their stringcounterpartsord() andchr().

Note that Unicode provides space for private encodings. Usage ofthese can cause different output representations on differentmachines. This problem is not a Python or Unicode problem, but amachine setup and maintenance one.

Comparison & Hash Value

Unicode objects should compare equal to other objects after theseother objects have been coerced to Unicode. For strings thismeans that they are interpreted as Unicode string using the<default encoding>.

Unicode objects should return the same hash value as their ASCIIequivalent strings. Unicode strings holding non-ASCII values arenot guaranteed to return the same hash values as the defaultencoded equivalent string representation.

When compared usingcmp() (orPyObject_Compare()) theimplementation should maskTypeErrors raised during the conversionto remain in synch with the string behavior. All other errorssuch asValueErrors raised during coercion of strings to Unicodeshould not be masked and passed through to the user.

In containment tests (‘a’ in u’abc’ and u’a’ in ‘abc’) both sidesshould be coerced to Unicode before applying the test. Errorsoccurring during coercion (e.g. None in u’abc’) should not bemasked.

Coercion

Using Python strings and Unicode objects to form new objectsshould always coerce to the more precise format, i.e. Unicodeobjects.

u+s:=u+unicode(s)s+u:=unicode(s)+u

All string methods should delegate the call to an equivalentUnicode object method call by converting all involved strings toUnicode and then applying the arguments to the Unicode method ofthe same name, e.g.

string.join((s,u),sep):=(s+sep)+usep.join((s,u)):=(s+sep)+u

For a discussion of %-formatting w/r to Unicode objects, seeFormatting Markers.

Exceptions

UnicodeError is defined in the exceptions module as a subclass ofValueError. It is available at the C level viaPyExc_UnicodeError. All exceptions related to Unicodeencoding/decoding should be subclasses ofUnicodeError.

Codecs (Coder/Decoders) Lookup

A Codec (see Codec Interface Definition) search registry should beimplemented by a module “codecs”:

codecs.register(search_function)

Search functions are expected to take one argument, the encodingname in all lower case letters and with hyphens and spacesconverted to underscores, and return a tuple of functions(encoder, decoder, stream_reader, stream_writer) taking thefollowing arguments:

encoder and decoder

These must be functions or methods which have the sameinterface as the.encode/.decode methods of Codec instances(see Codec Interface). The functions/methods are expected towork in a stateless mode.

stream_reader and stream_writer

These need to be factory functions with the followinginterface:

factory(stream,errors='strict')

The factory functions must return objects providing theinterfaces defined byStreamWriter/StreamReader resp. (seeCodec Interface). Stream codecs can maintain state.

Possible values for errors are defined in the Codec sectionbelow.

In case a search function cannot find a given encoding, it shouldreturn None.

Aliasing support for encodings is left to the search functions toimplement.

The codecs module will maintain an encoding cache for performancereasons. Encodings are first looked up in the cache. If notfound, the list of registered search functions is scanned. If nocodecs tuple is found, a LookupError is raised. Otherwise, thecodecs tuple is stored in the cache and returned to the caller.

To query the Codec instance the following API should be used:

codecs.lookup(encoding)

This will either return the found codecs tuple or raise aLookupError.

Standard Codecs

Standard codecs should live inside an encodings/ package directoryin the Standard Python Code Library. The__init__.py file of thatdirectory should include a Codec Lookup compatible search functionimplementing a lazy module based codec lookup.

Python should provide a few standard codecs for the most relevantencodings, e.g.

'utf-8':8-bitvariablelengthencoding'utf-16':16-bitvariablelengthencoding(little/bigendian)'utf-16-le':utf-16butexplicitlylittleendian'utf-16-be':utf-16butexplicitlybigendian'ascii':7-bitASCIIcodepage'iso-8859-1':ISO8859-1(Latin1)codepage'unicode-escape':SeeUnicodeConstructorsforadefinition'raw-unicode-escape':SeeUnicodeConstructorsforadefinition'native':DumpoftheInternalFormatusedbyPython

Common aliases should also be provided per default, e.g.‘latin-1’ for ‘iso-8859-1’.

Note: ‘utf-16’ should be implemented by using and requiring byteorder marks (BOM) for file input/output.

All other encodings such as the CJK ones to support Asian scriptsshould be implemented in separate packages which do not getincluded in the core Python distribution and are not a part ofthis proposal.

Codecs Interface Definition

The following base class should be defined in the module “codecs”.They provide not only templates for use by encoding moduleimplementors, but also define the interface which is expected bythe Unicode implementation.

Note that the Codec Interface defined here is well suitable for alarger range of applications. The Unicode implementation expectsUnicode objects on input for.encode() and.write() and characterbuffer compatible objects on input for.decode(). Output of.encode() and.read() should be a Python string and.decode() mustreturn an Unicode object.

First, we have the stateless encoders/decoders. These do not workin chunks as the stream codecs (see below) do, because allcomponents are expected to be available in memory.

classCodec:"""Defines the interface for stateless encoders/decoders.       The .encode()/.decode() methods may implement different       error handling schemes by providing the errors argument.       These string values are defined:         'strict'  - raise an error (or a subclass)         'ignore'  - ignore the character and continue with the next         'replace' - replace with a suitable replacement character;                     Python will use the official U+FFFD                     REPLACEMENT CHARACTER for the builtin Unicode                     codecs.    """defencode(self,input,errors='strict'):"""Encodes the object input and returns a tuple (output           object, length consumed).           errors defines the error handling to apply.  It           defaults to 'strict' handling.           The method may not store state in the Codec instance.           Use StreamCodec for codecs which have to keep state in           order to make encoding/decoding efficient.        """defdecode(self,input,errors='strict'):"""Decodes the object input and returns a tuple (output           object, length consumed).           input must be an object which provides the           bf_getreadbuf buffer slot.  Python strings, buffer           objects and memory mapped files are examples of objects           providing this slot.           errors defines the error handling to apply.  It           defaults to 'strict' handling.           The method may not store state in the Codec instance.           Use StreamCodec for codecs which have to keep state in           order to make encoding/decoding efficient.        """

StreamWriter andStreamReader define the interface for statefulencoders/decoders which work on streams. These allow processingof the data in chunks to efficiently use memory. If you havelarge strings in memory, you may want to wrap them withcStringIOobjects and then use these codecs on them to be able to do chunkprocessing as well, e.g. to provide progress information to theuser.

classStreamWriter(Codec):def__init__(self,stream,errors='strict'):"""Creates a StreamWriter instance.           stream must be a file-like object open for writing           (binary) data.           The StreamWriter may implement different error handling           schemes by providing the errors keyword argument.           These parameters are defined:             'strict' - raise a ValueError (or a subclass)             'ignore' - ignore the character and continue with the next             'replace'- replace with a suitable replacement character        """self.stream=streamself.errors=errorsdefwrite(self,object):"""Writes the object's contents encoded to self.stream.        """data,consumed=self.encode(object,self.errors)self.stream.write(data)defwritelines(self,list):"""Writes the concatenated list of strings to the stream           using .write().        """self.write(''.join(list))defreset(self):"""Flushes and resets the codec buffers used for keeping state.           Calling this method should ensure that the data on the           output is put into a clean state, that allows appending           of new fresh data without having to rescan the whole           stream to recover state.        """passdef__getattr__(self,name,getattr=getattr):"""Inherit all other methods from the underlying stream.        """returngetattr(self.stream,name)classStreamReader(Codec):def__init__(self,stream,errors='strict'):"""Creates a StreamReader instance.           stream must be a file-like object open for reading           (binary) data.           The StreamReader may implement different error handling           schemes by providing the errors keyword argument.           These parameters are defined:             'strict' - raise a ValueError (or a subclass)             'ignore' - ignore the character and continue with the next             'replace'- replace with a suitable replacement character;        """self.stream=streamself.errors=errorsdefread(self,size=-1):"""Decodes data from the stream self.stream and returns the           resulting object.           size indicates the approximate maximum number of bytes           to read from the stream for decoding purposes.  The           decoder can modify this setting as appropriate.  The           default value -1 indicates to read and decode as much           as possible.  size is intended to prevent having to           decode huge files in one step.           The method should use a greedy read strategy meaning           that it should read as much data as is allowed within           the definition of the encoding and the given size, e.g.           if optional encoding endings or state markers are           available on the stream, these should be read too.        """# Unsliced reading:ifsize<0:returnself.decode(self.stream.read())[0]# Sliced reading:read=self.stream.readdecode=self.decodedata=read(size)i=0while1:try:object,decodedbytes=decode(data)exceptValueError,why:# This method is slow but should work under pretty# much all conditions; at most 10 tries are madei=i+1newdata=read(1)ifnotnewdataori>10:raisedata=data+newdataelse:returnobjectdefreadline(self,size=None):"""Read one line from the input stream and return the           decoded data.           Note: Unlike the .readlines() method, this method           inherits the line breaking knowledge from the           underlying stream's .readline() method -- there is           currently no support for line breaking using the codec           decoder due to lack of line buffering.  Subclasses           should however, if possible, try to implement this           method using their own knowledge of line breaking.           size, if given, is passed as size argument to the           stream's .readline() method.        """ifsizeisNone:line=self.stream.readline()else:line=self.stream.readline(size)returnself.decode(line)[0]defreadlines(self,sizehint=0):"""Read all lines available on the input stream           and return them as list of lines.           Line breaks are implemented using the codec's decoder           method and are included in the list entries.           sizehint, if given, is passed as size argument to the           stream's .read() method.        """ifsizehintisNone:data=self.stream.read()else:data=self.stream.read(sizehint)returnself.decode(data)[0].splitlines(1)defreset(self):"""Resets the codec buffers used for keeping state.           Note that no stream repositioning should take place.           This method is primarily intended to be able to recover           from decoding errors.        """passdef__getattr__(self,name,getattr=getattr):""" Inherit all other methods from the underlying stream.        """returngetattr(self.stream,name)

Stream codec implementors are free to combine theStreamWriter andStreamReader interfaces into one class. Even combining all thesewith the Codec class should be possible.

Implementors are free to add additional methods to enhance thecodec functionality or provide extra state information needed forthem to work. The internal codec implementation will only use theabove interfaces, though.

It is not required by the Unicode implementation to use these baseclasses, only the interfaces must match; this allows writingCodecs as extension types.

As guideline, large mapping tables should be implemented usingstatic C data in separate (shared) extension modules. That waymultiple processes can share the same data.

A tool to auto-convert Unicode mapping files to mapping modulesshould be provided to simplify support for additional mappings(see References).

Whitespace

The.split() method will have to know about what is consideredwhitespace in Unicode.

Case Conversion

Case conversion is rather complicated with Unicode data, sincethere are many different conditions to respect. See

http://www.unicode.org/unicode/reports/tr13/

for some guidelines on implementing case conversion.

For Python, we should only implement the 1-1 conversions includedin Unicode. Locale dependent and other special case conversions(see the Unicode standard file SpecialCasing.txt) should be leftto user land routines and not go into the core interpreter.

The methods.capitalize() and.iscapitalized() should follow thecase mapping algorithm defined in the above technical report asclosely as possible.

Line Breaks

Line breaking should be done for all Unicode characters having theB property as well as the combinations CRLF, CR, LF (interpretedin that order) and other special line separators defined by thestandard.

The Unicode type should provide a.splitlines() method whichreturns a list of lines according to the above specification. SeeUnicode Methods.

Unicode Character Properties

A separate module “unicodedata” should provide a compact interfaceto all Unicode character properties defined in the standard’sUnicodeData.txt file.

Among other things, these properties provide ways to recognizenumbers, digits, spaces, whitespace, etc.

Since this module will have to provide access to all Unicodecharacters, it will eventually have to contain the data fromUnicodeData.txt which takes up around 600kB. For this reason, thedata should be stored in static C data. This enables compilationas shared module which the underlying OS can shared betweenprocesses (unlike normal Python code modules).

There should be a standard Python interface for accessing thisinformation so that other implementors can plug in their ownpossibly enhanced versions, e.g. ones that do decompressing of thedata on-the-fly.

Private Code Point Areas

Support for these is left to user land Codecs and not explicitlyintegrated into the core. Note that due to the Internal Formatbeing implemented, only the area between\uE000 and\uF8FF isusable for private encodings.

Internal Format

The internal format for Unicode objects should use a Pythonspecific fixed format <PythonUnicode> implemented as ‘unsignedshort’ (or another unsigned numeric type having 16 bits). Byteorder is platform dependent.

This format will hold UTF-16 encodings of the correspondingUnicode ordinals. The Python Unicode implementation will addressthese values as if they were UCS-2 values. UCS-2 and UTF-16 arethe same for all currently defined Unicode character points.UTF-16 without surrogates provides access to about 64k charactersand covers all characters in the Basic Multilingual Plane (BMP) ofUnicode.

It is the Codec’s responsibility to ensure that the data they passto the Unicode object constructor respects this assumption. Theconstructor does not check the data for Unicode compliance or useof surrogates.

Future implementations can extend the 32 bit restriction to thefull set of all UTF-16 addressable characters (around 1Mcharacters).

The Unicode API should provide interface routines from<PythonUnicode> to the compiler’s wchar_t which can be 16 or 32bit depending on the compiler/libc/platform being used.

Unicode objects should have a pointer to a cached Python stringobject <defenc> holding the object’s value using the <defaultencoding>. This is needed for performance and internal parsing(see Internal Argument Parsing) reasons. The buffer is filledwhen the first conversion request to the <default encoding> isissued on the object.

Interning is not needed (for now), since Python identifiers aredefined as being ASCII only.

codecs.BOM should return the byte order mark (BOM) for the formatused internally. The codecs module should provide the followingadditional constants for convenience and reference (codecs.BOMwill either beBOM_BE orBOM_LE depending on the platform):

BOM_BE:'\376\377'(correspondstoUnicodeU+0000FEFFinUTF-16onbigendianplatforms==ZEROWIDTHNO-BREAKSPACE)BOM_LE:'\377\376'(correspondstoUnicodeU+0000FFFEinUTF-16onlittleendianplatforms==definedasbeinganillegalUnicodecharacter)BOM4_BE:'\000\000\376\377'(correspondstoUnicodeU+0000FEFFinUCS-4)BOM4_LE:'\377\376\000\000'(correspondstoUnicodeU+0000FFFEinUCS-4)

Note that Unicode sees big endian byte order as being “correct”.The swapped order is taken to be an indicator for a “wrong”format, hence the illegal character definition.

The configure script should provide aid in deciding whether Pythoncan use the nativewchar_t type or not (it has to be a 16-bitunsigned type).

Buffer Interface

Implement the buffer interface using the <defenc> Python stringobject as basis forbf_getcharbuf and the internal buffer forbf_getreadbuf. Ifbf_getcharbuf is requested and the <defenc>object does not yet exist, it is created first.

Note that as special case, the parser marker “s#” will not returnraw Unicode UTF-16 data (which thebf_getreadbuf returns), butinstead tries to encode the Unicode object using the defaultencoding and then returns a pointer to the resulting string object(or raises an exception in case the conversion fails). This wasdone in order to prevent accidentally writing binary data to anoutput stream which the other end might not recognize.

This has the advantage of being able to write to output streams(which typically use this interface) without additionalspecification of the encoding to use.

If you need to access the read buffer interface of Unicodeobjects, use thePyObject_AsReadBuffer() interface.

The internal format can also be accessed using the‘unicode-internal’ codec, e.g. viau.encode('unicode-internal').

Pickle/Marshalling

Should have native Unicode object support. The objects should beencoded using platform independent encodings.

Marshal should use UTF-8 and Pickle should either chooseRaw-Unicode-Escape (in text mode) or UTF-8 (in binary mode) asencoding. Using UTF-8 instead of UTF-16 has the advantage ofeliminating the need to store a BOM mark.

Regular Expressions

Secret Labs AB is working on a Unicode-aware regular expressionmachinery. It works on plain 8-bit, UCS-2, and (optionally) UCS-4internal character buffers.

Also see

http://www.unicode.org/unicode/reports/tr18/

for some remarks on how to treat Unicode REs.

Formatting Markers

Format markers are used in Python format strings. If Pythonstrings are used as format strings, the following interpretationsshould be in effect:

'%s':ForUnicodeobjectsthiswillcausecoercionofthewholeformatstringtoUnicode.NotethatyoushoulduseaUnicodeformatstringtostartwithforperformancereasons.

In case the format string is an Unicode object, all parameters arecoerced to Unicode first and then put together and formattedaccording to the format string. Numbers are first converted tostrings and then to Unicode.

'%s':PythonstringsareinterpretedasUnicodestringusingthe<defaultencoding>.Unicodeobjectsaretakenasis.

All other string formatters should work accordingly.

Example:

u"%s%s"%(u"abc","abc")==u"abc abc"

Internal Argument Parsing

These markers are used by thePyArg_ParseTuple() APIs:

“U”

Check for Unicode object and return a pointer to it

“s”

For Unicode objects: return a pointer to the object’s<defenc> buffer (which uses the <default encoding>).

“s#”

Access to the default encoded version of the Unicode object(see Buffer Interface); note that the length relates tothe length of the default encoded string rather than theUnicode object length.

“t#”

Same as “s#”.

“es”

Takes two parameters: encoding (constchar*) and buffer(char**).

The input object is first coerced to Unicode in the usualway and then encoded into a string using the givenencoding.

On output, a buffer of the needed size is allocated andreturned through*buffer as NULL-terminated string. Theencoded may not contain embedded NULL characters. Thecaller is responsible for callingPyMem_Free() to free theallocated*buffer after usage.

“es#”

Takes three parameters: encoding (constchar*), buffer(char**) and buffer_len (int*).

The input object is first coerced to Unicode in the usualway and then encoded into a string using the givenencoding.

If*buffer is non-NULL,*buffer_len must be set tosizeof(buffer) on input. Output is then copied to*buffer.

If*buffer is NULL, a buffer of the needed size isallocated and output copied into it.*buffer is thenupdated to point to the allocated memory area. The calleris responsible for callingPyMem_Free() to free theallocated*buffer after usage.

In both cases*buffer_len is updated to the number ofcharacters written (excluding the trailing NULL-byte).The output buffer is assured to be NULL-terminated.

Examples:

Using “es#” with auto-allocation:

static PyObject *test_parser(PyObject *self,            PyObject *args){    PyObject *str;    const char *encoding = "latin-1";    char *buffer = NULL;    int buffer_len = 0;    if (!PyArg_ParseTuple(args, "es#:test_parser",                          encoding, &buffer, &buffer_len))        return NULL;    if (!buffer) {        PyErr_SetString(PyExc_SystemError,                        "buffer is NULL");        return NULL;    }    str = PyString_FromStringAndSize(buffer, buffer_len);    PyMem_Free(buffer);    return str;}

Using “es” with auto-allocation returning a NULL-terminated string:

static PyObject *test_parser(PyObject *self,            PyObject *args){    PyObject *str;    const char *encoding = "latin-1";    char *buffer = NULL;    if (!PyArg_ParseTuple(args, "es:test_parser",                          encoding, &buffer))        return NULL;    if (!buffer) {        PyErr_SetString(PyExc_SystemError,                        "buffer is NULL");        return NULL;    }    str = PyString_FromString(buffer);    PyMem_Free(buffer);    return str;}

Using “es#” with a pre-allocated buffer:

static PyObject *test_parser(PyObject *self,            PyObject *args){    PyObject *str;    const char *encoding = "latin-1";    char _buffer[10];    char *buffer = _buffer;    int buffer_len = sizeof(_buffer);    if (!PyArg_ParseTuple(args, "es#:test_parser",                          encoding, &buffer, &buffer_len))        return NULL;    if (!buffer) {        PyErr_SetString(PyExc_SystemError,                        "buffer is NULL");        return NULL;    }    str = PyString_FromStringAndSize(buffer, buffer_len);    return str;}

File/Stream Output

Since file.write(object) and most other stream writers use the“s#” or “t#” argument parsing marker for querying the data towrite, the default encoded string version of the Unicode objectwill be written to the streams (see Buffer Interface).

For explicit handling of files using Unicode, the standard streamcodecs as available through the codecs module should be used.

The codecs module should provide a short-cutopen(filename,mode,encoding) available which also assures thatmode contains the ‘b’ character when needed.

File/Stream Input

Only the user knows what encoding the input data uses, so nospecial magic is applied. The user will have to explicitlyconvert the string data to Unicode objects as needed or use thefile wrappers defined in the codecs module (see File/StreamOutput).

Unicode Methods & Attributes

All Python string methods, plus:

.encode([encoding=<defaultencoding>][,errors="strict"])-->seeUnicodeOutput.splitlines([include_breaks=0])-->breakstheUnicodestringintoalistof(Unicode)lines;returnsthelineswithlinebreaksincluded,ifinclude_breaksistrue.SeeLineBreaksforaspecificationofhowlinebreakingisdone.

Code Base

We should use Fredrik Lundh’s Unicode object implementation asbasis. It already implements most of the string methods neededand provides a well written code base which we can build upon.

The object sharing implemented in Fredrik’s implementation shouldbe dropped.

Test Cases

Test cases should follow those in Lib/test/test_string.py andinclude additional checks for the Codec Registry and the StandardCodecs.

References

• Unicode Consortium:http://www.unicode.org/
• Unicode FAQ:http://www.unicode.org/unicode/faq/
• Unicode 3.0:http://www.unicode.org/unicode/standard/versions/Unicode3.0.html
• Unicode-TechReports:http://www.unicode.org/unicode/reports/techreports.html
• Unicode-Mappings:ftp://ftp.unicode.org/Public/MAPPINGS/
• Introduction to Unicode (a little outdated by still nice to read):http://www.nada.kth.se/i18n/ucs/unicode-iso10646-oview.html
• For comparison:Introducing Unicode to ECMAScript (aka JavaScript) –http://www-4.ibm.com/software/developer/library/internationalization-support.html
• IANA Character Set Names:ftp://ftp.isi.edu/in-notes/iana/assignments/character-sets
• Discussion of UTF-8 and Unicode support for POSIX and Linux:http://www.cl.cam.ac.uk/~mgk25/unicode.html
• Encodings:
  • Overview:http://czyborra.com/utf/
  • UCS-2:http://www.uazone.org/multiling/unicode/ucs2.html
  • UTF-7: Defined inRFC 2152
  • UTF-8: Defined inRFC 2279
  • UTF-16:http://www.uazone.org/multiling/unicode/wg2n1035.html

History of this Proposal

[ed. note: revisions prior to 1.7 are available in the CVS historyof Misc/unicode.txt from the standard Python distribution. Allsubsequent history is available via the CVS revisions on thisfile.]