Unicode Type¶

These are the basic Unicode object types used for the Unicode implementation inPython:

Py_UCS4¶

Py_UCS2¶

Py_UCS1¶

These types are typedefs for unsigned integer types wide enough to containcharacters of 32 bits, 16 bits and 8 bits, respectively. When dealing withsingle Unicode characters, usePy_UCS4.

New in version 3.3.

Py_UNICODE¶

This is a typedef ofwchar_t, which is a 16-bit type or 32-bit typedepending on the platform.

Changed in version 3.3:In previous versions, this was a 16-bit type or a 32-bit type depending onwhether you selected a “narrow” or “wide” Unicode version of Python atbuild time.

PyASCIIObject¶

PyCompactUnicodeObject¶

PyUnicodeObject¶

These subtypes ofPyObject represent a Python Unicode object. Inalmost all cases, they shouldn’t be used directly, since all API functionsthat deal with Unicode objects take and returnPyObject pointers.

New in version 3.3.

PyTypeObjectPyUnicode_Type¶

This instance ofPyTypeObject represents the Python Unicode type. Itis exposed to Python code asstr.

The following APIs are really C macros and can be used to do fast checks and toaccess internal read-only data of Unicode objects:

intPyUnicode_Check(PyObject *o)¶

Return true if the objecto is a Unicode object or an instance of a Unicodesubtype. This function always succeeds.

intPyUnicode_CheckExact(PyObject *o)¶

Return true if the objecto is a Unicode object, but not an instance of asubtype. This function always succeeds.

intPyUnicode_READY(PyObject *o)¶

Ensure the string objecto is in the “canonical” representation. This isrequired before using any of the access macros described below.

Returns0 on success and-1 with an exception set on failure, which inparticular happens if memory allocation fails.

New in version 3.3.

Deprecated since version 3.10, will be removed in version 3.12:This API will be removed withPyUnicode_FromUnicode().

Py_ssize_tPyUnicode_GET_LENGTH(PyObject *o)¶

Return the length of the Unicode string, in code points.o has to be aUnicode object in the “canonical” representation (not checked).

New in version 3.3.

Py_UCS1*PyUnicode_1BYTE_DATA(PyObject *o)¶

Py_UCS2*PyUnicode_2BYTE_DATA(PyObject *o)¶

Py_UCS4*PyUnicode_4BYTE_DATA(PyObject *o)¶

Return a pointer to the canonical representation cast to UCS1, UCS2 or UCS4integer types for direct character access. No checks are performed if thecanonical representation has the correct character size; usePyUnicode_KIND() to select the right macro. Make surePyUnicode_READY() has been called before accessing this.

New in version 3.3.

PyUnicode_WCHAR_KIND¶

PyUnicode_1BYTE_KIND¶

PyUnicode_2BYTE_KIND¶

PyUnicode_4BYTE_KIND¶

Return values of thePyUnicode_KIND() macro.

New in version 3.3.

Deprecated since version 3.10, will be removed in version 3.12:PyUnicode_WCHAR_KIND is deprecated.

unsigned intPyUnicode_KIND(PyObject *o)¶

Return one of the PyUnicode kind constants (see above) that indicate how manybytes per character this Unicode object uses to store its data.o has tobe a Unicode object in the “canonical” representation (not checked).

New in version 3.3.

void*PyUnicode_DATA(PyObject *o)¶

Return a void pointer to the raw Unicode buffer.o has to be a Unicodeobject in the “canonical” representation (not checked).

New in version 3.3.

voidPyUnicode_WRITE(int kind, void *data,Py_ssize_t index,Py_UCS4 value)¶

Write into a canonical representationdata (as obtained withPyUnicode_DATA()). This macro does not do any sanity checks and isintended for usage in loops. The caller should cache thekind value anddata pointer as obtained from other macro calls.index is the index inthe string (starts at 0) andvalue is the new code point value which shouldbe written to that location.

New in version 3.3.

Py_UCS4PyUnicode_READ(int kind, void *data,Py_ssize_t index)¶

Read a code point from a canonical representationdata (as obtained withPyUnicode_DATA()). No checks or ready calls are performed.

New in version 3.3.

Py_UCS4PyUnicode_READ_CHAR(PyObject *o,Py_ssize_t index)¶

Read a character from a Unicode objecto, which must be in the “canonical”representation. This is less efficient thanPyUnicode_READ() if youdo multiple consecutive reads.

New in version 3.3.

PyUnicode_MAX_CHAR_VALUE(o)¶

Return the maximum code point that is suitable for creating another stringbased ono, which must be in the “canonical” representation. This isalways an approximation but more efficient than iterating over the string.

New in version 3.3.

Py_ssize_tPyUnicode_GET_SIZE(PyObject *o)¶

Return the size of the deprecatedPy_UNICODE representation, incode units (this includes surrogate pairs as 2 units).o has to be aUnicode object (not checked).

Deprecated since version 3.3, will be removed in version 3.12:Part of the old-style Unicode API, please migrate to usingPyUnicode_GET_LENGTH().

Py_ssize_tPyUnicode_GET_DATA_SIZE(PyObject *o)¶

Return the size of the deprecatedPy_UNICODE representation inbytes.o has to be a Unicode object (not checked).

Deprecated since version 3.3, will be removed in version 3.12:Part of the old-style Unicode API, please migrate to usingPyUnicode_GET_LENGTH().

Py_UNICODE*PyUnicode_AS_UNICODE(PyObject *o)¶

const char*PyUnicode_AS_DATA(PyObject *o)¶

Return a pointer to aPy_UNICODE representation of the object. Thereturned buffer is always terminated with an extra null code point. Itmay also contain embedded null code points, which would cause the stringto be truncated when used in most C functions. TheAS_DATA formcasts the pointer toconstchar*. Theo argument has to bea Unicode object (not checked).

Changed in version 3.3:This macro is now inefficient – because in many cases thePy_UNICODE representation does not exist and needs to be created– and can fail (returnNULL with an exception set). Try to port thecode to use the newPyUnicode_nBYTE_DATA() macros or usePyUnicode_WRITE() orPyUnicode_READ().

Deprecated since version 3.3, will be removed in version 3.12:Part of the old-style Unicode API, please migrate to using thePyUnicode_nBYTE_DATA() family of macros.

intPyUnicode_IsIdentifier(PyObject *o)¶

Return1 if the string is a valid identifier according to the languagedefinition, sectionIdentifiers and keywords. Return0 otherwise.

Changed in version 3.9:The function does not callPy_FatalError() anymore if the stringis not ready.

Unicode Character Properties¶

Unicode provides many different character properties. The most often needed onesare available through these macros which are mapped to C functions depending onthe Python configuration.

intPy_UNICODE_ISSPACE(Py_UCS4 ch)¶

Return1 or0 depending on whetherch is a whitespace character.

intPy_UNICODE_ISLOWER(Py_UCS4 ch)¶

Return1 or0 depending on whetherch is a lowercase character.

intPy_UNICODE_ISUPPER(Py_UCS4 ch)¶

Return1 or0 depending on whetherch is an uppercase character.

intPy_UNICODE_ISTITLE(Py_UCS4 ch)¶

Return1 or0 depending on whetherch is a titlecase character.

intPy_UNICODE_ISLINEBREAK(Py_UCS4 ch)¶

Return1 or0 depending on whetherch is a linebreak character.

intPy_UNICODE_ISDECIMAL(Py_UCS4 ch)¶

Return1 or0 depending on whetherch is a decimal character.

intPy_UNICODE_ISDIGIT(Py_UCS4 ch)¶

Return1 or0 depending on whetherch is a digit character.

intPy_UNICODE_ISNUMERIC(Py_UCS4 ch)¶

Return1 or0 depending on whetherch is a numeric character.

intPy_UNICODE_ISALPHA(Py_UCS4 ch)¶

Return1 or0 depending on whetherch is an alphabetic character.

intPy_UNICODE_ISALNUM(Py_UCS4 ch)¶

Return1 or0 depending on whetherch is an alphanumeric character.

intPy_UNICODE_ISPRINTABLE(Py_UCS4 ch)¶

Return1 or0 depending on whetherch is a printable character.Nonprintable characters are those characters defined in the Unicode characterdatabase as “Other” or “Separator”, excepting the ASCII space (0x20) which isconsidered printable. (Note that printable characters in this context arethose which should not be escaped whenrepr() is invoked on a string.It has no bearing on the handling of strings written tosys.stdout orsys.stderr.)

These APIs can be used for fast direct character conversions:

Py_UCS4Py_UNICODE_TOLOWER(Py_UCS4 ch)¶

Return the characterch converted to lower case.

Deprecated since version 3.3:This function uses simple case mappings.

Py_UCS4Py_UNICODE_TOUPPER(Py_UCS4 ch)¶

Return the characterch converted to upper case.

Deprecated since version 3.3:This function uses simple case mappings.

Py_UCS4Py_UNICODE_TOTITLE(Py_UCS4 ch)¶

Return the characterch converted to title case.

Deprecated since version 3.3:This function uses simple case mappings.

intPy_UNICODE_TODECIMAL(Py_UCS4 ch)¶

Return the characterch converted to a decimal positive integer. Return-1 if this is not possible. This macro does not raise exceptions.

intPy_UNICODE_TODIGIT(Py_UCS4 ch)¶

Return the characterch converted to a single digit integer. Return-1 ifthis is not possible. This macro does not raise exceptions.

doublePy_UNICODE_TONUMERIC(Py_UCS4 ch)¶

Return the characterch converted to a double. Return-1.0 if this is notpossible. This macro does not raise exceptions.

These APIs can be used to work with surrogates:

Py_UNICODE_IS_SURROGATE(ch)¶

Check ifch is a surrogate (0xD800<=ch<=0xDFFF).

Py_UNICODE_IS_HIGH_SURROGATE(ch)¶

Check ifch is a high surrogate (0xD800<=ch<=0xDBFF).

Py_UNICODE_IS_LOW_SURROGATE(ch)¶

Check ifch is a low surrogate (0xDC00<=ch<=0xDFFF).

Py_UNICODE_JOIN_SURROGATES(high, low)¶

Join two surrogate characters and return a single Py_UCS4 value.high andlow are respectively the leading and trailing surrogates in asurrogate pair.

Creating and accessing Unicode strings¶

To create Unicode objects and access their basic sequence properties, use theseAPIs:

PyObject*PyUnicode_New(Py_ssize_t size,Py_UCS4 maxchar)¶

Return value: New reference.

Create a new Unicode object.maxchar should be the true maximum code pointto be placed in the string. As an approximation, it can be rounded up to thenearest value in the sequence 127, 255, 65535, 1114111.

This is the recommended way to allocate a new Unicode object. Objectscreated using this function are not resizable.

New in version 3.3.

PyObject*PyUnicode_FromKindAndData(int kind, const void *buffer,Py_ssize_t size)¶

Return value: New reference.

Create a new Unicode object with the givenkind (possible values arePyUnicode_1BYTE_KIND etc., as returned byPyUnicode_KIND()). Thebuffer must point to an array ofsizeunits of 1, 2 or 4 bytes per character, as given by the kind.

New in version 3.3.

PyObject*PyUnicode_FromStringAndSize(const char *u,Py_ssize_t size)¶

Return value: New reference.

Create a Unicode object from the char bufferu. The bytes will beinterpreted as being UTF-8 encoded. The buffer is copied into the newobject. If the buffer is notNULL, the return value might be a sharedobject, i.e. modification of the data is not allowed.

Ifu isNULL, this function behaves likePyUnicode_FromUnicode()with the buffer set toNULL. This usage is deprecated in favor ofPyUnicode_New(), and will be removed in Python 3.12.

PyObject *PyUnicode_FromString(const char *u)¶

Return value: New reference.

Create a Unicode object from a UTF-8 encoded null-terminated char bufferu.

PyObject*PyUnicode_FromFormat(const char *format, ...)¶

Return value: New reference.

Take a Cprintf()-styleformat string and a variable number ofarguments, calculate the size of the resulting Python Unicode string and returna string with the values formatted into it. The variable arguments must be Ctypes and must correspond exactly to the format characters in theformatASCII-encoded string. The following format characters are allowed:

Format Characters	Type	Comment
%%	n/a	The literal % character.
%c	int	A single character,represented as a C int.
%d	int	Equivalent toprintf("%d").1
%u	unsigned int	Equivalent toprintf("%u").1
%ld	long	Equivalent toprintf("%ld").1
%li	long	Equivalent toprintf("%li").1
%lu	unsigned long	Equivalent toprintf("%lu").1
%lld	long long	Equivalent toprintf("%lld").1
%lli	long long	Equivalent toprintf("%lli").1
%llu	unsigned long long	Equivalent toprintf("%llu").1
%zd	Py_ssize_t	Equivalent toprintf("%zd").1
%zi	Py_ssize_t	Equivalent toprintf("%zi").1
%zu	size_t	Equivalent toprintf("%zu").1
%i	int	Equivalent toprintf("%i").1
%x	int	Equivalent toprintf("%x").1
%s	const char*	A null-terminated C characterarray.
%p	const void*	The hex representation of a Cpointer. Mostly equivalent toprintf("%p") except thatit is guaranteed to start withthe literal0x regardlessof what the platform’sprintf yields.
%A	PyObject*	The result of callingascii().
%U	PyObject*	A Unicode object.
%V	PyObject*,const char*	A Unicode object (which may beNULL) and a null-terminatedC character array as a secondparameter (which will be used,if the first parameter isNULL).
%S	PyObject*	The result of callingPyObject_Str().
%R	PyObject*	The result of callingPyObject_Repr().

An unrecognized format character causes all the rest of the format string to becopied as-is to the result string, and any extra arguments discarded.

Note

The width formatter unit is number of characters rather than bytes.The precision formatter unit is number of bytes for"%s" and"%V" (if thePyObject* argument isNULL), and a number ofcharacters for"%A","%U","%S","%R" and"%V"(if thePyObject* argument is notNULL).

1(1,2,3,4,5,6,7,8,9,10,11,12,13)

For integer specifiers (d, u, ld, li, lu, lld, lli, llu, zd, zi,zu, i, x): the 0-conversion flag has effect even when a precision is given.

Changed in version 3.2:Support for"%lld" and"%llu" added.

Changed in version 3.3:Support for"%li","%lli" and"%zi" added.

Changed in version 3.4:Support width and precision formatter for"%s","%A","%U","%V","%S","%R" added.

PyObject*PyUnicode_FromFormatV(const char *format, va_list vargs)¶

Return value: New reference.

Identical toPyUnicode_FromFormat() except that it takes exactly twoarguments.

PyObject*PyUnicode_FromEncodedObject(PyObject *obj, const char *encoding, const char *errors)¶

Return value: New reference.

Decode an encoded objectobj to a Unicode object.

bytes,bytearray and otherbytes-like objectsare decoded according to the givenencoding and using the error handlingdefined byerrors. Both can beNULL to have the interface use the defaultvalues (seeBuilt-in Codecs for details).

All other objects, including Unicode objects, cause aTypeError to beset.

The API returnsNULL if there was an error. The caller is responsible fordecref’ing the returned objects.

Py_ssize_tPyUnicode_GetLength(PyObject *unicode)¶

Return the length of the Unicode object, in code points.

New in version 3.3.

Py_ssize_tPyUnicode_CopyCharacters(PyObject *to,Py_ssize_t to_start,PyObject *from,Py_ssize_t from_start,Py_ssize_t how_many)¶

Copy characters from one Unicode object into another. This function performscharacter conversion when necessary and falls back tomemcpy() ifpossible. Returns-1 and sets an exception on error, otherwise returnsthe number of copied characters.

New in version 3.3.

Py_ssize_tPyUnicode_Fill(PyObject *unicode,Py_ssize_t start,Py_ssize_t length,Py_UCS4 fill_char)¶

Fill a string with a character: writefill_char intounicode[start:start+length].

Fail iffill_char is bigger than the string maximum character, or if thestring has more than 1 reference.

Return the number of written character, or return-1 and raise anexception on error.

New in version 3.3.

intPyUnicode_WriteChar(PyObject *unicode,Py_ssize_t index,Py_UCS4 character)¶

Write a character to a string. The string must have been created throughPyUnicode_New(). Since Unicode strings are supposed to be immutable,the string must not be shared, or have been hashed yet.

This function checks thatunicode is a Unicode object, that the index isnot out of bounds, and that the object can be modified safely (i.e. that itits reference count is one).

New in version 3.3.

Py_UCS4PyUnicode_ReadChar(PyObject *unicode,Py_ssize_t index)¶

Read a character from a string. This function checks thatunicode is aUnicode object and the index is not out of bounds, in contrast to the macroversionPyUnicode_READ_CHAR().

New in version 3.3.

PyObject*PyUnicode_Substring(PyObject *str,Py_ssize_t start,Py_ssize_t end)¶

Return value: New reference.

Return a substring ofstr, from character indexstart (included) tocharacter indexend (excluded). Negative indices are not supported.

New in version 3.3.

Py_UCS4*PyUnicode_AsUCS4(PyObject *u,Py_UCS4 *buffer,Py_ssize_t buflen, int copy_null)¶

Copy the stringu into a UCS4 buffer, including a null character, ifcopy_null is set. ReturnsNULL and sets an exception on error (inparticular, aSystemError ifbuflen is smaller than the length ofu).buffer is returned on success.

New in version 3.3.

Py_UCS4*PyUnicode_AsUCS4Copy(PyObject *u)¶

Copy the stringu into a new UCS4 buffer that is allocated usingPyMem_Malloc(). If this fails,NULL is returned with aMemoryError set. The returned buffer always has an extranull code point appended.

New in version 3.3.

Deprecated Py_UNICODE APIs¶

These API functions are deprecated with the implementation ofPEP 393.Extension modules can continue using them, as they will not be removed in Python3.x, but need to be aware that their use can now cause performance and memory hits.

PyObject*PyUnicode_FromUnicode(constPy_UNICODE *u,Py_ssize_t size)¶

Return value: New reference.

Create a Unicode object from the Py_UNICODE bufferu of the given size.umay beNULL which causes the contents to be undefined. It is the user’sresponsibility to fill in the needed data. The buffer is copied into the newobject.

If the buffer is notNULL, the return value might be a shared object.Therefore, modification of the resulting Unicode object is only allowed whenu isNULL.

If the buffer isNULL,PyUnicode_READY() must be called once thestring content has been filled before using any of the access macros such asPyUnicode_KIND().

Deprecated since version 3.3, will be removed in version 3.12:Part of the old-style Unicode API, please migrate to usingPyUnicode_FromKindAndData(),PyUnicode_FromWideChar(), orPyUnicode_New().

Py_UNICODE*PyUnicode_AsUnicode(PyObject *unicode)¶

Return a read-only pointer to the Unicode object’s internalPy_UNICODE buffer, orNULL on error. This will create thePy_UNICODE* representation of the object if it is not yetavailable. The buffer is always terminated with an extra null code point.Note that the resultingPy_UNICODE string may also containembedded null code points, which would cause the string to be truncated whenused in most C functions.

Deprecated since version 3.3, will be removed in version 3.12:Part of the old-style Unicode API, please migrate to usingPyUnicode_AsUCS4(),PyUnicode_AsWideChar(),PyUnicode_ReadChar() or similar new APIs.

Deprecated since version 3.3, will be removed in version 3.10.

PyObject*PyUnicode_TransformDecimalToASCII(Py_UNICODE *s,Py_ssize_t size)¶

Return value: New reference.

Create a Unicode object by replacing all decimal digits inPy_UNICODE buffer of the givensize by ASCII digits 0–9according to their decimal value. ReturnNULL if an exception occurs.

Deprecated since version 3.3, will be removed in version 3.11:Part of the old-stylePy_UNICODE API; please migrate to usingPy_UNICODE_TODECIMAL().

Py_UNICODE*PyUnicode_AsUnicodeAndSize(PyObject *unicode,Py_ssize_t *size)¶

LikePyUnicode_AsUnicode(), but also saves thePy_UNICODE()array length (excluding the extra null terminator) insize.Note that the resultingPy_UNICODE* stringmay contain embedded null code points, which would cause the string to betruncated when used in most C functions.

New in version 3.3.

Deprecated since version 3.3, will be removed in version 3.12:Part of the old-style Unicode API, please migrate to usingPyUnicode_AsUCS4(),PyUnicode_AsWideChar(),PyUnicode_ReadChar() or similar new APIs.

Py_UNICODE*PyUnicode_AsUnicodeCopy(PyObject *unicode)¶

Create a copy of a Unicode string ending with a null code point. ReturnNULLand raise aMemoryError exception on memory allocation failure,otherwise return a new allocated buffer (usePyMem_Free() to freethe buffer). Note that the resultingPy_UNICODE* string maycontain embedded null code points, which would cause the string to betruncated when used in most C functions.

New in version 3.2.

Please migrate to usingPyUnicode_AsUCS4Copy() or similar new APIs.

Py_ssize_tPyUnicode_GetSize(PyObject *unicode)¶

Return the size of the deprecatedPy_UNICODE representation, incode units (this includes surrogate pairs as 2 units).

Deprecated since version 3.3, will be removed in version 3.12:Part of the old-style Unicode API, please migrate to usingPyUnicode_GET_LENGTH().

PyObject*PyUnicode_FromObject(PyObject *obj)¶

Return value: New reference.

Copy an instance of a Unicode subtype to a new true Unicode object ifnecessary. Ifobj is already a true Unicode object (not a subtype),return the reference with incremented refcount.

Objects other than Unicode or its subtypes will cause aTypeError.

Locale Encoding¶

The current locale encoding can be used to decode text from the operatingsystem.

PyObject*PyUnicode_DecodeLocaleAndSize(const char *str,Py_ssize_t len, const char *errors)¶

Return value: New reference.

Decode a string from UTF-8 on Android and VxWorks, or from the currentlocale encoding on other platforms. The supportederror handlers are"strict" and"surrogateescape"(PEP 383). The decoder uses"strict" error handler iferrors isNULL.str must end with a null character butcannot contain embedded null characters.

UsePyUnicode_DecodeFSDefaultAndSize() to decode a string fromPy_FileSystemDefaultEncoding (the locale encoding read atPython startup).

This function ignores the Python UTF-8 mode.

See also

ThePy_DecodeLocale() function.

New in version 3.3.

Changed in version 3.7:The function now also uses the current locale encoding for thesurrogateescape error handler, except on Android. Previously,Py_DecodeLocale()was used for thesurrogateescape, and the current locale encoding wasused forstrict.

PyObject*PyUnicode_DecodeLocale(const char *str, const char *errors)¶

Return value: New reference.

Similar toPyUnicode_DecodeLocaleAndSize(), but compute the stringlength usingstrlen().

New in version 3.3.

PyObject*PyUnicode_EncodeLocale(PyObject *unicode, const char *errors)¶

Return value: New reference.

Encode a Unicode object to UTF-8 on Android and VxWorks, or to the currentlocale encoding on other platforms. Thesupported error handlers are"strict" and"surrogateescape"(PEP 383). The encoder uses"strict" error handler iferrors isNULL. Return abytes object.unicode cannotcontain embedded null characters.

UsePyUnicode_EncodeFSDefault() to encode a string toPy_FileSystemDefaultEncoding (the locale encoding read atPython startup).

This function ignores the Python UTF-8 mode.

See also

ThePy_EncodeLocale() function.

New in version 3.3.

Changed in version 3.7:The function now also uses the current locale encoding for thesurrogateescape error handler, except on Android. Previously,Py_EncodeLocale()was used for thesurrogateescape, and the current locale encoding wasused forstrict.

File System Encoding¶

To encode and decode file names and other environment strings,Py_FileSystemDefaultEncoding should be used as the encoding, andPy_FileSystemDefaultEncodeErrors should be used as the error handler(PEP 383 andPEP 529). To encode file names tobytes duringargument parsing, the"O&" converter should be used, passingPyUnicode_FSConverter() as the conversion function:

intPyUnicode_FSConverter(PyObject* obj, void* result)¶

ParseTuple converter: encodestr objects – obtained directly orthrough theos.PathLike interface – tobytes usingPyUnicode_EncodeFSDefault();bytes objects are output as-is.result must be aPyBytesObject* which must be released when it isno longer used.

New in version 3.1.

Changed in version 3.6:Accepts apath-like object.

To decode file names tostr during argument parsing, the"O&"converter should be used, passingPyUnicode_FSDecoder() as theconversion function:

intPyUnicode_FSDecoder(PyObject* obj, void* result)¶

ParseTuple converter: decodebytes objects – obtained eitherdirectly or indirectly through theos.PathLike interface – tostr usingPyUnicode_DecodeFSDefaultAndSize();strobjects are output as-is.result must be aPyUnicodeObject* whichmust be released when it is no longer used.

New in version 3.2.

Changed in version 3.6:Accepts apath-like object.

PyObject*PyUnicode_DecodeFSDefaultAndSize(const char *s,Py_ssize_t size)¶

Return value: New reference.

Decode a string usingPy_FileSystemDefaultEncoding and thePy_FileSystemDefaultEncodeErrors error handler.

IfPy_FileSystemDefaultEncoding is not set, fall back to thelocale encoding.

Py_FileSystemDefaultEncoding is initialized at startup from thelocale encoding and cannot be modified later. If you need to decode a stringfrom the current locale encoding, usePyUnicode_DecodeLocaleAndSize().

See also

ThePy_DecodeLocale() function.

Changed in version 3.6:UsePy_FileSystemDefaultEncodeErrors error handler.

PyObject*PyUnicode_DecodeFSDefault(const char *s)¶

Return value: New reference.

Decode a null-terminated string usingPy_FileSystemDefaultEncodingand thePy_FileSystemDefaultEncodeErrors error handler.

IfPy_FileSystemDefaultEncoding is not set, fall back to thelocale encoding.

UsePyUnicode_DecodeFSDefaultAndSize() if you know the string length.

Changed in version 3.6:UsePy_FileSystemDefaultEncodeErrors error handler.

PyObject*PyUnicode_EncodeFSDefault(PyObject *unicode)¶

Return value: New reference.

Encode a Unicode object toPy_FileSystemDefaultEncoding with thePy_FileSystemDefaultEncodeErrors error handler, and returnbytes. Note that the resultingbytes object may containnull bytes.

IfPy_FileSystemDefaultEncoding is not set, fall back to thelocale encoding.

Py_FileSystemDefaultEncoding is initialized at startup from thelocale encoding and cannot be modified later. If you need to encode a stringto the current locale encoding, usePyUnicode_EncodeLocale().

See also

ThePy_EncodeLocale() function.

New in version 3.2.

Changed in version 3.6:UsePy_FileSystemDefaultEncodeErrors error handler.

wchar_t Support¶

wchar_t support for platforms which support it:

PyObject*PyUnicode_FromWideChar(const wchar_t *w,Py_ssize_t size)¶

Return value: New reference.

Create a Unicode object from thewchar_t bufferw of the givensize.Passing-1 as thesize indicates that the function must itself compute the length,using wcslen.ReturnNULL on failure.

Py_ssize_tPyUnicode_AsWideChar(PyObject *unicode, wchar_t *w,Py_ssize_t size)¶

Copy the Unicode object contents into thewchar_t bufferw. At mostsizewchar_t characters are copied (excluding a possibly trailingnull termination character). Return the number ofwchar_t characterscopied or-1 in case of an error. Note that the resultingwchar_t*string may or may not be null-terminated. It is the responsibility of the callerto make sure that thewchar_t* string is null-terminated in case this isrequired by the application. Also, note that thewchar_t* stringmight contain null characters, which would cause the string to be truncatedwhen used with most C functions.

wchar_t*PyUnicode_AsWideCharString(PyObject *unicode,Py_ssize_t *size)¶

Convert the Unicode object to a wide character string. The output stringalways ends with a null character. Ifsize is notNULL, write the numberof wide characters (excluding the trailing null termination character) into*size. Note that the resultingwchar_t string might containnull characters, which would cause the string to be truncated when used withmost C functions. Ifsize isNULL and thewchar_t* stringcontains null characters aValueError is raised.

Returns a buffer allocated byPyMem_Alloc() (usePyMem_Free() to free it) on success. On error, returnsNULLand*size is undefined. Raises aMemoryError if memory allocationis failed.

New in version 3.2.

Changed in version 3.7:Raises aValueError ifsize isNULL and thewchar_t*string contains null characters.

Generic Codecs¶

These are the generic codec APIs:

PyObject*PyUnicode_Decode(const char *s,Py_ssize_t size, const char *encoding, const char *errors)¶

Return value: New reference.

Create a Unicode object by decodingsize bytes of the encoded strings.encoding anderrors have the same meaning as the parameters of the same namein thestr() built-in function. The codec to be used is looked upusing the Python codec registry. ReturnNULL if an exception was raised bythe codec.

PyObject*PyUnicode_AsEncodedString(PyObject *unicode, const char *encoding, const char *errors)¶

Return value: New reference.

Encode a Unicode object and return the result as Python bytes object.encoding anderrors have the same meaning as the parameters of the samename in the Unicodeencode() method. The codec to be used is looked upusing the Python codec registry. ReturnNULL if an exception was raised bythe codec.

PyObject*PyUnicode_Encode(constPy_UNICODE *s,Py_ssize_t size, const char *encoding, const char *errors)¶

Return value: New reference.

Encode thePy_UNICODE buffers of the givensize and return a Pythonbytes object.encoding anderrors have the same meaning as theparameters of the same name in the Unicodeencode() method. The codecto be used is looked up using the Python codec registry. ReturnNULL if anexception was raised by the codec.

Deprecated since version 3.3, will be removed in version 3.11:Part of the old-stylePy_UNICODE API; please migrate to usingPyUnicode_AsEncodedString().

UTF-8 Codecs¶

These are the UTF-8 codec APIs:

PyObject*PyUnicode_DecodeUTF8(const char *s,Py_ssize_t size, const char *errors)¶

Return value: New reference.

Create a Unicode object by decodingsize bytes of the UTF-8 encoded strings. ReturnNULL if an exception was raised by the codec.

PyObject*PyUnicode_DecodeUTF8Stateful(const char *s,Py_ssize_t size, const char *errors,Py_ssize_t *consumed)¶

Return value: New reference.

Ifconsumed isNULL, behave likePyUnicode_DecodeUTF8(). Ifconsumed is notNULL, trailing incomplete UTF-8 byte sequences will not betreated as an error. Those bytes will not be decoded and the number of bytesthat have been decoded will be stored inconsumed.

PyObject*PyUnicode_AsUTF8String(PyObject *unicode)¶

Return value: New reference.

Encode a Unicode object using UTF-8 and return the result as Python bytesobject. Error handling is “strict”. ReturnNULL if an exception wasraised by the codec.

const char*PyUnicode_AsUTF8AndSize(PyObject *unicode,Py_ssize_t *size)¶

Return a pointer to the UTF-8 encoding of the Unicode object, andstore the size of the encoded representation (in bytes) insize. Thesize argument can beNULL; in this case no size will be stored. Thereturned buffer always has an extra null byte appended (not included insize), regardless of whether there are any other null code points.

In the case of an error,NULL is returned with an exception set and nosize is stored.

This caches the UTF-8 representation of the string in the Unicode object, andsubsequent calls will return a pointer to the same buffer. The caller is notresponsible for deallocating the buffer. The buffer is deallocated andpointers to it become invalid when the Unicode object is garbage collected.

New in version 3.3.

Changed in version 3.7:The return type is nowconstchar* rather ofchar*.

const char*PyUnicode_AsUTF8(PyObject *unicode)¶

AsPyUnicode_AsUTF8AndSize(), but does not store the size.

New in version 3.3.

Changed in version 3.7:The return type is nowconstchar* rather ofchar*.

PyObject*PyUnicode_EncodeUTF8(constPy_UNICODE *s,Py_ssize_t size, const char *errors)¶

Return value: New reference.

Encode thePy_UNICODE buffers of the givensize using UTF-8 andreturn a Python bytes object. ReturnNULL if an exception was raised bythe codec.

Deprecated since version 3.3, will be removed in version 3.11:Part of the old-stylePy_UNICODE API; please migrate to usingPyUnicode_AsUTF8String(),PyUnicode_AsUTF8AndSize() orPyUnicode_AsEncodedString().

UTF-32 Codecs¶

These are the UTF-32 codec APIs:

PyObject*PyUnicode_DecodeUTF32(const char *s,Py_ssize_t size, const char *errors, int *byteorder)¶

Return value: New reference.

Decodesize bytes from a UTF-32 encoded buffer string and return thecorresponding Unicode object.errors (if non-NULL) defines the errorhandling. It defaults to “strict”.

Ifbyteorder is non-NULL, the decoder starts decoding using the given byteorder:

*byteorder==-1:littleendian*byteorder==0:nativeorder*byteorder==1:bigendian

If*byteorder is zero, and the first four bytes of the input data are abyte order mark (BOM), the decoder switches to this byte order and the BOM isnot copied into the resulting Unicode string. If*byteorder is-1 or1, any byte order mark is copied to the output.

After completion,*byteorder is set to the current byte order at the endof input data.

Ifbyteorder isNULL, the codec starts in native order mode.

ReturnNULL if an exception was raised by the codec.

PyObject*PyUnicode_DecodeUTF32Stateful(const char *s,Py_ssize_t size, const char *errors, int *byteorder,Py_ssize_t *consumed)¶

Return value: New reference.

Ifconsumed isNULL, behave likePyUnicode_DecodeUTF32(). Ifconsumed is notNULL,PyUnicode_DecodeUTF32Stateful() will not treattrailing incomplete UTF-32 byte sequences (such as a number of bytes not divisibleby four) as an error. Those bytes will not be decoded and the number of bytesthat have been decoded will be stored inconsumed.

PyObject*PyUnicode_AsUTF32String(PyObject *unicode)¶

Return value: New reference.

Return a Python byte string using the UTF-32 encoding in native byteorder. The string always starts with a BOM mark. Error handling is “strict”.ReturnNULL if an exception was raised by the codec.

PyObject*PyUnicode_EncodeUTF32(constPy_UNICODE *s,Py_ssize_t size, const char *errors, int byteorder)¶

Return value: New reference.

Return a Python bytes object holding the UTF-32 encoded value of the Unicodedata ins. Output is written according to the following byte order:

byteorder==-1:littleendianbyteorder==0:nativebyteorder(writesaBOMmark)byteorder==1:bigendian

If byteorder is0, the output string will always start with the Unicode BOMmark (U+FEFF). In the other two modes, no BOM mark is prepended.

IfPy_UNICODE_WIDE is not defined, surrogate pairs will be outputas a single code point.

ReturnNULL if an exception was raised by the codec.

Deprecated since version 3.3, will be removed in version 3.11:Part of the old-stylePy_UNICODE API; please migrate to usingPyUnicode_AsUTF32String() orPyUnicode_AsEncodedString().

UTF-16 Codecs¶

These are the UTF-16 codec APIs:

PyObject*PyUnicode_DecodeUTF16(const char *s,Py_ssize_t size, const char *errors, int *byteorder)¶

Return value: New reference.

Decodesize bytes from a UTF-16 encoded buffer string and return thecorresponding Unicode object.errors (if non-NULL) defines the errorhandling. It defaults to “strict”.

Ifbyteorder is non-NULL, the decoder starts decoding using the given byteorder:

*byteorder==-1:littleendian*byteorder==0:nativeorder*byteorder==1:bigendian

If*byteorder is zero, and the first two bytes of the input data are abyte order mark (BOM), the decoder switches to this byte order and the BOM isnot copied into the resulting Unicode string. If*byteorder is-1 or1, any byte order mark is copied to the output (where it will result ineither a\ufeff or a\ufffe character).

After completion,*byteorder is set to the current byte order at the endof input data.

Ifbyteorder isNULL, the codec starts in native order mode.

ReturnNULL if an exception was raised by the codec.

PyObject*PyUnicode_DecodeUTF16Stateful(const char *s,Py_ssize_t size, const char *errors, int *byteorder,Py_ssize_t *consumed)¶

Return value: New reference.

Ifconsumed isNULL, behave likePyUnicode_DecodeUTF16(). Ifconsumed is notNULL,PyUnicode_DecodeUTF16Stateful() will not treattrailing incomplete UTF-16 byte sequences (such as an odd number of bytes or asplit surrogate pair) as an error. Those bytes will not be decoded and thenumber of bytes that have been decoded will be stored inconsumed.

PyObject*PyUnicode_AsUTF16String(PyObject *unicode)¶

Return value: New reference.

Return a Python byte string using the UTF-16 encoding in native byteorder. The string always starts with a BOM mark. Error handling is “strict”.ReturnNULL if an exception was raised by the codec.

PyObject*PyUnicode_EncodeUTF16(constPy_UNICODE *s,Py_ssize_t size, const char *errors, int byteorder)¶

Return value: New reference.

Return a Python bytes object holding the UTF-16 encoded value of the Unicodedata ins. Output is written according to the following byte order:

byteorder==-1:littleendianbyteorder==0:nativebyteorder(writesaBOMmark)byteorder==1:bigendian

If byteorder is0, the output string will always start with the Unicode BOMmark (U+FEFF). In the other two modes, no BOM mark is prepended.

IfPy_UNICODE_WIDE is defined, a singlePy_UNICODE value may getrepresented as a surrogate pair. If it is not defined, eachPy_UNICODEvalues is interpreted as a UCS-2 character.

ReturnNULL if an exception was raised by the codec.

Deprecated since version 3.3, will be removed in version 3.11:Part of the old-stylePy_UNICODE API; please migrate to usingPyUnicode_AsUTF16String() orPyUnicode_AsEncodedString().

UTF-7 Codecs¶

These are the UTF-7 codec APIs:

PyObject*PyUnicode_DecodeUTF7(const char *s,Py_ssize_t size, const char *errors)¶

Return value: New reference.

Create a Unicode object by decodingsize bytes of the UTF-7 encoded strings. ReturnNULL if an exception was raised by the codec.

PyObject*PyUnicode_DecodeUTF7Stateful(const char *s,Py_ssize_t size, const char *errors,Py_ssize_t *consumed)¶

Return value: New reference.

Ifconsumed isNULL, behave likePyUnicode_DecodeUTF7(). Ifconsumed is notNULL, trailing incomplete UTF-7 base-64 sections will notbe treated as an error. Those bytes will not be decoded and the number ofbytes that have been decoded will be stored inconsumed.

PyObject*PyUnicode_EncodeUTF7(constPy_UNICODE *s,Py_ssize_t size, int base64SetO, int base64WhiteSpace, const char *errors)¶

Return value: New reference.

Encode thePy_UNICODE buffer of the given size using UTF-7 andreturn a Python bytes object. ReturnNULL if an exception was raised bythe codec.

Ifbase64SetO is nonzero, “Set O” (punctuation that has no otherwisespecial meaning) will be encoded in base-64. Ifbase64WhiteSpace isnonzero, whitespace will be encoded in base-64. Both are set to zero for thePython “utf-7” codec.

Deprecated since version 3.3, will be removed in version 3.11:Part of the old-stylePy_UNICODE API; please migrate to usingPyUnicode_AsEncodedString().

Unicode-Escape Codecs¶

These are the “Unicode Escape” codec APIs:

PyObject*PyUnicode_DecodeUnicodeEscape(const char *s,Py_ssize_t size, const char *errors)¶

Return value: New reference.

Create a Unicode object by decodingsize bytes of the Unicode-Escape encodedstrings. ReturnNULL if an exception was raised by the codec.

PyObject*PyUnicode_AsUnicodeEscapeString(PyObject *unicode)¶

Return value: New reference.

Encode a Unicode object using Unicode-Escape and return the result as abytes object. Error handling is “strict”. ReturnNULL if an exception wasraised by the codec.

PyObject*PyUnicode_EncodeUnicodeEscape(constPy_UNICODE *s,Py_ssize_t size)¶

Return value: New reference.

Encode thePy_UNICODE buffer of the givensize using Unicode-Escape andreturn a bytes object. ReturnNULL if an exception was raised by the codec.

Deprecated since version 3.3, will be removed in version 3.11:Part of the old-stylePy_UNICODE API; please migrate to usingPyUnicode_AsUnicodeEscapeString().

Raw-Unicode-Escape Codecs¶

These are the “Raw Unicode Escape” codec APIs:

PyObject*PyUnicode_DecodeRawUnicodeEscape(const char *s,Py_ssize_t size, const char *errors)¶

Return value: New reference.

Create a Unicode object by decodingsize bytes of the Raw-Unicode-Escapeencoded strings. ReturnNULL if an exception was raised by the codec.

PyObject*PyUnicode_AsRawUnicodeEscapeString(PyObject *unicode)¶

Return value: New reference.

Encode a Unicode object using Raw-Unicode-Escape and return the result asa bytes object. Error handling is “strict”. ReturnNULL if an exceptionwas raised by the codec.

PyObject*PyUnicode_EncodeRawUnicodeEscape(constPy_UNICODE *s,Py_ssize_t size)¶

Return value: New reference.

Encode thePy_UNICODE buffer of the givensize using Raw-Unicode-Escapeand return a bytes object. ReturnNULL if an exception was raised by the codec.

Deprecated since version 3.3, will be removed in version 3.11:Part of the old-stylePy_UNICODE API; please migrate to usingPyUnicode_AsRawUnicodeEscapeString() orPyUnicode_AsEncodedString().

Latin-1 Codecs¶

These are the Latin-1 codec APIs: Latin-1 corresponds to the first 256 Unicodeordinals and only these are accepted by the codecs during encoding.

PyObject*PyUnicode_DecodeLatin1(const char *s,Py_ssize_t size, const char *errors)¶

Return value: New reference.

Create a Unicode object by decodingsize bytes of the Latin-1 encoded strings. ReturnNULL if an exception was raised by the codec.

PyObject*PyUnicode_AsLatin1String(PyObject *unicode)¶

Return value: New reference.

Encode a Unicode object using Latin-1 and return the result as Python bytesobject. Error handling is “strict”. ReturnNULL if an exception wasraised by the codec.

PyObject*PyUnicode_EncodeLatin1(constPy_UNICODE *s,Py_ssize_t size, const char *errors)¶

Return value: New reference.

Encode thePy_UNICODE buffer of the givensize using Latin-1 andreturn a Python bytes object. ReturnNULL if an exception was raised bythe codec.

Deprecated since version 3.3, will be removed in version 3.11:Part of the old-stylePy_UNICODE API; please migrate to usingPyUnicode_AsLatin1String() orPyUnicode_AsEncodedString().

ASCII Codecs¶

These are the ASCII codec APIs. Only 7-bit ASCII data is accepted. All othercodes generate errors.

PyObject*PyUnicode_DecodeASCII(const char *s,Py_ssize_t size, const char *errors)¶

Return value: New reference.

Create a Unicode object by decodingsize bytes of the ASCII encoded strings. ReturnNULL if an exception was raised by the codec.

PyObject*PyUnicode_AsASCIIString(PyObject *unicode)¶

Return value: New reference.

Encode a Unicode object using ASCII and return the result as Python bytesobject. Error handling is “strict”. ReturnNULL if an exception wasraised by the codec.

PyObject*PyUnicode_EncodeASCII(constPy_UNICODE *s,Py_ssize_t size, const char *errors)¶

Return value: New reference.

Encode thePy_UNICODE buffer of the givensize using ASCII andreturn a Python bytes object. ReturnNULL if an exception was raised bythe codec.

Deprecated since version 3.3, will be removed in version 3.11:Part of the old-stylePy_UNICODE API; please migrate to usingPyUnicode_AsASCIIString() orPyUnicode_AsEncodedString().

Character Map Codecs¶

This codec is special in that it can be used to implement many different codecs(and this is in fact what was done to obtain most of the standard codecsincluded in theencodings package). The codec uses mappings to encode anddecode characters. The mapping objects provided must support the__getitem__() mapping interface; dictionaries and sequences work well.

These are the mapping codec APIs:

PyObject*PyUnicode_DecodeCharmap(const char *data,Py_ssize_t size,PyObject *mapping, const char *errors)¶

Return value: New reference.

Create a Unicode object by decodingsize bytes of the encoded stringsusing the givenmapping object. ReturnNULL if an exception was raisedby the codec.

Ifmapping isNULL, Latin-1 decoding will be applied. Elsemapping must map bytes ordinals (integers in the range from 0 to 255)to Unicode strings, integers (which are then interpreted as Unicodeordinals) orNone. Unmapped data bytes – ones which cause aLookupError, as well as ones which get mapped toNone,0xFFFE or'\ufffe', are treated as undefined mappings and causean error.

PyObject*PyUnicode_AsCharmapString(PyObject *unicode,PyObject *mapping)¶

Return value: New reference.

Encode a Unicode object using the givenmapping object and return theresult as a bytes object. Error handling is “strict”. ReturnNULL if anexception was raised by the codec.

Themapping object must map Unicode ordinal integers to bytes objects,integers in the range from 0 to 255 orNone. Unmapped characterordinals (ones which cause aLookupError) as well as mapped toNone are treated as “undefined mapping” and cause an error.

PyObject*PyUnicode_EncodeCharmap(constPy_UNICODE *s,Py_ssize_t size,PyObject *mapping, const char *errors)¶

Return value: New reference.

Encode thePy_UNICODE buffer of the givensize using the givenmapping object and return the result as a bytes object. ReturnNULL ifan exception was raised by the codec.

Deprecated since version 3.3, will be removed in version 3.11:Part of the old-stylePy_UNICODE API; please migrate to usingPyUnicode_AsCharmapString() orPyUnicode_AsEncodedString().

The following codec API is special in that maps Unicode to Unicode.

PyObject*PyUnicode_Translate(PyObject *str,PyObject *table, const char *errors)¶

Return value: New reference.

Translate a string by applying a character mapping table to it and return theresulting Unicode object. ReturnNULL if an exception was raised by thecodec.

The mapping table must map Unicode ordinal integers to Unicode ordinal integersorNone (causing deletion of the character).

Mapping tables need only provide the__getitem__() interface; dictionariesand sequences work well. Unmapped character ordinals (ones which cause aLookupError) are left untouched and are copied as-is.

errors has the usual meaning for codecs. It may beNULL which indicates touse the default error handling.

PyObject*PyUnicode_TranslateCharmap(constPy_UNICODE *s,Py_ssize_t size,PyObject *mapping, const char *errors)¶

Return value: New reference.

Translate aPy_UNICODE buffer of the givensize by applying acharactermapping table to it and return the resulting Unicode object.ReturnNULL when an exception was raised by the codec.

Deprecated since version 3.3, will be removed in version 3.11:Part of the old-stylePy_UNICODE API; please migrate to usingPyUnicode_Translate(). orgeneric codec based API

MBCS codecs for Windows¶

These are the MBCS codec APIs. They are currently only available on Windows anduse the Win32 MBCS converters to implement the conversions. Note that MBCS (orDBCS) is a class of encodings, not just one. The target encoding is defined bythe user settings on the machine running the codec.

PyObject*PyUnicode_DecodeMBCS(const char *s,Py_ssize_t size, const char *errors)¶

Return value: New reference.

Create a Unicode object by decodingsize bytes of the MBCS encoded strings.ReturnNULL if an exception was raised by the codec.

PyObject*PyUnicode_DecodeMBCSStateful(const char *s,Py_ssize_t size, const char *errors,Py_ssize_t *consumed)¶

Return value: New reference.

Ifconsumed isNULL, behave likePyUnicode_DecodeMBCS(). Ifconsumed is notNULL,PyUnicode_DecodeMBCSStateful() will not decodetrailing lead byte and the number of bytes that have been decoded will be storedinconsumed.

PyObject*PyUnicode_AsMBCSString(PyObject *unicode)¶

Return value: New reference.

Encode a Unicode object using MBCS and return the result as Python bytesobject. Error handling is “strict”. ReturnNULL if an exception wasraised by the codec.

PyObject*PyUnicode_EncodeCodePage(int code_page,PyObject *unicode, const char *errors)¶

Return value: New reference.

Encode the Unicode object using the specified code page and return a Pythonbytes object. ReturnNULL if an exception was raised by the codec. UseCP_ACP code page to get the MBCS encoder.

New in version 3.3.

PyObject*PyUnicode_EncodeMBCS(constPy_UNICODE *s,Py_ssize_t size, const char *errors)¶

Return value: New reference.

Encode thePy_UNICODE buffer of the givensize using MBCS and returna Python bytes object. ReturnNULL if an exception was raised by thecodec.

Deprecated since version 3.3, will be removed in version 4.0:Part of the old-stylePy_UNICODE API; please migrate to usingPyUnicode_AsMBCSString(),PyUnicode_EncodeCodePage() orPyUnicode_AsEncodedString().

Methods & Slots¶