Deprecated since Python 3.3

Py_UNICODE and APIs using it has been deprecated since Python 3.3.

Inefficient

All of these APIs are implemented usingPyUnicode_FromWideChar.So these APIs are inefficient when user want to encode Unicodeobject.

Not used widely

When searching from the top 4000 PyPI packages[1], only pyodbc usethese APIs.

• PyUnicode_EncodeUTF8()
• PyUnicode_EncodeUTF16()

pyodbc uses these APIs to encode Unicode object into bytes object.So it is easy to fix it.[2]

ReplacePy_UNICODE* withPyObject*

As described in the “Alternative APIs” section, some APIs don’t havepublic alternative APIs acceptingPyObject*unicode input.And some public alternative APIs have restrictions like missingerrors andbyteorder parameters.

Instead of removing deprecated APIs, we can reuse their names foralternative public APIs.

Since we have private alternative APIs already, it is just renamingfrom private name to public and deprecated names.

Pros:

• We have a more consistent API set.

Cons:

• Backward incompatible.
• We have more public APIs to maintain for rare use cases.
• Existing public APIs are enough for most use cases, andPyUnicode_AsEncodedString() can be used in other cases.

ReplacePy_UNICODE* withPy_UCS4*

We can replacePy_UNICODE withPy_UCS4 and undeprecatethese APIs.

UTF-8, UTF-16, UTF-32 encoders supportPy_UCS4 internally.SoPyUnicode_EncodeUTF8(),PyUnicode_EncodeUTF16(), andPyUnicode_EncodeUTF32() can avoid to create a temporary Unicodeobject.

Pros:

• We can avoid creating temporary Unicode object when encoding fromPy_UCS4* into bytes object with UTF-8, UTF-16, UTF-32 codecs.

Cons:

• Backward incompatible.
• We have more public APIs to maintain for rare use cases.
• Other Python implementations that want to support Python/C API needto support these APIs too.
• If we change the Unicode internal representation to UTF-8 in thefuture, we need to keep UCS-4 support only for these APIs.

ReplacePy_UNICODE* withwchar_t*

We can replacePy_UNICODE withwchar_t. SincePy_UNICODEis typedef ofwchar_t already, this is status quo.

On platforms wheresizeof(wchar_t)==4, we can avoid to create atemporary Unicode object when encoding fromwchar_t* to bytesobjects using UTF-8, UTF-16, and UTF-32 codec, like the “ReplacePy_UNICODE* withPy_UCS4*” idea.

Pros:

• Backward compatible.
• We can avoid creating temporary Unicode object when encode fromPy_UCS4* into bytes object with UTF-8, UTF-16, UTF-32 codecson platform wheresizeof(wchar_t)==4.

Cons:

• Although Windows is the most major platform that useswchar_theavily, these APIs need to create a temporary Unicode objectalways becausesizeof(wchar_t)==2 on Windows.
• We have more public APIs to maintain for rare use cases.
• Other Python implementations that want to support Python/C API needto support these APIs too.
• If we change the Unicode internal representation to UTF-8 in thefuture, we need to keep UCS-4 support only for these APIs.

Emit runtime warning

In addition to existing compiler warning, emitting runtimeDeprecationWarning is suggested.

But these APIs doesn’t release GIL for now. Emitting a warning fromsuch APIs is not safe. See this example.

PyObject*u=PyList_GET_ITEM(list,i);//uisborrowedreference.PyObject*b=PyUnicode_EncodeUTF8(PyUnicode_AS_UNICODE(u),PyUnicode_GET_SIZE(u),NULL);//Assumesuisstilllivingreference.PyObject*t=PyTuple_Pack(2,u,b);Py_DECREF(b);returnt;

If we emit Python warning fromPyUnicode_EncodeUTF8(), warningfilters and other threads may change thelist andu can bea dangling reference afterPyUnicode_EncodeUTF8() returned.

Objections

• Removing these APIs removes ability to use codec without temporaryUnicode.
  • Codecs can not encode Unicode buffer directly without temporaryUnicode object since Python 3.3. All these APIs creates temporaryUnicode object for now. So removing them doesn’t reduce anyabilities.
• Why not remove decoder APIs too?
  • They are part of stable ABI.
  • PyUnicode_DecodeASCII() andPyUnicode_DecodeUTF8() areused very widely. Deprecating them is not worth enough.
  • Decoder APIs can decode from byte buffer directly, withoutcreating temporary bytes object. On the other hand, encoder APIscan not avoid temporary Unicode object.