Before you start¶

There are some things you can do before you start the actual work of addingPython 3 support. The first one is to remove any usage of some old aliases youdon’t need any more. For example theRO macro has been removed. It was onlya shorthand forREADONLY, so if you usedRO in your code you can replaceit withREADONLY instead.

Other common redefinitions arestatichere andstaticforward. They areworkarounds for compatibility with certain compilers. For well behavingcompilers they are just redefinitions ofstatic so in Python 3 they are nowgone since there are now have well behaving compilers on all platforms thatCPython supports. If you use them in your code, you can replace themwithstatic.

Another change you can do as preparation is to move away fromPyClassObject.That’s the long deprecated “old-style classes” which are removed in Python 3.You should be able to move over toPyTypeObject with no big problem.

Object initialization¶

One of the less obvious errors encountered when compiling C extensions under Python 3is the error"missingbracesaroundinitializer", which you will get when youinitialize a Python class. You can indeed get around that by placing a couple ofbraces in the correct places, but a better solution is to replace thePyObject_HEAD_INIT macro to thePyVarObject_HEAD_INIT macro. The changewas done to conform with C’s strict aliasing rules, you can find moreinformation in PEP 3123[1] if you are interested.

Code that previously looked like this:

static PyTypeObject mytype = {    PyObject_HEAD_INIT(NULL)    0,    ...};

Should now look like this:

static PyTypeObject mytype = {    PyVarObject_HEAD_INIT(NULL, 0)    ...};

This will work in Python 2.6 and 2.7 as well. If you need to support Python2.5 or earlier, you can simply define thePyVarObject_HEAD_INIT macroif it’s missing:

#ifndef PyVarObject_HEAD_INIT    #define PyVarObject_HEAD_INIT(type, size) \        PyObject_HEAD_INIT(type) size,#endif

Another change in the object header is that thePyObject_HEAD macro haschanged so thatob_type is now in a nested structure. This means you nolonger can pick theob_type directly from the struct, so code likeob->ob_type stops working. You should replace this withPy_TYPE(ob).ThePy_TYPE macro doesn’t appear until Python 2.6, so to support earlierversions we make another#ifndef:

#ifndef Py_TYPE    #define Py_TYPE(ob) (((PyObject*)(ob))->ob_type)#endif

In both cases the definitions above are taken directly from the Python 2.6headers, where they are defined for forward compatibility purposes withPython 3. They work well in earlier Python versions as well, so this is a trickyou can use as a general rule; if you need to use a macro that is defined inPython 3 and Python 2.6, just steal the Python 2.6 or Python 2.7definition and put it inside an#ifndef.

Module initialization¶

Another big change is in the module initialization. There are many changes here,and as a result the module initialization usually ends up as part of theC extension with the most pre-processor commands or macros.

The family of functions to initialize modules, such asPy_InitModule3 aregone. Instead, you should usePyModule_Create. WherePy_InitModule3 tooka couple of parametersPyModule_Create needs aPyModuleDef struct. Ifyou want to support Python 2 you need to wrap this code with an#ifPY_MAJOR_VERSION>=3, both when you define the struct and when you use it.

#if PY_MAJOR_VERSION >= 3    static struct PyModuleDef moduledef = {        PyModuleDef_HEAD_INIT,        "themodulename",     /* m_name */        "This is a module",  /* m_doc */        -1,                  /* m_size */        module_functions,    /* m_methods */        NULL,                /* m_reload */        NULL,                /* m_traverse */        NULL,                /* m_clear */        NULL,                /* m_free */    };#endif...#if PY_MAJOR_VERSION >= 3    m = PyModule_Create(&moduledef);#else    m = Py_InitModule3("themodulename",        module_functions, "This is a module");#endif

If you want to separate the#if statements from the code you can make amacro definition. I’ve used this one, although it doesn’t support the extrafunctions like reload and traverse:

#if PY_MAJOR_VERSION >= 3    #define MOD_DEF(ob, name, doc, methods) \        static struct PyModuleDef moduledef = { \            PyModuleDef_HEAD_INIT, name, doc, -1, methods, }; \        ob = PyModule_Create(&moduledef);#else    #define MOD_DEF(ob, name, doc, methods) \        ob = Py_InitModule3(name, methods, doc);#endif

The definition of the module initialization function has also changed.In Python 2 you declared a function to initialize the module like this:

PyMODINIT_FUNC init<yourmodulename>(void)

In Python 3 this has changed to:

PyMODINIT_FUNC PyInit_<yourmodulename>(void)

It’s not just the name that has changed; it’s also the value ofPyMODINIT_FUNC. In Python 2 it’s typicallyvoid while in Python 3 itnow returns aPyObject*. You have to returnNULL if an error happenedand you need to return the module object if initialization succeeded. There arevarious ways of dealing with this if you need both Python 3 and Python 2support, starting with using multiple#ifPY_MAJOR_VERSION>=3 in thefunction. However, that gets ugly, especially in the function definition:

PyMODINIT_FUNC#if PY_MAJOR_VERSION >= 3PyInit_<yourmodulename>(void)#elseinit<yourmodulename>(void)#endif{...

It works, but it is not very readable code. It gets slightly better by usinga macro:g

#if PY_MAJOR_VERSION >= 3    #define MOD_INIT(name) PyMODINIT_FUNC PyInit_##name(void)#else    #define MOD_INIT(name) PyMODINIT_FUNC init##name(void)#endifMODINIT(themodulename){...}

But you still have to either have#if statements in the function todetermine if you should return a value or not, or make yet another macrofor that.

Another option is to define three functions. Firstly the actual moduleinitialization function, returning aPyObject* and then two wrappers. Onefor Python 3 that calls the first and returns the value and one for Python 2that calls the module initialization without returning a value:

// Python 3 module initializationstatic PyObject *moduleinit(void){    MOD_DEF(m, "themodulename",            "This is the module docstring",    module_methods)    if (m == NULL)        return NULL;    if (PyModule_AddObject(m, "hookable",           (PyObject *)&hookabletype) < 0)        return NULL;    return m;}#if PY_MAJOR_VERSION < 3    PyMODINIT_FUNC initthemodulename(void)    {        moduleinit();    }#else    PyMODINIT_FUNC PyInit_themodulename(void)    {        return moduleinit();    }#endif

As you see the module initialization will in any case end up with a lot of#ifPY_MAJOR_VERSION>=3. A complete example of all these#if statements isthis, taken fromzope.proxy:

#if PY_MAJOR_VERSION >= 3  static struct PyModuleDef moduledef = {    PyModuleDef_HEAD_INIT,    "_zope_proxy_proxy", /* m_name */    module___doc__,      /* m_doc */    -1,                  /* m_size */    module_functions,    /* m_methods */    NULL,                /* m_reload */    NULL,                /* m_traverse */    NULL,                /* m_clear */    NULL,                /* m_free */  };#endifstatic PyObject *moduleinit(void){    PyObject *m;#if PY_MAJOR_VERSION >= 3    m = PyModule_Create(&moduledef);#else    m = Py_InitModule3("_zope_proxy_proxy",                        module_functions, module___doc__);#endif    if (m == NULL)        return NULL;    if (empty_tuple == NULL)        empty_tuple = PyTuple_New(0);    ProxyType.tp_free = _PyObject_GC_Del;    if (PyType_Ready(&ProxyType) < 0)        return NULL;    Py_INCREF(&ProxyType);    PyModule_AddObject(m, "ProxyBase", (PyObject *)&ProxyType);    if (api_object == NULL) {        api_object = PyCObject_FromVoidPtr(&wrapper_capi, NULL);        if (api_object == NULL)        return NULL;    }    Py_INCREF(api_object);    PyModule_AddObject(m, "_CAPI", api_object);  return m;}#if PY_MAJOR_VERSION < 3    PyMODINIT_FUNC    init_zope_proxy_proxy(void)    {        moduleinit();    }#else    PyMODINIT_FUNC    PyInit__zope_proxy_proxy(void)    {        return moduleinit();    }#endif

If you don’t like all the version tests, you can put all of these togetherbefore the function definition and use macros for anything that differs. Hereis the samezope.proxy module, after I replaced all the#if tests withone block of definitions in the beginning:

#if PY_MAJOR_VERSION >= 3  #define MOD_ERROR_VAL NULL  #define MOD_SUCCESS_VAL(val) val  #define MOD_INIT(name) PyMODINIT_FUNC PyInit_##name(void)  #define MOD_DEF(ob, name, doc, methods) \          static struct PyModuleDef moduledef = { \            PyModuleDef_HEAD_INIT, name, doc, -1, methods, }; \          ob = PyModule_Create(&moduledef);#else  #define MOD_ERROR_VAL  #define MOD_SUCCESS_VAL(val)  #define MOD_INIT(name) void init##name(void)  #define MOD_DEF(ob, name, doc, methods) \          ob = Py_InitModule3(name, methods, doc);#endifMOD_INIT(_zope_proxy_proxy){    PyObject *m;    MOD_DEF(m, "_zope_proxy_proxy", module___doc__,            module_functions)    if (m == NULL)        return MOD_ERROR_VAL;    if (empty_tuple == NULL)        empty_tuple = PyTuple_New(0);    ProxyType.tp_free = _PyObject_GC_Del;    if (PyType_Ready(&ProxyType) < 0)        return MOD_ERROR_VAL;    Py_INCREF(&ProxyType);    PyModule_AddObject(m, "ProxyBase", (PyObject *)&ProxyType);    if (api_object == NULL) {        api_object = PyCObject_FromVoidPtr(&wrapper_capi, NULL);        if (api_object == NULL)        return MOD_ERROR_VAL;    }    Py_INCREF(api_object);    PyModule_AddObject(m, "_CAPI", api_object);    return MOD_SUCCESS_VAL(m);}

This is by far my preferred version, for stylistic reasons, but ultimately it’sa matter of taste and coding style if you prefer the in-line#if statementsor if you like to use many#define macros. So you choose what fits best withyour coding style.

Changes in Python¶

The changes in Python are of course reflected in the C API. These are usuallyeasy to handle. A typical example here is the unification ofint andlong types. Although in Python it behaves like thelong type is gone,it’s actually theint type that has been removed and thelong typerenamed. However, in the C API it hasn’t been renamed. That means that all thefunctions that returned Pythonint objects are now gone and you need toreplace them with the functions that returns Pythonlong objects. This meansthatPyInt_FromLong must be replaced withPyLong_FromLong,PyInt_FromString withPyLong_FromString etc. If you need tokeep Python 2 compatibility you have to replace it conditionally:

#if PY_MAJOR_VERSION >= 3    PyModule_AddObject(m, "val", PyLong_FromLong(2));#else    PyModule_AddObject(m, "val", PyInt_FromLong(2));#endif

Also in this case a#define makes for cleaner code if you need to do itmore than once:

#if PY_MAJOR_VERSION >= 3    #define PyInt_FromLong PyLong_FromLong#endifPyModule_AddObject(m, "val", PyInt_FromLong(2));

In Python 3.2 the CObject API was removed. It was replaced with the CapsuleAPI, which is also available for Python 2.7 and 3.1. For the simple usecase ofjust wrapping a C value the change is simple:

#if PY_MAJOR_VERSION < 3    c_api = PyCObject_FromVoidPtr ((void *)pointer_to_value, NULL);#else    c_api = PyCapsule_New((void *)pointer_to_value, NULL, NULL);#endif

Other things changed in Python that you are likely to encounter is the removalof the support for__cmp__() methods. The_typeobject structure used fordefining a new type includes a place for a__cmp__() method definition. It’sstill there in Python 3 for compatibility but it’s now ignored. Thecmpfunctype definition and thePyObject_Compare functions have been removed aswell. The only way to get full Python 3 compatibility here is to implement richcomparison functionality. There is support for that back to Python 2.1, sothere is no problem with backwards compatibility.

Strings and Unicode¶

The changes in strings, Unicode and bytes are of course one of the biggestchanges also when writing C extensions. In the C API, as with integers,there has been no renaming amongst the strings and theunicode type is stillcalledunicode. Thestr type and all accompanying support functions aregone and the newbytes type has replaced it.

This means that if your extension returns or handles binary data you will inPython 2 get and returnPyString objects, while you in Python 3 willhandlePyBytes objects. Where you handle text data you should inPython 2 accept bothPyString andPyUnicode while in Python 3 onlyPyUnicode is relevant. This can be handled with the same techniques as forint andlong above, you can either make two versions of the code andchoose between them with an#if, or you can redefine the missingPyString functions in Python 3 as eitherPyBytes orPyUnicodedepending on what you need.

Footnotes