Using C libraries

Note

This page uses two different syntax variants:

  • Cython specificcdef syntax, which was designed to make type declarationsconcise and easily readable from a C/C++ perspective.

  • Pure Python syntax which allows static Cython type declarations inpure Python code,followingPEP-484 type hintsandPEP 526 variable annotations.

    To make use of C data types in Python syntax, you need to import the specialcython module in the Python module that you want to compile, e.g.

    importcython

    If you use the pure Python syntax we strongly recommend you use a recentCython 3 release, since significant improvements have been made herecompared to the 0.29.x releases.

Apart from writing fast code, one of the main use cases of Cython isto call external C libraries from Python code. As Cython codecompiles down to C code itself, it is actually trivial to call Cfunctions directly in the code. The following gives a completeexample for using (and wrapping) an external C library in Cython code,including appropriate error handling and considerations aboutdesigning a suitable API for Python and Cython code.

Imagine you need an efficient way to store integer values in a FIFOqueue. Since memory really matters, and the values are actuallycoming from C code, you cannot afford to create and store Pythonint objects in a list or deque. So you look out for a queueimplementation in C.

After some web search, you find the C-algorithms library[CAlg] anddecide to use its double ended queue implementation. To make thehandling easier, however, you decide to wrap it in a Python extensiontype that can encapsulate all memory management.

[CAlg]

Simon Howard, C Algorithms library,https://fragglet.github.io/c-algorithms/

Defining external declarations

You can download CAlghere.

The C API of the queue implementation, which is defined in the headerfilec-algorithms/src/queue.h, essentially looks like this:

queue.h
/* queue.h */typedefstruct_QueueQueue;typedefvoid*QueueValue;Queue*queue_new(void);voidqueue_free(Queue*queue);intqueue_push_head(Queue*queue,QueueValuedata);QueueValuequeue_pop_head(Queue*queue);QueueValuequeue_peek_head(Queue*queue);intqueue_push_tail(Queue*queue,QueueValuedata);QueueValuequeue_pop_tail(Queue*queue);QueueValuequeue_peek_tail(Queue*queue);intqueue_is_empty(Queue*queue);

To get started, the first step is to redefine the C API in a.pxdfile, say,cqueue.pxd:

cqueue.pxd
cdefexternfrom"c-algorithms/src/queue.h":ctypedefstructQueue:passctypedefvoid*QueueValueQueue*queue_new()voidqueue_free(Queue*queue)intqueue_push_head(Queue*queue,QueueValuedata)QueueValuequeue_pop_head(Queue*queue)QueueValuequeue_peek_head(Queue*queue)intqueue_push_tail(Queue*queue,QueueValuedata)QueueValuequeue_pop_tail(Queue*queue)QueueValuequeue_peek_tail(Queue*queue)bintqueue_is_empty(Queue*queue)

Note how these declarations are almost identical to the header filedeclarations, so you can often just copy them over. However, you donot need to provideall declarations as above, just those that youuse in your code or in other declarations, so that Cython gets to seea sufficient and consistent subset of them. Then, consider adaptingthem somewhat to make them more comfortable to work with in Cython.

Specifically, you should take care of choosing good argument namesfor the C functions, as Cython allows you to pass them as keywordarguments. Changing them later on is a backwards incompatible APImodification. Choosing good names right away will make thesefunctions more pleasant to work with from Cython code.

One noteworthy difference to the header file that we use above is thedeclaration of theQueue struct in the first line.Queue isin this case used as anopaque handle; only the library that iscalled knows what is really inside. Since no Cython code needs toknow the contents of the struct, we do not need to declare itscontents, so we simply provide an empty definition (as we do not wantto declare the_Queue type which is referenced in the C header)[1].

[1]

There’s a subtle difference betweencdefstructQueue:passandctypedefstructQueue:pass. The former declares atype which is referenced in C code asstructQueue, whilethe latter is referenced in C asQueue. This is a Clanguage quirk that Cython is not able to hide. Most modern Clibraries use thectypedef kind of struct.

Another exception is the last line. The integer return value of thequeue_is_empty() function is actually a C boolean value, i.e. theonly interesting thing about it is whether it is non-zero or zero,indicating if the queue is empty or not. This is best expressed byCython’sbint type, which is a normalint type when used in Cbut maps to Python’s boolean valuesTrue andFalse whenconverted to a Python object. This way of tightening declarations ina.pxd file can often simplify the code that uses them.

It is good practice to define one.pxd file for each library thatyou use, and sometimes even for each header file (or functional group)if the API is large. That simplifies their reuse in other projects.Sometimes, you may need to use C functions from the standard Clibrary, or want to call C-API functions from CPython directly. Forcommon needs like this, Cython ships with a set of standard.pxdfiles that provide these declarations in a readily usable way that isadapted to their use in Cython. The main packages arecpython,libc andlibcpp. The NumPy library also has a standard.pxd filenumpy, as it is often used in Cython code. SeeCython’sCython/Includes/ source package for a complete list ofprovided.pxd files.

Writing a wrapper class

After declaring our C library’s API, we can start to design the Queueclass that should wrap the C queue. It will live in a file calledqueue.pyx/queue.py.[2]

[2]

Note that the name of the.pyx/.py file must be different fromthecqueue.pxd file with declarations from the C library,as both do not describe the same code. A.pxd file next toa.pyx/.py file with the same name defines exporteddeclarations for code in the.pyx/.py file. As thecqueue.pxd file contains declarations of a regular Clibrary, there must not be a.pyx/.py file with the same namethat Cython associates with it.

Here is a first start for the Queue class:

queue.py
fromcython.cimportsimportcqueue@cython.cclassclassQueue:_c_queue:cython.pointer[cqueue.Queue]def__cinit__(self):self._c_queue=cqueue.queue_new()

Note that it says__cinit__ rather than__init__. While__init__ is available as well, it is not guaranteed to be run (forinstance, one could create a subclass and forget to call theancestor’s constructor). Because not initializing C pointers oftenleads to hard crashes of the Python interpreter, Cython provides__cinit__ which isalways called immediately on construction,before CPython even considers calling__init__, and whichtherefore is the right place to initialise static attributes(cdef fields) of the new instance. However, as__cinit__ iscalled during object construction,self is not fully constructed yet,and one must avoid doing anything withself but assigning to staticattributes (cdef fields).

Note also that the above method takes no parameters, although subtypesmay want to accept some. A no-arguments__cinit__() method is aspecial case here that simply does not receive any parameters thatwere passed to a constructor, so it does not prevent subclasses fromadding parameters. If parameters are used in the signature of__cinit__(), they must match those of any declared__init__method of classes in the class hierarchy that are used to instantiatethe type.

Memory management

Before we continue implementing the other methods, it is important tounderstand that the above implementation is not safe. In caseanything goes wrong in the call toqueue_new(), this code willsimply swallow the error, so we will likely run into a crash later on.According to the documentation of thequeue_new() function, theonly reason why the above can fail is due to insufficient memory. Inthat case, it will returnNULL, whereas it would normally return apointer to the new queue.

The Python way to get out of this is to raise aMemoryError[3].We can thus change the init function as follows:

queue.py
fromcython.cimportsimportcqueue@cython.cclassclassQueue:_c_queue=cython.declare(cython.pointer[cqueue.Queue])def__cinit__(self):self._c_queue=cqueue.queue_new()ifself._c_queueiscython.NULL:raiseMemoryError()
[3]

In the specific case of aMemoryError, creating a newexception instance in order to raise it may actually fail becausewe are running out of memory. Luckily, CPython provides a C-APIfunctionPyErr_NoMemory() that safely raises the rightexception for us. Cython automaticallysubstitutes this C-API call whenever you writeraiseMemoryError orraiseMemoryError(). If you use an olderversion, you have to cimport the C-API function from the standardpackagecpython.exc and call it directly.

The next thing to do is to clean up when the Queue instance is nolonger used (i.e. all references to it have been deleted). To thisend, CPython provides a callback that Cython makes available as aspecial method__dealloc__(). In our case, all we have to do isto free the C Queue, but only if we succeeded in initialising it inthe init method:

def__dealloc__(self):ifself._c_queueisnotcython.NULL:cqueue.queue_free(self._c_queue)

Compiling and linking

At this point, we have a working Cython module that we can test. Tocompile it, we need to configure asetup.py script for setuptools.Here is the most basic script for compiling a Cython module

fromsetuptoolsimportExtension,setupfromCython.Buildimportcythonizesetup(ext_modules=cythonize([Extension("queue",["queue.py"])]))

To build against the external C library, we need to make sure Cython finds the necessary libraries.There are two ways to archive this. First we can tell setuptools where to findthe c-source to compile thequeue.c implementation automatically. Alternatively,we can build and install C-Alg as system library and dynamically link it. The latter is usefulif other applications also use C-Alg.

Static Linking

To build the c-code automatically we need to include compiler directives inqueue.pyx/queue.py

# distutils: sources = c-algorithms/src/queue.c# distutils: include_dirs = c-algorithms/src/importcythonfromcython.cimportsimportcqueue@cython.cclassclassQueue:_c_queue=cython.declare(cython.pointer[cqueue.Queue])def__cinit__(self):self._c_queue=cqueue.queue_new()ifself._c_queueiscython.NULL:raiseMemoryError()def__dealloc__(self):ifself._c_queueisnotcython.NULL:cqueue.queue_free(self._c_queue)

Thesources compiler directive gives the path of the Cfiles that setuptools is going to compile andlink (statically) into the resulting extension module.In general all relevant header files should be found ininclude_dirs.Now we can build the project using:

$pythonsetup.pybuild_ext-i

And test whether our build was successful:

$python-c'import queue; Q = queue.Queue()'

Dynamic Linking

Dynamic linking is useful, if the library we are going to wrap is alreadyinstalled on the system. To perform dynamic linking we first need tobuild and install c-alg.

To build c-algorithms on your system:

$cdc-algorithms$shautogen.sh$./configure$make

to install CAlg run:

$makeinstall

Afterwards the file/usr/local/lib/libcalg.so should exist.

Note

This path applies to Linux systems and may be different on other platforms,so you will need to adapt the rest of the tutorial depending on the pathwherelibcalg.so orlibcalg.dll is on your system.

In this approach we need to tell the setup script to link with an external library.To do so we need to extend the setup script to install change the extension setup from

ext_modules=cythonize([Extension("queue",["queue.py"])])

to

ext_modules=cythonize([Extension("queue",["queue.py"],libraries=["calg"])])

Now we should be able to build the project using:

$pythonsetup.pybuild_ext-i

If thelibcalg is not installed in a ‘normal’ location, users can provide therequired parameters externally by passing appropriate C compilerflags, such as:

CFLAGS="-I/usr/local/otherdir/calg/include"\LDFLAGS="-L/usr/local/otherdir/calg/lib"\pythonsetup.pybuild_ext-i

Before we run the module, we also need to make sure thatlibcalg is intheLD_LIBRARY_PATH environment variable, e.g. by setting:

$exportLD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/lib

Once we have compiled the module for the first time, we can now importit and instantiate a new Queue:

$exportPYTHONPATH=.$python-c'import queue; Q = queue.Queue()'

However, this is all our Queue class can do so far, so let’s make itmore usable.

Mapping functionality

Before implementing the public interface of this class, it is goodpractice to look at what interfaces Python offers, e.g. in itslist orcollections.deque classes. Since we only need a FIFOqueue, it’s enough to provide the methodsappend(),peek() andpop(), and additionally anextend() method to add multiplevalues at once. Also, since we already know that all values will becoming from C, it’s best to provide onlycdef/@cfunc methods for now, andto give them a straight C interface.

In C, it is common for data structures to store data as avoid* towhatever data item type. Since we only want to storeint values,which usually fit into the size of a pointer type, we can avoidadditional memory allocations through a trick: we cast ourint valuestovoid* and vice versa, and store the value directly as thepointer value.

Here is a simple implementation for theappend() method:

@cython.cfuncdefappend(self,value:cython.int):cqueue.queue_push_tail(self._c_queue,cython.cast(cython.p_void,value))

Again, the same error handling considerations as for the__cinit__() method apply, so that we end up with thisimplementation instead:

@cython.cfuncdefappend(self,value:cython.int):ifnotcqueue.queue_push_tail(self._c_queue,cython.cast(cython.p_void,value)):raiseMemoryError()

Adding anextend() method should now be straight forward:

@cython.cfuncdefextend(self,values:cython.p_int,count:cython.size_t):"""Append all ints to the queue.    """value:cython.intforvalueinvalues[:count]:# Slicing pointer to limit the iteration boundaries.self.append(value)

This becomes handy when reading values from a C array, for example.

So far, we can only add data to the queue. The next step is to writethe two methods to get the first element:peek() andpop(),which provide read-only and destructive read access respectively.To avoid compiler warnings when castingvoid* toint directly,we use an intermediate data type that is big enough to hold avoid*.Here,Py_ssize_t:

@cython.cfuncdefpeek(self)->cython.int:returncython.cast(cython.Py_ssize_t,cqueue.queue_peek_head(self._c_queue))@cython.cfuncdefpop(self)->cython.int:returncython.cast(cython.Py_ssize_t,cqueue.queue_pop_head(self._c_queue))

Normally, in C, we risk losing data when we convert a larger integer typeto a smaller integer type without checking the boundaries, andPy_ssize_tmay be a larger type thanint. But since we control how values are addedto the queue, we already know that all values that are in the queue fit intoanint, so the above conversion fromvoid* toPy_ssize_t toint(the return type) is safe by design.

Handling errors

Now, what happens when the queue is empty? According to thedocumentation, the functions return aNULL pointer, which istypically not a valid value. But since we are simply casting to andfrom ints, we cannot distinguish anymore if the return value wasNULL because the queue was empty or because the value stored inthe queue was0. In Cython code, we want the first case toraise an exception, whereas the second case should simply return0. To deal with this, we need to special case this value,and check if the queue really is empty or not:

@cython.cfuncdefpeek(self)->cython.int:value:cython.int=cython.cast(cython.Py_ssize_t,cqueue.queue_peek_head(self._c_queue))ifvalue==0:# this may mean that the queue is empty, or# that it happens to contain a 0 valueifcqueue.queue_is_empty(self._c_queue):raiseIndexError("Queue is empty")returnvalue

Note how we have effectively created a fast path through the method inthe hopefully common cases that the return value is not0. Onlythat specific case needs an additional check if the queue is empty.

If thepeek function was a Python function returning aPython object value, CPython would simply returnNULL internallyinstead of a Python object to indicate an exception, which wouldimmediately be propagated by the surrounding code. The problem isthat the return type isint and anyint value is a valid queueitem value, so there is no way to explicitly signal an error to thecalling code.

The only way calling code can deal with this situation is to call:c:function:`PyErr_Occurred()` when returning from a function to check if anexception was raised, and if so, propagate the exception. Thisobviously has a performance penalty. Cython therefore uses a dedicated valuethat it implicitly returns in the case of anexception, so that the surrounding code only needs to check for anexception when receiving this exact value.

By default, the value-1 is used as the exception return value.All other return values will be passed through almostwithout a penalty, thus again creating a fast path for ‘normal’values. SeeError return values for more details.

Now that thepeek() method is implemented, thepop() methodalso needs adaptation. Since it removes a value from the queue,however, it is not enough to test if the queue is emptyafter theremoval. Instead, we must test it on entry:

@cython.cfuncdefpop(self)->cython.int:ifcqueue.queue_is_empty(self._c_queue):raiseIndexError("Queue is empty")returncython.cast(cython.Py_ssize_t,cqueue.queue_pop_head(self._c_queue))

The return value for exception propagation is declared exactly as forpeek().

Lastly, we can provide the Queue with an emptiness indicator in thenormal Python way by implementing the__bool__() special method(note that Python 2 calls this method__nonzero__, whereas Cythoncode can use either name):

def__bool__(self):returnnotcqueue.queue_is_empty(self._c_queue)

Note that this method returns eitherTrue orFalse as wedeclared the return type of thequeue_is_empty() function asbint incqueue.pxd.

Testing the result

Now that the implementation is complete, you may want to write sometests for it to make sure it works correctly. Especially doctests arevery nice for this purpose, as they provide some documentation at thesame time. To enable doctests, however, you need a Python API thatyou can call. C methods are not visible from Python code, and thusnot callable from doctests.

A quick way to provide a Python API for the class is to change themethods fromcdef/@cfunc tocpdef/@ccall. This willlet Cython generate two entry points, one that is callable from normalPython code using the Python call semantics and Python objects as arguments,and one that is callable from C code with fast C semantics and without requiringintermediate argument conversion from or to Python types. Note thatcpdef/@ccall methods ensure that they can be appropriately overriddenby Python methods even when they are called from Cython. This adds a tiny overheadcompared tocdef/@cfunc methods.

Now that we have both a C-interface and a Python interface for ourclass, we should make sure that both interfaces are consistent.Python users would expect anextend() method that accepts arbitraryiterables, whereas C users would like to have one that allows passingC arrays and C memory. Both signatures are incompatible.

We will solve this issue by considering that in C, the API could alsowant to support other input types, e.g. arrays oflong orchar,which is usually supported with differently named C API functions such asextend_ints(),extend_longs(),extend_chars(), etc. This allowsus to free the method nameextend() for the duck typed Python method,which can accept arbitrary iterables.

The following listing shows the complete implementation that usescpdef/@ccall methods where possible:

queue.py
fromcython.cimportsimportcqueuefromcythonimportcast@cython.cclassclassQueue:"""A queue class for C integer values.    >>> q = Queue()    >>> q.append(5)    >>> q.peek()    5    >>> q.pop()    5    """_c_queue=cython.declare(cython.pointer[cqueue.Queue])def__cinit__(self):self._c_queue=cqueue.queue_new()ifself._c_queueiscython.NULL:raiseMemoryError()def__dealloc__(self):ifself._c_queueisnotcython.NULL:cqueue.queue_free(self._c_queue)@cython.ccalldefappend(self,value:cython.int):ifnotcqueue.queue_push_tail(self._c_queue,cast(cython.p_void,cast(cython.Py_ssize_t,value))):raiseMemoryError()# The `cpdef` feature is obviously not available for the original "extend()"# method, as the method signature is incompatible with Python argument# types (Python does not have pointers).  However, we can rename# the C-ish "extend()" method to e.g. "extend_ints()", and write# a new "extend()" method that provides a suitable Python interface by# accepting an arbitrary Python iterable.@cython.ccalldefextend(self,values):forvalueinvalues:self.append(value)@cython.cfuncdefextend_ints(self,values:cython.p_int,count:cython.size_t):value:cython.intforvalueinvalues[:count]:# Slicing pointer to limit the iteration boundaries.self.append(value)@cython.ccall@cython.exceptval(-1,check=True)defpeek(self)->cython.int:value:cython.int=cast(cython.Py_ssize_t,cqueue.queue_peek_head(self._c_queue))ifvalue==0:# this may mean that the queue is empty,# or that it happens to contain a 0 valueifcqueue.queue_is_empty(self._c_queue):raiseIndexError("Queue is empty")returnvalue@cython.ccall@cython.exceptval(-1,check=True)defpop(self)->cython.int:ifcqueue.queue_is_empty(self._c_queue):raiseIndexError("Queue is empty")returncast(cython.Py_ssize_t,cqueue.queue_pop_head(self._c_queue))def__bool__(self):returnnotcqueue.queue_is_empty(self._c_queue)

Now we can test our Queue implementation using a python script,for example heretest_queue.py:

importtimeimportqueueQ=queue.Queue()Q.append(10)Q.append(20)print(Q.peek())print(Q.pop())print(Q.pop())try:print(Q.pop())exceptIndexErrorase:print("Error message:",e)# Prints "Queue is empty"i=10000values=range(i)start_time=time.time()Q.extend(values)end_time=time.time()-start_timeprint("Adding {} items took {:1.3f} msecs.".format(i,1000*end_time))foriinrange(41):Q.pop()Q.pop()print("The answer is:")print(Q.pop())

As a quick test with 10000 numbers on the author’s machine indicates,using this Queue from Cython code with Cint values is about fivetimes as fast as using it from Cython code with Python object values,almost eight times faster than using it from Python code in a Pythonloop, and still more than twice as fast as using Python’s highlyoptimisedcollections.deque type from Cython code with Pythonintegers.

Callbacks

Let’s say you want to provide a way for users to pop values from thequeue up to a certain user defined event occurs. To this end, youwant to allow them to pass a predicate function that determines whento stop, e.g.:

defpop_until(self,predicate):whilenotpredicate(self.peek()):self.pop()

Now, let us assume for the sake of argument that the C queueprovides such a function that takes a C callback function aspredicate. The API could look as follows:

/*Ctypeofapredicatefunctionthattakesaqueuevalueandreturns*-1forerrors*0forreject*1foraccept*/typedefint(*predicate_func)(void*user_context,QueueValuedata);/*Popvaluesaslongasthepredicateevaluatestotrueforthem,*returns-1ifthepredicatefailedwithanerrorand0otherwise.*/intqueue_pop_head_until(Queue*queue,predicate_funcpredicate,void*user_context);

It is normal for C callback functions to have a genericvoid*argument that allows passing any kind of context or state through theC-API into the callback function. We will use this to pass our Pythonpredicate function.

First, we have to define a callback function with the expectedsignature that we can pass into the C-API function:

@cython.cfunc@cython.exceptval(check=False)defevaluate_predicate(context:cython.p_void,value:cqueue.QueueValue)->cython.int:"Callback function that can be passed as predicate_func"try:# recover Python function object from void* argumentfunc=cython.cast(object,context)# call function, convert result into 0/1 for True/Falsereturnbool(func(cython.cast(int,value)))except:# catch any Python errors and return error indicatorreturn-1

Note

@cfunc functions in pure python are defined as@exceptval(-1,check=True)by default. Sinceevaluate_predicate() should be passed to function as parameter,we need to turn off exception checking entirely.

The main idea is to pass a pointer (a.k.a. borrowed reference) to thefunction object as the user context argument. We will call the C-APIfunction as follows:

defpop_until(self,python_predicate_function):result=cqueue.queue_pop_head_until(self._c_queue,evaluate_predicate,cython.cast(cython.p_void,python_predicate_function))ifresult==-1:raiseRuntimeError("an error occurred")

The usual pattern is to first cast the Python object reference intoavoid* to pass it into the C-API function, and then castit back into a Python object in the C predicate callback function.The cast tovoid* creates a borrowed reference. On the castto<object>, Cython increments the reference count of the objectand thus converts the borrowed reference back into an owned reference.At the end of the predicate function, the owned reference goes outof scope again and Cython discards it.

The error handling in the code above is a bit simplistic. Specifically,any exceptions that the predicate function raises will essentially bediscarded and only result in a plainRuntimeError() being raisedafter the fact. This can be improved by storing away the exceptionin an object passed through the context parameter and re-raising itafter the C-API function has returned-1 to indicate the error.