What we’ll do¶

Let’s create an extension module calledspam[1],which will include a Python interface to the Cstandard library functionsystem().This function is defined instdlib.h.It takes a C string as argument, runs the argument as a systemcommand, and returns a result value as an integer.A manual page forsystem() might summarize it this way:

#include<stdlib.h>intsystem(constchar*command);

Note that like many functions in the C standard library,this function is already exposed in Python.In production, useos.system() orsubprocess.run()rather than the module you’ll write here.

We want this function to be callable from Python as follows:

>>>importspam>>>status=spam.system("whoami")User Name>>>status0

Start with the headers¶

Begin by creating a directory for this tutorial, and switching to iton the command line.Then, create a file namedspammodule.c in your directory.[2]

In this file, we’ll include two headers:Python.h to pull inall declarations of the Python C API, andstdlib.h for thesystem() function.[3]

Add the following lines tospammodule.c:

#include<Python.h>#include<stdlib.h>     // for system()

Be sure to putstdlib.h, and any other standard library includes,afterPython.h.On some systems, Python may define some pre-processor definitionsthat affect the standard headers.

Running your build tool¶

With only the includes in place, your extension won’t do anything.Still, it’s a good time to compile it and try to import it.This will ensure that your build tool works, so that you can makeand test incremental changes as you follow the rest of the text.

CPython itself does not come with a tool to build extension modules;it is recommended to use a third-party project for this.In this tutorial, we’ll usemeson-python.(If you want to use another one, seeAppendix: Other build tools.)

meson-python requires defining a “project” using two extra files.

First, addpyproject.toml with these contents:

[build-system]build-backend='mesonpy'requires=['meson-python'][project]# Placeholder project information# (change this before distributing the module)name='sampleproject'version='0'

Then, createmeson.build containing the following:

project('sampleproject','c')py=import('python').find_installation(pure:false)py.extension_module('spam',# name of the importable Python module'spammodule.c',# the C source fileinstall:true,)

Now, build install theproject in the current directory (.) viapip:

python-mpipinstall.

Note that you will need to run this command again every time you change yourextension.Unlike Python, C has an explicit compilation step.

When your extension is compiled and installed, start Python and try toimport it.This should fail with the following exception:

>>>importspamTraceback (most recent call last):...ImportError:dynamic module does not define module export function (PyModExport_spam or PyInit_spam)

Module export hook¶

The exception you got when you tried to import the module told you that Pythonis looking for a “module export function”, also known as amodule export hook.Let’s define one.

First, add a prototype below the#include lines:

PyMODEXPORT_FUNCPyModExport_spam(void);

ThePyMODEXPORT_FUNC macro declares the function’sreturn type, and adds any special linkage declarations neededto make the function visible and usable when CPython loads it.

After the prototype, add the function itself.For now, make it returnNULL:

PyMODEXPORT_FUNCPyModExport_spam(void){returnNULL;}

Compile and load the module again.You should get a different error this time.

>>>importspamTraceback (most recent call last):...SystemError:module export hook for module 'spam' failed without setting an exception

Simply returningNULL isnot correct behavior for an export hook,and CPython complains about it.That’s good – it means that CPython found the function!Let’s now make it do something useful.

The slot table¶

Rather thanNULL, the export hook should return the information needed tocreate a module.Let’s start with the basics: the name and docstring.

The information should be defined in astatic array ofPyModuleDef_Slot entries, which are essentially key-value pairs.Define this array just before your export hook:

staticPyModuleDef_Slotspam_slots[]={{Py_mod_name,"spam"},{Py_mod_doc,"A wonderful module with an example function"},{0,NULL}};

For bothPy_mod_name andPy_mod_doc, the values are Cstrings – that is, NUL-terminated, UTF-8 encoded byte arrays.

Note the zero-filled sentinel entry at the end.If you forget it, you’ll trigger undefined behavior.

The array is defined asstatic – that is, not visible outside this.c file.This will be a common theme.CPython only needs to access the export hook; all global variablesand all other functions should generally bestatic, so that they don’tclash with other extensions.

Return this array from your export hook instead ofNULL:

PyMODEXPORT_FUNCPyModExport_spam(void){returnspam_slots;}

Now, recompile and try it out:

>>>importspam>>>print(spam)<module 'spam' from '/home/encukou/dev/cpython/spam.so'>

You have an extension module!Tryhelp(spam) to see the docstring.

The next step will be adding a function.

Exposing a function¶

To expose thesystem() C function directly to Python,we’ll need to write a layer of glue code to convert arguments from Pythonobjects to C values, and the C return value back to Python.

One of the simplest ways to write glue code is a “METH_O” function,which takes two Python objects and returns one.All Python objects – regardless of the Python type – are represented in Cas pointers to thePyObject structure.

Add such a function above the slots array:

staticPyObject*spam_system(PyObject*self,PyObject*arg){Py_RETURN_NONE;}

For now, we ignore the arguments, and use thePy_RETURN_NONEmacro, which expands to areturn statement that properly returnsa PythonNone object.

Recompile your extension to make sure you don’t have syntax errors.We haven’t yet addedspam_system to the module, so you might get awarning thatspam_system is unused.

staticPyMethodDefspam_methods[]={{.ml_name="system",.ml_meth=spam_system,.ml_flags=METH_O,.ml_doc="Execute a shell command.",},{NULL,NULL,0,NULL}/* Sentinel */};

staticPyModuleDef_Slotspam_slots[]={{Py_mod_name,"spam"},{Py_mod_doc,"A wonderful module with an example function"},{Py_mod_methods,spam_methods},{0,NULL}};

>>>importspam>>>print(spam.system)<built-in function system>>>>print(spam.system('whoami'))None

>>>print(spam.system('too','many','arguments'))Traceback (most recent call last):...TypeError:spam.system() takes exactly one argument (3 given)

Returning an integer¶

Now, let’s take a look at the return value.Instead ofNone, we’ll wantspam.system to return a number – that is,a Pythonint object.Eventually this will be the exit code of a system command,but let’s start with a fixed value, say,3.

The Python C API provides a function to create a Pythonint objectfrom a Cint value:PyLong_FromLong().[5]

To call it, replace thePy_RETURN_NONE with the following 3 lines:

staticPyObject*spam_system(PyObject*self,PyObject*arg){intstatus=3;PyObject*result=PyLong_FromLong(status);returnresult;}

Recompile, restart the Python interpreter again,and check that the function now returns 3:

>>>importspam>>>spam.system('whoami')3

Accepting a string¶

Finally, let’s handle the function argument.

Our C function,spam_system(), takes two arguments.The first one,PyObject*self, will be set to thespam moduleobject.This isn’t useful in our case, so we’ll ignore it.

The other one,PyObject*arg, will be set to the object that the userpassed from Python.We expect that it should be a Python string.In order to use the information in it, we will needto convert it to a C value – in this case, a C string (constchar*).

There’s a slight type mismatch here: Python’sstr objects storeUnicode text, but C strings are arrays of bytes.So, we’ll need toencode the data, and we’ll use the UTF-8 encoding for it.(UTF-8 might not always be correct for system commands, but it’s whatstr.encode() uses by default,and the C API has special support for it.)

The function to encode a Python string into a UTF-8 buffer is namedPyUnicode_AsUTF8()[6].Call it like this:

staticPyObject*spam_system(PyObject*self,PyObject*arg){constchar*command=PyUnicode_AsUTF8(arg);intstatus=3;PyObject*result=PyLong_FromLong(status);returnresult;}

IfPyUnicode_AsUTF8() is successful,command will point to theresulting array of bytes.This buffer is managed by thearg object, which means we don’t need to freeit, but we must follow some rules:

• We should only use the buffer inside thespam_system function.Whenspam_system returns,arg and the buffer it manages might begarbage-collected.

• We must not modify it. This is why we useconst.

IfPyUnicode_AsUTF8() wasnot successful, it returns aNULLpointer.When callingany Python C API, we always need to handle such error cases.The way to do this in general is left for later chapters of this documentation.For now, be assured that we are already handling errors fromPyLong_FromLong() correctly.

For thePyUnicode_AsUTF8() call, the correct way to handle errors isreturningNULL fromspam_system.Add anif block for this:

staticPyObject*spam_system(PyObject*self,PyObject*arg){constchar*command=PyUnicode_AsUTF8(arg);if(command==NULL){returnNULL;}intstatus=3;PyObject*result=PyLong_FromLong(status);returnresult;}

That’s it for the setup.Now, all that is left is calling the C library functionsystem() withthechar* buffer, and using its result instead of the3:

staticPyObject*spam_system(PyObject*self,PyObject*arg){constchar*command=PyUnicode_AsUTF8(arg);if(command==NULL){returnNULL;}intstatus=system(command);PyObject*result=PyLong_FromLong(status);returnresult;}

Compile your module, restart Python, and test.This time, you should see your username – the output of thewhoamisystem command:

>>>importspam>>>result=spam.system('whoami')User Name>>>result0

You might also want to test error cases:

>>>importspam>>>result=spam.system('nonexistent-command')sh: line 1: nonexistent-command: command not found>>>result32512>>>spam.system(3)Traceback (most recent call last):...TypeError:bad argument type for built-in operation

The result¶

Congratulations!You have written a complete Python C API extension module,and completed this tutorial!

Here is the entire source file, for your convenience:

/// Includes#include<Python.h>#include<stdlib.h>     // for system()/// Implementation of spam.systemstaticPyObject*spam_system(PyObject*self,PyObject*arg){constchar*command=PyUnicode_AsUTF8(arg);if(command==NULL){returnNULL;}intstatus=system(command);PyObject*result=PyLong_FromLong(status);returnresult;}/// Module method tablestaticPyMethodDefspam_methods[]={{.ml_name="system",.ml_meth=spam_system,.ml_flags=METH_O,.ml_doc="Execute a shell command.",},{NULL,NULL,0,NULL}/* Sentinel */};/// Module slot tablestaticPyModuleDef_Slotspam_slots[]={{Py_mod_name,"spam"},{Py_mod_doc,"A wonderful module with an example function"},{Py_mod_methods,spam_methods},{0,NULL}};/// Export hook prototypePyMODEXPORT_FUNCPyModExport_spam(void);/// Module export hookPyMODEXPORT_FUNCPyModExport_spam(void){returnspam_slots;}

Appendix: Other build tools¶

You should be able to follow this tutorial – except theRunning your build tool section itself – with a build tool otherthanmeson-python.

The Python Packaging User Guide has alist of recommended tools;be sure to choose one for the C language.

// A workaroundvoid*PyInit_spam(void){returnNULL;}

gcc--shared$(python3-config--cflags--ldflags)spammodule.c-ospam.so