Introduction
This is the first part of a two-part series on Python. This article will cover the basic techniques for extending Python using C/C++. The second article will cover the techniques for embedding the Python interpreter in your C/C++ application. While I don't often use Python extensions directly, the information covered here is an absolute requirement for embedded use.
This article is not meant to be an introduction to Python. It assumes a working knowledge of the language.
You will need a Python distribution to build and use the sample. There are two major distributors of Python:Python.org andActiveState.com. Either one will suffice, but my preference these days is the ActiveState one. They've compiled the help files into a Windows HTML help file that I find easier to navigate, than the basic distribution. Plus it comes with all of the Windows extension libraries.
They both come withinclude andlibs directories, so you will not need to download the source code. The source will be necessary, however, in the second part of this series.
Extensions - What are they?
Python's motto is "Batteries included". By this they mean that, out of the package, Python does a lot. It comes with many extra modules that give the user, access to such features as sockets, CGI, URL parsing and HTTP support, XML processing, MIME, threads, and even XML-based RPC. Despite all of these extras, we will always need/want to customize.
Extension modules come in two forms: native Python, and C/C++ dynamically linked libraries (DLLs). Native Python modules are simply Python scripts that are available to be imported by user scripts. Creating native Python modules is as simple as writing a Python script.
C/C++ extension modules are compiled DLLs with a standard exported function that handles module initialization and registration. This article covers these DLL-based extension modules.
The API
Python is written in C, and the authors have been kind enough to expose and document most of the interpreter internals. It is through this API that we gain the access we need, to extend the language.
Python objects
Every object, nay every value, in Python is represented internally as aPyObject.PyObject is a structure that defines all of the handler entry points, and maintains type information and reference counts. One of the fundamentals of Python extension programming is that, whenever you manipulate any Python object in C/C++, you will be manipulating aPyObject.
That being said, you will rarely use thePyObject API routines. Instead, you will use the API routines that apply to the specific Python type being manipulated. Please see the Python C/C++ API documentation for specifics.
Reference counting
Python handles basic memory management with a reference counting mechanism. Each object has a reference count that gets incremented when the object is copied, and decremented when the object reference is dropped.
class doodle: # etc.d = doodle() # reference count =1e = d # reference count =2del(d) # reference count =1del(e) # reference count =0object deleted
When manipulating Python objects in C/C++, one must be conscious of the reference counts. Certain functions (marked in the Python C API documents) return new objects; others return borrowed references. Use thePy_INCREF() andPy_DECREF() macros to assist you. Also, I suggest that you mark each questionable API call with a comment noting the return type.
PyObject *pList = PyList_New(5);// new reference...PyObject *pItem = PyList_GetItem(2);// borrowed reference...Py_DECREF(pList);
Reference counts are important! I have spent many hours trying to track down memory leaks only to find that I didn'tPy_DECREF() an object when I should have.
Python types
There are six major native data types in Python: integers, floats, strings, tuples, lists and dictionaries. Python supports a variety of other native types (complex, long integers, etc), the use of which will be left as the proverbial exercise for the reader.
Integers, floats and strings
These are just what you would expect. The only thing you need to know is how to build and manipulate them.
// build an integerPyObject *pInt = Py_BuildValue("i",147);// new referenceassert(PyInt_Check(pInt));int i = PyInt_AsLong(pInt);Py_DECREF(pInt);// build a floatPyObject *pFloat = Py_BuildValue("f",3.14159f);// new referenceassert(PyFloat_Check(pFloat));float f = PyFloat_AsDouble(pFloat);Py_DECREF(pFloat);// build a stringPyObject *pString = Py_BuildValue("s","yabbadabbadoo");// new referenceassert(PyString_Check(pString);int nLen = PyString_Size(pString);char *s = PyString_AsString(pString);Py_DECREF(pString);Tuples
Tuples are fixed-length immutable arrays. When a Python script calls a C/C++ extension method, all non-keyword arguments are passed in a tuple. Needless to say, parsing this tuple tends to be the first thing done in your methods.
Here is a mish-mash of tuple use:
// create the tuplePyObject *pTuple = PyTuple_New(3);// new referenceassert(PyTuple_Check(pTuple));assert(PyTuple_Size(pTuple) ==3);// set the itemPyTuple_SetItem(pTuple,0, Py_BuildValue("i",1));PyTuple_SetItem(pTuple,1, Py_BuildValue("f",2.0f));PyTuple_SetItem(pTuple,2, Py_BuildValue("s","three"));// parse tuple itemsint i;float f;char *s;if(!PyArg_ParseTuple(pTuple,"ifs", &i, &f, &s)) PyErr_SetString(PyExc_TypeError,"invalid parameter");// cleanupPy_DECREF(pTuple);PyArg_ParseTuple() is probably one of the most commonly used API functions. The second parameter is a string that dictates the types of objects expected in the tuple.ifs means: integer, float, string. Please see the API documentation for a detailed explanation, and a list of the other type characters.
Lists
Lists are like STL vectors. They allow random access and iteration over stored objects. Here is an example of typical list use:
// create the listPyObject *pList = PyList_New(5);// new referenceassert(PyList_Check(pList));// set some initial valuesfor(int i =0; i<5; ++i) PyList_SetItem(pList, i, Py_BuildValue("i", i));// insert an itemPyList_Insert(pList,3, Py_BuildValue("s","inserted"));// append an itemPyList_Append(pList, Py_BuildValue("s","appended"));// sort the listPyList_Sort(pList);// reverse the listPyList_Reverse(pList);// fetch and manipulate a list slicePyObject *pSlice = PyList_GetSlice(pList,2,4);// new referencefor(int j =0; j< PyList_Size(pSlice); ++j) { PyObject *pValue = PyList_GetItem(pList, j); assert(pValue);}Py_DECREF(pSlice);// cleanupPy_DECREF(pList);Dictionaries
Dictionaries are the equivalent of STL maps. They map keys to values. Here is an example of typical dictionary use:
// create the dictionaryPyObject *pDict = PyDict_New();// new referenceassert(PyDict_Check(pDict));// add a few named valuesPyDict_SetItemString(pDict,"first", Py_BuildValue("i",1));PyDict_SetItemString(pDict,"second", Py_BuildValue("f",2.0f));// enumerate all named valuesPyObject *pKeys = PyDict_Keys();// new referencefor(int i =0; i< PyList_Size(pKeys); ++i) { PyObject *pKey = PyList_GetItem(pKeys, i);// borrowed reference PyObject *pValue = PyDict_GetItem(pDict, pKey);// borrowed reference assert(pValue);}Py_DECREF(pKeys);// remove a named valuePyDict_DelItemString(pDict,"second");// cleanupPy_DECREF(pDict);Extension concepts
An extension module typically consists of three parts: the actual exported functions, the method table and the initialization function.
First, we will look at a simple example of an extension module. Then we will examine each part individually.
A typical extension module will look like this:
// exported yinkle() functionstatic PyObject *wanklib_yinkle(PyObject *pSelf, PyObject *pArgs){char *szString;int nInt;float fFloat; PyObject *pList;if(!PyArg_ParseTuple(pArgs,"sifo", &szString, &nInt, &fFloat, &pList)) { PyErr_SetString(PyExc_TypeError,"yinkle() invalid parameter");return NULL; }if(!PyList_Check(pList)) { PyErr_SetString(PyExc_TypeError,"yinkle() fourth parameter must be a list");return NULL; } PyList_Append(pList, Py_BuildObject("f", strlen(szString) * nInt / fFloat)); Py_INCREF(Py_None);return Py_None;}// wanklib method tablestatic PyMethodDef WankLibMethods[] = { {"yinkle", wanklib_yinkle, METH_VARARGS,"Do a bit of stuff."}, {NULL, NULL,0, NULL}};// wanklib initialization functionvoid initwanklib(void) {// initialize module PyObject *pModule = Py_InitModule("wanklib", WankLibMethods);// fetch module dictionary and add non-function symbols PyObject *pDict = PyModule_GetDict(pModule); PyDict_SetItemString(pDict,"eleven", Py_BuildValue("i",147)); PyDict_SetItemString(pDict,"doubleyew", Py_BuildValue("s","kay"));}Initialization function
Every extension module must export a function calledinitmodule. When a Python script requests that the module be imported, Python queries the library for that exact named function and calls it. The initialization function is responsible for telling Python about the functions, variables and classes that it exports.
Method table
The initialization function will call the Python routinePy_InitModule() to register the module methods. It will pass the name by which the new module will be known, and a table describing the exported methods. Each table entry consists of four parts: the callable name string, the function itself, a parameter describing how parameters will be passed, and a documentation string. The last entry in the table needs to be a sentinel withNULL entries.
The name string is the name by which the method will be callable from Python. The parameter type marker can beMETH_VARARGS orMETH_KEYWORDS.METH_VARARGS is the standard way of passing parameters; they arrive packaged in a tuple. SpecifyingMETH_KEYWORDS requests that, named parameters be passed in a dictionary.
Methods
All extension methods have the same prototype (given that they are marked asMETH_VARARGS):
PyObject *method(PyObject *pSelf, PyObject *pArgs);
All extension methods must return aPyObject pointer. If the function has no real return value, you must return a pointer to the global "None" object, after incrementing its reference:
PyObject *method(PyObject *pSelf, PyObject *pArgs) { Py_INCREF(Py_None);return Py_None;}To signify that an error has occurred and to throw a Python exception, you must returnNULL and set the error string:
PyObject *method(PyObject *pSelf, PyObject *pArgs) { PyErr_SetString(PyExc_StandardError,"something bad happened");return NULL;}The first argument to extension methods is a "self" pointer and is really only valid when you are building custom classes. These will be detailed in the next article.
The second argument is a tuple containing each parameter in order. As mentioned above, parsing this tuple is usually the first thing that happens.
Variables
Each Python module has a dictionary of local objects. In order to export variables from your module, all you need to do is add them to this dictionary.Py_InitModule() returns a pointer to the initialized module.PyModule_GetDict() retrieves the local object dictionary.
PyObject *pModule = Py_InitModule("wanklib", wankLibMethods);PyObject *pDict = PyModule_GetDict(pModule);PyDict_SetItemString(pDict,"someVar", Py_BuildValue("i",147));Implementation
In Windows, Python extensions are simply DLL files with a known exported symbol (the initialization function). In order to build an extension, you must create a Win32 dynamic linked library project in Visual Studio. ChooseA DLL that exports some symbols, so you have a bit of a template from which to work. I'm sure you could build an extension using the MFC AppWizard but I've never tried it and don't intend to.
Simple extensions can be built in a single file, and will follow the layout shown above in the example.
All Python API declarations are accessed by including one file:Python.h. It is located in the include subdirectory below your Python installation. Rather than hard coding the path, it's much more desirable to add the directory to yourTools/Options/Directories list. While you're at it, add thelibs subdirectory to the list of Library Files search paths as well.
There is no need to explicitly link the Python library. ThePython.h include file uses a pragma to force the proper linkage.
NOTE: The pragma inPython.h forces the linkage of the debug build of Python:Python22_d.lib (version may be different depending on the version you have installed). If you haven't downloaded the Python source code, you likely don't have this library. Your choices are to download and build the debug versions, or to build your extensions in release mode.
In order to remove the C++ name mangling, you need to define your initialization function asextern"C".
And lastly, once compiled, place your DLL file in theDLLs subdirectory off of your Python installation. It will get picked up automatically when you try to import it.
Beyond that, you should be ready to go.
Example
The example that I have included simply wraps the Mersenne Twister pseudo random number generator. I have used what appears to be original code by the "inventors", Makoto Matsumoto and Takuji Nishimura. Themtprng module provides two methods:sgenrand() to seed the generator, andgenrand() to generate a number on [0,1]. The code compiles with VS6 and SP5, with Python 2.2 installed, although any Python version beyond 1.5.3 should be fine.
Good luck!
History
- November 21 2002 - Created.
News 
Suggestion 
Question 
Bug 
Answer 
Joke 
Rant 
Admin