Embedding Cython modules in C/C++ applications

This is a stub documentation page. PRs very welcome.

Quick links:

Initialising your main module

Most importantly, DO NOT call the module init function instead of importingthe module. This is not the right way to initialise an extension module.(It was always wrong but used to work before, but since Python 3.5, it iswrongand no longer works.)

For details, see the documentation of themodule init functionin CPython andPEP 489 regarding the moduleinitialisation mechanism in CPython 3.5 and later.

ThePyImport_AppendInittab()function in CPython allows registering statically (or dynamically) linked extensionmodules for later imports. An example is given in the documentation of the moduleinit function that is linked above.

Embedding example code

The following is a simple example that shows the main steps for embedding aCython module (embedded.pyx) in Python 3.x.

First, here is a Cython module that exports a C function to be called by externalcode. Note that thesay_hello_from_python() function is declared aspublicto export it as a linker symbol that can be used by other C files, which in thiscase isembedded_main.c.

# embedded.pyx# The following two lines are for test purposes only, please ignore them.# distutils: sources = embedded_main.c# tag: py3only# tag: no-cppTEXT_TO_SAY='Hello from Python!'cdefpublicintsay_hello_from_python()except-1:print(TEXT_TO_SAY)return0

The Cmain() function of your program could look like this:

 1/* embedded_main.c */ 2 3/* This include file is automatically generated by Cython for 'public' functions. */ 4#include"embedded.h" 5 6#ifdef __cplusplus 7extern"C"{ 8#endif 910int11main(intargc,char*argv[])12{13PyObject*pmodule;14wchar_t*program;1516program=Py_DecodeLocale(argv[0],NULL);17if(program==NULL){18fprintf(stderr,"Fatal error: cannot decode argv[0], got %d arguments\n",argc);19exit(1);20}2122/* Add a built-in module, before Py_Initialize */23if(PyImport_AppendInittab("embedded",PyInit_embedded)==-1){24fprintf(stderr,"Error: could not extend in-built modules table\n");25exit(1);26}2728/* Pass argv[0] to the Python interpreter */29Py_SetProgramName(program);3031/* Initialize the Python interpreter.  Required.32       If this step fails, it will be a fatal error. */33Py_Initialize();3435/* Optionally import the module; alternatively,36       import can be deferred until the embedded script37       imports it. */38pmodule=PyImport_ImportModule("embedded");39if(!pmodule){40PyErr_Print();41fprintf(stderr,"Error: could not import module 'embedded'\n");42gotoexit_with_error;43}4445/* Now call into your module code. */46if(say_hello_from_python()<0){47PyErr_Print();48fprintf(stderr,"Error in Python code, exception was printed.\n");49gotoexit_with_error;50}5152/* ... */5354/* Clean up after using CPython. */55PyMem_RawFree(program);56Py_Finalize();5758return0;5960/* Clean up in the error cases above. */61exit_with_error:62PyMem_RawFree(program);63Py_Finalize();64return1;65}6667#ifdef __cplusplus68}69#endif

(Adapted from theCPython documentation.)

Instead of writing such amain() function yourself, you can also letCython generate one into your module’s C file with thecython--embedoption. Or use thecython_freezescript to embed multiple modules. See theembedding demo programfor a complete example setup.

Be aware that your application will not contain any external dependencies thatyou use (including Python standard library modules) and so may not be truly portable.If you want to generate a portable application we recommend using a specializedtool (e.g.PyInstallerorcx_freeze) to find andbundle these dependencies.

Troubleshooting

Here’s some of the things that can go wrong when embedding Cython code.

Not initializing the Python interpreter

Cython doesn’t compile to “pure stand-alone C code”. Instead Cython compiles to a bunch ofPython C API calls that depend on the Python interpreter. Therefore, in your main functionyoumust initialize the Python interpreter withPy_Initialize(). You should do thisas early as possible in yourmain() function.

Very occasionally you may get away without it, for exceptionally simple programs. Thisis pure luck, and you should not rely on it. There is no “safe subset” of Cython that’sdesigned to run without the interpreter.

The consequence of not initializing the Python interpreter is likely to be crashes.

You should only initialize the interpreter once - a lot of modules, including most Cythonmodules and Numpy, don’t currently like being imported multiple times. Therefore if you’redoing occasional Python/Cython calculations in a larger program what youdon’t do is:

voidrun_calculation(){Py_Initialize();//UsePython/CythoncodePy_Finalize();}

The chances are you will get mystery unexplained crashes.

Not setting the Python path

If your module imports anything (and possibly even if it doesn’t) then it’ll needthe Python path set so it knows where to look for modules. Unlikely the standaloneinterpreter, embedded Python doesn’t set this up automatically.

PySys_SetPath(...) is the easiest way of doing this (just afterPy_Initialize()ideally). You could also usePySys_GetObject("path") and then append to thelist that it returns.

if you forget to do this you will likely see import errors.

Not importing the Cython module

Cython doesn’t create standalone C code - it creates C code that’s designed to beimported as a Cython module. The “import” function sets up a lot of the basicinfrastructure necessary for you code to run. For example, strings are initializedat import time, and built-ins likeprint are found and stashed within yourCython module.

Therefore, if you decide to skip the initialization and just go straight torunning your public functions you will likely experience crashes (even forsomething as simple as using a string).

InitTab

The preferred way to set up an extension module so that it’s availablefor import in modern Python (>=3.5) is to use theinittab mechanismwhich is detailed inelsewhere in the documentation. This should be donebeforePy_Initialize().

Forcing single-phase

If for some reason you aren’t able to add your module to the inittab beforePython is initialized (a common reason is trying to import another Cythonmodule built into a single shared library) then you can disablemulti-phase initialization by definingCYTHON_PEP489_MULTI_PHASE_INIT=0for your C compiler (for gcc this would be-DCYTHON_PEP489_MULTI_PHASE_INIT=0at the command line). If you do this then you can run the module initfunction directly (PyInit_<module_name> on Python 3).This reallyisn’t the preferred option.

Working with multi-phase

It is possible to run the multi-phase initialization manually yourself.One of the Cython developers has written aguide showing how to do this.However, he considers it sufficiently hacky that it is only linked here,and not reproduced directly. It is an option though, if you’re unable touse the inittab mechanism before initializing the interpreter.

Problems with multiprocessing and pickle

If you try to usemultiprocessing while using a Cython module embedded intoan executable it will likely fail with errors related to the pickle module.multiprocessing often uses pickle to serialize and deserialize data tobe run in another interpreter. What happens depends on the multiprocessing“start method”. However, on the “spawn” start method used on Windows, it starts afresh copy of thePython interpreter (rather than a fresh copy of your embedded program)and then tries to import your Cython module. Since your Cython module is onlyavailable by the inittab mechanism and not be a regular import then that importfails.

The solution likely involves settingmultiprocessing.set_executable to point to yourembedded program then modifying that program to handle the--multiprocessing-fork command-line argument that multiprocessing passesto the Python interpreter. You may also need to callmultiprocessing.freeze_support().

At the moment that solution is untested so you should treat multiprocessingfrom an embedded Cython executable as unsupported.