Movatterモバイル変換

[0]ホーム
URL:

ContentsMenuExpandLight modeDark modeAuto light/dark, in light modeAuto light/dark, in dark modeSkip to content
Python Developer's Guide
Logo
Python Developer's Guide
Back to top

Standard library extension modules

In this section, we explain how to configure and compile the CPython projectwith a Cextension module. We will not explain how to write a Cextension module and prefer to give you some links where you can read gooddocumentation:

Some modules in the standard library, such asdatetime orpickle,have identical implementations in C and Python; the C implementation, whenavailable, is expected to improve performance (such extension modules arecommonly referred to asaccelerator modules).

Other modules mainly implemented in Python may import a C helper extensionproviding implementation details (for instance, thecsv module usesthe internal_csv module defined inModules/_csv.c).

Classifying extension modules

Extension modules can be classified into two categories:

  • Abuilt-in extension module is a module built and shipped withthe Python interpreter. A built-in module isstatically linkedinto the interpreter, thereby lacking a__file__ attribute.

    See also

    sys.builtin_module_names — names of built-in modules.

    Built-in modules are built with thePy_BUILD_CORE_BUILTINmacro defined.

  • Ashared (ordynamic) extension module is built as a shared library(.so or.dll file) and isdynamically linked into the interpreter.

    In particular, the module’s__file__ attribute contains the pathto the.so or.dll file.

    Shared modules are built with thePy_BUILD_CORE_MODULEmacro defined. Using thePy_BUILD_CORE_BUILTIN macroinstead causes anImportError when importing the module.

Note

Informally, built-in extension modules can be regarded asrequiredwhile shared extension modules areoptional in the sense that theymight be supplied, overridden or disabled externally.

Usually, accelerator modules are built asshared extension modules,especially if they already have a pure Python implementation.

According toPEP 399,new extension modules MUST provide a working andtested pure Python implementation, unless a special dispensation fromtheSteering Council is given.

Adding an extension module to CPython

Assume that the standard library contains a pure Python modulefoowith the followingfoo.greet() function:

Lib/foo.py
defgreet():return"Hello World!"

Instead of using the Python implementation offoo.greet(), we want touse its corresponding C extension implementation exposed in the_foomodule. Ideally, we want to modifyLib/foo.py as follows:

Lib/foo.py
try:# use the C implementation if possiblefrom_fooimportgreetexceptImportError:# fallback to the pure Python implementationdefgreet():return"Hello World!"

Note

Accelerator modules shouldnever be imported directly. The convention isto mark them as private implementation details with the underscore prefix(namely,_foo in this example).

In order to incorporate the accelerator module, we need to determine:

  • where to update the CPython project tree with the extension module source code,

  • which files to modify to configure and compile the CPython project, and

  • whichMakefile rules to invoke at the end.

Updating the CPython project tree

Usually, accelerator modules are added in theModules directory ofthe CPython project. If more than one file is needed for the extension module,it is more convenient to create a sub-directory inModules.

In the simplest example where the extension module consists of one file, it maybe placed inModules asModules/_foomodule.c. For a non-trivialexample of the extension module_foo, we consider the following workingtree:

By convention, the source file containing the extension module implementationis called<NAME>module.c, where<NAME> is the name of the module thatwill be later imported (in our case_foo). In addition, the directorycontaining the implementation should also be named similarly.

Modules/_foo/helper.h
#ifndef _FOO_HELPER_H#define _FOO_HELPER_H#include"Python.h"typedefstruct{/* ... */}foomodule_state;staticinlinefoomodule_state*get_foomodule_state(PyObject*module){void*state=PyModule_GetState(module);assert(state!=NULL);return(foomodule_state*)state;}/* Helper used in Modules/_foo/_foomodule.c * but implemented in Modules/_foo/helper.c. */externPyObject*_Py_greet_fast(void);#endif// _FOO_HELPER_H

Tip

Functions or data that do not need to be shared across different C sourcefiles should be declaredstatic to avoid exporting their symbols fromlibpython.

If symbols need to be exported, their names must start withPy or_Py. This can be verified bymakesmelly. For more details,please refer to the section onChanging Python’s C API.

Modules/_foo/helper.c
#include"_foomodule.h"PyObject*_Py_greet_fast(void){returnPyUnicode_FromString("Hello World!");}
Modules/_foo/_foomodule.c
#include"helper.h"#include"clinic/_foomodule.c.h"/* Functions for the extension module's state */staticintfoomodule_exec(PyObject*module){// imports, static attributes, exported classes, etcreturn0;}staticintfoomodule_traverse(PyObject*m,visitprocvisit,void*arg){foomodule_state*st=get_foomodule_state(m);// call Py_VISIT() on the state attributesreturn0;}staticintfoomodule_clear(PyObject*m){foomodule_state*st=get_foomodule_state(m);// call Py_CLEAR() on the state attributesreturn0;}staticvoidfoomodule_free(void*m){(void)foomodule_clear((PyObject*)m);}/* Implementation of publicly exported functions. *//*[clinic input]module foo[clinic start generated code]*//*[clinic end generated code: output=... input=...]*//*[clinic input]foo.greet -> object[clinic start generated code]*/staticPyObject*foo_greet_impl(PyObject*module)/*[clinic end generated code: output=... input=...]*/{return_Py_greet_fast();}/* Exported module's data */staticPyMethodDeffoomodule_methods[]={// macro in 'clinic/_foomodule.c.h' after running 'make clinic'FOO_GREET_METHODDEF{NULL,NULL}};staticstructPyModuleDef_Slotfoomodule_slots[]={// 'foomodule_exec' may be NULL if the state is trivial{Py_mod_exec,foomodule_exec},{Py_mod_multiple_interpreters,Py_MOD_PER_INTERPRETER_GIL_SUPPORTED},{Py_mod_gil,Py_MOD_GIL_NOT_USED},{0,NULL},};staticstructPyModuleDeffoomodule={PyModuleDef_HEAD_INIT,.m_name="_foo",.m_doc="some doc",// or NULL if not needed.m_size=sizeof(foomodule_state),.m_methods=foomodule_methods,.m_slots=foomodule_slots,.m_traverse=foomodule_traverse,// or NULL if the state is trivial.m_clear=foomodule_clear,// or NULL if the state is trivial.m_free=foomodule_free,// or NULL if the state is trivial};PyMODINIT_FUNCPyInit__foo(void){returnPyModuleDef_Init(&foomodule);}

Tip

Recall that thePyInit_<NAME> function must be suffixed by themodule name<NAME> used in import statements (here_foo),and which usually coincides withPyModuleDef.m_name.

Other identifiers such as those used inArgument Clinicinputs do not have such naming requirements.

Configuring the CPython project

Now that we have added our extension module to the CPython source tree,we need to update some configuration files in order to compile the CPythonproject on different platforms.

UpdatingModules/Setup.{bootstrap,stdlib}.in

Depending on whether the extension module is required to get a functioninginterpreter or not, we updateModules/Setup.bootstrap.in orModules/Setup.stdlib.in. In the former case, the extensionmodule is necessarily built as a built-in extension module.

Tip

For accelerator modules,Modules/Setup.stdlib.in should bepreferred overModules/Setup.bootstrap.in.

For built-in extension modules, updateModules/Setup.bootstrap.inby adding the following line after the*static* marker:

*static*..._foo _foo/_foomodule.c _foo/helper.c...

The syntax is<NAME><SOURCES> where<NAME> is the name of themodule used inimport statements and<SOURCES> is the listof space-separated source files.

For other extension modules, updateModules/Setup.stdlib.inby adding the following line after the*@MODULE_BUILDTYPE@* markerbut before the*shared* marker:

*@MODULE_BUILDTYPE@*...@MODULE__FOO_TRUE@_foo _foo/_foomodule.c _foo/helper.c...*shared*

The@MODULE_<NAME_UPPER>_TRUE@<NAME> marker expects<NAME_UPPER> tobe the upper-cased form of<NAME>, where<NAME> has the same meaningas before (in our case,<NAME_UPPER> and<NAME> are_FOO and_foo respectively). The marker is followed by the list of source files.

If the extension module must be built as ashared module, put the@MODULE__FOO_TRUE@_foo line after the*shared* marker:

...*shared*...@MODULE__FOO_TRUE@_foo _foo/_foomodule.c _foo/helper.c

Updatingconfigure.ac

  • Locate theSRCDIRS variable and add the following line:

    AC_SUBST([SRCDIRS])SRCDIRS="\...Modules/_foo \..."

    Note

    This step is only needed when adding new source directories tothe CPython project.

  • Find the section containingPY_STDLIB_MOD andPY_STDLIB_MOD_SIMPLEusages and add the following line:

    dnl always enabled extension modules...PY_STDLIB_MOD_SIMPLE([_foo], [-I\$(srcdir)/Modules/_foo], [])...

    ThePY_STDLIB_MOD_SIMPLE macro takes as arguments:

    • the module name<NAME> used inimport statements,

    • the compiler flags (CFLAGS), and

    • the linker flags (LDFLAGS).

    If the extension module may not be enabled or supported depending on thehost configuration, use thePY_STDLIB_MOD macro instead, which takesas arguments:

    • the module name<NAME> used inimport statements,

    • a boolean indicating whether the extension isenabled or not,

    • a boolean indicating whether the extension issupported or not,

    • the compiler flags (CFLAGS), and

    • the linker flags (LDFLAGS).

    For instance, enabling the_foo extension on Linux platforms, butonly providing support for 32-bit architecture, is achieved as follows:

    PY_STDLIB_MOD([_foo],              [test "$ac_sys_system" = "Linux"],              [test "$ARCH_RUN_32BIT" = "true"],              [-I\$(srcdir)/Modules/_foo], [])

    More generally, the host’s configuration status of the extension isdetermined as follows:

    Enabled

    Supported

    Status

    true

    true

    yes

    true

    false

    missing

    false

    true or false

    disabled

    The extension status isn/a if the extension is marked unavailableby thePY_STDLIB_MOD_SET_NA macro. To mark an extension as unavailable,find the usages ofPY_STDLIB_MOD_SET_NA inconfigure.ac andadd the following line:

    dnl Modules that are not available on some platformsAS_CASE([$ac_sys_system],    ...    [PLATFORM_NAME], [PY_STDLIB_MOD_SET_NA([_foo])],    ...)

Tip

Consider reading the comments and configurations for existing modulesinconfigure.ac for guidance on adding new external builddependencies for extension modules that need them.

UpdatingMakefile.pre.in

If needed, add the following line to the section for module dependencies:

########################################################################### Module dependencies and platform-specific files...MODULE__FOO_DEPS=$(srcdir)/Modules/_foo/helper.h...

TheMODULE_<NAME_UPPER>_DEPS variable follows the same namingrequirements as the@MODULE_<NAME_UPPER>_TRUE@<NAME> marker.

Updating MSVC project files

We describe the minimal steps for compiling on Windows using MSVC.

  • UpdatePC/config.c:

    ...// add the entry point prototypeexternPyObject*PyInit__foo(void);...// update the entry points tablestruct_inittab_PyImport_Inittab[]={...{"_foo",PyInit__foo},...{0,0}};...

    Each item in_PyImport_Inittab consists of the module name to import,here_foo, with the correspondingPyInit_* function correctlysuffixed.

  • UpdatePCbuild/pythoncore.vcxproj:

    <!-- group with header files ..\Modules\<MODULE>.h --><ItemGroup>...<ClIncludeInclude="..\Modules\_foo\helper.h"/>...</ItemGroup><!-- group with source files ..\Modules\<MODULE>.c --><ItemGroup>...<ClCompileInclude="..\Modules\_foo\_foomodule.c"/><ClCompileInclude="..\Modules\_foo\helper.c"/>...</ItemGroup>

  • UpdatePCbuild/pythoncore.vcxproj.filters:

    <!-- group with header files ..\Modules\<MODULE>.h --><ItemGroup>...<ClIncludeInclude="..\Modules\_foo\helper.h"><Filter>Modules\_foo</Filter></ClInclude>...</ItemGroup><!-- group with source files ..\Modules\<MODULE>.c --><ItemGroup>...<ClCompileInclude="..\Modules\_foo\_foomodule.c"><Filter>Modules\_foo</Filter></ClCompile><ClCompileInclude="..\Modules\_foo\helper.c"><Filter>Modules\_foo</Filter></ClCompile>...<ItemGroup>

Tip

Header files use<ClInclude> tags, whereassource files use<ClCompile> tags.

Compiling the CPython project

Now that the configuration is in place, it remains to compile the project:

makeregen-configure./configuremakeregen-allmakeregen-stdlib-module-namesmake

Tip

Usemake-jN to speed-up compilation by utilizing as many CPU coresas possible, whereN is as many CPU cores you want to spare (and havememory for). Be careful usingmake-j with no argument, as this putsno limit on the number of jobs, and compilation can sometimes use up alot of memory (like when building with LTO).

  • makeregen-configure updates theconfigure script.

    Theconfigure script must be generated using a specific versionofautoconf. To that end, theTools/build/regen-configure.shscript which theregen-configure rule is based on either requires Dockeror Podman, the latter being assumed by default.

    Tip

    We recommend installingPodmaninstead of Docker since the former does not require a background serviceand avoids creating files owned by theroot user in some cases.

  • makeregen-all is responsible for regenerating header files andinvoking other scripts, such asArgument Clinic.Execute this rule if you do not know which files should be updated.

  • makeregen-stdlib-module-names updates the standard module names, making_foo discoverable and importable viaimport_foo.

  • The finalmake step is generally not needed since the previousmakeinvocations may completely rebuild the project, but it could be needed insome specific cases.

Troubleshooting

This section addresses common issues that you may face when followingthis example of adding an extension module.

No rule to make targetregen-configure

This usually happens after runningmakedistclean (which removestheMakefile). The solution is to regenerate theconfigurescript as follows:

./configure# for creating the 'Makefile' filemakeregen-configure# for updating the 'configure' script./configure# for updating the 'Makefile' file

If missing, theconfigure script can be regeneratedby executingTools/build/regen-configure.sh:

./Tools/build/regen-configure.sh# create an up-to-date 'configure'./configure# create an up-to-date 'Makefile'

makeregen-configure and missing permissions with Docker

If Docker complains about missing permissions, this Stack Overflow postcould be useful in solving the issue:How to fix docker: permission denied. Alternatively, you may tryusingPodman.

MissingPy_BUILD_CORE define when using internal headers

By default, the CPythonStable ABI is exposed via#include"Python.h". In some cases, this may be insufficientand internal headers fromInclude/internal are needed;in particular, those headers require thePy_BUILD_COREmacro to be defined.

To that end, one should define thePy_BUILD_CORE_BUILTINor thePy_BUILD_CORE_MODULE macro depending on whether theextension module is built-in or shared. Using either of the two macrosimpliesPy_BUILD_CORE and gives access to CPython internals:

Definition ofPy_BUILD_CORE_BUILTIN
#ifndef Py_BUILD_CORE_MODULE#  define Py_BUILD_CORE_BUILTIN 1#endif
Definition ofPy_BUILD_CORE_MODULE
#ifndef Py_BUILD_CORE_BUILTIN#  define Py_BUILD_CORE_MODULE 1#endif

Tips

In this section, we give some tips for improving the quality ofextension modules meant to be included in the standard library.

Restricting to the Limited API

In order for non-CPython implementations to benefit from new extension modules,it is recommended to use theLimited API. Instead ofexposing the entire Stable ABI, define thePy_LIMITED_API macrobefore the#include"Python.h" directive:

Using the 3.13 Limited API.
#include"pyconfig.h"    // Py_GIL_DISABLED#ifndef Py_GIL_DISABLED#  define Py_LIMITED_API 0x030d0000#endif#include"Python.h"

This makes the extension module non-CPython implementation-friendly byremoving the dependencies to CPython internals.

On this page
[8]ページ先頭
©2009-2026 Movatter.jp