For downstream package authors#
This document aims to explain some best practices for authoring a package thatdepends on NumPy.
Understanding NumPy’s versioning and API/ABI stability#
NumPy uses a standard,PEP 440 compliant, versioning scheme:major.minor.bugfix
. Amajor release is highly unusual and if it happensit will most likely indicate an ABI break. NumPy 1.xx releases happened from2006 to 2023; NumPy 2.0 in early 2024 is the first release which changed theABI (minor ABI breaks for corner cases may have happened a few times in minorreleases).Minor versions are released regularly, typically every 6 months. Minorversions contain new features, deprecations, and removals of previouslydeprecated code.Bugfix releases are made even more frequently; they do notcontain any new features or deprecations.
It is important to know that NumPy, like Python itself and most otherwell known scientific Python projects, doesnot use semantic versioning.Instead, backwards incompatible API changes require deprecation warnings for atleast two releases. For more details, seeNEP 23 — Backwards compatibility and deprecation policy.
NumPy has both a Python API and a C API. The C API can be used directly or viaCython, f2py, or other such tools. If your package uses the C API, then ABI(application binary interface) stability of NumPy is important. NumPy’s ABI isforward but not backward compatible. This means: binaries compiled against agiven target version of NumPy’s C API will still run correctly with newer NumPyversions, but not with older versions.
Modules can also be safely built against NumPy 2.0 or later inCPython’s abi3 mode, which allowsbuilding against a single (minimum-supported) version of Python but beforward compatible higher versions in the same series (e.g.,3.x
).This can greatly reduce the number of wheels that need to be built anddistributed. For more information and examples, see thecibuildwheel docs.
Testing against the NumPy main branch or pre-releases#
For large, actively maintained packages that depend on NumPy, we recommendtesting against the development version of NumPy in CI. To make this easy,nightly builds are provided as wheels athttps://anaconda.org/scientific-python-nightly-wheels/. Example install command:
pipinstall-U--pre--only-binary:all:-ihttps://pypi.anaconda.org/scientific-python-nightly-wheels/simplenumpy
This helps detect regressions in NumPy that need fixing before the next NumPyrelease. Furthermore, we recommend to raise errors on warnings in CI for thisjob, either all warnings or otherwise at leastDeprecationWarning
andFutureWarning
. This gives you an early warning about changes in NumPy toadapt your code.
If you want to test your own wheel builds against the latest NumPy nightlybuild and you’re usingcibuildwheel
, you may need something like this inyour CI config file:
CIBW_ENVIRONMENT:"PIP_PRE=1 PIP_EXTRA_INDEX_URL=https://pypi.anaconda.org/scientific-python-nightly-wheels/simple"
Adding a dependency on NumPy#
Build-time dependency#
Note
Before NumPy 1.25, the NumPy C-API wasnot exposed in a backwardscompatible way by default. This means that when compiling with a NumPyversion earlier than 1.25 you have to compile with the oldest version youwish to support. This can be done by usingoldest-supported-numpy.Please see theNumPy 1.24 documentation.
If a package either uses the NumPy C API directly or it uses some other toolthat depends on it like Cython or Pythran, NumPy is abuild-time dependencyof the package.
By default, NumPy will expose an API that is backwards compatible with theoldest NumPy version that supports the currently oldest compatible Pythonversion. NumPy 1.25.0 supports Python 3.9 and higher and NumPy 1.19 is thefirst version to support Python 3.9. Thus, we guarantee that, when usingdefaults, NumPy 1.25 will expose a C-API compatible with NumPy 1.19.(the exact version is set within NumPy-internal header files).
NumPy is also forward compatible for all minor releases, but a major releasewill require recompilation (see NumPy 2.0-specific advice further down).
The default behavior can be customized for example by adding:
#define NPY_TARGET_VERSION NPY_1_22_API_VERSION
before including any NumPy headers (or the equivalent-D
compiler flag) inevery extension module that requires the NumPy C-API.This is mainly useful if you need to use newly added API at the cost of notbeing compatible with older versions.
If for some reason you wish to compile for the currently installed NumPyversion by default you can add:
#ifndef NPY_TARGET_VERSION#define NPY_TARGET_VERSION NPY_API_VERSION#endif
Which allows a user to override the default via-DNPY_TARGET_VERSION
.This define must be consistent for each extension module (use ofimport_array()
) and also applies to the umath module.
When you compile against NumPy, you should add the proper version restrictionsto yourpyproject.toml
(see PEP 517). Since your extension will not becompatible with a new major release of NumPy and may not be compatible withvery old versions.
For conda-forge packages, please seehere.
as of now, it is usually as easy as including:
host:-numpyrun:-{{pin_compatible('numpy')}}
Runtime dependency & version ranges#
NumPy itself and many core scientific Python packages have agreed on a schedulefor dropping support for old Python and NumPy versions:NEP29. Werecommend all packages depending on NumPy to follow the recommendations in NEP29.
Forrun-time dependencies, specify version bounds usinginstall_requires
insetup.py
(assuming you usenumpy.distutils
orsetuptools
to build).
Most libraries that rely on NumPy will not need to set an upperversion bound: NumPy is careful to preserve backward-compatibility.
That said, if you are (a) a project that is guaranteed to releasefrequently, (b) use a large part of NumPy’s API surface, and (c) isworried that changes in NumPy may break your code, you can set anupper bound of<MAJOR.MINOR+N
with N no less than 3, andMAJOR.MINOR
being the current release of NumPy[*]. If you use the NumPyC API (directly or via Cython), you can also pin the current majorversion to prevent ABI breakage. Note that setting an upper bound onNumPy mayaffect the ability of your library to be installedalongside other, newer packages.
The reason for settingN=3
is that NumPy will, on therare occasion where it makes breaking changes, raise warningsfor at least two releases. (NumPy releases about once every sixmonths, so this translates to a window of at least a year;hence the subsequent requirement that your project releases atleast on that cadence.)
Note
SciPy has more documentation on how it builds wheels and deals with itsbuild-time and runtime dependencieshere.
NumPy and SciPy wheel build CI may also be useful as a reference, it can befoundhere for NumPy andhere for SciPy.
NumPy 2.0-specific advice#
NumPy 2.0 is an ABI-breaking release, however it does contain support forbuilding wheels that work on both 2.0 and 1.xx releases. It’s important to understand that:
When you build wheels for your package using a NumPy 1.xx version at buildtime, thosewill not work with NumPy 2.0.
When you build wheels for your package using a NumPy 2.x version at buildtime, thosewill work with NumPy 1.xx.
The first time the NumPy ABI for 2.0 is guaranteed to be stable will be therelease of the first release candidate for 2.0 (i.e., 2.0.0rc1). Our advice forhandling your dependency on NumPy is as follows:
In the main (development) branch of your package, do not add any constraints.
If you rely on the NumPy C API (e.g. via direct use in C/C++, or via Cythoncode that uses NumPy), add a
numpy<2.0
requirement in yourpackage’s dependency metadata for releases / in release branches. Do thisuntil numpy2.0.0rc1
is released and you can target that.Rationale: the NumPy C ABI will change in 2.0, so any compiled extensionmodules that rely on NumPy will break; they need to be recompiled.If you rely on a large API surface from NumPy’s Python API, also consideradding the same
numpy<2.0
requirement to your metadata until you aresure your code is updated for changes in 2.0 (i.e., when you’ve testedthings work against2.0.0rc1
).Rationale: we will do a significant API cleanup, with many aliases anddeprecated/non-recommended objects being removed (see, e.g.,NumPy 2.0 migration guideandNEP 52 — Python API cleanup for NumPy 2.0),so unless you only usemodern/recommended functions and objects, your code is likely to require atleast some adjustments.Plan to do a release of your own packages which depend on
numpy
shortlyafter the first NumPy 2.0 release candidate is released (probably around 1Feb 2024).Rationale: at that point, you can release packages that will work with both2.0 and 1.X, and hence your own end users will not be seeing much/anydisruption (you wantpipinstallmypackage
to continue working on theday NumPy 2.0 is released).Once
2.0.0rc1
is available, you can adjust your metadata inpyproject.toml
in the way outlined below.
There are two cases: you need to keep compatibility with numpy 1.xx while alsosupporting 2.0, or you are able to drop numpy 1.xx support for new releases ofyour package and support >=2.0 only. The latter is simpler, but may be morerestrictive for your users. In that case, simply addnumpy>=2.0
(ornumpy>=2.0.0rc1
) to your build and runtime requirements and you’re good togo. We’ll focus on the “keep compatibility with 1.xx and 2.x” now, which is alittle more involved.
Example for a package using the NumPy C API (via C/Cython/etc.) which wants to supportNumPy 1.23.5 and up:
[build-system]build-backend=...requires=[# Note for packagers: this constraint is specific to wheels# for PyPI; it is also supported to build against 1.xx still.# If you do so, please ensure to include a `numpy<2.0`# runtime requirement for those binary packages."numpy>=2.0.0rc1",...][project]dependencies=["numpy>=1.23.5",]
We recommend that you have at least one CI job which builds/installs via a wheel,and then runs tests against the oldest numpy version that the package supports.For example:
-name:Build wheel via wheel, then install itrun:|python -m build # This will pull in numpy 2.0 in an isolated envpython -m pip install dist/*.whl-name:Test against oldest supported numpy versionrun:|python -m pip install numpy==1.23.5# now run test suite
The above only works once NumPy 2.0 is available on PyPI. If you want to testagainst a NumPy 2.0-dev wheel, you have to use a numpy nightly build (seethis section higher up) or build numpy from source.