Recommended development setup#

Since NumPy contains parts written in C and Cython that need to becompiled before use, make sure you have the necessary compilers and Pythondevelopment headers installed - seeBuilding from source. BuildingNumPy as of version2.0 requires C11 and C++17 compliant compilers.

Having compiled code also means that importing NumPy from the developmentsources needs some additional steps, which are explained below. For the restof this chapter we assume that you have set up your git repo as described inWorking with scikit-image source code.

Using virtual environments#

A frequently asked question is “How do I set up a development version of NumPyin parallel to a released version that I use to do my job/research?”.

One simple way to achieve this is to install the released version insite-packages, by using pip or conda for example, and setup the development version in a virtual environment.

If you use conda, we recommend creating a separate virtual environment fornumpy development using theenvironment.yml file in the root of the repo(this will create the environment and install all development dependencies atonce):

$ conda env create -f environment.yml  # `mamba` works too for this command$ conda activate numpy-dev

If you installed Python some other way than conda, first installvirtualenv (optionally usevirtualenvwrapper), then create yourvirtualenv (namednumpy-dev here), activate it, and install all projectdependencies with:

$ virtualenv numpy-dev$ source numpy-dev/bin/activate # activate virtual environment$ python -m pip install -r requirements/all_requirements.txt

Now, whenever you want to switch to the virtual environment, you can use thecommandsourcenumpy-dev/bin/activate, anddeactivate to exit from thevirtual environment and back to your previous shell.

Building from source#

SeeBuilding from source.

Testing builds#

Before running the tests, first install the test dependencies:

$ python -m pip install -r requirements/test_requirements.txt$ python -m pip install asv # only for running benchmarks

To build the development version of NumPy and run tests, spawninteractive shells with the Python import paths properly set up etc., use thespin utility. To run tests, doone of:

$ spin test -v$ spin test numpy/random  # to run the tests in a specific module$ spin test -v -t numpy/_core/tests/test_nditer.py::test_iter_c_order

This builds NumPy first, so the first time it may take a few minutes.

You can also usespinbench for benchmarking. Seespin--help for morecommand line options.

Additional arguments may be forwarded topytest by passing the extraarguments after a bare--. For example, to run a test method with the--pdb flag forwarded to the target, run the following:

$ spin test -t numpy/tests/test_scripts.py::test_f2py -- --pdb

You can alsomatch test names using python operators by passing the-kargument to pytest:

$ spin test -v -t numpy/_core/tests/test_multiarray.py -- -k "MatMul and not vector"

To run “doctests” – to check that the code examples in the documentation is correct –use thecheck-docs spin command. It relies on thescipy-docs package, whichprovides several additional features on top of the standard librarydoctestpackage. Installscipy-doctest and run one of:

$ spin check-docs -v$ spin check-docs numpy/linalg$ spin check-docs -v -- -k 'det and not slogdet'

Other build options#

For more options including selecting compilers, setting custom compiler flagsand controlling parallelism, seeCompiler selection and customizing a build(from the SciPy documentation.)

Running tests#

Besides usingspin, there are various ways to run the tests. Insidethe interpreter, tests can be run like this:

>>>np.test()>>>np.test('full')# Also run tests marked as slow>>>np.test('full',verbose=2)# Additionally print test name/fileAn example of a successful test :``4686 passed, 362 skipped, 9 xfailed, 5 warnings in 213.99 seconds``

Or a similar way from the command line:

$ python -c "import numpy as np; np.test()"

Tests can also be run withpytestnumpy, however then the NumPy-specificplugin is not found which causes strange side effects.

Running individual test files can be useful; it’s much faster than running thewhole test suite or that of a whole module (example:np.random.test()).This can be done with:

$ python path_to_testfile/test_file.py

That also takes extra arguments, like--pdb which drops you into the Pythondebugger when a test fails or an exception is raised.

Running tests withtox is also supported. For example, to build NumPy andrun the test suite with Python 3.9, use:

$ tox -e py39

For more extensive information, seeTesting guidelines.

Note: do not run the tests from the root directory of your numpy git repo withoutspin,that will result in strange test errors.

Running linting#

Lint checks can be performed on newly added lines of Python code.

Install all dependent packages using pip:

$ python -m pip install -r requirements/linter_requirements.txt

To run lint checks before committing new code, run:

$ python tools/linter.py

To check all changes in newly added Python code of current branch with target branch, run:

$ python tools/linter.py

If there are no errors, the script exits with no message. In case of errors,check the error message for details:

$ python tools/linter.py./numpy/_core/tests/test_scalarmath.py:34:5: E303 too many blank lines (3)1       E303 too many blank lines (3)

It is advisable to run lint checks before pushing commits to a remote branchsince the linter runs as part of the CI pipeline.

For more details on Style Guidelines:

• Python Style Guide

• NEP 45 — C style guide

Rebuilding & cleaning the workspace#

Rebuilding NumPy after making changes to compiled code can be done with thesame build command as you used previously - only the changed files will bere-built. Doing a full build, which sometimes is necessary, requires cleaningthe workspace first. The standard way of doing this is (note: deletes anyuncommitted files!):

$ git clean -xdf

When you want to discard all changes and go back to the last commit in therepo, use one of:

$ git checkout .$ git reset --hard

Debugging#

Another frequently asked question is “How do I debug C code inside NumPy?”.First, ensure that you have gdb installed on your system with the Pythonextensions (often the default on Linux). You can see which version ofPython is running inside gdb to verify your setup:

(gdb)python>importsys>print(sys.version_info)>endsys.version_info(major=3,minor=7,micro=0,releaselevel='final',serial=0)

Most python builds do not include debug symbols and are built with compileroptimizations enabled. To get the best debugging experience using a debug buildof Python is encouraged, seeAdvanced debugging tools.

In terms of debugging, NumPy also needs to be built in a debug mode. You need to usedebug build type and disable optimizations to make sure-O0 flag is usedduring object building. Note that NumPy should NOT be installed in your environmentbefore you build with thespinbuild command.

To generate source-level debug information within the build process run:

$ spin build --clean -- -Dbuildtype=debug -Ddisable-optimization=true

Next you need to write a Python script that invokes the C code whose executionyou want to debug. For instancemytest.py:

importnumpyasnpx=np.arange(5)np.empty_like(x)

Note that your test file needs to be outside the NumPy clone you have. Now, you canrun:

$ spin gdb /path/to/mytest.py

In case you are using clang toolchain:

$ spin lldb /path/to/mytest.py

And then in the debugger:

(gdb)breakarray_empty_like(gdb)run

lldb counterpart:

(lldb)breakpointset--namearray_empty_like(lldb)run

The execution will now stop at the corresponding C function and you can stepthrough it as usual. A number of useful Python-specific commands are available.For example to see where in the Python code you are, usepy-list, to see thepython traceback, usepy-bt. For more details, seeDebuggingWithGdb. Here are some commonly used commands:

• list: List specified function or line.

• next: Step program, proceeding through subroutine calls.

• step: Continue program being debugged, after signal or breakpoint.

• print: Print value of expression EXP.

Rich support for Python debugging requires that thepython-gdb.py scriptdistributed with Python is installed in a path where gdb can find it. If youinstalled your Python build from your system package manager, you likely donot need to manually do anything. However, if you built Python from source,you will likely need to create a.gdbinit file in your home directorypointing gdb at the location of your Python installation. For example, aversion of python installed viapyenvneeds a.gdbinit file with the following contents:

add-auto-load-safe-path ~/.pyenv

Building NumPy with a Python built with debug support (on Linux distributionstypically packaged aspython-dbg) is highly recommended.

Understanding the code & getting started#

The best strategy to better understand the code base is to pick something youwant to change and start reading the code to figure out how it works. When indoubt, you can ask questions on the mailing list. It is perfectly okay if yourpull requests aren’t perfect, the community is always happy to help. As avolunteer project, things do sometimes get dropped and it’s totally fine toping us if something has sat without a response for about two to four weeks.

So go ahead and pick something that annoys or confuses you about NumPy,experiment with the code, hang around for discussions or go through thereference documents to try to fix it. Things will fall in place and soonyou’ll have a pretty good understanding of the project as a whole. Good Luck!