Running tests from inside Python#

You can test an installed NumPy bynumpy.test, for example,To run NumPy’s full test suite, use the following:

>>>importnumpy>>>numpy.test(label='slow')

The test method may take two or more arguments; the firstlabel is astring specifying what should be tested and the secondverbose is aninteger giving the level of output verbosity. See the docstringnumpy.testfor details. The default value forlabel is ‘fast’ - whichwill run the standard tests. The string ‘full’ will run the full batteryof tests, including those identified as being slow to run. Ifverboseis 1 or less, the tests will just show information messages about the teststhat are run; but if it is greater than 1, then the tests will also providewarnings on missing tests. So if you want to run every test and getmessages about which modules don’t have tests:

>>>numpy.test(label='full',verbose=2)# or numpy.test('full', 2)

Finally, if you are only interested in testing a subset of NumPy, forexample, the_core module, use the following:

>>>numpy._core.test()

Running tests from the command line#

If you want to build NumPy in order to work on NumPy itself, use thespin utility.

To run NumPy’s full test suite:

$ spin test -m full

Testing a subset of NumPy:

$ spin test -t numpy/_core/tests

For detailed info on testing, seeTesting builds

Running tests in multiple threads#

To help with stress testing NumPy for thread safety, the test suite can be run underpytest-run-parallel. To installpytest-run-parallel:

$ pip install pytest-run-parallel

To run the test suite in multiple threads:

$ spin test -p auto # have pytest-run-parallel detect the number of available cores$ spin test -p 4 # run each test under 4 threads$ spin test -p auto -- --skip-thread-unsafe=true # run ONLY tests that are thread-safe

When you write new tests, it is worth testing to make sure they do not failunderpytest-run-parallel, since the CI jobs make use of it. Some tips on how towrite thread-safe tests can be foundhere.

Running doctests#

NumPy documentation contains code examples, “doctests”. To check that the examplesare correct, install thescipy-doctest package:

$ pip install scipy-doctest

and run one of:

$ spin check-docs -v$ spin check-docs numpy/linalg$ spin check-docs -- -k 'det and not slogdet'

Note that the doctests are not run when you usespintest.

Other methods of running tests#

Run tests using your favourite IDE such asvscode orpycharm

Using C code in tests#

NumPy exposes a richC-API . These are tested using c-extensionmodules written “as-if” they know nothing about the internals of NumPy, ratherusing the official C-API interfaces only. Examples of such modules are testsfor a user-definedrational dtype in_rational_tests or the ufuncmachinery tests in_umath_tests which are part of the binary distribution.Starting from version 1.21, you can also write snippets of C code in tests thatwill be compiled locally into c-extension modules and loaded into python.

numpy.testing.extbuild.build_and_import_extension(modname,functions,*,prologue='',build_dir=None,include_dirs=None,more_init='')#

Build and imports a c-extension modulemodname from a list of functionfragmentsfunctions.

Parameters:

functionslist of fragments

Each fragment is a sequence of func_name, calling convention, snippet.

prologuestring

Code to precede the rest, usually extra#include or#definemacros.

build_dirpathlib.Path

Where to build the module, usually a temporary directory

include_dirslist

Extra directories to find include files when compiling

more_initstring

Code to appear in the module PyMODINIT_FUNC

Returns:

out: module

The module will have been loaded and is ready for use

Examples

Try it in your browser!

>>>functions=[("test_bytes","METH_O","""    if ( !PyBytesCheck(args)) {        Py_RETURN_FALSE;    }    Py_RETURN_TRUE;""")]>>>mod = build_and_import_extension("testme", functions)>>>assert not mod.test_bytes('abc')>>>assert mod.test_bytes(b'abc')

Go BackOpen In Tab

Labeling tests#

Unlabeled tests like the ones above are run in the defaultnumpy.test() run. If you want to label your test as slow - andtherefore reserved for a fullnumpy.test(label='full') run, youcan label it withpytest.mark.slow:

importpytest@pytest.mark.slowdeftest_big(self):print('Big, slow test')

Similarly for methods:

classtest_zzz:@pytest.mark.slowdeftest_simple(self):assert_(zzz()=='Hello from zzz')

Setup and teardown methods#

NumPy originally used xunit setup and teardown, a feature ofpytest. We now encouragethe usage of setup and teardown methods that are called explicitly by the tests thatneed them:

classTestMe:defsetup(self):print('doing setup')return1defteardown(self):print('doing teardown')deftest_xyz(self):x=self.setup()assertx==1self.teardown()

This approach is thread-safe, ensuring tests can run underpytest-run-parallel.Using pytest setup fixtures (such as xunit setup methods) is generally not thread-safeand will likely cause thread-safety test failures.

pytest supports more general fixture at various scopes which may be usedautomatically via special arguments. For example, the special argument nametmp_path is used in tests to create temporary directories. However,fixtures should be used sparingly.

Parametric tests#

One very nice feature ofpytest is the ease of testing across a rangeof parameter values using thepytest.mark.parametrize decorator. For example,suppose you wish to testlinalg.solve for all combinations of threearray sizes and two data types:

@pytest.mark.parametrize('dimensionality',[3,10,25])@pytest.mark.parametrize('dtype',[np.float32,np.float64])deftest_solve(dimensionality,dtype):np.random.seed(842523)A=np.random.random(size=(dimensionality,dimensionality)).astype(dtype)b=np.random.random(size=dimensionality).astype(dtype)x=np.linalg.solve(A,b)eps=np.finfo(dtype).epsassert_allclose(A@x,b,rtol=eps*1e2,atol=0)assertx.dtype==np.dtype(dtype)

Doctests#

Doctests are a convenient way of documenting the behavior of a functionand allowing that behavior to be tested at the same time. The outputof an interactive Python session can be included in the docstring of afunction, and the test framework can run the example and compare theactual output to the expected output.

The doctests can be run by adding thedoctests argument to thetest() call; for example, to run all tests (including doctests)for numpy.lib:

>>>importnumpyasnp>>>np.lib.test(doctests=True)

The doctests are run as if they are in a fresh Python instance whichhas executedimportnumpyasnp. Tests that are part of a NumPysubpackage will have that subpackage already imported. E.g. for a testinnumpy/linalg/tests/, the namespace will be created such thatfromnumpyimportlinalg has already executed.

tests/#

Rather than keeping the code and the tests in the same directory, weput all the tests for a given subpackage in atests/subdirectory. For our example, if it doesn’t already exist you willneed to create atests/ directory innumpy/xxx/. So the pathfortest_yyy.py isnumpy/xxx/tests/test_yyy.py.

Once thenumpy/xxx/tests/test_yyy.py is written, its possible torun the tests by going to thetests/ directory and typing:

pythontest_yyy.py

Or if you addnumpy/xxx/tests/ to the Python path, you could runthe tests interactively in the interpreter like this:

>>>importtest_yyy>>>test_yyy.test()

__init__.py andsetup.py#

Usually, however, adding thetests/ directory to the python pathisn’t desirable. Instead it would better to invoke the test straightfrom the modulexxx. To this end, simply place the following linesat the end of your package’s__init__.py file:

...deftest(level=1,verbosity=1):fromnumpy.testingimportTesterreturnTester().test(level,verbosity)

You will also need to add the tests directory in the configurationsection of your setup.py:

...defconfiguration(parent_package='',top_path=None):...config.add_subpackage('tests')returnconfig...

Now you can do the following to test your module:

>>>importnumpy>>>numpy.xxx.test()

Also, when invoking the entire NumPy test suite, your tests will befound and run:

>>>importnumpy>>>numpy.test()# your tests are included and run automatically!

Known failures & skipping tests#

Sometimes you might want to skip a test or mark it as a known failure,such as when the test suite is being written before the code it’smeant to test, or if a test only fails on a particular architecture.

To skip a test, simply useskipif:

importpytest@pytest.mark.skipif(SkipMyTest,reason="Skipping this test because...")deftest_something(foo):...

The test is marked as skipped ifSkipMyTest evaluates to nonzero,and the message in verbose test output is the second argument given toskipif. Similarly, a test can be marked as a known failure byusingxfail:

importpytest@pytest.mark.xfail(MyTestFails,reason="This test is known to fail because...")deftest_something_else(foo):...

Of course, a test can be unconditionally skipped or marked as a knownfailure by usingskip orxfail without argument, respectively.

A total of the number of skipped and known failing tests is displayedat the end of the test run. Skipped tests are marked as'S' inthe test results (or'SKIPPED' forverbose>1), and knownfailing tests are marked as'x' (or'XFAIL' ifverbose>1).

Tests on random data#

Tests on random data are good, but since test failures are meant to exposenew bugs or regressions, a test that passes most of the time but failsoccasionally with no code changes is not helpful. Make the random datadeterministic by setting the random number seed before generating it.Userng=numpy.random.RandomState(some_number) to set a seed on alocal instance ofnumpy.random.RandomState.

Alternatively, you can useHypothesis to generate arbitrary data.Hypothesis manages both Python’s and Numpy’s random seeds for you, andprovides a very concise and powerful way to describe data (includinghypothesis.extra.numpy, e.g. for a set of mutually-broadcastable shapes).

The advantages over random generation include tools to replay and sharefailures without requiring a fixed seed, reportingminimal examples foreach failure, and better-than-naive-random techniques for triggering bugs.

Writing thread-safe tests#

Writing thread-safe tests may require some trial-and-error. Generally you shouldfollow the guidelines stated so far, especially when it comes tosetup methods andseeding random data.Explicit setup and the usage of local RNG are thread-safe practices. Here are tipsfor some other common problems you may run into.

Usingpytest.mark.parametrize may occasionally cause thread-safety issues.To fix this, you can usecopy():

@pytest.mark.parametrize('dimensionality',[3,10,25])@pytest.mark.parametrize('dtype',[np.float32,np.float64])deftest_solve(dimensionality,dtype):dimen=dimensionality.copy()d=dtype.copy()# use these copied variables instead...

If you are testing something that is inherently thread-unsafe, you can label yourtest withpytest.mark.thread_unsafe so that it will run under a single threadand not cause test failures:

@pytest.mark.thread_unsafe(reason="reason this test is thread-unsafe")deftest_thread_unsafe():...

Some examples of what should be labeled as thread-unsafe:

• Usage ofsys.stdout andsys.stderr

• Mutation of global data, like docstrings, modules, garbage collectors, etc.

• Tests that require a lot of memory, since they could cause crashes.

Additionally, somepytest fixtures are thread-unsafe, such asmonkeypatch andcapsys. However,pytest-run-parallel will automatically mark these asthread-unsafe if you decide to use them. Some fixtures have been patched to bethread-safe, liketmp_path.

Documentation fornumpy.test#

numpy.test(label='fast',verbose=1,extra_argv=None,doctests=False,coverage=False,durations=-1,tests=None)#

Pytest test runner.

A test function is typically added to a package’s __init__.py like so:

fromnumpy._pytesttesterimportPytestTestertest=PytestTester(__name__).testdelPytestTester

Calling this test function finds and runs all tests associated with themodule and all its sub-modules.

Parameters:

module_namemodule name

The name of the module to test.

Attributes:

module_namestr

Full path to the package to test.

Notes

Unlike the previousnose-based implementation, this class is notpublicly exposed as it performs somenumpy-specific warningsuppression.