Running tests from inside Python#

You can test an installed NumPy bynumpy.test, for example,To run NumPy’s full test suite, use the following:

>>>importnumpy>>>numpy.test(label='slow')

The test method may take two or more arguments; the firstlabel is astring specifying what should be tested and the secondverbose is aninteger giving the level of output verbosity. See the docstringnumpy.testfor details. The default value forlabel is ‘fast’ - whichwill run the standard tests. The string ‘full’ will run the full batteryof tests, including those identified as being slow to run. Ifverboseis 1 or less, the tests will just show information messages about the teststhat are run; but if it is greater than 1, then the tests will also providewarnings on missing tests. So if you want to run every test and getmessages about which modules don’t have tests:

>>>numpy.test(label='full',verbose=2)# or numpy.test('full', 2)

Finally, if you are only interested in testing a subset of NumPy, forexample, the_core module, use the following:

>>>numpy._core.test()

Running tests from the command line#

If you want to build NumPy in order to work on NumPy itself, use thespinutility. To run NumPy’s full test suite:

$ spin test -m full

Testing a subset of NumPy:

$ spin test -t numpy/_core/tests

For detailed info on testing, seeTesting builds

Running doctests#

NumPy documentation contains code examples, “doctests”. To check that the examplesare correct, install thescipy-doctest package:

$ pip install scipy-doctest

and run one of:

$ spin check-docs -v$ spin check-docs numpy/linalg$ spin check-docs -- -k 'det and not slogdet'

Note that the doctests are not run when you usespintest.

Other methods of running tests#

Run tests using your favourite IDE such asvscode orpycharm

Using C code in tests#

NumPy exposes a richC-API . These are tested using c-extensionmodules written “as-if” they know nothing about the internals of NumPy, ratherusing the official C-API interfaces only. Examples of such modules are testsfor a user-definedrational dtype in_rational_tests or the ufuncmachinery tests in_umath_tests which are part of the binary distribution.Starting from version 1.21, you can also write snippets of C code in tests thatwill be compiled locally into c-extension modules and loaded into python.

numpy.testing.extbuild.build_and_import_extension(modname,functions,*,prologue='',build_dir=None,include_dirs=None,more_init='')#

Build and imports a c-extension modulemodname from a list of functionfragmentsfunctions.

Parameters:

functionslist of fragments

Each fragment is a sequence of func_name, calling convention, snippet.

prologuestring

Code to precede the rest, usually extra#include or#definemacros.

build_dirpathlib.Path

Where to build the module, usually a temporary directory

include_dirslist

Extra directories to find include files when compiling

more_initstring

Code to appear in the module PyMODINIT_FUNC

Returns:

out: module

The module will have been loaded and is ready for use

Examples

Try it in your browser!

>>>functions=[("test_bytes","METH_O","""    if ( !PyBytesCheck(args)) {        Py_RETURN_FALSE;    }    Py_RETURN_TRUE;""")]>>>mod = build_and_import_extension("testme", functions)>>>assert not mod.test_bytes('abc')>>>assert mod.test_bytes(b'abc')

Go BackOpen In Tab

Labeling tests#

Unlabeled tests like the ones above are run in the defaultnumpy.test() run. If you want to label your test as slow - andtherefore reserved for a fullnumpy.test(label='full') run, youcan label it withpytest.mark.slow:

importpytest@pytest.mark.slowdeftest_big(self):print('Big, slow test')

Similarly for methods:

classtest_zzz:@pytest.mark.slowdeftest_simple(self):assert_(zzz()=='Hello from zzz')

Easier setup and teardown functions / methods#

Testing looks for module-level or class method-level setup and teardownfunctions by name; thus:

defsetup_module():"""Module-level setup"""print('doing setup')defteardown_module():"""Module-level teardown"""print('doing teardown')classTestMe:defsetup_method(self):"""Class-level setup"""print('doing setup')defteardown_method():"""Class-level teardown"""print('doing teardown')

Setup and teardown functions to functions and methods are known as “fixtures”,and they should be used sparingly.pytest supports more general fixture at various scopes which may be usedautomatically via special arguments. For example, the special argument nametmpdir is used in test to create a temporary directory.

Parametric tests#

One very nice feature ofpytest is the ease of testing across a rangeof parameter values using thepytest.mark.parametrize decorator. For example,suppose you wish to testlinalg.solve for all combinations of threearray sizes and two data types:

@pytest.mark.parametrize('dimensionality',[3,10,25])@pytest.mark.parametrize('dtype',[np.float32,np.float64])deftest_solve(dimensionality,dtype):np.random.seed(842523)A=np.random.random(size=(dimensionality,dimensionality)).astype(dtype)b=np.random.random(size=dimensionality).astype(dtype)x=np.linalg.solve(A,b)eps=np.finfo(dtype).epsassert_allclose(A@x,b,rtol=eps*1e2,atol=0)assertx.dtype==np.dtype(dtype)

Doctests#

Doctests are a convenient way of documenting the behavior of a functionand allowing that behavior to be tested at the same time. The outputof an interactive Python session can be included in the docstring of afunction, and the test framework can run the example and compare theactual output to the expected output.

The doctests can be run by adding thedoctests argument to thetest() call; for example, to run all tests (including doctests)for numpy.lib:

>>>importnumpyasnp>>>np.lib.test(doctests=True)

The doctests are run as if they are in a fresh Python instance whichhas executedimportnumpyasnp. Tests that are part of a NumPysubpackage will have that subpackage already imported. E.g. for a testinnumpy/linalg/tests/, the namespace will be created such thatfromnumpyimportlinalg has already executed.

tests/#

Rather than keeping the code and the tests in the same directory, weput all the tests for a given subpackage in atests/subdirectory. For our example, if it doesn’t already exist you willneed to create atests/ directory innumpy/xxx/. So the pathfortest_yyy.py isnumpy/xxx/tests/test_yyy.py.

Once thenumpy/xxx/tests/test_yyy.py is written, its possible torun the tests by going to thetests/ directory and typing:

pythontest_yyy.py

Or if you addnumpy/xxx/tests/ to the Python path, you could runthe tests interactively in the interpreter like this:

>>>importtest_yyy>>>test_yyy.test()

__init__.py andsetup.py#

Usually, however, adding thetests/ directory to the python pathisn’t desirable. Instead it would better to invoke the test straightfrom the modulexxx. To this end, simply place the following linesat the end of your package’s__init__.py file:

...deftest(level=1,verbosity=1):fromnumpy.testingimportTesterreturnTester().test(level,verbosity)

You will also need to add the tests directory in the configurationsection of your setup.py:

...defconfiguration(parent_package='',top_path=None):...config.add_subpackage('tests')returnconfig...

Now you can do the following to test your module:

>>>importnumpy>>>numpy.xxx.test()

Also, when invoking the entire NumPy test suite, your tests will befound and run:

>>>importnumpy>>>numpy.test()# your tests are included and run automatically!

Known failures & skipping tests#

Sometimes you might want to skip a test or mark it as a known failure,such as when the test suite is being written before the code it’smeant to test, or if a test only fails on a particular architecture.

To skip a test, simply useskipif:

importpytest@pytest.mark.skipif(SkipMyTest,reason="Skipping this test because...")deftest_something(foo):...

The test is marked as skipped ifSkipMyTest evaluates to nonzero,and the message in verbose test output is the second argument given toskipif. Similarly, a test can be marked as a known failure byusingxfail:

importpytest@pytest.mark.xfail(MyTestFails,reason="This test is known to fail because...")deftest_something_else(foo):...

Of course, a test can be unconditionally skipped or marked as a knownfailure by usingskip orxfail without argument, respectively.

A total of the number of skipped and known failing tests is displayedat the end of the test run. Skipped tests are marked as'S' inthe test results (or'SKIPPED' forverbose>1), and knownfailing tests are marked as'x' (or'XFAIL' ifverbose>1).

Tests on random data#

Tests on random data are good, but since test failures are meant to exposenew bugs or regressions, a test that passes most of the time but failsoccasionally with no code changes is not helpful. Make the random datadeterministic by setting the random number seed before generating it. Useeither Python’srandom.seed(some_number) or NumPy’snumpy.random.seed(some_number), depending on the source of random numbers.

Alternatively, you can useHypothesis to generate arbitrary data.Hypothesis manages both Python’s and Numpy’s random seeds for you, andprovides a very concise and powerful way to describe data (includinghypothesis.extra.numpy, e.g. for a set of mutually-broadcastable shapes).

The advantages over random generation include tools to replay and sharefailures without requiring a fixed seed, reportingminimal examples foreach failure, and better-than-naive-random techniques for triggering bugs.

Documentation fornumpy.test#

numpy.test(label='fast',verbose=1,extra_argv=None,doctests=False,coverage=False,durations=-1,tests=None)#

Pytest test runner.

A test function is typically added to a package’s __init__.py like so:

fromnumpy._pytesttesterimportPytestTestertest=PytestTester(__name__).testdelPytestTester

Calling this test function finds and runs all tests associated with themodule and all its sub-modules.

Parameters:

module_namemodule name

The name of the module to test.

Notes

Unlike the previousnose-based implementation, this class is notpublicly exposed as it performs somenumpy-specific warningsuppression.

Attributes:

module_namestr

Full path to the package to test.