Repository files navigation

The plugin adds functionality to py.test to recognise and collect Jupyternotebooks. The intended purpose of the tests is to determine whether executionof the stored inputs match the stored outputs of the.ipynb file. Whilst alsoensuring that the notebooks are running without errors.

The tests were designed to ensure that Jupyter notebooks (especially those forreference and documentation), are executing consistently.

Each cell is taken as a test, a cell that doesn't reproduce the expectedoutput will fail.

Seedocs/source/index.ipynb for the full documentation.

Available on PyPi:

pip install nbval

or install the latest version from cloning the repository and running:

pip install .

from the main directory. To uninstall:

pip uninstall nbval

The extension looks through every cell that contains code in an IPython notebookand then thepy.test system compares the outputs stored in the notebookwith the outputs of the cells when they are executed. Thus, the notebook itself isused as a testing function.The output lines when executing the notebook can be sanitized passing anextra option and file, when calling thepy.test command. This fileis a usual configuration file for theConfigParser library.

Regarding the execution, roughly, the script initiates anIPython Kernel with ashell andaniopub sockets. Theshell is needed to execute the cells inthe notebook (it sends requests to the Kernel) and theiopub providesan interface to get the messages from the outputs. The contentsof the messages obtained from the Kernel are organised in dictionarieswith different information, such as time stamps of executions,cell data types, cell types, the status of the Kernel, username, etc.

In general, the functionality of the IPython notebook system isquite complex, but a detailed explanation of the messagesand how the system works, can be found here

https://jupyter-client.readthedocs.io/en/latest/messaging.html#messaging

To execute this plugin, you need to executepy.test with thenbval flagto differentiate the testing from the usual python files:

py.test --nbval

You can also specify--nbval-lax, which runs notebooks and checks forerrors, but only compares the outputs of cells with a#NBVAL_CHECK_OUTPUTmarker comment.

py.test --nbval-lax

The commands above will execute all the.ipynb files and 'pytest' tests in the current folder.Specify-p no:python if you would like to execute notebooks only. Alternatively, you can execute a specific notebook:

py.test --nbval my_notebook.ipynb

By default, each.ipynb file will be executed using the kernelspecified in its metadata. You can override this behavior by passingeither--nbval-kernel-name mykernel to run all the notebooks usingmykernel, or--current-env to use a kernel in the same environmentin which pytest itself was launched.

If the output lines are going to be sanitized, an extra flag,--nbval-sanitize-withtogether with the path to a confguration file with regex expressions, must be passed,i.e.

py.test --nbval my_notebook.ipynb --nbval-sanitize-with path/to/my_sanitize_file

wheremy_sanitize_file has the following structure.

[Section1]regex: [a-z]*replace: abcdregex: [1-9]*replace: 0000[Section2]regex: fooreplace: bar

Theregex option contains the expression that is going to be matched in the outputs, andreplace is the string that will replace theregex match. Currently, the sectionnames do not have any meaning or influence in the testing system, it will takeall the sections and replace the corresponding options.

To use notebooks to generate coverage for imported code, use the pytest-cov plugin.nbval should automatically detect the relevant options and configure itself with it.

nbval is compatible with the pytest-xdist plugin for parallel running of tests. It doeshowever require the use of the--dist loadscope flag to ensure that all cells of onenotebook are run on the same kernel.

The narrative documentation for nbval can be found athttps://nbval.readthedocs.io.

Thepy.test system help can be obtained withpy.test -h, which willshow all the flags that can be passed to the command, such as theverbose-v option. Nbval's options can be found under theJupyter Notebook validation section.

This plugin was inspired by Andrea Zonca's py.test plugin for collecting unittests in the IPython notebooks (https://github.com/zonca/pytest-ipynb).

The original prototype was based on the template inhttps://gist.github.com/timo/2621679 and the code of a testing systemfor notebookshttps://gist.github.com/minrk/2620735 which weintegrated and mixed with thepy.test system.

We acknowledge financial support from

• OpenDreamKit Horizon 2020 European Research Infrastructures project (#676541),http://opendreamkit.org

• EPSRC's Centre for Doctoral Training in Next GenerationComputational Modelling,http://ngcm.soton.ac.uk (#EP/L015382/1) andEPSRC's Doctoral Training Centre in Complex System Simulation((EP/G03690X/1),

• The Gordon and Betty Moore Foundation through Grant GBMF #4856, by theAlfred P. Sloan Foundation and by the Helmsley Trust.

2014 - 2017 David Cortes-Ortuno, Oliver Laslett, T. Kluyver, VidarFauske, Maximilian Albert, MinRK, Ondrej Hovorka, Hans Fangohr