11.. _testing :
22
3- ============================
4- Developer's tips for testing
5- ============================
3+ =======
4+ Testing
5+ =======
66
7- Matplotlib's testing infrastructure depends onpytest _. The tests are in
8- :file: `lib/matplotlib/tests `, and customizations to the pytest testing
9- infrastructure are in:mod: `matplotlib.testing `.
7+ Matplotlib uses thepytest _ framework.
8+
9+ The tests are in:file: `lib/matplotlib/tests `, and customizations to the pytest
10+ testing infrastructure are in:mod: `matplotlib.testing `.
1011
1112.. _pytest :http://doc.pytest.org/en/latest/
12- .. _pytest-flake8 :https://pypi.org/project/pytest-flake8/
1313.. _pytest-xdist :https://pypi.org/project/pytest-xdist/
14- .. _pytest-timeout :https://pypi.org/project/pytest-timeout/
15- .. _flake8 :https://pypi.org/project/flake8/
1614
1715
1816.. _testing_requirements :
@@ -24,14 +22,23 @@ To run the tests you will need to
2422:ref: `set up Matplotlib for development <installing_for_devs >`. Note in
2523particular the:ref: `additional dependencies <test-dependencies >` for testing.
2624
25+ ..note ::
26+
27+ We will assume that you want to run the tests in a development setup.
28+
29+ While you can run the tests against a regular installed version of
30+ Matplotlib, this is a far less common use case. You still need the
31+ :ref: `additional dependencies <test-dependencies >` for testing.
32+ You have to additionally get the reference images from the repository,
33+ because they are not distributed with pre-built Matplotlib packages.
34+
2735Running the tests
2836-----------------
2937
30- Running thetests is simple. Make sure you have pytest installed and run::
38+ In theroot directory of your development repository run::
3139
3240 pytest
3341
34- in the root directory of the repository.
3542
3643pytest can be configured via a lot of `command-line parameters `_. Some
3744particularly useful ones are:
@@ -40,11 +47,7 @@ particularly useful ones are:
4047``-v `` or ``--verbose `` Be more verbose
4148``-n NUM `` Run tests in parallel over NUM
4249 processes (requirespytest-xdist _)
43- ``--timeout=SECONDS `` Set timeout for results from each test
44- process (requirespytest-timeout _)
4550``--capture=no `` or ``-s `` Do not capture stdout
46- ``--flake8 `` Check coding standards usingflake8 _
47- (requirespytest-flake8 _)
4851============================= ===========
4952
5053To run a single test from the command line, you can provide a file path,
@@ -53,15 +56,6 @@ not need to be installed, but Matplotlib should be)::
5356
5457 pytest lib/matplotlib/tests/test_simplification.py::test_clipping
5558
56- or, if tests are installed, a dot-separated path to the module, optionally
57- followed by the function separated by two colons, such as::
58-
59- pytest --pyargs matplotlib.tests.test_simplification::test_clipping
60-
61- If you want to run the full test suite, but want to save wall time try
62- running the tests in parallel::
63-
64- pytest --verbose -n 5
6559
6660An alternative implementation that does not look at command line arguments
6761and works from within Python is to run the tests from the Matplotlib library
@@ -106,11 +100,6 @@ For numpy use::
106100 import numpy as np
107101 np.random.seed(19680801)
108102
109- and Python's random number generator::
110-
111- import random
112- random.seed(19680801)
113-
114103The seed is John Hunter's birthday.
115104
116105Writing an image comparison test
@@ -151,26 +140,6 @@ See the documentation of `~matplotlib.testing.decorators.image_comparison` and
151140`~matplotlib.testing.decorators.check_figures_equal ` for additional information
152141about their use.
153142
154- Known failing tests
155- -------------------
156-
157- If you're writing a test, you may mark it as a known failing test with the
158- :func: `pytest.mark.xfail ` decorator. This allows the test to be added to the
159- test suite and run on the buildbots without causing undue alarm. For example,
160- although the following test will fail, it is an expected failure::
161-
162- import pytest
163-
164- @pytest.mark.xfail
165- def test_simple_fail():
166- '''very simple example test that should fail'''
167- assert 1 + 1 == 3
168-
169- Note that the first argument to the:func: `~pytest.mark.xfail ` decorator is a
170- fail condition, which can be a value such as True, False, or may be a
171- dynamically evaluated expression. If a condition is supplied, then a reason
172- must also be supplied with the ``reason='message' `` keyword argument.
173-
174143Creating a new module in matplotlib.tests
175144-----------------------------------------
176145