Running¶

The shortest, simplest way of running the test suite is the following commandfrom the root directory of your checkout (after you havebuilt Python):

./python-mtest

./python.exe-mtest

.\python.bat -m test

This will run the majority of tests, but exclude a small portion of them; theseexcluded tests use special kinds of resources: for example, accessing theInternet, or trying to play a sound or to display a graphical interface onyour desktop. They are disabled by default so that running the test suiteis not too intrusive. To enable some of these additional tests (and forother flags which can help debug various issues such as reference leaks), readthe help text:

./python-mtest-h

./python.exe-mtest-h

.\python.bat -m test -h

If you want to run a single test file, simply specify the test file name(without the extension) as an argument. You also probably want to enableverbose mode (using-v), so that individual failures are detailed:

./python-mtest-vtest_abc

./python.exe-mtest-vtest_abc

.\python.bat -m test -v test_abc

To run a single test case, use theunittest module, providing the importpath to the test case:

./python-munittest-vtest.test_abc.TestABC_Py

./python.exe-munittest-vtest.test_abc.TestABC_Py

.\python.bat -m unittest -v test.test_abc.TestABC_Py

Some test modules also support direct invocation,which might be useful for IDEs and local debugging:

./pythonLib/test/test_typing.py

./python.exeLib/test/test_typing.py

.\python.bat Lib/test/test_typing.py

But, there are several important notes:

1. This way of running tests exists onlyfor local developer needs and is discouraged for anything else

2. Some modules do not support it at all. One example istest_importlib.In other words: if some module does not haveunittest.main(), thenmost likely it does not support direct invocation.

If you have a multi-core or multi-CPU machine, you can enable parallel testingusing several Python processes so as to speed up things:

./python-mtest-j0

./python.exe-mtest-j0

.\python.bat -m test -j0

Finally, if you want to run tests under a more strenuous set of settings, youcan runtest as:

./python-bb-E-Wd-mtest-r-w-uall

./python.exe-bb-E-Wd-mtest-r-w-uall

.\python.bat -bb -E -Wd -m test -r -w -uall

The various extra flags passed to Python cause it to be much stricter aboutvarious things (the-Wd flag should be-Werror at some point, but thetest suite has not reached a point where all warnings have been dealt with andso we cannot guarantee that a bug-free Python will properly complete a test runwith-Werror). The-r flag to the test runner causes it to run tests ina more random order which helps to check that the various tests do not interferewith each other. The-w flag causes failing tests to be run again to seeif the failures are transient or consistent.The-uall flag allows the use of all availableresources so as to not skip tests requiring, for example, Internet access.

To check for reference leaks (only needed if you modified C code),you can enable reference leak checking during testing using the-R flag.For example, using the command:

python-mtest<test_name>-R:

This default setting performs a few initial warm-up runs to stabilize the reference count,followed by additional runs to check for leaks.

If you want more control over the number of runs, you can specifywarmups andrepeats explicitly:

python-mtest<test_name>-R<warmups>:<repeats>

For instance,-R3:2 will first run the test 3 times to settle down thereference count, and then run it 2 more times to check for leaks.

You can also execute theTools/scripts/run_tests.py script as found in aCPython checkout. The script tries to balance speed with thoroughness. But ifyou want the most thorough tests you should use the strenuous approach shownabove.

Writing¶

Writing tests for Python is much like writing tests for your own code. Testsneed to be thorough, fast, isolated, consistently repeatable, and as simple aspossible. We try to have tests both for normal behaviour and for errorconditions. Tests live in theLib/test directory, where every file thatincludes tests has atest_ prefix.

One difference with ordinary testing is that you are encouraged to rely on thetest.support module. It contains various helpers that are tailored toPython’s test suite and help smooth out common problems such as platformdifferences, resource consumption and cleanup, or warnings management.That module is not suitable for use outside of the standard library.

When you are adding tests to an existing test file, it is also recommendedthat you study the other tests in that file; it will teach you which precautionsyou have to take to make your tests robust and portable.

For tests of the C API, see Tests sections inChanging Python’s C API.

Benchmarks¶

Benchmarking is useful to test that a change does not degrade performance.

The Python Benchmark Suitehas a collection of benchmarks for all Python implementations. Documentationabout running the benchmarks is in theREADME.txt of the repo.