- Getting started
- Development workflow
- Issues and triaging
- Documentation
- Testing and buildbots
- Development tools
- Core developers
- CPython’s internals
- Status of Python versions
- Python Contributor’s Guide (draft)
- [Plan for the Contributor’s Guide]
- Introduction
- The CPython project
- Issues and triaging
- Documentation contributions
- Code contributions
- Core team
- Accessibility, design, and user success
- Security and infrastructure contributions
- Workflows
Increase test coverage¶
Python development follows a practice that all semantic changes and additionsto the language andstdlib are accompanied byappropriate unit tests. Unfortunately Python was in existence for a long timebefore the practice came into effect. This has left chunks of the stdlibuntested which is not a desirable situation to be in.
A good, easy way to become acquainted with Python’s code and to help out is tohelp increase the test coverage for Python’s stdlib. Ideally we would like tohave 100% coverage, but any increase is a good one. Do realize, though, thatgetting 100% coverage is not always possible. There could be platform-specificcode that simply will not execute for you, errors in the output, etc. You canuse your judgement as to what should and should not be covered, but beingconservative and assuming something should be covered is generally a good ruleto follow.
Choosing what module you want to increase test coverage for can be done in acouple of ways.You can simply run the entire test suite yourself with coverage turnedon and see what modules need help. This has the drawback of running the entiretest suite under coverage measuring which takes some time to complete, but youwill have an accurate, up-to-date notion of what modules need the most work.
Another is to follow the examples below and simply see whatcoverage your favorite module has. This is “stabbing in the dark”, though, andso it might take some time to find a module that needs coverage help.
Do make sure, though, that for any module you do decide to work on that you runcoverage for just that module. This will make sure you know how good theexplicit coverage of the module is from its own set of tests instead of fromimplicit testing by other code that happens to use the module.
Common gotchas¶
Please realize that coverage reports on modules already imported before coveragedata starts to be recorded will be wrong. Typically you can tell a module fallsinto this category by the coverage report saying that global statements thatwould obviously be executed upon import have gone unexecuted while localstatements have been covered. In these instances you can ignore the globalstatement coverage and simply focus on the local statement coverage.
When writing new tests to increase coverage, do take note of the style of testsalready provided for a module (for example, whitebox, blackbox, etc.). Assome modules are primarily maintained by a single core developer they may havea specific preference as to what kind of test is used (for example, whitebox) andprefer that other types of tests not be used (for example, blackbox). When in doubt,stick with whitebox testing in order to properly exercise the code.
Measuring coverage¶
It should be noted that a quirk of running coverage over Python’s own stdlib isthat certain modules are imported as part of interpreter startup. Those modulesrequired by Python itself will not be viewed as executed by the coverage toolsand thus look like they have very poor coverage (for example, thestatmodule). In these instances the module will appear to not have any coverage ofglobal statements but will have proper coverage of local statements (for example,function definitions will not be traced, but the function bodies will).Calculating the coverage of modules in this situation will simply requiremanually looking at what local statements were not executed.
Using coverage.py¶
One of the most popular third-party coverage tools iscoverage.py whichprovides very nice HTML output along with advanced features such asbranch coverage. If you prefer to stay with tools onlyprovided by the stdlib then you canuse test.regrtest.
Install coverage¶
By default, pip will not install into the in-development version of Python youjust built, and this built version of Python will not see packages installedinto your default version of Python. One option is to use a virtual environmentto install coverage.
Run:
./python-mvenv../cpython-venvsource../cpython-venv/bin/activatepipinstallcoverageOnmost macOS systems run:
./python.exe-mvenv../cpython-venvsource../cpython-venv/bin/activatepipinstallcoverageRun:
python.bat -m venv ..\\cpython-venv..\\cpython-venv\\Scripts\\activate.batpip install coverage
You can now use python without the ./ for the rest of these instructions, aslong as your venv is activated. For more info on venv seeVirtual Environment documentation.
If this does not work for you for some reason, you should try using thein-development version of coverage.py to see if it has been updated as needed.To do this you should clone/check out the development version of coverage.py:
gitclonehttps://github.com/nedbat/coveragepy
You will need to use the full path to the installation.
Another option is to use an installed copy of coverage.py, if you already haveit. For this, you will again need to use the full path to that installation.
Basic usage¶
The following command will tell you if your copy of coverage works (substituteCOVERAGEDIR with the directory where your clone exists, for example,../coveragepy):
./pythonCOVERAGEDIR
Coverage.py will print out a little bit of helper text verifying thateverything is working. If you are using an installed copy, you can do thefollowing instead (note this must be installed using the built copy of Python,such as by venv):
./python-mcoverage
The rest of the examples on how to use coverage.py will assume you are using acloned copy, but you can substitute the above and all instructions should stillbe valid.
To run the test suite under coverage.py, do the following:
./pythonCOVERAGEDIRrun--pylibLib/test/regrtest.py
To run only a single test, specify the module/package being testedin the--source flag (so as to prune the coverage reporting to only themodule/package you are interested in) and then append the name of the test youwish to run to the command:
./pythonCOVERAGEDIRrun--pylib--source=abcLib/test/regrtest.pytest_abc
To see the results of the coverage run, you can view a text-based report with:
./pythonCOVERAGEDIRreport
You can use the--show-missing flag to get a list of lines that were notexecuted:
./pythonCOVERAGEDIRreport--show-missing
But one of the strengths of coverage.py is its HTML-based reports which letyou visually see what lines of code were not tested:
./python COVERAGEDIR html -i --include=`pwd`/Lib/* --omit="Lib/test/*,Lib/*/tests/*"
This will generate an HTML report in a directory namedhtmlcov whichignores any errors that may arise and ignores modules for which test coverage isunimportant (for example, tests, temp files, etc.). You can then open thehtmlcov/index.html file in a web browser to view the coverage results alongwith pages that visibly show what lines of code were or were not executed.
Branch coverage¶
For the truly daring, you can use another powerful feature of coverage.py:branch coverage. Testing every possible branch path through code, while a greatgoal to strive for, is a secondary goal to getting 100% linecoverage for the entire stdlib (for now).
If you decide you want to try to improve branch coverage, simply add the--branch flag to your coverage run:
./pythonCOVERAGEDIRrun--pylib--branch<argumentstoruntest(s)>
This will lead to the report stating not only what lines were not covered, butalso what branch paths were not executed.
Using test.regrtest¶
If you prefer to rely solely on the stdlib to generate coverage data, you cando so by passing the appropriate flags totest (along withany other flags you want to):
./python -m test --coverage -D `pwd`/coverage_data <test arguments>
Do note the argument to-D; if you do not specify an absolute path to whereyou want the coverage data to end up it will go somewhere you don’t expect.
Note
If you are running coverage over the entire test suite, make sure toadd-xtest_importlibtest_runpytest_trace to exclude those tests asthey trigger exceptions during coverage; seepython/cpython#54750 andpython/cpython#55200.
Once the tests are done you will find the directory you specified containsfiles for each executed module along with which lines were executed how manytimes.
Filing the issue¶
Once you have increased coverage,you need to create an issue on theissue tracker andsubmit apull request.
Measuring coverage of C code with gcov and lcov¶
It’s also possible to measure the function, line and branch coverage ofPython’s C code. Right now only GCC withgcov is supported. In order tocreate an instrumented build of Python with gcov, run:
makecoverage
.\make coverage
Then run some code and gather coverage data with thegcov command. Inorder to create a HTML report you can installlcov. The command:
makecoverage-lcov
.\make coverage-lcov
assembles coverage data, removes 3rd party and system libraries and finallycreates a report. You can skip both steps and just run:
makecoverage-report
.\make coverage-report
if you like to generate a coverage report for Python’s stdlib tests. It takesabout 20 to 30 minutes on a modern computer.
Note
Multiple test jobs may not work properly. C coverage reporting has onlybeen tested with a single test process.