mutmut - python mutation tester¶
Mutmut is a mutation testing system for Python, with a strong focus on easeof use. If you don’t know what mutation testing is try starting withthis article.
Some highlight features:
Found mutants can be applied on disk with a simple command making it veryeasy to work with the results
Remembers work that has been done, so you can work incrementally
Knows which tests to execute, speeding up mutation testing
Interactive terminal based UI
Parallel and fast execution
If you want to mutate code outside of functions, you can try using mutmut 2,which has a different execution model than mutmut 3+.
Requirements¶
Mutmut must be run on a system withfork support. This means that if you wantto run on windows, you must run inside WSL.
Install and run¶
You can get started with a simple:
pip install mutmutmutmut run
This will by run pytest on tests in the “tests” or “test” folder andit will try to figure out where the code to mutate is.
You can stop the mutation run at any time and mutmut will restart where youleft off. It will continue where it left off, and re-test functions that weremodified since last run.
To work with the results, usemutmut browse where you can see the mutants,retest them when you’ve updated your tests.
You can also write a mutant to disk from thebrowse interface, or viamutmut apply <mutant>. You shouldREALLY have the file you mutate undersource code control and committed before you apply a mutant!
Configuration¶
Insetup.cfg in the root of your project you can configure mutmut if you need to:
[mutmut]paths_to_mutate=src/tests_dir=tests/
If you usepyproject.toml, you must specify the paths as array in atool.mutmut section:
[tool.mutmut]paths_to_mutate=["src/"]tests_dir=["tests/"]
See below for more options for configuring mutmut.
Wildcards for testing mutants¶
Unix filename pattern matching style on mutants is supported. Example:
mutmut run "my_module*"mutmut run "my_module.my_function*"
In thebrowse TUI you can pressf to retest a function, andm to retestan entire module.
“also copy” files¶
To run the full test suite some files are often needed above the tests and thesource. You can configure to copy extra files that you need by addingdirectories and files toalso_copy in yoursetup.cfg:
also_copy=iommi/snapshots/conftest.py
Limit stack depth¶
In big code bases some functions are called incidentally by huge swaths of thecodebase, but you really don’t want tests that hit those executions to countfor mutation testing purposes. Incidentally tested functions lead to slowmutation testing as hundreds of tests can be checked for things that shouldhave clean and fast unit tests, and it leads to bad test suites as anyintroduced bug in those base functions will lead to many tests that fail whichare hard to understand how they relate to the function with the change.
You can configure mutmut to only count a test as being relevant for a functionif the stack depth from the test to the function is below some limit. In yoursetup.cfg add:
max_stack_depth=8
A lower value will increase mutation speed and lead to more localized tests,but will also lead to more surviving mutants that would otherwise have beencaught.
Exclude files from mutation¶
You can exclude files from mutation insetup.cfg:
do_not_mutate=*__tests.py
Whitelisting¶
You can mark lines like this:
some_code_here()# pragma: no mutate
to stop mutation on those lines. Some cases we’ve found where you need towhitelist lines are:
The version string on your library. You really shouldn’t have a test for this :P
Optimizing break instead of continue. The code runs fine when mutating breakto continue, but it’s slower.
Example mutations¶
Integer literals are changed by adding 1. So 0 becomes 1, 5 becomes 6, etc.
< is changed to<=
break is changed to continue and vice versa
In general the idea is that the mutations should be as subtle as possible.See__init__.py for the full list.
Workflow¶
This section describes how to work with mutmut to enhance your test suite.
Run mutmut withmutmut run. A full run is preferred but if you’re justgetting started you can exit in the middle and start working with what youhave found so far.
Show the mutants withmutmut browse
Find a mutant you want to work on and write a test to try to kill it.
Pressr to rerun the mutant and see if you successfully managed to kill it.
Mutmut keeps the data of what it has done and the mutants in themutants/directory.If you want to make sure you run a full mutmut run you can deletethis directory to start from scratch.
Contributing to Mutmut¶
If you wish to contribute to Mutmut, please see ourcontributing guide.