Repository files navigation

Pytest plugin to randomly order tests and controlrandom.seed.

Testing a Django project?Check out my bookSpeed Up Your Django Tests which covers loads of ways to write faster, more accurate tests.

All of these features are on by default but can be disabled with flags.

• Randomly shuffles the order of test items. This is done first at the level ofmodules, then at the level of test classes (if you have them), then at theorder of functions. This also works with things like doctests.

• Generates a base random seed or accepts one for reproduction with--randomly-seed.The base random seed is printed at the start of the test run, and can be passed in to repeat a failure caused by test ordering or random data.

• At the start of the test run, and before each test setup, run, and teardown, it resets Python’s global random seed to a fixed value, usingrandom.seed().The fixed value is derived from the base random seed, the pytest test ID, and an offset for setup or teardown.This ensures each test gets a different but repeatable random seed.

• pytest-randomly also resets several libraries’ random states at the start ofevery test, if they are installed:

  • factory boy

  • Faker

    Thefaker pytest fixture is also affected, as pytest-randomly definesthefaker_seed fixture.

  • Model Bakery

  • NumPy

    Only itslegacy random state is affected.

• If additional random generators are used, they can be registered under thepytest_randomly.random_seederentry point andwill have their seed reset at the start of every test. Register a functionthat takes the current seed value.

• Works withpytest-xdist.

Randomness in testing can be quite powerful to discover hidden flaws in thetests themselves, as well as giving a little more coverage to your system.

By randomly ordering the tests, the risk of surprising inter-test dependenciesis reduced - a technique used in many places, for example Google's C++ testrunnergoogletest.Research suggests that "dependent tests do exist in practice" and a randomorder of test executions can effectively detect such dependencies[1].Alternatively, a reverse order of test executions, as provided bypytest-reverse, may find less dependenttests but can achieve a better benefit/cost ratio.

By resetting the random seed to a repeatable number for each test, tests cancreate data based on random numbers and yet remain repeatable, for examplefactory boy's fuzzy values. This is good for ensuring that tests specify thedata they need and that the tested system is not affected by any data that isfilled in randomly due to not being specified.

I have written ablog post covering the history ofpytest-randomly,including how it started life as the nose pluginnose-randomly.

Additionally, I appeared on the Test and Code podcast totalk aboutpytest-randomly.

Install with:

python -m pip install pytest-randomly

Python 3.10 to 3.14 supported.

Pytest will automatically find the plugin and use it when you runpytest.The output will start with an extra line that tells you the random seed that isbeing used:

$ pytest...platform darwin -- Python ...Using --randomly-seed=1553614239...

If the tests fail due to ordering or randomly created data, you can restartthem with that seed using the flag as suggested:

pytest --randomly-seed=1234

Or more conveniently, use the special valuelast:

pytest --randomly-seed=last

(This only works if pytest’s cacheprovider plugin has not been disabled.)

Since the ordering is by module, then by class, you can debug inter-testpollution failures by narrowing down which tests are being run to find the badinteraction by rerunning just the module/class:

pytest --randomly-seed=1234 tests/module_that_failed/

You can disable behaviours you don't like with the following flags:

• --randomly-dont-reset-seed - turn off the reset ofrandom.seed() atthe start of every test
• --randomly-dont-reorganize - turn off the shuffling of the order of tests

The plugin appears to Pytest with the name 'randomly'. To disable italtogether, you can use the-p argument, for example:

pytest -p no:randomly

To fix the order of some tests, use thepytest-order plugin.See itsdocumentation section on usage with pytest-randomly.

If you're using a different randomness generator in your third party package,you can register an entrypoint to be called every timepytest-randomlyreseeds. Implement the entrypointpytest_randomly.random_seeder, referringto a function/callable that takes one argument, the new seed (int).

For example in yoursetup.cfg:

[options.entry_points]pytest_randomly.random_seeder =mypackage = mypackage.reseed

Then implementreseed(new_seed).