There are at least 15 known ways to run the Python test suite. IMO it's too much, it's hard to keep all these code paths consistent. IMOpython -m test should be enough to fit all use cases.

Maybe we need a--ci,--buildbot, or--strict option which would run tests is "strict" mode.

Portable way to run tests:

./python -m test./python -m test.regrtest./python -m test.libregrtest./python -m test.autotest./python Lib/test/regrtest.py./python Lib/test/autotest.py./python -c 'from test import autotest'

The main drawback is that running./python doesn't work when Python is built with./configure --enable-shared, and it doesn't work when Python is cross-compiled (WASM/WASI buildbots). Moreover, on macOS,./python.exe must be run instead. On Windows, it's justpython.

Unix, from Python source code:

make buildbottest TESTOPTS="..."make testmake testallmake testuniversalmake hostrunnertest./python Tools/scripts/run_tests.py

Windows, from Python source code:

PCbuild\rt.batTools\buildbot\test.bat

• Lib/test/autotest.py just runstest.libregrtest.main().

• Lib/test/regrtest.py runstest.libregrtest.main()and removeLib/test/ fromsys.pathand make__file__ an absolute path.

• PCbuild\rt.bat:

  • python: pass-u -Wd -E -bb options
  • regrtest: pass options passed tort.bat (except of options specific to rt.bat)
  • most options are to get the path to the python.exe program in PCbuild/

• Tools\buildbot\test.bat runsPCbuild\rt.bat:

  • rt.bat: pass-q -d options.
  • regrtest: pass-j1 -uall -rwW --slowest --timeout=1200 --fail-env-changed options.
  • ARM32 SSH pass-unetwork -udecimal -usubprocess -uurlfetch -utzdata -rwW --slowest --timeout=1200 --fail-env-changed options.

• make buildbottest runsTools/scripts/run_tests.py:

  • python: pass$(TESTPYTHONOPTS) options.
  • regrtest: pass-j 1 -u all -W --slowest --fail-env-changed --timeout=$(TESTTIMEOUT) $(TESTOPTS) options.

• Tools/scripts/run_tests.py:

  • python: pass-u -W default -bb -E options -- also cross-compilation options, not detailed here
  • regrtest: pass-r -w -j0 -u all,-largefile,-audio,-gui options, add-n on Windows.

GitHub Action workflow uses,.github/workflows/build.yml:

• Windows x86:.\PCbuild\rt.bat -p Win32 -d -q -uall -u-cpu -rwW --slowest --timeout=1200 -j0
• Windows x64:.\PCbuild\rt.bat -p x64 -d -q -uall -u-cpu -rwW --slowest --timeout=1200 -j0
• macOS:make buildbottest TESTOPTS="-j4 -uall,-cpu"
• Ubuntu:xvfb-run make buildbottest TESTOPTS="-j4 -uall,-cpu"
• Address Sanitizer:xvfb-run make buildbottest TESTOPTS="-j4 -uall,-cpu"

Buildbot use,master/custom/factories.py:

• UnixBuild and UnixCrossBuild:make buildbottest TESTOPTS="..." TESTPYTHONOPTS="..." TESTTIMEOUT="..." with code to select TESTOPTS, TESTPYTHONOPTS and TESTTIMEOUT.
• UnixInstalledBuild:/path/to/python -m test ... with code to select-m test options.
• BaseWindowsBuild:Tools\buildbot\test.bat ... with code to select.... options

Linked PRs

• gh-109566: Run buildbot tests with --fail-rerun #109567
• gh-109566, regrtest: Add --fast-ci and --slow-ci options #109570
• gh-109566: Fix typo in PCbuild/rt.bat #109701
• gh-109566: regrtest reexecutes the process #109909
• gh-109566: Fix regrtest code adding Python options #109926
• gh-109566: Fix regrtest Python options for WASM/WASI #109954
• gh-109566: regrtest doesn't enable --rerun if --python is used #109969
• gh-109566: regrtest _add_python_opts() handles KeyboardInterrupt #110062
• gh-109566: PCbuild/rt.bat now uses --fast-ci #110120
• gh-109566: regrtest --fast-ci no longer enables --nowindow #110121
• gh-109566: Remove make testall #110122