26.8.test — Regression tests package for Python

Note

Thetest package is meant for internal use by Python only. It isdocumented for the benefit of the core developers of Python. Any use ofthis package outside of Python’s standard library is discouraged as codementioned here can change or be removed without notice between releases ofPython.

Thetest package contains all regression tests for Python as well as themodulestest.support andtest.regrtest.test.support is used to enhance your tests whiletest.regrtest drives the testing suite.

Each module in thetest package whose name starts withtest_ is atesting suite for a specific module or feature. All new tests should be writtenusing theunittest ordoctest module. Some older tests arewritten using a “traditional” testing style that compares output printed tosys.stdout; this style of test is considered deprecated.

See also

Moduleunittest

Writing PyUnit regression tests.

Moduledoctest

Tests embedded in documentation strings.

26.8.1.Writing Unit Tests for thetest package

It is preferred that tests that use theunittest module follow a fewguidelines. One is to name the test module by starting it withtest_ and endit with the name of the module being tested. The test methods in the test moduleshould start withtest_ and end with a description of what the method istesting. This is needed so that the methods are recognized by the test driver astest methods. Also, no documentation string for the method should be included. Acomment (such as#TestsfunctionreturnsonlyTrueorFalse) should be usedto provide documentation for test methods. This is done because documentationstrings get printed out if they exist and thus what test is being run is notstated.

A basic boilerplate is often used:

importunittestfromtestimportsupportclassMyTestCase1(unittest.TestCase):# Only use setUp() and tearDown() if necessarydefsetUp(self):...codetoexecuteinpreparationfortests...deftearDown(self):...codetoexecutetocleanupaftertests...deftest_feature_one(self):# Test feature one....testingcode...deftest_feature_two(self):# Test feature two....testingcode......moretestmethods...classMyTestCase2(unittest.TestCase):...samestructureasMyTestCase1......moretestclasses...if__name__=='__main__':unittest.main()

This code pattern allows the testing suite to be run bytest.regrtest,on its own as a script that supports theunittest CLI, or via thepython-munittest CLI.

The goal for regression testing is to try to break code. This leads to a fewguidelines to be followed:

  • The testing suite should exercise all classes, functions, and constants. Thisincludes not just the external API that is to be presented to the outsideworld but also “private” code.

  • Whitebox testing (examining the code being tested when the tests are beingwritten) is preferred. Blackbox testing (testing only the published userinterface) is not complete enough to make sure all boundary and edge casesare tested.

  • Make sure all possible values are tested including invalid ones. This makessure that not only all valid values are acceptable but also that impropervalues are handled correctly.

  • Exhaust as many code paths as possible. Test where branching occurs and thustailor input to make sure as many different paths through the code are taken.

  • Add an explicit test for any bugs discovered for the tested code. This willmake sure that the error does not crop up again if the code is changed in thefuture.

  • Make sure to clean up after your tests (such as close and remove all temporaryfiles).

  • If a test is dependent on a specific condition of the operating system thenverify the condition already exists before attempting the test.

  • Import as few modules as possible and do it as soon as possible. Thisminimizes external dependencies of tests and also minimizes possible anomalousbehavior from side-effects of importing a module.

  • Try to maximize code reuse. On occasion, tests will vary by something as smallas what type of input is used. Minimize code duplication by subclassing abasic test class with a class that specifies the input:

    classTestFuncAcceptsSequencesMixin:func=mySuperWhammyFunctiondeftest_func(self):self.func(self.arg)classAcceptLists(TestFuncAcceptsSequencesMixin,unittest.TestCase):arg=[1,2,3]classAcceptStrings(TestFuncAcceptsSequencesMixin,unittest.TestCase):arg='abc'classAcceptTuples(TestFuncAcceptsSequencesMixin,unittest.TestCase):arg=(1,2,3)

    When using this pattern, remember that all classes that inherit fromunittest.TestCase are run as tests. TheMixin class in the example abovedoes not have any data and so can’t be run by itself, thus it does notinherit fromunittest.TestCase.

See also

Test Driven Development

A book by Kent Beck on writing tests before code.

26.8.2.Running tests using the command-line interface

Thetest package can be run as a script to drive Python’s regressiontest suite, thanks to the-m option:python -m test. Underthe hood, it usestest.regrtest; the callpython -mtest.regrtest used in previous Python versions still works. Running thescript by itself automatically starts running all regression tests in thetest package. It does this by finding all modules in the package whosename starts withtest_, importing them, and executing the functiontest_main() if present or loading the tests viaunittest.TestLoader.loadTestsFromModule iftest_main does not exist. Thenames of tests to execute may also be passed to the script. Specifying a singleregression test (python -m test test_spam) will minimize output andonly print whether the test passed or failed.

Runningtest directly allows what resources are available fortests to use to be set. You do this by using the-u command-lineoption. Specifyingall as the value for the-u option enables allpossible resources:python -m test -uall.If all but one resource is desired (a more common case), acomma-separated list of resources that are not desired may be listed afterall. The commandpython -m test -uall,-audio,-largefilewill runtest with all resources except theaudio andlargefile resources. For a list of all resources and more command-lineoptions, runpython -m test -h.

Some other ways to execute the regression tests depend on what platform thetests are being executed on. On Unix, you can runmake test at thetop-level directory where Python was built. On Windows,executingrt.bat from yourPCBuild directory will run allregression tests.

26.9.test.support — Utilities for the Python test suite

Thetest.support module provides support for Python’s regressiontest suite.

Note

test.support is not a public module. It is documented here to helpPython developers write tests. The API of this module is subject to changewithout backwards compatibility concerns between releases.

This module defines the following exceptions:

exceptiontest.support.TestFailed

Exception to be raised when a test fails. This is deprecated in favor ofunittest-based tests andunittest.TestCase’s assertionmethods.

exceptiontest.support.ResourceDenied

Subclass ofunittest.SkipTest. Raised when a resource (such as anetwork connection) is not available. Raised by therequires()function.

Thetest.support module defines the following constants:

test.support.verbose

True when verbose output is enabled. Should be checked when moredetailed information is desired about a running test.verbose is set bytest.regrtest.

test.support.is_jython

True if the running interpreter is Jython.

test.support.TESTFN

Set to a name that is safe to use as the name of a temporary file. Anytemporary file that is created should be closed and unlinked (removed).

Thetest.support module defines the following functions:

test.support.forget(module_name)

Remove the module namedmodule_name fromsys.modules and delete anybyte-compiled files of the module.

test.support.is_resource_enabled(resource)

ReturnTrue ifresource is enabled and available. The list ofavailable resources is only set whentest.regrtest is executing thetests.

test.support.requires(resource,msg=None)

RaiseResourceDenied ifresource is not available.msg is theargument toResourceDenied if it is raised. Always returnsTrue if called by a function whose__name__ is'__main__'.Used when tests are executed bytest.regrtest.

test.support.findfile(filename,subdir=None)

Return the path to the file namedfilename. If no match is foundfilename is returned. This does not equal a failure since it could be thepath to the file.

Settingsubdir indicates a relative path to use to find the filerather than looking directly in the path directories.

test.support.run_unittest(*classes)

Executeunittest.TestCase subclasses passed to the function. Thefunction scans the classes for methods starting with the prefixtest_and executes the tests individually.

It is also legal to pass strings as parameters; these should be keys insys.modules. Each associated module will be scanned byunittest.TestLoader.loadTestsFromModule(). This is usually seen in thefollowingtest_main() function:

deftest_main():support.run_unittest(__name__)

This will run all tests defined in the named module.

test.support.run_doctest(module,verbosity=None)

Rundoctest.testmod() on the givenmodule. Return(failure_count,test_count).

Ifverbosity isNone,doctest.testmod() is run with verbosityset toverbose. Otherwise, it is run with verbosity set toNone.

test.support.check_warnings(*filters,quiet=True)

A convenience wrapper forwarnings.catch_warnings() that makes iteasier to test that a warning was correctly raised. It is approximatelyequivalent to callingwarnings.catch_warnings(record=True) withwarnings.simplefilter() set toalways and with the option toautomatically validate the results that are recorded.

check_warnings accepts 2-tuples of the form("messageregexp",WarningCategory) as positional arguments. If one or morefilters areprovided, or if the optional keyword argumentquiet isFalse,it checks to make sure the warnings are as expected: each specified filtermust match at least one of the warnings raised by the enclosed code or thetest fails, and if any warnings are raised that do not match any of thespecified filters the test fails. To disable the first of these checks,setquiet toTrue.

If no arguments are specified, it defaults to:

check_warnings(("",Warning),quiet=True)

In this case all warnings are caught and no errors are raised.

On entry to the context manager, aWarningRecorder instance isreturned. The underlying warnings list fromcatch_warnings() is available via the recorder object’swarnings attribute. As a convenience, the attributes of the objectrepresenting the most recent warning can also be accessed directly throughthe recorder object (see example below). If no warning has been raised,then any of the attributes that would otherwise be expected on an objectrepresenting a warning will returnNone.

The recorder object also has areset() method, which clears thewarnings list.

The context manager is designed to be used like this:

withcheck_warnings(("assertion is always true",SyntaxWarning),("",UserWarning)):exec('assert(False, "Hey!")')warnings.warn(UserWarning("Hide me!"))

In this case if either warning was not raised, or some other warning wasraised,check_warnings() would raise an error.

When a test needs to look more deeply into the warnings, rather thanjust checking whether or not they occurred, code like this can be used:

withcheck_warnings(quiet=True)asw:warnings.warn("foo")assertstr(w.args[0])=="foo"warnings.warn("bar")assertstr(w.args[0])=="bar"assertstr(w.warnings[0].args[0])=="foo"assertstr(w.warnings[1].args[0])=="bar"w.reset()assertlen(w.warnings)==0

Here all warnings will be caught, and the test code tests the capturedwarnings directly.

Changed in version 3.2:New optional argumentsfilters andquiet.

test.support.captured_stdin()
test.support.captured_stdout()
test.support.captured_stderr()

A context managers that temporarily replaces the named stream withio.StringIO object.

Example use with output streams:

withcaptured_stdout()asstdout,captured_stderr()asstderr:print("hello")print("error",file=sys.stderr)assertstdout.getvalue()=="hello\n"assertstderr.getvalue()=="error\n"

Example use with input stream:

withcaptured_stdin()asstdin:stdin.write('hello\n')stdin.seek(0)# call test code that consumes from sys.stdincaptured=input()self.assertEqual(captured,"hello")
test.support.temp_dir(path=None,quiet=False)

A context manager that creates a temporary directory atpath andyields the directory.

Ifpath isNone, the temporary directory is created usingtempfile.mkdtemp(). Ifquiet isFalse, the context managerraises an exception on error. Otherwise, ifpath is specified andcannot be created, only a warning is issued.

test.support.change_cwd(path,quiet=False)

A context manager that temporarily changes the current workingdirectory topath and yields the directory.

Ifquiet isFalse, the context manager raises an exceptionon error. Otherwise, it issues only a warning and keeps the currentworking directory the same.

test.support.temp_cwd(name='tempcwd',quiet=False)

A context manager that temporarily creates a new directory andchanges the current working directory (CWD).

The context manager creates a temporary directory in the currentdirectory with namename before temporarily changing the currentworking directory. Ifname isNone, the temporary directory iscreated usingtempfile.mkdtemp().

Ifquiet isFalse and it is not possible to create or changethe CWD, an error is raised. Otherwise, only a warning is raisedand the original CWD is used.

test.support.temp_umask(umask)

A context manager that temporarily sets the process umask.

test.support.can_symlink()

ReturnTrue if the OS supports symbolic links,Falseotherwise.

@test.support.skip_unless_symlink

A decorator for running tests that require support for symbolic links.

@test.support.anticipate_failure(condition)

A decorator to conditionally mark tests withunittest.expectedFailure(). Any use of this decorator shouldhave an associated comment identifying the relevant tracker issue.

@test.support.run_with_locale(catstr,*locales)

A decorator for running a function in a different locale, correctlyresetting it after it has finished.catstr is the locale category asa string (for example"LC_ALL"). Thelocales passed will be triedsequentially, and the first valid locale will be used.

test.support.make_bad_fd()

Create an invalid file descriptor by opening and closing a temporary file,and returning its descriptor.

test.support.import_module(name,deprecated=False)

This function imports and returns the named module. Unlike a normalimport, this function raisesunittest.SkipTest if the modulecannot be imported.

Module and package deprecation messages are suppressed during this importifdeprecated isTrue.

New in version 3.1.

test.support.import_fresh_module(name,fresh=(),blocked=(),deprecated=False)

This function imports and returns a fresh copy of the named Python moduleby removing the named module fromsys.modules before doing the import.Note that unlikereload(), the original module is not affected bythis operation.

fresh is an iterable of additional module names that are also removedfrom thesys.modules cache before doing the import.

blocked is an iterable of module names that are replaced withNonein the module cache during the import to ensure that attempts to importthem raiseImportError.

The named module and any modules named in thefresh andblockedparameters are saved before starting the import and then reinserted intosys.modules when the fresh import is complete.

Module and package deprecation messages are suppressed during this importifdeprecated isTrue.

This function will raiseImportError if the named module cannot beimported.

Example use:

# Get copies of the warnings module for testing without affecting the# version being used by the rest of the test suite. One copy uses the# C implementation, the other is forced to use the pure Python fallback# implementationpy_warnings=import_fresh_module('warnings',blocked=['_warnings'])c_warnings=import_fresh_module('warnings',fresh=['_warnings'])

New in version 3.1.

test.support.bind_port(sock,host=HOST)

Bind the socket to a free port and return the port number. Relies onephemeral ports in order to ensure we are using an unbound port. This isimportant as many tests may be running simultaneously, especially in abuildbot environment. This method raises an exception if thesock.family isAF_INET andsock.type isSOCK_STREAM, and the socket hasSO_REUSEADDR orSO_REUSEPORT set on it.Tests should never set these socket options for TCP/IP sockets.The only case for setting these options is testing multicasting viamultiple UDP sockets.

Additionally, if theSO_EXCLUSIVEADDRUSE socket option isavailable (i.e. on Windows), it will be set on the socket. This willprevent anyone else from binding to our host/port for the duration of thetest.

test.support.find_unused_port(family=socket.AF_INET,socktype=socket.SOCK_STREAM)

Returns an unused port that should be suitable for binding. This isachieved by creating a temporary socket with the same family and type asthesock parameter (default isAF_INET,SOCK_STREAM),and binding it to the specified host address (defaults to0.0.0.0)with the port set to 0, eliciting an unused ephemeral port from the OS.The temporary socket is then closed and deleted, and the ephemeral port isreturned.

Either this method orbind_port() should be used for any testswhere a server socket needs to be bound to a particular port for theduration of the test.Which one to use depends on whether the calling code is creating a pythonsocket, or if an unused port needs to be provided in a constructoror passed to an external program (i.e. the-accept argument toopenssl’s s_server mode). Always preferbind_port() overfind_unused_port() where possible. Using a hard coded port isdiscouraged since it can make multiple instances of the test impossible torun simultaneously, which is a problem for buildbots.

test.support.load_package_tests(pkg_dir,loader,standard_tests,pattern)

Generic implementation of theunittestload_tests protocol foruse in test packages.pkg_dir is the root directory of the package;loader,standard_tests, andpattern are the arguments expected byload_tests. In simple cases, the test package’s__init__.pycan be the following:

importosfromtest.supportimportload_package_testsdefload_tests(*args):returnload_package_tests(os.path.dirname(__file__),*args)
test.support.detect_api_mismatch(ref_api,other_api,*,ignore=())

Returns the set of attributes, functions or methods ofref_api notfound onother_api, except for a defined list of items to beignored in this check specified inignore.

By default this skips private attributes beginning with ‘_’ butincludes all magic methods, i.e. those starting and ending in ‘__’.

New in version 3.5.

test.support.check__all__(test_case,module,name_of_module=None,extra=(),blacklist=())

Assert that the__all__ variable ofmodule contains all public names.

The module’s public names (its API) are detected automaticallybased on whether they match the public name convention and were defined inmodule.

Thename_of_module argument can specify (as a string or tuple thereof) whatmodule(s) an API could be defined in in order to be detected as a publicAPI. One case for this is whenmodule imports part of its public API fromother modules, possibly a C backend (likecsv and its_csv).

Theextra argument can be a set of names that wouldn’t otherwise be automaticallydetected as “public”, like objects without a proper__module__attribute. If provided, it will be added to the automatically detected ones.

Theblacklist argument can be a set of names that must not be treated as part ofthe public API even though their names indicate otherwise.

Example use:

importbarimportfooimportunittestfromtestimportsupportclassMiscTestCase(unittest.TestCase):deftest__all__(self):support.check__all__(self,foo)classOtherTestCase(unittest.TestCase):deftest__all__(self):extra={'BAR_CONST','FOO_CONST'}blacklist={'baz'}# Undocumented name.# bar imports part of its API from _bar.support.check__all__(self,bar,('bar','_bar'),extra=extra,blacklist=blacklist)

New in version 3.6.

Thetest.support module defines the following classes:

classtest.support.TransientResource(exc,**kwargs)

Instances are a context manager that raisesResourceDenied if thespecified exception type is raised. Any keyword arguments are treated asattribute/value pairs to be compared against any exception raised within thewith statement. Only if all pairs match properly againstattributes on the exception isResourceDenied raised.

classtest.support.EnvironmentVarGuard

Class used to temporarily set or unset environment variables. Instances canbe used as a context manager and have a complete dictionary interface forquerying/modifying the underlyingos.environ. After exit from thecontext manager all changes to environment variables done through thisinstance will be rolled back.

Changed in version 3.1:Added dictionary interface.

EnvironmentVarGuard.set(envvar,value)

Temporarily set the environment variableenvvar to the value ofvalue.

EnvironmentVarGuard.unset(envvar)

Temporarily unset the environment variableenvvar.

classtest.support.SuppressCrashReport

A context manager used to try to prevent crash dialog popups on tests thatare expected to crash a subprocess.

On Windows, it disables Windows Error Reporting dialogs usingSetErrorMode.

On UNIX,resource.setrlimit() is used to setresource.RLIMIT_CORE’s soft limit to 0 to prevent coredump filecreation.

On both platforms, the old value is restored by__exit__().

classtest.support.WarningsRecorder

Class used to record warnings for unit tests. See documentation ofcheck_warnings() above for more details.

classtest.support.FakePath(path)

Simplepath-like object. It implements the__fspath__()method which just returns thepath argument. Ifpath is an exception,it will be raised in__fspath__().