runtests.pl - run one or more test cases
runtests.pl [options] [tests]
runtests.pl runs one, several or all the existing test casesin curl's test suite. It is often called from the root Makefile of thecurl package with 'make test'.
Specify which test(s) to run by specifying test numbers orkeywords.
If no test number or keyword is given, all existing tests that thescript can find are considered for running. You can specify single testcases to run by specifying test numbers space-separated, like1 3 5 7 11, and you can specify a range of tests like45 to 67.
Specify tests to not run with a leading exclamation point, like!66, which runs all available tests except number 66.
Prefix a test number with a tilde (~) to still run it, but ignore theresults.
It is also possible to specify tests based on a keyword describingthe test(s) to run, likeFTPS. The keywords are stringsused in the individual tests.
Features are included as keywords with thefeat: prefix(e.g.,feat:debug). Specify a feature to run only testsrequiring it, or exclude tests using!feat:<feature>,like!feat:proxy, to disable tests which depend on thatfeature.
You can also specify keywords with a leading exclamation point andthe keyword or phrase, like "!HTTP NTLM auth" to run all testsexcept those using this keyword. Remember that theexclamation marks and spaces need to be quoted somehow when entered atmany command shells.
Prefix a keyword with a tilde (~) to still run it, but ignore theresults.
When running without-s (short output), for instancewhen running runtests.pl directly rather than via make, each test emitsa pair of lines like this:
Test 0045...[simple HTTP Location: without protocol in initial URL]--pd---e-v- OK (45 out of 1427, remaining: 16:08, took 6.188s, duration: 00:31)the first line contains the test number and a description. On thesecond line, the characters at the beginning are flags indicating whichaspects of curl's behavior were checked by the test:
s stdoutr stderrp protocold datau uploadP proxyo outpute exit codem memoryv valgrindE the test was run event-basedThe remainder of the second line contains the test result, currenttest sequence, total number of tests to be run and an estimated amountof time to complete the test run.
-aContinue running the rest of the test cases even if one test fails.By default, the test script stops as soon as an error is detected.
-ac \<curl\>Provide a path to a curl binary to talk to APIs (currently only CItest APIs).
-amDisplay test results in automake style output(PASS/FAIL: [number] [name]).
-c \<curl\>Provide a path to a custom curl binary to run the tests with. Defaultis the curl executable in the build tree.
-dEnable protocol debug: have the servers display protocol output. Ifused in conjunction with parallel testing, it is difficult to associatethe logs with the specific test being run.
-E \<exclude_file\>Load theexclude_file with additional reasons whycertain tests should be skipped. Useful when testing with external HTTPproxies in which case some of the tests are not appropriate.
The file contains colon-delimited lines. The first field contains thetype of exclusion, the second field contains a pattern and the finalfield contains the reason why matching tests should be skipped. Theexclusion types arekeyword,test, andtool.
-e or--test-eventRun the test event-based (if possible). This makes runtests invokecurl with --test-event option. This option only works if both curl andlibcurl were built debug-enabled.
-fForce the test to run even if mentioned in DISABLED.
-gRun the given test(s) with gdb. This is best used on a single testcase and curl built --disable-shared. This then fires up gdb withcommand line set to run the specified test case. Simply (set abreak-point and) type 'run' to start.
-glRun the given test(s) with lldb. This is best used on a single testcase and curl built --disable-shared. This then fires up lldb withcommand line set to run the specified test case. Simply (set abreak-point and) type 'run' to start.
-gwRun the given test(s) with gdb as a windowed application.
-h, --helpDisplays a help text about this program's command line options.
-j[num]Spawn the given number of processes to run tests in. This defaults to0 to run tests serially within a single process. Using a number greaterthan one allows multiple tests to run in parallel, speeding up a testrun. The optimum number is dependent on the system and set of tests torun, but 7 times the number of CPU cores is a good figure to start with,or 1.3 times if Valgrind is in use, or 5 times for torture tests.Enabling parallel tests is not recommended in conjunction with the -goption.
-kKeep output and log files in log/ after a test run, even if no errorwas detected. Useful for debugging.
-L \<file\>Load and execute the specified file which should contain perl code.This option allows one to changeruntests.pl behavior byoverwriting functions and variables and is useful when testing externalproxies using curl's regression test suite.
-lLists all test case names.
-nDisable the check for and use of valgrind.
--no-debuginfodDelete theDEBUGINFOD_URLS variable if that is defined.Makes valgrind, gdb etc not able to use this functionality.
-o \<variablename=value\>Overwrite the specified internalvariable withvalue. Useful to change variables that did not get adedicated flag to change them. Check the source to see which variablesare available.
-P \<proxy\>Use the specified HTTP proxy when executing tests, even if the teststhemselves do not specify a proxy. This option allows one to testexternal proxies using curl's regression test suite.
-pPrints out all files in the log directory to stdout when a test casefails. Practical when used in the automated and distributed tests sincethen the people checking the failures and the reasons for them might nothave physical access to the machine and logs.
-RRun the tests in a scrambled, or randomized, order instead ofsequentially.
The random seed initially set for this is fixed per month and can beset with--seed.
-rDisplay run time statistics. (Requires thePerl Time::HiRes module)
-rfDisplay full run time statistics. (Requires thePerl Time::HiRes module)
--repeat=[num]This repeats the given set of test numbers this many times. If notest numbers are given, it repeats ALL tests this many times. It addsthe new repeated sequence at the end of the initially given one.
If-R option is also used, the scrambling is doneafter the repeats have extended the test sequence.
-sShorter output. Speaks less than default.
--seed=[num]When using--shallow or-R that randomize certainaspects of the behavior, this option can set the initial seed. If notset, the random seed is set based on the currently set local year andmonth and the first line of the "curl -V" output.
--shallow=[num]Used together with-t. This limits the number oftests to fail in torture mode to no more thannum pertest case. If this reduces the amount, the script randomly discardsentries to fail until the amount isnum.
The random seed initially set for this is fixed per month and can beset with--seed.
-t[num]Selects atorture test for the given tests. Thismakes runtests.pl first run the tests once and count the number ofmemory allocations made. It then reruns the test that number of times,each time forcing one of the allocations to fail until all allocationshave been tested. By settingnum you can force the allocationwith that number to be set to fail at once instead of looping througheveryone, which is handy when debugging and then often in combinationwith-g.
--test-duphandlePasses the--test-duphandle option to curl when invoked.This command line option only exists in debug builds and runs curlnormally, but duplicates the easy handle before the transfer and use theduplicate instead of the original handle. This verifies that theduplicate works exactly as good as the original handle.
Because of how the curl tool uses a share object to store and keepsome data, not everything is however perfectly copied in the duplicate.In particular HSTS data is not. A specific test case can be set to avoidusing--test-duphandle by disabling it on a per testbasis.
-uError instead of warning on server unexpectedly alive.
-vEnable verbose output. Speaks more than by default. If used inconjunction with parallel testing, it is difficult to associate the logswith the specific test being run.
-vc \<curl\>Provide a path to a custom curl binary to run when verifying that theservers running are indeed our test servers. Default is the curlexecutable in the build tree.
Many tests have conditions that must be met before the test case canrun fine. They could depend on built-in features in libcurl or featurespresent in the operating system or even in third-party libraries thatcurl may or may not use.
The test script checks most of these by itself to determine when itis safe to attempt to run each test. Those which cannot be run due tofailed requirements are simply skipped and listed at the completion ofall test cases. In some unusual configurations, the test script cannotmake the correct determination for all tests. In these cases, theproblematic tests can be skipped using the "!keyword" skip featuredocumented earlier.
The simplest way to write test cases is to start with a similarexisting test, save it with a new number and then adjust it to fit.There is an attempt to document the test case file format intests/FILEFORMAT.md.