Synopsis

codeql test run [--threads=<num>] [--ram=<MB>] <options>... -- <test|dir>...

codeql test run [--threads=<num>] [--ram=<MB>] <options>... -- <test|dir>...

Description

Run unit tests for QL queries.

Options

Primary Options

<test|dir>...

Each argument is one of:

• A.ql or.qlref file that defines a test to run.
• A directory which will be searched recursively for tests to run.

--failing-exitcode=<code>

[Advanced] Set the exit code to produce if any failures areencountered. Usually 1, but tooling that parses the output may find ituseful to set it to 0.

--format=<fmt>

Select output format. Possible choices:

text(default): A human-readable textual rendering.

json: A streamed JSON array of test result objects.

betterjson: A streamed JSON array of event objects.

jsonz: A stream of zero-terminated JSON test result objects.

betterjsonz: A stream of zero-terminated JSON event objects.

For thebetterjson andbetterjsonz formats, each event has atypeproperty specifying the type of the event. New event types may be addedin the future, so consumers should ignore any event with an unrecognizedkind property.

--[no-]keep-databases

[Advanced] Preserve the databases extracted to run the test queries,even where all tests in a directory pass. (The database will always beleft present when there are tests thatfail).

--[no-]fast-compilation

[Deprecated] [Advanced] Omit particularly slow optimization stepswhen compiling test queries.

--[no-]learn

[Advanced] When a test produces unexpected output, instead of failingit, update its.expected file to match the actual output, such that itpasses. Tests can still fail in this mode, for example if creation of atest database to query does not succeed.

--consistency-queries=<dir>

[Advanced] A directory with consistency queries that will be run foreach test database. These queries should not produce any output (exceptwhen they find a problem) unless the test directory includes aCONSISTENCY subdirectory with a.expected file. This is mostlyuseful for testing extractors.

--[no-]check-databases

[Advanced] Runcodeql dataset check over each test database created and report a failure if it detects inconsistencies. This is useful when testing extractors. If the check is (temporarily!) expected to fail for a particular database, place aDB-CHECK.expected file in the test directory.

--[no-]show-extractor-output

[Advanced] Show the output from extractor scripts that create testdatabases. This can be useful while developing or editing test cases.Beware that it can cause duplicated or malformed output if you use thiswith multiple threads!

-M, --ram=<MB>

Set total amount of RAM the test runner should be allowed to use.

--slice=<N/M>

[Advanced] Divide the test cases intoM roughly equal-sized slicesand process only theNth of them. This can be used for manualparallelization of the testing process.

--[no-]strict-test-discovery

[Advanced] Only use queries that can be strongly identified as tests.This mode tries to distinguish between.ql files that define unittests and.ql files that are meant to be useful queries. This optionis used by tools, such as IDEs, that need to identify all unit tests ina directory tree without depending on previous knowledge of how thefiles in it are arranged.

Within a QL pack whoseqlpack.yml declares atests directory, all.ql files in that directory are considered tests, and.ql filesoutside it are ignored. In a QL pack that doesn't declare atestsdirectory, a.ql file is identified as a test only if it has acorresponding.expected file.

For consistency,.qlref files are limited by the same rules as.qlfiles even though a.qlref file cannot really be a non-test.

Options to find libraries and extractors used by the tests

--search-path=<dir>[:<dir>...]

A list of directories under which QL packs may be found. Each directorycan either be a QL pack (or bundle of packs containing a.codeqlmanifest.json file at the root) or the immediate parent of oneor more such directories.

If the path contains more than one directory, their order definesprecedence between them: when a pack name that must be resolved ismatched in more than one of the directory trees, the one given firstwins.

Pointing this at a checkout of the open-source CodeQL repository oughtto work when querying one of the languages that live there.

If you have checked out the CodeQL repository as a sibling of theunpacked CodeQL toolchain, you don't need to give this option; suchsibling directories will always be searched for QL packs that cannot befound otherwise. (If this default does not work, it is stronglyrecommended to set up--search-path once and for all in a per-userconfiguration file).

(Note: On Windows the path separator is;).

--additional-packs=<dir>[:<dir>...]

If this list of directories is given, they will be searched for packsbefore the ones in--search-path. The order between these doesn'tmatter; it is an error if a pack name is found in two different placesthrough this list.

This is useful if you're temporarily developing a new version of a packthat also appears in the default path. On the other hand, it isnotrecommended to override this option in a config file; some internalactions will add this option on the fly, overriding any configuredvalue.

(Note: On Windows the path separator is;).

--library-path=<dir>[:<dir>...]

[Advanced] An optional list of directories that will be added to theraw import search path for QL libraries. This should only be used ifyou're using QL libraries that have not been packaged as QL packs.

(Note: On Windows the path separator is;).

--dbscheme=<file>

[Advanced] Explicitly define which dbscheme queries should be compiledagainst. This should only be given by callers that are extremely surewhat they're doing.

--compilation-cache=<dir>

[Advanced] Specify an additional directory to use as a compilationcache.

--no-default-compilation-cache

[Advanced] Don't use compilation caches in standard locations such asin the QL pack containing the query or in the CodeQL toolchaindirectory.

Options for configuring the CodeQL package manager

--registries-auth-stdin

Authenticate to GitHub Enterprise Server Container registries by passinga comma-separated list of <registry_url>=<token> pairs.

For example, you can passhttps://containers.GHEHOSTNAME1/v2/=TOKEN1,https://containers.GHEHOSTNAME2/v2/=TOKEN2to authenticate to two GitHub Enterprise Server instances.

This overrides the CODEQL_REGISTRIES_AUTH and GITHUB_TOKEN environmentvariables. If you only need to authenticate to the github.com Containerregistry, you can instead authenticate using the simpler--github-auth-stdin option.

--github-auth-stdin

Authenticate to the github.com Container registry by passing agithub.com GitHub Apps token or personal access token via standardinput.

To authenticate to GitHub Enterprise Server Container registries, pass--registries-auth-stdin or use the CODEQL_REGISTRIES_AUTH environmentvariable.

This overrides the GITHUB_TOKEN environment variable.

Options to control query compilation

--no-release-compatibility

[Advanced] Use the newest compiler features, at the cost ofportability.

From time to time, new QL language features and evaluator optimizationswill be supported by the QL evaluator a few releases before they areenabled by default in the QL compiler. This helps ensure that theperformance you experience when developing queries in the newest CodeQLrelease can be matched by slightly older releases that may still be inuse for Code Scanning or CI integrations.

If you do not care about your queries being compatible with other(earlier or later) CodeQL releases, you can sometimes achieve a smallamount of extra performance by using this flag to enable recentimprovements in the compiler early.

In releases where there are no recent improvements to enable, thisoption silently does nothing. Thus it is safe to set it once and for allin your global CodeQL config file.

Available sincev2.11.1.

Options that control the evaluation of test queries

--[no-]tuple-counting

[Advanced] Display tuple counts for each evaluation step in the queryevaluator logs. If the--evaluator-log option is provided, tuplecounts will be included in both the text-based and structured JSON logsproduced by the command. (This can be useful for performanceoptimization of complex QL code).

--timeout=<seconds>

[Advanced] Set the timeout length for query evaluation, in seconds.

The timeout feature is intended to catch cases where a complex querywould take "forever" to evaluate. It is not an effective way to limitthe total amount of time the query evaluation can take. The evaluationwill be allowed to continue as long as each separately timed part of thecomputation completes within the timeout. Currently these separatelytimed parts are "RA layers" of the optimized query, but that mightchange in the future.

If no timeout is specified, or is given as 0, no timeout will be set(except forcodeql test run, where the default timeout is 5 minutes).

-j, --threads=<num>

Use this many threads to evaluate queries.

Defaults to 1. You can pass 0 to use one thread per core on the machine,or -N to leaveN cores unused (except still use at least onethread).

Options for controlling outputting of structured evaluator logs

--evaluator-log=<file>

[Advanced] Output structured logs about evaluator performance to thegiven file. The format of this log file is subject to change with nonotice, but will be a stream of JSON objects separated by either twonewline characters (by default) or one if the--evaluator-log-minifyoption is passed. Please usecodeql generate log-summary <file> toproduce a more stable summary of this file, and avoid parsing the filedirectly. The file will be overwritten if it already exists.

--evaluator-log-minify

[Advanced] If the--evaluator-log option is passed, also passingthis option will minimize the size of the JSON log produced, at theexpense of making it much less human readable.

Options for checking imported TRAP

--[no-]check-undefined-labels

[Advanced] Report errors for undefined labels.

--[no-]check-unused-labels

[Advanced] Report errors for unused labels.

--[no-]check-repeated-labels

[Advanced] Report errors for repeated labels.

--[no-]check-redefined-labels

[Advanced] Report errors for redefined labels.

--[no-]check-use-before-definition

[Advanced] Report errors for labels used before they're defined.

--[no-]fail-on-trap-errors

[Advanced] Exit non-zero if an error occurs during trap import.

--[no-]include-location-in-star

[Advanced] Construct entity IDs that encode the location in the TRAPfile they came from. Can be useful for debugging of TRAP generators, buttakes up a lot of space in the dataset.

--[no-]linkage-aware-import

[Advanced] Controls whethercodeql dataset import is linkage-aware(default) or not. On projects where this part of database creationconsumes too much memory, disabling this option may help them progressat the expense of database completeness.

Available sincev2.15.3.

Common options

-h, --help

Show this help text.

-J=<opt>

[Advanced] Give option to the JVM running the command.

(Beware that options containing spaces will not be handled correctly.)

-v, --verbose

Incrementally increase the number of progress messages printed.

-q, --quiet

Incrementally decrease the number of progress messages printed.

--verbosity=<level>

[Advanced] Explicitly set the verbosity level to one of errors,warnings, progress, progress+, progress++, progress+++. Overrides-vand-q.

--logdir=<dir>

[Advanced] Write detailed logs to one or more files in the givendirectory, with generated names that include timestamps and the name ofthe running subcommand.

(To write a log file with a name you have full control over, insteadgive--log-to-stderr and redirect stderr as desired.)

--common-caches=<dir>

[Advanced] Controls the location of cached data on disk that willpersist between several runs of the CLI, such as downloaded QL packs andcompiled query plans. If not set explicitly, this defaults to adirectory named.codeql in the user's home directory; it will becreated if it doesn't already exist.

Available sincev2.15.2.