Repository files navigation

A unit testing framework for CMake.

Travis CI (Ubuntu)	AppVeyor (Windows)	Coverage	Biicode	Licence

Because CMake is a powerful and battle-tested language for writing build systemsfor large-scale C++ projects, but its dynamic nature makes it easy to makeundetectable errors which later ship as bugs that either you, or the users ofyour macros, need to work around. We have to put a lot of logic inside outCMake scripts sometimes, like propogation of global state, if conditions forvarious options and host system configurations and loops over variable argumentlists.

It is something you want to get right the first time rather than having toscratch your head about later with the lack of debugging tools for CMakescripts.

cmake-unit is written entirely using the CMake language and should work acrossall platforms where CMake is supported. It had been tested on:

• Windows (Visual Studio 2010, 2012, 2013, NMake)
• Mac OS X (XCode, Ninja, Make)
• Ubuntu (Ninja, Make)

cmake-unit should be included as a submodule in your project and comes withthree files.

CMakeUnitRunner contains the main "runner" script for loading and executingtest scripts. Include this file in the/CMakeLists.txt.

Tests are defined inline as functions. They are automatically discovered bycmake-unit and the name of the function must be in the format of${your_namespace}_test_{test_name}. Within each test function arefunction definitions used to control each "phase" of the test's build. Afterthese functions are called, a call tocmake_unit_configure_test ties allthe phases functions together into a single test.

Test functions are subdivided into "phases", each phase having its ownscript that can be run in place of the default. Usually you will want tooverride the CONFIGURE or VERIFY phases in order to provide your ownproject set-up and verification scripts. The build of each project goesthrough the following phases, in order:

• PRECONFIGURE
• CLEAN
• INVOKE_CONFIGURE
• CONFIGURE
• INVOKE_BUILD
• INVOKE_TEST
• VERIFY
• COVERAGE

For each phase, a name of a function can be provided which will "override"the default function called for that phase. Some phases are called withindifferent CMake invocations, so you shouldn't assume that state can beshared between the phase functions.

The name of each phase is a keyword argument tocmake_unit_configure_test.Following the phase name, further options can be specified for each phase.Some common options are:

• COMMAND: The name of a function to run when this phase is encountered.
• ALLOW_FAIL: A keyword specifying that this phase is permitted to fail(and further, that no phase after this one should be run).

This is the first phase that is run for the test. It cannot be overridden.It does some initial setup for the test itself, including writing outa special driver script which will be used to invoke this test at CTest time.

Two options are exposed to this unit for this phase,SKIP_GENERATOR_REGEXandSKIP_SYSTEM_REGEX. A list of regular expressions can be provided foreach option, which will cause the test not to run when the regex matchesCMAKE_SYSTEM andCMAKE_GENERATOR respectively.

This phase is responsible for cleaning the build directory of the test. Bydefault, it callscmake_unit_invoke_clean, which just removes the testproject'sCMAKE_BINARY_DIR.

This phase is responsible for writing out a stub/CMakeLists.txt and jumpinginvokingcmake on the resulting project folder. By default it will callcmake_unit_invoke_configure. The written out/CMakeLists.txt will do somesetup for the test project, including calling theproject command.

cmake_unit_invoke_configure will not configure any languages by default. Thisis to prevent unnecessary overhead when testing on platforms where configuringlanguage support is quite slow (for instance, Visual Studio and XCode). Insteadof overriding the command, usually the only action you will need to take ifyou need language support is to set theLANGUAGES option (eg, toC CXX).

This phase is responsible for actually configuring the project. Any commandsrun inside this phase are effectively run as though CMake was configuringa project by processing a/CMakeLists.txt, so the full range of commandsare available. Usually you will want to override theCOMMAND and configureyour project as required (or make assertions).

This phase is responsible for invokingcmake --build. Usually theCOMMANDwill not need to be overridden, but if the build can fail or if the projectshould not be built at all, thenALLOW_FAIL orCOMMAND NONE should bespecified respectively.

TheTARGET option allows you to specify a custom target to build insteadof the default one.

This phase is responsible for invokingctest. Usually theCOMMAND willnot need to be overridden, unless you need to invokectestin a special way.

This phase is responsible for verifying that the configure, build and teststeps went the way you expected. It is executed after the final step of theconfigure-build-test cycle is completed for this project.

You can inspect the standard output and error of each of these steps. Use thecmake_unit_get_log_for command in order to fetch the path to these log files.

This phase is responsible for collecting tracefile output and turning it intoline-coverage statistics. It is not overridable.

Here is an example of how a test looks in practice:

function (namespace_test_one)    function (_namespace_configure)        cmake_unit_create_simple_library (library SHARED FUNCTIONS function)        cmake_unit_create_simple_executable (executable)        target_link_libraries (executable library)        cmake_unit_assert_that (executable is_linked_to library)    endfunction ()    function (_namespace_verify)        cmake_unit_get_log_for (INVOKE_BUILD OUTPUT BUILD_OUTPUT)        cmake_unit_assert_that ("${BUILD_OUTPUT}"                                file_contents any_line                                matches_regex                                "^.*executable.*$")    endfunction ()    cmake_unit_configure_test (INVOKE_CONFIGURE LANGUAGES C CXX                               CONFIGURE COMMAND _namespace_configure                               VERIFY COMMAND _namespace_verify)endfunction ()

The_namespace_configure and_namespace_verify functions are defined withinthenamespace_test_one function. They are passed to theCOMMAND keyword fortheCONFIGURE andVERIFY phases oncmake_unit_configure_test.

LANGUAGES C CXX is passed toINVOKE_CONFIGURE. This ensures that compilersare tested and CMake is set up to build and link C and C++ binary code.

If there's no need to build and test the test project, or to verify it, youcan usecmake_unit_configure_config_only_test in place ofcmake_unit_configure_test. This will passINVOKE_BUILD COMMAND NONE andINVOKE_TEST COMMAND NONE tocmake_unit_configure_test along with whateveroptions you specify.

cmake_unit_init is what handles the registration and running of eachdiscovered test function. It takes a namespace as the argument to the keywordNAMESPACE. This is the name each test is prefixed with (followed by _test).Any function matching the pattern ^${namespace}test.*$ will be automaticallyregistered. It also takes a list of files considered to be candidates forcode coverage asCOVERAGE_FILES.

As an example, see the following:

cmake_unit_init (NAMESPACE namespace                 COVERAGE_FILES "${PROJECT_DIR}/Module.cmake")

CMakeUnit contains matchers and a generalcmake_unit_assert_that function.You can use them in both the configure and verify stages. Ifthe script hits an assertion failure, it will callmessage (SEND_ERROR).

The following matchers are available at the time of writing this documentation

• is_true: Matches if the passed variable name has a value that is booleantrue.
• is_false: Matches if the passed variable name has a value that is booleanfalse.
• target_exists: Matches if the target provided as the first argumentexists.
• variable_contains: Matches if a substring is present in the value ofthe value of the variable name.
• compare_as: Matches if the variable specified satisfies theparameters provided. A variable name, type, comparator statement(EQUALLESSGREATER) and value to compare against can be provided.
• matches_regex: Matches if the value of the variable provided, whentreated as a string matches the regex provided in the second argument.
• is_defined: Matches any variable that is defined
• executes_with_success: For a command and each of its arguments encapsulatedin the list passed-by-variable-name (as opposed to by value), check if itexecuted with success.
• is_linked_to: Matches if the target has a link library that matchesthe name specified by the second argument. It does regex matching to ensurethat in the default case, libraries with slightly inexact names betweenplatforms are still matched against.
• list_contains_value: Checks inside specified variable name containinga list to see if any item contains a value satisfying the criteria.
• has_property_with_value: Matches if the item specified with the item typespecified has property with a value and type specified which matches theprovided comparator.
• has_property_containing_value: Likehas_property_with_value but looksinside items in a list held by the property.
• exists_as_file: Matches if file exists on the filesystem.
• file_contents: Matches if the contents of a file match the matcherand arguments provided afterwards.
• any_line: Matches if any line of a multi-line string matches the followingmatcher and its arguments.
• not: Matches if the item specified does not match the following matcher.

cmake-unit can be extended with your own matchers. To do this, you willneed to write a "callable" function in your project's namespace, for example

function (my_namespace_equal_to_seven)    list (GET CALLER_ARGN 0 VARIABLE)    list (GET CALLER_ARGN -1 RESULT_VALUE)    set (${RESULT_VALUE} "to be equal to 7" PARENT_SCOPE)    if ("${${VARIABLE}}" EQUAL 7)        set (${RESULT_VALUE} TRUE PARENT_SCOPE)    endif ()endfunction ()

Then you will need to register your project's namespace as a namespacecontaining matchers

cmake_unit_register_matcher_namespace (my_namespace)

You can start using your matcher like so:

cmake_unit_assert_that (VARIABLE equal_to_seven)

The functionmy_namespace_equal_to_seven is acallable function abidingby the calling convention set out below. Its first argument will always bethe variable-to-be-matched. Depending on what your matcher does, this may bea value or a variable name. The last variable is always the "result variable",which is the name of the variable that you will need to set in the parentscope to indicate the matcher status. By convention, matchers shouldset this variable to a sentence fragment that would provide a sensibleexplanation of what happened in the sentence "Expected VARIABLE ..." incase there was a mismatch. Otherwise, the variable should be set to TRUE.

Each of these functions is a default for a phase of a cmake-unit test's buildcycle. If they are overridden, they can be "chained up" to.CALLER_ARGN willbe passed implicitly.

Cleans the projectBINARY_DIRECTORY (as specified inCALLER_ARGN).

Creates a/CMakeLists.txt for this project which does some initial setup andthen jumps to the function defined forCONFIGURE.cmake is invoked on abuild directory for the folder containing the created/CMakeLists.txt.

Invokescmake --build on this project.

Invokesctest -C Debug on this project.

cmake-unit also provides a few utility functions to make writing tests easier.

Escape all characters from INPUT and store in OUTPUT_VARIABLE

Reliably returns the binary and source directories for this test. You shoulduse this instead ofCMAKE_CURRENT_SOURCE_DIR andCMAKE_CURRENT_BINARY_DIRwhere possible, as it will be correct in every phase.

Writes out a source file, for use withadd_library,add_executable or sourcescanners during the configure phase.

If the source is detected as a header based on theNAME property such that itdoes not have a C or C++ extension, then header guards will be written andfunction definitions will not be included.

• [Optional]NAME: Name of the source file. May include slashes which willbe interpreted as a subdirectory relative toCMAKE_CURRENT_SOURCE_DIR.The default is Source.cpp
• [Optional]FUNCTIONS_EXPORT_TARGET: The target that this source file isbuilt for. Generally this is used if it is necessary to export functionsfrom this source file. cmake_unit_create_simple_library usesthis argument for instance.
• [Optional]INCLUDES: A list of files, relative or absolute paths, to#include
• [Optional]DEFINES: A list of#defines (macro name only)
• [Optional]FUNCTIONS: A list of functions.
• [Optional]PREPEND_CONTENTS: Contents to include in the file afterINCLUDES,DEFINES and Function Declarations, but before FunctionDefinitions
• [Optional]INCLUDE_DIRECTORIES: A list of directories such that, if anentry in the INCLUDES list has the same directory name as an entry inINCLUDE_DIRECTORIES then the entry will be angle-brackets<include> withthe path relative to that include directory.

Generates a source file, for use withadd_library,add_executableor source scanners during the build phase.

If the source is detected as a header based on theNAME property such thatit does not have a C or C++ extension, then header guards will be writtenand function definitions will not be included.

• TARGET_RETURN: Variable to store the name of the target this source filewill be generated on
• [Optional]NAME: Name of the source file. May include slashes which willbe interpreted as a subdirectory relative toCMAKE_CURRENT_SOURCE_DIR.The default is Source.cpp
• [Optional]FUNCTIONS_EXPORT_TARGET: The target that this source file isbuilt for. Generally this is used if it is necessary to export functionsfrom this source file. cmake_unit_create_simple_library usesthis argument for instance.
• [Optional]INCLUDES: A list of files, relative or absolute paths, to#include
• [Optional]DEFINES: A list of#defines (macro name only)
• [Optional]FUNCTIONS: A list of functions.
• [Optional]PREPEND_CONTENTS: Contents to include in the file afterINCLUDES,DEFINES and Function Declarations, but before FunctionDefinitions
• [Optional]INCLUDE_DIRECTORIES: A list of directories such that, if anentry in the INCLUDES list has the same directory name as an entry inINCLUDE_DIRECTORIES then the entry will be angle-brackets<include> withthe path relative to that include directory.

These functions can be used to generate binary targets such as simpleexecutables and libraries. There will only be a single source file perexecutable or library generated.

Creates a simple executable by the name "NAME" which will always have a "main"function.

• NAME: Name of executable
• [Optional]INCLUDES: A list of files, relative or absolute paths, to#include
• [Optional]DEFINES: A list of#defines (macro name only)
• [Optional]FUNCTIONS: A list of functions.
• [Optional]PREPEND_CONTENTS: Contents to include in the file afterINCLUDES,DEFINES and Function Declarations, but before FunctionDefinitions
• [Optional]INCLUDE_DIRECTORIES: A list of directories such that, if anentry in the INCLUDES list has the same directory name as an entry inINCLUDE_DIRECTORIES then the entry will be angle-brackets<include> withthe path relative to that include directory.

Creates a simple executable by the name "NAME" which will always have a "main"function.

• NAME: Name of executable
• [Optional]INCLUDES: A list of files, relative or absolute paths, to#include
• [Optional]DEFINES: A list of#defines (macro name only)
• [Optional]FUNCTIONS: A list of functions.
• [Optional]PREPEND_CONTENTS: Contents to include in the file afterINCLUDES,DEFINES and Function Declarations, but before FunctionDefinitions
• [Optional]INCLUDE_DIRECTORIES: A list of directories such that, if anentry in the INCLUDES list has the same directory name as an entry inINCLUDE_DIRECTORIES then the entry will be angle-brackets<include> withthe path relative to that include directory.

For an exports fileEXPORTS and targetTARGET, finds the location of atarget from an already generatedEXPORTS file.

This function should be run in the verify stage in order to determine thelocation of a binary or library built by CMake. The initial configurestep should runexport (TARGETS ...) in order to generate this file.

This function should always be used where a binary or library needs tobe invoked after build. Different platforms put the completed binariesin different places and also give them a different name. This functionwill resolve all those issues.

• EXPORTS: Full path toEXPORTS file to read
• TARGET: Name ofTARGET as it will be found in theEXPORTS file
• LOCATION_RETURN: Variable to write target'sLOCATION property into.

Exports the currentCMAKE_CFG_INTDIR variable (known at configure-time)and writes it into the file specified atLOCATION. This file could be readafter the build to determine theCMAKE_CFG_INTDIR property

• LOCATION: Filename to writeCMAKE_CFG_INTDIR variable to.

ReadsOUTPUT_FILE to import the value of theCMAKE_CFG_INTDIR propertyand stores the value inside ofLOCATION_RETURN. This should be run in theverify phase to get theCMAKE_CFG_INTDIR property for the configure phasegenerator. Usecmake_unit_export_cfg_int_dir in the configure phaseto export theCMAKE_CFG_INTDIR property.

• OUTPUT_FILE: Filename to readCMAKE_CFG_INTDIR variable from.
• LOCATION_RETURN: Variable to storeCMAKE_CFG_INTDIR value into.

Gets theLOG_TYPE log forPHASE and stores it in the variable specifiedinLOG_FILE_RETURN. The returned log is a path to a file. Valid valuesfor theLOG_TYPE parameter areERROR andOUTPUT.

CMakeTraceToLCov is a script that converts a tracefile generated by usingCMAKE_UNIT_LOG_COVERAGE=ON into a Linux Test Project Coverage (LCov)compatible file.

CMakeTraceToLCov should be run in script mode from the toplevel sourcedirectory where CMake scripts are to be kept.

There are two cache options which must be set prior to use:

• TRACEFILE: Path to a tracefile generated by usingCMAKE_UNIT_LOG_COVERAGE=ON
• LCOV_OUTPUT: Path to filename where LCov output file should be stored.

The following issues are known at the time of writing:

• #55 : Custom Command output on Visual Studio Generatorsnot available
• #56 : cmake-unit overrides add_custom_command
• #57 : Coverage file paths may not contain squarebrackets ([])

cmake-unit uses some clever hacks under the hood in order to achieve its"streamlined" test definition syntax.

Test auto-discovery, dynamic test loading and custom phase specification isall achieved through the ability to call arbitrary functions. CMake doesn'toffer any syntax to do so, but there is a back door using a debugging featurecalledvariable_watch.

Obviously,variable_watch provides its own arguments to the called function,which is not entirely what we want. However, CMake makes it relatively easy toestablish a kind of "calling convention" for these called functions. Usage ofkeyword arguments withCMakeParseArgumentsis pretty common for most modules. We defined a variable calledCALLER_ARGNwhich functions just likeARGN would in a normal function call. All argumentsare passed as keywords.

variable_watch can only be used to register a callback for one variable at atime, so if a function is to be called multiple times, then a register needsto be maintained mapping function names to variable names.

All of this is encapsulated within thecmake_call_function command. This ishosted as a separate block on biicode. It is hoped that eventually this canbecome part of the core CMake syntax.

Test function discovery is by using the "hidden"COMMANDS property ofGLOBAL scope. This provides a list of all defined commands at the timeof retrieving the property.