About QUTest

QUTest (pronounced "cutest") is auniversal trace-based testing harness (a.k.a.testing framework), which is specifically designed for embedded systems, but also supports testing of embedded code on host computers ("dual targeting"). As aunit testing harness, QUTest is the fundamental tooling forTest-Driven Development (TDD)↑ andBehavior-Driven Development (BDD)↑ ofQP/C/C++ applications↑, which is a highly recommended best-practice. As an universal testing harness, QUTest can also be used forintegration testing (on embedded hardware only).

Note

Even though QUTest has been primarily designed for testing ofevent-driven systems↑, it is universal and can also be used to testany embedded C or C++ code not necessarily based on theQP real-time event frameworks↑. To demonstrate this capability, QUTest comes with a few examples from the conventionalUnity testing framework↑, which among others demonstrates such techniques as usingtest doubles↑ andmocks↑ with QUTest.

Trace-Based Testing (TBT)

QUTest is atrace-based testing (TBT) framework, which works similarly to "debugging with printf", where you instrument the code to output information about its execution. You then run the code under test (CUT) with a controlled set of inputs, and examine the produced output from the generated trace to determine whether the code under test operates correctly. The main differences from using ‘printf’ are:

1. A much more efficientQP/Spy software tracing mechanism is used instead, and
2. Both generating the inputs and checking the test outputs are fullyautomated.

The process of testing embedded code with QUTest involves the following components:

1. TheTarget, which runs an instrumentedtest fixture code written in C or C++, whose job is to exercise yourCodeUnderTest (CUT). Please note that atest fixture only exercises the CUT and reports the results to the host usingQP/Spy, but thetest fixture doesnot check if the CUT operates "correctly".
   

   Note

   Of course, thetest fixture is written in the same programming language as the CUT (C or C++), and it links statically with the CUT. Several examples oftest fixtures are explained in theQUTest Tutorial. The details oftest fixtures are described inQUTest Fixture Reference.

2. TheQSPY "back-end" application running on the host, which receives the tracing data from the Target and also opens a communication channel for the QUTest "front-end".
3. TheQUTest "front-end" also running on the host that executestest scripts. Thetest scripts (written in Python) drive the tests and check the QSPY output against the expectations ("test-assertions").
   

   Note

   Several examples oftest scripts are explained in theQUTest Tutorial. The details oftest scripts are described inQUTest Script Reference.

   

   Remarks

   The separation between CUT execution and checking the test results has many benefits. One of the most important ones is that CUT execution (test fixture) and checking the results (test script) can be done indifferent programming languages. To this end, QUTest provides support for writing thetest scripts in Python.

   The general QUTest structure just described corresponds to running tests on an embedded Target. But QUTest can also execute tests on thehost computer. In that case (shown in the figure below), thetest fixture is a host executable that communicates with theQSPY host application via a TCP/IP socket. In this case, all QUTest components execute on the host computer.

QUTest with Host Executable

Remarks

To work effectively with QUTest, you need to understandhow much output to expect from any given input. Additionally, you need to know how QUTest ensures that for every input to thetest fixture, the generated QSPY output matches all test expectations and that there are no extra expectations or missing expectations after every command from thetest script. These subjects are explained in the sectionRun-to-Completion Processing.

What's Special about QUTest?

Unlike other existing unit testing harnesses for embedded systems (e.g.,Unity↑ orCppUTest↑) QUTest isnot based onxUnit↑. Instead, QUTest is based on the principle oftrace-based testing, which is particularly suitable fordeeply embedded andevent-driven systems in general. Here is a list of QUTest's unique features, specifically designed for this purpose:

• QUTest separatesexecution of the CUT (Code Under Test) fromchecking of the "test assertions". The embedded target is concerned only with running atest fixture that exercises the CUT and produces the QP/Spy trace, but it doesnot check the test expectations ("test assertions"). Verifying the correctness of test expectations is performed on thehost computer by means oftest scripts.
• The QUTest approach is moreintuitive for embedded developers, because it is conceptually like automated "debugging by printf" that most embedded developers use extensively. This approach also simplifies the development of all sorts oftest doubles↑, includingmocks↑,without breaking encapsulation of the CUT.
• QUTest is a unique test harness on the embedded market that supportsscripting. QUTesttest scripts run exclusively on the host, which skips compiling and uploading the code to the Target and thus shortens the testing micro-cycle.

Note

QUTest supportstest scripts written inPython (3.3+).

• QUTest supportsresetting the Target for each test, if needed. This goes far beyond providing testsetup() andteardown() functions that other test fixtures offer (and of course QUTest supports as well). A clean reset of the Target avoids erroneous tests that implicitly rely on side effects from previously executed code. This is particularly important for embedded systems and for state machines, so that each test can start from a known reset condition.
• QUTest supportstesting Design by Contract (assertions in C or C++, not to be confused with "test assertions") in the CUT. This is a carefully designed, unique feature of QUTest not available in other test harnesses. A successful test of DbC might actually mean breaking an assertion in the Target code.
• QUTesttest fixtures that run on the Targetdo not require dynamic memory allocation (malloc()/free() in C ornew/delete in C++). This means that you don't need to commit any of your precious embedded RAM to the heap (you can set the heap size to zero), and you don't need to link the heap management code. Avoiding dynamic memory allocation is one of the best practices of real-time embedded programming, which you don't need to compromise to run QUTest.
• QUTesttest fixtures that run on the Targetdo not require non-local jumps (setjmp()()/longjmp() in C orthrow/catch in C++), which are needed by other test harnesses to discontinue failing tests. QUTesttest fixtures do not need to discontinue failing tests, because they don't check test expectations ("test assertions"), so atest fixture does not "know" if it is failing or passing. Should a test fixture crash on the Target, it simply waits for the target reset commanded by atest script.
• QUTesttest fixtures can be based on the actualapplication code. For example, you can reuse the samemain() function in atest fixture and in your final application. This means that you can either grow yourtest fixture into a final application through TDD, or you can more easily add unit or integration tests to an existing application.

Note

Even though QUTest is particularly suitable for running tests on deeply embedded targets, it also fully supports runningthe same tests on yourhost computer (Windows, Linux, and macOS are supported). In fact, running the tests as much as possible on the host and thus avoiding the target hardware bottleneck is the highly recommended best practice of embedded TDD. QUTest supportsfully-automated testing, both on the embedded target and on the host computer.

Installation & Use