English

Kernel Testing Guide

There are a number of different tools for testing the Linux kernel, so knowingwhen to use each of them can be a challenge. This document provides a roughoverview of their differences, and how they fit together.

Writing and Running Tests

The bulk of kernel tests are written using either the kselftest or KUnitframeworks. These both provide infrastructure to help make running tests andgroups of tests easier, as well as providing helpers to aid in writing newtests.

If you’re looking to verify the behaviour of the Kernel — particularly specificparts of the kernel — then you’ll want to use KUnit or kselftest.

The Difference Between KUnit and kselftest

KUnit (KUnit - Linux Kernel Unit Testing) is an entirely in-kernel systemfor “white box” testing: because test code is part of the kernel, it can accessinternal structures and functions which aren’t exposed to userspace.

KUnit tests therefore are best written against small, self-contained partsof the kernel, which can be tested in isolation. This aligns well with theconcept of ‘unit’ testing.

For example, a KUnit test might test an individual kernel function (or even asingle codepath through a function, such as an error handling case), ratherthan a feature as a whole.

This also makes KUnit tests very fast to build and run, allowing them to berun frequently as part of the development process.

There is a KUnit test style guide which may give further pointers inTest Style and Nomenclature

kselftest (Linux Kernel Selftests), on the other hand, islargely implemented in userspace, and tests are normal userspace scripts orprograms.

This makes it easier to write more complicated tests, or tests which need tomanipulate the overall system state more (e.g., spawning processes, etc.).However, it’s not possible to call kernel functions directly from kselftest.This means that only kernel functionality which is exposed to userspace somehow(e.g. by a syscall, device, filesystem, etc.) can be tested with kselftest. Towork around this, some tests include a companion kernel module which exposesmore information or functionality. If a test runs mostly or entirely within thekernel, however, KUnit may be the more appropriate tool.

kselftest is therefore suited well to tests of whole features, as these willexpose an interface to userspace, which can be tested, but not implementationdetails. This aligns well with ‘system’ or ‘end-to-end’ testing.

For example, all new system calls should be accompanied by kselftest tests.

Code Coverage Tools

The Linux Kernel supports two different code coverage measurement tools. Thesecan be used to verify that a test is executing particular functions or linesof code. This is useful for determining how much of the kernel is being tested,and for finding corner-cases which are not covered by the appropriate test.

Using gcov with the Linux kernel is GCC’s coverage testing tool, which can beused with the kernel to get global or per-module coverage. Unlike KCOV, itdoes not record per-task coverage. Coverage data can be read from debugfs,and interpreted using the usual gcov tooling.

KCOV: code coverage for fuzzing is a feature which can be built in to thekernel to allow capturing coverage on a per-task level. It’s therefore usefulfor fuzzing and other situations where information about code executed during,for example, a single syscall is useful.

Dynamic Analysis Tools

The kernel also supports a number of dynamic analysis tools, which attempt todetect classes of issues when they occur in a running kernel. These typicallyeach look for a different class of bugs, such as invalid memory accesses,concurrency issues such as data races, or other undefined behaviour likeinteger overflows.

Some of these tools are listed below:

These tools tend to test the kernel as a whole, and do not “pass” likekselftest or KUnit tests. They can be combined with KUnit or kselftest byrunning tests on a kernel with these tools enabled: you can then be surethat none of these errors are occurring during the test.

Some of these tools integrate with KUnit or kselftest and willautomatically fail tests if an issue is detected.

Static Analysis Tools

In addition to testing a running kernel, one can also analyze kernel source codedirectly (at compile time) usingstatic analysis tools. The toolscommonly used in the kernel allow one to inspect the whole source tree or justspecific files within it. They make it easier to detect and fix problems duringthe development process.

Sparse can help test the kernel by performing type-checking, lock checking,value range checking, in addition to reporting various errors and warnings whileexamining the code. See theSparse documentationpage for details on how to use it.

Smatch extends Sparse and provides additional checks for programming logicmistakes such as missing breaks in switch statements, unused return values onerror checking, forgetting to set an error code in the return of an error path,etc. Smatch also has tests against more serious issues such as integeroverflows, null pointer dereferences, and memory leaks. See the project page athttp://smatch.sourceforge.net/.

Coccinelle is another static analyzer at our disposal. Coccinelle is often usedto aid refactoring and collateral evolution of source code, but it can also helpto avoid certain bugs that occur in common code patterns. The types of testsavailable include API tests, tests for correct usage of kernel iterators, checksfor the soundness of free operations, analysis of locking behavior, and furthertests known to help keep consistent kernel usage. See theCoccinelle documentation page for details.

Beware, though, that static analysis tools suffer fromfalse positives.Errors and warns need to be evaluated carefully before attempting to fix them.

When to use Sparse and Smatch

Sparse does type checking, such as verifying that annotated variables do notcause endianness bugs, detecting places that use__user pointers improperly,and analyzing the compatibility of symbol initializers.

Smatch does flow analysis and, if allowed to build the function database, italso does cross function analysis. Smatch tries to answer questions like whereis this buffer allocated? How big is it? Can this index be controlled by theuser? Is this variable larger than that variable?

It’s generally easier to write checks in Smatch than it is to write checks inSparse. Nevertheless, there are some overlaps between Sparse and Smatch checks.

Strong points of Smatch and Coccinelle

Coccinelle is probably the easiest for writing checks. It works before thepre-processor so it’s easier to check for bugs in macros using Coccinelle.Coccinelle also creates patches for you, which no other tool does.

For example, with Coccinelle you can do a mass conversion fromkmalloc(x*size,GFP_KERNEL) tokmalloc_array(x,size,GFP_KERNEL), andthat’s really useful. If you just created a Smatch warning and try to push thework of converting on to the maintainers they would be annoyed. You’d have toargue about each warning if can really overflow or not.

Coccinelle does no analysis of variable values, which is the strong point ofSmatch. On the other hand, Coccinelle allows you to do simple things in a simpleway.