KCOV: code coverage for fuzzing¶
KCOV collects and exposes kernel code coverage information in a form suitablefor coverage-guided fuzzing. Coverage data of a running kernel is exported viathekcov debugfs file. Coverage collection is enabled on a task basis, andthus KCOV can capture precise coverage of a single system call.
Note that KCOV does not aim to collect as much coverage as possible. It aimsto collect more or less stable coverage that is a function of syscall inputs.To achieve this goal, it does not collect coverage in soft/hard interrupts(unless remove coverage collection is enabled, see below) and from someinherently non-deterministic parts of the kernel (e.g. scheduler, locking).
Besides collecting code coverage, KCOV can also collect comparison operands.See the “Comparison operands collection” section for details.
Besides collecting coverage data from syscall handlers, KCOV can also collectcoverage for annotated parts of the kernel executing in background kerneltasks or soft interrupts. See the “Remote coverage collection” section fordetails.
Prerequisites¶
KCOV relies on compiler instrumentation and requires GCC 6.1.0 or lateror any Clang version supported by the kernel.
Collecting comparison operands is supported with GCC 8+ or with Clang.
To enable KCOV, configure the kernel with:
CONFIG_KCOV=y
To enable comparison operands collection, set:
CONFIG_KCOV_ENABLE_COMPARISONS=y
Coverage data only becomes accessible once debugfs has been mounted:
mount -t debugfs none /sys/kernel/debug
Coverage collection¶
The following program demonstrates how to use KCOV to collect coverage for asingle syscall from within a test program:
#include<stdio.h>#include<stddef.h>#include<stdint.h>#include<stdlib.h>#include<sys/types.h>#include<sys/stat.h>#include<sys/ioctl.h>#include<sys/mman.h>#include<unistd.h>#include<fcntl.h>#include<linux/types.h>#define KCOV_INIT_TRACE _IOR('c', 1, unsigned long)#define KCOV_ENABLE _IO('c', 100)#define KCOV_DISABLE _IO('c', 101)#define COVER_SIZE (64<<10)#define KCOV_TRACE_PC 0#define KCOV_TRACE_CMP 1intmain(intargc,char**argv){intfd;unsignedlong*cover,n,i;/* A single fd descriptor allows coverage collection on a single * thread. */fd=open("/sys/kernel/debug/kcov",O_RDWR);if(fd==-1)perror("open"),exit(1);/* Setup trace mode and trace size. */if(ioctl(fd,KCOV_INIT_TRACE,COVER_SIZE))perror("ioctl"),exit(1);/* Mmap buffer shared between kernel- and user-space. */cover=(unsignedlong*)mmap(NULL,COVER_SIZE*sizeof(unsignedlong),PROT_READ|PROT_WRITE,MAP_SHARED,fd,0);if((void*)cover==MAP_FAILED)perror("mmap"),exit(1);/* Enable coverage collection on the current thread. */if(ioctl(fd,KCOV_ENABLE,KCOV_TRACE_PC))perror("ioctl"),exit(1);/* Reset coverage from the tail of the ioctl() call. */__atomic_store_n(&cover[0],0,__ATOMIC_RELAXED);/* Call the target syscall call. */read(-1,NULL,0);/* Read number of PCs collected. */n=__atomic_load_n(&cover[0],__ATOMIC_RELAXED);for(i=0;i<n;i++)printf("0x%lx\n",cover[i+1]);/* Disable coverage collection for the current thread. After this call * coverage can be enabled for a different thread. */if(ioctl(fd,KCOV_DISABLE,0))perror("ioctl"),exit(1);/* Free resources. */if(munmap(cover,COVER_SIZE*sizeof(unsignedlong)))perror("munmap"),exit(1);if(close(fd))perror("close"),exit(1);return0;}
After piping throughaddr2line the output of the program looks as follows:
SyS_readfs/read_write.c:562__fdget_posfs/file.c:774__fget_lightfs/file.c:746__fget_lightfs/file.c:750__fget_lightfs/file.c:760__fdget_posfs/file.c:784SyS_readfs/read_write.c:562
If a program needs to collect coverage from several threads (independently),it needs to open/sys/kernel/debug/kcov in each thread separately.
The interface is fine-grained to allow efficient forking of test processes.That is, a parent process opens/sys/kernel/debug/kcov, enables trace mode,mmaps coverage buffer, and then forks child processes in a loop. The childprocesses only need to enable coverage (it gets disabled automatically whena thread exits).
Comparison operands collection¶
Comparison operands collection is similar to coverage collection:
/* Same includes and defines as above. *//* Number of 64-bit words per record. */#define KCOV_WORDS_PER_CMP 4/* * The format for the types of collected comparisons. * * Bit 0 shows whether one of the arguments is a compile-time constant. * Bits 1 & 2 contain log2 of the argument size, up to 8 bytes. */#define KCOV_CMP_CONST (1 << 0)#define KCOV_CMP_SIZE(n) ((n) << 1)#define KCOV_CMP_MASK KCOV_CMP_SIZE(3)intmain(intargc,char**argv){intfd;uint64_t*cover,type,arg1,arg2,is_const,size;unsignedlongn,i;fd=open("/sys/kernel/debug/kcov",O_RDWR);if(fd==-1)perror("open"),exit(1);if(ioctl(fd,KCOV_INIT_TRACE,COVER_SIZE))perror("ioctl"),exit(1);/* * Note that the buffer pointer is of type uint64_t*, because all * the comparison operands are promoted to uint64_t. */cover=(uint64_t*)mmap(NULL,COVER_SIZE*sizeof(unsignedlong),PROT_READ|PROT_WRITE,MAP_SHARED,fd,0);if((void*)cover==MAP_FAILED)perror("mmap"),exit(1);/* Note KCOV_TRACE_CMP instead of KCOV_TRACE_PC. */if(ioctl(fd,KCOV_ENABLE,KCOV_TRACE_CMP))perror("ioctl"),exit(1);__atomic_store_n(&cover[0],0,__ATOMIC_RELAXED);read(-1,NULL,0);/* Read number of comparisons collected. */n=__atomic_load_n(&cover[0],__ATOMIC_RELAXED);for(i=0;i<n;i++){uint64_tip;type=cover[i*KCOV_WORDS_PER_CMP+1];/* arg1 and arg2 - operands of the comparison. */arg1=cover[i*KCOV_WORDS_PER_CMP+2];arg2=cover[i*KCOV_WORDS_PER_CMP+3];/* ip - caller address. */ip=cover[i*KCOV_WORDS_PER_CMP+4];/* size of the operands. */size=1<<((type&KCOV_CMP_MASK)>>1);/* is_const - true if either operand is a compile-time constant.*/is_const=type&KCOV_CMP_CONST;printf("ip: 0x%lx type: 0x%lx, arg1: 0x%lx, arg2: 0x%lx, ""size: %lu, %s\n",ip,type,arg1,arg2,size,is_const?"const":"non-const");}if(ioctl(fd,KCOV_DISABLE,0))perror("ioctl"),exit(1);/* Free resources. */if(munmap(cover,COVER_SIZE*sizeof(unsignedlong)))perror("munmap"),exit(1);if(close(fd))perror("close"),exit(1);return0;}
Note that the KCOV modes (collection of code coverage or comparison operands)are mutually exclusive.
Remote coverage collection¶
Besides collecting coverage data from handlers of syscalls issued from auserspace process, KCOV can also collect coverage for parts of the kernelexecuting in other contexts - so-called “remote” coverage.
Using KCOV to collect remote coverage requires:
Modifying kernel code to annotate the code section from where coverageshould be collected with
kcov_remote_startandkcov_remote_stop.Using
KCOV_REMOTE_ENABLEinstead ofKCOV_ENABLEin the userspaceprocess that collects coverage.
Bothkcov_remote_start andkcov_remote_stop annotations and theKCOV_REMOTE_ENABLE ioctl accept handles that identify particular coveragecollection sections. The way a handle is used depends on the context where thematching code section executes.
KCOV supports collecting remote coverage from the following contexts:
Global kernel background tasks. These are the tasks that are spawned duringkernel boot in a limited number of instances (e.g. one USB
hub_eventworker is spawned per one USB HCD).Local kernel background tasks. These are spawned when a userspace processinteracts with some kernel interface and are usually killed when the processexits (e.g. vhost workers).
Soft interrupts.
For #1 and #3, a unique global handle must be chosen and passed to thecorrespondingkcov_remote_start call. Then a userspace process must passthis handle toKCOV_REMOTE_ENABLE in thehandles array field of thekcov_remote_arg struct. This will attach the used KCOV device to the codesection referenced by this handle. Multiple global handles identifyingdifferent code sections can be passed at once.
For #2, the userspace process instead must pass a non-zero handle through thecommon_handle field of thekcov_remote_arg struct. This common handlegets saved to thekcov_handle field in the currenttask_struct andneeds to be passed to the newly spawned local tasks via custom kernel codemodifications. Those tasks should in turn use the passed handle in theirkcov_remote_start andkcov_remote_stop annotations.
KCOV follows a predefined format for both global and common handles. Eachhandle is au64 integer. Currently, only the one top and the lower 4 bytesare used. Bytes 4-7 are reserved and must be zero.
For global handles, the top byte of the handle denotes the id of a subsystemthis handle belongs to. For example, KCOV uses1 as the USB subsystem id.The lower 4 bytes of a global handle denote the id of a task instance withinthat subsystem. For example, eachhub_event worker uses the USB bus numberas the task instance id.
For common handles, a reserved value0 is used as a subsystem id, as suchhandles don’t belong to a particular subsystem. The lower 4 bytes of a commonhandle identify a collective instance of all local tasks spawned by theuserspace process that passed a common handle toKCOV_REMOTE_ENABLE.
In practice, any value can be used for common handle instance id if coverageis only collected from a single userspace process on the system. However, ifcommon handles are used by multiple processes, unique instance ids must beused for each process. One option is to use the process id as the commonhandle instance id.
The following program demonstrates using KCOV to collect coverage from bothlocal tasks spawned by the process and the global task that handles USB bus #1:
/* Same includes and defines as above. */structkcov_remote_arg{__u32trace_mode;__u32area_size;__u32num_handles;__aligned_u64common_handle;__aligned_u64handles[0];};#define KCOV_INIT_TRACE _IOR('c', 1, unsigned long)#define KCOV_DISABLE _IO('c', 101)#define KCOV_REMOTE_ENABLE _IOW('c', 102, struct kcov_remote_arg)#define COVER_SIZE (64 << 10)#define KCOV_TRACE_PC 0#define KCOV_SUBSYSTEM_COMMON (0x00ull << 56)#define KCOV_SUBSYSTEM_USB (0x01ull << 56)#define KCOV_SUBSYSTEM_MASK (0xffull << 56)#define KCOV_INSTANCE_MASK (0xffffffffull)staticinline__u64kcov_remote_handle(__u64subsys,__u64inst){if(subsys&~KCOV_SUBSYSTEM_MASK||inst&~KCOV_INSTANCE_MASK)return0;returnsubsys|inst;}#define KCOV_COMMON_ID 0x42#define KCOV_USB_BUS_NUM 1intmain(intargc,char**argv){intfd;unsignedlong*cover,n,i;structkcov_remote_arg*arg;fd=open("/sys/kernel/debug/kcov",O_RDWR);if(fd==-1)perror("open"),exit(1);if(ioctl(fd,KCOV_INIT_TRACE,COVER_SIZE))perror("ioctl"),exit(1);cover=(unsignedlong*)mmap(NULL,COVER_SIZE*sizeof(unsignedlong),PROT_READ|PROT_WRITE,MAP_SHARED,fd,0);if((void*)cover==MAP_FAILED)perror("mmap"),exit(1);/* Enable coverage collection via common handle and from USB bus #1. */arg=calloc(1,sizeof(*arg)+sizeof(uint64_t));if(!arg)perror("calloc"),exit(1);arg->trace_mode=KCOV_TRACE_PC;arg->area_size=COVER_SIZE;arg->num_handles=1;arg->common_handle=kcov_remote_handle(KCOV_SUBSYSTEM_COMMON,KCOV_COMMON_ID);arg->handles[0]=kcov_remote_handle(KCOV_SUBSYSTEM_USB,KCOV_USB_BUS_NUM);if(ioctl(fd,KCOV_REMOTE_ENABLE,arg))perror("ioctl"),free(arg),exit(1);free(arg);/* * Here the user needs to trigger execution of a kernel code section * that is either annotated with the common handle, or to trigger some * activity on USB bus #1. */sleep(2);/* * The load to the coverage count should be an acquire to pair with * pair with the corresponding write memory barrier (smp_wmb()) on * the kernel-side in kcov_move_area(). */n=__atomic_load_n(&cover[0],__ATOMIC_ACQUIRE);for(i=0;i<n;i++)printf("0x%lx\n",cover[i+1]);if(ioctl(fd,KCOV_DISABLE,0))perror("ioctl"),exit(1);if(munmap(cover,COVER_SIZE*sizeof(unsignedlong)))perror("munmap"),exit(1);if(close(fd))perror("close"),exit(1);return0;}