Key Points¶

• Understanding system resources necessary to build and run a workloadis important.

• Linux tracing and strace can be used to discover the system resourcesin use by a workload. The completeness of the system usage informationdepends on the completeness of coverage of a workload.

• Performance and security of the operating system can be analyzed withthe help of tools such as:perf,stress-ng,paxtest.

• Once we discover and understand the workload needs, we can focus on themto avoid regressions and use it to evaluate safety considerations.

Methodology¶

strace is adiagnostic, instructional, and debugging tool and can be used to discoverthe system resources in use by a workload. Once we discover and understandthe workload needs, we can focus on them to avoid regressions and use itto evaluate safety considerations. We use strace tool to trace workloads.

This method of tracing using strace tells us the system calls invoked bythe workload and doesn’t include all the system calls that can be invokedby it. In addition, this tracing method tells us just the code paths withinthese system calls that are invoked. As an example, if a workload opens afile and reads from it successfully, then the success path is the one thatis traced. Any error paths in that system call will not be traced. If thereis a workload that provides full coverage of a workload then the methodoutlined here will trace and find all possible code paths. The completenessof the system usage information depends on the completeness of coverage of aworkload.

The goal is tracing a workload on a system running a default kernel withoutrequiring custom kernel installs.

How do we gather fine-grained system information?¶

strace tool can be used to trace system calls made by a process and signalsit receives. System calls are the fundamental interface between anapplication and the operating system kernel. They enable a program torequest services from the kernel. For instance, the open() system call inLinux is used to provide access to a file in the file system. strace enablesus to track all the system calls made by an application. It lists all thesystem calls made by a process and their resulting output.

You can generate profiling data combining strace and perf record tools torecord the events and information associated with a process. This providesinsight into the process. “perf annotate” tool generates the statistics ofeach instruction of the program. This document goes over the details of howto gather fine-grained information on a workload’s usage of system resources.

We used strace to trace the perf, stress-ng, paxtest workloads to illustrateour methodology to discover resources used by a workload. This process canbe applied to trace other workloads.

Getting the system ready for tracing¶

Before we can get started we will show you how to get your system ready.We assume that you have a Linux distribution running on a physical systemor a virtual machine. Most distributions will include strace command. Let’sinstall other tools that aren’t usually included to build Linux kernel.Please note that the following works on Debian based distributions. Youmight have to find equivalent packages on other Linux distributions.

Install tools to build Linux kernel and tools in kernel repository.scripts/ver_linux is a good way to check if your system already hasthe necessary tools:

sudo apt-get install build-essential flex bison yaccsudo apt install libelf-dev systemtap-sdt-dev libslang2-dev libperl-dev libdw-dev

cscope is a good tool to browse kernel sources. Let’s install it now:

sudo apt-get install cscope

Install stress-ng and paxtest:

apt-get install stress-ngapt-get install paxtest

Workload overview¶

As mentioned earlier, we used strace to trace perf bench, stress-ng andpaxtest workloads to show how to analyze a workload and identify Linuxsubsystems used by these workloads. Let’s start with an overview of thesethree workloads to get a better understanding of what they do and how touse them.

What is strace and how do we use it?¶

As mentioned earlier, strace which is a useful diagnostic, instructional,and debugging tool and can be used to discover the system resources in useby a workload. It can be used:

• To see how a process interacts with the kernel.

• To see why a process is failing or hanging.

• For reverse engineering a process.

• To find the files on which a program depends.

• For analyzing the performance of an application.

• For troubleshooting various problems related to the operating system.

In addition, strace can generate run-time statistics on times, calls, anderrors for each system call and report a summary when program exits,suppressing the regular output. This attempts to show system time (CPU timespent running in the kernel) independent of wall clock time. We plan to usethese features to get information on workload system usage.

strace command supports basic, verbose, and stats modes. strace command whenrun in verbose mode gives more detailed information about the system callsinvoked by a process.

Running strace -c generates a report of the percentage of time spent in eachsystem call, the total time in seconds, the microseconds per call, the totalnumber of calls, the count of each system call that has failed with an errorand the type of system call made.

• Usage: strace <command we want to trace>

• Verbose mode usage: strace -v <command>

• Gather statistics: strace -c <command>

We used the “-c” option to gather fine-grained run-time statistics in useby three workloads we have chose for this analysis.

• perf

• stress-ng

• paxtest

What is cscope and how do we use it?¶

Now let’s look atcscope, a commandline tool for browsing C, C++ or Java code-bases. We can use it to findall the references to a symbol, global definitions, functions called by afunction, functions calling a function, text strings, regular expressionpatterns, files including a file.

We can use cscope to find which system call belongs to which subsystem.This way we can find the kernel subsystems used by a process when it isexecuted.

Let’s checkout the latest Linux repository and build cscope database:

git clone git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git linuxcd linuxcscope -R -p10  # builds cscope.out database before starting browse sessioncscope -d -p10  # starts browse session on cscope.out database

Note: Run “cscope -R -p10” to build the database and c”scope -d -p10” toenter into the browsing session. cscope by default cscope.out database.To get out of this mode press ctrl+d. -p option is used to specify thenumber of file path components to display. -p10 is optimal for browsingkernel sources.

What is perf and how do we use it?¶

Perf is an analysis tool based on Linux 2.6+ systems, which abstracts theCPU hardware difference in performance measurement in Linux, and providesa simple command line interface. Perf is based on the perf_events interfaceexported by the kernel. It is very useful for profiling the system andfinding performance bottlenecks in an application.

If you haven’t already checked out the Linux mainline repository, you can doso and then build kernel and perf tool:

git clone git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git linuxcd linuxmake -j3 allcd tools/perfmake

Note: The perf command can be built without building the kernel in therepository and can be run on older kernels. However matching the kerneland perf revisions gives more accurate information on the subsystem usage.

We used “perf stat” and “perf bench” options. For a detailed information onthe perf tool, run “perf -h”.

What is stress-ng and how do we use it?¶

As mentioned earlier, stress-ng is used for performing stress testing onthe kernel. It allows you to exercise various physical subsystems of thecomputer, as well as interfaces of the OS kernel, using stressor-s. Theyare available for CPU, CPU cache, devices, I/O, interrupts, file system,memory, network, operating system, pipelines, schedulers, and virtualmachines.

The netdev stressor starts N workers that exercise various netdevice ioctlcommands across all the available network devices. The following ioctls areexercised:

• SIOCGIFCONF, SIOCGIFINDEX, SIOCGIFNAME, SIOCGIFFLAGS

• SIOCGIFADDR, SIOCGIFNETMASK, SIOCGIFMETRIC, SIOCGIFMTU

• SIOCGIFHWADDR, SIOCGIFMAP, SIOCGIFTXQLEN

The following command runs the stressor:

stress-ng --netdev 1 -t 60 --metrics command.

We can use the perf record command to record the events and informationassociated with a process. This command records the profiling data in theperf.data file in the same directory.

Using the following commands you can record the events associated with thenetdev stressor, view the generated report perf.data and annotate the toview the statistics of each instruction of the program:

perf record stress-ng --netdev 1 -t 60 --metrics command.perf reportperf annotate

What is paxtest and how do we use it?¶

paxtest is a program that tests buffer overflows in the kernel. It testskernel enforcements over memory usage. Generally, execution in some memorysegments makes buffer overflows possible. It runs a set of programs thatattempt to subvert memory usage. It is used as a regression test suite forPaX, and will be useful to test other memory protection patches for thekernel.

paxtest provides kiddie and blackhat modes. The paxtest kiddie mode runsin normal mode, whereas the blackhat mode tries to get around the protectionof the kernel testing for vulnerabilities. We focus on the kiddie mode hereand combine “paxtest kiddie” run with “perf record” to collect CPU stacktraces for the paxtest kiddie run to see which function is calling otherfunctions in the performance profile. Then the “dwarf” (DWARF’s Call FrameInformation) mode can be used to unwind the stack.

The following command can be used to view resulting report in call-graphformat:

perf record --call-graph dwarf paxtest kiddieperf report --stdio

Tracing workloads¶

Now that we understand the workloads, let’s start tracing them.

strace -c perf bench all

strace -c  stress-ng --netdev 1 -t 60 --metrics

strace -c paxtest kiddie

Conclusion¶

This document is intended to be used as a guide on how to gather fine-grainedinformation on the resources in use by workloads using strace.

References¶

• Discovery Linux Kernel Subsystems used by OpenAPS

• ELISA-White-Papers-Discovering Linux kernel subsystems used by a workload

• strace

• perf

• paxtest README

• stress-ng

• Monitoring and managing system status and performance