Repository files navigation

LTP Next-Gen runner is a new version of therunltp script used by theLinux Test Project.

Host information    System: Linux    Node: susy    Kernel Release: 5.14.21-150400.24.33-default    Kernel Version: #1 SMP PREEMPT_DYNAMIC Fri Nov 4 13:55:06 UTC 2022 (76cfe60)    Machine Architecture: x86_64    Processor: x86_64    Temporary directory: /tmp/runltp.acer/tmpcwtket0mConnecting to SUT: hostDownloading suite: mathStarting suite: mathabs01: pass | tained  (0.005s)atof01: pass | tained  (0.005s)float_bessel: pass | tained  (0.702s)float_exp_log: pass | tained  (0.703s)float_iperb: pass | tained  (0.288s)float_power: pass | tained  (0.540s)float_trigo: pass | tained  (0.643s)fptest01: pass | tained  (0.020s)fptest02: pass | tained  (0.005s)nextafter01: pass | tained  (0.004s)Suite Name: mathTotal Run: 10Elapsed Time: 2.9 secondsPassed Tests: 22Failed Tests: 0Skipped Tests: 0Broken Tests: 0Warnings: 0Kernel Version: Linux 5.14.21-150400.24.33-default #1 SMP PREEMPT_DYNAMIC Fri Nov 4 13:55:06 UTC 2022 (76cfe60)CPU: x86_64Machine Architecture: x86_64RAM: 15569564 kBSwap memory: 2095424 kBDistro: opensuse-leapDistro Version: 15.4Disconnecting from SUT: host

Some basic commands are the following:

# run syscalls and dio testing suites on host./runltp-ng --run-suite syscalls dio# run syscalls and dio testing suites in qemu VM./runltp-ng --sut qemu:image=folder/image.qcow2 \    --run-suite syscalls dio# run syscalls and dio testing suites via SSH# NOTE: paramiko and scp packages must be installed in the system./runltp-ng --sut=ssh:host myhost.com:user=root:key_file=myhost_id_rsa \    --run-suite syscalls dio

It's possible to run a single command before running testing suites using--run-cmd option as following:

runltp-ng --run-cmd /mnt/testcases/kernel/systemcalls/bpf/bpf_prog02 \    --sut qemu:image=folder/image.qcow2:virtfs=/home/user/ltp \    --ltp-dir /mnt \    --run-suite syscalls dio

It can be used also to run a single command without running testing suites:

runltp-ng --run-cmd /mnt/testcases/kernel/systemcalls/bpf/bpf_prog02 \    --sut qemu:image=folder/image.qcow2

Every session has a temporary directory which can be found in/<TMPDIR>/runltp-of<username>. Inside this folder there's a symlinkcalledlatest, pointing to the latest session's temporary directory, and theapplication will rotate over 5 sessions.

For more information, checkout the following video at the SUSE Labs Conference2022:

To enable console on a tty device for a VM do:

• open/etc/default/grub
• addconsole=$tty_name, console=tty0 toGRUB_CMDLINE_LINUX
• rungrub-mkconfig -o /boot/grub/grub.cfg

Where$tty_name should bettyS0, unless virtio serial type is used (i.e.if you set theserial=virtio backend option, then usehvc0)

Sometimes we need to cover complex testing scenarios, where the SUT usesparticular protocols and infrastructures, in order to communicate with ourhost machine and to execute tests binaries.

For this reason,runltp-ng provides a plugin system to recognize custom SUTclass implementations inside theltp package folder. Please checkhost.pyorssh.py implementations for more details.

Once a new SUT class is implemented and placed inside theltp package folder,runltp-ng -s help command can be used to see if application correctlyrecognise it.

The application is validated usingpytest andpylint.To run unittests:

pytest

To run linting checks:

pylint --rcfile=pylint.ini ./ltp

The LTPrunltp code is hard to read, maintain and some of its partsare legacy features which are not supported anymore. But if we focus closer onthe results,runltp has done its job for a while since 2001. Nowadays, withnew automation systems, easily accesible virtualization and bigger computingpower,runltp became more and more obsolete, since its main goal was to testLinux Kernel on target and specific distro(s). Let's take a look at the issueswe have:

• it's hard to maintain and it's based on a mixture of bash/C, both hardto read and not maintained anymore
• it contains many features which are not used and they can be deprecated
• report files are custom format logs or HTML files which are both hard to parseinside i.e. an automation system
• if a test causes system crash, which is common for kernel tests, the toolcrashes and we loose most or even all results we obtained before itsexecution. This means we need to run it inside a virtualized system to besure that if system crashes, we won't loose control of the machine. And, inany case, we will loose testing report

The last point is really important, since in a world where cloud and embeddedsystems are having a big market, we need to provide a usable and a stable way totest Linux Kernel. Something thatrunltp is not able to achieve nowadays.

Cyril Hrubis started the first Perl prototype ofrunltp-ng(https://github.com/metan-ucw/runltp-ng/), a next generation tests runner thatallows to run tests on a host, as well as inside a Qemu instance or over a SSH.The tool provided results in a machine parsable format which were easy toconsume by automation systems.However as the community didn't like the choice of Perl programming language wedecided to switch from Perl to Python to take advantage of the Python communitysize, easier maintenance and packages.

In particular, we tried to focus on missing features and got rid of the oneswhich were not strictly needed. We ended up with a simple and light tool havingthe following features:

• test suites can run inside a virtualized system using Qemu or they can beexecuted via SSH protocol
• runner became more robust so it can gracefully handle kernel crashes andtained statuses of the kernel. At the moment, only Qemu supports this feature
• report file type is JSON by default, so it will be easier to parse withexternal tools and automation systems
• the user interface has been simplified, so we have two modes: quiet andverbose mode. The quiet mode is the default one and it shows only tests namesand their results on a list. Verbose mode is similar to the currentrunltpstdout

Nowadaysrunltp-ng is a simple and lightweight implementation that is based onPython 3.6+ and it doesn't have any dependency from external packages.Its skeleton is easy to understand and features can be added easily.

A missing feature that is currently under development is the possibility toexecute tests via LTX (experimental).

LTX is a small service that runs on target and it permits to communicate viamsgpack (https://msgpack.org/) in order to execute binaries on host in thefastest way as possible. Its development is currently maintained by RichardPalethorpe and we plan to make it the default LTP runner in the next future.When LTP metadata file will be completed, LTX will also permit to execute testsin parallel.

By taking in consideration previous topics, we can still provide an usable andsimple tool that can replace currentrunltp script inside the LTP upstream.Its usage is simple, pretty stable and we are starting to move forward intoa modern approach to schedule and run tests in the Linux Testing Project.