Repository files navigation

Install the compilers needed to build pandas in the system:

apt install gcc g++

Create a user to run the benchmarks, and clone this repository in its home.

Installpixi, which we use to manage the environment that runsasv. Note that the the environment to run the benchmarks is managed by asv and it isdifferent from the pixi environment:

curl -fsSL https://pixi.sh/install.sh| bash

Clone the pandas repository inside thepandas-benchmarks directory:

cd pandas-benchmarksgit clone https://github.com/pandas-dev/pandas.git

We usepixi to manage the environment and run the benchmarks:

pixi run bench

We may want to implement a script that runs benchmarks continually (a new run startswhen the previous finishes, indefinetly). But for now we are using cron.

To set up cron to run the benchmarks automatically we can use:

0 */3 * * * cd pandas-benchmarks && /home/bench/.pixi/bin/pixi run bench >> bench.log 2>&1

Note that the frequency should avoid starting a new job when the previoushas not finished, so if the benchmarks take 2.5 hours to complete, we shouldschedule the runs to for example every 3 hours.

To view the log of cron executions we can run:

grep CRON /var/log/syslog| grep"(bench)"

Everything that happens in the system while running the benchmarks causes animpact, meaning that benchmarks will run faster when there is not much noise,and will run slower when there is. For example, if the core running the benchmarkstakes care of an operating system interruption, this will cause a context switch,will flush the CPU caches, and the benchmark will take longer. Even if everybenchmark is run multiple times, this variance makes our results worse and likelyto cause false positives. This section is about trying to make the system morestable and reduce the variance of the execution time of benchmarks.

First thing we can do is to isolate the CPUs where the benchmarks run. This meansthat the operating system won't use the CPU unless a process is explicitly startedwith a CPU affinity to that core.

First, to check the cores available in the system we can run:

$ lscpu --all --extendedCPU NODE SOCKET CORE L1d:L1i:L2:L3 ONLINE    MAXMHZ   MINMHZ       MHZ  0    0      0    0 0:0:0:0          yes 4900.0000 800.0000 4798.3130  1    0      0    1 1:1:1:0          yes 4900.0000 800.0000 4603.2891  2    0      0    2 2:2:2:0          yes 4900.0000 800.0000 4000.0000  3    0      0    3 3:3:3:0          yes 4900.0000 800.0000 4000.0000  4    0      0    0 0:0:0:0          yes 4900.0000 800.0000 4000.0000  5    0      0    1 1:1:1:0          yes 4900.0000 800.0000 4000.0000  6    0      0    2 2:2:2:0          yes 4900.0000 800.0000 4782.7388  7    0      0    3 3:3:3:0          yes 4900.0000 800.0000 4000.0000

TheCPU column shows that the benchmarks server has 8 cores, and theCOREcolumn shows that those are using 4 different physical cores (every physicalcore is used by two separate pipelines or virtual cores, referred by Intelas hyperthreads). We need to isolate physical cores, so the OS does notexecute anything in the other pipeline either, which would also slow downthe benchmark execution.

To isolate CPUs we need to add parameters to the kernel. To do so, we editthe file/etc/default/grub and do these changes:

# Find this line:GRUB_CMDLINE_LINUX_DEFAULT="quiet splash"# Replace it with this line (add the parameters at the end):GRUB_CMDLINE_LINUX_DEFAULT="quiet splash isolcpus=3,7 nohz_full=3,7"

This will isolate the physical core 3, via its two virtual cores 3 and 7.It will also remove these cores from the operating system scheduler ticks.We can surely isolate more cores, for now we just start by one for simplicity.

For the changes to have an effect we first need to update the actual grubconfiguration with the changes in/etc/default/grub.d/50-cloudimg-settings.cfg.In general/etc/default/grub is used for grub settings, but OVH overwrites thecontent of that file with50-cloudimg-settings.cfg. Note that grub does not readdirectly from those files, so it is needed to executeupdate-grub orgrub-mkconfigwhich parse these files and write to/boot/grub.grub.cfg which is the one used bythe operating system. After executing one of those commands it is needed to restartthe system so the running kernel contains the new parameters. In practice this is assimple as tuning the next commands

$ sudo vim /etc/default/grub.d/50-cloudimg-settings.cfg# and make changes above$ sudo update-grub$ sudo reboot

Once the system is restarted we should check that the CPUs are indeedisolated as expected. This can be done checking the information in thenext files:

$ cat /sys/devices/system/cpu/isolated3,7

We can also see that the operating system is not running tasks in the isolated CPUsby generating process and checking CPU usage with htop:

$ apt install stress$ stress --cpu 8$ htop# in a different terminal

Isolation works for processes running in the user space, but not in the system space.Ideally, we would like to avoid interruptions running in our isolated kernel. Whilethis is a complex topic, and not all intererruptions can run in any core, to limit thenumber of cores every interruption runs in a general way, this command can be used:

forIRQ_AFFINITY_FILEin$(find. -name smp_affinity);doecho 77| sudo tee$IRQ_AFFINITY_FILE;done

Note that for some interruptions the command will fail. Also note that77 is a binarymask in hexadecimal representing0111 0111 (4th and 8th CPUs are not allowed to run theinterruption).

Modern CPUs are able to scale their frequency depending on work load or temperature. When a CPUis idle it will decrease its frequency to save energy. Also, when a CPU is busy and its temperatureincreases, it will eventually decrease its frequency so the temperature goes back to safe level.

Most of these frequency scaling technologies can be disabled via the system BIOS, but we do nothave control of it in the servers in a data center, and disabling them may make frequency slow, andthe benchmark suite take much longer to run (something like double the time based on past tests).

There are some things we have control of at runtime. We should be able to disable TurboBoost via:

echo 1| sudo tee /sys/devices/system/cpu/intel_pstate/no_turbo

We can also installcpufreq which gives informations and allow to control certain features with:

sudo apt install linux-tools-generic

While the system introduces noise to due to CPU scaling or our benchmark process being interruptedby other processes and interruptions, there are other sources of noise that cause variance in theresults of our benchmarks.

The main ones identifies are:

• I/O operations
• Unpredictable CPU cache misses
• Randomness (for example, our benchmarks on functions that check duplicates are affected by therandomness in the hashing functions for the used hash tables).