Pressure interface¶

Pressure information for each resource is exported through therespective file in /proc/pressure/ – cpu, memory, and io.

The format for CPU is as such:

some avg10=0.00 avg60=0.00 avg300=0.00 total=0

and for memory and IO:

some avg10=0.00 avg60=0.00 avg300=0.00 total=0full avg10=0.00 avg60=0.00 avg300=0.00 total=0

The “some” line indicates the share of time in which at least sometasks are stalled on a given resource.

The “full” line indicates the share of time in which all non-idletasks are stalled on a given resource simultaneously. In this stateactual CPU cycles are going to waste, and a workload that spendsextended time in this state is considered to be thrashing. This hassevere impact on performance, and it’s useful to distinguish thissituation from a state where some tasks are stalled but the CPU isstill doing productive work. As such, time spent in this subset of thestall state is tracked separately and exported in the “full” averages.

The ratios (in %) are tracked as recent trends over ten, sixty, andthree hundred second windows, which gives insight into short term eventsas well as medium and long term trends. The total absolute stall time(in us) is tracked and exported as well, to allow detection of latencyspikes which wouldn’t necessarily make a dent in the time averages,or to average trends over custom time frames.

Monitoring for pressure thresholds¶

Users can register triggers and use poll() to be woken up when resourcepressure exceeds certain thresholds.

A trigger describes the maximum cumulative stall time over a specifictime window, e.g. 100ms of total stall time within any 500ms window togenerate a wakeup event.

To register a trigger user has to open psi interface file under/proc/pressure/ representing the resource to be monitored and write thedesired threshold and time window. The open file descriptor should beused to wait for trigger events using select(), poll() or epoll().The following format is used:

<some|full> <stall amount in us> <time window in us>

For example writing “some 150000 1000000” into /proc/pressure/memorywould add 150ms threshold for partial memory stall measured within1sec time window. Writing “full 50000 1000000” into /proc/pressure/iowould add 50ms threshold for full io stall measured within 1sec time window.

Triggers can be set on more than one psi metric and more than one triggerfor the same psi metric can be specified. However for each trigger a separatefile descriptor is required to be able to poll it separately from others,therefore for each trigger a separate open() syscall should be made evenwhen opening the same psi interface file.

Monitors activate only when system enters stall state for the monitoredpsi metric and deactivates upon exit from the stall state. While system isin the stall state psi signal growth is monitored at a rate of 10 times pertracking window.

The kernel accepts window sizes ranging from 500ms to 10s, therefore minmonitoring update interval is 50ms and max is 1s. Min limit is set toprevent overly frequent polling. Max limit is chosen as a high enough numberafter which monitors are most likely not needed and psi averages can be usedinstead.

When activated, psi monitor stays active for at least the duration of onetracking window to avoid repeated activations/deactivations when system isbouncing in and out of the stall state.

Notifications to the userspace are rate-limited to one per tracking window.

The trigger will de-register when the file descriptor used to define thetrigger is closed.

Userspace monitor usage example¶

#include <errno.h>#include <fcntl.h>#include <stdio.h>#include <poll.h>#include <string.h>#include <unistd.h>/* * Monitor memory partial stall with 1s tracking window size * and 150ms threshold. */int main() {      const char trig[] = "some 150000 1000000";      struct pollfd fds;      int n;      fds.fd = open("/proc/pressure/memory", O_RDWR | O_NONBLOCK);      if (fds.fd < 0) {              printf("/proc/pressure/memory open error: %s\n",                      strerror(errno));              return 1;      }      fds.events = POLLPRI;      if (write(fds.fd, trig, strlen(trig) + 1) < 0) {              printf("/proc/pressure/memory write error: %s\n",                      strerror(errno));              return 1;      }      printf("waiting for events...\n");      while (1) {              n = poll(&fds, 1, -1);              if (n < 0) {                      printf("poll error: %s\n", strerror(errno));                      return 1;              }              if (fds.revents & POLLERR) {                      printf("got POLLERR, event source is gone\n");                      return 0;              }              if (fds.revents & POLLPRI) {                      printf("event triggered!\n");              } else {                      printf("unknown event received: 0x%x\n", fds.revents);                      return 1;              }      }      return 0;}

Cgroup2 interface¶

In a system with a CONFIG_CGROUP=y kernel and the cgroup2 filesystemmounted, pressure stall information is also tracked for tasks groupedinto cgroups. Each subdirectory in the cgroupfs mountpoint containscpu.pressure, memory.pressure, and io.pressure files; the format isthe same as the /proc/pressure/ files.

Per-cgroup psi monitors can be specified and used the same way assystem-wide ones.