Real-time application monitors

Description

Real-time applications may have design flaws such that they experienceunexpected latency and fail to meet their time requirements. Often, these flawsfollow a few patterns:

  • Page faults: A real-time thread may access memory that does not have amapped physical backing or must first be copied (such as for copy-on-write).Thus a page fault is raised and the kernel must first perform the expensiveaction. This causes significant delays to the real-time thread

  • Priority inversion: A real-time thread blocks waiting for a lower-prioritythread. This causes the real-time thread to effectively take on thescheduling priority of the lower-priority thread. For example, the real-timethread needs to access a shared resource that is protected by anon-pi-mutex, but the mutex is currently owned by a non-real-time thread.

Thertapp monitor detects these patterns. It aids developers to identifyreasons for unexpected latency with real-time applications. It is a container ofmultiple sub-monitors described in the following sections.

Monitor pagefault

Thepagefault monitor reports real-time tasks raising page faults. Itsspecification is:

RULE = always (RT imply not PAGEFAULT)

To fix warnings reported by this monitor,mlockall() ormlock() can be usedto ensure physical backing for memory.

This monitor may have false negatives because the pages used by the real-timethreads may just happen to be directly available during testing. To minimizethis, the system can be put under memory pressure (e.g. invoking the OOM killerusing a program that doesptr = malloc(SIZE_OF_RAM); memset(ptr, 0,SIZE_OF_RAM);) so that the kernel executes aggressive strategies to recycle asmuch physical memory as possible.

Monitor sleep

Thesleep monitor reports real-time threads sleeping in a manner that maycause undesirable latency. Real-time applications should only put a real-timethread to sleep for one of the following reasons:

  • Cyclic work: real-time thread sleeps waiting for the next cycle. For thiscase, only theclock_nanosleep syscall should be used withTIMER_ABSTIME(to avoid time drift) andCLOCK_MONOTONIC (to avoid the clock beingchanged). No other method is safe for real-time. For example, threadswaiting for timerfd can be woken by softirq which provides no real-timeguarantee.

  • Real-time thread waiting for something to happen (e.g. another threadreleasing shared resources, or a completion signal from another thread). Inthis case, only futexes (FUTEX_LOCK_PI, FUTEX_LOCK_PI2 or one ofFUTEX_WAIT_*) should be used. Applications usually do not use futexesdirectly, but use PI mutexes and PI condition variables which are built ontop of futexes. Be aware that the C library might not implement conditionalvariables as safe for real-time. As an alternative, the librtpi libraryexists to provide a conditional variable implementation that is correct forreal-time applications in Linux.

Beside the reason for sleeping, the eventual waker should also bereal-time-safe. Namely, one of:

  • An equal-or-higher-priority thread

  • Hard interrupt handler

  • Non-maskable interrupt handler

This monitor’s warning usually means one of the following:

  • Real-time thread is blocked by a non-real-time thread (e.g. due tocontention on a mutex without priority inheritance). This is priorityinversion.

  • Time-critical work waits for something which is not safe for real-time (e.g.timerfd).

  • The work executed by the real-time thread does not need to run at real-timepriority at all. This is not a problem for the real-time thread itself, butit is potentially taking the CPU away from other important real-time work.

Application developers may purposely choose to have their real-time applicationsleep in a way that is not safe for real-time. It is debatable whether that is aproblem. Application developers must analyze the warnings to make a properassessment.

The monitor’s specification is:

RULE = always ((RT and SLEEP) imply (RT_FRIENDLY_SLEEP or ALLOWLIST))RT_FRIENDLY_SLEEP = (RT_VALID_SLEEP_REASON or KERNEL_THREAD)                and ((not WAKE) until RT_FRIENDLY_WAKE)RT_VALID_SLEEP_REASON = FUTEX_WAIT                     or RT_FRIENDLY_NANOSLEEPRT_FRIENDLY_NANOSLEEP = CLOCK_NANOSLEEP                    and NANOSLEEP_TIMER_ABSTIME                    and NANOSLEEP_CLOCK_MONOTONICRT_FRIENDLY_WAKE = WOKEN_BY_EQUAL_OR_HIGHER_PRIO                or WOKEN_BY_HARDIRQ                or WOKEN_BY_NMI                or KTHREAD_SHOULD_STOPALLOWLIST = BLOCK_ON_RT_MUTEX         or FUTEX_LOCK_PI         or TASK_IS_RCU         or TASK_IS_MIGRATION

Beside the scenarios described above, this specification also handle somespecial cases:

  • KERNEL_THREAD: kernel tasks do not have any pattern that can be recognizedas valid real-time sleeping reasons. Therefore sleeping reason is notchecked for kernel tasks.

  • KTHREAD_SHOULD_STOP: a non-real-time thread may stop a real-time kernelthread by waking it and waiting for it to exit (kthread_stop()). Thiswakeup is safe for real-time.

  • ALLOWLIST: to handle known false positives with the kernel.

  • BLOCK_ON_RT_MUTEX is included in the allowlist due to its implementation.In the release path of rt_mutex, a boosted task is de-boosted before wakingthe rt_mutex’s waiter. Consequently, the monitor may see a real-time-unsafewakeup (e.g. non-real-time task waking real-time task). This is actuallyreal-time-safe because preemption is disabled for the duration.

  • FUTEX_LOCK_PI is included in the allowlist for the same reason asBLOCK_ON_RT_MUTEX.