Softlockup detector and hardlockup detector (aka nmi_watchdog)

The Linux kernel can act as a watchdog to detect both soft and hardlockups.

A ‘softlockup’ is defined as a bug that causes the kernel to loop inkernel mode for more than 20 seconds (see “Implementation” below fordetails), without giving other tasks a chance to run. The currentstack trace is displayed upon detection and, by default, the systemwill stay locked up. Alternatively, the kernel can be configured topanic; a sysctl, “kernel.softlockup_panic”, a kernel parameter,“softlockup_panic” (see “The kernel’s command-line parameters” fordetails), and a compile option, “BOOTPARAM_SOFTLOCKUP_PANIC”, areprovided for this.

A ‘hardlockup’ is defined as a bug that causes the CPU to loop inkernel mode for more than 10 seconds (see “Implementation” below fordetails), without letting other interrupts have a chance to run.Similarly to the softlockup case, the current stack trace is displayedupon detection and the system will stay locked up unless the defaultbehavior is changed, which can be done through a sysctl,‘hardlockup_panic’, a compile time knob, “BOOTPARAM_HARDLOCKUP_PANIC”,and a kernel parameter, “nmi_watchdog”(see “The kernel’s command-line parameters” for details).

The panic option can be used in combination with panic_timeout (thistimeout is set through the confusingly named “kernel.panic” sysctl),to cause the system to reboot automatically after a specified amountof time.

Implementation

The soft and hard lockup detectors are built on top of the hrtimer andperf subsystems, respectively. A direct consequence of this is that,in principle, they should work in any architecture where thesesubsystems are present.

A periodic hrtimer runs to generate interrupts and kick the watchdogjob. An NMI perf event is generated every “watchdog_thresh”(compile-time initialized to 10 and configurable through sysctl of thesame name) seconds to check for hardlockups. If any CPU in the systemdoes not receive any hrtimer interrupt during that time the‘hardlockup detector’ (the handler for the NMI perf event) willgenerate a kernel warning or call panic, depending on theconfiguration.

The watchdog job runs in a stop scheduling thread that updates atimestamp every time it is scheduled. If that timestamp is not updatedfor 2*watchdog_thresh seconds (the softlockup threshold) the‘softlockup detector’ (coded inside the hrtimer callback function)will dump useful debug information to the system log, after which itwill call panic if it was instructed to do so or resume execution ofother kernel code.

The period of the hrtimer is 2*watchdog_thresh/5, which means it hastwo or three chances to generate an interrupt before the hardlockupdetector kicks in.

As explained above, a kernel knob is provided that allowsadministrators to configure the period of the hrtimer and the perfevent. The right value for a particular environment is a trade-offbetween fast response to lockups and detection overhead.

By default, the watchdog runs on all online cores. However, on akernel configured with NO_HZ_FULL, by default the watchdog runs onlyon the housekeeping cores, not the cores specified in the “nohz_full”boot argument. If we allowed the watchdog to run by default onthe “nohz_full” cores, we would have to run timer ticks to activatethe scheduler, which would prevent the “nohz_full” functionalityfrom protecting the user code on those cores from the kernel.Of course, disabling it by default on the nohz_full cores means thatwhen those cores do enter the kernel, by default we will not beable to detect if they lock up. However, allowing the watchdogto continue to run on the housekeeping (non-tickless) cores meansthat we will continue to detect lockups properly on those cores.

In either case, the set of cores excluded from running the watchdogmay be adjusted via the kernel.watchdog_cpumask sysctl. Fornohz_full cores, this may be useful for debugging a case where thekernel seems to be hanging on the nohz_full cores.