9.1. Overview¶

On PCI Express, interrupts are represented with either MSI or inboundinterrupt messages (Assert_INTx/Deassert_INTx). The integrated IO-APIC in agiven Core IO converts the legacy interrupt messages from PCI Express toMSI interrupts. If the IO-APIC is disabled (via the mask bits in theIO-APIC table entries), the messages are routed to the legacy PCH. Thisin-band interrupt mechanism was traditionally necessary for systems thatdid not support the IO-APIC and for boot. Intel in the past has used theterm “boot interrupts” to describe this mechanism. Further, the PCI Expressprotocol describes this in-band legacy wire-interrupt INTx mechanism forI/O devices to signal PCI-style level interrupts. The subsequent paragraphsdescribe problems with the Core IO handling of INTx message routing to thePCH and mitigation within BIOS and the OS.

9.2. Issue¶

When in-band legacy INTx messages are forwarded to the PCH, they in turntrigger a new interrupt for which the OS likely lacks a handler. When aninterrupt goes unhandled over time, they are tracked by the Linux kernel asSpurious Interrupts. The IRQ will be disabled by the Linux kernel after itreaches a specific count with the error “nobody cared”. This disabled IRQnow prevents valid usage by an existing interrupt which may happen to sharethe IRQ line:

irq 19: nobody cared (try booting with the "irqpoll" option)CPU: 0 PID: 2988 Comm: irq/34-nipalk Tainted: 4.14.87-rt49-02410-g4a640ec-dirty #1Hardware name: National Instruments NI PXIe-8880/NI PXIe-8880, BIOS 2.1.5f1 01/09/2020Call Trace:<IRQ> ? dump_stack+0x46/0x5e ? __report_bad_irq+0x2e/0xb0 ? note_interrupt+0x242/0x290 ? nNIKAL100_memoryRead16+0x8/0x10 [nikal] ? handle_irq_event_percpu+0x55/0x70 ? handle_irq_event+0x4f/0x80 ? handle_fasteoi_irq+0x81/0x180 ? handle_irq+0x1c/0x30 ? do_IRQ+0x41/0xd0 ? common_interrupt+0x84/0x84</IRQ>handlers:irq_default_primary_handler threaded usb_hcd_irqDisabling IRQ #19

9.3. Conditions¶

The use of threaded interrupts is the most likely condition to triggerthis problem today. Threaded interrupts may not be reenabled after the IRQhandler wakes. These “one shot” conditions mean that the threaded interruptneeds to keep the interrupt line masked until the threaded handler has run.Especially when dealing with high data rate interrupts, the thread needs torun to completion; otherwise some handlers will end up in stack overflowssince the interrupt of the issuing device is still active.

9.4. Affected Chipsets¶

The legacy interrupt forwarding mechanism exists today in a number ofdevices including but not limited to chipsets from AMD/ATI, Broadcom, andIntel. Changes made through the mitigations below have been applied todrivers/pci/quirks.c

Starting with ICX there are no longer any IO-APICs in the Core IO’sdevices. IO-APIC is only in the PCH. Devices connected to the Core IO’sPCIe Root Ports will use native MSI/MSI-X mechanisms.

9.5. Mitigations¶

The mitigations take the form of PCI quirks. The preference has been tofirst identify and make use of a means to disable the routing to the PCH.In such a case a quirk to disable boot interrupt generation can beadded.[1]

Intel® 6300ESB I/O Controller Hub

Alternate Base Address Register:

BIE: Boot Interrupt Enable

0	Boot interrupt is enabled.
1	Boot interrupt is disabled.

Intel® Sandy Bridge through Sky Lake based Xeon servers:

Coherent Interface Protocol Interrupt Control

dis_intx_route2pch/dis_intx_route2ich/dis_intx_route2dmi2:

When this bit is set. Local INTx messages received from theIntel® Quick Data DMA/PCI Express ports are not routed to legacyPCH - they are either converted into MSI via the integrated IO-APIC(if the IO-APIC mask bit is clear in the appropriate entries)or cause no further action (when mask bit is set)

In the absence of a way to directly disable the routing, another approachhas been to make use of PCI Interrupt pin to INTx routing tables forpurposes of redirecting the interrupt handler to the rerouted interruptline by default. Therefore, on chipsets where this INTx routing cannot bedisabled, the Linux kernel will reroute the valid interrupt to its legacyinterrupt. This redirection of the handler will prevent the occurrence ofthe spurious interrupt detection which would ordinarily disable the IRQline due to excessive unhandled counts.[2]

The config option X86_REROUTE_FOR_BROKEN_BOOT_IRQS exists to enable (ordisable) the redirection of the interrupt handler to the PCH interruptline. The option can be overridden by either pci=ioapicreroute orpci=noioapicreroute.[3]

9.6. More Documentation¶

There is an overview of the legacy interrupt handling in several datasheets(6300ESB and 6700PXH below). While largely the same, it provides insightinto the evolution of its handling with chipsets.