25.3.1.Synchronizing all CPUs¶

The microcode engine which receives the microcode update is sharedbetween the two logical threads in a SMT system. Therefore, whenthe update is executed on one SMT thread of the core, the sibling“automatically” gets the update.

Since the microcode can “simulate” MSRs too, while the microcode updateis in progress, those simulated MSRs transiently cease to exist. Thiscan result in unpredictable results if the SMT sibling thread happens tobe in the middle of an access to such an MSR. The usual observation isthat such MSR accesses cause #GPs to be raised to signal that former arenot present.

The disappearing MSRs are just one common issue which is being observed.Any other instruction that’s being patched and gets concurrentlyexecuted by the other SMT sibling, can also result in similar,unpredictable behavior.

To eliminate this case, astop_machine()-based CPU synchronization wasintroduced as a way to guarantee that all logical CPUs will not executeany code but just wait in a spin loop, polling an atomic variable.

While this took care of device or external interrupts, IPIs includingLVT ones, such as CMCI etc, it cannot address other special interruptsthat can’t be shut off. Those are Machine Check (#MC), System Management(#SMI) and Non-Maskable interrupts (#NMI).

25.3.2.Machine Checks¶

Machine Checks (#MC) are non-maskable. There are two kinds of MCEs.Fatal un-recoverable MCEs and recoverable MCEs. While un-recoverableerrors are fatal, recoverable errors can also happen in kernel contextare also treated as fatal by the kernel.

On certain Intel machines, MCEs are also broadcast to all threads in asystem. If one thread is in the middle of executing WRMSR, a MCE will betaken at the end of the flow. Either way, they will wait for the threadperforming the wrmsr(0x79) to rendezvous in the MCE handler and shutdowneventually if any of the threads in the system fail to check in to theMCE rendezvous.

To be paranoid and get predictable behavior, the OS can choose to setMCG_STATUS.MCIP. Since MCEs can be at most one in a system, if anMCE was signaled, the above condition will promote to a system resetautomatically. OS can turn off MCIP at the end of the update for thatcore.

25.3.3.System Management Interrupt¶

SMIs are also broadcast to all CPUs in the platform. Microcode updaterequests exclusive access to the core before writing to MSR 0x79. So ifit does happen such that, one thread is in WRMSR flow, and the 2nd gotan SMI, that thread will be stopped in the first instruction in the SMIhandler.

Since the secondary thread is stopped in the first instruction in SMI,there is very little chance that it would be in the middle of executingan instruction being patched. Plus OS has no way to stop SMIs fromhappening.

25.3.4.Non-Maskable Interrupts¶

When thread0 of a core is doing the microcode update, if thread1 ispulled into NMI, that can cause unpredictable behavior due to thereasons above.

OS can choose a variety of methods to avoid running into this situation.

25.3.5.Is the microcode suitable for late loading?¶

Late loading is done when the system is fully operational and runningreal workloads. Late loading behavior depends on what the base patch onthe CPU is before upgrading to the new patch.

This is true for Intel CPUs.

Consider, for example, a CPU has patch level 1 and the update is topatch level 3.

Between patch1 and patch3, patch2 might have deprecated a software-visiblefeature.

This is unacceptable if software is even potentially using that feature.For instance, say MSR_X is no longer available after an update,accessing that MSR will cause a #GP fault.

Basically there is no way to declare a new microcode update suitablefor late-loading. This is another one of the problems that caused lateloading to be not enabled by default.