What is hwpoison?¶

Upcoming Intel CPUs have support for recovering from some memory errors(MCArecovery). This requires the OS to declare a page “poisoned”,kill the processes associated with it and avoid using it in the future.

This patchkit implements the necessary infrastructure in the VM.

To quote the overview comment:

High level machine check handler. Handles pages reported by thehardware as being corrupted usually due to a 2bit ECC memory or cachefailure.This focusses on pages detected as corrupted in the background.When the current CPU tries to consume corruption the currentlyrunning process can just be killed directly instead. This impliesthat if the error cannot be handled for some reason it's safe tojust ignore it because no corruption has been consumed yet. Insteadwhen that happens another machine check will happen.Handles page cache pages in various states. The tricky parthere is that we can access any page asynchronous to other VMusers, because memory failures could happen anytime and anywhere,possibly violating some of their assumptions. This is why this codehas to be extremely careful. Generally it tries to use normal lockingrules, as in get the standard locks, even if that means theerror handling takes potentially a long time.Some of the operations here are somewhat inefficient and have nonlinear algorithmic complexity, because the data structures have notbeen optimized for this case. This is in particular the casefor the mapping from a vma to a process. Since this case is expectedto be rare we hope we can get away with this.

The code consists of a the high level handler in mm/memory-failure.c,a new page poison bit and various checks in the VM to handle poisonedpages.

The main target right now is KVM guests, but it works for all kindsof applications. KVM support requires a recent qemu-kvm release.

For the KVM use there was need for a new signal type so thatKVM can inject the machine check into the guest with the properaddress. This in theory allows other applications to handlememory failures too. The expectation is that most applicationswon’t do that, but some very specialized ones might.

Failure recovery modes¶

There are two (actually three) modes memory failure recovery can be in:

vm.memory_failure_recovery sysctl set to zero:

All memory failures cause a panic. Do not attempt recovery.

early kill

(can be controlled globally and per process)Send SIGBUS to the application as soon as the error is detectedThis allows applications who can process memory errors in a gentleway (e.g. drop affected object)This is the mode used by KVM qemu.

late kill

Send SIGBUS when the application runs into the corrupted page.This is best for memory error unaware applications and defaultNote some pages are always handled as late kill.

User control¶

vm.memory_failure_recovery

See sysctl.txt

vm.memory_failure_early_kill

Enable early kill mode globally

PR_MCE_KILL

Set early/late kill mode/revert to system default

arg1: PR_MCE_KILL_CLEAR:

Revert to system default

arg1: PR_MCE_KILL_SET:

arg2 defines thread specific mode

PR_MCE_KILL_EARLY:

Early kill

PR_MCE_KILL_LATE:

Late kill

PR_MCE_KILL_DEFAULT

Use system global default

Note that if you want to have a dedicated thread which handlesthe SIGBUS(BUS_MCEERR_AO) on behalf of the process, you shouldcall prctl(PR_MCE_KILL_EARLY) on the designated thread. Otherwise,the SIGBUS is sent to the main thread.

PR_MCE_KILL_GET

return current mode

Testing¶

• madvise(MADV_HWPOISON, ....) (as root) - Poison a page in theprocess for testing

• hwpoison-inject module through debugfs/sys/kernel/debug/hwpoison/

  corrupt-pfn

  Inject hwpoison fault at PFN echoed into this file. This doessome early filtering to avoid corrupted unintended pages in test suites.

  unpoison-pfn

  Software-unpoison page at PFN echoed into this file. This waya page can be reused again. This only works for Linuxinjected failures, not for real memory failures. Once any hardwarememory failure happens, this feature is disabled.

  Note these injection interfaces are not stable and might change betweenkernel versions

  corrupt-filter-dev-major, corrupt-filter-dev-minor

  Only handle memory failures to pages associated with the filesystem defined by block device major/minor. -1U is thewildcard value. This should be only used for testing withartificial injection.

  corrupt-filter-memcg

  Limit injection to pages owned by memgroup. Specified by inodenumber of the memcg.

  Example:

  mkdir /sys/fs/cgroup/mem/hwpoisonusemem -m 100 -s 1000 &echo `jobs -p` > /sys/fs/cgroup/mem/hwpoison/tasksmemcg_ino=$(ls -id /sys/fs/cgroup/mem/hwpoison | cut -f1 -d' ')echo $memcg_ino > /debug/hwpoison/corrupt-filter-memcgpage-types -p `pidof init`   --hwpoison  # shall do nothingpage-types -p `pidof usemem` --hwpoison  # poison its pages

  corrupt-filter-flags-mask, corrupt-filter-flags-value

  When specified, only poison pages if ((page_flags & mask) ==value). This allows stress testing of many kinds ofpages. The page_flags are the same as in /proc/kpageflags. Theflag bits are defined in include/linux/kernel-page-flags.h anddocumented inExamining Process Page Tables

• Architecture specific MCE injector

  x86 has mce-inject, mce-test

  Some portable hwpoison test programs in mce-test, see below.

References¶

http://halobates.de/mce-lc09-2.pdf

Overview presentation from LinuxCon 09

git://git.kernel.org/pub/scm/utils/cpu/mce/mce-test.git

Test suite (hwpoison specific portable tests in tsrc)

git://git.kernel.org/pub/scm/utils/cpu/mce/mce-inject.git

x86 specific injector

Limitations¶

• Not all page types are supported and never will. Most kernel internalobjects cannot be recovered, only LRU pages for now.

---Andi Kleen, Oct 2009