Movatterモバイル変換

[0]ホーム
URL:

Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

 Sign up
Appearance settings
forked fromtorvalds/linux

Commit76b0438

Browse files
dwmw2KAGA-KOKO
authored andcommitted
x86/retpoline: Add initial retpoline support
Enable the use of -mindirect-branch=thunk-extern in newer GCC, and providethe corresponding thunks. Provide assembler macros for invoking the thunksin the same way that GCC does, from native and inline assembler.This adds X86_FEATURE_RETPOLINE and sets it by default on all CPUs. Insome circumstances, IBRS microcode features may be used instead, and theretpoline can be disabled.On AMD CPUs if lfence is serialising, the retpoline can be dramaticallysimplified to a simple "lfence; jmp *\reg". A future patch, after it hasbeen verified that lfence really is serialising in all circumstances, canenable this by setting the X86_FEATURE_RETPOLINE_AMD feature bit in additionto X86_FEATURE_RETPOLINE.Do not align the retpoline in the altinstr section, because there is noguarantee that it stays aligned when it's copied over the oldinstr duringalternative patching.[ Andi Kleen: Rename the macros, add CONFIG_RETPOLINE option, export thunks][ tglx: Put actual function CALL/JMP in front of the macros, convert to symbolic labels ][ dwmw2: Convert back to numeric labels, merge objtool fixes ]Signed-off-by: David Woodhouse <dwmw@amazon.co.uk>Signed-off-by: Thomas Gleixner <tglx@linutronix.de>Acked-by: Arjan van de Ven <arjan@linux.intel.com>Acked-by: Ingo Molnar <mingo@kernel.org>Cc: gnomes@lxorguk.ukuu.org.ukCc: Rik van Riel <riel@redhat.com>Cc: Andi Kleen <ak@linux.intel.com>Cc: Josh Poimboeuf <jpoimboe@redhat.com>Cc: thomas.lendacky@amd.comCc: Peter Zijlstra <peterz@infradead.org>Cc: Linus Torvalds <torvalds@linux-foundation.org>Cc: Jiri Kosina <jikos@kernel.org>Cc: Andy Lutomirski <luto@amacapital.net>Cc: Dave Hansen <dave.hansen@intel.com>Cc: Kees Cook <keescook@google.com>Cc: Tim Chen <tim.c.chen@linux.intel.com>Cc: Greg Kroah-Hartman <gregkh@linux-foundation.org>Cc: Paul Turner <pjt@google.com>Link:https://lkml.kernel.org/r/1515707194-20531-4-git-send-email-dwmw@amazon.co.uk
1 parent258c760 commit76b0438

File tree

8 files changed

+231
-0
lines changed

8 files changed

+231
-0
lines changed

‎arch/x86/Kconfig‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -429,6 +429,19 @@ config GOLDFISH
429429
def_bool y
430430
depends on X86_GOLDFISH
431431

432+
config RETPOLINE
433+
bool "Avoid speculative indirect branches in kernel"
434+
default y
435+
help
436+
Compile kernel with the retpoline compiler options to guard against
437+
kernel-to-user data leaks by avoiding speculative indirect
438+
branches. Requires a compiler with -mindirect-branch=thunk-extern
439+
support for full protection. The kernel may run slower.
440+
441+
Without compiler support, at least indirect branches in assembler
442+
code are eliminated. Since this includes the syscall entry path,
443+
it is not entirely pointless.
444+
432445
config INTEL_RDT
433446
bool "Intel Resource Director Technology support"
434447
default n

‎arch/x86/Makefile‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -235,6 +235,16 @@ KBUILD_CFLAGS += -Wno-sign-compare
235235
#
236236
KBUILD_CFLAGS += -fno-asynchronous-unwind-tables
237237

238+
# Avoid indirect branches in kernel to deal with Spectre
239+
ifdefCONFIG_RETPOLINE
240+
RETPOLINE_CFLAGS +=$(call cc-option,-mindirect-branch=thunk-extern -mindirect-branch-register)
241+
ifneq ($(RETPOLINE_CFLAGS),)
242+
KBUILD_CFLAGS +=$(RETPOLINE_CFLAGS) -DRETPOLINE
243+
else
244+
$(warning CONFIG_RETPOLINE=y, but not supported by the compiler. Toolchain update recommended.)
245+
endif
246+
endif
247+
238248
archscripts: scripts_basic
239249
$(Q)$(MAKE)$(build)=arch/x86/tools relocs
240250

‎arch/x86/include/asm/asm-prototypes.h‎

Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -11,7 +11,32 @@
1111
#include<asm/pgtable.h>
1212
#include<asm/special_insns.h>
1313
#include<asm/preempt.h>
14+
#include<asm/asm.h>
1415

1516
#ifndefCONFIG_X86_CMPXCHG64
1617
externvoidcmpxchg8b_emu(void);
1718
#endif
19+
20+
#ifdefCONFIG_RETPOLINE
21+
#ifdefCONFIG_X86_32
22+
#defineINDIRECT_THUNK(reg) extern asmlinkage void __x86_indirect_thunk_e ## reg(void);
23+
#else
24+
#defineINDIRECT_THUNK(reg) extern asmlinkage void __x86_indirect_thunk_r ## reg(void);
25+
INDIRECT_THUNK(8)
26+
INDIRECT_THUNK(9)
27+
INDIRECT_THUNK(10)
28+
INDIRECT_THUNK(11)
29+
INDIRECT_THUNK(12)
30+
INDIRECT_THUNK(13)
31+
INDIRECT_THUNK(14)
32+
INDIRECT_THUNK(15)
33+
#endif
34+
INDIRECT_THUNK(ax)
35+
INDIRECT_THUNK(bx)
36+
INDIRECT_THUNK(cx)
37+
INDIRECT_THUNK(dx)
38+
INDIRECT_THUNK(si)
39+
INDIRECT_THUNK(di)
40+
INDIRECT_THUNK(bp)
41+
INDIRECT_THUNK(sp)
42+
#endif/* CONFIG_RETPOLINE */

‎arch/x86/include/asm/cpufeatures.h‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -203,6 +203,8 @@
203203
#defineX86_FEATURE_PROC_FEEDBACK( 7*32+ 9)/* AMD ProcFeedbackInterface */
204204
#defineX86_FEATURE_SME( 7*32+10)/* AMD Secure Memory Encryption */
205205
#defineX86_FEATURE_PTI( 7*32+11)/* Kernel Page Table Isolation enabled */
206+
#defineX86_FEATURE_RETPOLINE( 7*32+12)/* Generic Retpoline mitigation for Spectre variant 2 */
207+
#defineX86_FEATURE_RETPOLINE_AMD( 7*32+13)/* AMD Retpoline mitigation for Spectre variant 2 */
206208
#defineX86_FEATURE_INTEL_PPIN( 7*32+14)/* Intel Processor Inventory Number */
207209
#defineX86_FEATURE_INTEL_PT( 7*32+15)/* Intel Processor Trace */
208210
#defineX86_FEATURE_AVX512_4VNNIW( 7*32+16)/* AVX-512 Neural Network Instructions */
Lines changed: 128 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,128 @@
1+
/* SPDX-License-Identifier: GPL-2.0 */
2+
3+
#ifndef__NOSPEC_BRANCH_H__
4+
#define__NOSPEC_BRANCH_H__
5+
6+
#include<asm/alternative.h>
7+
#include<asm/alternative-asm.h>
8+
#include<asm/cpufeatures.h>
9+
10+
#ifdef__ASSEMBLY__
11+
12+
/*
13+
* This should be used immediately before a retpoline alternative. It tells
14+
* objtool where the retpolines are so that it can make sense of the control
15+
* flow by just reading the original instruction(s) and ignoring the
16+
* alternatives.
17+
*/
18+
.macroANNOTATE_NOSPEC_ALTERNATIVE
19+
.Lannotate_\@:
20+
.pushsection .discard.nospec
21+
.long .Lannotate_\@- .
22+
.popsection
23+
.endm
24+
25+
/*
26+
* These are the bare retpoline primitives for indirect jmp and call.
27+
* Do not use these directly; they only exist to make the ALTERNATIVE
28+
* invocation below less ugly.
29+
*/
30+
.macroRETPOLINE_JMPreg:req
31+
call.Ldo_rop_\@
32+
.Lspec_trap_\@:
33+
pause
34+
jmp.Lspec_trap_\@
35+
.Ldo_rop_\@:
36+
mov\reg, (%_ASM_SP)
37+
ret
38+
.endm
39+
40+
/*
41+
* This is a wrapper around RETPOLINE_JMP so the called function in reg
42+
* returns to the instruction after the macro.
43+
*/
44+
.macroRETPOLINE_CALLreg:req
45+
jmp.Ldo_call_\@
46+
.Ldo_retpoline_jmp_\@:
47+
RETPOLINE_JMP \reg
48+
.Ldo_call_\@:
49+
call.Ldo_retpoline_jmp_\@
50+
.endm
51+
52+
/*
53+
* JMP_NOSPEC and CALL_NOSPEC macros can be used instead of a simple
54+
* indirect jmp/call which may be susceptible to the Spectre variant 2
55+
* attack.
56+
*/
57+
.macroJMP_NOSPECreg:req
58+
#ifdefCONFIG_RETPOLINE
59+
ANNOTATE_NOSPEC_ALTERNATIVE
60+
ALTERNATIVE_2__stringify(jmp*\reg),\
61+
__stringify(RETPOLINE_JMP \reg),X86_FEATURE_RETPOLINE,\
62+
__stringify(lfence;jmp*\reg),X86_FEATURE_RETPOLINE_AMD
63+
#else
64+
jmp*\reg
65+
#endif
66+
.endm
67+
68+
.macroCALL_NOSPECreg:req
69+
#ifdefCONFIG_RETPOLINE
70+
ANNOTATE_NOSPEC_ALTERNATIVE
71+
ALTERNATIVE_2__stringify(call*\reg),\
72+
__stringify(RETPOLINE_CALL \reg),X86_FEATURE_RETPOLINE,\
73+
__stringify(lfence;call*\reg),X86_FEATURE_RETPOLINE_AMD
74+
#else
75+
call*\reg
76+
#endif
77+
.endm
78+
79+
#else/* __ASSEMBLY__ */
80+
81+
#defineANNOTATE_NOSPEC_ALTERNATIVE\
82+
"999:\n\t"\
83+
".pushsection .discard.nospec\n\t"\
84+
".long 999b - .\n\t"\
85+
".popsection\n\t"
86+
87+
#if defined(CONFIG_X86_64)&& defined(RETPOLINE)
88+
89+
/*
90+
* Since the inline asm uses the %V modifier which is only in newer GCC,
91+
* the 64-bit one is dependent on RETPOLINE not CONFIG_RETPOLINE.
92+
*/
93+
# defineCALL_NOSPEC\
94+
ANNOTATE_NOSPEC_ALTERNATIVE\
95+
ALTERNATIVE(\
96+
"call *%[thunk_target]\n",\
97+
"call __x86_indirect_thunk_%V[thunk_target]\n",\
98+
X86_FEATURE_RETPOLINE)
99+
# defineTHUNK_TARGET(addr) [thunk_target] "r" (addr)
100+
101+
#elif defined(CONFIG_X86_32)&& defined(CONFIG_RETPOLINE)
102+
/*
103+
* For i386 we use the original ret-equivalent retpoline, because
104+
* otherwise we'll run out of registers. We don't care about CET
105+
* here, anyway.
106+
*/
107+
# defineCALL_NOSPEC ALTERNATIVE("call *%[thunk_target]\n",\
108+
" jmp 904f;\n"\
109+
" .align 16\n"\
110+
"901:call 903f;\n"\
111+
"902:pause;\n"\
112+
" jmp 902b;\n"\
113+
" .align 16\n"\
114+
"903:addl $4, %%esp;\n"\
115+
" pushl %[thunk_target];\n"\
116+
" ret;\n"\
117+
" .align 16\n"\
118+
"904:call 901b;\n",\
119+
X86_FEATURE_RETPOLINE)
120+
121+
# defineTHUNK_TARGET(addr) [thunk_target] "rm" (addr)
122+
#else/* No retpoline */
123+
# defineCALL_NOSPEC "call *%[thunk_target]\n"
124+
# defineTHUNK_TARGET(addr) [thunk_target] "rm" (addr)
125+
#endif
126+
127+
#endif/* __ASSEMBLY__ */
128+
#endif/* __NOSPEC_BRANCH_H__ */

‎arch/x86/kernel/cpu/common.c‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -905,6 +905,10 @@ static void __init early_identify_cpu(struct cpuinfo_x86 *c)
905905
setup_force_cpu_bug(X86_BUG_SPECTRE_V1);
906906
setup_force_cpu_bug(X86_BUG_SPECTRE_V2);
907907

908+
#ifdefCONFIG_RETPOLINE
909+
setup_force_cpu_cap(X86_FEATURE_RETPOLINE);
910+
#endif
911+
908912
fpu__init_system(c);
909913

910914
#ifdefCONFIG_X86_32

‎arch/x86/lib/Makefile‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -26,6 +26,7 @@ lib-y += memcpy_$(BITS).o
2626
lib-$(CONFIG_RWSEM_XCHGADD_ALGORITHM) += rwsem.o
2727
lib-$(CONFIG_INSTRUCTION_DECODER) += insn.o inat.o
2828
lib-$(CONFIG_RANDOMIZE_BASE) += kaslr.o
29+
lib-$(CONFIG_RETPOLINE) += retpoline.o
2930

3031
obj-y += msr.o msr-reg.o msr-reg-export.o hweight.o
3132

‎arch/x86/lib/retpoline.S‎

Lines changed: 48 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,48 @@
1+
/* SPDX-License-Identifier: GPL-2.0 */
2+
3+
#include<linux/stringify.h>
4+
#include<linux/linkage.h>
5+
#include<asm/dwarf2.h>
6+
#include<asm/cpufeatures.h>
7+
#include<asm/alternative-asm.h>
8+
#include<asm/export.h>
9+
#include<asm/nospec-branch.h>
10+
11+
.macro THUNK reg
12+
.section.text.__x86.indirect_thunk.\reg
13+
14+
ENTRY(__x86_indirect_thunk_\reg)
15+
CFI_STARTPROC
16+
JMP_NOSPEC %\reg
17+
CFI_ENDPROC
18+
ENDPROC(__x86_indirect_thunk_\reg)
19+
.endm
20+
21+
/*
22+
* Despite being an assembler file we can't just use .irp here
23+
* because __KSYM_DEPS__ only uses the C preprocessor and would
24+
* only see one instance of "__x86_indirect_thunk_\reg" rather
25+
* than one per register with the correct names. So we do it
26+
* the simple and nasty way...
27+
*/
28+
#define EXPORT_THUNK(reg) EXPORT_SYMBOL(__x86_indirect_thunk_## reg)
29+
#define GENERATE_THUNK(reg) THUNK reg ; EXPORT_THUNK(reg)
30+
31+
GENERATE_THUNK(_ASM_AX)
32+
GENERATE_THUNK(_ASM_BX)
33+
GENERATE_THUNK(_ASM_CX)
34+
GENERATE_THUNK(_ASM_DX)
35+
GENERATE_THUNK(_ASM_SI)
36+
GENERATE_THUNK(_ASM_DI)
37+
GENERATE_THUNK(_ASM_BP)
38+
GENERATE_THUNK(_ASM_SP)
39+
#ifdef CONFIG_64BIT
40+
GENERATE_THUNK(r8)
41+
GENERATE_THUNK(r9)
42+
GENERATE_THUNK(r10)
43+
GENERATE_THUNK(r11)
44+
GENERATE_THUNK(r12)
45+
GENERATE_THUNK(r13)
46+
GENERATE_THUNK(r14)
47+
GENERATE_THUNK(r15)
48+
#endif

0 commit comments

Comments
 (0)
[8]ページ先頭
©2009-2025 Movatter.jp