Closes#33338 optimize comparisons for const strings
Closes#7651 Generic Type Check Inlining
Closes#10303 have inlining heuristics look for cases where inlining might enable devirtualization
Closes#47434 Weird inlining of methods optimized by JIT
Closes#41923 Don't inline calls inside rarely used basic-blocks
Closes#33349 Utf8Formatter.TryFormat causes massive code bloat when used with a constant StandardFormat

This PR was split, the first part is#53670

Inlining is one of the most important compiler optimizations - it significantly widens opportunities for other optimizations such as Constant Folding, CSE, Devirtualization, etc. Unfortunately, there is no Silver Bullet to do it right and it's essentially a sort of an NP completeKnapsack problem, especially in JIT environments where we don't see the whole graph of all calls, new types can be loaded/added dynamically, and we're limited in time, e.g. we can't just try to import all the possible callees into JIT's IR nodes, apply all the existing optimizations we have and decide whether it was profitable or not - we only have time to quickly inspect raw IL and note some useful patterns (furthermore, we ask R2R to bake "noinline" into non-profitable methods forever).

It's important to note possibleregressions due to bad Inlining decisions:

• Significant codegen size increase - it can be noticeable for the AOT'd (R2R) code. There are cases where inlining actually reduces size of callers, but mostly it's a trade-off between size and performance.
• Always regresses JIT's throughput (we analyze/inline more and it's not cheap). Visible effect - it takes longer to warmup/start an app/process first request.
• CPUs have limited sets of registers, so if we inline a callee with a lot of temps inside - we might cause serious problems for the Register Allocator (that, just like the inliner, also uses heuristics 🙂).
• Bigger functions/loop bodies may not fit into CPU caches.
• Jit has a hard-coded limit of locals it can track (lclMAX_TRACKED=512) - a good example is thissharplab.io snippet.

Current approach in RyuJIT

In RyuJIT we have several strategies (or policies) such asDefaultPolicy,ModelPolicy (based on some ML model to estimate impact),SizePolicy (prefers size over perf) andProfilePolicy (heavily relies on PGO data). But onlyDefaultPolicy is used in Production. Its algorithm can be explained in 5 steps:

1. Make some quick pre-checks, e.g.:
   • Callee is less than 16 bytes of IL code or hasAggressiveInlining attribute - always inline
   • Callee is larger than 100 bytes of IL code or hasNoInlinine - always ignore
   • Callee has EH, stackalloc (and callsite is in a loop), etc - always ignore
   • Check that we don't exceed depth of calls, amount of basic-block and time budget.
2. Roughly estimate codegen size of the callsite (NOTE: we don't take current caller's size into account directly)
3. Roughly estimate codegen size of the inline candidate by using a state-machine that accepts raw IL code and applies some weight for all IL opcodes.
4. Inspectraw IL and try to find some useful observations (heuristics) and compose a so called "Benefit multiplier"
5. Make a decision using the following formula:bool toInline = callsiteNativeSize * BenefitMultiplier > calleeNativeSize
   An example with logs:
   
   (BTW, in this PR this method is inlined)

Things this PR tries to improve

1. Add more observations (it's done inInliner: new observations (don't impact inlining for now) #53670) on top of raw IL, such as
   • Inline candidate has some abstract class in its signature for one of the arguments and we pass something known at the callsite. If we inline it we'll be able to devirtualize all calls inside the callee even without the GDV (Guarded Devirtualization).
   • Inline candidate is generic when caller is not
   • Inline candidate has foldable unary/binary expressions and even branches when we pass a constant argument(s).
   • Inline candidate has foldable BOX/UNBOX operations (generics emit a lot of them).
   • Inline candidate has an expensive "X / Arg" expression and we pass a constant at the callsite for the Arg - if we inline it - we'll avoid expensiveidiv.
   • It now is able to recognize more potentially foldable branches, even patterns liketypeof(T1) == typeof(T2) orargStr.Length == 10 whenargStr is a string literal at the callsite. (NOTE: only during prejitting or when a method was promoted to tier1 naturally. Won't work for methods with loops or with TieredCompilation=0).
   • Inline candidates pass/returns a large struct by value - we might avoid redundant copies if we inline.
   • there are few more
     A small example that covers some of the new observations inliner makes:
     

Thanks to@tannergooding's work we can now resolve somecall tokens and recognize intrinsics.

2. RefactorfgFindJumpTarget andPushedStack: We pushed opcodes to that two-slots stack in order to recognize some patterns in the next (binary/unary) opcodes - unfortunately we ignored a lot of opcodes and it led to invalid state in that stack => invalid observations. Such invalid observations were the main source of regressions in my PR because some of them led to a better performance for some reason 😐.

3. Find more optimal multipliers for the observations (using micro/TE benchmarks, deep performance analysis of traces/regressions, example:[JIT] Improve inliner: new heuristics, rely on PGO data #52708 (comment)).

4. Rely on PGO data - the most important part. We try harder to inline what was recognized as hot. Unfortunately, there are cases where a profile can be misleading/polluted and mark hot or semi-hot blocks as cold:

   • We don't support context-sensitive instrumentation in Dynamic PGO mode (see example inPGO: Instrument cold blocks with profile validators #53840)
   • The Static PGO that we ship can be irrelevant for some apps.
   • We don't support deoptimizations so we can't re-collect profiles if something changes in the workflow over time.
   • Sometimes it's still makes sense to inline methods in cold blocks to improve type/escape analysis for the whole caller. Example:sharplab.io

5. Allow JIT to inline bigger functions if really needed: more than 5 basic-blocks and more than 100 bytes of IL.

Metodology

For now, I mostly run various benchmarks (TE, dotnet/performance, PowerShell, Kestrel, SignalR), analyze collected samples (flamegraphs) produced by profilers/dotnet trace/BDN itself in hot spots where I inspect all calls in the chains and if I see a method that looks like it should be inlined - I try to find what kind of observations (and corresponding benefit multipliers) can make it happen, example:

There are some thoughts on how to automatize it using ML/Genetic Algorithms.

Benchmarks

I usePerfLab machines for TE/SignalR/YARP benchmarks. Besides that, I have locally:

• Win - Windows 10 x64, Core i7 8700K (4 cores, 3.7Ghz, Coffee Lake, AVX2)
• Linux - Ubuntu 20.04 x64, Core i7 4930K (6 cores, 3.4Ghz, Ivy Bridge E, AVX1)
• Arm64 - macOS 11.5 arm64 (Mac Mini M1, arm64-v8.3(or higher?))

Also, I use different JIT modes:

• Default - Default arguments.
• NoPGO - same asDefault, but without static PGO collected from various benchmarks/workloads. I use this mode to estimate overall impact from PGO when compared againstFullPGO (in .NET 5.0 we didn't have any static profile so it kind of simulates net5.0).
• DynPGO:
  • DOTNET_TieredPGO=1 - collect profile in tier0.
  • DOTNET_TC_QuickJitForLoops=1 - so we also instrument methods with loops on tier0. By default they bypass tier0 to avoid "cold loops with hot bodies" problem (OSR is not enabled by default yet).
• FullPGO: same asDynPGO but with:
  • DOTNET_ReadyToRun=0 - ignore all AOT code and re-collect actual profile (with better class probes).

Results

It seems like this PR unlocks PGO and GDV potential in FullPGO mode. Many high-load benchmarks show +10-20% improvements. I tried to avoid startup/time-to-first-request regressions in the Default mode so the RPS numbers are not as great as they could be with a more aggressive inliner. In order to avoid size regressions in R2R, we use the "old" inliner - it leads to zero improvements when we benchmark R2R'd code inDOTNET_TieredCompilation=0 mode (e.g.dotnet/performance microbenchmarks). Also, prejitter bakes "noinline" into thousands of method using the old algorithm and kind of limits possibilities for the new one during prejitting.

The latest run of TE benchmarks on aspnet-citrine-lin:

^ some P90-P99 numbers regressed because, I believe, they include the warmup stage, so when I try to run a specific benchmark and ask it to run longer than 30 seconds - P99 numbers are greatly improved)
Also, don't expect large numbers from Platform-* benchmarks, those are heavily optimized by hands 😐.
UPD: the latest numbers:#52708 (comment)

NoPGO vs FullPGO (mainly to estimate impact of PGO and show room for improvements for the Default mode, NOTE: NoPGO numbers should be slightly better than the current results for .NET 5.0 in the Default mode):
UPD new numbers:#52708 (comment)

JIT ET events show that the improved inliner usually inlines 10-20% more functions, e.g.:
Plaintext-MVC (Default mode):

Orchard CMS (Default mode):

YARP Proxy (Default mode):