@dotnet/jit-contrib ,@AndyAyersMS

Expected impact is

• (short term) perf stability for benchmarks that run jitted code with loops
• (medium term) ability to optimize perf by adjusting alignment of code within methods

Would be good to highlight one or two benchmarks we really expect to stabilize with this (benchmarks with very simple timing payloads).

Note because of the way BenchmarkDotNet operates the main "driving" loop for most benchmark runs (which invokes a delegate that runs the method to benchmark) will be jitted and has a loop, so bypasses Tiering. One hypothesis we're exploring is that the varying alignment of this loop over time contributes to some of our observed day to day perf fluctuations.

Beyond that, once this is in, it would be good to do a retrospective study of benchmarks after we've gotten a few days worth of data, and see if we really did move the needle on stability. I had hoped to do that with#2249 but lab collection wasn't in great shape back then; looking back now it doesn't seem to have really had much impact, perhaps because it only affected Tier1 methods.

Any idea what a typical delta this causes on the overall size of the code segment?

I picked upLoopReturn benchmark which wasrecently found by BenchmarkDotNet to have 22% regression but isBimodel test. The benchmark itself has a tight loop on which I can do some experimentation. For the sake of argument, I removedthe only condition in the loop code to eliminate any branch prediction effect. The loop is 17 bytes long, so a single fetch of 32B line should bring the entire loop code into the decoder and cached in DSB.

Another thing that I experimented along with method alignment is the effect on the benchmark timing because of loop alignment within the method. With the help of Andy'sNopPadding branch, I added various padding ofnop to adjust loop header with alignments ranging from0x0 thru0xF to see if fixing the method alignment to 32-byte boundary helps overall execution timing and stability of the benchmark regardless of where loop is aligned. The results are positive.

All the measurements are done onIntel Core i7-8700 CPU 3.20GHz (Coffee Lake), 1 CPU, 12 logical and 6 physical cores.

Stability

Without the no-ops, the loop starts at offset0x3D, so adding 1 no-op moves the loop to0x3E and so forth. In below graph, X-axis is no. of nops padding I added in the method and Y-axis is the standard deviation of benchmark timings measured over 5 iterations. Key observation is that there is very less deviation with 32-byte alignment as compared to the default (16-byte alignment). With default alignment, for some nops padding, the benchmarks are stable, but in general they deviate if the loop's alignment is not favorable to the DSB line. As opposed to that, the 32-byte alignment provides stable timings with very little deviation, no matter what the loop alignment is. For 32-byte method alignment, when the loop's alignment gets to 0x40 (after adding 3 thru 15 nops), the performance even gets better. My theory is that after 3 nops are added in a method, the loop (17 bytes long) gets loaded entirely in a single fetch. For e.g. with 2 nops, the loop starts at0x3F address which is not a boundary of 32-byte alignment and so there might be two fetches needed to bring the loop to decoder. However after adding, for example, 5 nops, although the loop gets aligned to0x42, I assume the fetch happens starting from0x40 (because it is a 32-byte boundary), so the entire loop gets loaded even though non-loop uops between0x40~0x41 gets fetched too. This performance side-effect is guaranteed because the method is 32-byte aligned. So yes, I think this is the step in right direction.

Performance

I also tried measuring the performance between default vs. 32-byte alignment and I see some segment of my observation that I saw in stability. As soon as the loop gets aligned to 32-byte boundary (after adding 3+ nops), the performance with 32 byte alignment becomes better. So perhaps, to improve performance as Andy pointed out multiple times, it will be good to align hot loops at 32-byte boundary which in combination with method alignment will give us both stability and performance wins.

@adamsitnik , @danmosemsft

LoopReturn seems like a reasonable choice, though I am also interested cases where the benchmark payload does not have a loop. Maybe we can find one?

Here's the historical record showing the bimodal behavior, seems like most of the time it gets a bad alignment and is slow.

Pretty nice if we get perf for customers as well as stability for us. i didn't expect that: I assumed we would just move perf around when we made alignment stable. So great!

Note because of the way BenchmarkDotNet operates the main "driving" loop for most benchmark runs (which invokes a delegate that runs the method to benchmark) will be jitted and has a loop, so bypasses Tiering. One hypothesis we're exploring is that the varying alignment of this loop over time contributes to some of our observed day to day perf fluctuations.

If I remember correctly,@AndreyAkinshin has added a manual loop unrolling to this particular loop to minimize the effects of the alignment.

If you pass--keepFiles and open the auto-generated*.notcs file you should be able to see something like this:

privatevoidWorkloadActionUnroll(System.Int64invokeCount){for(System.Int64i=0;i<invokeCount;i++){consumer.Consume(workloadDelegate());consumer.Consume(workloadDelegate());consumer.Consume(workloadDelegate());consumer.Consume(workloadDelegate());consumer.Consume(workloadDelegate());consumer.Consume(workloadDelegate());consumer.Consume(workloadDelegate());consumer.Consume(workloadDelegate());consumer.Consume(workloadDelegate());consumer.Consume(workloadDelegate());consumer.Consume(workloadDelegate());consumer.Consume(workloadDelegate());consumer.Consume(workloadDelegate());consumer.Consume(workloadDelegate());consumer.Consume(workloadDelegate());consumer.Consume(workloadDelegate());}}

to disable it, you need to pass--unrollFactor 1:

privatevoidWorkloadActionUnroll(System.Int64invokeCount){for(System.Int64i=0;i<invokeCount;i++){consumer.Consume(workloadDelegate());}}

Key observation is that there is very less deviation with 32-byte alignment as compared to the default (16-byte alignment).

@kunalspathak this looks and sounds awesome!

it will be good to align hot loops at 32-byte boundary which in combination with method alignment will give us both stability and performance wins

this would be perfect!!

FWIW here are some benchmarks that I remember are very dependent on the loop alignment:

• System.Memory.Span<Char>.IndexOfValue
• System.Memory.Span<Byte>.BinarySearch

No loops, but super dependent on alignment onx86:

• Benchstone.BenchI.Fib.Test

Any idea what a typical delta this causes on the overall size of the code segment?

I added logging around the change to see how many methods get 32-byte aligned vs. the default alignment and then ran pmi on .NET libraries. There is definitely duplicate methods that might get counted in either of them, but on an average, 532,667 methods continued getting default alignment while 70,573 methods got 32-bytes alignment i.e. out of 603,240 total methods, 70,573 methods got 32-byte aligned which is approx.11% of them. This is close to what@AndyAyersMS anticipated in#2249 (comment).

LoopReturn seems like a reasonable choice, though I am also interested cases where the benchmark payload does not have a loop. Maybe we can find one?

I tried gathering the results forFib benchmark (the only one which is bimodel and has no-loops) and seeing mixed results.

master:

PR:

For x86 as well, I don't see much difference for Fib benchmark on my local machine.

master:

PR:

out of 603,240 total methods, 70,573 methods got 32-byte aligned

If we assume half of these would have been 32 byte aligned anyways, then we have an additional 35K instances of 16 bytes wasted in the code segment, so around 560K bytes of extra allocation.

I'd like to know how this compares to the total amount of code. What was the total code size for the 603,240 methods?

out of 603,240 total methods, 70,573 methods got 32-byte aligned

If we assume half of these would have been 32 byte aligned anyways, then we have an additional 35K instances of 16 bytes wasted in the code segment, so around 560K bytes of extra allocation.

That makes sense.

I'd like to know how this compares to the total amount of code. What was the total code size for the 603,240 methods?

The total code size is 51377607 bytes or around 51MB.

So perhaps as much as 1% of the code space is now padding. I suppose it's possible the code heap can fit some smaller methods into the larger padding voids.

Would be nice at some point to actually measure the heap sizes instead of trying to do these back of the envelope estimations.

Let's take this change and keep an eye on the perf lab runs to see if we do indeed see increased stability.

Would be nice at some point to actually measure the heap sizes instead of trying to do these back of the envelope estimations.

Yes, I wanted to get the heap size number for this one and spent some time debugging theallocCode and allocCodeRaw but in order to get accurate numbers from the appropriate heap (specially aroundallocCodeRaw andUnlockedAllocMemForCode_NoThrow) needed some extra special-casing because those methods are also called from runtime which I thought of deferring it given that I could get rough ballpark using other way. But it is on my TODO list to get the heap size.

This is pretty cool! 😄

@AndyAyersMS Do you think this could be related to the .NET perf regession from#36907?
Since your last hypothesis was about those switch branches in the hot path possibly not being aligned on .NET 5?

This work is a prerequisite for aligning branches -- we plan to explore doing that soon.

I'd expect this change might stabilize performance of your code, but would not necessarily optimize it.

Ah, I see, thank you for the clarification! 😊

Was just curious whether this was at least in part related to that, happy to hear branch alignment work is planned soon!