After years of work, Mono can now be built out of thedotnet/runtime repository in a .NET 5-compatible mode! This mode means numerous changes in the available APIs, managed and embedding, as well as internal runtime behavioral changes to better align Mono with CoreCLR and the .NET ecosystem.

One area with multiple highly impactful changes to the runtime internals is library loading. For managed assemblies, Mono now follows the algorithms outlined onthis page, which result from the removal ofAppDomains and the newAssemblyLoadContext APIs. The only exception to this is that Mono still supports bundles registered via the embedding API, and so the runtime will check that as part of the probing logic.

The managed loading changes are fairly clear and well documented, but unmanaged library loading has changed in numerous ways, some of them far more subtle.

Summary of changes

• New P/Invoke resolution algorithm
• Dropped support for DllMap
• Unmanaged library loading defaults toRTLD_LOCAL
• Added support forDefaultDllImportSearchPathsAttribute
• On non-Windows platforms, Mono and CoreCLR no longer attempt to probe for A/W variants of symbols
• Default loader log level changed from INFO to DEBUG, and new log entries added for the new algorithm

More detail where appropriate in the sections below.

Dropped support for DllMap

The new unmanaged loading algorithm makes no mention of DllMap, as Mono has removed its functionality almost entirely in .NET 5. DllMap’s XML config files have have been disabled on every platform out of security concerns. The DllMap embedding APIs are also disabled on desktop platforms, though this may change.

In place of DllMap, users are encouraged to utilize theNativeLibrary resolution APIs, which are set in managed code, and theruntime hosting properties, which are set by embedders with themonovm_initialize function.

We recognize that this does not sufficiently cover some existing mono/mono scenarios. If the NativeLibrary APIs are insufficient for your use case, pleasetell us about it! We’re always looking to improve our interop functionality, and in particular with .NET 6 will be evaluatingNativeLibrary, so community input would be greatly appreciated.

Unmanaged library loading defaults toRTLD_LOCAL

A more subtle, yet no less impactful change is that native library loading now defaults toRTLD_LOCAL to be consistent with CoreCLR and Windows, as opposed to our historical behavior ofRTLD_GLOBAL. What this means in practice is that on Unix-like platforms, libraries are no longer loaded into a single global namespace and when looking up symbols, the librarymust be correctly specified. This change prevents symbol collision, and will both break and enable various scenarios and libraries. For more information on the difference, seethe dlopen man page.

For an example: historically in Mono on Linux, it was possible to load libraryfoo containing symbolbar, and then invokebar with a P/Invoke like so:

// note the incorrect library name[DllImport("asdf")]public static extern int bar();

This will no longer work. For that P/Invoke to function correctly, the attribute would need to use the correct library name:[DllImport("foo")]. A lot of code in the wild that was using incorrect library names will need to be updated. However, this means that when loading two libraries containing the same symbol name, there is no longer a conflict.

There have been some embedding API changes as part of this.MONO_DL_MASK is no longer a full mask, asMONO_DL_GLOBAL has been introduced to specifyRTLD_GLOBAL. If bothMONO_DL_LOCAL andMONO_DL_GLOBAL, are set, Mono will use local. See mono/utils/mono-dl-fallback.h for more info.

This also means that dynamically linking libmonosgen and attempting to resolve Mono symbols fromdlopen(NULL, ...) will no longer work.__Internal has been preserved as a Mono-specific extension, but its meaning has been expanded. When P/Invoking into__Internal, the runtime will check bothdlopen(NULL) and the runtime library in the case that they differ, so that users attempting to call Mono APIs with__Internal will not have those calls break.

Added support forDefaultDllImportSearchPathsAttribute

Mono now supports theDefaultDllImportSearchPathsAttribute attribute, which can be found inSystem.Runtime.InteropServices. In particular, passingDllImportSearchPath.AssemblyDirectory is now required to have the loader search the executing assembly’s directory for native libraries, and the other Windows-specific loader options should be passed down when appropriate.

Fin

And that’s it! If you have any further questions, feel free to ping us onDiscord orGitter.

Note: This is a guest post by Jordi Mon Companys from Códice Software, a long-time Mono user, about how they used Mono to develop their flagship product.

Plastic SCMis a full version control stack. This means Plastic SCMcomprises a full repo management core, command line (until here it wouldbe the equivalent to bare Git), native GUIs on Linux, macOS and Windows,web interfaces, diff and merge tools (the equivalent to Meld, WinMergeor Kdiff3), and also a cloud hosting for repositories. Add Visual Studioplugins, integrations with the major Continuous Integration systems,IDEs and issue trackers.

Plastic SCM was first released in 2006 and didn't stop evolving in thelast 13+ years, with 2-3 new public versions every week for the last 2 years.

Overall Plastic SCM sums more than 1.5 million lines of code and 95% ofthem written in C#. This means we have extensively used Mono for everythingnon-Windows since the very early days, now almost a decade and a halfago.

And here goes the story.

Mono shone light down the cross-platform way

When the first lines of Plastic SCM were written back in September 2005,the decision to go for C# was already made. But we knew a new versioncontrol system could only be considered as a serious alternative if itwas truly cross-platform. Windows-only was not a choice.

Why then, we decided to go for .NET/C# instead of something moreportable like Java, or C/C++? The reason was clear: because Monoexisted. We had never decided to use C# if Mono hadn't been there already.It promised a way to have cross-platform .NET and we invested heavily onit. How did it work out? Read along fellow monkeys!

Fun with cross-platform WinForms

Code once, run everywhere. That's what we embraced when we starteddoing our first tests with WinForms on Linux.

With very minor changes, the full Windows version was able to run onLinux and macOS (through X11). We later rewrote most of the controls wewere using on WinForms to give them a more consistent look and feel:

We also did this as a workaround to basically skip some well-knownissues with some standard controls. Obviously, desktop GUIs were not abig thing in Mono, and we felt like pioneers finding our way through awild territory :-)

Mono on Solaris - a story from the good-old days

Something many won't know is that for a couple of years we were theunofficial maintainers of theMono port forSolaris.

We were lucky enough to hire a former Mono hacker, our friend DickPorter, who enrolled to help us porting Plastic SCM to exotic platformslike Solaris and HP-UX.

By that time, we still relied on WinForms everywhere, which was achallenge on its own.

You can see how Plastic SCM running on Solaris looked like:

And:

We were super excited about it because it allowed us to run Plastic SCMon some old Sun workstations we had around. And they featured SPARCCPUs, 64-bit big endian and everything. In fact, we found and protectedsome edge cases caused by big endian :-).

From Boehm to sgen

We were hit by some of the limitations of Boehm GC, so we happilyprovided the developers working on the new sgen collector with a memoryhungry Plastic SCM environment. We used to run some memory intensiveautomated tests for them so we mutually benefit from the effort.

This was mostly before everyone moved to the cloud, so we ran most ofthese tests in about 300 real machines controlled by our in-house PNUnittest environment.

Native GUIs

Depending on X11 to run our GUI on macOS wasn't greatly perceived byhardcore Apple users who prefer a smooth experience. So, we decided toradically change our approach to GUI development. We committed to createnative GUIs for each of our platforms.

• Windows would still benefit from the same original codebase. But,removing the common denominator requirements allowed us to introducenew controls and enrich the overall experience.

• The macOS GUI would be rewritten taking advantage of MonoMac, whichlater became XamarinMac, the technology we still use. It was goingto be an entirely new codebase that only shared the core operationswith Windows, while the entire intermediate layer would be developedfrom scratch.

• And finally, we decided to go for a native GTKSharp-based GUI forLinux. In fact, it would be a good exercise to see how much of thecommon layer could be actually shared between macOS and Linux. Itworked quite well.

Some takeaways from this new approach:

• We decided to entirely skip "visual designer tools". ResX onWinForms proved to be a nightmare when used cross-platform, anddepending on locating controls by hand with a designer on a canvaswasn't good to keep consistent margins, spacing and so on. So, wewent cowboy: every single GUI you see in Plastic SCM now (except theolder ones in WinForms) is basically coded, not designed. Everybutton is created with "new Button()", and so on. It can soundlike a slowdown, but it certainly pays off when maintaining code:you spend muchless time dealing with code thandesigners.

• We created our ownautomated GUI testenvironmentto test the Linux and macOS GUIs. There weren't any cross-platformsolutions for Mono, so we decided to create our own.

• We realized how much better GTK was and is than any other solutionfrom a programmer’s perspective. We love to code GTK. Yes, it isalso possibly the ugliest in visual terms of them all, but youcan't have it all :-)

This is how Plastic SCM looks now, enjoy:

But wait! Wouldn’t Mono affect performance?

Many of you might think: how can a version control be written inMono/C# and expect to compete against Git or Subversion or evenPerforce which are all written in C or a C/C++ combination?

Speed was an obsession for us since day one, and we found C# to bequite capable if used carefully. The only downside is that when you arein a C#/managed world you tend to think allocating memory is free andyou pay for it when that happens (something that radically changed withthe arrival of .NET Core and the entire Span<T> and their focus onmaking the platform a real option for highly scalable and performantsolutions). But, over the years we learned a few lessons, started to bemuch more aware of the importance of saving allocations, and the resultsbacked up that reasoning.

Below you can see how a 2019 version of Plastic SCM compares to Git anda major commercial version control competitor performing quite commonoperations:

As you can see, Plastic SCM consistently beats Git, which we believe isquite an achievement considering it is written in .NET/Mono/C# insteadof system-level C.

Heavy loaded servers

In terms of pure scalability, we also achieve quite good resultscompared to commercial version controls:

We don't compare to Git here since what we are running are purecentralized operations (direct checkin or commit if you prefer)something Git can't do. Plastic SCM can work in Git or SVN modes, localrepos or direct connection to a central server.

In fact, some of our highest loaded servers on production run onLinux/Mono serving more than 3000 developers on a big enterprise setup.A single server handles most of the load singlehandedly :-)

Mono is indeed for code monkeys

If you’ve read the whole use case you already know that we have beenusing Mono for the purpose of providing a full version control stacksince 2006! That is for almost 13 years, right after the company wasfounded and the first product of our portfolio was delivered.

After all this time it has helped us build and distribute the sameproduct across the different environments: a full stack version controlsystem that is pioneering software configuration management in manyareas. The results are there and hey, we are demanding: versatility,performance, scalability and resilience are not an option for ourclients, or us. Given the structural relevance of SCM tools to anysoftware project, it is paramount for Plastic SCM to deliver a solidproduct across all platforms, and we do it. To us Mono = cross-platformand that is a huge advantage since we can focus on functionality,roadmap and support while Mono makes the product one same experienceeverywhere. Mono is definitely a foundational part of toolkit.

By virtue of using LLVM, Mono has access to a wide suite of tools andoptimization backends. A lot of active research uses LLVM IR. One suchresearch project, Souper, tries to brute-force a search for missedoptimizations in our emitted code. The .NET community may have softwareprojects that benefit from using Souper directly to generate code, ratherthan waiting for us to find ways to automate those optimizations ourselves.This algorithm can generate code that would be very challengingfor a traditional compiler to find.

The Mono .NET VM is a rather nimble beast. Rather than requiring allusers to live with the performance characteristics of a given policy, weoften choose to create multiple backends and bindings that exploitwhat’s best of the native platform while presenting a commoninterface. Part of this is the choice between using an interpreter, aJust-In-Time compiler, or an Ahead-Of-Time compiler.

AOT compilation is attractive to some projects for thecombination of optimized code with low start-up time. This is theclassic advantage of native code over code from a JIT or interpreter.AOT code is often much worse than code from a JIT because of a need forindirection in code that references objects in run-time memory. It’s importantfor AOT code to exploit every possible optimization to make up for thisdisadvantage. For this, we increasingly rely on optimizations performed by LLVM.

LLVM’s optimization passes can analyze a program globally. It is able to seethrough layers of abstractions and identify repeated or needless operations in a program’sglobal flow. Likewise, it can examine the operations in a small segment of code andmake them perfect with respect to one another. Sometimes though, we fail to optimize code.Classic compilers work by analyzing the control-flow and dataflow of a programand matching on specific patterns such as stores to variables that aren’t used laterand constants that are stored to variables rather than being propagated everywherethey can be. If the pattern matches, the transformation can take place.Sometimes the code we feed into LLVM does not match the patternsof inefficiency that it looks for, and we don’t get an optimization.

What’s worse is that we don’t know that we hit this optimization blocker. Wedon’t know what we expect from code until it’s a problem and we’rereally spending time optimizing it. Spotting trends in generated machinecode across thousands of methods is incredibly labor intensive. Often onlyreally bad code that runs many many times will catch attention. Fixing everysingle missed optimization and finding every single missed optimization becomesa chicken-and-egg problem.

The solution to some manifestations of this problem is the use ofsuperoptimizers. The academic discipline of superoptimizers is veryold. The idea is to treat the code that was written as moreof a restriction, a specification. The superoptimizer generates a tonof native code and checks the ways in which it behavesdifferently than the written code. If it can generate a faster nativecode sequence than the compiler generated while keeping behavior exactly thesame, it wins.

This “exactly the same” part can be incredibly expensive if not donecorrectly. The computational effort involved has historically kept superoptimization from beingused very often. Since then, it has gotten a lot easier to run computationally intensive jobs.Computer hardware has become orders of magnitude more powerful.Theorems around equivalence checking and control-flowrepresentations made more powerful claims and used algorithms with better running times.We are therefore seeing superoptimization research reemerge at this time.

One superoptimizer in particular, namedSouper, hasreached maturity while interoperating with the industry standardcode generator (LLVM) and the industry standard SMT engine (Z3). It haskickstarted a renewed faith in researchers that superoptimization is a reasonablepolicy. It can take the LLVM IR that a compiler was going to feed into LLVM, andcompute better IR. This can sometimes take a lot of time, and thecode emitted is the result of a process that isn’t auditable. The pipeline isplacing total faith in Souper for the correctness of generatedcode.

It’s mostly useful for compiler engineers to use to tell thatoptimizations were missed, and to identify how to fix that usingconventional pattern matching over the program’s control-flow and dataflow graphs.That said, Souper offers the ability todrop in for clang and to generate the code that is run. Some projects areeager to make any trade-offs for performance that are acceptable.Other projects may want to get a feel for how fast they could runif they were to invest making sure Mono generates good code.If the compile time increase doesn’t discourage them, manyprojects may find some benefit in such an optimizing compiler.

I recommend that curious readers install Z3, get a checkout of

https://github.com/google/souper,

and complete the compilation process described in that documentation.

When AOTing code with Mono, they’re going to want to pass thecommandline flags named there into the---aot=llvmopts= argument.

As of the time of this writing, that is

llvmopts="-load /path/to/libsouperPass.so -souper -z3-path=/usr/bin/z3"

Mono will then allow Souper to step in during the middle of the LLVMcompilation and try it’s best at brute-forcing some better code. Ifthere’s anything short and fast that does the job better, it will befound.

It is frankly amazing that Mono can get such extensive optimizations simplyby compiling to LLVM IR. Without changing a single line of Mono’s source, wechanged our compilation pipeline in truly dramatic ways. This shows off thelack of expectations that Mono has about the layout of our generated code. This shows off theflexibility of LLVM as a code generation framework and to Mono as an embedded runtime.Embedders using Mono should considerusing our LLVM backend with this and other third-party LLVM optimization passes.Feedback about the impact of our research on real-world programs will help us decide what weshould be using by default.

During the 2018 Microsoft Hack Week, members of the Mono team explored the ideaof replacing the Mono’s code generation engine written in C with a codegeneration engine written in C#.

In this blog post we describe our motivation, the interface between the nativeMono runtime and the managed compiler and how we implemented the new managedcompiler in C#.

Motivation

Mono’s runtime and JIT compiler are entirely written in C, a highly portablelanguage that has served the project well.Yet, we feel jealous of our own users that get to write code in a high-levellanguage and enjoy the safety, the luxury and reap the benefits of writing codein a high-level language, while the Mono runtime continues to be written in C.

We decided to explore whether we could make Mono’s compilation engine pluggableand then plug a code generator written entirely in C#.If this were to work, we could more easily prototype, write new optimizationsand make it simpler for developers to safely try changes in the JIT.

This idea has been explored by research projects like theJikesRVM,Maxime andGraal for Java.In the .NET world, the Unity team wrote an IL compiler to C++ compiler calledil2cpp.They also experimented with amanagedJITrecently.

In this blog post, we discuss the prototype that we built.The code mentioned in this blog post can be found here:https://github.com/lambdageek/mono/tree/mjit/mcs/class/Mono.Compiler

Interfacing with the Mono Runtime

The Mono runtime provides various services, just-in-time compilation, assemblyloading, an IO interface, thread management and debugging capabilities.The code generation engine in Mono is calledmini and is used both for staticcompilation and just-in-time compilation.

Mono’s code generation has a number of dimensions:

• Code can be either interpreted, or compiled to native code
• When compiling to native code, this can be done just-in-time, or it can be batch compiled, also known as ahead-of-time compilation.
• Mono today has two code generators, the light and fastmini JIT engine, and the heavy duty engine based on the LLVM optimizing compiler. These two are not really completely unaware of the other, Mono’s LLVM support reuses many parts of themini engine.

This project started with a desire to make this division even more clear, andto swap up the native code generation engine in ‘mini’ with one that could becompletely implemented in a .NET language.In our prototype we used C#, but other languages like F# or IronPython could be used as well.

To move the JIT to the managed world, we introduced theICompiler interfacewhich must be implemented by your compilation engine, and it is invoked ondemand when a specific method needs to be compiled.

This is the interface that you must implement:

interface ICompiler {    CompilationResult CompileMethod (IRuntimeInformation runtimeInfo,                                     MethodInfo methodInfo,                                     CompilationFlags flags,                                     out NativeCodeHandle nativeCode);    string Name { get; }}

TheCompileMethod () receives aIRuntimeInformation reference, whichprovides services for the compiler as well as aMethodInfo that representsthe method to be compiled and it is expected to set thenativeCode parameterto the generated code information.

TheNativeCodeHandle merely represents the generated code address and its length.

This is theIRuntimeInformation definition, which shows the methods availableto theCompileMethod to perform its work:

interface IRuntimeInformation {    InstalledRuntimeCode InstallCompilationResult (CompilationResult result,                                                   MethodInfo methodInfo,                                                  NativeCodeHandle codeHandle);    object ExecuteInstalledMethod (InstalledRuntimeCode irc,                                   params object[] args);    ClassInfo GetClassInfoFor (string className);    MethodInfo GetMethodInfoFor (ClassInfo classInfo, string methodName);    FieldInfo GetFieldInfoForToken (MethodInfo mi, int token);    IntPtr ComputeFieldAddress (FieldInfo fi);    /// For a given array type, get the offset of the vector relative to the base address.    uint GetArrayBaseOffset(ClrType type);}

We currently have one implementation ofICompiler, we call it the the “BigStep” compiler.When wired up, this is what the process looks like when we compile a method with it:

Themini runtime can call into managed code viaCompileMethod upon acompilation request.For the code generator to do its work, it needs to obtain some informationabout the current environment.This information is surfaced by theIRuntimeInformation interface.Once the compilation is done, it will return a blob of native instructions tothe runtime.The returned code is then “installed” in your application.

Now there is a trick question: Who is going to compile the compiler?

The compiler written in C# is initially executed with one of the built-inengines (either theinterpreter,or the JIT engine).

The BigStep Compiler

Our firstICompiler implementation is called theBigStepcompiler.

This compiler was designed and implemented by a developer (Ming Zhou) notaffiliated with Mono Runtime Team.It is a perfect showcase of how the work we presented through this project canquickly enable a third-party to build their own compiler without much hassleinteracting with the runtime internals.

The BigStep compiler implements an IL to LLVM compiler.This was convenient to build the proof of concept and ensure that the designwas sound, while delegating all the hard compilation work to the LLVM compilerengine.

A lot can be said when it comes to the design and architecture of a compiler,but our main point here is to emphasize how easy it can be, with what we havejust introduced to Mono runtime, to bridge IL code with a customized backend.

The IL code is streamed into to the compiler interface through an iterator,with information such as op-code, index and parameters immediately available tothe user.See below for more details about the prototype.

Hosted Compiler

Another beauty of moving parts of the runtime to the managed side is that wecan test the JIT compilerwithout recompiling the native runtime, soessentially developing a normal C# application.

TheInstallCompilationResult () can be used to register compiled method withthe runtime and theExecuteInstalledMethod () are can be used to invoke amethod with the provided arguments.

Here is an example how this is used code:

public static int AddMethod (int a, int b) {  return a + b;}[Test]public void TestAddMethod (){  ClassInfo ci = runtimeInfo.GetClassInfoFor (typeof (ICompilerTests).AssemblyQualifiedName);  MethodInfo mi = runtimeInfo.GetMethodInfoFor (ci, "AddMethod");  NativeCodeHandle nativeCode;  CompilationResult result = compiler.CompileMethod (runtimeInfo, mi, CompilationFlags.None, out nativeCode);  InstalledRuntimeCode irc = runtimeInfo.InstallCompilationResult (result, mi, nativeCode);  int addition = (int) runtimeInfo.ExecuteInstalledMethod (irc, 1, 2);  Assert.AreEqual (addition, 3);}

We can ask the host VM for the actual result, assuming it’s our gold standard:

int mjitResult = (int) runtimeInfo.ExecuteInstalledMethod (irc, 666, 1337);int hostedResult = AddMethod (666, 1337);Assert.AreEqual (mjitResult, hostedResult);

This eases development of a compiler tremendously.

We don’t need to eat our own dog food during debugging, but when we feel readywe can flip a switch and use the compiler as our system compiler.This is actually what happens if you runmake -C mcs/class/Mono.Compiler run-testin themjit branch: We use thisAPI to test the managed compiler while running on the regular Mini JIT.

Native to Managed to Native: Wrapping Mini JIT intoICompiler

As part of this effort, we also wrapped Mono’s JIT in theICompiler interface.

MiniCompiler calls back into native code and invokes the regular Mini JIT.It works surprisingly well, however there is a caveat: Once back in the nativeworld, the Mini JIT doesn’t need to go throughIRuntimeInformation and justuses its old ways to retrieve runtime details.Though, we can turn this into an incremental process now: We can identify thoseparts, add them toIRuntimeInformation and change Mini JIT so that it usesthe new API.

Conclusion

We strongly believe in a long-term value of this project.A code base in managed code is more approachable for developers and thus easierto extend and maintain.Even if we never see this work upstream, it helped us to better understand theboundary between runtime and JIT compiler, and who knows, it might will help usto integrate RyuJIT into Mono one day 😉

We should also note thatIRuntimeInformation can be implemented by any other.NET VM: HelloCoreCLR folks 👋

If you are curious about this project, ping us on ourGitterchannel.

Appendix: Converting Stack-Based OpCodes into Register Operations

Since the target language was LLVM IR, we had to build a translator thatconverted the stack-based operations from IL into the register-based operationsof LLVM.

Since many potential target are register based, we decided to design aframework to make it reusable of the part where we interpret the IL logic. Tothis goal, we implemented an engine to turn the stack-based operations into theregister operations.

Consider theADD operation in IL.This operation pops two operands from the stack, performing addition and pushing back the result to the stack. This is documented in ECMA 335 as follows:

  Stack Transition:      ..., value1, value2 -> ..., result

The actual kind of addition that is performed depends on the types of thevalues in the stack.If the values are integers, the addition is an integer addition.If the values are floating point values, then the operation is a floating pointaddition.

To re-interpret this in a register-based semantics, we treat each pushed framein the stack as a different temporary value.This means if a frame is popped out and a new one comes in, although it has thesame stack depth as the previous one, it’s a new temporary value.

Each temporary value is assigned a unique name.Then an IL instruction can be unambiguously presented in a form using temporary names instead of stack changes.For example, theADD operation becomes

Temp3 := ADD Temp1 Temp2

Other than coming from the stack, there are other sources of data duringevaluation: local variables, arguments, constants and instruction offsets (usedfor branching).These sources are typed differently from the stack temporaries, so that thedownstream processor (to talk in a few) can properly map them into theircontext.

A third problem that might be common among those target languages is thejumping target for branching operations.IL’s branching operation assumes an implicit target should the result be taken:The next instruction.But branching operations in LLVM IR must explicitly declare the targets forboth taken and not-taken paths.To make this possible, the engine performs a pre-pass before the actualexecution, during which it gathers all the explicit and implicit targets.In the actual execution, it will emit branching instructions with both targets.

As we mentioned earlier, the execution engine is a common layer that merelytranslates the instruction to a more generic form.It then sends out each instruction toIOperationProcessor, an interface thatperforms actual translation.Comparing to the instruction received fromICompiler, the presentation here,OperationInfo, is much more consumable:In addition to the op codes, it has an array of the input operands, and a result operand:

public class OperationInfo{  ... ...  internal IOperand[] Operands { get; set; }  internal TempOperand Result { get; set; }  ... ...}

There are several types of the operands:ArgumentOperand,LocalOperand,ConstOperand,TempOperand,BranchTargetOperand, etc.Note that the result, if it exists, is always aTempOperand.The most important property onIOperand is itsName, which unambiguouslydefines the source of data in the IL runtime.If an operand with the same name comes in another operation, it unquestionablytells us the very same data address is targeted again.It’s paramount to the processor to accurately map each name to its own storage.

The processor handles each operand according to its type.For example, if it’s an argument operand, we might consider retrieving thevalue from the corresponding argument.An x86 processor may map this to a register.In the case of LLVM, we simply go to fetch it from a named value that ispre-allocated at the beginning of method construction.The resolution strategy is similar for other operands:

• LocalOperand: fetch the value from pre-allocated address
• ConstOperand: use the const value carried by the operand
• BranchTargetOperand: use the index carried by the operand

Since the temp value uniquely represents an expression stack frame from CLRruntime, it will be mapped to a register.Luckily for us, LLVM allows infinite number of registers, so we simply name anew one for each different temp operand.If a temp operand is reused, however, the very same register must as well.

We useLLVMSharp binding tocommunicate with LLVM.

Note: This is a guest post by Calvin Buckley (@NattyNarwhal on GitHub)introducing the community port of Mono to IBM AIX and IBM i. If you’d like tohelp with this community port please contact the maintainers on Gitter.

You might have noticed this in theMono 5.12 release notes,Mono now includes support forIBM AIX and IBM i; two verydifferent yet (mostly!) compatible operating systems. This post should serve asan introduction to this port.

What does it take to port Mono?

Porting Mono to a new operating system is not as hard as you might think!Pretty much the entire world is POSIX compliant these days, and Mono is a largeyet manageable codebase due to a low number of dependencies, use of plain C99,and an emphasis on portability. Most common processor architectures in use aresupported by the code generator, though more obscure ISAs will have some caveats.

Pretty much all of the work you do will be twiddling#ifdefs to accommodate forthe target platform’s quirks; such as missing or different preprocessordefinitions and functions, adding the platform to definitions so it issupported by core functionality, and occasionally having to tweak the runtimeor build system to handle when the system does something completely differentlythan others. In the case of AIX and IBM i, I had to do all of these things.

Where would I be without IBM?

For some background on what needed to happen, we can start by giving somebackground on our target platforms.

Both of our targets run on 64-bit PowerPC processors in big endian mode.Mono does support PowerPC,and Bernhard Urban maintains it. What is odd about the calling conventions onAIX (shared occasionally by Linux) is the use offunction descriptors, whichmeans that pointers to functions do not point to code, but instead point tometadata about them. This can cause bugs in the JIT if you are not careful toconsume or produce function descriptors instead of raw pointers when needed.Because the runtime is better tested on 64-bit PowerPC, and machines are fastenough that the extra overhead is not significant, we always build a 64-bitruntime.

In addition to a strange calling convention, AIX also has a different binaryformat - that means that currently, the ahead-of-time compiler does not work.While most Unix-like operating systems use ELF, AIX (and by extension, IBM ifor the purposes of this port) use XCOFF, a subset of the Windows PE binaryformat.

AIX is a Unix (descended from the System V rather than the BSD side of thefamily) that runs on PowerPC systems. Despite being a Unix, it has some quirksof its own, that I will describe in this article.

Unix? What’s a Unix?

IBM i (formerly known as i5/OS or OS/400) is decidedly not a Unix. Unlike Unix,it has an object-based filesystem where all objects are mapped into a singlehumongous address space, backed on disk known as single level storage – realmain storage (RAM) holds pages of objects “in use” and acts as a cache forobjects that reside permanently on disk. Instead of flat files, IBM i usesdatabase tables as the means to store data. (On IBM i, all files are databasetables, and a file is just one of the “object types” supported by IBM i; othersinclude libraries and programs.) Programs on IBM i are not simple nativebinaries, but instead are “encapsulated” objects that contain an intermediateform, called Machine Interface instructions, (similar to MSIL/CIL) that is thentranslated and optimized ahead-of-time for the native hardware (or upon firstuse); this also provides part of the security model and has allowed users totransition from custom CISC CPUs to enhanced PowerPC variants, without havingto recompile their programs from the original source code.

This sounds similar to running inside of WebAssembly rather than any kind ofUnix – So, then, how do you port programs dependent on POSIX? IBM i provides anenvironment called PASE (Portable Application Solutions Environment) thatprovides binary compatibility for AIX executables, for a large subset of theAIX ABI, within the IBM i. But Unix and IBM i are totally different; Unix hasfiles and per-process address spaces, and IBM i normally does not, so how doyou make these incongruent systems work?

To try to bridge the gap, IBM i also has an “Integrated File System” thatsupports byte-stream file objects in a true hierarchical file system directoryhierarchy. For running Unix programs that expect their own address space, IBM iprovides something called “teraspace” that provides a large private addressspace per process or job. This requires IBM i to completely changes the MMUmode and does a cache/TLB flush every time it enters and exits the Unix world,making system calls somewhat expensive; in particular, forking and I/O. Whilesome system calls are not implemented, there are more than enough to portnon-trivial AIX programs to the PASE environment, even with its quirks andperformance limitations. You could even build them entirely inside of the PASEenvironment.

A port to the native IBM i environment outputting MI code with the ahead oftime compiler has been considered, but would take a lot of work to write an MIbackend for the JIT, use the native APIs in the runtime, and handle how theenvironment is different from anything else Mono runs on. As such, I insteadPASE and AIX for the ease of porting existing POSIX compatible code.

What happened to port it?

The port came out of some IBM i users expressing an interest in wanting to run.NET programs on their systems. A friend of mine involved in the IBM icommunity had noticed I was working on a (mostly complete, but not fullyworking) Haiku port, and approached me to see if it could be done. Consideringthat that I now had experience with porting Mono to new platforms, and therewas already a PowerPC JIT, I decided to take the challenge.

The primary porting target was IBM i, with AIX support being a by-product.Starting by building on IBM i, I set up a chroot environment to work in,(chroot support was added to PASE fairly recently), setting up a toolchain withAIX packages. Initial bring-up of the port happened on IBM i, up to the pointwhere the runtime was built, but execution of generated code was not happening.One problem with building on IBM i, however, is that the performancelimitations really start to show. While building took the same amount of timeon the system I had access to (dual POWER6, taking about roughly 30 minutes tobuild the runtime) as AIX due to it mostly being computation, the configurescript was extremely impacted due to its emphasis on many small reads andwrites with lots of forking. Whereas it took AIX 5 minutes and Linux 2 minutesto run through the configure script, it took IBM i well over an hour to runthrough all of it. (Ouch!)

At this point, I submitted the initial branch as a pull request for review. Alot of back and forth went on to work on the underlying bugs as well asfollowing proper style and practices for Mono. I set up an AIX VM on themachine, and switched to cross-compiling from AIX to IBM i; targeting bothplatforms with the same source and binary. Because I was not building on IBM iany longer, I had to periodically copy binaries over to IBM i, to check if Monowas using missing libc functions or system calls, or if I had tripped on somebehaviour that PASE exhibits differently from AIX. With the improved iterationtime, I could start working on the actual porting work much more quickly.

To help with matters where I was unsure exactly how AIX worked, David Edelsohnfrom IBM helped by explaining how AIX handles things like calling conventions,libraries, issues with GCC, and best practices for dealing with porting thingsto AIX.

What needed to change?

There are some unique aspects of AIX and the subset that PASE provides, beyondthe usual#ifdef handling.

What did we start with?

One annoyance I had was how poor the GNU tools are on AIX. GNU binutils areeffectively useless on AIX, so I had to explicitly use IBM’s binutils, and dealwith some small problems related to autotools with environment variables andassumption of GNU ld features in makefiles. I had also dealt with some issuesin older versions of GCC (which is actually fairly well supported on AIX, allthings considered) that made me upgrade to a newer version. However, GCC’s“fixincludes” tool to try to mend GCC compatibility issues in system headerfiles in fact mangled them, causing them to be missing some definitions foundin libraries. (Sometimes they were in libc, but never defined in the headers inthe first place!)

Improper use of function pointers was sometimes a problem. Based on the adviceof Bernhard, there was a problem with the function descriptors#ifdefs, whichhad caused a mix-up interpreting function pointers as code. Once that had beenfixed, Mono was running generated code on AIX for the first time – quite asight to behold!

What’s a “naidnE?”

One particularly nerve-racking issue that bugged me while trying to bootstrapwas with the Decimal type returning a completely bogus value when dividing,causing a non-sense overflow condition. Because of constant inlining, thisoccurred when building the BCL, so it was hard to put off. With some carefuldebugging from my friend, comparing the variable state between x86 and PPC whendividing a decimal, we had determined exactly where the incorrect endiannesshandling had taken place and I had came up with a fix.

While Mono has historically handled different endianness just fine, Mono hasstarted to replace portions of its own home-grown BCL with CoreFX, (theopen-source Microsoft BCL) and it did not have the same rigor towardsendianness issues. Mono does patch CoreFX code, but it sometimes pulls in newcode that has not had endianness (or other such possible compatibility issues)worked out yet and thus requires further patching. In this case, the code hadalready been fixed for big endian before, but pulling in updated code fromCoreFX had created a new problem with endianness.

What’s “RTLD_MEMBER?”

On AIX, there are two ways to handle libraries. One is your typical System Vstyle linking with .so libraries; this isn’t used by default, but can beforced. The other way is the “native” way to do it, where objects are storedin an archive (.a) typically used for holding objects used for static linking.Because AIX always uses position-independent code, multiple objects arecombined into a single object and then inserted into the archive. You can thenaccess the library like normal. Using this technique, you can even fit multipleshared objects of the same version into a single archive! This took onlyminimal changes to support; I only had to adjust the dynamic library loader totell it to look inside of archive files, and some build system tweaks to pointit to the proper archive and objects to look for. (Annoyingly, we have tohardcode some version names of library objects. Even then, the build systemstill needs revision for cases when it assumes that library names are just thename and an extension.)

What’s “undefined behaviour?”

When Mono tries to access an object reference, and the reference (a pointer) isnull, (that is, zero) Mono does not normally check to see if the pointer isnull. On most operating systems, when a process accesses invalid memory such asa null pointer, it sends the process a signal (such as SIGSEGV) and if theprogram does not handle that signal, it will terminate the program. Normally,Mono registers a signal handler, and instead of checking for null, it wouldjust try to dereference a null pointer anyways to let the signal handlerinterrupt and return an exception to managed code instead. AIX doesn’t do that– it lets programs dereference null pointers anyway! What gives?

Accessing memory via a null pointer is not actually defined by the ANSI Cstandards – this is a case of a dreadedundefined behaviour. Mono relied onthe assumption that most operating systems did it in the typical way of sendinga signal to the process. What AIX instead does is to implement a “null page”mapped at 0x0 and accepts reads and writes to it. (You could also execute fromit, but since all zeroes is an invalid opcode on PowerPC, this does not do muchbut throw an illegal instruction signal at the process.) This is a historicaldecision, relating back to code optimizations implemented in older IBMcompilers made where they used speculative execution in compiler-generated codeduring the 1980s for improved performance when evaluating complex logicalexpressions. Because we cannot rely on handling a signal to catch the nulldereference, we can instead force the behaviour to always check if pointers arenull, (normally reserved for runtime debugging) to be on all the time.

What’s so boring about TLS?

BoringSSL is required to get modern TLS required by newer websites. The buildsystem, instead of autotools and make, is CMake based. Luckily, this workedfine on AIX, though I had to apply some massaging for it to do 64-bit librarymangling. For a while, I was stumped by an illegal instruction error, thatturned out to be due to not linking in pthread to the library, and it notwarning about it.

It turns out that even though BoringSSL was now working, one cipher suite(secp256r1) was not, so sites using that cipher were broken. To try to test it,I had gone “yak shaving” to build what was needed for the test harnessaccording to the README; Ninja and Go. I had a heck of a time trying to buildGo on a PPC Linux system to triage, but as it turned out, I did not actuallyneed it anyway – Mono had tweaked the build system so that it was not neededafter all; I just had to flip a CMake flag to let it build the tests and runthem manually. After figuring out what exactly was wrong, it turned out to bean endianness issue in an optimized path. A fix was attempted for it, but inthe end, only disabling it worked and let the cipher run fine on big endianPowerPC. Since the code came from Google code that has been rewritten in bothBoringSSL and OpenSSL upstream’s latest sources, it is due to be replaced thenext time Mono’s BoringSSL fork gets updated.

What else?

I had an issue with I/O getting some spurious and strange issues withthreading. Threads would complain that they had an unexpected errno of 0.(indicating success) What happened was that AIX does not assume that allprograms are thread-safe by default, so errno was not thread-local. One small#define later, and that was fixed. (Miguel de Icaza was amused that someoperating systems still consider thread safety to be an advanced feature. 🙂)

We also found a cosmetic issue with uname. Most Unices put their version in therelease field of the uname structure, and things like the kernel type in theversion field. AIX and PASE however, put the major version in the version field,and the minor version in the release field. A simplesprintf for the AIX casewas enough to fix this.

PASE has many quirks – this necessitated some patches to work arounddeficiencies; from bugs to unimplemented functions. I aim to target IBM i 7.1or newer, so I worked around some bugs that have been fixed in newer versions.A lot of this I cleaned up with some more preprocessor definitions.

What’s next?

Now that Mono runs on these platforms, there’s still a lot of work left to bedone. The ahead of time compiler needs to be reworked to emit XCOFF-compatiblecode, libgdiplus needs to be ported, Roslyn is broken on ppc64be, continuousintegration would be useful to detect build failures, the build system is stilla bit weird regarding AIX libraries, and plenty more where that came from.Despite all this, the fact the port works well enough already in its currentstate should provide a solid foundation to work with, going forward.