You might have noticed this in theMono 5.12 release notes,Mono now includes support forIBM AIX and IBM i; two verydifferent yet (mostly!) compatible operating systems. This post should serve asan introduction to this port.

What does it take to port Mono?

Porting Mono to a new operating system is not as hard as you might think!Pretty much the entire world is POSIX compliant these days, and Mono is a largeyet manageable codebase due to a low number of dependencies, use of plain C99,and an emphasis on portability. Most common processor architectures in use aresupported by the code generator, though more obscure ISAs will have some caveats.

Pretty much all of the work you do will be twiddling#ifdefs to accommodate forthe target platform’s quirks; such as missing or different preprocessordefinitions and functions, adding the platform to definitions so it issupported by core functionality, and occasionally having to tweak the runtimeor build system to handle when the system does something completely differentlythan others. In the case of AIX and IBM i, I had to do all of these things.

Where would I be without IBM?

For some background on what needed to happen, we can start by giving somebackground on our target platforms.

Both of our targets run on 64-bit PowerPC processors in big endian mode.Mono does support PowerPC,and Bernhard Urban maintains it. What is odd about the calling conventions onAIX (shared occasionally by Linux) is the use offunction descriptors, whichmeans that pointers to functions do not point to code, but instead point tometadata about them. This can cause bugs in the JIT if you are not careful toconsume or produce function descriptors instead of raw pointers when needed.Because the runtime is better tested on 64-bit PowerPC, and machines are fastenough that the extra overhead is not significant, we always build a 64-bitruntime.

In addition to a strange calling convention, AIX also has a different binaryformat - that means that currently, the ahead-of-time compiler does not work.While most Unix-like operating systems use ELF, AIX (and by extension, IBM ifor the purposes of this port) use XCOFF, a subset of the Windows PE binaryformat.

AIX is a Unix (descended from the System V rather than the BSD side of thefamily) that runs on PowerPC systems. Despite being a Unix, it has some quirksof its own, that I will describe in this article.

Unix? What’s a Unix?

IBM i (formerly known as i5/OS or OS/400) is decidedly not a Unix. Unlike Unix,it has an object-based filesystem where all objects are mapped into a singlehumongous address space, backed on disk known as single level storage – realmain storage (RAM) holds pages of objects “in use” and acts as a cache forobjects that reside permanently on disk. Instead of flat files, IBM i usesdatabase tables as the means to store data. (On IBM i, all files are databasetables, and a file is just one of the “object types” supported by IBM i; othersinclude libraries and programs.) Programs on IBM i are not simple nativebinaries, but instead are “encapsulated” objects that contain an intermediateform, called Machine Interface instructions, (similar to MSIL/CIL) that is thentranslated and optimized ahead-of-time for the native hardware (or upon firstuse); this also provides part of the security model and has allowed users totransition from custom CISC CPUs to enhanced PowerPC variants, without havingto recompile their programs from the original source code.

This sounds similar to running inside of WebAssembly rather than any kind ofUnix – So, then, how do you port programs dependent on POSIX? IBM i provides anenvironment called PASE (Portable Application Solutions Environment) thatprovides binary compatibility for AIX executables, for a large subset of theAIX ABI, within the IBM i. But Unix and IBM i are totally different; Unix hasfiles and per-process address spaces, and IBM i normally does not, so how doyou make these incongruent systems work?

To try to bridge the gap, IBM i also has an “Integrated File System” thatsupports byte-stream file objects in a true hierarchical file system directoryhierarchy. For running Unix programs that expect their own address space, IBM iprovides something called “teraspace” that provides a large private addressspace per process or job. This requires IBM i to completely changes the MMUmode and does a cache/TLB flush every time it enters and exits the Unix world,making system calls somewhat expensive; in particular, forking and I/O. Whilesome system calls are not implemented, there are more than enough to portnon-trivial AIX programs to the PASE environment, even with its quirks andperformance limitations. You could even build them entirely inside of the PASEenvironment.

A port to the native IBM i environment outputting MI code with the ahead oftime compiler has been considered, but would take a lot of work to write an MIbackend for the JIT, use the native APIs in the runtime, and handle how theenvironment is different from anything else Mono runs on. As such, I insteadPASE and AIX for the ease of porting existing POSIX compatible code.

What happened to port it?

The port came out of some IBM i users expressing an interest in wanting to run.NET programs on their systems. A friend of mine involved in the IBM icommunity had noticed I was working on a (mostly complete, but not fullyworking) Haiku port, and approached me to see if it could be done. Consideringthat that I now had experience with porting Mono to new platforms, and therewas already a PowerPC JIT, I decided to take the challenge.

The primary porting target was IBM i, with AIX support being a by-product.Starting by building on IBM i, I set up a chroot environment to work in,(chroot support was added to PASE fairly recently), setting up a toolchain withAIX packages. Initial bring-up of the port happened on IBM i, up to the pointwhere the runtime was built, but execution of generated code was not happening.One problem with building on IBM i, however, is that the performancelimitations really start to show. While building took the same amount of timeon the system I had access to (dual POWER6, taking about roughly 30 minutes tobuild the runtime) as AIX due to it mostly being computation, the configurescript was extremely impacted due to its emphasis on many small reads andwrites with lots of forking. Whereas it took AIX 5 minutes and Linux 2 minutesto run through the configure script, it took IBM i well over an hour to runthrough all of it. (Ouch!)

At this point, I submitted the initial branch as a pull request for review. Alot of back and forth went on to work on the underlying bugs as well asfollowing proper style and practices for Mono. I set up an AIX VM on themachine, and switched to cross-compiling from AIX to IBM i; targeting bothplatforms with the same source and binary. Because I was not building on IBM iany longer, I had to periodically copy binaries over to IBM i, to check if Monowas using missing libc functions or system calls, or if I had tripped on somebehaviour that PASE exhibits differently from AIX. With the improved iterationtime, I could start working on the actual porting work much more quickly.

To help with matters where I was unsure exactly how AIX worked, David Edelsohnfrom IBM helped by explaining how AIX handles things like calling conventions,libraries, issues with GCC, and best practices for dealing with porting thingsto AIX.

What needed to change?

There are some unique aspects of AIX and the subset that PASE provides, beyondthe usual#ifdef handling.

What did we start with?

One annoyance I had was how poor the GNU tools are on AIX. GNU binutils areeffectively useless on AIX, so I had to explicitly use IBM’s binutils, and dealwith some small problems related to autotools with environment variables andassumption of GNU ld features in makefiles. I had also dealt with some issuesin older versions of GCC (which is actually fairly well supported on AIX, allthings considered) that made me upgrade to a newer version. However, GCC’s“fixincludes” tool to try to mend GCC compatibility issues in system headerfiles in fact mangled them, causing them to be missing some definitions foundin libraries. (Sometimes they were in libc, but never defined in the headers inthe first place!)

Improper use of function pointers was sometimes a problem. Based on the adviceof Bernhard, there was a problem with the function descriptors#ifdefs, whichhad caused a mix-up interpreting function pointers as code. Once that had beenfixed, Mono was running generated code on AIX for the first time – quite asight to behold!

What’s a “naidnE?”

One particularly nerve-racking issue that bugged me while trying to bootstrapwas with the Decimal type returning a completely bogus value when dividing,causing a non-sense overflow condition. Because of constant inlining, thisoccurred when building the BCL, so it was hard to put off. With some carefuldebugging from my friend, comparing the variable state between x86 and PPC whendividing a decimal, we had determined exactly where the incorrect endiannesshandling had taken place and I had came up with a fix.

While Mono has historically handled different endianness just fine, Mono hasstarted to replace portions of its own home-grown BCL with CoreFX, (theopen-source Microsoft BCL) and it did not have the same rigor towardsendianness issues. Mono does patch CoreFX code, but it sometimes pulls in newcode that has not had endianness (or other such possible compatibility issues)worked out yet and thus requires further patching. In this case, the code hadalready been fixed for big endian before, but pulling in updated code fromCoreFX had created a new problem with endianness.

What’s “RTLD_MEMBER?”

On AIX, there are two ways to handle libraries. One is your typical System Vstyle linking with .so libraries; this isn’t used by default, but can beforced. The other way is the “native” way to do it, where objects are storedin an archive (.a) typically used for holding objects used for static linking.Because AIX always uses position-independent code, multiple objects arecombined into a single object and then inserted into the archive. You can thenaccess the library like normal. Using this technique, you can even fit multipleshared objects of the same version into a single archive! This took onlyminimal changes to support; I only had to adjust the dynamic library loader totell it to look inside of archive files, and some build system tweaks to pointit to the proper archive and objects to look for. (Annoyingly, we have tohardcode some version names of library objects. Even then, the build systemstill needs revision for cases when it assumes that library names are just thename and an extension.)

What’s “undefined behaviour?”

When Mono tries to access an object reference, and the reference (a pointer) isnull, (that is, zero) Mono does not normally check to see if the pointer isnull. On most operating systems, when a process accesses invalid memory such asa null pointer, it sends the process a signal (such as SIGSEGV) and if theprogram does not handle that signal, it will terminate the program. Normally,Mono registers a signal handler, and instead of checking for null, it wouldjust try to dereference a null pointer anyways to let the signal handlerinterrupt and return an exception to managed code instead. AIX doesn’t do that– it lets programs dereference null pointers anyway! What gives?

Accessing memory via a null pointer is not actually defined by the ANSI Cstandards – this is a case of a dreadedundefined behaviour. Mono relied onthe assumption that most operating systems did it in the typical way of sendinga signal to the process. What AIX instead does is to implement a “null page”mapped at 0x0 and accepts reads and writes to it. (You could also execute fromit, but since all zeroes is an invalid opcode on PowerPC, this does not do muchbut throw an illegal instruction signal at the process.) This is a historicaldecision, relating back to code optimizations implemented in older IBMcompilers made where they used speculative execution in compiler-generated codeduring the 1980s for improved performance when evaluating complex logicalexpressions. Because we cannot rely on handling a signal to catch the nulldereference, we can instead force the behaviour to always check if pointers arenull, (normally reserved for runtime debugging) to be on all the time.

What’s so boring about TLS?

BoringSSL is required to get modern TLS required by newer websites. The buildsystem, instead of autotools and make, is CMake based. Luckily, this workedfine on AIX, though I had to apply some massaging for it to do 64-bit librarymangling. For a while, I was stumped by an illegal instruction error, thatturned out to be due to not linking in pthread to the library, and it notwarning about it.

It turns out that even though BoringSSL was now working, one cipher suite(secp256r1) was not, so sites using that cipher were broken. To try to test it,I had gone “yak shaving” to build what was needed for the test harnessaccording to the README; Ninja and Go. I had a heck of a time trying to buildGo on a PPC Linux system to triage, but as it turned out, I did not actuallyneed it anyway – Mono had tweaked the build system so that it was not neededafter all; I just had to flip a CMake flag to let it build the tests and runthem manually. After figuring out what exactly was wrong, it turned out to bean endianness issue in an optimized path. A fix was attempted for it, but inthe end, only disabling it worked and let the cipher run fine on big endianPowerPC. Since the code came from Google code that has been rewritten in bothBoringSSL and OpenSSL upstream’s latest sources, it is due to be replaced thenext time Mono’s BoringSSL fork gets updated.

What else?

I had an issue with I/O getting some spurious and strange issues withthreading. Threads would complain that they had an unexpected errno of 0.(indicating success) What happened was that AIX does not assume that allprograms are thread-safe by default, so errno was not thread-local. One small#define later, and that was fixed. (Miguel de Icaza was amused that someoperating systems still consider thread safety to be an advanced feature. 🙂)

We also found a cosmetic issue with uname. Most Unices put their version in therelease field of the uname structure, and things like the kernel type in theversion field. AIX and PASE however, put the major version in the version field,and the minor version in the release field. A simplesprintf for the AIX casewas enough to fix this.

PASE has many quirks – this necessitated some patches to work arounddeficiencies; from bugs to unimplemented functions. I aim to target IBM i 7.1or newer, so I worked around some bugs that have been fixed in newer versions.A lot of this I cleaned up with some more preprocessor definitions.

What’s next?

Now that Mono runs on these platforms, there’s still a lot of work left to bedone. The ahead of time compiler needs to be reworked to emit XCOFF-compatiblecode, libgdiplus needs to be ported, Roslyn is broken on ppc64be, continuousintegration would be useful to detect build failures, the build system is stilla bit weird regarding AIX libraries, and plenty more where that came from.Despite all this, the fact the port works well enough already in its currentstate should provide a solid foundation to work with, going forward.