Moonlight 4 Preview 1 is out

Yesterday wereleasedMoonlight4, Preview 1.

This release of Moonlight completes the Moonlight 3 featureset and includes many features from Silverlight 4. Check outour release notes for the list of things thatarecurrentlymissing from our Silverlight 4 support.

Rendering

Moonlight rendering system uses a painter's algorithmcoupled with culling to reduce the amount of rasterizationthat needs to take place.

For example, if we had these objects all rendered at thesame location, on top of each other:

A simple implementation would render the triangle, then therectangle and then the circle, and we would get this result:

Moonlight optimizes this rendering by computing thebounding boxes of each element and determining which objectsare completely obscured. In this case, the triangle neverneeds to be rendered as it is fully covered.

Since we have the entire graph scene in memory, we can pushall the rendering to the X server without ever having to pulldata back from it.

Each visible element on Silverlight derives from the classUIElement and Moonlight tracks the bounding box for all ofeach individual element. As you compose elements, theaggregate bounding box is computed. For example, a canvasthat hosts these three UIElements would have a bounding boxthat contains all three of them:

New Rendering Features

With Silverlight 3, Microsoft introduced two largeimportant changes to the framework: 3D transformations onobjects and support for pixel shaders. Both of these areapplied to every visual element in Silverlight (this isimplemented in the class UIElement).

In addition to the properties inherited from Silverlight 2,UIElements now have two new properties: Projection and Effect.

The Projection property is a 3D matrix transformation (the3D variation of the 2D affine transform that is available inmost 2D graphics APIs). Silverlight exposes boththeraw3D matrix or a setofconvenientproperties that are easier to use and require nounderstanding of the interactions of the twelve elements ofthe 3D matrix(seethispage for an explanation).

Just like 2D affine APIs typically expose conveniencemethods to scale, rotate, skew and translate, thePlaneProjection properties allow developers to focus on thosecomponents.

You can see asamplehere.

Effects follow a similar pattern. The blur anddrop-shadow effects are given convenient names and accessors(BlurEffectandDropShadowEffectbut Silverlight exposes an API to create programmable pixelshaders that go beyond these two simple cases.

TheShaderEffectallows users to load pixel shaders written usingtheHighLevel Shader Language (HLSL). Here isasampleapp showing pixel shaders in action.

3D transformations and pixel shaders require that thecontents of a UIElement are rendered to an intermediate outputsurface. The 3D transformation and shader effect is appliedwhen this surface is composited onto the parent outputsurface. This compositing operation can be accelerated usinggraphics hardware.

From our previous example, the three elements would berendered into a 2D surface, and the actual transformation can be done in the hardware:

Finally, the third new rendering upgrade was theintroduction of a bitmap cache that can be applied to aUIElement. When a UIElement is flagged for being bitmapcached, the same kind of intermediate surface rendering andhardware accelerated compositing is performed as for elementswith 3D transformations or pixel shaders. The contents ofbitmap cache elements are also rendered once and kept on abitmap that is later composited. This can improve performancevastly for complex controls with many interlocking pieces:instead of computing and re-rendering the entire contentsevery time, the bitmap cache is used.

This of course has some visible effect. If you instructSilverlight to use a bitmap cache, and then you zoom-in thecontents, you will see the result get pixelated. You canlearn more aboutthisonthe BitmapCache documentation.

Moonlight's New Rendering Pipeline and GPU Acceleration

Both effects and projections can be implemented purely insoftware. Effects can be implemented by providing a smallinterpreter for HLSL code and projections by performing therendering in software and compositing the results.

David Reveman, the hacker behind Compiz joined theMoonlight team last year to implement the new renderingprimitives, but also to figure out how we could hardwareaccelerate Moonlight. The results of his work are availableon yesterday's release of Moonlight 4.

The rendering pipeline was modified. Now, whenever aUIElement has either a Projection, an Effect or has the theflag BitmapCache set the entire UIElement and its children arerendered into a surface that is then off-loaded for the GPU torender.

When OpenGL is available on the system, the composition ofUIElements is entirely done by the GPU.

Moonlight will use the GPU to accelerate for compositing,perspective transformations and pixel shaders if the hardwareis present without having to turn this on. Silverlight bydefault requires developers to opt into hardware accelerationand hasitsownset of features that it will hardware accelerate.

In general, Moonlight uses the GPU more than Silverlightdoes, except in the case of DeepZoom, where we still do notaccelerate this code path.

Gallium3D

Our new rendering pipeline is built using OpenGLandGallium3D.

Gallium is an engine that is typically used to implementOpenGL. In our case, we use the Gallium3D API when we needto fallback to software rendering 3D transforms on Linux.Otherwise we go through the OpenGL pipeline:

If we were to only support Linux/X11, Gallium3D would havebeen a better choice for us. But we want to allow theMoonlight runtime to be ported to other windowing systems(LikeWaylandon Linux) and other operating systems.

Room for Growth

Now that we have this 3D accelerated platform in place, itis easy to fine-tune specific UIElements to be hardwareaccelerated.

This first release did only the basics, but wecan now trivially use hardware decoders for video and havethat directly composited in hardware with the rest of a scene,or we can offload image transformations entirely to thehardware on a type-by-type basis and of course, DeepZoom.

Object Lifecycle

Objects in moonlight live in two spaces, low-levelcomponents live in C++ and are surfaced to C#. Typicallythis means that we have code that looks like this in C++:

//class MyElement : public UIElement {  protected:MyElement ();  private:        // fields and methods for MyElement go here}

In C# we create a mirror, like this:

public class DependencyObject {// Points to the C++ instance.IntPtr _native;}

When a user wrote in C# "new MyElement", we would P/Invokeinto C++ code that does "new MyElement", get a pointer back tothis instance and store it in the "_native" field.

In the other direction, if we created a C++ object and thenwe had to "surface" it to the managed world, we wouldinitialize the object based on our C++ "this" pointer.

We could create instances of MyElement in C++, and whenthis instance needs to be surfaces to the managed world, wewould create an instance of the managed object, and store thepointer to the underlying C++ object in the _native pointer.

In the Moonlight 2.0 days we used to have C++ objects thatwould only create managed objects ondemand.Atthe time, we did this to avoid creating thousands ofmanaged objects when loading a XAML file when only a handfulof those would be controlled by user code.

The Moonlight runtime, running in pure C++ code, surfacedobjects to the C# world and we tracked the object life cyclewith good old reference counts. But with Silverlight 2,we started to see problems with the design as it was possibleto end up with cycles. These cycles did not happen only inthe C++ side or the C# side, but they spanned the boundaries.This made it very hard to debug and it made it hard to keepMoonlight from not leaking memory.

Templates for example could create these cycles.

With Moonlight 4, we have landed a new life cyclemanagement system that works like this:

Every C++ object that we create always points to amanaged counterpart. Gone are the days where themanaged peer was created only when needed.
Every C++ instance field that points to aDependencyObject subclass goes through this cool C++templated class that notifies managed when thereference changes.
There are no ref/unref pairs surrounding stores toinstance fields in c++ anymore.

Now our base class in C++ has this:

// Our entire hierarchy exposed to managed code// derives from EventObjectclass EventObject {        GCHandle managed_handle;}

Now all the c++ objects exist and are kept alive solely bytheir managed peers (there are some rare exceptions for thingslike async calls) and the whole graph is traversable by Mono's GCbecause all stores to c++ instance fields result in a managedref between the respective peers.

With the new code, we no longer live in a world ofrefs/unrefs (again, except for some rare cases) and wheneverwe assign to a C++ field that points to a managed object we notifythe managed peer of the change.

We were not able to ship Moonlight 4 with our new garbagecollection system (Sgen) as we ran into a couple of hard totrack down bugs at the last minute, but we are going to switchit on again soon.

Future Work

There is still room for improvement, and now that we knowhow to cook this kind of code, the goal is to use Mono's newGC in Moonlight more extensively.

We want to teach SGen to scan the C++ heap and trackreferences to managed objects, dropping another potentialsource of problems and reducing the code needed. We wouldalso love to go back to only creating managed objects ondemand.

Platform Abstraction Layer

Moonlight was originally designed as a Linux-only, X11-onlyplugin for rendering Silverlight content. Developersconstantly ask us whether they could run Moonlight on platformXX that is either not Linux or does not use X11.

The amount of work to port Moonlight 2 to those kinds ofscenarios was so overwhelming that most people abandoned theefforts relatively quickly.

With Moonlight 4, we have introduced anewplatformabstraction layer that will make it simpler for developersto port the Moonlight engine to other platforms.

Whether you want hardware accelerated user experiences in yourvideo game or you want to put Moonlight on a the FreezeMaster10000 Domestic Fridge with an Internet Connection andSmoothStreaming running on a barebones ARM CPU, you can nowenjoy this in the comfort of your home.

We have done some minimal tests to exercise the API and canrun the Moonlight engine on both MacOS and Android. You canlook at exclusive footage of the animation test suite runningonOSXandonAndroid.

If you are like me, not much of a click-on-the-video kindof person, and would rather get a screenshot, youcanbaskon the smooth colors of this screenshot on Android orin this delightful test onMacOS.

We are currently not planning on completing that workourselves, so this is a fabulous opportunity for acaffeine-driven hacker to complete these ports.

Some possibilities, from the top of my head include beingable to use Silverlight to design parts of your userexperience for apps on the Mac AppStore(thinkMoonlightView in your MonoMac apps),or for your Android app.

Using Expression beats coding cute animations andfuturistic UIs by hand. That is why every major video gameembeds ScaleForm, the embeddable Flash runtime for handlingthe UI.

New XAML Parser

Our original XAML parser was written in C++, this workedfine for Moonlight 1, but with Moonlight 2 it started tobecome obvious that we were spending too much time callingback from C++ to C# to create managed objects.

This was acceptable up to version 2, but it no longerscaled with Moonlight 3. Too much time was spent going backand forth between the C++ world and the C# world. Thosefollowing the development of Moonlight would have noticed thatevery couple of weeks a new extra parameter to the xaml_loadfunction was added to deal with yet another callback.

The new XAML parser is entirely written in C#, is fasterand more reliable.