Posted onOct 6, 2020 • Originally published atlbarasti.com

Crystal JSON beyond the basics

#crystal #json #serialization #adt

Introduction

When modelling a business domain, you will often find yourself defining custom data types on the top of the language's primitives. If you've been exposed to some functional programming, you're likely to strive forsum types, in particular.

In Crystal, we can represent sum types as composite types inheriting from an abstract one. As a side benefit, this pattern makes it straightforward to encode (and decode) custom types into (and from) JSON, as hinted in theofficial documentation.

In this article, we'll look at how we can JSON-encode and decode sum types in Crystal using thejson module and its powerful macros.

We'll cover:

Automatic encoding withJSON::Serializable
Type resolution with discriminators
Encoding of nested composite data types
Considerations on the extensibility of this approach

Case study - a P2P client

Suppose we want to model events related to a peer to peer application. We'll focus on two domain events:

Connected event: a connection with a peer is established
Started event: a file piece download is started.

A common pattern in this scenario is to represent the various types of event as classes or structs inheriting from a baseEvent type. Events are inherently immutable, so it makes sense to model them as structs with getters.

aliasPeer=StringabstractstructEventendstructConnected<Eventgetterpeerdefinitialize(@peer:Peer);endendstructStarted<Eventgetterpeer,piecedefinitialize(@peer:Peer,@piece:UInt32);endend

In the snippet above

On line 1, aPeer is represented by a string - its IP address.
On line 3, we define an abstract structEvent which serves as base type for all the concrete event types. Mind that theabstract identifier makes it so thatEvent objects cannot be instantiated - meaningEvent.new won't compile.

JSON encoding

As we are contemplating our nicely designed events hierarchy, a requirement comes in saying that we need to persist all the P2P events processed by our application for auditing purposes. After an intense discussion with the team, we decide to go for the JSON format. Let's update our code so that we can turnEvent instances into JSON

require"json"abstractstructEventincludeJSON::Serializableend

Here we are importing the JSON package (line 1), and then simplymixing theJSON::Serializable moduleintoEvent (line 4).

Is that it? Well, let's see...

e0=Connected.new("0.0.0.0")#=> Connected(@peer="0.0.0.0")e1=Started.new("0.0.0.0",2)#=> Started(@peer="0.0.0.0", @piece=2)e0.to_json#=> {"peer":"0.0.0.0"}e1.to_json#=> {"peer":"0.0.0.0","piece":2}

Now, that's impressive! Simply including theJSON::Serializable module into the base type resulted in equipping its subtypes with working#to_json methods. The following works, too:

Connected.from_json(e0.to_json)#=> Connected(@peer="0.0.0.0")Started.from_json(e1.to_json)#=> Started(@peer="0.0.0.0", @piece=2)

This is nice, e.g. for testing purposes, but mind that the exact type of an event will likely be unknown to us at compile time, so what we'dactually like to run is

Event.from_json(e0.to_json)# raises "Error: can't instantiate abstract struct Event"

Unfortunately, this raises an error: by default, the deserializer defined within theJSON::Serializable module tries to instantiate anEvent object. As we mentioned above, this is not possible, due to the abstract nature of the type. So, where do we go from here?

Discriminators to the rescue

Here is an idea: in order to deserialize a JSON payload to the correct runtime type, we will attach some extra metadata about the event type to the JSON itself. We call this field adiscriminator.

Luckily, thejson module comes with an aptly nameduse_json_discriminator macro. This will give us the deserialization capability we are looking for, but it's up to us to make sure that the discriminator field is populated properly at serialization time.

Let's update our code to add support for discriminators.

abstractstructEventincludeJSON::Serializableuse_json_discriminator"type",{connected:Connected,started:Started}endstructConnected<Eventgetterpeergettertype="connected"definitialize(@peer:Peer);endendstructStarted<Eventgetterpeer,piecegettertype="started"definitialize(@peer:Peer,@piece:UInt32);endend

OK, what's going here?

On line 4, we calluse_json_discriminator by providing a mapping between a discriminator field value and a type. The deserializer expects the discriminator to appear under the"type" field, in this case.
lines 12 and 18 ensure that thetype field is populated according to the event type name.

You'll notice a correspondence between the value of eachtype field and the discriminator mapping.

Let's check how this affects our serializer.

e0.to_json#=> {"type":"connected","peer":"0.0.0.0"}e1.to_json#=> {"type":"started","peer":"0.0.0.0","piece":2}

Notice how the type metadata is now part of the generated JSON. This in turn makes the following work:

Event.from_json(e0.to_json)#=> Connected(@type="connected", @peer="0.0.0.0")Event.from_json(e1.to_json)#=> Started(@type="started", @peer="0.0.0.0", @piece=2)

Brilliant!

Composing composite types

The above works all right with composite types where fields are primitive types, but what if we were to define composite types on the top of other composite types? 😱

Let's expand the definition ofPeer to check this out:

structPeergetteraddress:Stringgetterport:Int32definitialize(@address,@port)endende0=Connected.new(Peer.new("0.0.0.0",8020))e1=Started.new(Peer.new("0.0.0.0",8020),2)

Now the following fails

e0.to_json# raises "Error: no overload matches 'Peer#to_json' with type JSON::Builder"

The compiler is pretty explicit here: it does not know how to turn aPeer object into JSON.

🤔 I know this one! Let's includeJSON::Serializable intoPeer.

structPeerincludeJSON::Serializablegetteraddress:Stringgetterport:Int32definitialize(@address,@port)endend

And now try

e0.to_json#=> {"type":"connected","peer":{"address":"0.0.0.0","port":8020}}e1.to_json#=> {"type":"started","peer":{"address":"0.0.0.0","port":8020},"piece":2}

Success! What about deserialization?

Event.from_json(s0)#=> Connected(@type="connected", @peer=Peer(@address="0.0.0.0", @port=8020))Event.from_json(s1)#=> Started(@type="started", @peer=Peer(@address="0.0.0.0", @port=8020), @piece=2)

Excellent! This is really all there is to it. Let's wrap up with an interesting trick and some more considerations on the extensibility of this method.

Adding Event subtypes

At present, both the base event type and its implementation are JSON-aware, meaning the code in both includes bits related to the JSON support.

This is not an issue in itself, but it feels like the JSON logic is leaking implementation details into theEvent subtypes definition. Could we make it so that anEvent implementer does not have to know about thetype field? After all, thetype getter returns a value that can be computed programmatically - in this case, a downcase version of the type name.

It turns out we can:

abstractstructEventincludeJSON::Serializableuse_json_discriminator"type",{connected:Connected,started:Started}macroinheritedgettertype:String={{@type.stringify.downcase}}endend

Wait, what is thismacro inherited on line 6 about? It's a special macrohook that injects the code in its body into any type inheriting fromEvent. This is exactly what we need, as it gives us the opportunity to inject thetype getter into each implementation ofEvent and set it to the type name, stringified and downcased. On line 7, note that occurrences of@type in a macro resolve to the name of the instantiating type.

Now the rest of the code looks like this:

structConnected<Eventgetterpeerdefinitialize(@peer:Peer);endendstructStarted<Eventgetterpeer,piecedefinitialize(@peer:Peer,@piece:UInt32);endend

No trace of JSON logic 🎉

Let's introduce a newEvent type to demonstrate this.

# An event indicating the completion of a file piece.structCompleted<Eventgetterpeer,piecedefinitialize(@peer:Peer,@piece:UInt32);endend

No extra logic on the implementer side, as expected, but mind that we still need to update the discriminator mapping:

abstractstructEventuse_json_discriminator"type",{connected:Connected,started:Started,completed:Completed}# ...end

Challenge. Can we avoid having to manually update the mapping?

Hint: the following macro generates exactly theNamedTupleLiteral we're looking for:

abstractstructEventmacrosubclasses{{% fornamein@type.subclasses%}      {{ name.stringify.downcase.id }}:{{name.id}},{% end%}}endendEvent.subclasses# => {connected: Connected, started: Started, completed: Completed}

Unfortunately, the following does not work.

use_json_discriminator"type",Event.subclasses#=> raises Error: mapping argument must be a HashLiteral or a NamedTupleLiteral, not Call

This is because at the time when theuse_json_discriminator macro is expanded,Event.subclasses hasn't been expanded, yet. I've seen this kind of issues arising often when working with macros: they can save you from writing a lot of code, but composing them can be frustratingly complicated.

Here is my recommendation: