Writing a DSL in Lua

DSLs, or domain specific languages, are programming languages that are designedto implement a set of features specific to a particular problem or field. Anexample could beMake, the build tool,which is a specially designed language for combining commands and files whilemanaging dependencies.

A lot of modern programming languages have so much flexibility in their syntaxthat it’s possible to build libraries that expose their own mini-languageswithin the host language. The definition of DSL has broadened to include thesekinds of libraries.

In this guide we'll build a DSL for generating HTML. It looks like this:

html{body{h1"Welcome to my Lua site",a{href="http://leafo.net","Go home"}}}

Before jumping in, here are some DSL building techniques:

Dropping the parenthesis

One of the cases for Lua as described in itsinitial public release (1996) isthat it makes a good configuration language. That’s still true to this day, andLua is friendly to building DSLs.

A unique part about Lua’s syntax is parenthesis are optional in some scenarioswhen calling functions. Terseness is important when building a DSL, andremoving superfluous characters is a good way to do that.

When calling a function that has a single argument of either a table literal ora string literal, the parenthesis are optional.

print"hello"--> print("hello")my_function{1,2,3}--> my_function({1,2,3})-- whitespace isn't needed, these also work:print"hello"--> print("hello")my_function{1,2,3}--> my_function({1,2,3})

This syntax has very high precedence, the same as if you were usingparenthesis:

tonumber"1234"+5-- > tonumber("1234") + 5

Chaining

Parenthesis-less invocation can be chained as long as each expression from theleft evaluates to a function (or a callable table). Here’s some example syntaxfor a hypothetical web routing framework:

match"/post-comment"{GET=function()-- render the formend,POST=function()-- save to databaseend}

If it’s not immediately obvious what’s going on, writing the parenthesis inwill clear things up. The precedence of the parenthesis-less invocation goesfrom left to right, so the above is equivalent to:

match("/post-comment")({...})

The pattern we would use to implement this syntax would look something likethis:

localfunctionmatch(path)print("match:",path)returnfunction(params)print("params:",params)-- both path and params are now availble for use hereendend

Using a recursive function constructor it’s possible to make chaining work forany length.

Using function environments

When interacting with a Lua module you regularly have to bring any functions orvalues into scope usingrequire. When working with a DSL, it’s nice to haveall the functionality available without having to manually load anything.

One option would be to make all the functions and values global variables, butit’s not recommended as it might interfere with other libraries.

Afunction environment can be used to change how a function resolves globalvariable references within its scope. This can be used to automatically exposea DSL’s functionality without polluting the regular global scope.

For the sake of this guide I'll assume thatsetfenv exists in the version ofLua we're using. If you're using 5.2 or above you'll need to provide you ownimplementation:Implementing setfenv in Lua 5.2, 5.3, and above

Here’s a functionrun_with_env that runs another function with a particularenvironment.

localfunctionrun_with_env(env,fn,...)setfenv(fn,env)fn(...)end

The environment passed will represent the DSL:

localdsl_env={move=function(x,y)print("I moved to",x,y)end,speak=function(message)print("I said",message)end}run_with_env(dsl_env,function()move(10,10)speak("I am hungry!")end)

In this trivial example the benefits might not be obvious, but typically yourDSL would be implemented in another module, and each place you invoke it is notnecessary to bring each function into scope manually, but rather activate thewhole sscope withrun_with_env.

Function environments also let you dynamically generate methods on the fly.Using the__index metamethod implemented as a function, any value can beprogrammatically created. This is how the HTML builder DSL will be created.

Implementing the HTML builder

Our goal is to make the following syntax work:

html{body{h1"Welcome to my Lua site",a{href="http://leafo.net","Go home"}}}

Each HTML tag is represented by a Lua function that will return the HTML stringrepresenting that tag with the correct attribute and content if necessary.

Although it would be possible to write code to generate all the HTML tagbuilder functions ahead of time, a function__index metamethod will be usedto generate them on the fly.

In order to run code in the context of our DSL, it must be packaged into afunction. Therender_html function will take that function and convert it toa HTML string:

render_html(function()returndiv{img{src="http://leafo.net/hi"}}end)-- > <div><img src="http://leafo.net/hi" /></div>

Theimg tag is self-closing, it has no separate close tag. HTML calls these“voidelements”.These will be treated differently in the implementation.

render_html might be implemented like this:

localfunctionrender_html(fn)setfenv(fn,setmetatable({},{__index=function(self,tag_name)returnfunction(opts)returnbuild_tag(tag_name,opts)endend}))returnfn()end

Thebuild_tag function is where all actual work is done. It takes the name ofthe tag, and the attributes and content as a single table.

This function could be optimized by caching the generated functions in theenvironment table.

Thevoidelements, asmentioned above, are defined as a simple set:

localvoid_tags={img=true,-- etc...}

The most efficient way to concatenate strings in regular Lua is to accumulatethem into a table then calltable.concat. Many calls totable.insertcould be used to append to this buffer table, but I prefer the followingfunction to allow multiple values to be appended at once:

localfunctionappend_all(buffer,...)fori=1,select("#",...)dotable.insert(buffer,(select(i,...)))endend-- example:--   local buffer = {}--   append_all(buffer, "a", "b", c)-- buffer now is {"a", "b", "c"}

append_all uses Lua’s built in functionselect to avoid any extraallocations by querying the varargs object instead of creating a new table.

Now the implementation ofbuild_tag:

localfunctionbuild_tag(tag_name,opts)localbuffer={"<",tag_name}iftype(opts)=="table"thenfork,vinpairs(opts)doiftype(k)~="number"thenappend_all(buffer," ",k,'="',v,'"')endendendifvoid_tags[tag_name]thenappend_all(buffer," />")elseappend_all(buffer,">")iftype(opts)=="table"thenappend_all(buffer,unpack(opts))elseappend_all(buffer,opts)endappend_all(buffer,"</",tag_name,">")endreturntable.concat(buffer)end

There are a couple interesting things here:

Theopts argument can either be a string literal or a table. When it’s atable it takes advantage of the fact that Lua tables are both hash tables andarrays at the same time. The hash table portion holds the attributes of theHTML element, and the array portion holds the contents of the element.

Checking if the key in apairs iteration is numeric is a quick way toapproximate isolating array like elements. It’s not perfect, but will work forthis case.

fork,vinpairs(opts)doiftype(k)~="number"then-- access hash table key and valuesendend

When the content of the tag is inserted into the buffer for the table basedopts, the following line is used:

append_all(buffer,unpack(opts))

Lua’s built in functionunpack converts the array values in a table tovarargs. This fits perfectly into theappend_all function defined above.

unpack istable.unpack in Lua 5.2 and above.

Closing

This simple implementation of an HTML builder that should give you a goodintroduction to building your own DSLs in Lua.

The HTML builder provided performs no HTML escaping. It’s not suitable forrendering untrusted input. If you're looking for a way to enhance the builderthen try adding html escaping. For example:

localunsafe_text=[[<script type="text/javascript">alert('hacked!')</script>]]render_html(function()returndiv(unsafe_text)end)-- should not return a functional script tag:-- <div>&lt;script type=&quot;text/javascript&quot;&gt;alert('hacked!')&lt;/script&gt;</div>