Creating Terraform-like configuration languages with HCL and Go

In the past year and a half, I’ve been working onAtlas, a databaseschema management tool that we’re developing atAriga.As part of this effort, I worked on implementing the infrastructure for theAtlas DDL, a data definition language that is the basis for Atlas’s declarative style workflow for managing database schemas.

The Atlas language is based onHCL,a toolkit for creating configuration languages with a neat and simpleinformation model and syntax.HCL was created at HashiCorp and used in popular tools such asTerraformandNomad. We chose HCL as the basis of our configuration language for multiple reasons:

• It has a base syntax that is clear and concise, easily readable by humans and machines.
• Popularized by Terraform and other projects in the DevOps / Infrastructure-as-Code space, we thought it would feel familiar to practitioners which are the one of the core audiences for our tool.
• It’s written in Go, making it super easy to integrate with the rest of our codebase atAriga.
• It has great support for extending the basic syntax into a full-blown DSL using functions, expressions and context variables.

PCL: The Pizza Configuration Language

In the rest of this post, we will demonstrate how to create a basic configurationlanguage using HCL and Go. To make this discussion entertaining, let’s imagine thatwe are creating a new PaC (Pizza-as-Code) product that lets users define their pizzain simple HCL-based configuration files and send them as orders to their nearbypizza place.

Orders and Contacts

Let’s start building our PaC configuration language by letting users define wherethey want their pizza delivered and who is the hungry contact waiting for the pizzato arrive. We’re aiming for something like:

contact{name="Sherlock Holmes"phone="+44 20 7224 3688"}address{street="221B Baker St"city="London"country="England"}

To capture this configuration, we will define a Go structOrder with sub-structsfor capturing theContact andAddress:

type(Orderstruct{Contact*Contact`hcl:"contact,block"`Address*Address`hcl:"address,block"`}Contactstruct{Namestring`hcl:"name"`Phonestring`hcl:"phone"`}Addressstruct{Streetstring`hcl:"street"`Citystring`hcl:"city"`Countrystring`hcl:"country"`})

The Go HCL codebase contains two packages with a fairly high-level API for decoding HCL documents into Go structs:hclsimple (GoDoc)andgohcl (GoDoc). Both packages rely on the user supplying Go struct field tags to map from the configuration file to the struct fields.

We will start the example by using the simpler one, with the surprisingname,hclsimple:

funcTestOrder(t*testing.T){varoOrderiferr:=hclsimple.DecodeFile("testdata/order.hcl",nil,&o);err!=nil{t.Fatalf("failed: %s",err)}require.EqualValues(t,Order{Contact:&Contact{Name:"Sherlock Holmes",Phone:"+44 20 7224 3688",},Address:&Address{Street:"221B Baker St",City:"London",Country:"England",},},o)}

Pizza sizes and toppings (using static values)

Next, let’s add the ability to order actual pizzas in our PaC application.To describe a pizza in our configuration language users should be able to dosomething like:

pizza{size=XLcount=1toppings=[olives,feta_cheese,onions,]}

Notice that to make our API more explicit, users do not pass string values to thesize ortoppings field, and instead they use pre-defined, staticidentifiers (called “variables” in the HCL internal API) such asXL orfeta_cheese.

To support this kind of behavior, we can pass anhcl.EvalContext (GoDoc),which provides the variables and functions that should be used to evaluate an expression.

To construct this context we’ll create thisctx() helper function:

funcctx()*hcl.EvalContext{vars:=make(map[string]cty.Value)for_,size:=range[]string{"S","M","L","XL"}{vars[size]=cty.StringVal(size)}for_,topping:=range[]string{"olives","onion","feta_cheese","garlic","tomatoe"}{vars[topping]=cty.StringVal(topping)}return&hcl.EvalContext{Variables:vars,}}

To use it we need to add thepizza block to our top levelOrder struct:

type(Orderstruct{Contact*Contact`hcl:"contact,block"`Address*Address`hcl:"address,block"`Pizzas[]*Pizza`hcl:"pizza,block"`}Pizzastruct{Sizestring`hcl:"size"`Countint`hcl:"count,optional"`Toppings[]string`hcl:"toppings,optional"`}// ... More types ...)

Here’s ourpizza block read usingctx() in action:

funcTestPizza(t*testing.T){varoOrderiferr:=hclsimple.DecodeFile("testdata/pizza.hcl",ctx(),&o);err!=nil{t.Fatalf("failed: %s",err)}require.EqualValues(t,Order{Pizzas:[]*Pizza{{Size:"XL",Toppings:[]string{"olives","feta_cheese","onion",},},},},o)}

How many pizzas to order? (Using functions in HCL)

The final conundrum in any pizza delivery order is of course, how many pizzasto order. To help our users out with this riddle, let’s level up our DSL andadd thefor_diners function that will take a number of diners and calculatefor the user how many pizzas should be ordered. This will look something like:

pizza {  size     = XL  count    = for_diners(3)  toppings = [    tomato  ]}

Based on the universally accepted heuristic that one should order 3 slices perdiner and round up, we can register the following function into ourEvalContext:

funcctx()*hcl.EvalContext{// .. Variables ..// Define a the "for_diners" function.spec:=&function.Spec{// Return a number.Type:function.StaticReturnType(cty.Number),// Accept a single input parameter, "diners", that is not-null number.Params:[]function.Parameter{{Name:"diners",Type:cty.Number,AllowNull:false},},// The function implementation.Impl:func(args[]cty.Value,_cty.Type)(cty.Value,error){d:=args[0].AsBigFloat()if!d.IsInt(){returncty.NilVal,fmt.Errorf("expected int got %q",d)}di,_:=d.Int64()neededSlices:=di*3returncty.NumberFloatVal(math.Ceil(float64(neededSlices)/8)),nil},}return&hcl.EvalContext{Variables:vars,Functions:map[string]function.Function{"for_diners":function.New(spec),},}}

Testing thefor_diners function out:

funcTestDiners(t*testing.T){varoOrderiferr:=hclsimple.DecodeFile("testdata/diners.hcl",ctx(),&o);err!=nil{t.Fatalf("failed: %s",err)}// For 3 diners, we expect 2 pizzas to be ordered.require.EqualValues(t,2,o.Pizzas[0].Count)}

Wrapping up

With these features, I think we can call it a day for this prototype of theworld’s first Pizza-as-Code product. As the source code for these examples isavailable onGitHub under an Apache 2.0license, I truly hope someone picks this up and builds this thing!

In this post we reviewed some basic things you can do to create a configuration language for your users using HCL. There’s a lot of other cool features we built into the Atlas language (such asinput variables,block referencing andblock polymorphism), so if you’re interestedso if you’re interested in reading more about it feel free to ping me onTwitter.