(OCaml 4.02 is entering a feature freeze, which makes it a good time to stopand take a look at what to expect for this release. This is the first of a fewblog pots where I’ll describe the features that strike me as notable.)
OCaml’s metaprogramming story is kind of messy.
The good news is that OCaml has an effective metaprogramming system. It’s calledcamlp4, and before complaining about it, I want to be clear how useful it is.At Jane Street, syntax extensions likesexplib,pa_compare andbinprothave made us more productive, allowing us to extend the language in ways thatsave us from having to write big piles of unmaintainable boilerplate.
Butcamlp4 has some serious warts, which mostly derive from where it sits inthe OCaml pipeline. In particular,camlp4 is an alternate front-end to thecompiler, with its own extensible parser that allows you to extend OCaml’ssyntax in any way you like. There are downsides, however, that derive fromhaving two different parsers. In particular,camlp4’s parser has its ownslightly different behavior, its independent set of bugs, and its own excitinglyobscure error messages.
camlp4’s separate parser might seem like a necessary evil. After all, syntaxextensions require changing the syntax, and you obviously can’t change thesyntax without a new parser.
Or can you? Lisp has a rich syntactic macro system but just one parser. Thus, inLisp, all macros are AST-to-AST transformations. Lisp’s macro system canimplement lots of different syntaxes within the world of s-expressions, sinces-expressions are so general and flexible.
OCaml’s syntax, on the other hand, is very specific and inflexible. It lets youparse OCaml’s syntax exactly, and any deviation is flagged as a syntax error.Almost any interesting syntax extension will require parsing programs that arenot syntactically valid OCaml programs.
That’s where extension points come in. Extension points are a collection ofextensions to OCaml’s grammar that adds a notation for annotations. With theseannotations, OCaml’s syntax becomes general enough to accommodate many differentsyntax extensions. Indeed, Alain Frisch, who is the main author and advocate ofthis change, organized a survey of existing camlp4-based syntax extensions, andmade sure that extension points were rich enough to accommodate them.
The big advantage of this approach is that it simplifies the process ofdeveloping the compiler (because you don’t need to maintain two independentimplementations of the parser) and because you only have one syntax fordevelopment tools to target. One of the wins we hope to get from this is thatIDE-like tools like Merlin should be able to more easily interact with code thatuses syntactic macros, like the codebase at Jane Street.
The downside, of course, is that to take advantage of this, you need to portyour existing syntax extensions to work against the (now extended) OCaml AST.Also, it means that the concrete syntax that was used in existing syntaxextensions will mostly need to change. No longer can we write
type t = int * string with sexp
Instead, we’ll need to write something like:
type t = int * string [@@sexp]
That said, we expect this change to be worth the implied churn.
p.s., it was noted that I didn’t do a great job of showing the flexibility thatextension points gives you. If you want to learn more about this,this goes over the major use-cases for syntaxextensions that were considered in the design of the annotation syntax, and howthey would be rendered in OCaml 4.02.
Yaron Minsky joined Jane Street back in 2002, and claims the dubious honorof having convinced the firm to start using OCaml.
