Changing the AST

Making changes to the AST is a common task when working on new F# compiler features or when working on developer tooling.
This document describes the process of making changes to the AST.

The easiest way to modify the AST is to start with the type definitions inSyntaxTree.fsi andSyntaxTree.fs and then let the compiler guide you to the places where you need to make changes.Let's look at an example: We want to extend the AST to include the range of the/ symbol in aSynRationalConst.Rational.

There are two solutions to choose from:- Add a new field to theRational union case- Add a dedicated trivia type to the union case which contains the new range and maybe move the existing ranges to the trivia type as well

The pros of introducing a dedicated trivia type are:- Having the additional information in a separate structure allows it to grow more easily over time. Adding new information to an existing trivia type won't disrupt most FCS consumers.
- It is clear that it is information that is not relevant for the compilation.

The cons are:- It can be a bit excessive to introduce for a single field.- The existing AST node might already contain fields that are historically more suited for trivia, but predate the SyntaxTrivia module.

In this example, we'll go with the first solution and add a new field nameddivRange to theRational union case as it felt a bit excessive to introduce a new trivia type for a single field.
But these are the type of decisions that need to be made when changing the AST.

typeSynRationalConst=// ...|Rationalofnumerator:int32*numeratorRange:range*divRange:range*// our new fielddenominator:int32*denominatorRange:range*range:range// ...

After modifyingSyntaxTree.fsi andSyntaxTree.fs, the compiler will report errors inpars.fsy. If not, thefsy file wasn't processed by the compilation. In this case, a rebuild ofFSharp.Compiler.Service.fsproj should help.
pars.fsy is the parser specification of F#, a list of rules that describe how to parse F# code. Don't be scared by the size of the file or the unfamiliar content.It's easier than it looks.The F# compiler uses a parser generator calledfsyacc to generate the parser from the specification inpars.fsy.Let's look at the most relevant syntax parts of a.fsy file:

rationalConstant:|INT32INFIX_STAR_DIV_MOD_OPINT32{if$2<>"/"thenreportParseErrorAt(rhsparseState2)(FSComp.SR.parsUnexpectedOperatorForUnitOfMeasure())iffst$3=0thenreportParseErrorAt(rhsparseState3)(FSComp.SR.parsIllegalDenominatorForMeasureExponent())if(snd$1)||(snd$3)thenerrorR(Error(FSComp.SR.lexOutsideThirtyTwoBitSigned(),lhsparseState))SynRationalConst.Rational(fst$1,rhsparseState1,fst$3,rhsparseState3,lhsparseState)}|// ...

The first line is the name of the rule,rationalConstant in this case. It's a so called non-terminal symbol in contrast to a terminal symbol likeINT32 orINFIX_STAR_DIV_MOD_OP. The individual cases of the rule are separated by|, they are called productions.

By now, you should be able to see the similarities between an fsyacc rule and the pattern matching you know from F#.
The code between the curly braces is the code that gets executed when the rule is matched and isreal F# code. After compilation, it ends up in.\artifacts\obj\FSharp.Compiler.Service\Debug\netstandard2.0\pars.fs, generated by fsyacc.

The first three lines do error checking and report errors if the input is invalid.Then the code calls theRational constructor ofSynRationalConst and passes some values to it. Here we need to make changes to adjust the parser to our modified type definition.
The values or symbols that matched the rule are available as$1,$2,$3 etc. in the code. As you can see,$1 is a tuple, consisting of the parsed number and a boolean indicating whether the number is a valid 32 bit signed integer or not.The code is executed in the context of the parser, so you can use theparseState variable, an instance ofIParseState, to access the current state of the parser. There are helper functions defined inParseHelpers.fs that make it easier to work with it.
rhs parseState 1 returns the range of the first symbol that matched the rule, hereINT32. So, it returns the range of23 in23/42.
rhs is short forright hand side.
Another helper isrhs2. Using it likerhs2 parseState 2 3 for example, returns the range covering the symbols from the second to the third symbol that matched the rule. Given23/42, it would return the range of/42.
lhs parseState returns the range of the whole rule,23/42 in our example.When parser recovery is of concern for a rule, it's preferred to userhs2 overlhs.

Circling back to our original example of adding a new field toSynRationalConst, we need to add a new parameter to the call of theRational constructor. We want to pass the range of the/ symbol, so we need to addrhs parseState 2 as the third parameter to the constructor call:

SynRationalConst.Rational(fst$1,rhsparseState1,rhsparseState2,fst$3,rhsparseState3,lhsparseState)

That's it. Adjusting the other constructor calls ofRational inpars.fsy should be enough to have a working parser again which returns the modified AST.
While fixing the remaining compiler errors outside ofpars.fsy, it's a good idea to use named access to the fields of theSynRationalConst.Rational union case instead of positional access. This way, the compilation won't fail if additional fields are added to the union case in the future.
After a successful compilation, you can run the parser tests inSyntaxTreeTests.fs to verify that everything works as expected.
It's likely that you'll need to update the baseline files as described inSyntaxTreeTests.fs.