We recently finished the implementation of the core parser algorithm. Details can be found inthis blog post.

While a lot of work has been automated with procedural macros, we still need to provide the keywords for a nodeinget_node_properties manually. For example, aSelectStmt node has theselect keyword as a property, and if there is afrom_clause, afrom keyword. This is implemented as

fn custom_handlers(node:&Node) ->TokenStream{match node.name.as_str(){"SelectStmt" =>quote!{        tokens.push(TokenProperty::from(Token::Select));if n.from_clause.len() >0{               tokens.push(TokenProperty::from(Token::From));}        ...},        ...}

Requirements

Setup a rust development environment:

1. Installrustup
2. Install nightly rust viarustup-init

rustup-init --default-toolchain nightly -y

3. Fork the repository

Run your first test

To make it as easy as possible for contributors to get into it, we prepared a simple test suite to quickly assert the tokens for a node.

1. open the filecrates/parser/src/codegen.rs. you will find a bunch of tests there, e.g.test_simple_select. In each test, thetest_get_node_properties helper is called . The parameters are a valid sql inputstr, the kind of the node we want to test for, and aVec<TokenProperty> with the properties we expect.
2. in a terminal,cd intocrates/parser
3. now, run a test with debugging enabledRUST_LOG=DEBUG cargo test test_simple_select. you will see both the entire graph, as well as the node you are testing against within the logs. The test should pass, since we are expecting only theSelect keyword.

Become familiar with the codebase

A lot of logic is generated using macros, which makes it harder to find the definition of struct, enums and functions. Most important in this case areSyntaxKind andTokenProperty.

SyntaxKind is an enum that holds a variant for everynode and token that Postgres supports, and some custom ones that we need for parsing such asEof (end of file). To figure out what variants exactly, you can look into the generated source ofpg_query.rs to find out how nodes and tokens are defined. This is also useful to figure out what properties each node has. Open the filecrates/parser/src/parse/libpg_query_node.rs and go to the definition ofuse pg_query::NodeEnum;. The file contains all nodes and tokens. Search forenum Token to find the latter.

TokenProperty is a struct that defines a property of a node. It is defined in the codemod forget_node_properties and holds either a stringvalue or akind or both. You can create aTokenProperty from a various types, e.g.TokenProperty::from(SyntaxKind::SelectStmt).

Add your own test

Let's say you want to add a test for theSelectStmt node to contain thefrom property when a from clause is used. We choose a simpleselect 1 from contact; statement as our input. Since there are many nodes within this statement, we have to pass the kind of the node that we want to test:SyntaxKind::SelectStmt. We expect two properties:Select, andFrom. The test now looks like:

#[test]fntest_select_with_from(){test_get_node_properties("select 1 from contact;",SyntaxKind::SelectStmt,vec![TokenProperty::from(SyntaxKind::Select),TokenProperty::from(SyntaxKind::From),],)}

Note thatSyntaxKind containsall nodes and token types. In this case, the implementation has already been done and the test should pass. If your sample contains a case that is not yet implemented, go tocrates/codegen/src/get_node_properties and add the missing implementation tocustom_handlers. You should be able to figure out what properties are relevant from the node definition generated by pg_query.rs (see above).

Any help is highly appreciated. Please let us know if you have problems setting everything up. We are very happy to support and improve this guide to maximise efficiency for contributors.