Repository files navigation

ReasonML native library for working with TextMate grammars

• esy install
• esy build

• esy '@test' install
• esy '@test' run

• esy '@bench' install
• esy '@bench' run

Latest docs are available here:https://onivim.github.io/reason-textmate/textmate/index.html

• esy '@docs' install
• esy '@docs' build
• esy '@docs' update

Benchmarks were averaged across 3 trials on my Windows 10 dev machine. It's surprising that the CSS is so much faster; it is possible there is a bug, although I compared the highlight output forbootstrap.css between VSCode and Onivim 2 using this library, and did not see differences.

The main bottleneck for performance is running the Oniguruma regular expressions (onig_search) - anything that can be done to reduce the number of times we need to run a search can greatly improve performance. There are some other optimizations that could be made, for example, we're not using theflambda optimizing compiler today - and we're doing some wasteful list manipulation and string manipulation for scopes. But this is pretty low overhead compared to the main bottleneck of evaluating the regular expression.

Most of the textmate grammar syntax is supported, but there are a few missing features we need for full parity:

• While conditions
• Nested patterns in capture groups

NOTE: These features are not used by the grammars in the benchmarks

openTextmate;// Create a grammar repositoryletgrammarRepository=GrammarRepository.ofFilePath("source.js","/path/to/js-grammar.json");// Create a tokenizerlettokenizer=Tokenizer.create(grammarRepository);// Tokenize a line. Tokenizing returns a scope stack and a set of tokens.let (scopeStack,tokens)=Tokenizer.tokenize(~lineNumber=0, ~scopeStack=None, ~scope="source.js", tokenizer,"console.log('Hello, world!')");// Print tokens:List.iter((token)=> print_endline("Token: "++Token.show(token), tokens);// Tokenize a second line, using the scope stack from the previous line.let (scopeStack,tokens)=Tokenizer.tokenize(~lineNumber=1, ~scopeStack=Some(scopeStack), ~scopeName, tokenizer,"console.log('Hello, again!')");

Contributions are welcome! We'd ❤️ help implementing the remainder of functionality.

New changes must:

• Add test coverage
• Pass all existing tests (`esy '@test' run)

MIT License

Copyright 2019 Outrun Labs, LLC