Repository files navigation

Most lexical analyzers are closely coupled with the language, the amount of code is relatively large. It's hard to pay attention to the essential principles of lexical analyzer.

In order to focus on the working principle of lexical analyzer , not to consider the small differences caused by different languages , an idea of making alexer project that is completely decoupled from the language was born.

lexer through the following two files, realize the decoupling of lexical analyzer and language

• src/lexer.js is the core part of lexical analyzer within 300 lines, includingISR andDFA
• src/lang/{lang}-define.jsis the language extension of lexical analyzer. Support different languages，such assrc/lang/c-define.js

From inputting the character sequence to generatingtoken after the analysis,lexer has complete steps for lexical analysis, and 12 token types for most language extensions

lexer supports different language extensions such asPython,Go, etc. How to make different language extensions, please checkContributions

• C ：A popular programming language，click here to see its lexical analysis
• SQL ：A popular database query language，click here to see its lexical analysis
• Goal ：A goal parser problem from leetCode ，click here to see its lexical analysis

The core mechanism of lexical analyzer is based on the state flow ofDFA. For this reason,lexer records detailed state flow log to achieve the following requirements of you

• Debug mode
• Automatically generateDFA state flow diagram

Aftergit clone command, no need for any dependencies, and no extra installation steps

If you need uselexer in your project, such as code editor, etc.

npm install chain-lexer

varchainLexer=require('chain-lexer');letlexer=chainLexer.cLexer;letstream="int a = 10;";lexer.start(stream);letparsedTokens=lexer.DFA.result.tokens;lexer=chainLexer.sqlLexer;stream="select * from test where id >= 10;";lexer.start(stream);parsedTokens=lexer.DFA.result.tokens;

Import thepackage/{lang}-lexer.min.js file, then visitlexer variable to get the object of lexical analyzer，and visitlexer.DFA.result.tokens to gettokens

// 1. The code that needs lexical analysisletstream="int a = 10;";// 2. Start lexical analysislexer.start(strem);// 3. After the lexical analysis is done, get the generated tokensletparsedTokens=lexer.DFA.result.tokens;// 4. Do what you want to doparsedTokens.forEach((token)=>{// ... ...});

TheProvide state flow log part in features，visitflowModel.result.paths will get the detail logs of state flow insidelexer. The data format is as follows

[{state:0,// now statech:"a",// read charnextSstate:2,// next statematch:true,// is matchend:false,// is last char},// ... ...]

In order to preview the process oflexer in real time, to debug and test, there is aindex.html file in the root directory of this project. Open it directly in your browser, and after entering the code will automatically output theToken generated afterlexer analysis, as shown in the figure below

inta=10;intb=20;intc=20;floatf=928.2332;charb='b';if(a==b){printf("Hello, World!");}elseif(b!=c){printf("Hello, World! Hello, World!");}else{printf("Hello!");}

or check theonline website

Documents about source code development, project design, unit testing, automated testing, development specifications, and how to make extensions in different languages, please readsource code explanation

• Add more new features
• Add more extensions/src/lang/{lang}-define.js

The project is released with the version number ofA-B-C，regarding release log, you can check theCHANGELOG or therelease record

• A：Major upgrade
• B：Minor upgrade
• C：bug fix / features / ...

If you have any problems or questions, pleasesubmit an issue