lexer

Star

A grammar describes the syntax of a programming language, and might be defined in Backus-Naur form (BNF). A lexer performs lexical analysis, turning text into tokens. A parser takes tokens and builds a data structure like an abstract syntax tree (AST). The parser is concerned with context: does the sequence of tokens fit the grammar? A compiler is a combined lexer and parser, built for a specific grammar.

Here are 1,939 public repositories matching this topic...

Language:All

Filter by language

All1,939 Python273 C236 C++226 Java193 Rust187 Go172 TypeScript114 JavaScript105 C#61 Yacc35

Sort:Most stars

Sort options

Most stars Fewest stars Most forks Fewest forks Recently updated Least recently updated

westes /flex

Star4k

The Fast Lexical Analyzer - scanner generator for lexing in C and C++

c flex lexer lexer-generator

UpdatedAug 18, 2025
C

alecthomas /participle

Sponsor

Star3.8k

A parser library for Go

go golang parser parser-library grammar ast lexer ebnf parser-api parser-framework participle

UpdatedFeb 7, 2026
Go

maciejhirsz /logos

Sponsor

Star3.4k

Create ridiculously fast Lexers

rust parser parsing lexer lexer-generator

UpdatedFeb 5, 2026
Rust

carthage-software /mago

Star2.8k

Mago is a toolchain for PHP that aims to provide a set of tools to help developers write better code.

php parser formatter linter static-analysis coding-standards lexer code-style type-checker code-analyzer

UpdatedFeb 7, 2026
Rust

Chevrotain /chevrotain

Star2.7k

Parser Building Toolkit for JavaScript

javascript open-source grammars typescript parsing parser-library tokenizer lexer

UpdatedFeb 7, 2026
TypeScript

ArthurSonzogni /Diagon

Sponsor

Star2.2k

Interactive ASCII art diagram generators. 🌟

markdown unicode parser cmake cpp ascii uml webassembly antlr ascii-diagram ascii-art webapp lexer uml-diagram generators webapplication arthursonzogni

UpdatedMay 16, 2025
C++

csstree /csstree

Star2.1k

A tool set for CSS including fast detailed parser, walker, generator and lexer based on W3C specs and browser implementations

css fast parser generator w3c ast css-parser lexer walker

UpdatedMay 12, 2025
JavaScript

chipsalliance /verible

Star1.8k

Verible is a suite of SystemVerilog developer tools, including a parser, style-linter, formatter and language server

productivity parser formatter analysis style-linter linter language-server-protocol syntax-tree lexer yacc systemverilog hacktoberfest lsp-server systemverilog-parser systemverilog-developer sv-lrm verible

UpdatedDec 22, 2025
C++

katef /libfsm

Sponsor

Star977

DFA regular expression library & friends

automata regexes fsm compiler regex regexp lex finite-state-machine lexer lexical-analysis compilers lexing compiler-principles lexer-generator nfa regex-validator finite-state-automata finite-state-machines dfa

UpdatedJan 27, 2026
C

ArashPartow /exprtk

Star926

C++ Mathematical Expression Parsing And Evaluation Libraryhttps://www.partow.net/programming/exprtk/index.html

language c-plus-plus parser compiler math math-expressions high-performance grammar mathematics ast numerical-calculations scientific-computing lexer mit-license expression-parser expression-evaluator optimization-algorithms mirrored-repository semantic-analyzer exprtk

UpdatedJan 31, 2026
C++

no-context /moo

Star874

Optimised tokenizer/lexer generator! 🐄 Uses /y for performance. Moo.

javascript regexp tokenizer lexer

UpdatedMay 16, 2023
JavaScript

wundergraph /graphql-go-tools

Star815

GraphQL Router / API Gateway framework written in Golang, focussing on correctness, extensibility, and high-performance. Supports Federation v1 & v2, Subscriptions & more.

graphql golang parser validation parsing linter printing ast introspection lexer execution ast-transformation ast-parser execution-engine graphql-tools complexity-analysis ast-printer ast-visitor ast-normalization introspection-query

UpdatedFeb 6, 2026
Go

JetBrains /Grammar-Kit

Star761

Grammar files support & parser/PSI generation for IntelliJ IDEA

parser intellij grammar lexer

UpdatedFeb 4, 2026
Java

jflex-de /jflex

Star626

The fast scanner generator for Java™ with full Unicode support

java flex parsing cup scanner regexp tokenizer grammar maven-plugin bazel-rules lexer yacc lexer-generator nfa dfa lexical-analyzer dfa-minimization scanner-generator

UpdatedJan 1, 2025
Java

Genivia /RE-flex

Star580

Yet another high-performance C++ regex library and lexical analyzer generator like Flex: extends Flex++ with Unicode support, indent/dedent anchors, lazy quantifiers, functions for lex and syntax error reporting and more. Seamlessly integrates with Bison and other parsers.

bison flex compiler regex regular-expression fuzzy-matching lex pcre lexer lexer-generator regex-pattern regular-expressions graphviz-dot lexical-analyzer regex-library regex-engine flex-regex

UpdatedJun 19, 2025
C++

zakirullin /tiny-compiler

Star579

A tiny compiler for a language featuring LL(2) grammar. With Lexer, Parser, ASM-like codegen and VM. Complex enough to give you a flavour of how the "real" thing works whilst not being a mere toy example

parser compiler asm ast lexer