lexer

Star

A grammar describes the syntax of a programming language, and might be defined in Backus-Naur form (BNF). A lexer performs lexical analysis, turning text into tokens. A parser takes tokens and builds a data structure like an abstract syntax tree (AST). The parser is concerned with context: does the sequence of tokens fit the grammar? A compiler is a combined lexer and parser, built for a specific grammar.

Here are 1,846 public repositories matching this topic...

Language:All

Filter by language

All1,846 Python252 C222 C++213 Java183 Rust175 Go164 TypeScript109 JavaScript101 C#63 Yacc35

Sort:Most stars

Sort options

Most stars Fewest stars Most forks Fewest forks Recently updated Least recently updated

westes /flex

Star3.9k

The Fast Lexical Analyzer - scanner generator for lexing in C and C++

c flex lexer lexer-generator

UpdatedAug 18, 2025
C

alecthomas /participle

Sponsor

Star3.8k

A parser library for Go

go golang parser parser-library grammar ast lexer ebnf parser-api parser-framework participle

UpdatedOct 17, 2025
Go

maciejhirsz /logos

Sponsor

Star3.3k

Create ridiculously fast Lexers

rust parser parsing lexer lexer-generator

UpdatedNov 4, 2025
Rust

Chevrotain /chevrotain

Star2.7k

Parser Building Toolkit for JavaScript

javascript open-source grammars typescript parsing parser-library tokenizer lexer

UpdatedNov 5, 2025
TypeScript

ArthurSonzogni /Diagon

Sponsor

Star2.1k

Interactive ASCII art diagram generators. 🌟

markdown unicode parser cmake cpp ascii uml webassembly antlr ascii-diagram ascii-art webapp lexer uml-diagram generators webapplication arthursonzogni

UpdatedMay 16, 2025
C++

csstree /csstree

Star2.1k

A tool set for CSS including fast detailed parser, walker, generator and lexer based on W3C specs and browser implementations

css fast parser generator w3c ast css-parser lexer walker

UpdatedMay 12, 2025
JavaScript

carthage-software /mago

Star1.9k

Mago is a toolchain for PHP that aims to provide a set of tools to help developers write better code.

php parser formatter linter static-analysis coding-standards lexer code-style type-checker code-analyzer

UpdatedNov 1, 2025
Rust

chipsalliance /verible

Star1.7k

Verible is a suite of SystemVerilog developer tools, including a parser, style-linter, formatter and language server

productivity parser formatter analysis style-linter linter language-server-protocol syntax-tree lexer yacc systemverilog hacktoberfest lsp-server systemverilog-parser systemverilog-developer sv-lrm verible

UpdatedAug 29, 2025
C++

katef /libfsm

Sponsor

Star968

DFA regular expression library & friends

automata regexes fsm compiler regex regexp lex finite-state-machine lexer lexical-analysis compilers lexing compiler-principles lexer-generator nfa regex-validator finite-state-automata finite-state-machines dfa

UpdatedNov 5, 2025
C

no-context /moo

Star871

Optimised tokenizer/lexer generator! 🐄 Uses /y for performance. Moo.

javascript regexp tokenizer lexer

UpdatedMay 16, 2023
JavaScript

ArashPartow /exprtk

Star859

C++ Mathematical Expression Parsing And Evaluation Libraryhttps://www.partow.net/programming/exprtk/index.html

language c-plus-plus parser compiler math math-expressions high-performance grammar mathematics ast numerical-calculations scientific-computing lexer mit-license expression-parser expression-evaluator optimization-algorithms mirrored-repository semantic-analyzer exprtk

UpdatedJun 30, 2025
C++

wundergraph /graphql-go-tools

Star790

GraphQL Router / API Gateway framework written in Golang, focussing on correctness, extensibility, and high-performance. Supports Federation v1 & v2, Subscriptions & more.

graphql golang parser validation parsing linter printing ast introspection lexer execution ast-transformation ast-parser execution-engine graphql-tools complexity-analysis ast-printer ast-visitor ast-normalization introspection-query

UpdatedOct 31, 2025
Go

JetBrains /Grammar-Kit

Star757

Grammar files support & parser/PSI generation for IntelliJ IDEA

parser intellij grammar lexer

UpdatedNov 3, 2025
Java

jflex-de /jflex

Star623

The fast scanner generator for Java™ with full Unicode support

java flex parsing cup scanner regexp tokenizer grammar maven-plugin bazel-rules lexer yacc lexer-generator nfa dfa lexical-analyzer dfa-minimization scanner-generator

UpdatedJan 1, 2025
Java

zakirullin /tiny-compiler

Star577

A tiny compiler for a language featuring LL(2) grammar. With Lexer, Parser, ASM-like codegen and VM. Complex enough to give you a flavour of how the "real" thing works whilst not being a mere toy example

parser compiler asm ast lexer

UpdatedMay 11, 2025
C

Genivia /RE-flex

Star572

Yet another high-performance C++ regex library and lexical analyzer generator like Flex: extends Flex++ with Unicode support, indent/dedent anchors, lazy quantifiers, functions for lex and syntax error reporting and more. Seamlessly integrates with Bison and other parsers.

bison flex compiler regex regular-expression fuzzy-matching lex pcre lexer lexer-generator regex-pattern regular-expressions graphviz-dot lexical-analyzer regex-library regex-engine flex-regex