word-segmentation

Ekphrasis is a text processing tool, geared towards text from social networks, such as Twitter or Facebook. Ekphrasis performs tokenization, word normalization, word segmentation (for splitting hashtags) and spell correction, using word statistics from 2 big corpora (english Wikipedia, twitter - 330mil english tweets).

nlp tokenizer text-processing semeval nlp-library word-segmentation spelling-correction tokenization text-segmentation spell-corrector word-normalization

UpdatedJun 2, 2025
Python

vncorenlp /VnCoreNLP

Star650

A Vietnamese natural language processing toolkit (NAACL 2018)

java nlp natural-language-processing parsing vietnamese python3 named-entity-recognition ner word-segmentation pos-tagging dependency-parsing pos-tagger vietnamese-nlp sentence-segmentation vietnamese-tokenizer vncorenlp word-segmenter rdrsegmenter vnmarmot

UpdatedFeb 12, 2023
Java

bab2min /Kiwi

Sponsor

Star650

Kiwi(지능형 한국어 형태소 분석기)

nlp cpp morphology korean word-segmentation morphological-analysis korean-text-processing korean-tokenizer korean-nlp

UpdatedDec 15, 2025
C++

JayYip /m3tl

Star546

BERT for Multitask Learning

nlp text-classification transformer named-entity-recognition pretrained-models part-of-speech ner word-segmentation bert cws encoder-decoder multi-task-learning multitask-learning

UpdatedApr 12, 2023
Jupyter Notebook

modelscope /AdaSeq

Star450

AdaSeq: An All-in-One Library for Developing State-of-the-Art Sequence Understanding Models

nlp natural-language-processing crf pytorch information-extraction named-entity-recognition chinese-nlp ner word-segmentation bert sequence-labeling relation-extraction natural-language-understanding entity-typing token-classification multi-modal-ner

UpdatedNov 15, 2023
Python

taishi-i /nagisa

Star409

A Japanese tokenizer based on recurrent neural networks

nlp natural-language-processing japanese tokenizer nlp-library word-segmentation dynet pos-tagging sequence-labeling

UpdatedOct 29, 2025
Python

ku-nlp /jumanpp

Star403

Juman++ (a Morphological Analyzer Toolkit)

nlp japanese tokenizer cjk word-segmentation pos-tagging part-of-speech-tagger morphological-analysis pos-tagger morphological-analyser juman

UpdatedOct 3, 2023
C++

jacksonllee /pycantonese

Sponsor

Star393

Cantonese Linguistics and NLP

python nlp natural-language-processing linguistics cantonese computational-linguistics word-segmentation jyutping pycantonese stop-words part-of-speech-tagging

UpdatedMay 23, 2024
Python

yongzhuo /Pytorch-NLU

Star351

中文文本分类、序列标注工具包（pytorch），支持中文长文本、短文本的多类、多标签分类任务，支持中文命名实体识别、词性标注、分词、抽取式文本摘要等序列标注任务。 Chinese text classification and sequence labeling toolkit, supports multi class and multi label classification, text similsrity, text summary and NER.

text-classification transformers python3 pytorch named-entity-recognition chinese-text-segmentation pretrained-models word-segmentation bert pos-tagging sequence-labeling chinese-text-classification