Tokenize2 is a plugin which allows your users to select multiple items from a predefined list or ajax, using autocompletion as they type to find each item. You may have seen a similar type of text entry when filling in the recipients field sending messages on facebook or tags on tumblr.

• UpdatedNov 30, 2022
• JavaScript

Unicode tokeniser. Ucto tokenizes text files: it separates words from punctuation, and splits sentences. It offers several other basic preprocessing steps such as changing case that you can all use to make your text suited for further processing such as indexing, part-of-speech tagging, or machine translation. Ucto comes with tokenisation rules …

• UpdatedFeb 8, 2025
• C++

Taiwanese Hokkien Transliterator and Tokeniser

• UpdatedAug 31, 2024
• Python

A fast, simple, multilingual tokenizer

• UpdatedMay 24, 2017
• Python

JavaScript Parser

• UpdatedSep 16, 2024
• TypeScript

Text segmenter and tokeniser for Danish, English and other languages. Reads an RTF or flat text file and outputs the text, one line per sentence & optionally tokenized.

• UpdatedDec 1, 2022
• C++

A Lightweight Word Piece Tokenizer

• UpdatedSep 27, 2022
• Python

Javascript port of HappyFunTokenizer.py by Christopher Potts and HappierFunTokenizing.py by H. Andrew Schwartz

• UpdatedFeb 29, 2024
• TypeScript

Taiwanese Hokkien Transliterator and Tokeniser

• UpdatedDec 27, 2024
• JavaScript

Converts BBC BASIC 2 source code into UEF files.

• UpdatedJan 28, 2025
• C++

Find out useful information about your Casio watch

• UpdatedAug 21, 2024
• Rust

A python and rust implementation of SentencePiece (A language-independent subword tokeniser and de-tokeniser developed by Google)

• UpdatedMar 7, 2025
• Rust