Movatterモバイル変換

Skip to content

#

tokenization

Here are 1,429 public repositories matching this topic...

Language:All

Filter by language

All1,429 Jupyter Notebook425 Python347 JavaScript84 TypeScript75 HTML47 Solidity46 Go42 Java39 Rust38 C++34

Sort:Most stars

Sort options

Most stars Fewest stars Most forks Fewest forks Recently updated Least recently updated

explosion /spaCy

💫 Industrial-strength Natural Language Processing (NLP) in Python

python nlp data-science machine-learning natural-language-processing ai deep-learning neural-network text-classification cython artificial-intelligence spacy named-entity-recognition neural-networks nlp-library tokenization entity-linking

UpdatedNov 27, 2025
Python

toon

toon-format /toon

🎒 Token-Oriented Object Notation (TOON) – Compact, human-readable, schema-aware JSON for LLM prompts. Spec, benchmarks, TypeScript SDK.

serialization tokenization data-format llm

UpdatedDec 15, 2025
TypeScript

AgentOps-AI /tokencost

Easy token price estimates for 400+ LLMs. TokenOps.

analytics price openai token price-tracker observability tokenization claude large-language-models llm

UpdatedSep 5, 2025
Python

NVIDIA /Cosmos-Tokenizer

A suite of image and video neural tokenizers

transformers diffusion tokenization

UpdatedFeb 11, 2025
Jupyter Notebook

lunasec-io /lunasec

LunaSec - Dependency Security Scanner that automatically notifies you about vulnerabilities like Log4Shell or node-ipc in your Pull Requests and Builds. Protect yourself in 30 seconds with the LunaTrace GitHub App:https://github.com/marketplace/lunatrace-by-lunasec/

security continuous-delivery dependency-analysis cybersecurity pci-dss web-security compliance scanning cve-scanning tokenization gdpr security-tools devsecops software-composition-analysis zero-trust soc2 sbom scanning-tool sbom-generator log4shell

UpdatedMay 2, 2024
TypeScript

databunker

securitybunker /databunker

Secure Vault for Customer PII/PHI/PCI/KYC Records

security privacy encryption database vault application-server compliance passportjs tokenization gdpr data-protection legaltech anonymization pii data-anonymization secure-storage privacy-by-design user-consent piidata ccpa

UpdatedNov 6, 2025
Go

RavenProject /Ravencoin

Ravencoin Core integration/staging tree

bitcoin raven blockchain token asset tokenization ravencoin

UpdatedMay 24, 2024
C

VKCOM /YouTokenToMe

Unsupervised text tokenizer focused on computational efficiency

nlp natural-language-processing word-segmentation tokenization bpe

UpdatedMar 29, 2024
C++

AmoDinho /datacamp-python-data-science-track

All the slides, accompanying code and exercises all stored in this repo. 🎈

python nlp data-science natural-language-processing neural-network scikit-learn pandas datascience neural-networks bokeh machinelearning tokenization datacamp-course datacamp datacamp-exercises datacamp-projects datacamp-solutions-python datacamp-python datacamp-machine-learning

UpdatedJul 17, 2023
Python

explosion /spacy-streamlit

👑 spaCy building blocks and visualizers for Streamlit apps

nlp machine-learning natural-language-processing text-classification spacy visualizer named-entity-recognition ner dependency-parsing tokenization word-vectors visualizers streamlit part-of-speech-tagging

UpdatedJul 29, 2024
Python

nlp-uoregon /trankit

Trankit is a Light-Weight Transformer-based Python Toolkit for Multilingual Natural Language Processing

multilingual nlp machine-learning natural-language-processing pytorch artificial-intelligence adapters deeplearning language-model universal-dependencies dependency-parsing tokenization lemmatization sentence-segmentation morphological-tagging part-of-speech-tagging xlm-roberta

UpdatedJul 22, 2025
Python

cbaziotis /ekphrasis

Ekphrasis is a text processing tool, geared towards text from social networks, such as Twitter or Facebook. Ekphrasis performs tokenization, word normalization, word segmentation (for splitting hashtags) and spell correction, using word statistics from 2 big corpora (english Wikipedia, twitter - 330mil english tweets).

nlp tokenizer text-processing semeval nlp-library word-segmentation spelling-correction tokenization text-segmentation spell-corrector word-normalization

UpdatedJun 2, 2025
Python

alasdairforsythe /tokenmonster

Ungreedy subword tokenizer and vocabulary trainer for Python, Go & Javascript

tokenizer vocabulary vocabulary-builder tokenize tokenization tokenisation tokenizing text-tokenization vocabulary-generator

UpdatedJul 2, 2024
Go

adobe /NLP-Cube

Natural Language Processing Pipeline - Sentence Splitting, Tokenization, Lemmatization, Part-of-speech Tagging and Dependency Parsing

parse machine-translation embeddings information-extraction dependency-parser universal-dependencies part-of-speech-tagger dependency-parsing tokenization lemmatization sentence-splitting nlp-cube language-pipeline

UpdatedNov 3, 2024
HTML

yooper /php-text-analysis

PHP Text Analysis is a library for performing Information Retrieval (IR) and Natural Language Processing (NLP) tasks using the PHP language

nlp php text-analysis php-language tokenization php-text-analysis

UpdatedDec 28, 2024
PHP

blockchain-bike-rental

alejoacosta74 /blockchain-bike-rental

Solidity based "BIKE RENTAL SHOP" on Ethereum network.

ethereum solidity tokenization ether truffle-framework

UpdatedSep 4, 2025
JavaScript

WorksApplications /sudachi.rs

Sudachi in Rust 🦀 and new generation of SudachiPy

python rust segmentation pos-tagging morphological-analysis tokenization sudachi nlp-libary

UpdatedJun 20, 2025
Rust

daac-tools /vibrato

🎤 vibrato: Viterbi-based accelerated tokenizer

nlp rust japanese tokenizer segmentation morphological-analysis tokenization

UpdatedNov 8, 2025
Rust

macmade /ClangKit

ClangKit provides an Objective-C frontend to LibClang. Source tokenization, diagnostics and fix-its are actually implemented.

c syntax-highlighting c-plus-plus parsing objective-c code llvm static-analysis clang source diagnostics tokenization

UpdatedAug 2, 2021
C

SaberaTalukder /TOTEM

The official code 👩‍💻 for - TOTEM: TOkenized Time Series EMbeddings for General Time Series Analysis

time-series representation-learning tokenization time-series-analysis time-series-forecasting time-series-anomaly-detection time-series-imputation foundation-models time-series-foundation-model

UpdatedFeb 20, 2025
Python

Improve this page

Add a description, image, and links to thetokenization topic page so that developers can more easily learn about it.

Curate this topic

Add this topic to your repo

To associate your repository with thetokenization topic, visit your repo's landing page and select "manage topics."

[8]ページ先頭

©2009-2025 Movatter.jp