A thin wrapper around the tiktoken-rs crate, allowing to encode text into Byte-Pair-Encoding (BPE) tokens and decode tokens back to text. This is useful to understand how Large Language Models (LLMs) perceive text.
| Version: | 0.0.7 |
| Depends: | R (≥ 4.2) |
| Suggests: | testthat (≥ 3.0.0) |
| Published: | 2025-04-14 |
| DOI: | 10.32614/CRAN.package.rtiktoken |
| Author: | David Zimmermann-Kollenda [aut, cre], Roger Zurawicki [aut] (tiktoken-rs Rust library), Authors of the dependent Rust crates [aut] (see AUTHORS file) rtiktoken author details |
| Maintainer: | David Zimmermann-Kollenda <david_j_zimmermann at hotmail.com> |
| BugReports: | https://github.com/DavZim/rtiktoken/issues |
| License: | MIT + fileLICENSE |
| URL: | https://davzim.github.io/rtiktoken/,https://github.com/DavZim/rtiktoken/ |
| NeedsCompilation: | yes |
| SystemRequirements: | Cargo (Rust's package manager), rustc >= 1.65.0 |
| Materials: | README,NEWS |
| CRAN checks: | rtiktoken results |
| Reference manual: | rtiktoken.html ,rtiktoken.pdf |
| Package source: | rtiktoken_0.0.7.tar.gz |
| Windows binaries: | r-devel:rtiktoken_0.0.7.zip, r-release:rtiktoken_0.0.7.zip, r-oldrel:rtiktoken_0.0.7.zip |
| macOS binaries: | r-release (arm64):rtiktoken_0.0.7.tgz, r-oldrel (arm64):rtiktoken_0.0.7.tgz, r-release (x86_64):rtiktoken_0.0.7.tgz, r-oldrel (x86_64):rtiktoken_0.0.7.tgz |
| Old sources: | rtiktoken archive |
Please use the canonical formhttps://CRAN.R-project.org/package=rtiktokento link to this page.