TONL (Token-Optimized Notation Language)

🎉 Latest Release: v2.5.2 - Documentation & Testing Excellence

📚v2.5.2 (December 20, 2025)

🔧v2.5.1 (December 11, 2025)

🛡️v2.5.0 (December 3, 2025)

🧪Testing Excellence:

📋 Table of Contents

Why TONL?

🚀 Quick Start

Installation

npm install tonl

Basic Usage

import{TONLDocument,encodeTONL,decodeTONL}from'tonl';// Create from JSONconstdoc=TONLDocument.fromJSON({users:[{id:1,name:"Alice",role:"admin",age:30},{id:2,name:"Bob",role:"user",age:25}]});// Query with JSONPath-like syntaxdoc.get('users[0].name');// 'Alice'doc.query('users[*].name');// ['Alice', 'Bob']doc.query('users[?(@.role == "admin")]');// [{ id: 1, ... }]doc.query('$..age');// All ages recursively// Aggregation (v2.4.0)doc.count('users[*]');// 2doc.sum('users[*]','age');// 55doc.avg('users[*]','age');// 27.5doc.groupBy('users[*]','role');// { admin: [...], user: [...] }doc.aggregate('users[*]').stats('age');// { count, sum, avg, min, max, stdDev }// Fuzzy Matching (v2.4.0)import{fuzzySearch,soundsLike}from'tonl/query';fuzzySearch('Jon',['John','Jane','Bob']);// [{ value: 'John', score: 0.75 }]soundsLike('Smith','Smyth');// true// Temporal Queries (v2.4.0)import{parseTemporalLiteral,isDaysAgo}from'tonl/query';parseTemporalLiteral('@now-7d');// 7 days agoisDaysAgo(someDate,30);// within last 30 days?// Modify datadoc.set('users[0].age',31);doc.push('users',{id:3,name:"Carol",role:"editor",age:28});// Navigate and iteratefor(const[key,value]ofdoc.entries()){console.log(key,value);}doc.walk((path,value,depth)=>{console.log(`${path}:${value}`);});// Exportconsttonl=doc.toTONL();constjson=doc.toJSON();awaitdoc.save('output.tonl');// Classic API (encode/decode)constdata={users:[{id:1,name:"Alice"}]};consttonlText=encodeTONL(data);constrestored=decodeTONL(tonlText);// Advanced Optimization (v2.0.1+)import{AdaptiveOptimizer,BitPacker,DeltaEncoder}from'tonl/optimization';// Automatic optimizationconstoptimizer=newAdaptiveOptimizer();constresult=optimizer.optimize(data);// Auto-selects best strategies// Specific optimizersconstpacker=newBitPacker();constpacked=packer.packBooleans([true,false,true]);constdelta=newDeltaEncoder();consttimestamps=[1704067200000,1704067201000,1704067202000];constcompressed=delta.encode(timestamps,'timestamp');

CLI Usage

🎮Interactive CLI (NEW v2.3.1)

# Interactive stats dashboardtonl stats data.json --interactivetonl stats data.json -i --theme neon# File comparison modetonl stats data.json --compare --theme matrix# Interactive explorationtonl stats --interactive# Launch without file for menu-driven exploration

📊Standard Commands

# Get started (shows help)tonl# Version infotonl --version# Encode JSON to TONL (perfect round-trip, quotes special keys)tonl encode data.json --out data.tonl --smart --stats# Encode with preprocessing (clean, readable keys)tonl encode data.json --preprocess --out data.tonl# Decode TONL to JSONtonl decode data.tonl --out data.json# Query datatonl query users.tonl"users[?(@.role == 'admin')]"tonl get data.json"user.profile.email"# Validate against schematonl validate users.tonl --schema users.schema.tonl# Format and prettifytonl format data.tonl --pretty --out formatted.tonl# Compare token coststonl stats data.json --tokenizer gpt-5

🎨Interactive Themes (v2.3.1)

# Available themes: default, neon, matrix, cyberpunktonl stats data.json -i --theme neon# Bright neon colorstonl stats data.json -i --theme matrix# Green matrix styletonl stats data.json -i --theme cyberpunk# Cyan/purple cyberpunktonl stats data.json -i --theme default# Clean terminal colors

⚖️File Comparison (v2.3.1)

# Compare JSON and TONL files side-by-sidetonl stats data.json --comparetonl stats data.json --compare --theme neon# Interactive comparison modetonl stats data.json -i --compare

📊 Format Overview

Arrays of Objects (Tabular Format)

{"users": [    {"id":1,"name":"Alice","role":"admin" },    {"id":2,"name":"Bob, Jr.","role":"user" },    {"id":3,"name":"Carol","role":"editor" }  ]}

#version 1.0users[3]{id:u32,name:str,role:str}:  1, Alice, admin  2, "Bob, Jr.", user  3, Carol, editor

Nested Objects

{"user": {"id":1,"name":"Alice","contact": {"email":"alice@example.com","phone":"+123456789"    },"roles": ["admin","editor"]  }}

#version 1.0user{id:u32,name:str,contact:obj,roles:list}:  id: 1  name: Alice  contact{email:str,phone:str}:    email: alice@example.com    phone: +123456789  roles[2]: admin, editor

✨ Complete Feature Set

🔄 Core Serialization

🔍 Query & Navigation API

✏️ Modification API

⚡ Performance & Indexing

🗜️ Advanced Optimization

✅ Schema & Validation

🛠️ Developer Tools

📊 Performance Comparison

🔒 Security & Quality

✅ Tests:          698+ tests passing (100% coverage)✅ Security:       All vulnerabilities fixed (100%)✅ Security Tests: 96 security tests passing✅ Code Quality:   TypeScript strict mode✅ Dependencies:   0 runtime dependencies✅ Bundle Size:    10.5 KB gzipped (browser)✅ Performance:    10-1600x faster than targets✅ Production:     Ready & Fully Secure

🎯 Use Cases

LLM Prompts

constprompt=`Analyze this user data:\n${doc.toTONL()}`;// 45% fewer tokens = lower API costs

Configuration Files

config{env:str,database:obj,features:list}:  env: production  database{host:str,port:u32,ssl:bool}:    host: db.example.com    port: 5432    ssl: true  features[3]: auth, analytics, caching

API Responses

app.get('/api/users',async(req,res)=>{constdoc=awaitTONLDocument.load('users.tonl');constfiltered=doc.query('users[?(@.active == true)]');res.type('text/tonl').send(encodeTONL(filtered));});

Data Pipelines

import{createEncodeStream,createDecodeStream}from'tonl/stream';createReadStream('huge.json').pipe(createDecodeStream()).pipe(transformStream).pipe(createEncodeStream({smart:true})).pipe(createWriteStream('output.tonl'));

Log Aggregation

logs[1000]{timestamp:i64,level:str,message:str,metadata:obj}:  1699564800, INFO, "User login", {user_id:123,ip:"192.168.1.1"}  1699564801, ERROR, "DB timeout", {query:"SELECT...",duration:5000}  ...

🌐 Browser Usage

ESM (Modern Browsers)

<scripttype="module">import{encodeTONL,decodeTONL}from'https://cdn.jsdelivr.net/npm/tonl@2.4.1/+esm';constdata={users:[{id:1,name:"Alice"}]};consttonl=encodeTONL(data);console.log(tonl);</script>

UMD (Universal)

<scriptsrc="https://unpkg.com/tonl@2.4.1/dist/browser/tonl.umd.js"></script><script>consttonl=TONL.encodeTONL({hello:"world"});console.log(tonl);</script>

📚 Complete API Reference

TONLDocument Class

// CreationTONLDocument.fromJSON(data)TONLDocument.parse(text)// Parse TONL stringTONLDocument.fromFile(filepath)// Async file loadTONLDocument.fromFileSync(filepath)// Sync file load// Querydoc.get(path:string)// Single valuedoc.query(query:string)// Multiple valuesdoc.exists(path:string)// Check existence// Modificationdoc.set(path:string,value: any)// Set valuedoc.delete(path:string)// Delete valuedoc.push(path:string,value: any)// Append to arraydoc.pop(path:string)// Remove last from arraydoc.merge(path:string,value:object)// Deep merge objects// Navigationdoc.entries()// Iterator<[key, value]>doc.keys()// Iterator<string>doc.values()// Iterator<any>doc.walk(callback:WalkCallback)// Tree traversaldoc.find(predicate:Predicate)// Find single valuedoc.findAll(predicate:Predicate)// Find all matchingdoc.some(predicate:Predicate)// Any matchdoc.every(predicate:Predicate)// All match// Indexingdoc.createIndex(name:string,path:string,type?)// Create indexdoc.dropIndex(name:string)// Remove indexdoc.getIndex(name:string)// Get index// Exportdoc.toTONL(options?:EncodeOptions)// Export as TONLdoc.toJSON()// Export as JSONdoc.save(filepath: string,options?)// Save to filedoc.size()// Size in bytesdoc.stats()// Statistics object

Encode/Decode API

// EncodingencodeTONL(data: any,options?:{delimiter?:","|"|"|"\t"|";";  includeTypes?:boolean;  version?:string;  indent?:number;  singleLinePrimitiveLists?:boolean;}):string// Smart encoding (auto-optimized)encodeSmart(data: any,options?:EncodeOptions):string// DecodingdecodeTONL(text:string,options?:{delimiter?:","|"|"|"\t"|";";  strict?:boolean;}):any

Schema API

import{parseSchema,validateTONL}from'tonl/schema';// Parse schemaconstschema=parseSchema(schemaText: string);// Validate dataconstresult=validateTONL(data:any,schema:Schema);if(!result.valid){result.errors.forEach(err=>{console.error(`${err.field}:${err.message}`);});}

Streaming API

import{createEncodeStream,createDecodeStream,encodeIterator,decodeIterator}from'tonl/stream';// Node.js streamscreateReadStream('input.json').pipe(createEncodeStream({smart:true})).pipe(createWriteStream('output.tonl'));// Async iteratorsforawait(constlineofencodeIterator(dataStream)){console.log(line);}

✅ Schema Validation

@schema v1@strict true@description "User management schema"# Define custom typesUser: obj  id: u32 required  username: str required min:3 max:20 pattern:^[a-zA-Z0-9_]+$  email: str required pattern:email lowercase:true  age: u32? min:13 max:150  roles: list<str> required min:1 unique:true# Root schemausers: list<User> required min:1totalCount: u32 required

🛠️ Development

Build & Test

# Install dependenciesnpm install# Build TypeScriptnpm run build# Run all tests (698+ tests)npmtest# Watch modenpm run dev# Clean build artifactsnpm run clean

Benchmarking

# Byte size comparisonnpm run bench# Token estimation (GPT-5, Claude 3.5, Gemini 2.0, Llama 4)npm run bench-tokens# Comprehensive performance analysisnpm run bench-comprehensive

CLI Development

# Install CLI locallynpm run link# Test commandstonl encode test.jsontonl query data.tonl"users[*].name"tonl format data.tonl --pretty# Test interactive features (v2.3.1+)tonl stats data.json --interactivetonl stats data.json -i --theme neontonl stats data.json --compare

🗺️ Roadmap

📖 Documentation

For Users

For Implementers (Other Languages)

🤝 Contributing

📄 License

🌟 Links

TONL: Making structured data LLM-friendly without sacrificing readability. 🚀

Built with ❤️ byErsin Koc