Repository files navigation

A lightweight implementation of theUnicode Text Segmentation (UAX #29)

• Spec compliant: Up-to-date Unicode data, verified by the official Unicode test suites and fuzzed with the nativeIntl.Segmenter, and maintaining 100% test coverage.

• Excellent compatibility: It works well on older browsers, edge runtimes, React Native (Hermes) and QuickJS.

• Zero-dependencies: It doesn't bloatnode_modules or the network bandwidth. Like a small minimal snippet.

• Small bundle size: It effectively compresses the Unicode data and provides a bundler-friendly format.

• Extremely efficient: It's carefully optimized for runtime performance, making it the fastest one in the ecosystem—outperforming even the built-inIntl.Segmenter.

• TypeScript: It's fully type-checked, and provides type definitions and JSDoc.

• ESM-first: It primarily supports ES modules, and still supports CommonJS.

Unicode® 16.0.0

Unicode® Standard Annex #29 -Revision 45 (2024-08-28)

Entries for Unicode text segmentation.

• unicode-segmenter/grapheme: Segments and countsextended grapheme clusters
• unicode-segmenter/intl-adapter:Intl.Segmenter adapter
• unicode-segmenter/intl-polyfill:Intl.Segmenter polyfill

And matchers for extra use cases.

• unicode-segmenter/emoji: Matches single codepoint emojis
• unicode-segmenter/general: Matches single codepoint alphanumerics

Utilities for text segmentation by extended grapheme cluster rules.

import{graphemeSegments}from'unicode-segmenter/grapheme';[...graphemeSegments('a̐éö̲\r\n')];// 0: { segment: 'a̐', index: 0, input: 'a̐éö̲\r\n' }// 1: { segment: 'é', index: 2, input: 'a̐éö̲\r\n' }// 2: { segment: 'ö̲', index: 4, input: 'a̐éö̲\r\n' }// 3: { segment: '\r\n', index: 7, input: 'a̐éö̲\r\n' }

import{splitGraphemes}from'unicode-segmenter/grapheme';[...splitGraphemes('#️⃣*️⃣0️⃣1️⃣2️⃣')];// 0: #️⃣// 1: *️⃣// 2: 0️⃣// 3: 1️⃣// 4: 2️⃣

import{countGraphemes}from'unicode-segmenter/grapheme';'👋 안녕!'.length;// => 6countGraphemes('👋 안녕!');// => 5'a̐éö̲'.length;// => 7countGraphemes('a̐éö̲');// => 3

graphemeSegments() exposes some knowledge identified in the middle of the process to support some useful cases.

For example, knowing theGrapheme_Cluster_Break category at the beginning and end of a segment can help approximately infer the applied boundary rule.

import{graphemeSegments,GraphemeCategory}from'unicode-segmenter/grapheme';function*matchEmoji(str){for(const{ segment, _catBegin}ofgraphemeSegments(input)){// `_catBegin` identified as Extended_Pictographic means the segment is emojiif(_catBegin===GraphemeCategory.Extended_Pictographic){yieldsegment;}}}[...matchEmoji('1🌷2🎁3💩4😜5👍')]// 0: 🌷// 1: 🎁// 2: 💩// 3: 😜// 4: 👍

Or build even more advanced one like an Unicode-awareTTY string width utility.

Intl.Segmenter API adapter (onlygranularity: "grapheme" available yet)

import{Segmenter}from'unicode-segmenter/intl-adapter';// Same API with the `Intl.Segmenter`constsegmenter=newSegmenter();

Intl.Segmenter API polyfill (onlygranularity: "grapheme" available yet)

// Apply polyfill to the `globalThis.Intl` object.import'unicode-segmenter/intl-polyfill';constsegmenter=newIntl.Segmenter();

Utilities for matching emoji-like characters.

import{isEmojiPresentation,// match \p{Emoji_Presentation}isExtendedPictographic,// match \p{Extended_Pictographic}}from'unicode-segmenter/emoji';isEmojiPresentation('😍'.codePointAt(0));// => trueisEmojiPresentation('♡'.codePointAt(0));// => falseisExtendedPictographic('😍'.codePointAt(0));// => trueisExtendedPictographic('♡'.codePointAt(0));// => true

Utilities for matching alphanumeric characters.

import{isLetter,// match \p{L}isNumeric,// match \p{N}isAlphabetic,// match \p{Alphabetic}isAlphanumeric,// match [\p{N}\p{Alphabetic}]}from'unicode-segmenter/general';

unicode-segmenter uses only fundamental features of ES2015, making it compatible with most browsers.

To ensure compatibility, the runtime should support:

• String.prototype.codePointAt()
• Generators
• Modules

If the runtime doesn't support these features, it can easily be fulfilled with tools like Babel.

SinceHermes doesn't support theIntl.Segmenter API yet,unicode-segmenter is a good alternative.

unicode-segmenter is compiled into small & efficient Hermes bytecode than other JavaScript libraries. See thebenchmark for details.

unicode-segmenter aims to be lighter and faster than alternatives in the ecosystem while fully spec compliant. So the benchmark is tracking several libraries' performance, bundle size, and Unicode version compliance.

• graphemer@1.4.0 (34.4M+ weekly downloads on NPM)
• grapheme-splitter@1.0.4 (6.3M+ weekly downloads on NPM)
• @formatjs/intl-segmenter@11.7.10 (10K+ weekly downloads on NPM)
• WebAssembly build ofunicode-segmentation@1.12.0 with minimum bindings
• Built-inIntl.Segmenter API

• @formatjs/intl-segmenter handles grapheme, word, and sentence, but it's not tree-shakable.
• unicode-segmentation size contains only minimum WASM binary and its bindings to execute benchmarking. It will increases to expose more features.
• Intl.Segmenter's Unicode data depends on the host, and may not be up-to-date.
• Intl.Segmenter may not be available insome old browsers, edge runtimes, or embedded environments.

• It would be compressed when included as an app asset.

Here is a brief explanation, and you can seearchived benchmark results.

Performance in Node.js/Bun/Deno:unicode-segmenter/grapheme has best-in-class performance.

• 8~35x faster than other JavaScript libraries.
• 3~5x faster than WASM binding of the Rust'sunicode-segmentation.
• 2~3x faster than built-inIntl.Segmenter.

Performance in Browsers: The performance in browser environments varies greatly due to differences in browser engines, which makes benchmarking inconsistent, but:

• Still significantly faster than other JavaScript libraries.
• Generally outperforms the built-in in the most browser environments, except the Firefox.

Performance in React Native:unicode-segmenter/grapheme is still faster than alternatives when compiled to Hermes bytecode. It's 3~8x faster thangraphemer and 20~26x faster thangrapheme-splitter, with the performance gap increasing with input size.

Performance in QuickJS:unicode-segmenter/grapheme is the only usable library in terms of performance.

Instead of trusting these claims, you can tryyarn perf:grapheme directly in your environment or build your own benchmark.

• The Rust Unicode team (@unicode-rs):
  The initial implementation was ported manually fromunicode-segmentation library.

• Marijn Haverbeke (@marijnh):
  Inspired a technique that can greatly compress Unicode data table fromhis library.

MIT