Repository files navigation

Async Japanese tokenizer native plugin for React Native that works on iOS and Android.It tokenizes Japanese text in background thread so it won't block UI. It comes in handy for such as indexing a lot of texts for full-text search.

It utilizesCFStringTokenizer to tokenize strings into words.

The implementation is based onTinySegmenter. It is lightweight word segmenter which works without dictionaries.

$ npm install react-native-japanese-tokenizer --save

$ react-native link react-native-japanese-tokenizer

1. In XCode, in the project navigator, right clickLibraries ➜Add Files to [your project's name]
2. Go tonode_modules ➜react-native-japanese-tokenizer and addRNJapaneseTokenizer.xcodeproj
3. In Xcode, in the project navigator, select your project. AddlibRNJapaneseTokenizer.a to your project'sBuild Phases ➜Link Binary With Libraries
4. Run your project (Cmd+R)

1. Open upandroid/app/src/main/java/[...]/MainActivity.java

• Addimport dog.craftz.japanese_tokenizer.RNJapaneseTokenizerPackage; to the imports at the top of the file
• Addnew RNJapaneseTokenizerPackage() to the list returned by thegetPackages() method

2. Append the following lines toandroid/settings.gradle:

   include ':react-native-japanese-tokenizer'project(':react-native-japanese-tokenizer').projectDir = new File(rootProject.projectDir,  '../node_modules/react-native-japanese-tokenizer/android')

3. Insert the following lines inside the dependencies block inandroid/app/build.gradle:

    compile project(':react-native-japanese-tokenizer')

UsingPromise:

importTokenizerfrom'react-native-japanese-tokenizer'vartext="週休七日で働きたい"Tokenizer.tokenize(text).then(tokens=>{  ...})

async/await:

importTokenizerfrom'react-native-japanese-tokenizer'vartext="週休七日で働きたい"consttokens=awaitTokenizer.tokenize(text)

Also check outthe test to learn more.

MIT License. By Takuya Matsuyama (@craftzdog).

Note that this module is based on following libraries with respective license. Thank you for the fantastic works!

• TinySegmenter: Javascriptだけで実装されたコンパクトな分かち書きソフトウェア
  • BSD-3-Clause
• TinySegmenter for Java by takscape
  • Public Domain