Warning

TL;DR

$ pip install sudachipy sudachidict_core$echo"高輪ゲートウェイ駅"| sudachipy高輪ゲートウェイ駅名詞,固有名詞,一般,*,*,*高輪ゲートウェイ駅EOS$echo"高輪ゲートウェイ駅"| sudachipy -m A高輪名詞,固有名詞,地名,一般,*,*高輪ゲートウェイ名詞,普通名詞,一般,*,*,*ゲートウェー駅名詞,普通名詞,一般,*,*,*駅EOS$echo"空缶空罐空きカン"| sudachipy -a空缶名詞,普通名詞,一般,*,*,*空き缶空缶アキカン0空罐名詞,普通名詞,一般,*,*,*空き缶空罐アキカン0空きカン名詞,普通名詞,一般,*,*,*空き缶空きカンアキカン0EOS

Setup

Step 1. Install SudachiPy

$ pip install sudachipy

Step 2. Get a Dictionary

$ pip install sudachidict_core

Usage: As a command

$echo"外国人参政権"| sudachipy外国人参政権名詞,普通名詞,一般,*,*,*外国人参政権EOS$echo"外国人参政権"| sudachipy -m A外国名詞,普通名詞,一般,*,*,*外国人接尾辞,名詞的,一般,*,*,*人参政名詞,普通名詞,一般,*,*,*参政権接尾辞,名詞的,一般,*,*,*権EOS

$ sudachipy tokenize -husage: sudachipy tokenize [-h] [-r file] [-m {A,B,C}] [-o file] [-s string]                          [-a] [-d] [-v]                          [file [file ...]]Tokenize Textpositional arguments:  file           text writtenin utf-8optional arguments:-h, --help     show thishelp message andexit-r file        the setting filein JSON format  -m {A,B,C}     the mode of splitting-o file        the output file-s string      sudachidicttype-a             print all of the fields-d             print the debug information  -v, --version  print sudachipy version

Output

$echo"外国人参政権"| sudachipy -a外国人参政権名詞,普通名詞,一般,*,*,*外国人参政権外国人参政権ガイコクジンサンセイケン0EOS

echo"阿quei"| sudachipy -a阿名詞,普通名詞,一般,*,*,*阿阿-1(OOV)quei名詞,普通名詞,一般,*,*,*queiquei-1(OOV)EOS

Usage: As a Python package

fromsudachipyimporttokenizerfromsudachipyimportdictionarytokenizer_obj=dictionary.Dictionary().create()

# Multi-granular Tokenizationmode=tokenizer.Tokenizer.SplitMode.C[m.surface()formintokenizer_obj.tokenize("国家公務員",mode)]# => ['国家公務員']mode=tokenizer.Tokenizer.SplitMode.B[m.surface()formintokenizer_obj.tokenize("国家公務員",mode)]# => ['国家', '公務員']mode=tokenizer.Tokenizer.SplitMode.A[m.surface()formintokenizer_obj.tokenize("国家公務員",mode)]# => ['国家', '公務', '員']

# Morpheme informationm=tokenizer_obj.tokenize("食べ",mode)[0]m.surface()# => '食べ'm.dictionary_form()# => '食べる'm.reading_form()# => 'タベ'm.part_of_speech()# => ['動詞', '一般', '*', '*', '下一段-バ行', '連用形-一般']

# Normalizationtokenizer_obj.tokenize("附属",mode)[0].normalized_form()# => '付属'tokenizer_obj.tokenize("SUMMER",mode)[0].normalized_form()# => 'サマー'tokenizer_obj.tokenize("シュミレーション",mode)[0].normalized_form()# => 'シミュレーション'

Dictionary Edition

Dictionary option: command line

$ pip install sudachidict_small$echo"外国人参政権"| sudachipy -s small

$ pip install sudachidict_full$echo"外国人参政権"| sudachipy -s full

Dictionary option: Python package

classDictionary(config_path=None,resource_dir=None,dict_type=None)

fromsudachipyimporttokenizerfromsudachipyimportdictionary# default: sudachidict_coretokenizer_obj=dictionary.Dictionary().create()# The dictionary given by the `systemDict` key in the config file (/path/to/sudachi.json) will be usedtokenizer_obj=dictionary.Dictionary(config_path="/path/to/sudachi.json").create()# The dictionary specified by `dict_type` will be set.tokenizer_obj=dictionary.Dictionary(dict_type="core").create()# sudachidict_core (same as default)tokenizer_obj=dictionary.Dictionary(dict_type="small").create()# sudachidict_smalltokenizer_obj=dictionary.Dictionary(dict_type="full").create()# sudachidict_full# The dictionary specified by `dict_type` overrides those defined in the config path.# In the following code, `sudachidict_full` will be used regardless of a dictionary defined in the config file.tokenizer_obj=dictionary.Dictionary(config_path="/path/to/sudachi.json",dict_type="full").create()

Dictionary in The Setting File

{    "systemDict" : "relative/path/to/system.dic",    ...}

$ sudachipy -r path/to/sudachi.json

User Dictionary

{"userDict" :["relative/path/to/user.dic"],    ...}

$ sudachipy -r path/to/sudachi.json

$ sudachipy ubuild -husage: sudachipy ubuild [-h] [-d string] [-o file] [-s file] file [file ...]Build User Dictionarypositional arguments:  filesource files with CSV format (one or more)optional arguments:  -h, --help  show thishelp message andexit  -d string   description comment to be embedded on dictionary  -o file     output file (default: user.dic)  -s file     system dictionary path (default: system core dictionary path)

Customized System Dictionary

$ sudachipy build -husage: sudachipy build [-h] [-o file] [-d string] -m file file [file ...]Build Sudachi Dictionarypositional arguments:  filesource files with CSV format (one of more)optional arguments:  -h, --help  show thishelp message andexit  -o file     output file (default: system.dic)  -d string   description comment to be embedded on dictionaryrequired named arguments:  -m file     connection matrix file with MeCab's matrix.def format

{    "systemDict" : "relative/path/to/system.dic",    ...}

$ sudachipy -r path/to/sudachi.json

For Developers

Cython Build

$ python setup.py build_ext --inplace

Code Format

Test

Contact