@ManyTheFish Thank you for writing the details.

for instance ダメ, is also spelled 駄目, or だめ (I didn't find a better example, sorry 🙏)

It's fine. This is enough for Japanese to understand 👍

Language detection only with Kanji/Hanzi strings

I researched various things to see if whatlang could handle it, but it might be better not to expect too much.
This is because, as you know, Japanese kanji overlaps with Chinese, and the ratio is very large.

I'm just thinking it might be better to think of another way.

Normalization

I think Unicode NKFC is enough for Japanese normalization. ( e.g.https://github.com/unicode-rs/unicode-normalization )

It's convert...

Indexing

I am very happy to be able to do ambiguous searches in hiragana, so I agree.

Since Lindera holds Pronounce by Katakana, I feel that it can be achieved by devising the indexing.

About the Japanese input method.

This is a supplement in the hope that it will be of some help.

Japanese keyboard seems to only contain Hiragana characters. And so, Katakana and Kanji characters are written in Hiragana by the user, then the computer will suggest a Katakana or a Kanji version of the written text.

There are two input methods for Japanese: "Romaji" and "Kana".

Romaji (Input by alhpabets)	Kana (Input hiragana directly)

These can switch by Japanese IME option.

Most Japanese choose romaji input because it is easy to learn the keyboard layout.

Hello all!
I come back to you to make some updates.
ForHacktoberfest we've created several issues to enhance the Japanese support of Meilisearch:

• Implement a Compatibility Decomposition Normalizer charabia#139:

A simple normalization allowing Meilisearch to retrieve "same" characters with different Unicode code points likeｶﾞ andガ.

• Implement a Japanese specialized Normalizer charabia#131:

Allow Meilisearch to retrieve Hiragana words with Katakana search queries likeダメ andだめ

• Add an allowlist to the tokenizer builder charabia#132:

the first step to enhance Meilisearch Language detection by forcing Charabia to choose in a sub-set of Language avoiding it to detect Chinese when it is not in the list

• Store detected Language per document during indexing milli#646:

the second step to enhance Meilisearch Language detection by detecting languages during document indexing and storing it.

All these issues are open to external contributions during the whole month, so don't hesitate to contribute! 🧑‍💻
I'll be glad to help you if you have any questions.

This is another step in enhancing Japanese Language support, depending on future feedback, we will be able to go further.

Thanks for all your feedback! ✍️ 🇯🇵

Hi all,

I also put here the comment I wrote onmeilisearch/charabia#139 .

This character normalization seems to be performed after tokenization, but in some cases, it is better to perform character normalization before tokenization in Japanese.

For example, this is a case where there is no problem even after tokenization:

$ echo "私はメガネを買いました。" | lindera私名詞,代名詞,一般,*,*,*,私,ワタシ,ワタシは助詞,係助詞,*,*,*,*,は,ハ,ワメガネ名詞,一般,*,*,*,*,メガネ,メガネ,メガネを助詞,格助詞,一般,*,*,*,を,ヲ,ヲ買い動詞,自立,*,*,五段・ワ行促音便,連用形,買う,カイ,カイまし助動詞,*,*,*,特殊・マス,連用形,ます,マシ,マシた助動詞,*,*,*,特殊・タ,基本形,た,タ,タ。記号,句点,*,*,*,*,。,。,。EOS

$ echo "私はﾒｶﾞﾈを買いました。" | lindera私名詞,代名詞,一般,*,*,*,私,ワタシ,ワタシは助詞,係助詞,*,*,*,*,は,ハ,ワﾒｶﾞﾈUNKを助詞,格助詞,一般,*,*,*,を,ヲ,ヲ買い動詞,自立,*,*,五段・ワ行促音便,連用形,買う,カイ,カイまし助動詞,*,*,*,特殊・マス,連用形,ます,マシ,マシた助動詞,*,*,*,特殊・タ,基本形,た,タ,タ。記号,句点,*,*,*,*,。,。,。EOS

Half-widthﾒｶﾞﾈ is not a problem because it is tokenized as a single unknown word, although it does not exist in the Japanese morphological dictionary (IPADIC). Of course, if normalization has been done in advance, morphological analysis can be used to retrieve the part-of-speech information of words from the dictionary accurately.

But the following cases can be problematic.

$ echo "私は時給1000円です。" | lindera私名詞,代名詞,一般,*,*,*,私,ワタシ,ワタシは助詞,係助詞,*,*,*,*,は,ハ,ワ時給名詞,一般,*,*,*,*,時給,ジキュウ,ジキュー1000UNK円名詞,接尾,助数詞,*,*,*,円,エン,エンです助動詞,*,*,*,特殊・デス,基本形,です,デス,デス。記号,句点,*,*,*,*,。,。,。EOS

$ echo "私は時給１０００円です。" | lindera私名詞,代名詞,一般,*,*,*,私,ワタシ,ワタシは助詞,係助詞,*,*,*,*,は,ハ,ワ時給名詞,一般,*,*,*,*,時給,ジキュウ,ジキュー１名詞,数,*,*,*,*,１,イチ,イチ０名詞,数,*,*,*,*,０,ゼロ,ゼロ０名詞,数,*,*,*,*,０,ゼロ,ゼロ０名詞,数,*,*,*,*,０,ゼロ,ゼロ円名詞,接尾,助数詞,*,*,*,円,エン,エンです助動詞,*,*,*,特殊・デス,基本形,です,デス,デス。記号,句点,*,*,*,*,。,。,。EOS

Since full-width numbers are already registered in the morphological dictionary (IPADIC), each number becomes a single token, so a full-width１０００ cannot be searched for1000.
For this reason, it is common for search engines that handle Japanese to perform character normalization before tokenization.

If possible, I would like you to consider a way to perform character normalization before tokenization.
Thanks!

@ManyTheFish
This is a slide that I created previously that explains how tokenizer works in Japanese search engines for software engineers who are not familiar with the Japanese language.

I hope it will be of some help to you.

https://speakerdeck.com/mosuka/the-importance-of-morphological-analysis-in-japanese-search-engines

I have published a simple application that I had made to confirm that Meilisearch works in Japanese.

https://meilisearch-example-jp.miiton.dev/

Hello people!
I've been putting this message off for a long time, I'm sorry.
I want to make a complete state of the current Japanese Language support in Meilisearch.

The current behavior

Language Detection

Today, we are usingwhatlang-rs to detect the Script and the Language in a text, Language detection is really important for Japanese Language support mainly to make the difference with the Chinese Language when only Kanjis are used in a text, for example, a small title or a query.

Segmentation

To segment Japanese text, we are currently usingLindera based on a Viterbi algorithm using the IPA dictionaries. Thanks to@mosuka for maintaining it. (a small explanation of Japanese segmentation)

Normalization

So far, we only normalize Japanese characters by replacing them with their decomposed compatible form, to give an example, half-width kanas are converted into kanas. To know more about this, I put below some documentation about it below:
https://unicode.org/reports/tr15/

The remaining issues we should tackle in the future

1. The Language detection library we are using is efficient in detecting the Script used, however, it's not sufficient to make the difference between Japanese and Chinese when only Kanjis are used in a text. There are some things that could be done to enhance this:
   a) Contribute to thewhatlang-rs library, afirst issue has been created by@miiton that could help, however, if someone sees a better approach, don't hesitate to suggest it.
   b) Replace the Library, here I didn't make any comparisons with other ones.
   c) Add a configuration in Meilisearch allowing the user to authorize some Language to be detected or not.
2. The compatibility decomposition normalization is made after segmentation, however,@mosuka suggested finding a way to put it before,here is the link to the comment.
3. We had a discussion with@mosuka about converting all the Katakana characters into Hiragana characters during the normalization, this would be able to raise the recall, by allowing any form of a word to find the other one (ダメ <-> だめ). This feature is currently deactivated in Meilisearch buthere is the link to the discussion.

Prototypes

There is a prototype of Meilisearch deactivating completely the Chinese support, this way we avoid Language detection mistakes, in addition, this prototype activates the katakana-to-hiragana conversion, if you want to try this prototype I put the link to it:
meilisearch/meilisearch#3588

Thanks!

Handling of Proper Nouns in Japanese

@ManyTheFish

Is the issue of not being able to search for proper nouns that are not in ipadic already being discussed, such as Chinese language support, or etc?

ref:misskey-dev/misskey/issues/10845

Target Contents

# echo "Misskeyはしゅいろさんがつくったのだよ" | linderaMisskey UNKは      助詞,係助詞,*,*,*,*,は,ハ,ワし      動詞,自立,*,*,サ変・スル,連用形,する,シ,シゅいろさんがつくったのだよ      UNK# echo "Misskeyはしゅいろさんが開発しています" | linderaMisskey UNKは      助詞,係助詞,*,*,*,*,は,ハ,ワし      動詞,自立,*,*,サ変・スル,連用形,する,シ,シゅいろさんが    UNK開発    名詞,サ変接続,*,*,*,*,開発,カイハツ,カイハツし      動詞,自立,*,*,サ変・スル,連用形,する,シ,シて      助詞,接続助詞,*,*,*,*,て,テ,テい      動詞,非自立,*,*,一段,連用形,いる,イ,イます    助動詞,*,*,*,特殊・マス,基本形,ます,マス,マス

Search Word:しゅいろ

# echo "しゅいろ" | linderaし      動詞,自立,*,*,サ変・スル,連用形,する,シ,シゅいろ  UNK

Results

❌Meilisearch (v1.1.1 & prototype-japanese-2)

id:2 no hits

# curl -s -H "Authorization: Bearer hoge" -H "Content-Type: application/json" -X POST localhost:7700/indexes/hogehoge/search -d '{"q":"しゅいろ","attributesToHighlight":["msg"]}' | jq -r '.hits[] | [.id, ._formatted.msg] | @tsv'1       Misskeyはし<em>ゅいろ</em>さんが開発<em>し</em>ています

⭕Algolia

⭕Manticore Search

mysql> CREATE TABLE notes(id bigint, msg text) charset_table = 'non_cjk' ngram_len = '1' ngram_chars = 'cjk';mysql> INSERT INTO notes(id, msg) VALUES (1, 'Misskeyはしゅいろさんが開発しています');mysql> INSERT INTO notes(id, msg) VALUES (2, 'Misskeyはしゅいろさんがつくったのだよ');mysql> SELECT id, msg, highlight() FROM notes WHERE match('しゅいろ');+------+------------------------------------------------------+---------------------------------------------------------------------------+| id   | msg                                                  | highlight()                                                               |+------+------------------------------------------------------+---------------------------------------------------------------------------+|    2 | Misskeyはしゅいろさんがつくったのだよ                | Misskeyは<b>しゅいろ</b>さんがつくったのだよ                              ||    1 | Misskeyはしゅいろさんが開発しています                | Misskeyは<b>しゅいろ</b>さんが開発<b>し</b>て<b>い</b>ます                |+------+------------------------------------------------------+---------------------------------------------------------------------------+

Hello everyone 👋

An update on Meilisearch and the Japanese support

New release V1.3 🦁

v1.3 has been released today 🦁 including a change in the Japanese segmentation, Meilisearch now relies on UniDic instead of IPADIC to segment Japanese words which should increase the amount of document retrieved by Meilisearch.

We still encounter difficulties when a dataset contains small documents with kanji-only fields, if you don't manage to retrieve documents containing kanji-only fields with the official Meilisearch version, please try theJapanese specialized docker image that deactivates other Language support.

A preview of V1.4 👀

We just released a 🧪 prototype that allows the users to customize how Meilisearch tokenizes documents and queries, and we'd love your feedback.
Additionally to the already existingstopWords andsynonyms this prototype provides:

• nonSeparatorTokens: allows removing some tokens from the default list of separators
• separatorTokens: allows to add some tokens to the list of separators
• dictionary: allows to override the word segmentation on the list of words defined in the dictionary

How to get the prototype?

Using docker, use the following command:

docker run -p 7700:7700 -v $(pwd)/meili_data:/meili_data getmeili/meilisearch:prototype-tokenizer-customization-2

From source, compile Meilisearch on theprototype-tokenizer-customization-2 tag

How to use the prototype?

You can find some examples below, or look at the originalPR for more info.

We know that thenonSeparatorTokens andseparatorTokens aren't as useful in Japanese as in other languages, however, thedictionary setting should allow defining Japanese proper nouns, company names, or any specific words to better tokenize documents related to a specific use case and retrieve them easily!

⚠️ We do NOT recommend using this prototype in production. This is for test purposes only.

Feedback and bug reporting when using this prototype are encouraged! Thanks in advance for your involvement. It means a lot to us ❤️

Facet Search is not working as I expected in Japanese.

I am trying Facet Search on aJapanese demo site I have published, but it doesn't seem to work the way I want it to.

I am trying to narrow down the prefectures... the example of Osaka-fu is easy to understand.

( Meilisearch version:prototype-japanese-5 )

If I type in大阪, it returns大分県, 大阪府.

I think it is because of the ambiguous search that大分県 is hit, but I would like the results to be returned in the order of大阪府, 大分県 at least.

If I type in大阪府, it returns京都府, 大分県, 大阪府.

I think it is a little strange for a "filter" to increase the number of results even though the number of input characters is increased.

Hi! Thank you guys for your fantastic work on improving Japanese support.

I tried the Docker imagemeilisearch/prototype-japanese-7, which works really well in my case (Drupal search API + Meilisearch backend).

I have two questions:

1. I noticed the image is numbered 0-7. Where can I find information about new releases so I can use the latest one?
2. Is there a configuration option in Meilisearch to explicitly set the fallback tokenizer to the Japanese one?

My apologies if this is not the right place to ask. Thank you for any input!

Thanks for the Great Tools.
We were experimenting with this getmeili/meilisearch:prototype-japanese-7 image in our Team. And It seems like it still has some incorrect results.

My search query was '大学' and it is showing results for '大垣' as well which doesn't make sense.

こんにちは！
以下のような
meilisearch:prototype-japanese-7
https://hub.docker.com/layers/getmeili/meilisearch/prototype-japanese-7/images/sha256-0cbcaafc43d3db7d7e934e4d1340c18312b775e4426db75c46bd0d580f7b9cfd?context=explore
Docker Imageの日本語特化検索機能はMeilisearch Cloudでは利用されているのでしょうか？
何がしたいかと言いますと、Meilisearch Cloudでは、こちらのDocker Imageのように日本語に特化した検索はできますでしょうか？

Hello!
like below
meilisearch:prototype-japanese-7
https://hub.docker.com/layers/getmeili/meilisearch/prototype-japanese-7/images/sha256-0cbcaafc43d3db7d7e934e4d1340c18312b775e4426db75c46bd0d580f7b9cfd?context=explore
Is Docker Image's Japanese-specific search function used in Meilisearch Cloud?
What I want to do is, with Meilisearch Cloud, is it possible to search specifically for Japanese like this Docker Image?

Hello All,
The Japanese Language-specialized docker image, up to date with the last Meilisearch version (v1.6.0), has been released:

$ docker pull getmeili/meilisearch:prototype-japanese-9

If you want more information about the last release, I put below the link to it:
https://github.com/meilisearch/meilisearch/releases/tag/v1.6.0

Below is the PR with all the Japanese Language-specialized docker images:
meilisearch/meilisearch#3882

See you!

Specify an user dictionary for Japanese:

I am impressed with the creation of such a wonderful search engine! 😀

Japanese does not perform word segmentation using spaces, so a good dictionary is necessary to determine word boundaries.

Particularly for new words or proper nouns, registration of words in a user dictionary may be required.

Lindera has a feature to specify a user dictionary,
and I am experimenting to specify a user dictionary via environment variable.
meilisearch/charabia@main...kamiyn:charabia:users/kamiyn/japanese_user_dictionary

If this is appropriate, I would like to make PR with it.

Hello All,
The Japanese Language-specialized docker image, up to date with the last Meilisearch version (v1.8.1), has been released:

$ docker pull getmeili/meilisearch:prototype-japanese-11

If you want more information about the last release, I put below the link to it:

• v1.8.0
• v1.8.1

Below is the PR with all the Japanese Language-specialized docker images:
meilisearch/meilisearch#3882

See you!

Thanks for the great support & great software

We started usinggetmeili/meilisearch:prototype-japanese-11 in production, however I found that the default configuration doesn't seems to support hiragana and katakana search, for example, searchingニホン should also returnにほん、.

Is there anyway to get back this features ?

For those who try Meilisearch for the first time: you should try v1.10.2 (or any later versions) with"locales": ["jpn"] first today.prototype-japanese-* images are based on older versions.
If you're usingprototype-japanese-12 now for example:

-12 →-184 → (dump migration) →-13 → (dump migration) →v1.10.2(→(dump migration)→1.11→…)

@ManyTheFish

I would like to enable Lindera's character filters and token filters in Charabia.
For example, if I use Lindera's RegexCharacterFilter or JapaneseStopTagsFilter, the correct offset and the other values are recorded in Lindera's Token, but only Lindera's token.text is passed to Charabia.

I am thinking that if I don't create a Charabia token at this time based on values other than text recorded by Lindera's token, the term will be out of position due to highlighting, etc.

I made it possible to describe Lindera settings in YAML, we would be very happy if this could be accomplished, allowing Japanese-specific string handling to be configured by out of Meilisearch using environment variable.

Is there a better way to do this?

https://github.com/meilisearch/charabia/blob/467b3e4e767c58d54a8f2974813e3ed6ac6c9795/charabia/src/segmenter/japanese.rs#L33-L36

hi all, thanks for the great support for this (niche) direction, but i'd like to know if there's any plan to rebase/bump the Japanese supporttag/branch against the newest developments, e.g. given thenew indexer is much faster?