Movatterモバイル変換


[0]ホーム

URL:


Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

Sign up
Appearance settings

Try out caching and ASCII special casing for other segmenters #80

Open
@Manishearth

Description

@Manishearth

@cessen in#77 and#79 implemented a complementary pair of optimizations for grapheme segmentation: one which optimizes the binary search to stay in the found region, and one which handles ASCII cases. They work well together, even for non-Latin text, because the caching is much more efficient when common punctuation/spaces are handled directly and do not invalidate the cache.

It might be worth doing the same thing for word, line, and sentence breaking. In some of these cases a table lookup instead of if branches may work out better, though I think if branches will typically be faster.

It may also be worth tweaking the binary search so that even if the character is not in the cached range, it can searchnearby first. This is useful for e.g. Indic scripts, where you're going to have a rapid mix of categories, but all the code points are nearby.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions


      [8]ページ先頭

      ©2009-2025 Movatter.jp