- Notifications
You must be signed in to change notification settings - Fork61
Open
Description
It would be nice to segment character iterators, especially for interoperability with theunicode-normalization
crate. This could provide a solution to#7 when/ifio::Chars
stabilizes. In particular, I'd like to write a tokenizer like this:
let input:BufRead =my_input();let tokens = input.chars().nfkc().split_word_bounds();
One issue I see is that most of the public structs provide anas_str
method that returns "the underlying data (the part yet to be iterated) as a slice of the original string". This obviously won't work with streaming types.
Metadata
Metadata
Assignees
Labels
No labels