Didn't have a real look, but looks thorough! Mainly to link that there is some ancient PR that introduces it for old strings ingh-20694.
I think this ufunc version (which copies) is better though (both implementation and behavior wise, of course implementing it as a ufunc would have been hard back then).

Thanks@seberg for taking a look!

I had a couple of questions:

1. is the API design acceptable? I intentionally madeslice only take positional arguments to mimicbuiltins.slice's behavior. Or should we support keyword arguments, likenp.strings.slice(a, step=-1)?
2. For variable-length strings we end up doing three passes over the string: (1) to computenum_codepoints, (2) to computeoutsize, (3) to copy each codepoint. I cannot see a way to avoid this, but maybe someone has an idea?
3. Negative steps, where "step < 1" are tricky, because we could potentially read past the beginning of the string by accident. I would be thankful if someone could double-check the logic I wrote there.

We can always start with only positional and relax it if someone feels strongly (I also wondered briefly if we should accept slices, but probably better not).

I am hoping@lysnikolaou or@ngoldbaum will have time to dive in (but I would not expect this to happen very quickly necessarily).
In general, I would think you could at least bring it down to 2 passes (merge the first two), although it may need a custom new method on the buffer helper implementation?
(I'll assume that small, usually one, steps are typically, and we shouldn't worry about optimizing large steps much.)

Neat, thanks for the ping. I'd like to spend some time looking closely at this. Do you mind if I push directly to this branch with fixes?

pypy has a neat optimization that allows O(1) indexing into UTF-8 strings by storing effectively a reverse index into the string. There are also some clever tricks to bring the memory overhead down. Adopting something like that would help for performance on operations like this that assume indexing into strings is cheap.

Neat, thanks for the ping. I'd like to spend some time looking closely at this. Do you mind if I push directly to this branch with fixes?

Sure, that sounds fine to me!

pypy has a neat optimization that allows O(1) indexing into UTF-8 strings by storing effectively a reverse index into the string. There are also some clever tricks to bring the memory overhead down. Adopting something like that would help for performance on operations like this that assume indexing into strings is cheap.

Oh, that's clever! Curious to see the details. We should probably add some benchmarks? It's probably worth to have a dedicated codepath forstep==1.

So there's a repo here:https://github.com/pypy/fast-utf8-methods, the pypy UTF-8 code also lives in pypy here:https://github.com/pypy/pypy/tree/6742499bbf3fc0aa63702fe4aa27147e11050c74/rpython/rlib/fastutf8

@mattip might know if there's a technical explanation somewhere of how this works. I'm not aware of a blog post or paper.

This is more of a long-term idea. I'm not sure if the memory overhead of building the index is worth it - not everyone wants to do fast string indexing by character.

I'm not sure if the memory overhead of building the index is worth it - not everyone wants to do fast string indexing by character.

I don't think there is a technical summary of the SIMD utf8 code. The code actually was never merged into PyPy, it was an experiment that one developer tried to push forward but was never completed. PyPy doesbuild an index of every 4th codepoint which is only allocated and built on-demand when a utf-8 string is sliced.

Reminder to myself to look through this sometime next week.

This also needs a release note.

Just a head's up I'm on Christmas break right now. I'm
planning to look at this in the new year.

@lysnikolaou is there any chance I can get you to do a high-level once-over on this PR? There's no need to go through the new C++ code in detail, I just did that.

@mhvk could you give this one a once-over if you have a little time?

I rebased on main and squashed the commits. Is this ready to be merged?

I rebased on main and squashed the commits. Is this ready to be merged?

I'll have Nathan have another look over, but likely. The CircleCI failure is real and we should address it (a tiny thing): The documentation is slightly wrong somewhere:

1669     Examples1670     --------1671     >>> import numpy as np1672     >>> a = np.array(['hello', 'world'])1673     >>> np.char.slice(a, 2)UNEXPECTED EXCEPTION: AttributeError("module 'numpy.char' has no attribute 'slice'")

I rebased on main and squashed the commits. Is this ready to be merged?

I'll have Nathan have another look over, but likely. The CircleCI failure is real and we should address it (a tiny thing): The documentation is slightly wrong somewhere:

1669     Examples1670     --------1671     >>> import numpy as np1672     >>> a = np.array(['hello', 'world'])1673     >>> np.char.slice(a, 2)UNEXPECTED EXCEPTION: AttributeError("module 'numpy.char' has no attribute 'slice'")

Ah, looks like doctest running was fixed in one of the commits included in the rebase!

We had decided to not addnp.char.slice since it's supposedly a deprecated module, but I had not fixed the doctests.

Everything is passing now.

Thanks for your patience on getting this reviewed and merged@ArvidJB. If you spot other bugs or missing features innp.strings please don’t hesitate to follow up with more issues or PRs!

Thanks,@ArvidJB, really nice!