copying a comment@nenb made inthis zulip discussion:

The first thing that caught my eye was that you are using numpy character codes. What was the motivation for this? numpy character codes are not extensible in their current format, and lead to issues like:jax-ml/ml_dtypes#41.

A feature of the character code is that it provides a way to distinguish parametric types likeU* from parametrized instances of those types (likeU3). Defining a class with the character codeU means instances of the class can be initialized with a "length" parameter, and then we can makeU2,U3, etc, as instances of the same class. If instead we bind a concrete numpy dtype as class attributes, we need a separate class forU2,U3, etc, which is undesirable. I do think I can work around this, but I figured the explanation might be helpful.

Summarising from azulip discussion:

@nenb: How is the endianness of a dtype handled?

@d-v-b: In v3, endianness is specified by the codecs. The exact same data type could be decoded to big or little endian representations based on the state of a codec. This is very much not how v2 does it -- v2 puts the endianness in the dtype.

Proposed solution: Makeendianness an attribute on in the dtype instance. This will be an implementation detail used byzarr-python to handle endianness, but won't be part of the dtype on disk (as requuired by the spec).

Summarising from azulip discussion:

@nenb: How is the endianness of a dtype handled?

@d-v-b: In v3, endianness is specified by the codecs. The exact same data type could be decoded to big or little endian representations based on the state of a codec. This is very much not how v2 does it -- v2 puts the endianness in the dtype.

Proposed solution: Makeendianness an attribute on in the dtype instance. This will be an implementation detail used byzarr-python to handle endianness, but won't be part of the dtype on disk (as requuired by the spec).

Thanks for the summary! I have implemented the proposed solution.

Nice work@d-v-b!

Huge props to@d-v-b for pulling this across the finish line. This was a huge and highly-impactful lift. 🙌 🙌 🙌 🙌

For those that are following this thread, expect this to come out in Zarr 3.1.

@ngoldbaum - thanks for all your work on NumPy. We'd love open the conversation about supporting writing to Zarr directly from NumPy. Can you advise on where the best place to do that is?

@d-v-b Not sure if it's a bug or not, but here is something interesting.

Run this using, say, zarr 3.0.8:

importzarr,numcodecsz=zarr.open("foo.zarr",zarr_format=2)z.create_array("bar", (10,),dtype=object,filters=[numcodecs.VLenUTF8()],fill_value="")z["bar"][...]# array(['', '', '', '', '', '', '', '', '', ''], dtype=object)

Then after this PR:

importzarr,numcodecsz=zarr.open("foo.zarr")z["bar"][...]# array(['', '', '', '', '', '', '', '', '', ''], dtype=StringDType())

i.e., the dtype is now a string dtype. For us it was a simple fix, but not sure if it is (a) intentional or (b) indicative of some other issue.

@d-v-b Not sure if it's a bug or not, but here is something interesting.

Run this using, say, zarr 3.0.8:

importzarr,numcodecsz=zarr.open("foo.zarr",zarr_format=2)z.create_array("bar", (10,),dtype=object,filters=[numcodecs.VLenUTF8()],fill_value="")z["bar"][...]# array(['', '', '', '', '', '', '', '', '', ''], dtype=object)

Then after this PR:

importzarr,numcodecsz=zarr.open("foo.zarr")z["bar"][...]# array(['', '', '', '', '', '', '', '', '', ''], dtype=StringDType())

i.e., the dtype is now a string dtype. For us it was a simple fix, but not sure if it is (a) intentional or (b) indicative of some other issue.

Numpy 2.0 introduced a new data type specifically for variable length strings. that's theStringDtype() you are seeing in the numpy array. I suspect that prior to my PR, zarr-python was not usingStringDtype when reading variable-length strings in zarr v2. But it should have been, IMO, so I think the current behavior is an improvement.

Agreed. The only reason it came up was that we were implicitly relying on the nullability of the object dtype in strings. So we'll transition now away from that, I think

I think the numpy string dtype hasnullability semantics, that might be worth investigating

Was this intentionally merged into main, and not the 3.1.0 branch?

Was this intentionally merged into main, and not the 3.1.0 branch?

Yes, but we can revisit that decision. I was under the impression that we were going to merge the 3.1 prs directly into main, but if we want to do a 3.0.9 release first, then we can revert this merge.

👍 - I'll delete the 3.1.0 branch then, and create a 3.0.x branch from before when this was merged.

thanks for all your work on NumPy. We'd love open the conversation about supporting writing to Zarr directly from NumPy. Can you advise on where the best place to do that is?

Sorry for the delay on this. We can schedule a call, maybe?

We have a biweekly NumPy community call you could drop into or we come to as well but unfortunately I need to miss the first half hour of that. Seethe NumPy community calendar for meeting links.

There's also a NumPy developer slack I can send you an invite to.