@nenb thank you so much for working on this. I really like the ideas in here -- it looks like a great addition to the library, and I will try to give this close examination, and I recommend the other @zarr-developers/python-core-devs do the same. This is a big, forward-looking change so I imagine we will have a lot of opinions on multiple scales here.

Our weekly dev meeting is tomorrow (here's thecalendar info, the particular meeting I'm referring to is thezarr-python developers meeting), if the timing works for you, would it be possible for you to attend and give an overview of this PR?

@nenb thank you so much for working on this. I really like the ideas in here -- it looks like a great addition to the library, and I will try to give this close examination, and I recommend the other @zarr-developers/python-core-devs do the same. This is a big, forward-looking change so I imagine we will have a lot of opinions on multiple scales here.

Our weekly dev meeting is tomorrow (here's thecalendar info, the particular meeting I'm referring to is thezarr-python developers meeting), if the timing works for you, would it be possible for you to attend and give an overview of this PR?

@d-v-b Thanks for being so receptive! I'm happy to attend and give an overview.

What's likely the best format to present at the meeting (I haven't attended before). If I give a 5 minute overview (including some motivation from the AI-side of things), and then leave 10 minutes for questions, would that work? Is there anything specific that you think it's worth focusing on e.g. extension mechanism?

@nenb that plan sounds good. I'm looking forward to it!

@d-v-b@normanrz Thanks so much for the support during today's meeting, I'm very excited about the potential forzarr to help accelerate the AI field!

My current thinking is to close this PR (as it was just intended as an illustration of what I was hoping to gain support for), and then sketch out how to proceed elsewhere. Would you agree?

If so, could you give me a few pointers about how best to support coordination of this work please e.g. where to document a proposed list of tasks, who needs to be tagged etc.?

As mentioned, I'm very flexible as to how the work proceeds! My main interests are i) allowing users to be able to register their own dtypes, ii) decoupling thezarr-python internal dtype API from thenumpy dtype API (because I thinknumpy dtype API is not appropriate for AI work at the moment) and iii) trying to achieve these things in the next couple of weeks.

I know this last point is a little artificial, but as mentioned during the meeting earlier, a lot of this field moves fast at the moment!

A heads up of a discussion aroundZEP9 & extension naming:

https://ossci.zulipchat.com/#narrow/channel/423692-Zarr/topic/ZEP9.3A.20extension.20points

/me reminds himself yet again to look for a bot to do these cross-linkings...

@nenb could you explain the motivation for centering the design around classes that have no instance attributes? E.g., you have

classFloat16(ZarrDType):zarr_spec_format="3"experimental=Falseendianness="little"byte_count=2to_numpy=np.dtype('float16')

With this design,__init__ is a no-op, so from what I can tell there's not really a difference betweenFloat16 andFloat16()

An alternative implementation would be:

float16=ZarrDType(zarr_spec_format=3,experimental=False, ...)

I'm guessing there was a reason why you didn't opt for the "specify the dtype in__init__" approach?

@d-v-b
TL;DR
I don't remember there being anything important behind this. From memory, I was just trying to hack something together quickly, and I think this is what I ended up with. I had a couple of 'requirements' in mind, which are described in details below, and satisfying these were more important to me than any of the (throwaway) code that I wrote here.

Details
Requirements that I had:

• once a dtype has been created, don't allow patching the attributes later
• don't allow creation of a dtype instance with invalid attributes or with missing required attributes
• make sure that the dtype can be discovered via the current entrypoint mechanism

For the first point, I thought it would lead to weird and confusing situations if users were able to (for example) change thebyte_count to a different value after creation. I thought it best to make it as difficult as possible to modify the dtypes after creation, because they are fundamental to a lot of behaviour. (Personal choice)

For the second point, this was just my attempt at re-attempting Pydantic with metaclasses and convoluted code. 🤡 I think it's useful though to prevent creation/importing bad dtype classes, for the same reason I mentioned in the previous paragraph.

I didn't want to make many changes to the entrypoint code, as I felt it would distract from what I was trying to show here about the dtypes. This is probably why I ended up with the weird design for classes that you mentioned i.e. I just kept hacking until I got everything to work with the current entrypoint mechanism. Obviously, this is not a requirement when doing this work properly.

I would personally be happy with any code that can satisfy the above three requirements. And I'm happy to discuss any of these requirements if they considered unhelpful for whatever reason!

@nenb what are you thoughts on the following things:

• handling parametric dtypes, like [datetime64[ms],datetime64[s], ...], [S1,S2, ...]? For the datettimes the set of units is small enough that these can be created in advance, but the fixed length types like<S | V><length> are harder to pre-generate. If a user shows up with a concrete numpy dtype likeS2, it's possible to create the corresponding class at runtime with something liketype(f"FixedLengthString{length}", (ZarrDType,), {...kwargs}), but this feels a bit clunky compared to creating an instance of a variable-length string class, e.g.VarLenString(length=2).
• I'm not sure we should flag experimental vs non experimental dtypes. Any experimentalness should be represented by the context in which the dtype is used, not the dtype itself, because e.g. extension dtypes will mature, and then cease to be experimental, and that should not require changing the API of the dtype object itself.
• I'm not sure the zarr format should be part of the dtype metadata. We should instead have a zarr-format-independent representation of dtypes, and treat the zarr format as a serialization issue.
• Endianness is tricky because it isn't part of the dtype model of zarr v3 -- in v3,endianness is specified by a codec, not the dtype itself. So from a v3 POV, little-endian uint16 and big-endian uint16 specify two arrays with the same dtype but different codecs. This might mean that any dtype object has to takeendianness as a parameter when serializing or generating numpy dtypes.

@d-v-b (I'll answer the rest of this message tomorrow)

handling parametric dtypes ...

Looking at the Arrowdata types (Arrow is reasonably mature at this point, I think that it's probably reasonable to adopt their design decisions, unless there is a clear reason not to?), it looks like having a type parameter is sensible. I don't think I gave this enough thought previously. I think an explicit reference to type parameters in the Zarr v3 spec is also lacking. What are your thoughts about type parameters?

A potential challenge that I see would be in naming these parametric dtypes. Do you see issues with assigning the parameters in theconfiguration field e.g.

{  "data_type": {    "name": "DateTime",    "configuration: {        "unit": "ms",        "timezone": "UTC"        }  }}

Something similar could happen for fixed-width strings. Is it critical that a dtype be identified by its name, or is it allowed to stuff identifying information inconfiguration like this?

The alternative seems a rather crude naming scheme where we attach the parameters to the type name.

it's possible to create the corresponding class at runtime ...

For (numpy) data types such as fixed-width strings and bytestrings (codesU andS respectively), I was leaning towards creating them at runtime. I hadn't considered how to handle more parametric dtypes more generally (see previous paragraph), which was an oversight on my part.

I personally don't see an issue with creating parametric dtypes at runtime, but open to change my mind on this.

Looking at the Arrowdata types (Arrow is reasonably mature at this point, I think that it's probably reasonable to adopt their design decisions, unless there is a clear reason not to?), it looks like having a type parameter is sensible. I don't think I gave this enough thought previously. I think an explicit reference to type parameters in the Zarr v3 spec is also lacking. What are your thoughts about type parameters?

You're right that the Zarr V3 spec is not really clear about parameters for data types. On one hand it says that externally-defined data types can be identified with a{"name": <name>, "configuration": {...}} JSON object, and as you note the configuration is a natural place to put type parameters. On the other hand, the v3 spec already defines a parametric data type (r*, fixed-size bytes, nearly the same as the numpyV<num_bytes> dtype), and in this case the string name contains the parameters. So it seems like "stringly typed" and typed are both on the table.

In terms of personal preference I would lean toward putting parameters in aconfiguration object, and it seems like we are aligned there. But if someone really wants superficial similarity to numpy, then this would argue for baking parameters in the string name of the dtype.

From an API POV, each variation of a parametrized type could be a separate class, or instances of a class:

classS1:capacity:ClassVar[Literal[1]]=1classS2:capacity:ClassVar[Literal[2]]=2s1=S1()s2=S2()...

vs

@dataclassclassS:capacity:ints1=S(capacity=1)s2=S(capacity=2)

I'm leaning toward the second formulation, but I'd be open to feedback for what the first is better / preferred

I'm leaning toward the second formulation, but I'd be open to feedback for what the first is better / preferred

I vote for this as well. If there is more than one type parameter, the first formulation could be confusing.

I'm not sure we should flag experimental vs non experimental dtypes. Any experimentalness should be represented by the context in which the dtype is used, not the dtype itself, because e.g. extension dtypes will mature, and then cease to be experimental, and that should not require changing the API of the dtype object itself.

I had in mind that this would be a private part of the dtype API, so that someone could inspect a dtype at runtime and know whether or not it was part of the spec. Let's say I wanted to write code that only applied to experimental dtypes (e.g. emit warnings). How would code be able to infer if a dtype was experimental or not at runtime? Remember that the definition of what is a core type can change from one day to the next, but the (zarr-python) code itself will still be the same. The only way I can solve this is for the creator to signal the intent via an attribute likeexperimental. Perhaps what I'm saying is that experimentalness is a function of how the dtype iscreated rather than how it isused? This might need more discussion though, I'm not terribly confident in what I've written.

I'm not sure the zarr format should be part of the dtype metadata. We should instead have a zarr-format-independent representation of dtypes, and treat the zarr format as a serialization issue.

I think this is another variation of the last paragraph. How do I know what format a dtype belongs to at runtime? I could imagine this being useful at some point in the future (but maybe in the very distant future 😆). I don't have strong opinions on this though, and if we were able to infer the format of a dtype at runtime another way, I would also be happy with that.

Endianness is tricky because it isn't part of the dtype model of zarr v3 -- in v3,endianness is specified by a codec, not the dtype itself. So from a v3 POV, little-endian uint16 and big-endian uint16 specify two arrays with the same dtype but different codecs. This might mean that any dtype object has to take endianness as a parameter when serializing or generating numpy dtypes.

Yeah, I remember this causing me quite a few headaches. I think given all thezarr-python code currently out there, folks are going to be trying to do things like<i2 for quite a while into the future. I added theendianess attribute as a (private) part of the API, to try and helpzarr-python developers support this use-case. Basically,zarr-python could use this attribute to track the endianess of the dtypein memory. So, when I create my array inzarr-python using>i2,zarr-python is still able to support this use-case by creating a big-endian dtypein memory. But, when serialising to disk, it would follow the spec and ultimately store this information in the codec. It would basically be an implementation detail ofzarr-python.

I think I remember you pointing out in earlier issues, that using codecs in this way diverges from thenumpy dtype model, and this makes it difficult forzarr-python, give its close historical relationship withnumpy. Adding the attribute on the dtype was my way of trying to support this. Again, it would ultiamtely be serialised to disk as part of the codec, but it would be part of the (Python) dtype private API as well, to continue support use cases like creating arrays with>i2 and<i2.

Let me know if this isn't clear. Given the amount of issues related to endianness for v3 inzarr-python, I would love to try and solve as many of these as possible with a new dtype API.

with thespiritual successor to this pr merged, can we close this?

Closing after merge of#2874