This could be exposed as what I call a "subscript method", i.e. syntax likearr.lazy[10:20] orarr.lazy.oindex[10:20].

Note that while__setitem__ can return a value, when using__setitem__ in the normal way, asx[a] = b, which is the whole point of__setitem__, there is no way to access the return value. Consequently, it does not make sense for__setitem__ to return aFuture-like type. In TensorStore, while__getitem__ just does lazy indexing,__setitem__ always does a synchronous assignment. However, when using Transactions (https://google.github.io/tensorstore/python/api/tensorstore.Transaction.html#),__setitem__ synchronously assigns within the transaction, but the separate commit operation (which can be done asynchronously) is needed to actually do the write. A similar transaction mechanism implemented in zarr-python may be one way to use__setitem__ even when async behavior is desired.

A natural way to implement this type of composition of indexing operations would be use something similar to the IndexTransform (https://google.github.io/tensorstore/index_space.html#index-transform) concept used in TensorStore.

I would second not changing the core API. I think it's super useful that zarr andh5py are basically interchangeable at the API level, and wouldn't want to break that. Also don't want to introduce a large barrier to upgrading.

I also lean toward keeping the behavior as is at this time unless we have a strong argument for how zarr-python could optimize indexing operations at load time.

I would look closely at Xarray's"Named-Array" and Lazy Indexing classes. We have talked for a long time about how these classes would be very useful outside of Xarray and this seems like a great place to test that idea. cc@andersy005 who has been working hard on this topic lately.

Honestly, theNamedArray object is going to fit so well with Zarr V3 that we should think more about how to integrate them directly.

The way I was imagining it, users who don't want a numpy array would not be callingnp.array(s0) -- they would instead callmy_non_numpy_library.array(s0), e.g.cupy.array(s0)

This would require xarray, dask et al. to look atmeta_array then. Seems OK in principle. This is the relevant line in dask:https://github.com/dask/dask/blob/4eb026cc9c495a9e64130764498f5ff8b61d7602/dask/array/core.py#L106

zarr-python already usesmeta_array (for chunks that should decompress to cupy arrays), so hopefully xarray and dask are OK with it in practice, too!

Yeah +1 to what@d-v-b says and using meta-array. From the perspective of napari and any general array processing libraries, a.load is very frustrating because it may work sometimes or in other cases (e.g. plain numpy arrays).np.array(lazy_or_gpu_or_whatever_array) shouldalways work, imho. Some libraries want aforce= keyword and it's on my perpetual to-do list to make a NEP for that. 😮‍💨

napari and any general array processing libraries, a .load is very frustrating because it may work sometimes or in other cases (e.g. plain numpy arrays).

If napari used xarray we would paper over this for you 😉

To convert an arbitrary type to a numpy array, numpy supportsarray:https://numpy.org/devdocs/user/basics.interoperability.html

For synchronous conversion that is probably adequate, but indeed there is no standard for async conversion. Tensorstore uses read.

Yes sorry I should have written "in-memory array" rather than numpy array - see Deepak's comment#1603 (comment)

I see --- so the goal is to be able to somehow configure for aZarrArray object the associated "memory region" (i.e. system memory or the memory of a particular attached accelerator device, or perhaps a particular system memory NUMA node) and possibly associated representation (i.e. regular dense array or particular sparse array representation), and this would constrain the output array forread calls where an existing output array is not specified.

Presumably the associated memory type and representation would also influence how the entire read operation is done (and therefore would require rather deep integration), since if it is just reading into a dense system memory array and then copying at the end, there isn't much benefit.

I can see how this may be useful but I am also wary of trying to solve this problem immediately unless someone has a concrete proposal of how it will work, since it also adds a lot of complexity and the design will likely require careful consideration of how the API might be used. For example when using accelerator devices it is often desired to issue operations asynchronously, in a way that is not necessarily compatible with Python asyncio.

In terms of high-level API, I could imagine that instead of configuring the output array constraints on theZarrArray object itself, the output array constraints are specified when callingread. Whether that is more natural for users and/or implementations is unclear to me, though.

In any case I would agree that it would be valuable to standardize synchronous and asynchronousread andwrite APIs,

Thanks for the feedback@mullenkamp. I want to emphasize that it will always be easy to convert zarr-backed data into a numpy array. But from my experience, eagerly converting large datasets into numpy arrays is not a design decision that reduces complexity.

I work with multi-TB datasets. In zarr today, I can basically never run this line of code:my_array[1:]. If I run that line, my machine will lock up as zarr attempts to sequentially load the entire multi-TB dataset into memory before converting it to a numpy array. My system will run out of memory before this completes. So I have to use dask, or some other library, for the simple task of slicing my array, whichdrastically increases the complexity of my workflow. That's what we want to fix: zarr users with large arrays should be able to callmy_array[1:] and get back something useful without locking up their computer.

We know that most zarr users (myself included) want an easy path to get a numpy array. That path will always be there. But the current design of zarr puts an undue burden on users with big datasets, and the proposed design actually simplifies the situation considerably for those users.

When we do start writing code to implement this design, it will be extremely useful to get feedback, so I hope you try out what we end up writing and give us your thoughts!

@mullenkamp - thanks for sharing your thoughts here.

Putting my Xarray developer hat on briefly, I would love to hear more about the problems you ran into (memory, cache management, etc). Any chance you could open a separate discussion on this set of pain points?

Hi@d-v-b . I really appreciate your reply and feedback.

Personally, if someone would writemy_array[1:] in their code when they wanted to do some computation on the data, then the code should fail and the user shouldn't write such code when it's obvious that it will return an in-memory numpy array. As with h5py, they should iterate through the chunks and process the data accordingly. Just like with my normal use-cases, they likely would want to then save their results to another persistent store possibly in the same shape as the original. The user would process the data in chunks, then immediately write each chunk to a "file".

I do recognise that zarr (and the developers) might be more Dask-minded in the sense of wanting to let Dask handle the data and processing. So doingmy_array[1:] followed by some computation that dask will handle magically seems to be the route that zarr (and xarray) has chosen. Am I correct? I do want to better understand the data handling and processing coding logic of zarr and xarray.

Thanks!

Hi@jhamman .
Sure. I'll open a discussion in the xarray repo. I and many of my colleagues use xarray a lot. So I'm very happy to provide feedback if it'll help.

xarray and/or dask are still the state of the art in terms of lazy loading.