I would say it's not really the loop or the function call that will make the difference, but (as other benchmarking already found) memcopies, cpython API calls and syscalls. For instance

def g(n):    for i in range(n):        pass

10.2ms

def g(n):    for i in range(n):        b[10:20]

(where b is a bytes object)
61ms

Hi@jni, ha! Sorry I didn't "connect the dots" between your blog and your GitHub ID! It's a small world, huh?!? Thanks loads for re-doing the benchmarks!

@martindurant, I completely agree that memcopies are surprisingly slow. As you've mentioned in the past, the ideal case would be that - for uncompressed Zarr chunks - we can do exactly one copy: from storage to the final numpy array (ideally using DMA). If we have batched processing of chunks, then there's at least a chance that, even for chunks that need to be "scattered" throughout the final numpy array, it'd still be possible to do that using DMA (e.g. usingreadv to read different parts of a single chunk into different memory locations, with a single system call).

I was not really advocating for a low level super fix like that, but by all means try it. I think the python readinto and decompress_into are probably enough, but the point is that the "batch" concept I don't think will make much difference so long as high latency reads are awaited together. I consider this the low-hanging fruit, together with not reading bytes we don't need.

All contingent on contiguous memory blocks. Where striding is at issue, we probably can't do better than numpy copies, unless we write our own decompression wrapper capable of "getting every fourth uncompressed byte" or similar pattern.

On the topic of where to put this in the stack...

I feel I'm too new to the Zarr-Python world to have a particularly strong grasp of the pros and cons of where to place this. But I will give it more thought.

One option thatI'm considering for my Zarr Rust experiments, is to have a method which - ported to Python - would be something like:

classStore:asyncdefget_items(self,keys:Iterable[str],transform:Optional[Callable]=None   )->list[bytes]:"""Get multiple chunks.   Args:       keys: List of chunk keys.       transform: A function that will be applied to every chunk. The function must take two arguments:               - chunk_id: int               - chunk_data: bytes           And it must return bytes or None (if, for example, the processing function moves data to a final array)   """

Then theArray can ask theStore to perform any arbitrary transform on each chunk (e.g. decompress and copy to the final numpy array). My motivation is twofold: I want processing to start as soon as each chunk is available, using a threadpool (one thread per CPU core). And I wantall processing (decompression, maybe some numerical transform, and copying to the final array) to happen in very quick succession, to maximise that chance that the chunk stays in CPU cache.

So, this kind of dodges the question of "where to put decompression". TheStore can perform any arbitrary computation. But the decompression algorithm(s) wouldn't by defined in theStore.

But I don't know if that's appropriate for Zarr-Python. Nor have I actually implemented this in Rust yet, so I don't know if it's a good idea for Rust, either!

Were you followingrfsspec? That does the concurrent reads; since cramjam is rust-python, I thought it could nicely "decompress into" numpy buffers in CPU threads driven by tokio.