Interesting, thanks for the link and explanation.

This can still work for virtualization but only if byte ranges are addressible in the virtualization scheme, and the byte range for all the boundary chunks has been calculated.

Hmm I think that alone is not sufficient. For example, I have the addressable byte ranges for boundary chunks but encounter a reshape error in the ArrayBytes codec because the buffer length is less than a full chunk's buffer length. IIUC one option is to add a BytesBytes (e.g.,Pad) codec in between any compressors and the ArrayBytes codec that pads the buffer to match a full chunk and make sure that thePad codec aligns well with the defined ArrayBytes codec so that the correct bytes are truncated. Seems a bit risky though. Do you know off-hand where in the codec pipeline the truncation happens?

An alternative would be to propose a chunk grid extension, but that risks limiting interoperability.

I guess we could also define a customArrayBytes codec that pads before reshaping, again not sure if this is way too hacky though.

ah and I was wrong about the byte addressing thing -- on the encoding side, Ithink partial chunks are padded to full size before the codec pipeline runs? and if so, there's no way a byte range can be helpful, because the entire padded chunk will be compressed.