Uh oh!
There was an error while loading.Please reload this page.
- Notifications
You must be signed in to change notification settings - Fork366
How to decrease traffic when writing into region of zarr archive#3089
-
we are initialize empty zarrs with shape: dataset.to_zarr(store,region=region,mode="r+",compute=True,consolidated=Trueifzarr_format<3elseFalse,zarr_format=zarr_format,safe_chunks=safe_chunks, ) During writing along the first dimension day by day, the whole chunk (aka dataset) needs to be loaded to write into region. This causes a lot of traffic. We would like to go into s3 object store with the process but we need to decrease the traffic. Do you know how to achive this? The previous solution used append mode, which causes chunk sizes (1, 96, 50, 50) and very long queries against the dataset. Best regards |
BetaWas this translation helpful?Give feedback.
All reactions
Replies: 1 comment 2 replies
-
No one ? Am I the only facing issues with region write? |
BetaWas this translation helpful?Give feedback.
All reactions
-
Define "traffic". |
BetaWas this translation helpful?Give feedback.
All reactions
❤️ 1
-
I mean the fact, to load the whole chunk to write into it. |
BetaWas this translation helpful?Give feedback.