Movatterモバイル変換


[0]ホーム

URL:


Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

Sign up
Appearance settings

How to decrease traffic when writing into region of zarr archive#3089

Unanswered
meteoDaniel asked this question inQ&A
Discussion options

we are initialize empty zarrs with shape:
(31, 96, 1000, 1000) and chunks : (31, 96, 50, 50) because we need all data from dim 1 and 2 in each query.
Now we write into region again:

dataset.to_zarr(store,region=region,mode="r+",compute=True,consolidated=Trueifzarr_format<3elseFalse,zarr_format=zarr_format,safe_chunks=safe_chunks,    )

During writing along the first dimension day by day, the whole chunk (aka dataset) needs to be loaded to write into region. This causes a lot of traffic.

We would like to go into s3 object store with the process but we need to decrease the traffic. Do you know how to achive this?

The previous solution used append mode, which causes chunk sizes (1, 96, 50, 50) and very long queries against the dataset.

Best regards
Daniel

You must be logged in to vote

Replies: 1 comment 2 replies

Comment options

No one ? Am I the only facing issues with region write?

You must be logged in to vote
2 replies
@rabernat
Comment options

Define "traffic".

@meteoDaniel
Comment options

I mean the fact, to load the whole chunk to write into it.
E.g. when I use a remote s3 this would result in additional traffic and i need to download it complete just to write some additional data into a region.
Or am I wrong with this assumption?

Sign up for freeto join this conversation on GitHub. Already have an account?Sign in to comment
Category
Q&A
Labels
None yet
2 participants
@meteoDaniel@rabernat

[8]ページ先頭

©2009-2025 Movatter.jp