Movatterモバイル変換


[0]ホーム

URL:


Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

Sign up
Appearance settings

Shared memory cache across multiple processes?#3488

Unanswered
golmschenk asked this question inQ&A
Discussion options

Hello! Is there an easy way in Zarr to have the in-memory cache be shared by multiple processes? I have several processes that need to load a subset of a large dataset. Caching the subset would be useful in my case. However, if each process had a separate copy of that subset, I would still run out of memory in my case. So I would like to have the multiple processes share the same underlying cache if possible. If there's a known way in which this would work using built-in Zarr capabilities, that would be ideal. I can resortmanually creating the shared memory object if needed, but there are a lot of pitfalls going through that method, so I'd like to avoid it if possible. If any one knowns if this is already existing functionality or should be easy to implement using Zarr capabilities, please let me know! Or, if you're someone that is pretty confident they know that this is probably not easy to do with existing Zarr capabilities, that would be useful to know too! Thank you for you time!

You must be logged in to vote

Replies: 1 comment 3 replies

Comment options

I'm not familiar with anyone doing this yet, but it does seem reasonable to try.https://docs.python.org/3/library/multiprocessing.shared_memory.html includes an example of using NumPy with shared memory.

I'm not immediately sure whether you'd need a custom Store (for reading data into shared memory) or a custom Buffer class, or both.

You must be logged in to vote
3 replies
@golmschenk
Comment options

I also just converted the code to Zarr 3, and it seems caching capabilities (shared or not) are limited there currently. So, for the moment, it seems I'll need to do caching on my own. Thanks!

@TomAugspurger
Comment options

#3366 might be of interest.

@golmschenk
Comment options

For future readers' reference,#3366 was just merged. This does not directly provide shared caching across processes. But does allow separate cache stores. It might be possible to share the cache store via the Python builtin shared memory tools (though I haven't checked this in any detail yet to see if Zarr stores will have issues with this).

Sign up for freeto join this conversation on GitHub. Already have an account?Sign in to comment
Category
Q&A
Labels
None yet
2 participants
@golmschenk@TomAugspurger

[8]ページ先頭

©2009-2025 Movatter.jp