Uh oh!
There was an error while loading.Please reload this page.
- Notifications
You must be signed in to change notification settings - Fork366
Shared memory cache across multiple processes?#3488
Uh oh!
There was an error while loading.Please reload this page.
Uh oh!
There was an error while loading.Please reload this page.
-
Hello! Is there an easy way in Zarr to have the in-memory cache be shared by multiple processes? I have several processes that need to load a subset of a large dataset. Caching the subset would be useful in my case. However, if each process had a separate copy of that subset, I would still run out of memory in my case. So I would like to have the multiple processes share the same underlying cache if possible. If there's a known way in which this would work using built-in Zarr capabilities, that would be ideal. I can resortmanually creating the shared memory object if needed, but there are a lot of pitfalls going through that method, so I'd like to avoid it if possible. If any one knowns if this is already existing functionality or should be easy to implement using Zarr capabilities, please let me know! Or, if you're someone that is pretty confident they know that this is probably not easy to do with existing Zarr capabilities, that would be useful to know too! Thank you for you time! |
BetaWas this translation helpful?Give feedback.
All reactions
Replies: 1 comment 3 replies
-
I'm not familiar with anyone doing this yet, but it does seem reasonable to try.https://docs.python.org/3/library/multiprocessing.shared_memory.html includes an example of using NumPy with shared memory. I'm not immediately sure whether you'd need a custom Store (for reading data into shared memory) or a custom Buffer class, or both. |
BetaWas this translation helpful?Give feedback.
All reactions
-
I also just converted the code to Zarr 3, and it seems caching capabilities (shared or not) are limited there currently. So, for the moment, it seems I'll need to do caching on my own. Thanks! |
BetaWas this translation helpful?Give feedback.
All reactions
-
#3366 might be of interest. |
BetaWas this translation helpful?Give feedback.
All reactions
👍 1
-
For future readers' reference,#3366 was just merged. This does not directly provide shared caching across processes. But does allow separate cache stores. It might be possible to share the cache store via the Python builtin shared memory tools (though I haven't checked this in any detail yet to see if Zarr stores will have issues with this). |
BetaWas this translation helpful?Give feedback.