Movatterモバイル変換

[0]ホーム
URL:

Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

 Sign up
Appearance settings

Commit9e8b50a

Browse files
kylebarrond-v-bjhammandcherianmaxrjones
authored
obstore-based Store implementation (#1661)
Implementation of the Zarr Store API using obstore.---------Co-authored-by: Davis Bennett <davis.v.bennett@gmail.com>Co-authored-by: Joe Hamman <joe@earthmover.io>Co-authored-by: Deepak Cherian <dcherian@users.noreply.github.com>Co-authored-by: Max Jones <14077947+maxrjones@users.noreply.github.com>Co-authored-by: Deepak Cherian <deepak@cherian.net>Co-authored-by: Tom Augspurger <tom.augspurger88@gmail.com>
1 parent8c24819 commit9e8b50a

File tree

9 files changed

+567
-23
lines changed

9 files changed

+567
-23
lines changed

‎.pre-commit-config.yaml‎

Lines changed: 7 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -8,9 +8,9 @@ repos:
88
-repo:https://github.com/astral-sh/ruff-pre-commit
99
rev:v0.9.9
1010
hooks:
11-
-id:ruff
12-
args:["--fix", "--show-fixes"]
13-
-id:ruff-format
11+
-id:ruff
12+
args:["--fix", "--show-fixes"]
13+
-id:ruff-format
1414
-repo:https://github.com/codespell-project/codespell
1515
rev:v2.4.1
1616
hooks:
@@ -19,8 +19,8 @@ repos:
1919
-repo:https://github.com/pre-commit/pre-commit-hooks
2020
rev:v5.0.0
2121
hooks:
22-
-id:check-yaml
23-
-id:trailing-whitespace
22+
-id:check-yaml
23+
-id:trailing-whitespace
2424
-repo:https://github.com/pre-commit/mirrors-mypy
2525
rev:v1.15.0
2626
hooks:
@@ -31,9 +31,10 @@ repos:
3131
-packaging
3232
-donfig
3333
-numcodecs[crc32c]
34-
-numpy==2.1# until https://github.com/numpy/numpy/issues/28034 is resolved
34+
-numpy==2.1# until https://github.com/numpy/numpy/issues/28034 is resolved
3535
-typing_extensions
3636
-universal-pathlib
37+
-obstore>=0.5.1
3738
# Tests
3839
-pytest
3940
-repo:https://github.com/scientific-python/cookie

‎changes/1661.feature.rst‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
Add experimental ObjectStore storage class based on obstore.

‎docs/conf.py‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -369,6 +369,7 @@ def setup(app: sphinx.application.Sphinx) -> None:
369369
"python": ("https://docs.python.org/3/",None),
370370
"numpy": ("https://numpy.org/doc/stable/",None),
371371
"numcodecs": ("https://numcodecs.readthedocs.io/en/stable/",None),
372+
"obstore": ("https://developmentseed.org/obstore/latest/",None),
372373
}
373374

374375

‎docs/user-guide/storage.rst‎

Lines changed: 37 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -47,7 +47,7 @@ Explicit Store Creation
4747

4848
In some cases, it may be helpful to create a store instance directly. Zarr-Python offers four
4949
built-in store::class:`zarr.storage.LocalStore`,:class:`zarr.storage.FsspecStore`,
50-
:class:`zarr.storage.ZipStore`, and:class:`zarr.storage.MemoryStore`.
50+
:class:`zarr.storage.ZipStore`,:class:`zarr.storage.MemoryStore`,and:class:`zarr.storage.ObjectStore`.
5151

5252
Local Store
5353
~~~~~~~~~~~
@@ -99,6 +99,42 @@ Zarr data (metadata and chunks) to a dictionary.:
9999
>>>zarr.create_array(store=store,shape=(2,),dtype='float64')
100100
<Array memory://... shape=(2,) dtype=float64>
101101

102+
Object Store
103+
~~~~~~~~~~~~
104+
105+
:class:`zarr.storage.ObjectStore` stores the contents of the Zarr hierarchy using any ObjectStore
106+
`storage implementation<https://developmentseed.org/obstore/latest/api/store/>`_, including AWS S3 (:class:`obstore.store.S3Store`), Google Cloud Storage (:class:`obstore.store.GCSStore`), and Azure Blob Storage (:class:`obstore.store.AzureStore`). This store is backed by `obstore<https://developmentseed.org/obstore/latest/>`_, which
107+
builds on the production quality Rust library `object_store<https://docs.rs/object_store/latest/object_store/>`_.
108+
109+
110+
>>>from zarr.storageimport ObjectStore
111+
>>>from obstore.storeimport MemoryStore
112+
>>>
113+
>>>store= ObjectStore(MemoryStore())
114+
>>>zarr.create_array(store=store,shape=(2,),dtype='float64')
115+
<Array object_store://... shape=(2,) dtype=float64>
116+
117+
Here's an example of using ObjectStore for accessing remote data:
118+
119+
>>>from zarr.storageimport ObjectStore
120+
>>>from obstore.storeimport S3Store
121+
>>>
122+
>>>s3_store= S3Store('noaa-nwm-retro-v2-zarr-pds',skip_signature=True,region="us-west-2")
123+
>>>store= zarr.storage.ObjectStore(store=s3_store,read_only=True)
124+
>>>group= zarr.open_group(store=store,mode='r')
125+
>>>group.info
126+
Name :
127+
Type : Group
128+
Zarr format : 2
129+
Read-only : True
130+
Store type : ObjectStore
131+
No. members : 12
132+
No. arrays : 12
133+
No. groups : 0
134+
135+
..warning::
136+
The:class:`zarr.storage.ObjectStore` class is experimental.
137+
102138
.. _user-guide-custom-stores:
103139

104140
Developing custom stores

‎pyproject.toml‎

Lines changed: 12 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -64,6 +64,7 @@ keywords = ["Python", "compressed", "ndimensional-arrays", "zarr"]
6464
# User extras
6565
remote = [
6666
"fsspec>=2023.10.0",
67+
"obstore>=0.5.1",
6768
]
6869
gpu = [
6970
"cupy-cuda12x",
@@ -114,6 +115,12 @@ Discussions = "https://github.com/zarr-developers/zarr-python/discussions"
114115
Documentation ="https://zarr.readthedocs.io/"
115116
Homepage ="https://github.com/zarr-developers/zarr-python"
116117

118+
[dependency-groups]
119+
dev = [
120+
"ipykernel>=6.29.5",
121+
"pip>=25.0.1",
122+
]
123+
117124
[tool.coverage.report]
118125
exclude_lines = [
119126
"pragma: no cover",
@@ -129,7 +136,9 @@ omit = [
129136

130137
[tool.hatch]
131138
version.source ="vcs"
132-
build.hooks.vcs.version-file ="src/zarr/_version.py"
139+
140+
[tool.hatch.build]
141+
hooks.vcs.version-file ="src/zarr/_version.py"
133142

134143
[tool.hatch.envs.test]
135144
dependencies = [
@@ -165,7 +174,7 @@ run-hypothesis = "run-coverage --hypothesis-profile ci --run-slow-hypothesis tes
165174
list-env ="pip list"
166175

167176
[tool.hatch.envs.doctest]
168-
features = ["test","optional"]
177+
features = ["test","optional","remote"]
169178
description ="Test environment for doctests"
170179

171180
[tool.hatch.envs.doctest.scripts]
@@ -211,6 +220,7 @@ dependencies = [
211220
'universal_pathlib @ git+https://github.com/fsspec/universal_pathlib',
212221
'typing_extensions @ git+https://github.com/python/typing_extensions',
213222
'donfig @ git+https://github.com/pytroll/donfig',
223+
'obstore @ git+https://github.com/developmentseed/obstore@main#subdirectory=obstore',
214224
# test deps
215225
'zarr[test]',
216226
]

‎src/zarr/storage/__init__.py‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,7 @@
88
fromzarr.storage._localimportLocalStore
99
fromzarr.storage._loggingimportLoggingStore
1010
fromzarr.storage._memoryimportGpuMemoryStore,MemoryStore
11+
fromzarr.storage._obstoreimportObjectStore
1112
fromzarr.storage._wrapperimportWrapperStore
1213
fromzarr.storage._zipimportZipStore
1314

@@ -17,6 +18,7 @@
1718
"LocalStore",
1819
"LoggingStore",
1920
"MemoryStore",
21+
"ObjectStore",
2022
"StoreLike",
2123
"StorePath",
2224
"WrapperStore",

0 commit comments

Comments
 (0)
[8]ページ先頭
©2009-2025 Movatter.jp