Movatterモバイル変換


[0]ホーム

URL:


Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

Sign up
Appearance settings

Compatibility policy for zarr-python v3#2097

TomAugspurger started this conversation inGeneral
Discussion options

Apologies if this has been discussed already, but is there policy written down anywhere with respect tocode compatibility written against zarr 2.x? Concretely, I wonder if making this snippet using xarray is a design goal of zarr-python v3?

importxarrayasxrimportpathlibimportshutilp=pathlib.Path("out.zarr")shutil.rmtree(p,ignore_errors=True)ds=xr.tutorial.open_dataset("air_temperature")ds.to_zarr(p)

That runs fine with zarr 2.18.2; Should it work with zarr-python 3.x without modification to xarray? To give you an idea of the kinds of changes needed to make this work:

  • store is passed as a positional arg, while it's keyword only on v3 (scikit-learn had a nice decorator for deprecating positional arguments, so if we wantstore to be keyword only we can deprecate it gracefully without too much effort)
  • xarray usesAttributes.put(dict). Easy to alias to.update (with a deprecation if we want)
  • xarray accesses various attributes from theGroup object (store, read_only, synchronizer). No idea if these still make sense in the v3 codebase.
  • Group.create previously took some keyword arguments (shape, dtype, fill_value, chunks) that seem more like Array arguments. Reading throughzarr.v2.Group.create, I guess that creates an Array in the group, rather than a new group. Is that deliberate?

There's some additional examples of this in some issues (#2023,#2021)

One point worth discussing: is there value in code-level compatibility if zarr-python 3.x is (presumably) writing zarr-v3 by default? IMO yes, but perhaps not.

You must be logged in to vote

Replies: 1 comment 2 replies

Comment options

One point worth discussing: is there value in code-level compatibility if zarr-python 3.x is (presumably) writing zarr-v3 by default? IMO yes, but perhaps not.

Short answer, if we wanted to preserve code-level compatibility, we would be targeting a 2.x release and not a 3.x release.

The longer answer is that supporting zarr version 2and zarr version 3and preserving code compatibility with zarr-python v2 (which closely models zarr version 2) is not an easy path. So far, zarr-python 3.x has optimized for supporting zarr version 2 and zarr version 3, at the cost of changing zarr-python 2.x APIs. I think these are the right priorities (but others might disagree). Keep in mind that zarr v3 is a different format from zarr v2. Expecting old APIs to work for a new format seems like a stretch.

That doesn't mean we should wantonly break stuff, and we are open to a variety of solutions for minimizing the changes. The simplest thing is for users who depend on zarr-python 2.x APIs to pin their dependencies to that version of zarr-python. More complicated solutions could be including all of zarr-python 2.x inside a v2 namespace in zarr-python 3.x, or publishing zarr-python 2.x as a stand-alonezarr2 package, etc.

You must be logged in to vote
2 replies
@TomAugspurger
Comment options

TomAugspurgerAug 18, 2024
Maintainer Author

It's an interesting thought exercise for what has a longer tail: software or data files? :)

Overall I think I agree: Prioritize APIs that make sense for Zarr v3, but don't wantonly break stuff. To make this a bit more concrete I published#2098 as a draft of what it takes to getxarray.Dataset.to_zarr working (probably, I haven't verified stuff 100%).

@jhamman
Comment options

I have created#2102 to help address some of these questions. Contributions / feedback on that PR is welcome.

Sign up for freeto join this conversation on GitHub. Already have an account?Sign in to comment
Category
General
Labels
None yet
3 participants
@TomAugspurger@jhamman@d-v-b

[8]ページ先頭

©2009-2025 Movatter.jp