Movatterモバイル変換


[0]ホーム

URL:


Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

Sign up
Appearance settings

Use unsigned bytes to back Buffer#2738

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to ourterms of service andprivacy statement. We’ll occasionally send you account related emails.

Already on GitHub?Sign in to your account

Merged
d-v-b merged 6 commits intozarr-developers:mainfromQuLogic:unsigned-buffer
Apr 19, 2025

Conversation

@QuLogic
Copy link
Contributor

This makes compressors consistent with v2, and seems more correct than signed bytes.

Fixes#2735

Note that this does not implement accepting both signed and unsigned, as there wasn't a conclusion in#2735 about that.

TODO:

  • Add unit tests and/or doctests in docstrings
  • Add docstrings and API docs for any new/modified user-facing classes and functions
  • New/modified features documented indocs/user-guide/*.rst
  • Changes documented indocs/release-notes.rst
  • GitHub Actions have all passed
  • Test coverage is 100% (Codecov passes)

madsbk reacted with thumbs up emoji
@QuLogic
Copy link
ContributorAuthor

PS, going to ignore tests for now.

This makes compressors consistent with v2, and buffers consistents with`bytes` types.Fixeszarr-developers#2735
@github-actionsgithub-actionsbot added the needs release notesAutomatically applied to PRs which haven't added release notes labelJan 25, 2025
@dstansby
Copy link
Contributor

I just fixed uploading code coverage for the GPU tests at#2767, so hopefully when I merge inmain here it should improve the patch coverage 🤞

@maxrjones
Copy link
Member

maxrjones commentedApr 11, 2025
edited
Loading

Hi@QuLogic and@dstansby, thanks for working on this! Could I help at all with resolving the codecov issue? I'd be super excited to see this completed to enable further trying out imagecodecs with Zarr V3 (e.g.,https://github.com/maxrjones/virtual-tiff/pull/4).

@jakirkham
Copy link
Member

(clicking update branch in case that fixes it)

@jakirkham
Copy link
Member

Closing and reopening to hopefully fix RTD (it has agit checkout error)

@jakirkham
Copy link
Member

Ok better RTD checked out the code 😌

@jakirkham
Copy link
Member

Looks like Codecov got stuck somewhere. Let's try another close/reopen

@maxrjones
Copy link
Member

I don't think that the codecov/patch target will be hit unless the overall coverage actually increases, rather than remains constant. The issue is that two of the lines changed (L87 and L110) are not tested in main either -https://app.codecov.io/gh/zarr-developers/zarr-python/blob/main/src%2Fzarr%2Fcore%2Fbuffer%2Fgpu.py, so the PR isn't decreasing code coverage but still will not hit the 80% target for code coverage for patches (i.e., all new/modified lines of code). Do you prefer to ignore the codecov/patch target or to include unit tests for adding and creating zero length buffers?

@jakirkham
Copy link
Member

If someone can add some tests, that would be welcome

What was concerning to me was one Codecov job says "failing after 1s". Though this may be a non-issue and just the report percentage (as you suggested)

@maxrjones
Copy link
Member

If someone can add some tests, that would be welcome

I added test coverage for the GPU buffer in#2978. However, the test for the GPU buffer with the sharding codec is failing. Debugging that failure would require a cuda-compatable GPU but I don't have one easily accessible. I am wondering if you'd consider merging this PR prior to addressing the failures in#2978 for a few reasons:

  1. The impact on the CPU buffer is fully tested and much more commonly used
  2. The sharding codec failure with the GPU buffer is independent of this PR, such that these changes would not introduce new bugs (Add more unit tests for GPU buffer #2978)
  3. Using unsigned bytes in the arrays in the buffer class would unblock imagecodecs usage for both V2 arrays (Buffer uses signed bytes with v2 compressors #2735) and V3 arrays (e.g.,https://github.com/maxrjones/virtual-tiff/blob/main/src/virtual_tiff/imagecodecs.py, which currently requires installing from this PR)

Thanks for your consideration, and@d-v-b for the helpful discussion and recommendation to post this suggestion at the Zarr-Python dev call today!

@d-v-b
Copy link
Contributor

thanks for pushing this along@maxrjones. Based on the changes in this PR, I'm happy merging this regardless of what codecov says at the moment.

Copy link
Contributor

@d-v-bd-v-b left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

to repeat what I said in thelinked issue, the particular numpy dtype we use in our buffer implementation should be considered an implementation detail by consumers. But for the same reason we can alter that dtype as needed, e.g. to make life easier for imagecodecs users, which is what this PR does. I'm going to approve and merge this despite the low test coverage because adding those tests depends on getting the gpu buffer to work with sharding, which is IMO out of scope for this PR.

@d-v-bd-v-b merged commitbb55f0c intozarr-developers:mainApr 19, 2025
29 of 30 checks passed
d-v-b added a commit to d-v-b/zarr-python that referenced this pull requestApr 20, 2025
This makes compressors consistent with v2, and buffers consistents with`bytes` types.Fixeszarr-developers#2735Co-authored-by: Davis Bennett <davis.v.bennett@gmail.com>Co-authored-by: David Stansby <dstansby@gmail.com>Co-authored-by: jakirkham <jakirkham@gmail.com>
tomwhite added a commit to tomwhite/zarr-python that referenced this pull requestApr 21, 2025
rapids-botbot pushed a commit to rapidsai/kvikio that referenced this pull requestApr 22, 2025
This adds compatibility for zarr-python 3.0.7, which changed the dtype for representing bytes inzarr-developers/zarr-python#2738.Authors:  - Tom Augspurger (https://github.com/TomAugspurger)Approvers:  - Tianyu Liu (https://github.com/kingcrimsontianyu)  - Benjamin Zaitlen (https://github.com/quasiben)URL:#699
d-v-b added a commit that referenced this pull requestMay 14, 2025
* Avoid memory copy in obstore write* Add as_bytes_like method to Buffer* Add changelog entry* No need to take unsigned bytes view following#2738* Change method name to `as_buffer_like`---------Co-authored-by: jakirkham <jakirkham@gmail.com>Co-authored-by: Davis Bennett <davis.v.bennett@gmail.com>
@QuLogicQuLogic deleted the unsigned-buffer branchAugust 23, 2025 09:41
Sign up for freeto join this conversation on GitHub. Already have an account?Sign in to comment

Reviewers

@d-v-bd-v-bd-v-b approved these changes

Assignees

No one assigned

Labels

needs release notesAutomatically applied to PRs which haven't added release notes

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Buffer uses signed bytes with v2 compressors

5 participants

@QuLogic@dstansby@maxrjones@jakirkham@d-v-b

[8]ページ先頭

©2009-2025 Movatter.jp