I do think that block constraints make IPFS a deal-breaker for Zarr performance, however, IPLD is a really interesting protocol and I think an integration with it for a torrent-based Zarr store would be really cool to see.

Thanks for sharing the dclimate project; it's exciting and validating to see such an effort!

Thanks so much for sharing and it's exciting to see that others see value in this effort! On the performance note, what are the speeds you're looking at achieving? We've seen without any optimizations 10-20mb/s when making requests to remote gateways and 260mb/s when running on local ipfs nodes. We think there are many areas for improvement around here such as divide and conquer algorithms withRAPIDE

For climate, weather, and geospatial datasets, read speeds in the ballpark of 3 GBPS are what we're shooting for:https://earthmover.io/blog/icechunk#:~:text=new%20async%20API.-,performance,-At%20this%20stage

I don't know the network characteristics of torrents, but I imagine large block sizes would help achieve this goal.

Ah fair enough, this is a good and I think achievable goal. Appreciate you sharing that link and hope to have something to share in the not too distant future. For reference IPFS team members have reported in the low GB/s range with the previous numbers reported above using the IPFS public network on a home network.

Edit: With a private cluster and/or an enterprise server pipe we expect to see speeds exceeding that of S3. tl;dr your download bandwidth can be completely saturated given enough peers.

I wasn't too familiar with Bittorrent w.r.t its implementation but it seems like the "block" size is essentially equivalent for torrents

With that said I'm not sure what the advantage would be for torrents over IPFS (given the latter already exists), or rather the drawback to IPFS necessitating a torrent store if blocksize is a concern. Open to any suggestions or ideas here though.

We literally had this discussion on academic torrents in our group so exciting to see this form

Really interesting Rich! The sciop.net catalog is really validating. Further, from reading your notebook, I wonder if what we need is not a new Zarr store by a torrent-based filesystem (say, in fsspec).

Sorry I missed this. Extending Zarr stores with torrents for p2p distribution is an interesting idea. Do I understand you correctly? Can you say more here?

No problem. I am trying to understand if usingTorrent ZarrStore would imply that the data represented this way would be inherently read-only / constant, given that modifying it (e.g.: incrementally adding new variables at later time) would change the.torrent metadata and pieces it contains, therefore making it a "different torrent"?

I see what you mean. I am hoping for the Zarr stores to be writeable, but optimized for read access. It's possible that the torrent format is not amenable to writes/updates. If so, thanks for calling this out early.

That said, I bet there could be someinteresting tricks for constancy and mutability under distributed replication.