- Notifications
You must be signed in to change notification settings - Fork14
feat: ability to open a hash as a seekable bao file#72
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to ourterms of service andprivacy statement. We’ll occasionally send you account related emails.
Already on GitHub?Sign in to your account
base:main
Are you sure you want to change the base?
Uh oh!
There was an error while loading.Please reload this page.
Conversation
Documentation for this PR has been generated and is available at:https://n0-computer.github.io/iroh-blobs/pr/72/docs/iroh_blobs/ Last updated: 2025-03-11T19:08:54Z |
maboesanman commentedMar 11, 2025 • edited
Loading Uh oh!
There was an error while loading.Please reload this page.
edited
Uh oh!
There was an error while loading.Please reload this page.
Upon reading the signature pubasyncfnopen(&self,hash:Hash) ->Result<impl std::io::Read + std::io::Seek>{todo!()} I would personally expect this to be pending until the entire blob is downloaded, then give me readers that won't error (except under extreme cases like file permissions changing or storage devices being physically removed), but it sounds like the implementation (as described in the PR) gives you a future which resolves much more eagerly, independent of whether the blob is downloaded. This seems like it would be surprising and potentially frustrating if passing this to some code that expects read failures to be unlikely. Maybe this api could be adjusted to something like: pubasyncfnopen_partially_downloaded(&self,hash:Hash) ->Result<impl std::io::Read + std::io::Seek>{// current implementation of the open function}pubasyncfnopen(&self,hash:Hash) ->Result<impl std::io::Read + std::io::Seek>{let download_complete_fut =some_future();// resolves when the download is completelet reader_future =open_partially_downloaded();let(_, reader) =join!(download_complete_fut, reader_future).await;Ok(reader?)} |
rklaehn commentedMar 12, 2025 • edited
Loading Uh oh!
There was an error while loading.Please reload this page.
edited
Uh oh!
There was an error while loading.Please reload this page.
Yeah, naming is TBD. This is for the YOLO case where you just want to try to read. I think that is a quite valid use case. The other variant is also possible, but since it requires interaction with the store and the downloader it needs a living store and protocol handler, whereas this gives you a fully independent "file" that stays alive even if the store is dropped. |
Uh oh!
There was an error while loading.Please reload this page.
Description
Adds the ability to open apartial or complete hash as a file that supports std::io::Read and std::io::Seek. If you read at a position where the data is not present, you will get an io error, since all reads are validated.
The created file will be completely independent of the store, you can use it even when the store is dropped.
This relies on some experimental code in bao-tree. I will also support it in the new store.
The underlying implementation is using positioned_io::ReadAt, and I also have a wrapper similar to tokio::io::Cursor that wraps the file so that it impls tokio:io::AsyncRead and tokio::io::AsyncSeek.
Breaking Changes
None?
Notes & open questions
This is very WIP
Change checklist