- Notifications
You must be signed in to change notification settings - Fork1.3k
-
Hi, I can't really seem to find how to do the following (in ZIO 2.0), which feels like it should be easy (.. famous last words): Given for example with say a size of 68 megabytes, I would like to produce a series of Traversing those (8mb) sub-streams would be done sequentially so that the original stream is also just traversed once from front to back (it could also come from the network and avoiding buffering would be good). If I use Any hints as to which functions would be useful for this, which methods to look at for inspiration, or which approach I could try to take, would be highly appreciated. [Why? Splitting large files when storing them in SeaweedFS] |
BetaWas this translation helpful?Give feedback.
All reactions
Replies: 2 comments
-
I have something working by using traitStreamSplitter {defgetNext:ZIO[Any,Throwable,Option[ZStream[Any,Throwable,Byte]]]}objectStreamSplitter {defmake(input:ZStream[Any,Throwable,Byte],splitSize:Int)=for { pull<- input.toPull leftover<-Ref.make(Option.empty[Chunk[Byte]]) finished<-Ref.make(false) }yield {defpuller(readRef:Ref[Int])= {for { read<- readRef.get result<- (read== splitSize)match {casetrue=>ZIO.fail(Option.empty[Throwable])case _=>for { lf<- leftover.get n<- (lfmatch {caseSome(s)=>for { _<- leftover.update(_=>None) }yield scase _=> pull }).either res<- nmatch {caseLeft(e)=>for { _<- finished.update(_=>true) result<-ZIO.fail(e) }yield resultcaseRight(r)=> {if (read+ r.size> splitSize) {valmissing= (splitSize- read).toIntvaltoReturn= r.take(missing)valtoRemain= r.drop(missing)for { _<- leftover.update(_=>Some(toRemain)) _<- readRef.update(_=> splitSize) }yield toReturn }else {for { _<- readRef.update(_=> read+ r.size) }yield r } } } }yield res } }yield result }newStreamSplitter {defgetNext= finished.get.map{casetrue=>Nonecasefalse=>Some(ZStream.fromPull(for { read<-Ref.make[Int](0) }yield puller(read))) } } }} Any style tips or improvement suggestions of course welcome, for example:
|
BetaWas this translation helpful?Give feedback.
All reactions
-
I think you just want to use |
BetaWas this translation helpful?Give feedback.