Repository files navigation

A simple way to create a Node.JS Transform stream which processes inparallel. You can limit the concurrency (default is 16) and order isnot preserved (so chunks/objects can end up in a different order tothe order they started in if the transform functions take differentamounts of time).

Built usingthrough2 and has thesame API with the addition of amaxConcurrency option.

Non-objectMode streams are supported for completeness but I'm notsure they'd be useful for anything.

Written by Thomas Parslow(almostobsolete.net andtomparslow.co.uk) as part of Active Inbox(activeinboxhq.com).

npm install --save through2-concurrent

Process lines from a CSV in parallel. The order the results end up intheall variable is not deterministic.

varthrough2Concurrent=require('through2-concurrent');varall=[];fs.createReadStream('data.csv').pipe(csv2()).pipe(through2Concurrent.obj({maxConcurrency:10},function(chunk,enc,callback){varself=this;someThingAsync(chunk,function(newChunk){self.push(newChunk);callback();});})).on('data',function(data){all.push(data)}).on('end',function(){doSomethingSpecial(all)})

Fixed or improved stuff? Great! Send me a pull requestthrough GitHubor get in touch on Twitter@almostobsolete or email attom@almostobsolete.net