It seems like it'd be nice to have support for running reductions in parallel.

There are some disadvantages, so this should definitely be opt-in. One issue is that there may be dependencies between chunks which need to be handled in some fashion. When you have a file with chunks like this:

ABCD

It may be that you can successfully removeC only if you have previously removedA, or that you will succeed in removingD only if you didnot already removeB. If you run the reductions where you removeA,B,C, andD in parallel, this will produce nondeterministic results.

Nondeterministic results would be acceptable for a lot of use cases, though, so I think there'd be value in supporting the feature.

Would the other users of lithium find this useful? Has anyone taken a look at doing this already?