Repository files navigation

lerna-parallelism is an extension oflerna that adds supportfor deterministically partitioning packages to allow parallel execution acrossmultiple workers, likeCircleCI'sparallelism feature.

• Introduction
  • Use Case
  • Quick Example
• Installation
• Usage

Speeding up monorepo CI pipelines via multi-worker parallelization.

You maintain a largelerna monorepo with lots of individual packages.In CI you run a script / command for many or all of the packages.

# Execute the `test` script for all packages.lerna run \# Prefix each log line with the package name.# https://github.com/lerna/lerna/tree/main/commands/run#--stream  --stream \# Run packages sequentially to avoid interleaved log output and resource contention.# https://github.com/lerna/lerna/blob/main/core/global-options/README.md#--concurrency  --concurrency 1 \test

Your CI workflow is taking far too long, because you process all packagessequentially on a single CI worker instance / node / VM.

Your CI service supports spinning up multiple worker instances, but you don'tknow how telllerna to distribute the workload across these instances. Youcould use--scope and hard-code all package names, but this isdifficult to maintain, as packages are added, renamed or removed.

lerna-parallelism makes it easy to split the workload — the packages to beprocessed — dynamically and deterministically across separate worker instances.

lerna-parallelism run \# Keep your `lerna` options just like before.  --stream \  --concurrency 1 \# Divide the list of packages to process into 4 equal-sized chunks.# This should be equal to the total number of worker instances.  --split 4 \# Take only the first chunk (zero-based).# This should be the index of the individual worker instance you're running on.  --partition 0 \# Run the `test` script for all packages in that chunk only.test

yarn add -D lerna-parallelism# And then run via:yarn lerna-parallelism ...

yarn global add lerna-parallelism# orvolta install lerna-parallelism# And then run via:lerna-parallelism ...

• Global Options
  • Partition Count:--split,--partition
  • Distribution Strategy
    • --distribute-by count
    • --distribute-by weight
• Commands
  • lerna-parallelism run
  • Other Commands

lerna-parallelism adds a few CLI options on top oflerna. The followingoptions are globally understood by all commands supported bylerna-parallelism.

All other command options behave just like the upstreamlerna version of therespective command.

Split the list of packages into--split n chunks and take the chunk withzero-based index--partition n.

• --split n: The number of split partitions.Defaults to$CIRCLE_NODE_TOTAL.
  This should be equal to the total number of worker instances.
• --partition n: Which partition to execute, zero-based.Defaults$CIRCLE_NODE_INDEX.
  This should be the index of the individual worker instance you're running on.

Algorithm to use for deciding which package goes into which partition.

Default. Split into even-sized partitions.

The packages are taken in order as emitted bylerna and simply split into--split n partitions of equal size.

If the total package count cannot be distributed evenly, some partition(s) maybe slightly smaller than the other partition(s).

Example: 26 packages (nameda–z) are split into--split 4 partitions.

• --partition 0 (size = 7):a,b,c,d,e,f,g
• --partition 1 (size = 7):h,i,j,k,l,m,n
• --partition 2 (size = 6):o,p,q,r,s,t
• --partition 3 (size = 6):u,v,w,x,y,z

Assign a weight to each package and split into even-weighted partitions.

The packages are ordered by "weight" in descending order and iterativelyassigned to the current lightest partition. This results in partitions withapproximately equal weight, but uneven size by total package count.

This strategy is useful to account for different CI runtimes per package.

Each package's weight is read from thelernaPackageWeight property from itspackage.json. If missing, it defaults to1. You can change the propertylookup name with the--packageWeightKey option, like--packageWeightKey testRuntime. It just has to be anumber.

You're expected to add these weights yourself, where necessary. Basing theweights off of the package's CI runtime is sensible. Keep in mind, that you alsoneed to maintain these weights: you should adjust the weights from time to time,e.g. when adding new tests or build steps to a package, that increase it'soverall CI runtime.

Example: 9 packages are split into--split 3 partitions using thefollowing weights.

• --partition 0 (size = 2, weight = 10):a (9),h (1)
• --partition 1 (size = 3, weight = 11):b (5),d (5),i (1)
• --partition 2 (size = 4, weight = 11):c (5),e (2),f (2),g (2)

The following commands fromlerna are supported:

lerna run

Run an npm script in each package that contains that script.

For instance, this executes the last of four partitions. It also passes along--stream &--concurrency 1 to prefix log lines with the package name.

lerna-parallelism run \  --stream \  --concurrency 1 \  --split 4 \  --partition 3 \test

For some commands, likelerna bootstrap, splitting makes no sense. For someothers, it does, specifically:

• lerna changed
• lerna exec
• lerna list
• lerna publish

If you'd like to see support for these commands as well, feel free to submit apull request!

This project is licensed under theISC License.