mentioning#46 for visibility

the current version works as expected, the only issue is the log flickering with multiple invocations of nix writing to stdout in parallel. I was considering utilising the raw-internal-json output similarly to how nix-output-monitor does it but that might be out of scope here 🤔

activation is fully synchronous but that is usually the part that takes the least amount of time

The new approach seems good 👍

there are a still lot of unwraps that need to be handled better, this pr is still far from finished but progress has been made:

I added some progress indicators using the indicatif crate for each concurrent build 🥳

This is what they look like (image the spinner spinning and the text changing as nix builds stuff:

For remote and builds each host gets its own progress spinner that is reused for all of that hosts profiles since they are built in order anyways. This could be adjusted for multi-profile hosts since indicatif also supports nested tree-like progress bars.

I changed some data structures around to make working with them in async contexts easier.
The nested references are somewhat ok until you have to make them work in a async context where you need send objects. deep cloning by hand really isn't a great solution and it really doesnt make any difference performance-wise to add some clones here and there.
Nobody is going to deploy to >10k hosts and if they would, cloning the deploy data a few times would be the least of their problems.
This also removes a bunch of generic lifetime specifiers which I am not sad about.

With these two last commits I would declare this PR somewhat usable 🥳

If anyone wants to try it out with their own setups I would be very happy to hear how it works for you, I am always open for feedback of any kind

Hi all,
I tried this out, with my setup in ManoftheSea/SeaofDirac/trunk, on targets [littlecreek, crunchbits, technetium]. I saw that it ran the checks for all the systems, which meant building them locally, then proceeded with pushing profiles to the systems in parallel (which resulted in building them remotely, I believe). After all three were ready, they deployed in sequence, and as it was just an update, they all completed happily.

This works well for these three systems, which are a pair of VPSes and a local (to me) server; were I trying to deploy all the systems in my home network, I would prefer instead that my development machine (personal laptop) be able to push the profiles to a single server (technetium), so that it can then act as a substituter for e.g. aluminium and nickel. Maybe provide a "build-host" attribute per-host, with a default of the hostself? e.g. deploy.nickel.buildHost = "technetium"; deploy.littlecreek.buildHost = "littlecreek";

Hi there, our team atPyro utilized this PR and observed a massive speedup deploying to a our node cluster and for our initial deployments we see a huge speed increase of at minimum 2x, however there might be a problem.

I'm deploying from Ubuntu WSL, so that might be the culprit but it seems after every deployment in the same terminal session only half will be working in parallel, and then half again, until its back to single threaded (even though the spinners are still there). It might not be halfing but there is at minimum diminishing returns over multiple command invocations.

Here's a successful one if you're curious, I haven't been able to record the above yet though.
I figured with such a big cluster our network would likely be prime testing ground for this PR.

I'm really excited for this to get merged and I'll test anything necessary to speed things along.

https://asciinema.org/a/dirahL9YYWFaT0lORkU98nX8s

P.s. A huge potential speedup would be processing checks in a parallel fashion as well, as that takes up a huge chunk of time, though I'm not sure how viable this is.

Hi there, our team atPyro utilized this PR and observed a massive speedup deploying to a our node cluster and for our initial deployments we see a huge speed increase of at minimum 2x

oh that is great to hear! thanks for the feedback.

it seems after every deployment in the same terminal session only half will be working in parallel, and then half again, until its back to single threaded (even though the spinners are still there). It might not be halfing but there is at minimum diminishing returns over multiple command invocations.

hmm, I'll have to investigate how that might happen. I'll add some more debug output for each thread so it'll easier to debug on your side. This pr pretty much just spawns a bunch of threads using high-level abstractions so I'm curious how that might happen 🤔

P.s. A huge potential speedup would be processing checks in a parallel fashion as well, as that takes up a huge chunk of time, though I'm not sure how viable this is.

unfortunately, the checks are limited by nix's eval speed. there is some work going on on detsys's side (see e.g.https://determinate.systems/posts/parallel-nix-eval/) but afaiu nix flake checks are currently single-threaded in nature.

you could attempt to get around that by checking/building all systems in parallel batches usinggnu parallel although I am not sure how feasible that is. I do not really have access to hardware that could build even two systems in parallel...

just a small update for everybody interested in this: I am still working on it, didnt have all too much time recently. This weekend i've got some more time so I'll address all of rvem's points above 🫡

Thank you for the update! Our team was considering taking over this PR and aiding its development, but its good to know there's still activity. We use this branch religiously and are free to test any new functionality / bug fixes.

We use this branch religiously and are free to test any new functionality / bug fixes.

that is a huge number of hosts, thanks for a large scale test on this!

did you encounter the issue with the deployments not running in parallel again? I could not find anything in the code that could lead to such an issue

This looks really good ! Any further help testing needed to land this ?