Problem

Currently, the outcome of a prebuild reconciliation loop is a set of provisioner jobs, either for creating or deleting prebuilt workspaces. When the number of prebuild-related jobs exceeds the number of available provisioner daemons, the provisioner queue can become overloaded, resulting in delayed human-initiated jobs.

With PR#18933, human-initiated jobs are prioritized over system jobs (i.e., prebuild-related jobs). Since job preemption is not supported, this means that even though human-initiated jobs have higher priority in the queue, they must still wait for a provisioner daemon to become available. If all daemons are busy processing prebuild-related jobs, human-initiated jobs will experience delays.

This issue becomes more significant when prebuild-related jobs are long-running. Additionally, because the prebuild reconciliation loop handles tuples such as(template, version, preset), new template versions can trigger additional prebuild jobs, further increasing the queue load (e.g., prebuilds for both version A and version B).