Currently, the outcome of a prebuild reconciliation loop is a set of provisioner jobs, either for creating or deleting prebuilt workspaces. When the number of prebuild-related jobs exceeds the number of available provisioner daemons, the provisioner queue can become overloaded, resulting in delayed human-initiated jobs.

With PR#18933, human-initiated jobs are prioritized over system jobs (i.e., prebuild-related jobs). Since job preemption is not supported, this means that even though human-initiated jobs have higher priority in the queue, they must still wait for a provisioner daemon to become available. If all daemons are busy processing prebuild-related jobs, human-initiated jobs will experience delays.

This issue becomes more significant when prebuild-related jobs are long-running. Additionally, because the prebuild reconciliation loop handles tuples such as(template, version, preset), new template versions can trigger additional prebuild jobs, further increasing the queue load (e.g., prebuilds for both version A and version B).

Proposed Solution

Issue#18972 (and sub-issue#19937) proposes introducing a limit on the maximum number of pending reconciliation actions that the reconciler can have ongoing at any given time. The prebuild reconciliation loop is asystem-wide process that handles all presets present in the system. In a simple Coder deployment, for example, a single organization with no provisioner tags, this approach effectively mitigates the problem. Since there is only one queue for handling provisioner jobs, limiting the number of pending actions to a value lower than the number of available provisioner daemons ensures that some daemons remain available for human-initiated jobs.
The main drawback of this approach is that prebuild status may take longer to align with the template definitions, as completing all prebuilds will likely require multiple reconciliation loop iterations.

However, Coder supports multiple organizations and multiple provisioner tag sets within an organization. This means that in more complex deployments, there can bemultiple provisioner queues, one per organization, and within each organization, one per tag set.

For example, in the following deployment:

orgA has two tag sets:dev andprod
orgB has one tag set:dev

This results in three provisioner queues:

(orgA, dev)
(orgA, prod)
(orgB, dev)

With each queue having its own set of provisioner daemons. In such scenarios, a system-wide limit on prebuild actions might still overload one of these queues if there is an imbalance in the number of daemons per queue.
Ideally, the system-wide limit should be defined based on thesmallest provisioner daemon set in the system. This ensures that even the least-resourced queue can maintain available daemons for human-initiated jobs. However, this approach can also limit throughput in other queues with more provisioner daemons, potentially leading to underutilization of resources.

In conclusion, we can consider three possible scopes for throttling the number of pending prebuild actions:

System-wide
Organization-wide
Tag-wide

For simplicity, the POC presented in this PR implements the system-wide throttling approach.

Atag-wide throttling model would likely address the issue more precisely, since it would cap the number of in-progress prebuild jobs per provisioner queue (i.e., per organization and tag set). This approach isolates smaller daemon pools and preserves headroom where it matters most.
However, it also introduces significant complexity and raises architectural questions, for example, whether the prebuild reconciliation loop should be aware of queue topology and per-queue capacity. This adds overhead and tighter coupling between components, increasing maintenance costs and blurring the separation of responsibilities.

Anorganization-wide solution could represent a practical middle ground. It would reduce the burden caused by prebuild jobs and could be built on top of the system-wide approach proposed in this POC. However, it would not fully guarantee that the issue is avoided if the configured limit does not account for the capacity of the least-resourced daemon set within the organization.

Implementation changes

The prebuild reconciliation loop handles presets as tuples in the form of(template, version, preset).
For example, if there is a templateT1 with versionV1 that has one presetP1, and a versionV2 that has two presets (P1 andP2), the reconciliation loop will handle the following presets:

(T1, V1, P1)
(T1, V2, P1)
(T1, V2, P2)

Each of these tuples is considered a preset in the following sections.

Current implementation

In the current implementation, each preset is handled by a dedicated goroutine.
Each goroutine:

Retrieves the state of the preset’s prebuild instances.
Calculates the actions required for the deployed state to match the template definition.
Executes those actions by creating prebuild-related provisioner jobs (for creating or deleting prebuilt workspaces).

goroutine:  > ReconcilePreset()    preset.CalculateState()    preset.CalculateActions()    for each action:      > executeReconciliationAction()        > Write Transaction:           Builder.build() // DB.InsertProvisionerJob() and DB.InsertWorkspaceBuild()           PublishProvisionerJobEvent()

Once the jobs are created, the provisionerd server takes over and processes them from the queue.
The goroutine only ensures that the job creation is scheduled, it does not wait for the prebuilds to complete.

POC implementation

(Note: The solution proposed in this PR is a POC implementation, meaning that the implementation might not be the best and can definitely be improved. The goal is to validate the design and assess the level of refactoring required, rather than to deliver a production-ready solution. There are several TODOs in the code with ideas for improvements and possible alternatives)

To minimize refactoring, the POC retains the overall structure of the current implementation but introduces a new component: a prebuild scheduler. The scheduler is responsible for scheduling provisioner jobs and limiting the number of jobs created during each reconciliation loop.

Under this new design:

The preset goroutines still compute the state and determine the full list of required actions (to match the template definition).
Instead of scheduling all actions directly, they pass them to the scheduler.
The scheduler enforces a limit on the number of actions that can be scheduled per loop, based on a configurable value (currently defined as a constant inscheduler.go for simplicity) and the number of currently pending prebuild jobs.

goroutine:  > ReconcilePreset()    preset.CalculateState()    preset.CalculateActions()scheduler (after all goroutines finish):  for each action:    > executeReconciliationAction()      > Write Transaction:        Builder.build() // DB.InsertProvisionerJob() and DB.InsertWorkspaceBuild()        PublishProvisionerJobEvent()

Currently, the scheduler executes actions sequentially, but scheduling could likely be performed concurrently in the future to improve throughput once the throttling mechanism and concurrency boundaries are better defined.

Future Improvements

These changes focus on the producer side, specifically, the prebuild reconciliation loop that creates jobs.
A complementary improvement would target the consumer side: when a provisioner daemon requests a job, if the next job in the queue is a system job (i.e., prebuild-related), provisionerd could assess the current daemon capacity and, if capacity is constrained or at risk, choose not to assign that job.
This would introduce consumer-side backpressure, helping to prevent overload and smooth overall throughput across the system.