Problem

Right now, you can easily overwhelm your available provisioners by creating a sufficient number of templates withprebuild_count > 0. This will cause the prebuilds reconciler to create a number of provisioner jobs to create or delete prebuilt workspaces. If you create enough templates and template versions, the churn can overwhelm your provisioners and cause users to need to wait for existing prebuild-related jobs to complete.

Proposed solutions

1. fix: prioritise human-initiated builds over prebuilds #18933 modifies theAcquireProvisionerJob query toalways prioritise human-created provisioner jobs. This will reduce the overall wait time for users at the expense of delaying prebuilds.

2. Restrict the maximum number of pending reconciliation actions that the reconciler can have ongoing at any given time. We could intelligently default this to a percentage of the active provisioners. It would slow down the overall provisioning time for prebuilds but ensure there is available capacity to immediately pick up human-initiated jobs. (Note: it would also make sense for the reconciler to prioritise delete actions over create actions here)

3. Allow separating prebuilds onto their own tagged provisioners. Right now, it is not possible to usedata.coder_workspace.me.is_prebuild as an input tocoder_workspace_tags, but our parsing logic for this could be updated to enable this use-case so that administrators could have a completely separate pool of provisioners dedicated to prebuilt workspaces.