Oct 9, 2025 · Oct 9, 2025 · Oct 9, 2025 · Oct 9, 2025 · Oct 9, 2025 · Oct 9, 2025
diff --git a/docs/admin/templates/extending-templates/prebuilt-workspaces.md b/docs/admin/templates/extending-templates/prebuilt-workspaces.md

 If a quota is exceeded, the prebuilt workspace will fail provisioning the same way other workspaces do.

 ### Managing prebuild provisioning queues

 Prebuilt workspaces can overwhelm a Coder deployment, causing significant delays when users and template administrators attempt to create new workspaces or manage their templates. This can happen in two scenarios:

 1. **Organic overload**: Not enough provisioners to meet the deployment's needs
 2. **Broken template**: A template that mistakenly requests too many prebuilt workspaces

 In the second case, it can be difficult to fix the situation because you cannot upload a corrected template version while the provisioners are overloaded.

 The troubleshooting steps below will help you resolve this situation:

 - Pause prebuilt workspace reconciliation to stop the problem from getting worse
 - Check how many prebuild jobs are clogging your provisioner queue
 - Cancel excess prebuild jobs to free up provisioners for human users
 - Fix any problematic templates that are causing the issue
 - Resume prebuilt reconciliation once everything is back to normal

 If your Coder deployment is exhibiting the above symptoms, follow these instructions to verify and then rectify the situation:

 First, run:

 ```bash
 coder prebuilds pause
 ```

 This prevents further pollution of your provisioner queues by stopping the prebuilt workspaces feature from scheduling new creation jobs. Jobs that have already been enqueued will still be processed.

 **Important**: Remember to run `coder prebuilds resume` once all impact has been mitigated (see the last step in this section).

 Next, run:

 ```bash
 coder provisioner jobs list --status=pending --initiator=prebuilds
 ```

 This will show a list of all pending jobs that have been enqueued by the prebuilt workspace system. The length of this list indicates whether prebuilt workspaces have overwhelmed your Coder deployment.

 Human-initiated jobs have priority over pending prebuild jobs, but running prebuild jobs cannot be preempted. A long list of pending prebuild jobs increases the likelihood that all provisioners are already occupied when a user wants to create a workspace. This increases the likelihood that users will experience delays waiting for the next available provisioner.

 To ensure that the next available provisioner will be given to a human-initiated job, run:

 ```bash
 coder provisioner jobs list --status=pending --initiator=prebuilds | jq -r '.[].id' | xargs -n1 -P2 -I{} coder provisioner jobs cancel {}
 ```

 This will clear the provisioner queue of all jobs that were not initiated by a human being, which increases the probability that a provisioner will be available when the next human operator needs it. It does not cancel running provisioner jobs, so there may still be some delay in processing new provisioner jobs until a provisioner completes its current job.

 At this stage, most prebuild related impact will have been mitigated. There may still be a bugged template version, but it will no longer pollute provisioner queues with prebuilt workspace jobs. If the latest version of a template is also broken for reasons unrelated to prebuilds, then users are able to create workspaces using a previous template version. Some running jobs may have been initiated by the prebuild system, but these cannot be cancelled without potentially orphaning resources that have already been deployed by Terraform. Depending on your deployment and template provisioning times, it might be best to upload a new template version and wait for it to be processed organically.

 If you need to expedite the processing of human-related jobs at the cost of some infrastructure housekeeping, you can run:

 ```bash
 coder provisioner jobs list --status=running --initiator=prebuilds | jq -r '.[].id' | xargs -n1 -P2 -I{} coder provisioner jobs cancel {}
 ```

 This will cancel running prebuild jobs (orphaning any resources that have already been deployed) and immediately make room for human-initiated jobs.

 Once the provisioner queue has been cleared and all templates have been fixed, resume prebuild reconciliation by running:

 ```bash
 coder prebuilds resume
 ```

 This re-enables the prebuilt workspaces feature and allows the reconciliation loop to resume normal operation. The system will begin creating new prebuilt workspaces according to your template configurations.

 ### Template configuration best practices

 #### Preventing resource replacement