It is possible for prebuilt workspaces to overwhelm a Coder deployment to the point that users and template administrators experience significant delays when they attempt to create new workspaces or manage their templates. This may happen organically, if there are not enough provisioners to meet the needs of the Coder deployment. It may also happen if a broken template is uploaded, which mistakenly requests too many prebuilt workspaces. In the latter case, it may be hard to correct the situation because a fixed template version cannot be uploaded while the Coder deployment's provisioners are overloaded.

Prebuilt workspaces can overwhelm a Coder deployment, causing significant delays when users and template administrators attempt to create new workspaces or manage their templates. This can happen in two scenarios:

1.**Organic overload**: Not enough provisioners to meet the deployment's needs

2.**Broken template**: A template that mistakenly requests too many prebuilt workspaces

In the second case, it can be difficult to fix the situation because you cannot upload a corrected template version while the provisioners are overloaded.

The troubleshooting steps below will help you resolve this situation:

* Pause prebuilt workspace reconciliation to stop the problem from getting worse

* Check how many prebuild jobs are clogging your provisioner queue

* Cancel excess prebuild jobs to free up provisioners for human users

* Fix any problematic templates that are causing the issue

* Resume prebuilt reconciliation once everything is back to normal

If your Coder deployment is exhibiting the above symptoms, follow these instructions to verify and then rectify the situation:

coder prebuilds pause

This prevents further pollution of your provisioner queues. Specifically, it will prevent the prebuilt workspaces feature from scheduling new prebuilt workspace creation jobs. Jobs that have already been enqueued will still be processed at this point. Regardless of the steps taken in the rest of your troubleshooting process, remember to run`coder prebuilds resume` once all impact has been mitigated. See the last step in this section for more information.

This prevents further pollution of your provisioner queues by stopping the prebuilt workspaces feature from scheduling new creation jobs. Jobs that have already been enqueued will still be processed.

**Important**: Remember to run`coder prebuilds resume` once all impact has been mitigated (see the last step in this section).

Next, run:

coder provisionerjobs list --status=pending --initiator=prebuilds

This will show a list of all pending jobs that have been enqueued by the prebuilt workspace system. The length of this list is an indicator of whether prebuilt workspaces have overwhelmed your Coder deployment. Human initiated jobs do have priority over pending prebuild jobs, but a prebuild job that is already running will not be preempted to process a human initiated job. An extended list of pending prebuild jobs therefore increases the likelihood that all provisioners may already be occopied when a human user would like to create a workspace. This then transitively increases the likelihood that users will experience delays waiting for the next available provisioner to become available when they try to manage their workspaces or templates.

This will show a list of all pending jobs that have been enqueued by the prebuilt workspace system. The length of this list indicates whether prebuilt workspaces have overwhelmed your Coder deployment.

Human-initiated jobs have priority over pending prebuild jobs, but running prebuild jobs cannot be preempted. A long list of pending prebuild jobs increases the likelihood that all provisioners are already occupied when a user wants to create a workspace. This increases the likelihood that users will experience delays waiting for the next available provisioner.

To ensure that the next available provisioner will be given to a humaninitiated job, run:

To ensure that the next available provisioner will be given to a human-initiated job, run:

coder provisionerjobs list --status=pending --initiator=prebuilds| jq'...'| xargs -n1 -P2 -I{} coder provisionerjobs cancel {}

coder provisionerjobs list --status=pending --initiator=prebuilds| jq-r'.[].id'| xargs -n1 -P2 -I{} coder provisionerjobs cancel {}

This will clear the provisioner queue of all jobs that were not initiated by a human being, which increases the probability that a provisioner will be available when the next human operator needs it. It does not cancel running provisioner jobs, so there may still be some delay in processing new provisioner jobsat leastuntil a provisioner completes its current job.

This will clear the provisioner queue of all jobs that were not initiated by a human being, which increases the probability that a provisioner will be available when the next human operator needs it. It does not cancel running provisioner jobs, so there may still be some delay in processing new provisioner jobs until a provisioner completes its current job.

At this stage, most prebuild related impact will have been mitigated. There may still be a bugged template version, but it will no longer pollute provisioner queues with prebuilt workspace jobs. If the latest version of a template is also broken for reasons unrelated to prebuilds, then users are able to create workspaces using a previous template version. Some running jobs may have been initiated by the prebuild system, but these cannot be cancelled without potentially orphaning resources that have already been deployed by Terraform. Depending on your deployment and template provisioning times, it might be best to upload a new template version and wait for it to be processed organically.

If you need to expedite the processing of human-related jobs at the cost of some infrastructurehouse-keeping, then you can run:

If you need to expedite the processing of human-related jobs at the cost of some infrastructurehousekeeping, you can run:

coder provisionerjobs list --status=running --initiator=prebuilds| jq'...'| xargs -n1 -P2 -I{} coder provisionerjobs cancel {}

coder provisionerjobs list --status=running --initiator=prebuilds| jq-r'.[].id'| xargs -n1 -P2 -I{} coder provisionerjobs cancel {}

This will cancel running prebuild jobs (orphaning any resources that have already been deployed) and immediately make room for human-initiated jobs.

Once the provisioner queue has been cleared and all templates have been fixed,remember toresume prebuild reconciliation by running:

Once the provisioner queue has been cleared and all templates have been fixed, resume prebuild reconciliation by running:

coder prebuilds resume

This re-enables the prebuilt workspaces feature and allows the reconciliation loop to resume normal operation. The system will begin creating new prebuilt workspaces according to your template configurations.