If a quota is exceeded, the prebuilt workspace will fail provisioning the same way other workspaces do.

It is possible for prebuilt workspaces to overwhelm a Coder deployment to the point that users and template administrators experience significant delays when they attempt to create new workspaces or manage their templates. This may happen organically, if there are not enough provisioners to meet the needs of the Coder deployment. It may also happen if a broken template is uploaded, which mistakenly requests too many prebuilt workspaces. In the latter case, it may be hard to correct the situation because a fixed template version cannot be uploaded while the Coder deployment's provisioners are overloaded.

If your Coder deployment is exhibiting the above symptoms, follow these instructions to verify and then rectify the situation:

First, run:

coder prebuilds pause

This prevents further pollution of your provisioner queues. Specifically, it will prevent the prebuilt workspaces feature from scheduling new prebuilt workspace creation jobs. Jobs that have already been enqueued will still be processed at this point. Regardless of the steps taken in the rest of your troubleshooting process, remember to run`coder prebuilds resume` once all impact has been mitigated. See the last step in this section for more information.

Next, run:

coder provisionerjobs list --status=pending --initiator=prebuilds

This will show a list of all pending jobs that have been enqueued by the prebuilt workspace system. The length of this list is an indicator of whether prebuilt workspaces have overwhelmed your Coder deployment. Human initiated jobs do have priority over pending prebuild jobs, but a prebuild job that is already running will not be preempted to process a human initiated job. An extended list of pending prebuild jobs therefore increases the likelihood that all provisioners may already be occopied when a human user would like to create a workspace. This then transitively increases the likelihood that users will experience delays waiting for the next available provisioner to become available when they try to manage their workspaces or templates.

To ensure that the next available provisioner will be given to a human initiated job, run:

coder provisionerjobs list --status=pending --initiator=prebuilds| jq'...'| xargs -n1 -P2 -I{} coder provisionerjobs cancel {}

This will clear the provisioner queue of all jobs that were not initiated by a human being, which increases the probability that a provisioner will be available when the next human operator needs it. It does not cancel running provisioner jobs, so there may still be some delay in processing new provisioner jobs at least until a provisioner completes its current job.

At this stage, most prebuild related impact will have been mitigated. There may still be a bugged template version, but it will no longer pollute provisioner queues with prebuilt workspace jobs. If the latest version of a template is also broken for reasons unrelated to prebuilds, then users are able to create workspaces using a previous template version. Some running jobs may have been initiated by the prebuild system, but these cannot be cancelled without potentially orphaning resources that have already been deployed by Terraform. Depending on your deployment and template provisioning times, it might be best to upload a new template version and wait for it to be processed organically.

If you need to expedite the processing of human-related jobs at the cost of some infrastructure house-keeping, then you can run:

coder provisionerjobs list --status=running --initiator=prebuilds| jq'...'| xargs -n1 -P2 -I{} coder provisionerjobs cancel {}

This will cancel running prebuild jobs (orphaning any resources that have already been deployed) and immediately make room for human-initiated jobs.

Once the provisioner queue has been cleared and all templates have been fixed, remember to resume prebuild reconciliation by running:

coder prebuilds resume