This PR introduces an optimization to automatically cancel pending prebuild-related jobs from non-active template versions in the reconciliation loop.

Problem

Currently, when a template is configured with more prebuild instances than available provisioners, the provisioner queue can become flooded with pending prebuild jobs. This issue is worsened when provisioning/deprovisioning operations take a long time.

When the prebuild reconciliation loop generates jobs faster than provisioners can process them, pending jobs accumulate in the queue. Since prebuilt workspaces should always run the latest active template version, pending prebuild jobs from non-active versions become obsolete once a new version is promoted.

Solution

The reconciliation loop cancels pending prebuild-related jobs from non-active template versions that match the following criteria:

Build number: 1 (initial build created by the reconciliation loop)
Job status:pending
Not yet picked up by a provisioner (worker_id isNULL)
Owned by the prebuilds system user
Workspace transition:start

This prevents the queue from being cluttered with stale prebuild jobs that would provision workspaces on an outdated template version that would consequently need to be deprovisioned.

Changes

Added new SQL queryCountPendingNonActivePrebuilds to identify presets with pending jobs from non-active versions
Added new SQL queryUpdatePrebuildProvisionerJobWithCancel to cancel jobs for a specific preset
New reconciliation action typeActionTypeCancelPending handles the cancellation logic
Cancellation is non-blocking: failures to cancel prebuild jobs are logged as errors and don't prevent other reconciliation actions

Follow-up PR

Canceling pending prebuild jobs leaves workspaces in a Canceled state. While no Terraform resources need to be destroyed (since jobs were canceled before provisioning started), these database records should still be cleaned up. This will be addressed in a follow-up PR.

Closes:#20242

github-actionsbot assignedssncferreira

Oct 20, 2025

ssncferreira force-pushed thessncferreira/feat-cancel-pending-prebuilds branch 6 times, most recently fromf1a8e32 to9cfb23aCompare

October 21, 2025 11:58

ssncferreira added2 commits

October 21, 2025 17:28

feat: cancel pending prebuilds on template publish

ed0e3ab

test: add cancel pending prebuilds to template version active version

29cc88e

ssncferreira force-pushed thessncferreira/feat-cancel-pending-prebuilds branch from9cfb23a to29cc88eCompare

October 21, 2025 17:28

chore: minor fixes

51fa3ef

ssncferreira force-pushed thessncferreira/feat-cancel-pending-prebuilds branch from73cf135 to51fa3efCompare

October 21, 2025 18:04

ssncferreira commented

Oct 21, 2025

View reviewed changes

coderd/templateversions.go OutdatedShow resolvedHide resolved

ssncferreira commented

Oct 21, 2025

View reviewed changes

coderd/templateversions.go OutdatedShow resolvedHide resolved

ssncferreira requested review fromDanielleMaywood,SasSwart andjohnstcn

October 21, 2025 19:01

ssncferreira marked this pull request as ready for review

October 21, 2025 19:04

johnstcn reviewed

Oct 21, 2025

View reviewed changes

Copy link

Member

johnstcn left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

Mostly LGTM, but I'm not 100% convinced about the row locking?

coderd/database/queries/provisionerjobs.sql OutdatedShow resolvedHide resolved

coderd/database/queries/templates.sql OutdatedShow resolvedHide resolved

coderd/database/modelmethods.go OutdatedShow resolvedHide resolved

DanielleMaywood reviewed

Oct 22, 2025

View reviewed changes

coderd/database/dbauthz/dbauthz.go OutdatedShow resolvedHide resolved

coderd/database/dbauthz/dbauthz.go Outdated

		func (q*querier)UpdatePrebuildProvisionerJobWithCancel(ctx context.Context,arg database.UpdatePrebuildProvisionerJobWithCancelParams) ([]uuid.UUID,error) {
		// This is a system-only operation for canceling pending prebuild-related jobs
		// when a new template version is promoted.
		iferr:=q.authorizeContext(ctx,policy.ActionRead,rbac.ResourceSystem);err!=nil {

Copy link

Contributor

DanielleMaywoodOct 22, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

note(non-blocking): just want to noterbac.ResourceSystem usage here

Copy link

Member

johnstcnOct 23, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

If we givesubjectPrebuildsOrchestrator the ability to read/update provisioner jobs, we should be able to userbac.ResourceProvisionerJob here.

Copy link

ContributorAuthor

ssncferreiraOct 23, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

Yeah, permissions are a bit tricky here.

There is aResourceProvisionerJobs that we could use here, but that would mean that all users with this permissions would be abel to execute it.

The way it is done forUpdateProvisionerJobWithCancelByID, for instance, is to check theUpdate permissions on the workspace. Since it is a prebuild, we could also do a
if err := q.authorizeContext(ctx, policy.ActionUpdate, rbac.ResourcePrebuiltWorkspace); err != nil { Maybe this one makes the most sense 🤔

Copy link

Member

johnstcnOct 23, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

That's not a bad approximation actually 👍

Copy link

ContributorAuthor

ssncferreiraOct 23, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

Added therbac.ResourcePrebuiltWorkspace update check indcb6de1

coderd/database/queries/provisionerjobs.sql OutdatedShow resolvedHide resolved

coderd/templateversions.go OutdatedShow resolvedHide resolved

ssncferreira removed the request for review fromSasSwart

October 22, 2025 10:25

Copy link

ContributorAuthor

ssncferreira commentedOct 22, 2025

Note: Moving this PR back to draft.

After internal discussion, we've decided to move the logic to cancel pending prebuilds to the prebuilds reconciliation loop rather than handling it in thepatchActiveTemplateVersion endpoint.

This PR will be updated to reflect these changes:

Prebuild cancellation logic will be moved to the reconciliation loop
Implementation for deleting cancelled prebuilds will be handled in a separate PR for simplicity

The PR will be marked ready for review once the new implementation is in place.

ssncferreira marked this pull request as draft

October 22, 2025 10:58

ssncferreira added7 commits

October 22, 2025 11:27

chore: remove pending prebuild job cancellation from template promote…

4d677f0

… endpoint

fix: SQL query formatting

836ba0f

Merge remote-tracking branch 'origin/main' into ssncferreira/feat-can…

5ffe5b2

…cel-pending-prebuilds

chore: move pending prebuilds cancellation logic to reconciliation loop

fac079e

fix: minor fixes

2ce4b7f

fix: minor fixes

95825cb

Merge remote-tracking branch 'origin/main' into ssncferreira/feat-can…

095dd80

…cel-pending-prebuilds

ssncferreira added3 commits

October 23, 2025 10:15

test: add prebuilds tests

d4a7dba

Merge remote-tracking branch 'origin/main' into ssncferreira/feat-can…

dbb79f3

…cel-pending-prebuilds

chore: fix duplicate name test

8ea986e

ssncferreira marked this pull request as ready for review

October 23, 2025 10:28

chore: fix imports

0224665

ssncferreira requested review fromSasSwart andjohnstcn

October 23, 2025 10:37

johnstcn reviewed

Oct 23, 2025

View reviewed changes

coderd/database/dbauthz/dbauthz.go Outdated

		func (q*querier)UpdatePrebuildProvisionerJobWithCancel(ctx context.Context,arg database.UpdatePrebuildProvisionerJobWithCancelParams) ([]uuid.UUID,error) {
		// This is a system-only operation for canceling pending prebuild-related jobs
		// when a new template version is promoted.
		iferr:=q.authorizeContext(ctx,policy.ActionRead,rbac.ResourceSystem);err!=nil {

Copy link

Member

johnstcnOct 23, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

If we givesubjectPrebuildsOrchestrator the ability to read/update provisioner jobs, we should be able to userbac.ResourceProvisionerJob here.

coderd/database/queries/prebuilds.sql

Comment on lines +300 to +301

		-- Cancels all pending provisioner jobs for prebuilt workspaces on a specific preset from an
		-- inactive template version.

Copy link

Member

johnstcnOct 23, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

nit: This comment is slightly misleading, the query doesn't check that the preset relates to an inactive template version. It's currently the responsibility of the caller to verify that@preset_id is not related to the current active version ID.

Copy link

ContributorAuthor

ssncferreiraOct 23, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

That is a good catch! I've updated the query to filter out active template versions:0497f8b

coderd/prebuilds/preset_snapshot_test.goShow resolvedHide resolved

ssncferreira added3 commits

October 23, 2025 11:21

fix: UpdatePrebuildProvisionerJobWithCancel permissions

dcb6de1

fix: UpdatePrebuildProvisionerJobWithCancel to filter by non-active t…

0497f8b

…emplate version

Merge remote-tracking branch 'origin/main' into ssncferreira/feat-can…

b635573

…cel-pending-prebuilds

johnstcn approved these changes

Oct 23, 2025

View reviewed changes

coderd/database/queries/prebuilds.sqlShow resolvedHide resolved

enterprise/coderd/prebuilds/reconcile_test.go Outdated

		funcTestCancelPendingPrebuilds(t*testing.T) {
		t.Parallel()

		for_,tt:=range []struct {

Copy link

Member

johnstcnOct 23, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

nit, non-blocking: Since these tests all boil down to "only these jobs should be canceled and nothing else", I think we could collapse this to a single test case with all of the "should not cancel" data in one. The rationale for this is twofold:

It reduces the number of test databases we have to create
It gives us better assurance that only the jobs we care about are being canceled in the presence of more antagonist data.