How does this tag interact with other tags? Let's say I already have provisioners with separate tags to process only aws jobs in one pool and only gke in another. Or perhaps I have a provisioner deployed in each region.

Would I need to add a new provisioner key for each of these sets but with theis_prebuild flag included?

Multiplecoder_workspace_tags are cumulative:https://github.com/coder/coder/blob/main/coderd/dynamicparameters/tags_internal_test.go#L200-L242

Multiple instances of thecoder_workspace_tags data source cannot clobber existing tag values.

How does this tag interact with other tags?

It depends on the tags that the user has defined in their template. I didn’t go into much detail here because that could make it harder to understand. Basically, the provisioner key needs to include all the tags already specified in the template, plus this new one.

Would I need to add a new provisioner key for each of these sets but with the is_prebuild flag included?

In your example, if you want to update your provisioners deployed in AWS and GKE, you would need to create 2 new provisioner key with those same tags and the additionalis_prebuild tag.

For instance, in the dogfood template we already have two static tags:

data "coder_workspace_tags" "tags" {  tags = {    "cluster" = "dogfood-v2"    "env"     = "gke"  }}

For the provisioners to handle prebuild jobs for this template, the key must be created with:--tag cluster=dogfood-v2 --tag env=gke --tag is_prebuild=true

I get your point. If we're being overly verbose it will be hard to understand. On the other hand, I don't know that what you said above will be obvious to readers without additional context.

Don't feel pressured to change anything at my insistence, but perhaps consider whether there is a way to illustrate this interaction simply.

What are the failure scenarios for this approach?
If prebuild specific provisioners fail to deploy for a specific tagset, will those jobs remain pending? Will that result in any kind of back pressure where the pending jobs impact the reconciliation loop?

The jobs will remain in a pending state as there will be no provisioner to pick them up. My understanding is that the reconcilation loop should detect that there are in-progress jobs and not completely spam the queue (ref:https://github.com/coder/coder/blob/main/coderd/prebuilds/preset_snapshot.go#L394-L410) but do correct me if I'm not understanding correctly.

What@johnstcn said, the prebuild jobs would remain in a pending state, waiting for a provisioner daemon with matching tags to become available. The reconciliation loop already accounts for pending prebuilds (jobs in the queue) and subtracts them from the desired count, so this wouldn’t impact the loop itself.

The only side effect is that the queue for prebuild-related jobs could continue to grow if new template versions are imported or existing prebuilds are claimed.

Should I add another step here to validate the status of the provisioners from Coder’s perspective, for example, by checking the Provisioners page or running thecoder provisioner list CLI command?

I think that would be a good idea, but I'll leave the decision up to you.