This is a nice idea, I like the simplicity of prioritizing human-initiated jobs.
That said, I’m a bit concerned about the risk ofprebuild starvation under heavy load. If I understand the query correctly, we’re first ordering by whether the job is human-initiated (higher priority), and then bycreated_at within each group. That meansprebuild jobs will only be picked up once all human-initiated jobs have been processed.

Would it make sense to consider an ordering strategy that also takes into account how long a job has been waiting in the queue? I know we don’t have an explicitenqueued_at field, but maybecreated_at could serve that purpose. That way, we could still prioritize human jobs, but allow older prebuilds to eventually be picked up and get processed, avoiding starvation while maintaining the intended prioritization.

That means prebuild jobs will only be picked up once all human-initiated jobs have been processed.

Correct. The assumption here is that there will never be a constant stream of human-initiated jobs. If this is the case, then the deployment is undersized and requires more provisioners.

Would it make sense to consider an ordering strategy that also takes into account how long a job has been waiting in the queue?

I'm not sure the added complexity is worthwhile here. How likely is it for users to be creating a constant stream of provisioner jobs similar in volume to that of the prebuilds reconciler?

Also, we need to decide how long is 'too long' for a prebuild to wait? 1 minute? 10 minutes? Does it need to be configurable?

Finally, after the initial waiting period elapses, users will still end up in the situation where they will be stuck waiting on a bunch of provisioner jobs (as the prebuilds reconciler creates provisioner jobs in batches).

My take is thatif you are running into this problem in the first place, you need more provisioners.

I do like the suggestion of leveragingcoder_workspace_tags to conditionally send prebuilds to a separate set of provisioners, but it appears thattfparse is not able to evaluate that right now.

I had the same starvation concern in the original PR, but your point convinced me@johnstcn.

Do we need to update that query that tells us the queue position too?

oof maybe yea Done