Related to#15843
Related to#12331

If you have a job that is permanently stuck in queue due to e.g. incorrect provisioner tags, it will impact the "position in queue" for every workspace that enters the job queue.

This results in incorrect labels on workspaces saying "Position in queue; 21" when in reality they are position 1 in queue with the current set of available provisioners.

To fix this we could add a new case to the existing hangdetector code to cancel jobs that have been stuck pending for an hour or so. The queue position calculation logic should probably also be made smarter (see#15843).

https://codercom.slack.com/archives/C06T78M7X7H/p1738927399726959