Context

We want to place a tight SLO around coder workspace build times, so we can detect regression. However, buffy GPU instances often take a much longer time to start/stop than general instance, which frequently triggered our SLO alerts, even though it's only because of a few (expected) slow GPU builds. This is caused by the metrics we are usingcoderd_provisionerd_job_timing_seconds not having a dimension for template name (as we have a separate template for GPU and another for general instances).

Looking closer at the code, this metrics is also not the correct one to use either, because a Job can actually be many different things, not just a workspace build.

Intent

This PR introduces a new prometheus metrics forworkspace_build_timing_seconds, which specifically reports workspace build times. To reduce cardinality, this metrics excludesworkspace_name andworkspace_owner that are present on theworkspace_builds_total metrics.

This'd allow us to have different (and tight) SLOs for each of our template (GPU vs non-GPU) by filtering on thetemplate_name (optionallytemplate_version tag) as well as the workspace transition (as we noticedstop is often slower thanstart, but users don't care a lot aboutstop transitions).