NotificationsYou must be signed in to change notification settings
Fork907
Star10k

feat: fetch prebuilds metrics state in background#17792

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to ourterms of service andprivacy statement. We’ll occasionally send you account related emails.

Already on GitHub?Sign in to your account

Jump to bottom

Merged

dannykopping merged 4 commits intomainfromdk/debounce-metrics-collection

May 13, 2025

Merged

feat: fetch prebuilds metrics state in background#17792

dannykopping merged 4 commits intomainfromdk/debounce-metrics-collection

May 13, 2025

+109 −24

Conversation

Copy link

Contributor

dannykopping commentedMay 13, 2025•
edited
Loading

Collect() is called whenever the/metrics endpoint is hit to retrieve metrics.

The queries used in prebuilds metrics collection are quite heavy, and we want to avoid having them running concurrently / too often to keep db load down.

Here I'm moving towards a background retrieval of the state required to set the metrics, which gets invalidated every interval.

Also introducescoderd_prebuilt_workspaces_metrics_last_updated which operators can use to determine when these metrics go stale.

See#17789 as well.

feat: fetch prebuilds metrics state in background

562b56d

Signed-off-by: Danny Kopping <dannykopping@gmail.com>

github-actionsbot assigneddannykopping

May 13, 2025

dannykopping force-pushed thedk/debounce-metrics-collection branch from5da546e toe73dae6Compare

May 13, 2025 12:29

chore: improvements

fcbfb7f

Signed-off-by: Danny Kopping <dannykopping@gmail.com>

dannykopping force-pushed thedk/debounce-metrics-collection branch frome73dae6 tofcbfb7fCompare

May 13, 2025 12:29

dannykopping marked this pull request as ready for review

May 13, 2025 12:35

dannykopping requested review fromjohnstcn anddeansheather

May 13, 2025 12:35

deansheather approved these changes

May 13, 2025

View reviewed changes

enterprise/coderd/prebuilds/metricscollector.goShow resolvedHide resolved

enterprise/coderd/prebuilds/metricscollector.go OutdatedShow resolvedHide resolved

enterprise/coderd/prebuilds/metricscollector.goShow resolvedHide resolved

enterprise/coderd/prebuilds/metricscollector.go

		@@ -55,20 +57,34 @@ var (
		labels,
		nil,
		)
		lastUpdateDesc = prometheus.NewDesc(
		"coderd_prebuilt_workspaces_metrics_last_updated",
		"The unix timestamp when the metrics related to prebuilt workspaces were last updated; these metrics are cached.",

Copy link

Member

deansheatherMay 13, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

Is unix timestamp easy to alert on? Like can you do something likeunix_now() - metric_value > 1000 or something in grafana and co? If not, it might be better if this was a durationsince the last successful fetch instead.

Copy link

Member

johnstcnMay 13, 2025•
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

+1 from me for duration since last successful fetch

Copy link

ContributorAuthor

dannykoppingMay 13, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

The idiomatic approach is to use unix timestamps, seeprometheus_config_last_reload_success_timestamp_seconds.

Copy link

Member

deansheatherMay 13, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

So I guess we have an existing metric for the coder server start timestamp?

Copy link

ContributorAuthor

dannykoppingMay 13, 2025•
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

I don't think so (or at least not one we export), but I think as long as this metric is updated relative to itselfandup is taken into consideration, it should be useful.

enterprise/coderd/prebuilds/metricscollector.go OutdatedShow resolvedHide resolved

enterprise/coderd/prebuilds/reconcile.goShow resolvedHide resolved

johnstcn reviewed

May 13, 2025

View reviewed changes

enterprise/coderd/prebuilds/metricscollector.go OutdatedShow resolvedHide resolved

enterprise/coderd/prebuilds/reconcile.go OutdatedShow resolvedHide resolved

dannykopping mentioned this pull request

May 13, 2025

chore: optimize workspace_latest_builds view query#17789

Merged

dannykopping added2 commits

May 13, 2025 16:43

Merge branch 'main' of github.com:/coder/coder into dk/debounce-metri…

e3591c2

…cs-collection

chore: review feedback

35cebf6

Signed-off-by: Danny Kopping <dannykopping@gmail.com>

dannykopping merged commitb2a1de9 intomain

May 13, 2025

34 checks passed

dannykopping deleted the dk/debounce-metrics-collection branch

May 13, 2025 18:27

github-actionsbot locked and limited conversation to collaborators

May 13, 2025

Labels

None yet

3 participants

Movatterモバイル変換

Navigation Menu

Search code, repositories, users, issues, pull requests...

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat: fetch prebuilds metrics state in background#17792

feat: fetch prebuilds metrics state in background#17792

Conversation

dannykopping commentedMay 13, 2025•
edited
Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

deansheatherMay 13, 2025

Choose a reason for hiding this comment

Uh oh!

johnstcnMay 13, 2025•
edited
Loading

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dannykoppingMay 13, 2025

Choose a reason for hiding this comment

Uh oh!

deansheatherMay 13, 2025

Choose a reason for hiding this comment

Uh oh!

dannykoppingMay 13, 2025•
edited
Loading

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Movatterモバイル変換

feat: fetch prebuilds metrics state in background#17792

feat: fetch prebuilds metrics state in background#17792

Conversation

dannykopping commentedMay 13, 2025• editedLoading Uh oh!There was an error while loading.Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

deansheatherMay 13, 2025

Choose a reason for hiding this comment

Uh oh!

johnstcnMay 13, 2025• editedLoading Uh oh!There was an error while loading.Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dannykoppingMay 13, 2025

Choose a reason for hiding this comment

Uh oh!

deansheatherMay 13, 2025

Choose a reason for hiding this comment

Uh oh!

dannykoppingMay 13, 2025• editedLoading Uh oh!There was an error while loading.Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

dannykopping commentedMay 13, 2025•
edited
Loading

johnstcnMay 13, 2025•
edited
Loading

dannykoppingMay 13, 2025•
edited
Loading