NotificationsYou must be signed in to change notification settings
Fork1.1k
Star11.4k

WIP perf: improve performance of metricsAggregator path by reducing memory allocations#20724

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to ourterms of service andprivacy statement. We’ll occasionally send you account related emails.

Already on GitHub?Sign in to your account

Jump to bottom

Open

cstyan wants to merge2 commits intomain

base:main

Choose a base branch

fromcallum/asPrometheus-perf

Open

WIP perf: improve performance of metricsAggregator path by reducing memory allocations#20724

cstyan wants to merge2 commits intomainfromcallum/asPrometheus-perf

+133 −37

Conversation

Copy link

Contributor

cstyan commentedNov 11, 2025

NOTE: this is still WIP as it introduces in-memory caching of some data, and there is currently no cleanup of that cache. To be decided; do we want to keep the cache or not based on benchmark results (see below), and if so do we want to introduce LRU caching or TTL caching.

Again, poking around in our profiling data this seemed like a quick win. This will be more impactful in larger organizations/with larger #'s of agents per workspace.

TheMetricsAggregatorRun function and theasPrometheus function, which previously was attached to theannotatedMetric type, are the source of ~11% of our memory allocations and ~7.5% of total memory allocated (24h period). This path itself is only ~1.6% of our CPU time but it should also have a knock on affect on garbage collection which is about ~7.6% of CPU time.

The optimizations are relatively straightforward:

preallocate the string slice for the final label set with the correct length based on the base labels and extra labels lengths
moveasPrometheus from theannotatedMetric toMetricsAggregator so we can cache metrics description structs for metrics (should be a big win, agents likely push the same metrics repeatedly over time)
use a stack allocated slice in asPrometheus of a 'reasonable size' (16 in this case) to avoid heap allocation (most of the time) for smaller label sets (in terms of # of pairs)

Benchmark Results:

goos: linuxgoarch: amd64pkg: github.com/coder/coder/v2/coderd/prometheusmetricscpu: AMD EPYC 9254 24-Core Processor                                               │ /workspace/bench/baseline.txt │  /workspace/bench/opt_stackbuf.txt  │  /workspace/bench/opt_nocache.txt   │                               │            sec/op             │    sec/op     vs base               │    sec/op     vs base               │_asPrometheus/base4_extra0-48                     2.128µ ± ∞ ¹   1.384µ ± ∞ ¹  -34.96% (p=0.008 n=5)   2.660µ ± ∞ ¹  +25.00% (p=0.016 n=5)_asPrometheus/base4_extra2-48                     3.246µ ± ∞ ¹   1.842µ ± ∞ ¹  -43.25% (p=0.008 n=5)   3.492µ ± ∞ ¹        ~ (p=0.151 n=5)_asPrometheus/base4_extra5-48                     5.423µ ± ∞ ¹   2.596µ ± ∞ ¹  -52.13% (p=0.008 n=5)   5.560µ ± ∞ ¹        ~ (p=0.421 n=5)_asPrometheus/base4_extra10-48                    7.168µ ± ∞ ¹   3.747µ ± ∞ ¹  -47.73% (p=0.008 n=5)   8.239µ ± ∞ ¹  +14.94% (p=0.008 n=5)_asPrometheus/base2_extra5-48                     3.636µ ± ∞ ¹   2.073µ ± ∞ ¹  -42.99% (p=0.008 n=5)   3.766µ ± ∞ ¹        ~ (p=0.310 n=5)geomean                                           3.962µ         2.199µ        -44.50%                 4.375µ        +10.42%¹ need >= 6 samples for confidence interval at level 0.95                               │ /workspace/bench/baseline.txt │  /workspace/bench/opt_stackbuf.txt   │   /workspace/bench/opt_nocache.txt   │                               │             B/op              │     B/op       vs base               │     B/op       vs base               │_asPrometheus/base4_extra0-48                     1184.0 ± ∞ ¹     816.0 ± ∞ ¹  -31.08% (p=0.008 n=5)    1008.0 ± ∞ ¹  -14.86% (p=0.008 n=5)_asPrometheus/base4_extra2-48                     1496.0 ± ∞ ¹     912.0 ± ∞ ¹  -39.04% (p=0.008 n=5)    1288.0 ± ∞ ¹  -13.90% (p=0.008 n=5)_asPrometheus/base4_extra5-48                    2.375Ki ± ∞ ¹   1.219Ki ± ∞ ¹  -48.68% (p=0.008 n=5)   2.125Ki ± ∞ ¹  -10.53% (p=0.008 n=5)_asPrometheus/base4_extra10-48                   3.125Ki ± ∞ ¹   1.766Ki ± ∞ ¹  -43.50% (p=0.008 n=5)   2.797Ki ± ∞ ¹  -10.50% (p=0.008 n=5)_asPrometheus/base2_extra5-48                    1.539Ki ± ∞ ¹   1.000Ki ± ∞ ¹  -35.03% (p=0.008 n=5)   1.383Ki ± ∞ ¹  -10.15% (p=0.008 n=5)geomean                                          1.808Ki         1.088Ki        -39.79%                 1.590Ki        -12.01%¹ need >= 6 samples for confidence interval at level 0.95                               │ /workspace/bench/baseline.txt │ /workspace/bench/opt_stackbuf.txt  │  /workspace/bench/opt_nocache.txt  │                               │           allocs/op           │  allocs/op   vs base               │  allocs/op   vs base               │_asPrometheus/base4_extra0-48                      33.00 ± ∞ ¹   20.00 ± ∞ ¹  -39.39% (p=0.008 n=5)   29.00 ± ∞ ¹  -12.12% (p=0.008 n=5)_asPrometheus/base4_extra2-48                      41.00 ± ∞ ¹   25.00 ± ∞ ¹  -39.02% (p=0.008 n=5)   37.00 ± ∞ ¹   -9.76% (p=0.008 n=5)_asPrometheus/base4_extra5-48                      56.00 ± ∞ ¹   34.00 ± ∞ ¹  -39.29% (p=0.008 n=5)   52.00 ± ∞ ¹   -7.14% (p=0.008 n=5)_asPrometheus/base4_extra10-48                     76.00 ± ∞ ¹   49.00 ± ∞ ¹  -35.53% (p=0.008 n=5)   72.00 ± ∞ ¹   -5.26% (p=0.008 n=5)_asPrometheus/base2_extra5-48                      44.00 ± ∞ ¹   28.00 ± ∞ ¹  -36.36% (p=0.008 n=5)   41.00 ± ∞ ¹   -6.82% (p=0.008 n=5)geomean                                            47.95         29.76        -37.94%                 43.99         -8.25%¹ need >= 6 samples for confidence interval at level 0.95

cstyan added2 commits

November 11, 2025 23:00

add benchmark that explicitly measures asPrometheus

784469d

Signed-off-by: Callum Styan <callumstyan@gmail.com>

optimize metrics aggregator asPrometheus path via reducing chance of

67a0d0d

slice growth + unnecsesary repeated metric description creation.Signed-off-by: Callum Styan <callumstyan@gmail.com>

github-actionsbot assignedcstyan

Nov 11, 2025

Labels

None yet

Movatterモバイル変換

Navigation Menu

Search code, repositories, users, issues, pull requests...

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

WIP perf: improve performance of metricsAggregator path by reducing memory allocations#20724

Are you sure you want to change the base?

WIP perf: improve performance of metricsAggregator path by reducing memory allocations#20724

Conversation

cstyan commentedNov 11, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants