- Notifications
You must be signed in to change notification settings - Fork1k
Commit53e8e9c
authored
fix: reduce cost of prebuild failure (#17697)
Relates to#17432### Part 1:Notes:- `GetPresetsAtFailureLimit` SQL query is added, which is similar to`GetPresetsBackoff`, they use same CTEs: `filtered_builds`,`time_sorted_builds`, but they are still different.- Query is executed on every loop iteration. We can consider markingspecific preset as permanently failed as an optimization to avoidexecuting query on every loop iteration. But I decided don't do it fornow.- By default `FailureHardLimit` is set to 3.- `FailureHardLimit` is configurable. Setting it to zero - means thathard limit is disabled.### Part 2Notes:- `PrebuildFailureLimitReached` notification is added.- Notification is sent to template admins.- Notification is sent only the first time, when hard limit is reached.But it will `log.Warn` on every loop iteration.- I introduced this enum:```sqlCREATE TYPE prebuild_status AS ENUM ( 'normal', -- Prebuilds are working as expected; this is the default, healthy state. 'hard_limited', -- Prebuilds have failed repeatedly and hit the configured hard failure limit; won't be retried anymore. 'validation_failed' -- Prebuilds failed due to a non-retryable validation error (e.g. template misconfiguration); won't be retried.);````validation_failed` not used in this PR, but I think it will be used innext one, so I wanted to save us an extra migration.- Notification looks like this:<img width="472" alt="image"src="https://github.com/user-attachments/assets/e10efea0-1790-4e7f-a65c-f94c40fced27"/>### Latest notification views:<img width="463" alt="image"src="https://github.com/user-attachments/assets/11310c58-68d1-4075-a497-f76d854633fe"/><img width="725" alt="image"src="https://github.com/user-attachments/assets/6bbfe21a-91ac-47c3-a9d1-21807bb0c53a"/>1 parente1934fe commit53e8e9c
File tree
32 files changed
+1160
-60
lines changed- cli/testdata
- coderd
- apidoc
- database
- dbauthz
- dbmem
- dbmetrics
- dbmock
- migrations
- queries
- notifications
- testdata/rendered-templates
- smtp
- webhook
- prebuilds
- codersdk
- docs/reference/api
- enterprise/coderd/prebuilds
- site/src/api
32 files changed
+1160
-60
lines changedOriginal file line number | Diff line number | Diff line change | |
---|---|---|---|
| |||
704 | 704 |
| |
705 | 705 |
| |
706 | 706 |
| |
| 707 | + | |
| 708 | + | |
| 709 | + | |
| 710 | + |
Some generated files are not rendered by default. Learn more aboutcustomizing how changed files appear on GitHub.
Some generated files are not rendered by default. Learn more aboutcustomizing how changed files appear on GitHub.
Original file line number | Diff line number | Diff line change | |
---|---|---|---|
| |||
2226 | 2226 |
| |
2227 | 2227 |
| |
2228 | 2228 |
| |
| 2229 | + | |
| 2230 | + | |
| 2231 | + | |
| 2232 | + | |
| 2233 | + | |
| 2234 | + | |
| 2235 | + | |
| 2236 | + | |
| 2237 | + | |
2229 | 2238 |
| |
2230 | 2239 |
| |
2231 | 2240 |
| |
| |||
4201 | 4210 |
| |
4202 | 4211 |
| |
4203 | 4212 |
| |
| 4213 | + | |
| 4214 | + | |
| 4215 | + | |
| 4216 | + | |
| 4217 | + | |
| 4218 | + | |
| 4219 | + | |
| 4220 | + | |
| 4221 | + | |
| 4222 | + | |
| 4223 | + | |
| 4224 | + | |
| 4225 | + | |
| 4226 | + | |
| 4227 | + | |
| 4228 | + | |
| 4229 | + | |
| 4230 | + | |
4204 | 4231 |
| |
4205 | 4232 |
| |
4206 | 4233 |
| |
|
Original file line number | Diff line number | Diff line change | |
---|---|---|---|
| |||
4924 | 4924 |
| |
4925 | 4925 |
| |
4926 | 4926 |
| |
| 4927 | + | |
| 4928 | + | |
| 4929 | + | |
| 4930 | + | |
| 4931 | + | |
4927 | 4932 |
| |
4928 | 4933 |
| |
4929 | 4934 |
| |
| |||
4971 | 4976 |
| |
4972 | 4977 |
| |
4973 | 4978 |
| |
| 4979 | + | |
4974 | 4980 |
| |
4975 | 4981 |
| |
| 4982 | + | |
| 4983 | + | |
| 4984 | + | |
| 4985 | + | |
| 4986 | + | |
| 4987 | + | |
| 4988 | + | |
| 4989 | + | |
| 4990 | + | |
| 4991 | + | |
| 4992 | + | |
| 4993 | + | |
| 4994 | + | |
| 4995 | + | |
| 4996 | + | |
| 4997 | + | |
| 4998 | + | |
| 4999 | + | |
| 5000 | + | |
| 5001 | + | |
| 5002 | + | |
| 5003 | + | |
| 5004 | + | |
| 5005 | + | |
| 5006 | + | |
4976 | 5007 |
| |
4977 | 5008 |
| |
4978 | 5009 |
| |
|
Original file line number | Diff line number | Diff line change | |
---|---|---|---|
| |||
4287 | 4287 |
| |
4288 | 4288 |
| |
4289 | 4289 |
| |
| 4290 | + | |
4290 | 4291 |
| |
4291 | 4292 |
| |
4292 | 4293 |
| |
| |||
4352 | 4353 |
| |
4353 | 4354 |
| |
4354 | 4355 |
| |
| 4356 | + | |
| 4357 | + | |
| 4358 | + | |
| 4359 | + | |
4355 | 4360 |
| |
4356 | 4361 |
| |
4357 | 4362 |
| |
| |||
9089 | 9094 |
| |
9090 | 9095 |
| |
9091 | 9096 |
| |
| 9097 | + | |
9092 | 9098 |
| |
9093 | 9099 |
| |
9094 | 9100 |
| |
| |||
10917 | 10923 |
| |
10918 | 10924 |
| |
10919 | 10925 |
| |
| 10926 | + | |
| 10927 | + | |
| 10928 | + | |
| 10929 | + | |
| 10930 | + | |
| 10931 | + | |
| 10932 | + | |
| 10933 | + | |
| 10934 | + | |
| 10935 | + | |
| 10936 | + | |
| 10937 | + | |
| 10938 | + | |
| 10939 | + | |
| 10940 | + | |
| 10941 | + | |
| 10942 | + | |
| 10943 | + | |
| 10944 | + | |
10920 | 10945 |
| |
10921 | 10946 |
| |
10922 | 10947 |
| |
|
Some generated files are not rendered by default. Learn more aboutcustomizing how changed files appear on GitHub.
Some generated files are not rendered by default. Learn more aboutcustomizing how changed files appear on GitHub.
Some generated files are not rendered by default. Learn more aboutcustomizing how changed files appear on GitHub.
Lines changed: 1 addition & 0 deletions
Original file line number | Diff line number | Diff line change | |
---|---|---|---|
| |||
| 1 | + |
0 commit comments
Comments
(0)