- Notifications
You must be signed in to change notification settings - Fork2k
create db vacuum service to periodically delete old prefect resources#19280
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to ourterms of service andprivacy statement. We’ll occasionally send you account related emails.
Already on GitHub?Sign in to your account
base:main
Are you sure you want to change the base?
Conversation
codspeed-hqbot commentedOct 24, 2025 • edited
Loading Uh oh!
There was an error while loading.Please reload this page.
edited
Uh oh!
There was an error while loading.Please reload this page.
CodSpeed Performance ReportMerging#19280 willnot alter performanceComparing Summary
|
levzem commentedOct 25, 2025
zzstoatzz commentedOct 27, 2025
hi@levzem - i appreciate the PR and have a couple initial thoughts:
can you create an issue where you lay out your design here and why? this would be a non-trivial introduction to the server |
levzem commentedOct 27, 2025
@zzstoatzz thanks for the quick turnaround. I agree it should be off by default, that was a copy pasta mistake on my end. I have tagged you in a design proposal in#16054. Look forward to hearing your thoughts. |
zzstoatzz commentedOct 27, 2025
great thank you@levzem - marking this as a draft for the time being! will give feedback on the issue |
This pull request is stale because it has been open 14 days with no activity. To keep this pull request open remove stale label or comment. |
This pull request was closed because it has been stale for 14 days with no activity. If this pull request is important or you have more to add feel free to re-open it. |
zzstoatzz commentedDec 15, 2025
hi@levzem - sorry for the delay here if you're willing, can you check out thenew background servicesimplemented in docket? notably, |
Closes#16054
This PR adds a new LoopService called DBVacuum that allows for the automatic clean up of Prefect resources that are older than a configurable retention period. This is to help keep the Prefect DB from overflowing with old flow runs/task runs/etc.
The DBVacuum deletes the following:
Task runs are handled by the DELETE CASCADE rules setup on the flow run table.
The DBVacuum queries a limited amount of parent flow runs (those that do not have a parent_task_run_id) that are older than the retention and deletes them. It then deletes logs that no longer have any flow runs associated with them and finally performs the same deletion for artifacts.
The deletion of a parent flow run will delete child task runs and set their child flow runs parent_task_run_id to null, which will get picked up on the next round of the loop. This gives a guarantee that eventually all flow runs and their sub flow runs will be deleted (along with the associated logs and artifacts).
Checklist
<link to issue>"mint.json.