- Notifications
You must be signed in to change notification settings - Fork1.1k
perf: optimize GetTemplateAppInsightsByTemplate by pre-filtering apps based on start/end times#20669
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to ourterms of service andprivacy statement. We’ll occasionally send you account related emails.
Already on GitHub?Sign in to your account
Open
cstyan wants to merge2 commits intomainChoose a base branch fromcallum/app-insights-optimize
base:main
Could not load branches
Branch not found:{{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline, and old review comments may become outdated.
+50 −20
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.Learn more about bidirectional Unicode characters
start/end timesSigned-off-by: Callum Styan <callumstyan@gmail.com>
Sign up for freeto join this conversation on GitHub. Already have an account?Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
In this PR we're optimizing the
GetTemplateAppInsightsByTemplatequery by pre-filtering out apps which do not have an active session during the start/end time window.Note: as of Nov 4 this is our most expensive query internally in terms of DB load though it only is called ~900 times in a 24h period it has a 1s average execution time and minimum of ~400ms.
This query currently looks at all entries in
workspace_app_stats(IIUC each row is an app session), then splits them into per-minute buckets before filtering out buckets that do not fall within our start/end time window. This leads to expensive query processing time to do sequential scans and joins for various data that will eventually just be thrown away due to being outside the time range.Instead, we can pre-filter out themajority of the buckets that need to be thrown away by only retrieving entries from
workspace_app_statsfor sessions where at least some portion of the sessions active time range is within our start/end time range. We keep the existing filter of buckets to ensure we still filter out buckets that fall outside the time range.The default time window for the query is 5 minutes, so start =
time.Now() - 5mand end =time.Now(). UsingEXPLAINand a specific 5 minute time window from earlier today (2025-11-03 18:00–18:05 UTC) the difference is pretty obvious:I compared/verified the output of the queries, the filtered rows and distinct app names/template IDs, and that
terminalis still not considered an app.