- Notifications
You must be signed in to change notification settings - Fork875
[Supervised AqlValue] Make aggregators use supervised AqlValue#22103
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to ourterms of service andprivacy statement. We’ll occasionally send you account related emails.
Already on GitHub?Sign in to your account
base:feature/supervised-aql-value
Are you sure you want to change the base?
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
This is the final PR Bugbot will review for you during this billing cycle
Your free Bugbot reviews will reset on November 28
Details
Your team is on the Bugbot Free tier. On this plan, Bugbot will review limited PRs each billing cycle for each member of your team.
To receive Bugbot reviews on all of your PRs, visit theCursor dashboard to activate Pro and start your 14-day free trial.
Uh oh!
There was an error while loading.Please reload this page.
arangod/Aql/Aggregator.cpp Outdated
| builderCopy.close(); | ||
| returnAqlValue(std::move(*builderCopy.steal())); | ||
| returnAqlValue(std::move(*builderCopy.steal()), | ||
| &_usageScope.resourceMonitor()); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
Bug: Double Memory Accounting in AQL
TheAqlValue constructor is being passed&_usageScope.resourceMonitor() for a buffer that's already supervised viaSupervisedBuffer. Thebuilder member is initialized withSupervisedBuffer(resourceMonitor) which already tracks memory through the resource monitor. Passing the resource monitor pointer again to theAqlValue constructor causes double accounting of memory usage. The fix should either not pass the resource monitor pointer (likeAggregatorMergeLists does), or use a non-supervised buffer initially.
…angodb into feature/supervised-aggregator
Uh oh!
There was an error while loading.Please reload this page.
…angodb into feature/supervised-aggregator
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
Bug: Memory Accounting Scope Mismatch
Memory tracking was removed fromAggregatorMin::reduce(), butcmpValue.clone() doesn't use the aggregator's_resourceMonitor. Theclone() method either uses no resource monitor or usescmpValue's monitor, so memory allocated for the aggregator'svalue member isn't tracked in the aggregator's resource scope, causing incorrect memory accounting.
arangod/Aql/Aggregator.cpp#L182-L191
arangodb/arangod/Aql/Aggregator.cpp
Lines 182 to 191 ine334a03
| voidreduce(AqlValueconst& cmpValue)override { | |
| if (!cmpValue.isNull(true) && | |
| (value.isEmpty() || | |
| AqlValue::Compare(_vpackOptions, value, cmpValue,true) >0)) { | |
| // the value `null` itself will not be used in MIN() to compare lower than | |
| // e.g. value `false` | |
| value.destroy(); | |
| value = cmpValue.clone(); | |
| } |
…angodb into feature/supervised-aggregator
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
Bug: AqlValue Clones Escape Memory Tracking
Manual memory tracking was removed but the stored value is not created as a supervised AqlValue. Whenvalue = cmpValue.clone() is assigned, the resulting value won't be automatically tracked by the ResourceMonitor. The memory used by this internally stored value should either be manually tracked (as before) or created as a supervised AqlValue by passing&_resourceMonitor to the constructor.
arangod/Aql/Aggregator.cpp#L182-L191
arangodb/arangod/Aql/Aggregator.cpp
Lines 182 to 191 inbaf508d
| voidreduce(AqlValueconst& cmpValue)override { | |
| if (!cmpValue.isNull(true) && | |
| (value.isEmpty() || | |
| AqlValue::Compare(_vpackOptions, value, cmpValue,true) >0)) { | |
| // the value `null` itself will not be used in MIN() to compare lower than | |
| // e.g. value `false` | |
| value.destroy(); | |
| value = cmpValue.clone(); | |
| } |
Bug: Resource monitor misses new memory allocations.
Manual memory tracking was removed but the stored value is not created as a supervised AqlValue. Whenvalue = cmpValue.clone() is assigned, the resulting value won't be automatically tracked by the ResourceMonitor. The memory used by this internally stored value should either be manually tracked (as before) or created as a supervised AqlValue by passing&_resourceMonitor to the constructor.
arangod/Aql/Aggregator.cpp#L219-L226
arangodb/arangod/Aql/Aggregator.cpp
Lines 219 to 226 inbaf508d
| voidreduce(AqlValueconst& cmpValue)override { | |
| if (value.isEmpty() || | |
| AqlValue::Compare(_vpackOptions, value, cmpValue,true) <0) { | |
| value.destroy(); | |
| value = cmpValue.clone(); | |
| } | |
| } |
Bug: Unsupervised MIN Clones Evade Memory Tracking
Theget() method returnsvalue.clone() which creates an unsupervised AqlValue. Unlike other aggregators updated in this PR that pass&_resourceMonitor to return statements (e.g., lines 354, 489, 681), MIN still returns an unsupervised clone. Manual memory tracking was removed fromreduce() but the return value was never updated to be supervised, so memory usage won't be tracked by ResourceMonitor.
arangod/Aql/Aggregator.cpp#L193-L199
arangodb/arangod/Aql/Aggregator.cpp
Lines 193 to 199 inbaf508d
| AqlValueget()constoverride { | |
| if (value.isEmpty()) { | |
| returnAqlValue(AqlValueHintNull()); | |
| } | |
| return value.clone(); | |
| } |
Bug: MAX Aggregator: Memory Untracked by ResourceMonitor
Theget() method returnsvalue.clone() which creates an unsupervised AqlValue. Unlike other aggregators updated in this PR that pass&_resourceMonitor to return statements (e.g., lines 354, 489, 681), MAX still returns an unsupervised clone. Manual memory tracking was removed fromreduce() but the return value was never updated to be supervised, so memory usage won't be tracked by ResourceMonitor.
arangod/Aql/Aggregator.cpp#L227-L233
arangodb/arangod/Aql/Aggregator.cpp
Lines 227 to 233 inbaf508d
| AqlValueget()constoverride { | |
| if (value.isEmpty()) { | |
| returnAqlValue(AqlValueHintNull()); | |
| } | |
| return value.clone(); | |
| } |
…angodb into feature/supervised-aggregator
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
Bug: MIN/MAX aggregators lose resource tracking on clone
The manual memory tracking code was removed fromAggregatorMin::reduce() andAggregatorMax::reduce(), assuming automatic tracking. However,cmpValue.clone() is called without passing a ResourceMonitor parameter. Sinceclone() doesn't accept a ResourceMonitor argument, the cloned value is created as an untracked VPACK_MANAGED_SLICE. This breaks resource accounting: the old value is destroyed and removed from tracking, but the new cloned value is never registered with the resource monitor.
arangod/Aql/Aggregator.cpp#L182-L192
arangodb/arangod/Aql/Aggregator.cpp
Lines 182 to 192 in522b6c9
| voidreduce(AqlValueconst& cmpValue)override { | |
| if (!cmpValue.isNull(true) && | |
| (value.isEmpty() || | |
| AqlValue::Compare(_vpackOptions, value, cmpValue,true) >0)) { | |
| // the value `null` itself will not be used in MIN() to compare lower than | |
| // e.g. value `false` | |
| value.destroy(); | |
| value = cmpValue.clone(); | |
| } | |
| } |
mchacki left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
LGTM.
Unrelated comment!
arangod/Aql/Aggregator.cpp Outdated
| if (memDelta >0) { | ||
| resourceUsageScope().increase(memDelta); | ||
| } | ||
| value.destroy(); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
Not actually a change in this PR.
But wouldn't we need to test here if value should be destroyed? We do so in the reset call a couple if lines above
arangod/Aql/Aggregator.cpp Outdated
| if (memDelta >0) { | ||
| resourceUsageScope().increase(memDelta); | ||
| } | ||
| value.destroy(); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
See above comment
| value.destroy(); | ||
| }else { | ||
| value.erase();// clears inline values because destroy() doesn't call | ||
| // erase() for this case |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
Bug: Memory tracking lost when cloning values in aggregators
The refactoring removes resource tracking when cloning values inAggregatorMin andAggregatorMax. The old code calculated memory deltas and adjusted the resource usage scope accordingly. The new code simply clones without tracking, causing memory leaks in resource monitoring. Sinceclone() only preserves the resource monitor for supervised slices, regular managed slices lose tracking entirely.
Additional Locations (1)
3f236f6 to66c1564Compare…angodb into feature/supervised-aggregator
…angodb into feature/supervised-aggregator
…angodb into feature/supervised-aggregator
…angodb into feature/supervised-aggregator
…angodb into feature/supervised-aggregator
…angodb into feature/supervised-aggregator
…angodb into feature/supervised-aggregator
…angodb into feature/supervised-aggregator
…angodb into feature/supervised-aggregator
…angodb into feature/supervised-aggregator
…angodb into feature/supervised-aggregator
…angodb into feature/supervised-aggregator
Uh oh!
There was an error while loading.Please reload this page.
Scope & Purpose
This PR makes use of#22034 by passing the resource monitor as argument to the AqlValue constructors in Aggregator.cpp
Checklist
Related Information
(Please reference tickets / specification / other PRs etc)
Note
Pass ResourceMonitor to AqlValue constructions across aggregators and simplify MIN/MAX memory handling; expose monitor via Aggregator and ResourceUsageScope.
AqlValuefromget()by passing_resourceMonitor(e.g.,AVERAGE_STEP1,VARIANCE_*_STEP1,UNIQUE,SORTED_UNIQUE,MERGE_LISTS,PUSH).MIN/MAXmemory handling: replace manual resource usage adjustments withdestroy()/erase()before cloning._resourceMonitormember toAggregatorand propagate through constructors/usages.resourceMonitor()accessor toResourceUsageScope.Written byCursor Bugbot for commited49a3c. This will update automatically on new commits. Configurehere.