- Notifications
You must be signed in to change notification settings - Fork875
[Supervised AqlValue] Implementation of supervised AqlValue#22034
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to ourterms of service andprivacy statement. We’ll occasionally send you account related emails.
Already on GitHub?Sign in to your account
base:devel
Are you sure you want to change the base?
Conversation
…COMPILE, just pushed to have the current changes)
…stead of first byte of payload
| auto len =static_cast<std::uint64_t>(buffer.size()); | ||
| initFromSlice(velocypack::Slice(buffer.data()), | ||
| static_cast<velocypack::ValueLength>(len), rm); | ||
| } |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
Bug: New buffer constructor missing empty buffer validation
The newAqlValue(velocypack::Buffer<uint8_t> const& buffer, ResourceMonitor*) constructor is missing theTRI_ASSERT(len >= 1) validation that exists in the equivalent move constructor at line 1337. If an empty buffer is passed, creating aVPackSlice frombuffer.data() would read from potentially invalid memory, causing undefined behavior. The move constructor properly validates withTRI_ASSERT(size >= 1),TRI_ASSERT(size == slice.byteSize()), andTRI_ASSERT(!slice.isExternal()), but these checks are absent from the new const-ref overload.
| auto len =static_cast<std::uint64_t>(buffer.size()); | ||
| initFromSlice(velocypack::Slice(buffer.data()), | ||
| static_cast<velocypack::ValueLength>(len), rm); | ||
| } |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
Bug: Missing input validation in const buffer constructor
The newAqlValue(velocypack::Buffer<uint8_t> const& buffer, ResourceMonitor*) constructor is missing the input validation assertions that the analogous move constructor has. The move constructor (lines 1336-1340) includesTRI_ASSERT(size >= 1),TRI_ASSERT(size == slice.byteSize()), andTRI_ASSERT(!slice.isExternal()). The new const reference constructor directly callsinitFromSlice without any validation. If an empty buffer is passed, this could lead to undefined behavior when creating aVPackSlice from potentially null data and callinginitFromSlice with length zero.
arangod/Aql/AqlValue.cpp Outdated
| voidAqlValue::setSupervisedData(AqlValueType at, MemoryOriginType mot, | ||
| velocypack::ValueLength length) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
mot should never be anything other thanNew. I would actually prefer to remove the parameter and verify this on the caller side if appropriate.
arangod/Aql/AqlValue.h Outdated
| static_assert(std::is_copy_constructible_v<AqlValue>); | ||
| static_assert(std::is_copy_assignable_v<AqlValue>); | ||
| static_assert(std::is_trivially_move_constructible_v<AqlValue>); | ||
| static_assert(std::is_trivially_copy_assignable_v<AqlValue>); | ||
| static_assert(std::is_trivially_move_assignable_v<AqlValue>); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
These should remain unchanged - we still need AqlValues to be trivially copyable.
arangod/Aql/AqlValue.h Outdated
| booloperator==(AqlValueconst& a, AqlValueconst& b)noexcept; | ||
| booloperator!=(AqlValueconst& a, AqlValueconst& b)noexcept; | ||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
Why do we introduce these operators as part of this PR?
arangod/Aql/AqlValue.cpp Outdated
| auto mot =static_cast<MemoryOriginType>( | ||
| other._data.supervisedSliceMeta.getOrigin()); | ||
| auto len =static_cast<velocypack::ValueLength>( | ||
| other._data.supervisedSliceMeta.getLength()); | ||
| setSupervisedData(VPACK_SUPERVISED_SLICE, mot, len); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
As previously mentioned, memory origin for our supervised values must always beNew. We can assert that here, but should otherwise not care about it. In particularsetSupervisedType should not take a MemoryOriginType parameter.
Uh oh!
There was an error while loading.Please reload this page.
arangod/Aql/AqlValue.cpp Outdated
| if (type() == VPACK_MANAGED_SLICE) { | ||
| TRI_ASSERT(_data.managedSliceMeta.getLength() == | ||
| VPackSlice(_data.managedSliceMeta.pointer).byteSize()); | ||
| } |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
| if (type() == VPACK_MANAGED_SLICE) { | |
| TRI_ASSERT(_data.managedSliceMeta.getLength() == | |
| VPackSlice(_data.managedSliceMeta.pointer).byteSize()); | |
| } | |
| TRI_ASSERT(type() != VPACK_MANAGED_SLICE) || _data.managedSliceMeta.getLength() == | |
| VPackSlice(_data.managedSliceMeta.pointer).byteSize()); |
| AqlValue(velocypack::Slice slice, velocypack::ValueLength length); | ||
| explicitAqlValue(velocypack::Slice slice, velocypack::ValueLength length =0, | ||
| arangodb::ResourceMonitor* rm =nullptr); | ||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
Why do we now have a default value for length?
I would rather leave the constructor that takes just a slice and also extend that one to take the ResourceMonitor. It is only forwarding to the one that takes the length, which we can still do with the ResourceMonitor. That way we can also simplify a lot of places where we currently explicitly passbyteSize.
| TRI_ASSERT(length >=1); | ||
| TRI_ASSERT(!slice.isExternal()); | ||
| initFromSlice(slice, length, rm); | ||
| } |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
Bug: Tests pass invalid length=0 causing assertion failure and incorrect behavior
The constructorAqlValue(slice, length, rm) at line 1399 assertsTRI_ASSERT(length >= 1), but numerous tests pass0 for length (e.g.,AqlValue a1(sliceA, 0, &resourceMonitor)). In debug builds this assertion fails. In release builds,initFromSlice doesn't properly handlelength == 0: it skips the supervised/managed path (since0 > 15 is false) and memcpys 0 bytes into the inline buffer, leaving garbage data. Theif (length > 0) check at line 1491 suggests0 was intended to mean "auto-calculate from slice.byteSize()", but this logic is missing. Tests expectVPACK_SUPERVISED_SLICE but would getVPACK_INLINE with corrupted content.
Additional Locations (2)
| }; | ||
| if ((isSup(ta) &&isMan(tb)) || (isSup(tb) &&isMan(ta))) { | ||
| return a.slice(ta).binaryEquals(b.slice(tb)); | ||
| } |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
Bug: Hash and equality contract violated for cross-type comparisons
Theequal_to<AqlValue> implementation allows content-based equality betweenVPACK_SUPERVISED_SLICE andVPACK_MANAGED_SLICE/VPACK_MANAGED_STRING types, but thehash<AqlValue> uses pointer-based hashing for all these types. This violates the fundamental contract that equal objects must have the same hash. A managed slice and supervised slice with identical content will compare equal viabinaryEquals, but have different hashes since they point to different memory addresses. When used instd::unordered_set<AqlValue> (used inInputAqlItemRow::cloneToBlock), this can cause elements to not be found or duplicate entries to appear.
Additional Locations (1)
| voidinitFromSlice(velocypack::Slice slice, velocypack::ValueLength length); | ||
| voidinitFromSlice(velocypack::Slice slice, | ||
| velocypack::ValueLength length =0, | ||
| ResourceMonitor* rm =nullptr); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
Bug: Private function default parameter creates unsafe API
The privateinitFromSlice function now has a defaultlength = 0 parameter, but the implementation doesn't auto-compute the length from the slice when 0 is passed. If called without an explicit length,memcpy copies 0 bytes and leaves the inline slice data uninitialized while setting the type toVPACK_INLINE. The old signature required length explicitly. All current callers pass valid lengths, but the default creates a dangerous API where future code relying on it would produce corruptedAqlValue objects containing garbage data for arrays, objects, and strings.
Additional Locations (1)
| return t == T::VPACK_MANAGED_SLICE || t == T::VPACK_MANAGED_STRING; | ||
| }; | ||
| if ((isSup(ta) &&isMan(tb)) || (isSup(tb) &&isMan(ta))) { | ||
| return a.slice(ta).binaryEquals(b.slice(tb)); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
Bug: Hash/equality invariant violation for cross-type AqlValue comparison
Theequal_to<AqlValue> operator now allowsVPACK_SUPERVISED_SLICE to compare equal toVPACK_MANAGED_SLICE/VPACK_MANAGED_STRING based on content viabinaryEquals. However,hash<AqlValue> uses pointer identity for these types. This violates the hash/equality invariant: ifa == b, thenhash(a) == hash(b). When supervised and managed slices with identical content are used instd::unordered_set<AqlValue> (e.g., inInputAqlItemRow::cloneToBlock), lookups will fail because they hash to different buckets despite being equal.
Additional Locations (1)
…s to acomodate this removal
| return t == T::VPACK_MANAGED_SLICE || t == T::VPACK_MANAGED_STRING; | ||
| }; | ||
| if ((isSup(ta) &&isMan(tb)) || (isSup(tb) &&isMan(ta))) { | ||
| return a.slice(ta).binaryEquals(b.slice(tb)); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
Bug: Hash-equality contract violation for cross-type comparisons
Thestd::equal_to<AqlValue> specialization allows content-based equality betweenVPACK_SUPERVISED_SLICE andVPACK_MANAGED_SLICE/VPACK_MANAGED_STRING types, butstd::hash<AqlValue> uses pointer-based hashing for all these types. This violates the C++ standard requirement that ifa == b thenhash(a) == hash(b). The codebase usesstd::unordered_set<AqlValue> inInputAqlItemRow::cloneToBlock(), where this inconsistency can cause duplicate elements that should be deduplicated, or failed lookups for elements that exist in the set.
Uh oh!
There was an error while loading.Please reload this page.
Scope & Purpose
This PR introduces memory monitoring to the AqlValue payload when it surpasses the 16 bytes, and it needs to be stored on the heap.
Now, when an AqlValue object allocates space on the heap for its payload, also info about the query's resource monitor is squeezed in, so the memory consumption corresponding to this payload is accounted until its destruction.
Checklist
Related Information
(Please reference tickets / specification / other PRs etc)
Note
Introduce
VPACK_SUPERVISED_SLICEAqlValuethat tracks heap payload viaResourceMonitor, adjustAqlItemBlockownership/clone logic, and add comprehensive tests.AqlValuevariant: AddVPACK_SUPERVISED_SLICEstoring aResourceMonitor*prefix and payload; implement supervised allocate/deallocate, memory accounting (memoryUsageincludes prefix), and metadata packing.std::string_view,VPackSlice,Buffer,AqlValueHintSliceCopy) to optionally takeResourceMonitor*;initFromSlicesupports supervised copies whenrmprovided.slice(),data(),destroy(),clone(),materialize(),hash,Compare, predicates (is*), conversions (to*), and accessors (at,get*,_key/_id/_from/_to) handle supervised values and copy-on-demand withrm.std::hash/std::equal_tofor supervised slices (pointer equality within type; content-equality across managed↔supervised).AqlValue(other, data)incloneDataAndMoveShadow; caching keyed by payload pointer.destroyValue, if value not in_valueCount, assume ownership moved and onlyerase().Aql/AqlValueSupervisedTest.cppcovering allocation, accounting, semantics, and APIs; updateBasics/SupervisedBufferTest.cppaccordingly; include intests/CMakeLists.txt.Written byCursor Bugbot for commitf735927. This will update automatically on new commits. Configurehere.