Movatterモバイル変換


[0]ホーム

URL:


Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

Sign up
Appearance settings

[Supervised AqlValue] Implementation of supervised AqlValue#22034

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to ourterms of service andprivacy statement. We’ll occasionally send you account related emails.

Already on GitHub?Sign in to your account

Open
cpjulia wants to merge175 commits intodevel
base:devel
Choose a base branch
Loading
fromfeature/supervised-aql-value

Conversation

@cpjulia
Copy link
Contributor

@cpjuliacpjulia commentedOct 10, 2025
edited by cursorbot
Loading

Scope & Purpose

This PR introduces memory monitoring to the AqlValue payload when it surpasses the 16 bytes, and it needs to be stored on the heap.
Now, when an AqlValue object allocates space on the heap for its payload, also info about the query's resource monitor is squeezed in, so the memory consumption corresponding to this payload is accounted until its destruction.

  • 💩 Bugfix
  • 🍕 New feature
  • 🔥 Performance improvement
  • 🔨 Refactoring/simplification

Checklist

  • Tests
    • Regression tests
    • C++Unit tests
    • integration tests
    • resilience tests
  • 📖 CHANGELOG entry made
  • 📚 documentation written (release notes, API changes, ...)
  • Backports
    • Backport for 3.12.0:(Please link PR)
    • Backport for 3.11:(Please link PR)
    • Backport for 3.10:(Please link PR)

Related Information

(Please reference tickets / specification / other PRs etc)

  • Docs PR:
  • Enterprise PR:
  • GitHub issue / Jira ticket:
  • Design document:

Note

IntroduceVPACK_SUPERVISED_SLICEAqlValue that tracks heap payload viaResourceMonitor, adjustAqlItemBlock ownership/clone logic, and add comprehensive tests.

  • AQL Core:
    • NewAqlValue variant: AddVPACK_SUPERVISED_SLICE storing aResourceMonitor* prefix and payload; implement supervised allocate/deallocate, memory accounting (memoryUsage includes prefix), and metadata packing.
    • Constructors/Init: Extend constructors (std::string_view,VPackSlice,Buffer,AqlValueHintSliceCopy) to optionally takeResourceMonitor*;initFromSlice supports supervised copies whenrm provided.
    • APIs updated:slice(),data(),destroy(),clone(),materialize(),hash,Compare, predicates (is*), conversions (to*), and accessors (at,get*,_key/_id/_from/_to) handle supervised values and copy-on-demand withrm.
    • Equality/Hash: Definestd::hash/std::equal_to for supervised slices (pointer equality within type; content-equality across managed↔supervised).
  • AqlItemBlock:
    • Cloning/Sharing: Support sharing supervised payload viaAqlValue(other, data) incloneDataAndMoveShadow; caching keyed by payload pointer.
    • Ownership: IndestroyValue, if value not in_valueCount, assume ownership moved and onlyerase().
  • Tests:
    • AddAql/AqlValueSupervisedTest.cpp covering allocation, accounting, semantics, and APIs; updateBasics/SupervisedBufferTest.cpp accordingly; include intests/CMakeLists.txt.

Written byCursor Bugbot for commitf735927. This will update automatically on new commits. Configurehere.

@cpjuliacpjulia changed the titleFeature/supervised aql value[DO NOT MERGE] Feature/supervised aql valueOct 10, 2025
@cpjuliacpjulia added this to thedevel milestoneOct 21, 2025
cpjuliaand others added12 commitsOctober 21, 2025 12:40
auto len =static_cast<std::uint64_t>(buffer.size());
initFromSlice(velocypack::Slice(buffer.data()),
static_cast<velocypack::ValueLength>(len), rm);
}
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

Bug: New buffer constructor missing empty buffer validation

The newAqlValue(velocypack::Buffer<uint8_t> const& buffer, ResourceMonitor*) constructor is missing theTRI_ASSERT(len >= 1) validation that exists in the equivalent move constructor at line 1337. If an empty buffer is passed, creating aVPackSlice frombuffer.data() would read from potentially invalid memory, causing undefined behavior. The move constructor properly validates withTRI_ASSERT(size >= 1),TRI_ASSERT(size == slice.byteSize()), andTRI_ASSERT(!slice.isExternal()), but these checks are absent from the new const-ref overload.

Fix in Cursor Fix in Web

auto len =static_cast<std::uint64_t>(buffer.size());
initFromSlice(velocypack::Slice(buffer.data()),
static_cast<velocypack::ValueLength>(len), rm);
}
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

Bug: Missing input validation in const buffer constructor

The newAqlValue(velocypack::Buffer<uint8_t> const& buffer, ResourceMonitor*) constructor is missing the input validation assertions that the analogous move constructor has. The move constructor (lines 1336-1340) includesTRI_ASSERT(size >= 1),TRI_ASSERT(size == slice.byteSize()), andTRI_ASSERT(!slice.isExternal()). The new const reference constructor directly callsinitFromSlice without any validation. If an empty buffer is passed, this could lead to undefined behavior when creating aVPackSlice from potentially null data and callinginitFromSlice with length zero.

Fix in Cursor Fix in Web

@arangodbarangodb deleted a comment fromcursorbotDec 15, 2025
@arangodbarangodb deleted a comment fromcursorbotDec 15, 2025
@arangodbarangodb deleted a comment fromcursorbotDec 15, 2025
@arangodbarangodb deleted a comment fromcursorbotDec 15, 2025
@arangodbarangodb deleted a comment fromcursorbotDec 15, 2025
@arangodbarangodb deleted a comment fromcursorbotDec 15, 2025
@arangodbarangodb deleted a comment fromcursorbotDec 15, 2025
@arangodbarangodb deleted a comment fromcursorbotDec 15, 2025
@arangodbarangodb deleted a comment fromcursorbotDec 15, 2025
@arangodbarangodb deleted a comment fromcursorbotDec 15, 2025
Comment on lines 1565 to 1566
voidAqlValue::setSupervisedData(AqlValueType at, MemoryOriginType mot,
velocypack::ValueLength length) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

mot should never be anything other thanNew. I would actually prefer to remove the parameter and verify this on the caller side if appropriate.

Comment on lines 577 to 580
static_assert(std::is_copy_constructible_v<AqlValue>);
static_assert(std::is_copy_assignable_v<AqlValue>);
static_assert(std::is_trivially_move_constructible_v<AqlValue>);
static_assert(std::is_trivially_copy_assignable_v<AqlValue>);
static_assert(std::is_trivially_move_assignable_v<AqlValue>);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

These should remain unchanged - we still need AqlValues to be trivially copyable.

Comment on lines 585 to 587
booloperator==(AqlValueconst& a, AqlValueconst& b)noexcept;
booloperator!=(AqlValueconst& a, AqlValueconst& b)noexcept;

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

Why do we introduce these operators as part of this PR?

Comment on lines 1191 to 1195
auto mot =static_cast<MemoryOriginType>(
other._data.supervisedSliceMeta.getOrigin());
auto len =static_cast<velocypack::ValueLength>(
other._data.supervisedSliceMeta.getLength());
setSupervisedData(VPACK_SUPERVISED_SLICE, mot, len);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

As previously mentioned, memory origin for our supervised values must always beNew. We can assert that here, but should otherwise not care about it. In particularsetSupervisedType should not take a MemoryOriginType parameter.

Comment on lines 1378 to 1381
if (type() == VPACK_MANAGED_SLICE) {
TRI_ASSERT(_data.managedSliceMeta.getLength() ==
VPackSlice(_data.managedSliceMeta.pointer).byteSize());
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

Suggested change
if (type() == VPACK_MANAGED_SLICE) {
TRI_ASSERT(_data.managedSliceMeta.getLength() ==
VPackSlice(_data.managedSliceMeta.pointer).byteSize());
}
TRI_ASSERT(type() != VPACK_MANAGED_SLICE) || _data.managedSliceMeta.getLength() ==
VPackSlice(_data.managedSliceMeta.pointer).byteSize());

AqlValue(velocypack::Slice slice, velocypack::ValueLength length);
explicitAqlValue(velocypack::Slice slice, velocypack::ValueLength length =0,
arangodb::ResourceMonitor* rm =nullptr);

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

Why do we now have a default value for length?
I would rather leave the constructor that takes just a slice and also extend that one to take the ResourceMonitor. It is only forwarding to the one that takes the length, which we can still do with the ResourceMonitor. That way we can also simplify a lot of places where we currently explicitly passbyteSize.

TRI_ASSERT(length >=1);
TRI_ASSERT(!slice.isExternal());
initFromSlice(slice, length, rm);
}
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

Bug: Tests pass invalid length=0 causing assertion failure and incorrect behavior

The constructorAqlValue(slice, length, rm) at line 1399 assertsTRI_ASSERT(length >= 1), but numerous tests pass0 for length (e.g.,AqlValue a1(sliceA, 0, &resourceMonitor)). In debug builds this assertion fails. In release builds,initFromSlice doesn't properly handlelength == 0: it skips the supervised/managed path (since0 > 15 is false) and memcpys 0 bytes into the inline buffer, leaving garbage data. Theif (length > 0) check at line 1491 suggests0 was intended to mean "auto-calculate from slice.byteSize()", but this logic is missing. Tests expectVPACK_SUPERVISED_SLICE but would getVPACK_INLINE with corrupted content.

Additional Locations (2)

Fix in Cursor Fix in Web

};
if ((isSup(ta) &&isMan(tb)) || (isSup(tb) &&isMan(ta))) {
return a.slice(ta).binaryEquals(b.slice(tb));
}
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

Bug: Hash and equality contract violated for cross-type comparisons

Theequal_to<AqlValue> implementation allows content-based equality betweenVPACK_SUPERVISED_SLICE andVPACK_MANAGED_SLICE/VPACK_MANAGED_STRING types, but thehash<AqlValue> uses pointer-based hashing for all these types. This violates the fundamental contract that equal objects must have the same hash. A managed slice and supervised slice with identical content will compare equal viabinaryEquals, but have different hashes since they point to different memory addresses. When used instd::unordered_set<AqlValue> (used inInputAqlItemRow::cloneToBlock), this can cause elements to not be found or duplicate entries to appear.

Additional Locations (1)

Fix in Cursor Fix in Web

voidinitFromSlice(velocypack::Slice slice, velocypack::ValueLength length);
voidinitFromSlice(velocypack::Slice slice,
velocypack::ValueLength length =0,
ResourceMonitor* rm =nullptr);
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

Bug: Private function default parameter creates unsafe API

The privateinitFromSlice function now has a defaultlength = 0 parameter, but the implementation doesn't auto-compute the length from the slice when 0 is passed. If called without an explicit length,memcpy copies 0 bytes and leaves the inline slice data uninitialized while setting the type toVPACK_INLINE. The old signature required length explicitly. All current callers pass valid lengths, but the default creates a dangerous API where future code relying on it would produce corruptedAqlValue objects containing garbage data for arrays, objects, and strings.

Additional Locations (1)

Fix in Cursor Fix in Web

return t == T::VPACK_MANAGED_SLICE || t == T::VPACK_MANAGED_STRING;
};
if ((isSup(ta) &&isMan(tb)) || (isSup(tb) &&isMan(ta))) {
return a.slice(ta).binaryEquals(b.slice(tb));
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

Bug: Hash/equality invariant violation for cross-type AqlValue comparison

Theequal_to<AqlValue> operator now allowsVPACK_SUPERVISED_SLICE to compare equal toVPACK_MANAGED_SLICE/VPACK_MANAGED_STRING based on content viabinaryEquals. However,hash<AqlValue> uses pointer identity for these types. This violates the hash/equality invariant: ifa == b, thenhash(a) == hash(b). When supervised and managed slices with identical content are used instd::unordered_set<AqlValue> (e.g., inInputAqlItemRow::cloneToBlock), lookups will fail because they hash to different buckets despite being equal.

Additional Locations (1)

Fix in Cursor Fix in Web

return t == T::VPACK_MANAGED_SLICE || t == T::VPACK_MANAGED_STRING;
};
if ((isSup(ta) &&isMan(tb)) || (isSup(tb) &&isMan(ta))) {
return a.slice(ta).binaryEquals(b.slice(tb));
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

Bug: Hash-equality contract violation for cross-type comparisons

Thestd::equal_to<AqlValue> specialization allows content-based equality betweenVPACK_SUPERVISED_SLICE andVPACK_MANAGED_SLICE/VPACK_MANAGED_STRING types, butstd::hash<AqlValue> uses pointer-based hashing for all these types. This violates the C++ standard requirement that ifa == b thenhash(a) == hash(b). The codebase usesstd::unordered_set<AqlValue> inInputAqlItemRow::cloneToBlock(), where this inconsistency can cause duplicate elements that should be deduplicated, or failed lookups for elements that exist in the set.

Additional Locations (1)

Fix in Cursor Fix in Web

Sign up for freeto join this conversation on GitHub. Already have an account?Sign in to comment

Reviewers

@mpoetermpoetermpoeter requested changes

@cursorcursor[bot]cursor[bot] left review comments

Requested changes must be addressed to merge this pull request.

Assignees

No one assigned

Labels

Projects

None yet

Milestone

devel

Development

Successfully merging this pull request may close these issues.

4 participants

@cpjulia@mpoeter@knakatasf

[8]ページ先頭

©2009-2025 Movatter.jp