- Notifications
You must be signed in to change notification settings - Fork38
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to ourterms of service andprivacy statement. We’ll occasionally send you account related emails.
Already on GitHub?Sign in to your account
feat: basic BoolBuffer / BoolBufferMut#2456
base:develop
Are you sure you want to change the base?
Conversation
Can we call this BitBuffer? |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
Think we'll also want iterators and vectorized constructors (i.e. BooleanBuffer::collect_bool)
| Self { | ||
| buffer: self.buffer.clone(), | ||
| len, | ||
| offset: self.offset + start, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
Do we want to guarantee that we only hold a bit_offset? i.e.offset < 8?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
Hmm, does that help much?
25fc4bd to67729e2CompareSadly |
Where are we at with this? Need someone to takeover while you poke at FFI bindings? |
Maybe? I'm no longer sure there's a strong path forward that gives us something better than Arrow's BooleanBufferBuilder. The big issue is that mapping a bitset on top of uninit memory causes UB when you try and set bits in-place. While it generally seems harmless it is technically a violation of Rust's memory model so not guaranteed to work across platforms. |
Why does the memory have to be uninitialized? Even the ability to into_mut and update bits is valuable from an API perspective. |
until i find a better way to do this
I will take over this pr |
454875c toe26e316Compare
It's very annoying to do random in-place setting of bits in Arrow BooleanBufferBuilder. It's very append focused. For something like SparseArray in particular, you want to create a BoolArray against uninitialized memory and only set the bits corresponding to positions of non-null values.