Related to#574 , draft for the v3 Rust bindings. If we're gonna break APIs, let's just go all the way :D

The goal is to prevent unsoundness from the safe Rust interface, turning as many illegal operations as possible into compile errors , and catch the others at runtime.

Sorry for the mega-PR, but I don't know if it makes sense to split this up since all changes kind of affect everything.

Error handling, type safety

In general: existing methods onVectorType becomeunsafe fn xxx_unchecked, and safe variants are added that check all required invariants.

There's some ambiguity as to what is actually invalid/unsafe here. As far as I can tell, using an Index created with oneScalarKind (sayint8) as one of a different type (like inserting anf64 vector) does not actually cause any memory unsafety. It can garble data though, which I'd say is enough to make the operationunsafe (pretty common in the ecosystem AFAIK?).

So I've added a check that the casts that will be done on the C++ make sense, but it would be good to have more opinions on which ones are ok and which ones are almost certainly mistakes. Some are easy (f32 tof16,bf16 is ok,f64 tob1x8 is not for example), others less so.

SafeIndex views into file, buffer

Currently you can have anIndex reading a buffer passed in by the user, which will hopefully stay live as long as theIndex. Though the method is marked unsafe, it's easy to create a dangling pointer situation which is unfortunate in Rust of all places. Also, these immutableIndex views will throw exceptions if any mutating methods are called on them.

My proposed solution is to separate the regular, mutableIndex that owns its memory and the immutableIndexView into different types. This wayIndexView can be associated with a lifetime so the backing buffer can't be dropped while it's being used, and theIndexView object (holding a pointer) can not be moved between threads.

While we're at it, splitting mutating and readonly methods onIndex into different traits allowsIndexView to only expose read methods and prevent some runtime exceptions this way.

Custom metric closure shenanigans

This part could maybe be split into a different PR. First a lot of duplicated code can be written in a generic way, which is always nice. There's also a memory leak pointed out in#629 (fixed by properly dropping the currently set metric closure when it's overwritten).

I only noticed that because I wanted to see how ergonomics could be improved here. Currently custom metrics are boxed closures taking*const T arguments, which would be very nice to instead turn into&[T] slices of correct length/dimensionality. Also, closures are currently double-boxed becauseBox<dyn Fn..> is a wide pointer, andmetric_punned_t wants a regular pointer, so there's three (if I'm counting correctly) pointer dereferences per call to the metric function (trampoline -> pointer to wide pointer -> dynamic dispatch fordyn Fn trait object).

Ithink it's possible to add a shim in there to turn the pointers into slices without adding another indirection (currentlyclosure_address is a pointer to theBox<dyn>, when it could instead point to the trait object/wide pointer instead and the trampoline casts that into a valid trait object reference). But I'm not sure if it's worth it when you could also just allow the user a regularfn (&[T], &[T]) -> Distance pointer, which avoid a lot of that and covers many (most?) use cases just fine.

FFI and closures is a difficult topic though so it would be cool for other to weigh in and check what I'm saying is correct.