How much value would it be if llama.cpp exported the mmap stuff as a library?

How much value would it be if llama.cpp exported the mmap stuff as a library?

I don't think it'd help that much right now. The mmap part itself is more-or-less straightforward; replacing the current alloc+memcpy code with a buffer managed externally will be much trickier.

Have you experimented with MMaping then copying to GPU?
In my experience. I've restricted MMapping only to CPU inference & loading. MMap -> copy to GPU became a bottleneck for some reason (I assume page size potentially?)

Have you experimented with MMaping then copying to GPU? In my experience. I've restricted MMapping only to CPU inference & loading. MMap -> copy to GPU became a bottleneck for some reason (I assume page size potentially?)

Not yet. Right now I'm just reusing the I/O buffer; adding a separate code path to deliver the mapped area directly to the backend just to avoid a memcpy sounded like too much change for too little potential gain.

That behavior you describe sounds... odd. At least on Linux, large dynamically-allocated memory areas use mmap as backend anyway, so they should behave the same. Maybe it's a difference betweenfile -backed and anonymous mappings.

The direct use of the memory-mapped area for the tensor weights seems a lot more involved, so I think it' best to keep this simple for now, and add that support on a follow-up PR.

I changed the command-line to enable it by default, since that's the llama.cpp approach, but I can change it back if an opt-in is preferable.