Wow! Thanks a lot for the detailed explanation and rationale behind the current design. I do have some further questions if that's ok.

It's better to queue outside of Postgres than inside.

If I understand correctly, the choice to load a model per connection in the database is working as model replication and the pooler is basically load-balancing requests and reusing the alreadywarm models. That's all great if theN versions model do fit on the hardware! but let's say you only can afford to load 1 model in your GPU at a time ( this is usually the case for LLM) which is equivalent to using 1 connection and queueing the request in the pooler. How much context switching price are we paying by not having N connections and 1 separate model? I would think that these connections might need to filter on additional data that needs to be pulled in shared memory etc ...

Many models have a lock around them. Models running in Python are locked by both the GIL.

I don't really get this point, Python GIL prevents multithreading not multiprocessing. I might be wrong here or have a simplistic view on PG, but as I understand it postgres spawns a process per connection. This means that either way we need to have synchronization across processes not threads if we need to have a shared model architecture 🤔 ?

We've implemented a background worker that can share models across databases not just connections so that our serverless users can have even cheaper access to the latest LLMs, but we haven't open-sourced this yet, since we're still testing it internally

This sounds really cool but isn't it a little bit against the idea of postgresML of colocating compute and data? Okay, I might be pedantic here 😄 and as a data scientist I do understand that it these huge models might need to be separated, but on the other hand there is a push to optimize model inference speed for LLM and also use alternative accelerators which would match perfectly with postgresML.

I'd love to see some benchmarks and configurations of LLMs in these runtimes to see how much we can improve things.

I have contributed to thellm crate for llama2 support and would love to take a swing at this. Should I open an issue referencing this ?

Thanks again, I really appreciate the response you provided👍🏼

It's better to queue outside of Postgres than inside.

If I understand correctly, the choice to load a model per connection in the database is working as model replication and the pooler is basically load-balancing requests and reusing the already warm models. That's all great if the N versions model do fit on the hardware! but let's say you only can afford to load 1 model in your GPU at a time ( this is usually the case for LLM) which is equivalent to using 1 connection and queueing the request in the pooler. How much context switching price are we paying by not having N connections and 1 separate model? I would think that these connections might need to filter on additional data that needs to be pulled in shared memory etc ...

I agree, this is an interesting question, and it depends on the complexity of the VIEW being used for inference, but I think in the case that you only have the resources for 1 copy of the model, it is likely a large model, i.e. > 10GB. Even on a GPU, models at this scale take > 50ms (depending on number of tokens), which is significantly longer than SSD access times to page data in, so I'd suspect even in this case, you should be able to achieve > 90% GPU utilization with a single model/connection queuing outside the database. There could be other stalls though, like Python tokenization for smaller models, so any examples would be informative for us to optimize further. I still think Python is the likely low hanging fruit in this case.

Many models have a lock around them. Models running in Python are locked by both the GIL.

I don't really get this point, Python GIL prevents multithreading not multiprocessing. I might be wrong here or have a simplistic view on PG, but as I understand it postgres spawns a process per connection. This means that either way we need to have synchronization across processes not threads if we need to have a shared model architecture 🤔 ?

Correct. When I was writing this sentence, I was thinking about achieving inference with the weights in truly shared memory, across multiple threads in a single process (either connection or multithreaded background worker), e.g. connections can do parallel scans.

We've implemented a background worker that can share models across databases not just connections so that our serverless users can have even cheaper access to the latest LLMs, but we haven't open-sourced this yet, since we're still testing it internally

This sounds really cool but isn't it a little bit against the idea of postgresML of colocating compute and data? Okay, I might be pedantic here 😄 and as a data scientist I do understand that it these huge models might need to be separated, but on the other hand there is a push to optimize model inference speed for LLM and also use alternative accelerators which would match perfectly with postgresML.

Yep, you've nailed the thesis, which is another reason why this work hasn't been open sourced while we're in the exploration phase. Have you seen YANDEX's recent work ontabr. Tight data/model integration is only going to get more and more important.

I'd love to see some benchmarks and configurations of LLMs in these runtimes to see how much we can improve things.

I have contributed to the llm crate for llama2 support and would love to take a swing at this. Should I open an issue referencing this ?

That'd be a great starting point to flush out the best path forward.

Thanks for your thoughts and contributing to the discussion!