The question "Does it Scale?" has become somewhat of a meme in software engineering. There is a good reason for it though, because most businesses plan for success. If your app, online store, or SaaS becomes popular, you want to be sure that the system powering it can serve all your new customers.

At PostgresML, we are very concerned with scale. Our engineering background took us through scaling PostgreSQL to 100 TB+, so we're certain that it scales, but could we scale machine learning alongside it?

If you're familiar with how one runs PostgreSQL at scale, you can skip straight to the[results](broken-reference).

If you're familiar with how one runs PostgreSQL at scale, you can skip straight to the[results](broken-reference/).

Part of our thesis, and the reason why we chose Postgres as our host for machine learning, is that scaling machine learning inference is very similar to scaling read queries in a typical database cluster.

Inference speed varies based on the model complexity (e.g.`n_estimators` for XGBoost) and the size of the dataset (how many features the model uses), which is analogous to query complexity and table size in the database world and, as we'll demonstrate further on, scaling the latter is mostly a solved problem.

<figure><imgsrc="../.gitbook/assets/scaling-postgresml-3.svg"alt=""><figcaption><p><em>System Architecture</em></p></figcaption></figure>

| Component| Description|

| ---------| ---------------------------------------------------------------------------------------------------------|

| Clients| Regular Postgres clients|

PostgresML bypasses that limitation because of how Postgres itself handles concurrency:

<figure><imgsrc="../.gitbook/assets/postgres-multiprocess-2.png"alt=""><figcaption></figcaption></figure>

_PostgresML concurrency_

<div>

<figure><imgsrc="../.gitbook/assets/1M-RPS-latency.png"alt=""><figcaption><p>Latency</p></figcaption></figure>

<div>

<figure><imgsrc="../.gitbook/assets/1M-RPS-batching-latency%20(1).png"alt=""><figcaption></figcaption></figure>

<figure><imgsrc="../.gitbook/assets/1M-RPS-batching-latency (1)(1).png"alt=""><figcaption></figcaption></figure>

<figure><imgsrc="../.gitbook/assets/1M-RPS-batching-throughput%20(1).png"alt=""><figcaption></figcaption></figure>

<figure><imgsrc="../.gitbook/assets/1M-RPS-batching-throughput (1)(1).png"alt=""><figcaption></figcaption></figure>

</div>

The architectural implementations for these projects is significantly different. PostgresML takes a data centric approach with Postgres as the provider for both storage_and_ compute. To provide horizontal scalability for inference, the PostgresML team has also created[PgCat](https://github.com/postgresml/pgcat) to distribute workloads across many Postgres databases. On the other hand, MindsDB takes a service oriented approach that connects to various databases over the network.

<figure><imgsrc="../.gitbook/assets/mindsdb-architecture.png"alt=""><figcaption></figcaption></figure>

<figure><imgsrc="../.gitbook/assets/mindsdb-architecture (1).png"alt=""><figcaption></figcaption></figure>

|| MindsDB| PostgresML|

| -------------| -------------| ----------|

| On Premise| ✅| ✅|

| Web UI| ✅| ✅|

The difference in architecture leads to different tradeoffs and challenges. There are already hundreds of ways to get data into and out of a Postgres database, from just about every other service, language and platform that makes PostgresML highly compatible with other application workflows. On the other hand, the MindsDB Python service accepts connections from specifically supported clients like`psql` and provides a pseudo-SQL interface to the functionality. The service will parse incoming MindsDB commands that look similar to SQL (but are not), for tasks like configuring database connections, or doing actual machine learning. These commands typically have what looks like a sub-select, that will actually fetch data over the wire from configured databases for Machine Learning training and inference.

| translation\_en\_to\_es| t5-base| 1573| 1148| 294|

| summarization| sshleifer/distilbart-cnn-12-6| 4289| 3450| 479|

There is a general trend, the larger and slower the model is, the more work is spent inside libtorch, the less the performance of the rest matters, but for interactive models and use cases there is a significant difference. We've tried to cover the most generous use case we could between these two. If we were to compare XGBoost or other classical algorithms, that can have sub millisecond prediction times in PostgresML, the 20ms Python service overhead of MindsDB just to parse the incoming query would be hundreds of times slower.