*[Import your data](introduction/getting-started/import-your-data/README.md)

*[CSV](introduction/getting-started/import-your-data/csv.md)

*[Foreign Data Wrapper](introduction/getting-started/import-your-data/foreign-data-wrapper.md)

*[Machine Learning](introduction/machine-learning/README.md)

*[Natural Language Processing](introduction/machine-learning/natural-language-processing/README.md)

*[Embeddings](introduction/machine-learning/natural-language-processing/embeddings.md)

*[Fill Mask](introduction/machine-learning/natural-language-processing/fill-mask.md)

*[Question Answering](introduction/machine-learning/natural-language-processing/question-answering.md)

*[Summarization](introduction/machine-learning/natural-language-processing/summarization.md)

*[Text Classification](introduction/machine-learning/natural-language-processing/text-classification.md)

*[Text Generation](introduction/machine-learning/natural-language-processing/text-generation.md)

*[Text-to-Text Generation](introduction/machine-learning/natural-language-processing/text-to-text-generation.md)

*[Token Classification](introduction/machine-learning/natural-language-processing/token-classification.md)

*[Translation](introduction/machine-learning/natural-language-processing/translation.md)

*[Zero-shot Classification](introduction/machine-learning/natural-language-processing/zero-shot-classification.md)

*[Supervised Learning](introduction/machine-learning/supervised-learning/README.md)

*[Data Pre-processing](introduction/machine-learning/supervised-learning/data-pre-processing.md)

*[Regression](introduction/machine-learning/supervised-learning/regression.md)

*[Classification](introduction/machine-learning/supervised-learning/classification.md)

*[Hyperparameter Search](introduction/machine-learning/supervised-learning/hyperparameter-search.md)

*[Joint Optimization](introduction/machine-learning/supervised-learning/joint-optimization.md)

*[Unsupervised Learning](introduction/machine-learning/unsupervised-learning.md)

*[SDKs](introduction/machine-learning/sdks/README.md)

*[Overview](introduction/machine-learning/sdks/overview.md)

*[Getting Started](introduction/machine-learning/sdks/getting-started.md)

*[OpenSourceAI](introduction/machine-learning/sdks/opensourceai.md)

*[Collections](introduction/machine-learning/sdks/collections.md)

*[Pipelines](introduction/machine-learning/sdks/pipelines.md)

*[Search](introduction/machine-learning/sdks/search.md)

*[Tutorials](introduction/machine-learning/sdks/tutorials/README.md)

*[Semantic Search](introduction/machine-learning/sdks/tutorials/semantic-search.md)

*[Semantic Search using Instructor model](introduction/machine-learning/sdks/tutorials/semantic-search-using-instructor-model.md)

*[Extractive Question Answering](introduction/machine-learning/sdks/tutorials/extractive-question-answering.md)

*[Summarizing Question Answering](introduction/machine-learning/sdks/tutorials/summarizing-question-answering.md)

*[APIs](introduction/apis/README.md)

*[SQL Extensions](introduction/apis/sql-extensions/README.md)

*[pgml.deploy()](introduction/apis/sql-extensions/pgml.deploy.md)

*[pgml.embed()](introduction/apis/sql-extensions/pgml.embed.md)

*[pgml.generate()](introduction/apis/sql-extensions/pgml.generate.md)

*[pgml.predict()](introduction/apis/sql-extensions/pgml.predict/README.md)

*[Batch Predictions](introduction/apis/sql-extensions/pgml.predict/batch-predictions.md)

*[pgml.train()](introduction/apis/sql-extensions/pgml.train/README.md)

*[Regression](introduction/apis/sql-extensions/pgml.train/regression.md)

*[Classification](introduction/apis/sql-extensions/pgml.train/classification.md)

*[Clustering](introduction/apis/sql-extensions/pgml.train/clustering.md)

*[Data Pre-processing](introduction/apis/sql-extensions/pgml.train/data-pre-processing.md)

*[Hyperparameter Search](introduction/apis/sql-extensions/pgml.train/hyperparameter-search.md)

*[Joint Optimization](introduction/apis/sql-extensions/pgml.train/joint-optimization.md)

*[pgml.transform()](introduction/apis/sql-extensions/pgml.transform/README.md)

*[Fill Mask](introduction/apis/sql-extensions/pgml.transform/fill-mask.md)

*[Question Answering](introduction/apis/sql-extensions/pgml.transform/question-answering.md)

*[Summarization](introduction/apis/sql-extensions/pgml.transform/summarization.md)

*[Text Classification](introduction/apis/sql-extensions/pgml.transform/text-classification.md)

*[Text Generation](introduction/apis/sql-extensions/pgml.transform/text-generation.md)

*[Text-to-Text Generation](introduction/apis/sql-extensions/pgml.transform/text-to-text-generation.md)

*[Token Classification](introduction/apis/sql-extensions/pgml.transform/token-classification.md)

*[Translation](introduction/apis/sql-extensions/pgml.transform/translation.md)

*[Zero-shot Classification](introduction/apis/sql-extensions/pgml.transform/zero-shot-classification.md)

*[pgml.tune()](introduction/apis/sql-extensions/pgml.tune.md)

*[Client SDKs](introduction/apis/client-sdks/README.md)

*[Overview](introduction/apis/client-sdks/getting-started.md)

*[OpenSourceAI](introduction/apis/client-sdks/opensourceai.md)

*[Collections](introduction/apis/client-sdks/collections.md)

*[Pipelines](introduction/apis/client-sdks/pipelines.md)

*[Search](introduction/apis/client-sdks/search.md)

*[Tutorials](introduction/apis/client-sdks/tutorials/README.md)

*[Semantic Search](introduction/apis/client-sdks/tutorials/semantic-search.md)

*[Semantic Search using Instructor model](introduction/apis/client-sdks/tutorials/semantic-search-using-instructor-model.md)

*[Extractive Question Answering](introduction/apis/client-sdks/tutorials/extractive-question-answering.md)

*[Summarizing Question Answering](introduction/apis/client-sdks/tutorials/summarizing-question-answering.md)

*[Chatbots](use-cases/chatbots.md)

*[Search](use-cases/improve-search-results-with-machine-learning.md)

*[Embeddings](use-cases/embeddings/README.md)

*[Generating LLM embeddings with open source models in PostgresML](use-cases/embeddings/generating-llm-embeddings-with-open-source-models-in-postgresml/README.md)

*[Tuning vector recall while generating query embeddings in the database](use-cases/embeddings/generating-llm-embeddings-with-open-source-models-in-postgresml/tuning-vector-recall-while-generating-query-embeddings-in-the-database.md)

*[Generating LLM embeddings with open source models](use-cases/embeddings/generating-llm-embeddings-with-open-source-models-in-postgresml.md)

*[Tuning vector recall while generating query embeddings in the database](use-cases/embeddings/tuning-vector-recall-while-generating-query-embeddings-in-the-database.md)

*[Personalize embedding results with application data in your database](use-cases/embeddings/personalize-embedding-results-with-application-data-in-your-database.md)

*[Fraud Detection](use-cases/fraud-detection.md)

*[Recommendation Engine](use-cases/recommendation-engine.md)

*[Time-series Forecasting](use-cases/time-series-forecasting.md)

PostgresML adds extensions to the PostgreSQL database, as well as providing separate Client SDKs in JavaScript and Python that leverage the database to implement common ML & AI use cases. 

The extensions provide all of the ML & AI functionality via SQL APIs, like training and inference. They are designed to be used directly for all ML practitioners who implement dozens of different use cases on their own machine learning models. 

We also provide Client SDKs that implement the best practices on top of the SQL APIs, to ease adoption and implement common application use cases in applications, like chatbots or search engines.

Postgres is designed to be_**extensible**_. This has created a rich open-source ecosystem of additional functionality built around the core project. Some[extensions](https://www.postgresql.org/docs/current/contrib.html) are include in the base Postgres distribution, but others are also available via the[PostgreSQL Extension Network](https://pgxn.org/).\

There are 2 foundational extensions included in a PostgresML deployment that provide functionality inside the database through SQL APIs.

***pgml** - provides Machine Learning and Artificial Intelligence APIs with access to more than 50 ML algorithms to train classification, clustering and regression models on your own data, or you can perform dozens of tasks with thousands of models downloaded from HuggingFace.

***pgvector** - provides indexing and search functionality on vectors, in addition to the traditional application database storage, including JSON and plain text, provided by PostgreSQL.

Learn more about developing with the[sql-extensions](sql-extensions/"mention")

PostgresML provides client SDKs that streamline ML & AI use cases in both JavaScript and Python. With these SDKs, you can seamlessly manage various database tables related to documents, text chunks, text splitters, LLM (Language Model) models, and embeddings. By leveraging the SDK's capabilities, you can efficiently index LLM embeddings using pgvector with HNSW for fast and accurate queries.

These SDKs delegate all work to the extensions running in the database, which minimizes software and hardware dependencies that need to be maintained at the application layer, as well as securing data and models inside the data center. Our SDKs minimize data transfer to maximize performance, efficiency, security and reliability.

Learn more about developing with the[client-sdks](client-sdks/"mention")

SDK streamlines the development of vector search applications by abstracting away thecomplexities of database management and indexing. Here's an overview of how the SDK works:

SDK streamlines the development of vector search applications by abstracting away the complexities of database management and indexing. Here's an overview of how the SDK works:

***Automatic Document and Text Chunk Management**: The SDK provides a convenient interface to manage documents and pipelines, automatically handling chunking and embedding for you. You can easily organize and structure your text data within the PostgreSQL database.

***Open Source Model Integration**: With the SDK, you can seamlessly incorporate a wide range of open source models to generate high-quality embeddings. These models capture the semantic meaning of text and enable powerful analysis and search capabilities.

***Embedding Indexing**: The Python SDK utilizes the PgVector extension to efficiently index the embeddings generated by the open source models. This indexing process optimizes search performance and allows for fast and accurate retrieval of relevant results.

***Querying and Search**: Once the embeddings are indexed, you can perform vector-based searches on the documents and text chunks stored in the PostgreSQL database. The SDK provides intuitive methods for executing queries and retrieving search results.

Collections are the organizational building blocks of the SDK. They manage all documents and related chunks, embeddings, tsvectors, and pipelines.

{% tabs %}

{% tab title="Python" %}

collection= Collection("test_collection")

{% endtab %}

{% tab title="JavaScript" %}

collection=pgml.newCollection("test_collection")

{% endtab %}

{% tab title="Python" %}

collection= Collection("test_collection")

{% endtab %}

{% endtabs %}

Create a Collection that reads from a different database than that set by the environment variable`DATABASE_URL`.

{% tabs %}

{% tab title="Python" %}

collection= Collection("test_collection",CUSTOM_DATABASE_URL)

{% endtab %}

{% tab title="Javascript" %}

collection=pgml.newCollection("test_collection",CUSTOM_DATABASE_URL)

{% endtab %}

{% tab title="Python" %}

collection= Collection("test_collection",CUSTOM_DATABASE_URL)

{% endtab %}

{% endtabs %}

**Upsert documents with metadata**

{% tabs %}

{% tab title="JavaScript" %}

constdocuments= [

{

id:"Document One",

text:"document one contents...",

},

{

id:"Document Two",

text:"document two contents...",

},

];

awaitcollection.upsert_documents(documents);

{% endtab %}

{% tab title="Python" %}

documents= [

await collection.upsert_documents(documents)

{% endtab %}

{% tab title="JavaScript" %}

constdocuments= [

{

id:"Document One",

text:"document one contents...",

},

{

id:"Document Two",

text:"document two contents...",

},

];

awaitcollection.upsert_documents(documents);

{% endtab %}

{% endtabs %}