- Notifications
You must be signed in to change notification settings - Fork7
pgvector support for Elixir
License
pgvector/pgvector-elixir
Folders and files
| Name | Name | Last commit message | Last commit date | |
|---|---|---|---|---|
Repository files navigation
pgvector support for Elixir
Add this line to your application’smix.exs underdeps:
{:pgvector,"~> 0.3.0"}
And follow the instructions for your database library:
Or check out some examples:
- Embeddings with OpenAI
- Binary embeddings with Cohere
- Sentence embeddings with Bumblebee
- Hybrid search with Bumblebee (Reciprocal Rank Fusion)
- Sparse search with Bumblebee
- Horizontal scaling with Citus
- Bulk loading with
COPY
Createlib/postgrex_types.ex with:
Postgrex.Types.define(MyApp.PostgrexTypes,Pgvector.extensions()++Ecto.Adapters.Postgres.extensions(),[])
And add toconfig/config.exs:
config:my_app,MyApp.Repo,types:MyApp.PostgrexTypes
Create a migration
mix ecto.gen.migration create_vector_extension
with:
defmoduleMyApp.Repo.Migrations.CreateVectorExtensiondouseEcto.Migrationdefupdoexecute"CREATE EXTENSION IF NOT EXISTS vector"enddefdowndoexecute"DROP EXTENSION vector"endend
Run the migration
mix ecto.migrate
You can now use thevector type in future migrations
createtable(:items)doadd:embedding,:vector,size:3end
Also supports:halfvec,:bit, and:sparsevec
Update the model
schema"items"dofield:embedding,Pgvector.Ecto.Vectorend
Also supportsPgvector.Ecto.HalfVector,Pgvector.Ecto.Bit, andPgvector.Ecto.SparseVector
Insert a vector
aliasMyApp.{Repo,Item}Repo.insert(%Item{embedding:[1,2,3]})
Get the nearest neighbors
importEcto.QueryimportPgvector.Ecto.QueryRepo.all(fromiinItem,order_by:l2_distance(i.embedding,^Pgvector.new([1,2,3])),limit:5)
Also supportsmax_inner_product,cosine_distance,l1_distance,hamming_distance, andjaccard_distance
Convert a vector to a list or Nx tensor
item.embedding|>Pgvector.to_list()item.embedding|>Pgvector.to_tensor()
Add an approximate index in a migration
createindex("items",["embedding vector_l2_ops"],using::hnsw)# orcreateindex("items",["embedding vector_l2_ops"],using::ivfflat,options:"lists = 100")
Usevector_ip_ops for inner product andvector_cosine_ops for cosine distance
Register the extension
Postgrex.Types.define(MyApp.PostgrexTypes,Pgvector.extensions(),[])
And pass it tostart_link
{:ok,pid}=Postgrex.start_link(types:MyApp.PostgrexTypes)
Enable the extension
Postgrex.query!(pid,"CREATE EXTENSION IF NOT EXISTS vector",[])
Create a table
Postgrex.query!(pid,"CREATE TABLE items (embedding vector(3))",[])
Insert a vector
Postgrex.query!(pid,"INSERT INTO items (embedding) VALUES ($1)",[[1,2,3]])
Get the nearest neighbors
Postgrex.query!(pid,"SELECT * FROM items ORDER BY embedding <-> $1 LIMIT 5",[[1,2,3]])
Convert a vector to a list or Nx tensor
vector|>Pgvector.to_list()vector|>Pgvector.to_tensor()
Add an approximate index
Postgrex.query!(pid,"CREATE INDEX ON items USING hnsw (embedding vector_l2_ops)",[])# orPostgrex.query!(pid,"CREATE INDEX ON items USING ivfflat (embedding vector_l2_ops) WITH (lists = 100)",[])
Usevector_ip_ops for inner product andvector_cosine_ops for cosine distance
Create a vector from a list
vec=Pgvector.new([1,2,3])
Or an Nx tensor
vec=Nx.tensor([1.0,2.0,3.0])|>Pgvector.new()
Get a list
list=vec|>Pgvector.to_list()
Get an Nx tensor
tensor=vec|>Pgvector.to_tensor()
Create a half vector from a list
vec=Pgvector.HalfVector.new([1,2,3])
Or an Nx tensor
vec=Nx.tensor([1.0,2.0,3.0],type::f16)|>Pgvector.HalfVector.new()
Get a list
list=vec|>Pgvector.to_list()
Get an Nx tensor
tensor=vec|>Pgvector.to_tensor()
Create a sparse vector from a list
vec=Pgvector.SparseVector.new([1,2,3])
Or an Nx tensor
vec=Nx.tensor([1.0,2.0,3.0])|>Pgvector.SparseVector.new()
Or a map of non-zero elements
elements=%{0=>1.0,2=>2.0,4=>3.0}vec=Pgvector.SparseVector.new(elements,6)
Note: Indices start at 0
Get the number of dimensions
dim=vec|>Pgvector.SparseVector.dimensions()
Get the indices of non-zero elements
indices=vec|>Pgvector.SparseVector.indices()
Get the values of non-zero elements
values=vec|>Pgvector.SparseVector.values()
Get a list
list=vec|>Pgvector.to_list()
Get an Nx tensor
tensor=vec|>Pgvector.to_tensor()
Lists must be converted toPgvector structs for Ecto distance functions.
# beforel2_distance(i.embedding,[1,2,3])# afterl2_distance(i.embedding,^Pgvector.new([1,2,3]))
View thechangelog
Everyone is encouraged to help improve this project. Here are a few ways you can help:
- Report bugs
- Fix bugs andsubmit pull requests
- Write, clarify, or fix documentation
- Suggest or add new features
To get started with development:
git clone https://github.com/pgvector/pgvector-elixir.gitcd pgvector-elixirmix deps.getcreatedb pgvector_elixir_testmixtest
To run an example:
cd examples/loadingmix deps.getcreatedb pgvector_examplemix run example.exsAbout
pgvector support for Elixir
Resources
License
Security policy
Uh oh!
There was an error while loading.Please reload this page.
Stars
Watchers
Forks
Packages0
Uh oh!
There was an error while loading.Please reload this page.
Contributors5
Uh oh!
There was an error while loading.Please reload this page.