Vector Search
A vector search finds the approximate or exact nearest neighbors to a given query vector.
- In a recommendation system or search engine, you can find similar records to the one you searched.
- In LLM and other AI applications, each data point can be represented byembeddings generated from existing models, following which the search returns the most relevant features.
Distance metrics
Distance metrics are a measure of the similarity between a pair of vectors.Currently, LanceDB supports the following metrics:
| Metric | Description |
|---|---|
l2 | Euclidean / l2 distance |
cosine | Cosine Similarity |
dot | Dot Production |
hamming | Hamming Distance |
Note
Thehamming metric is only available for binary vectors.
Exhaustive search (kNN)
If you do not create a vector index, LanceDB exhaustively scans theentire vector spaceand computes the distance to every vector in order to find the exact nearest neighbors. This is effectively a kNN search.
uri="data/sample-lancedb"async_db=awaitlancedb.connect_async(uri)data=[{"vector":row,"item":f"item{i}"}fori,rowinenumerate(np.random.random((10_000,1536)).astype("float32"))]async_tbl=awaitasync_db.create_table("vector_search_async",data=data)(await(awaitasync_tbl.search(np.random.random((1536)))).limit(10).to_list())By default,l2 will be used as metric type. You can specify the metric type ascosine ordot if required.
Approximate nearest neighbor (ANN) search
To perform scalable vector retrieval with acceptable latencies, it's common to build a vector index.While the exhaustive search is guaranteed to always return 100% recall, the approximate nature ofan ANN search means that using an index often involves a trade-off between recall and latency.
See theIVF_PQ index for a deeper description of howIVF_PQindexes work in LanceDB.
Binary vector
LanceDB supports binary vectors as a data type, and has the ability to search binary vectors with hamming distance. The binary vectors are stored as uint8 arrays (every 8 bits are stored as a byte):
Note
The dim of the binary vector must be a multiple of 8. A vector of dim 128 will be stored as a uint8 array of size 16.
importlancedbimportnumpyasnpimportpyarrowaspaimportpytestdb=lancedb.connect("data/binary_lancedb")schema=pa.schema([pa.field("id",pa.int64()),# for dim=256, lance stores every 8 bits in a byte# so the vector field should be a list of 256 / 8 = 32 bytespa.field("vector",pa.list_(pa.uint8(),32)),])tbl=db.create_table("my_binary_vectors",schema=schema)data=[]foriinrange(1024):vector=np.random.randint(0,2,size=256)# pack the binary vector into bytes to save spacepacked_vector=np.packbits(vector)data.append({"id":i,"vector":packed_vector,})tbl.add(data)query=np.random.randint(0,2,size=256)packed_query=np.packbits(query)tbl.search(packed_query).distance_type("hamming").to_arrow()importlancedbimportnumpyasnpimportpyarrowaspaimportpytestdb=awaitlancedb.connect_async("data/binary_lancedb")schema=pa.schema([pa.field("id",pa.int64()),# for dim=256, lance stores every 8 bits in a byte# so the vector field should be a list of 256 / 8 = 32 bytespa.field("vector",pa.list_(pa.uint8(),32)),])tbl=awaitdb.create_table("my_binary_vectors",schema=schema)data=[]foriinrange(1024):vector=np.random.randint(0,2,size=256)# pack the binary vector into bytes to save spacepacked_vector=np.packbits(vector)data.append({"id":i,"vector":packed_vector,})awaittbl.add(data)query=np.random.randint(0,2,size=256)packed_query=np.packbits(query)await(awaittbl.search(packed_query)).distance_type("hamming").to_arrow()import*aslancedbfrom"@lancedb/lancedb";import{Field,FixedSizeList,Int32,Schema,Uint8}from"apache-arrow";constschema=newSchema([newField("id",newInt32(),true),newField("vec",newFixedSizeList(32,newField("item",newUint8()))),]);constdata=lancedb.makeArrowTable(Array(1_000).fill(0).map((_,i)=>({// the 256 bits would be store in 32 bytes,// if your data is already in this format, you can skip the packBits stepid:i,vec:lancedb.packBits(Array(256).fill(i%2)),})),{schema:schema},);consttbl=awaitdb.createTable("binary_table",data);awaittbl.createIndex("vec",{config:lancedb.Index.ivfFlat({numPartitions:10,distanceType:"hamming",}),});constquery=Array(32).fill(1).map(()=>Math.floor(Math.random()*255));constresults=awaittbl.query().nearestTo(query).limit(10).toArrow();// --8<-- [end:search_binary_dataexpect(results.numRows).toBe(10);}});});Multivector type
LanceDB supports multivector type, this is useful when you have multiple vectors for a single item (e.g. with ColBert and ColPali).
You can index on a column with multivector type and search on it, the query can be single vector or multiple vectors. If the query is multiple vectorsmq, the similarity (distance) from it to any multivectormv in the dataset, is defined as:
wheresim is the similarity function (e.g. cosine).
For now, onlycosine metric is supported for multivector search.The vector value type can befloat16,float32 orfloat64.
importlancedbimportnumpyasnpimportpyarrowaspadb=lancedb.connect("data/multivector_demo")schema=pa.schema([pa.field("id",pa.int64()),# float16, float32, and float64 are supportedpa.field("vector",pa.list_(pa.list_(pa.float32(),256))),])data=[{"id":i,"vector":np.random.random(size=(2,256)).tolist(),}foriinrange(1024)]tbl=db.create_table("my_table",data=data,schema=schema)# only cosine similarity is supported for multi-vectorstbl.create_index(metric="cosine")# query with single vectorquery=np.random.random(256).astype(np.float16)tbl.search(query).to_arrow()# query with multiple vectorsquery=np.random.random(size=(2,256))tbl.search(query).to_arrow()importlancedbimportnumpyasnpimportpyarrowaspadb=awaitlancedb.connect_async("data/multivector_demo")schema=pa.schema([pa.field("id",pa.int64()),# float16, float32, and float64 are supportedpa.field("vector",pa.list_(pa.list_(pa.float32(),256))),])data=[{"id":i,"vector":np.random.random(size=(2,256)).tolist(),}foriinrange(1024)]tbl=awaitdb.create_table("my_table",data=data,schema=schema)# only cosine similarity is supported for multi-vectorsawaittbl.create_index(column="vector",config=IvfPq(distance_type="cosine"))# query with single vectorquery=np.random.random(256)awaittbl.query().nearest_to(query).to_arrow()# query with multiple vectorsquery=np.random.random(size=(2,256))awaittbl.query().nearest_to(query).to_arrow()Search with distance range
You can also search for vectors within a specific distance range from the query vector. This is useful when you want to find vectors that are not just the nearest neighbors, but also those that are within a certain distance. This can be done by using thedistance_range method.
importlancedbimportnumpyasnpdb=lancedb.connect("data/distance_range_demo")data=[{"id":i,"vector":np.random.random(256),}foriinrange(1024)]tbl=db.create_table("my_table",data=data)query=np.random.random(256)# Search for the vectors within the range of [0.1, 0.5)tbl.search(query).distance_range(0.1,0.5).to_arrow()# Search for the vectors with the distance less than 0.5tbl.search(query).distance_range(upper_bound=0.5).to_arrow()# Search for the vectors with the distance greater or equal to 0.1tbl.search(query).distance_range(lower_bound=0.1).to_arrow()importlancedbimportnumpyasnpdb=awaitlancedb.connect_async("data/distance_range_demo")data=[{"id":i,"vector":np.random.random(256),}foriinrange(1024)]tbl=awaitdb.create_table("my_table",data=data)query=np.random.random(256)# Search for the vectors within the range of [0.1, 0.5)await(awaittbl.search(query)).distance_range(0.1,0.5).to_arrow()# Search for the vectors with the distance less than 0.5await(awaittbl.search(query)).distance_range(upper_bound=0.5).to_arrow()# Search for the vectors with the distance greater or equal to 0.1await(awaittbl.search(query)).distance_range(lower_bound=0.1).to_arrow()Output search results
LanceDB returns vector search results via different formats commonly used in python.Let's create a LanceDB table with a nested schema:
fromdatetimeimportdatetimeimportlancedbfromlancedb.pydanticimportVector,LanceModelfromlancedb.queryimportBooleanQuery,BoostQuery,MatchQuery,OccurimportnumpyasnpimportpyarrowaspafrompydanticimportBaseModelclassMetadata(BaseModel):source:strtimestamp:datetimeclassDocument(BaseModel):content:strmeta:MetadataclassLanceSchema(LanceModel):id:strvector:Vector(1536)payload:Document# Let's add 100 sample rows to our datasetdata=[LanceSchema(id=f"id{i}",vector=np.random.randn(1536),payload=Document(content=f"document{i}",meta=Metadata(source=f"source{i%10}",timestamp=datetime.now()),),)foriinrange(100)]# Synchronous clienttbl=db.create_table("documents",data=data)fromdatetimeimportdatetimeimportlancedbfromlancedb.pydanticimportVector,LanceModelfromlancedb.queryimportBooleanQuery,BoostQuery,MatchQuery,OccurimportnumpyasnpimportpyarrowaspafrompydanticimportBaseModelclassMetadata(BaseModel):source:strtimestamp:datetimeclassDocument(BaseModel):content:strmeta:MetadataclassLanceSchema(LanceModel):id:strvector:Vector(1536)payload:Document# Let's add 100 sample rows to our datasetdata=[LanceSchema(id=f"id{i}",vector=np.random.randn(1536),payload=Document(content=f"document{i}",meta=Metadata(source=f"source{i%10}",timestamp=datetime.now()),),)foriinrange(100)]async_tbl=awaitasync_db.create_table("documents_async",data=data)As a PyArrow table
Usingto_arrow() we can get the results back as a pyarrow Table.This result table has the same columns as the LanceDB table, withthe addition of an_distance column for vector search or ascorecolumn for full text search.
As a Pandas DataFrame
You can also get the results as a pandas dataframe.
While other formats like Arrow/Pydantic/Python dicts have a naturalway to handle nested schemas, pandas can only store nested data as apython dict column, which makes it difficult to support nested references.So for convenience, you can also tell LanceDB to flatten a nested schemawhen creating the pandas dataframe.
If your table has a deeply nested struct, you can control how many levelsof nesting to flatten by passing in a positive integer.
Note
flatten is not yet supported with our asynchronous client.
As a list of Python dicts
You can of course return results as a list of python dicts.
As a list of Pydantic models
We can add data using Pydantic models, and we can certainlyretrieve results as Pydantic models
Note
to_pydantic() is not yet supported with our asynchronous client.
Note that in this case the extra_distance field is discarded sinceit's not part of the LanceSchema.