@lancedb/lancedb •Docs
@lancedb/lancedb / VectorQuery
Class: VectorQuery
A builder used to construct a vector search
This builder can be reused to execute the query many times.
See
Extends
StandardQueryBase<NativeVectorQuery>
Properties
inner
Inherited from
StandardQueryBase.inner
Methods
addQueryVector()
Parameters
- vector:
IntoVector
Returns
analyzePlan()
Executes the query and returns the physical query plan annotated with runtime metrics.
This is useful for debugging and performance analysis, as it shows how the query was executedand includes metrics such as elapsed time, rows processed, and I/O statistics.
Returns
Promise<string>
A query execution plan with runtime metrics for each step.
Example
import*aslancedbfrom"@lancedb/lancedb"constdb=awaitlancedb.connect("./.lancedb");consttable=awaitdb.createTable("my_table",[{vector:[1.1,0.9],id:"1"},]);constplan=awaittable.query().nearestTo([0.5,0.2]).analyzePlan();Exampleoutput(withruntimemetricsinlined):AnalyzeExecverbose=true,metrics=[]ProjectionExec:expr=[id@3asid,vector@0asvector,_distance@2as_distance],metrics=[output_rows=1,elapsed_compute=3.292µs]Take:columns="vector, _rowid, _distance, (id)",metrics=[output_rows=1,elapsed_compute=66.001µs,batches_processed=1,bytes_read=8,iops=1,requests=1]CoalesceBatchesExec:target_batch_size=1024,metrics=[output_rows=1,elapsed_compute=3.333µs]GlobalLimitExec:skip=0,fetch=10,metrics=[output_rows=1,elapsed_compute=167ns]FilterExec:_distance@2ISNOTNULL,metrics=[output_rows=1,elapsed_compute=8.542µs]SortExec:TopK(fetch=10),expr=[_distance@2ASCNULLSLAST],metrics=[output_rows=1,elapsed_compute=63.25µs,row_replacements=1]KNNVectorDistance:metric=l2,metrics=[output_rows=1,elapsed_compute=114.333µs,output_batches=1]LanceScan:uri=/path/to/data, projection=[vector], row_id=true, row_addr=false, ordered=false, metrics=[output_rows=1, elapsed_compute=103.626µs, bytes_read=549, iops=2, requests=2]Inherited from
StandardQueryBase.analyzePlan
bypassVectorIndex()
If this is called then any vector index is skipped
An exhaustive (flat) search will be performed. The query vector willbe compared to every vector in the table. At high scales this can beexpensive. However, this is often still useful. For example, skippingthe vector index can give you ground truth results which you can use tocalculate your recall to select an appropriate value for nprobes.
Returns
column()
Set the vector column to query
This controls which column is compared to the query vector supplied inthe call to
Parameters
- column:
string
Returns
See
This parameter must be specified if the table has more than one columnwhose data type is a fixed-size-list of floats.
distanceRange()
Parameters
lowerBound?:
numberupperBound?:
number
Returns
distanceType()
Set the distance metric to use
When performing a vector search we try and find the "nearest" vectors accordingto some kind of distance metric. This parameter controls which distance metric touse. See
Parameters
- distanceType:
"l2"|"cosine"|"dot"
Returns
See
IvfPqOptions.distanceType for more details on the differentdistance metrics available.
Note: if there is a vector index then the distance type used MUST match the distancetype used to train the vector index. If this is not done then the results will beinvalid.
By default "l2" is used.
ef()
Set the number of candidates to consider during the search
This argument is only used when the vector column has an HNSW index.If there is no index then this value is ignored.
Increasing this value will increase the recall of your query but willalso increase the latency of your query. The default value is 1.5*limit.
Parameters
- ef:
number
Returns
execute()
Execute the query and return the results as an
Parameters
- options?:
Partial<QueryExecutionOptions>
Returns
AsyncGenerator<RecordBatch<any>,void,unknown>
See
- AsyncIteratorof
- RecordBatch.
By default, LanceDb will use many threads to calculate results and, whenthe result set is large, multiple batches will be processed at one time.This readahead is limited however and backpressure will be applied if thisstream is consumed slowly (this constrains the maximum memory used by asingle query)
Inherited from
StandardQueryBase.execute
explainPlan()
Generates an explanation of the query execution plan.
Parameters
- verbose:
boolean=falseIf true, provides a more detailed explanation. Defaults to false.
Returns
Promise<string>
A Promise that resolves to a string containing the query execution plan explanation.
Example
import*aslancedbfrom"@lancedb/lancedb"constdb=awaitlancedb.connect("./.lancedb");consttable=awaitdb.createTable("my_table",[{vector:[1.1,0.9],id:"1"},]);constplan=awaittable.query().nearestTo([0.5,0.2]).explainPlan();Inherited from
StandardQueryBase.explainPlan
fastSearch()
Skip searching un-indexed data. This can make search faster, but will missany data that is not yet indexed.
UseTable#optimize to index all un-indexed data.
Returns
this
Inherited from
StandardQueryBase.fastSearch
filter()
A filter statement to be applied to this query.
Parameters
- predicate:
string
Returns
this
See
where
Deprecated
Usewhere instead
Inherited from
StandardQueryBase.filter
fullTextSearch()
Parameters
query:
string|FullTextQueryoptions?:
Partial<FullTextSearchOptions>
Returns
this
Inherited from
StandardQueryBase.fullTextSearch
limit()
Set the maximum number of results to return.
By default, a plain search has no limit. If this method is notcalled then every valid row from the table will be returned.
Parameters
- limit:
number
Returns
this
Inherited from
StandardQueryBase.limit
maximumNprobes()
Set the maximum number of probes used.
This controls the maximum number of partitions that will be searched. If thisnumber is greater than minimumNprobes then the excess partitions willonly besearched if we have not found enough results. This can be useful when there isa narrow filter to allow these queries to spend more time searching and avoidpotential false negatives.
Parameters
- maximumNprobes:
number
Returns
minimumNprobes()
Set the minimum number of probes used.
This controls the minimum number of partitions that will be searched. Thisparameter will impact every query against a vector index, regardless of thefilter. Seenprobes for more details. Higher values will increase recallbut will also increase latency.
Parameters
- minimumNprobes:
number
Returns
nprobes()
Set the number of partitions to search (probe)
This argument is only used when the vector column has an IVF PQ index.If there is no index then this value is ignored.
The IVF stage of IVF PQ divides the input into partitions (clusters) ofrelated values.
The partition whose centroids are closest to the query vector will beexhaustiely searched to find matches. This parameter controls how manypartitions should be searched.
Increasing this value will increase the recall of your query but willalso increase the latency of your query. The default value is 20. Thisdefault is good for many cases but the best value to use will depend onyour data and the recall that you need to achieve.
For best results we recommend tuning this parameter with a benchmark againstyour actual data to find the smallest possible value that will still giveyou the desired recall.
For more fine grained control over behavior when you have a very narrow filteryou can useminimumNprobes andmaximumNprobes. This method sets boththe minimum and maximum to the same value.
Parameters
- nprobes:
number
Returns
offset()
Set the number of rows to skip before returning results.
This is useful for pagination.
Parameters
- offset:
number
Returns
this
Inherited from
StandardQueryBase.offset
outputSchema()
Returns the schema of the output that will be returned by this query.
This can be used to inspect the types and names of the columns that will bereturned by the query before executing it.
Returns
Promise<Schema<any>>
An Arrow Schema describing the output columns.
Inherited from
StandardQueryBase.outputSchema
postfilter()
If this is called then filtering will happen after the vector search instead ofbefore.
By default filtering will be performed before the vector search. This is howfiltering is typically understood to work. This prefilter step does add someadditional latency. Creating a scalar index on the filter column(s) canoften improve this latency. However, sometimes a filter is too complex or scalarindices cannot be applied to the column. In these cases postfiltering can beused instead of prefiltering to improve latency.
Post filtering applies the filter to the results of the vector search. This meanswe only run the filter on a much smaller set of data. However, it can cause thequery to return fewer thanlimit results (or even no results) if none of the nearestresults match the filter.
Post filtering happens during the "refine stage" (described in more detail in
Returns
See
VectorQuery#refineFactor). This means that setting a higher refinefactor can often help restore some of the results lost by post filtering.
refineFactor()
A multiplier to control how many additional rows are taken during the refine step
This argument is only used when the vector column has an IVF PQ index.If there is no index then this value is ignored.
An IVF PQ index stores compressed (quantized) values. They query vector is comparedagainst these values and, since they are compressed, the comparison is inaccurate.
This parameter can be used to refine the results. It can improve both improve recalland correct the ordering of the nearest results.
To refine results LanceDb will first perform an ANN search to find the nearestlimit *refine_factor results. In other words, ifrefine_factor is 3 andlimit is the default (10) then the first 30 results will be selected. LanceDbthen fetches the full, uncompressed, values for these 30 results. The results arethen reordered by the true distance and only the nearest 10 are kept.
Note: there is a difference between calling this method with a value of 1 and nevercalling this method at all. Calling this method with any value will have an impacton your search latency. When you call this method with arefine_factor of 1 thenLanceDb still needs to fetch the full, uncompressed, values so that it can potentiallyreorder the results.
Note: if this method is NOT called then the distances returned in the _distance columnwill be approximate distances based on the comparison of the quantized query vectorand the quantized result vectors. This can be considerably different than the truedistance between the query vector and the actual uncompressed vector.
Parameters
- refineFactor:
number
Returns
rerank()
Parameters
- reranker:
Reranker
Returns
select()
Return only the specified columns.
By default a query will return all columns from the table. However, this can havea very significant impact on latency. LanceDb stores data in a columnar fashion. Thismeans we can finely tune our I/O to select exactly the columns we need.
As a best practice you should always limit queries to the columns that you need. If youpass in an array of column names then only those columns will be returned.
You can also use this method to create new "dynamic" columns based on your existing columns.For example, you may not care about "a" or "b" but instead simply want "a + b". This is oftenseen in the SELECT clause of an SQL query (e.g.SELECT a+b FROM my_table).
To create dynamic columns you can pass in a Map
For example, an SQL query might stateSELECT a + b AS combined, c. The equivalentinput to this method would be:
Parameters
- columns:
string|string[] |Record<string,string> |Map<string,string>
Returns
this
Example
newMap([["combined","a + b"],["c","c"]])Columnswillalwaysbereturnedintheordergiven,evenifthatorderisdifferentthantheorderusedwhenaddingthedata.Notethatyoucanpassina`Record<string, string>`(e.g.anobjectliteral).Thismethoduses`Object.entries`whichshouldpreservetheinsertionorderoftheobject.However,objectinsertionorderiseasytogetwrongand`Map`ismorefoolproof.Inherited from
StandardQueryBase.select
toArray()
Collect the results as an array of objects.
Parameters
- options?:
Partial<QueryExecutionOptions>
Returns
Promise<any[]>
Inherited from
StandardQueryBase.toArray
toArrow()
Collect the results as an Arrow
Parameters
- options?:
Partial<QueryExecutionOptions>
Returns
Promise<Table<any>>
See
ArrowTable.
Inherited from
StandardQueryBase.toArrow
where()
A filter statement to be applied to this query.
The filter should be supplied as an SQL query string. For example:
Parameters
- predicate:
string
Returns
this
Example
x>10y>0ANDy<100x>5ORy='test'Filteringperformancecanoftenbeimprovedbycreatingascalarindexonthefiltercolumn(s).Inherited from
StandardQueryBase.where
withRowId()
Whether to return the row id in the results.
This column can be used to match results between different queries. Forexample, to match results from a full text search and a vector search inorder to perform hybrid search.
Returns
this
Inherited from
StandardQueryBase.withRowId