- Notifications
You must be signed in to change notification settings - Fork352
metadata and generic filters in vector search#689
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to ourterms of service andprivacy statement. We’ll occasionally send you account related emails.
Already on GitHub?Sign in to your account
Uh oh!
There was an error while loading.Please reload this page.
Conversation
montanalow left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
I think there is a quickly growing need to move to a builder pattern, to prevent having to duplicate this functionality for example on a text_search or fuzzy_search function. We need a better API that more cleanly encapsulates recall, ranking and filtering, reranking and refiltering, and finally passing as an input to another SQL function or recalling from the database to the application.
| ifmetadata_filter: | ||
| cte_select_statement+= ( | ||
| " AND documents.metadata @> {metadata_filter}".format( |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
This is nice. We should also add a GIN index todocuments.metadata.
I have added two filters to vector_search:
metadata_filteruses basic containment operator@>andgeneric_filtertakes pretty much anything that the user wants to filter ondocumentstable using SQL syntax.generic_filtershould give us quite a bit of coverage including timestamps and nested metadata.Added three tests.