LangChain 🦜️🔗↗

LangChain is a framework designed for building applications with large language models (LLMs) by chaining together various components. It supports a range of functionalities including memory, agents, and chat models, enabling developers to create context-aware applications.

LangChain streamlines these stages (in figure above) by providing pre-built components and tools for integration, memory management, and deployment, allowing developers to focus on application logic rather than underlying complexities.

Integration ofLangchain withLanceDB enables applications to retrieve the most relevant data by comparing query vectors against stored vectors, facilitating effective information retrieval. It results in better and context aware replies and actions by the LLMs.

Quick Start

You can load your document data using langchain's loaders, for this example we are usingTextLoader andOpenAIEmbeddings as the embedding model. Checkout Complete example here -LangChain demo

importosfromlangchain.document_loadersimportTextLoaderfromlangchain.vectorstoresimportLanceDBfromlangchain_openaiimportOpenAIEmbeddingsfromlangchain_text_splittersimportCharacterTextSplitteros.environ["OPENAI_API_KEY"]="sk-..."loader=TextLoader("../../modules/state_of_the_union.txt")# Replace with your data pathdocuments=loader.load()documents=CharacterTextSplitter().split_documents(documents)embeddings=OpenAIEmbeddings()docsearch=LanceDB.from_documents(documents,embeddings)query="What did the president say about Ketanji Brown Jackson"docs=docsearch.similarity_search(query)print(docs[0].page_content)

Documentation

In the above exampleLanceDB vector store class object is created usingfrom_documents() method which is aclassmethod and returns the initialized class object.

You can also useLanceDB.from_texts(texts: List[str],embedding: Embeddings) class method.

The exhaustive list of parameters forLanceDB vector store are :

db_url="db://lang_test"# url of db you createdapi_key="xxxxx"# your API keyregion="us-east-1-dev"# your selected regionvector_store=LanceDB(uri=db_url,api_key=api_key,#(dont include for local API)region=region,#(dont include for local API)embedding=embeddings,table_name='langchain_test'# Optional)

Methods

add_texts()

This method turn texts into embedding and add it to the database.

It returns list of ids of the added texts.

vector_store.add_texts(texts=['test_123'],metadatas=[{'source':'wiki'}])#Additionaly, to explore the table you can load it into a df or save it in a csv file:tbl=vector_store.get_table()print("tbl:",tbl)pd_df=tbl.to_pandas()pd_df.to_csv("docsearch.csv",index=False)# you can also create a new vector store object using an older connection object:vector_store=LanceDB(connection=tbl,embedding=embeddings)

create_index()

This method creates a scalar(for non-vector cols) or a vector index on a table.

For index creation make sure your table has enough data in it. An ANN index is ususally not needed for datasets ~100K vectors. For large-scale (>1M) or higher dimension vectors, it is beneficial to create an ANN index.

# for creating vector indexvector_store.create_index(vector_col='vector',metric='cosine')# for creating scalar index(for non-vector columns)vector_store.create_index(col_name='text')

similarity_search()

This method performs similarity search based ontext query.

Return documents most similar to the querywithout relevance scores.

docs=docsearch.similarity_search(query)print(docs[0].page_content)

similarity_search_by_vector()

The method returns documents that are most similar to the specifiedembedding (query) vector.

It does not provide relevance scores.

docs=docsearch.similarity_search_by_vector(query)print(docs[0].page_content)

similarity_search_with_score()

Returns documents most similar to thequery string along with their relevance scores.

It gets called by base class'ssimilarity_search_with_relevance_scores which selects relevance score based on our_select_relevance_score_fn.

docs=docsearch.similarity_search_with_relevance_scores(query)print("relevance score - ",docs[0][1])print("text- ",docs[0][0].page_content[:1000])

similarity_search_by_vector_with_relevance_scores()

Similarity search usingquery vector.

The method returns documents most similar to the specified embedding (query) vector, along with their relevance scores.

docs=docsearch.similarity_search_by_vector_with_relevance_scores(query_embedding)print("relevance score - ",docs[0][1])print("text- ",docs[0][0].page_content[:1000])

max_marginal_relevance_search()

This method returns docs selected using the maximal marginal relevance(MMR).Maximal marginal relevance optimizes for similarity to query AND diversity among selected documents.

Similarly,max_marginal_relevance_search_by_vector() function returns docs most similar to the embedding passed to the function using MMR. instead of a string query you need to pass the embedding to be searched for.

result=docsearch.max_marginal_relevance_search(query="text")result_texts=[doc.page_contentfordocinresult]print(result_texts)## search by vector :result=docsearch.max_marginal_relevance_search_by_vector(embeddings.embed_query("text"))result_texts=[doc.page_contentfordocinresult]print(result_texts)

add_images()

This method ddds images by automatically creating their embeddings and adds them to the vectorstore.

It returns list of IDs of the added images.

vec_store.add_images(uris=image_uris)# here image_uris are local fs paths to the images.