Pinecone (sparse)
Pinecone is a vector database with broad functionality.
This notebook shows how to use functionality related to thePinecone vector database.
Setup
To use thePineconeSparseVectorStore you first need to install the partner package, as well as the other packages used throughout this notebook.
%pip install-qU"langchain-pinecone==0.2.5"
[33mWARNING: pinecone 6.0.2 does not provide the extra 'async'[0m[33m
[0m
Credentials
Create a new Pinecone account, or sign into your existing one, and create an API key to use in this notebook.
import os
from getpassimport getpass
from pineconeimport Pinecone
# get API key at app.pinecone.io
os.environ["PINECONE_API_KEY"]= os.getenv("PINECONE_API_KEY")or getpass(
"Enter your Pinecone API key: "
)
# initialize client
pc= Pinecone()
Enter your Pinecone API key: ··········
Initialization
Before initializing our vector store, let's connect to a Pinecone index. If one named index_name doesn't exist, it will be created.
from pineconeimport AwsRegion, CloudProvider, Metric, ServerlessSpec
index_name="langchain-sparse-vector-search"# change if desired
model_name="pinecone-sparse-english-v0"
ifnot pc.has_index(index_name):
pc.create_index_for_model(
name=index_name,
cloud=CloudProvider.AWS,
region=AwsRegion.US_EAST_1,
embed={
"model": model_name,
"field_map":{"text":"chunk_text"},
"metric": Metric.DOTPRODUCT,
},
)
index= pc.Index(index_name)
print(f"Index `{index_name}` host:{index.config.host}")
Index `langchain-sparse-vector-search` host: https://langchain-sparse-vector-search-yrrgefy.svc.aped-4627-b74a.pinecone.io
For our sparse embedding model we usepinecone-sparse-english-v0, we initialize it like so:
from langchain_pinecone.embeddingsimport PineconeSparseEmbeddings
sparse_embeddings= PineconeSparseEmbeddings(model=model_name)
Now that our Pinecone index and embedding model are both ready, we can initialize our sparse vector store in LangChain:
from langchain_pineconeimport PineconeSparseVectorStore
vector_store= PineconeSparseVectorStore(index=index, embedding=sparse_embeddings)
Manage vector store
Once you have created your vector store, we can interact with it by adding and deleting different items.
Add items to vector store
We can add items to our vector store by using theadd_documents function.
from uuidimport uuid4
from langchain_core.documentsimport Document
documents=[
Document(
page_content="I had chocolate chip pancakes and scrambled eggs for breakfast this morning.",
metadata={"source":"social"},
),
Document(
page_content="The weather forecast for tomorrow is cloudy and overcast, with a high of 62 degrees.",
metadata={"source":"news"},
),
Document(
page_content="Building an exciting new project with LangChain - come check it out!",
metadata={"source":"social"},
),
Document(
page_content="Robbers broke into the city bank and stole $1 million in cash.",
metadata={"source":"news"},
),
Document(
page_content="Wow! That was an amazing movie. I can't wait to see it again.",
metadata={"source":"social"},
),
Document(
page_content="Is the new iPhone worth the price? Read this review to find out.",
metadata={"source":"website"},
),
Document(
page_content="The top 10 soccer players in the world right now.",
metadata={"source":"website"},
),
Document(
page_content="LangGraph is the best framework for building stateful, agentic applications!",
metadata={"source":"social"},
),
Document(
page_content="The stock market is down 500 points today due to fears of a recession.",
metadata={"source":"news"},
),
Document(
page_content="I have a bad feeling I am going to get deleted :(",
metadata={"source":"social"},
),
]
uuids=[str(uuid4())for _inrange(len(documents))]
vector_store.add_documents(documents=documents, ids=uuids)
['95b598af-c3dc-4a8a-bdb7-5d21283e5a86',
'838614a5-5635-4efd-9ac3-5237a37a542b',
'093fd11f-c85b-4c83-83f0-117df64ff442',
'fb3ba32f-f802-410a-ad79-56f7bce938fe',
'75cde9bf-7e91-4f06-8bae-c824dab16a08',
'9de8f769-d604-4e56-b677-ee333cbc8e34',
'f5f4ae97-88e6-4669-bcf7-87072bb08550',
'f9f82811-187c-4b25-85b5-7a42b4da3bff',
'ce45957c-e8fc-41ef-819b-1bd52b6fc815',
'66cacc6f-b8e2-441b-9f7f-468788aad88f']
Delete items from vector store
We can delete records from our vector store using thedelete method, providing it with a list of document IDs to delete.
vector_store.delete(ids=[uuids[-1]])
Query vector store
Once we have loaded our documents into the vector store we're most likely ready to begin querying. There are various method for doing this in LangChain.
First, we'll see how to perform a simple vector search by querying ourvector_store directly via thesimilarity_search method:
results= vector_store.similarity_search("I'm building a new LangChain project!", k=3)
for resin results:
print(f"*{res.page_content} [{res.metadata}]")
* Building an exciting new project with LangChain - come check it out! [{'source': 'social'}]
* Building an exciting new project with LangChain - come check it out! [{'source': 'social'}]
* LangGraph is the best framework for building stateful, agentic applications! [{'source': 'social'}]
We can also addmetadata filtering to our query to limit our search based on various criteria. Let's try a simple filter to limit our search to include only records withsource=="social":
results= vector_store.similarity_search(
"I'm building a new LangChain project!",
k=3,
filter={"source":"social"},
)
for resin results:
print(f"*{res.page_content} [{res.metadata}]")
* Building an exciting new project with LangChain - come check it out! [{'source': 'social'}]
* Building an exciting new project with LangChain - come check it out! [{'source': 'social'}]
* LangGraph is the best framework for building stateful, agentic applications! [{'source': 'social'}]
When comparing these results, we can see that our first query returned a different record from the"website" source. In our latter, filtered, query — this is no longer the case.
Similarity Search and Scores
We can also search while returning the similarity score in a list of(document, score) tuples. Where thedocument is a LangChainDocument object containing our text content and metadata.
results= vector_store.similarity_search_with_score(
"I'm building a new LangChain project!", k=3,filter={"source":"social"}
)
for doc, scorein results:
print(f"[SIM={score:3f}]{doc.page_content} [{doc.metadata}]")
[SIM=12.959961] Building an exciting new project with LangChain - come check it out! [{'source': 'social'}]
[SIM=12.959961] Building an exciting new project with LangChain - come check it out! [{'source': 'social'}]
[SIM=1.942383] LangGraph is the best framework for building stateful, agentic applications! [{'source': 'social'}]
As a Retriever
In our chains and agents we'll often use the vector store as aVectorStoreRetriever. To create that, we use theas_retriever method:
retriever= vector_store.as_retriever(
search_type="similarity_score_threshold",
search_kwargs={"k":3,"score_threshold":0.5},
)
retriever
VectorStoreRetriever(tags=['PineconeSparseVectorStore', 'PineconeSparseEmbeddings'], vectorstore=<langchain_pinecone.vectorstores_sparse.PineconeSparseVectorStore object at 0x7c8087b24290>, search_type='similarity_score_threshold', search_kwargs={'k': 3, 'score_threshold': 0.5})
We can now query our retriever using theinvoke method:
retriever.invoke(
input="I'm building a new LangChain project!",filter={"source":"social"}
)
/usr/local/lib/python3.11/dist-packages/langchain_core/vectorstores/base.py:1082: UserWarning: Relevance scores must be between 0 and 1, got [(Document(id='093fd11f-c85b-4c83-83f0-117df64ff442', metadata={'source': 'social'}, page_content='Building an exciting new project with LangChain - come check it out!'), 6.97998045), (Document(id='54f8f645-9f77-4aab-b9fa-709fd91ae3b3', metadata={'source': 'social'}, page_content='Building an exciting new project with LangChain - come check it out!'), 6.97998045), (Document(id='f9f82811-187c-4b25-85b5-7a42b4da3bff', metadata={'source': 'social'}, page_content='LangGraph is the best framework for building stateful, agentic applications!'), 1.471191405)]
self.vectorstore.similarity_search_with_relevance_scores(
[Document(id='093fd11f-c85b-4c83-83f0-117df64ff442', metadata={'source': 'social'}, page_content='Building an exciting new project with LangChain - come check it out!'),
Document(id='54f8f645-9f77-4aab-b9fa-709fd91ae3b3', metadata={'source': 'social'}, page_content='Building an exciting new project with LangChain - come check it out!'),
Document(id='f9f82811-187c-4b25-85b5-7a42b4da3bff', metadata={'source': 'social'}, page_content='LangGraph is the best framework for building stateful, agentic applications!')]
Usage for retrieval-augmented generation
For guides on how to use this vector store for retrieval-augmented generation (RAG), see the following sections:
API reference
For detailed documentation of all features and configurations head to the API reference:https://python.langchain.com/api_reference/pinecone/vectorstores_sparse/langchain_pinecone.vectorstores_sparse.PineconeSparseVectorStore.html#langchain_pinecone.vectorstores_sparse.PineconeSparseVectorStore
Sparse Embeddings:https://python.langchain.com/api_reference/pinecone/embeddings/langchain_pinecone.embeddings.PineconeSparseEmbeddings.html
Related
- Vector storeconceptual guide
- Vector storehow-to guides