TheVPC-SC security controls and CMEK are supported by Vertex AI RAG Engine. Data residency and AXT security controls aren't supported.

This page shows you how to connect your RAG corpus to yourPineconedatabase.

You can also follow along using this notebookVertex AI RAG Engine with Pinecone.

You can use your Pinecone database instance with Vertex AI RAG Engine to index, andconduct a vector-based similarity search. A similarity search is a way to findpieces of text that are similar to the text that you're looking for, whichrequires the use of anembedding model. The embedding model produces vector datafor each piece of text being compared. The similarity search is used to retrievesemantic contexts forgrounding to return themost accurate content from your LLM.

With Vertex AI RAG Engine, you can continue to use your fully-managed vector databaseinstance, which you're responsible for provisioning. Vertex AI RAG Engine uses yourvector database for storage, index management, and search.

Consider whether to use Pinecone with Vertex AI RAG Engine

Consider whether using the Pinecone database is the best choice for your RAGapplication by reviewing the following:

• You must create, configure, and manage the scaling of your Pinecone databaseinstance.

• Vertex AI RAG Engine uses the default namespace on your index. Ensure that thisnamespace isn't modifiable by anything else.

• You must provide a Pinecone API key, which allows Vertex AI RAG Engine to interactwith the Pinecone database. Vertex AI RAG Engine doesn't store and manage yourPinecone API key. Instead, you must do the following:

  • Store your key in the Google Cloud Secret Manager.
  • Grant your project's service account permissions to access your secret.
  • Provide Vertex AI RAG Engine access to your secret's resource name.
  • When you interact with your RAG corpus, Vertex AI RAG Engine accesses yoursecret resource using your service account.

• RAG corpus and the Pinecone index have a one-to-one mapping. Thisassociation is made as part of theCreateRagCorpus APIcall or theUpdateRagCorpus API call.

Create your Pinecone index

To create your Pinecone index, you must follow these steps:

1. See thePinecone quickstartguide to get theindex configurations that must be specified on your index to make the indexcompatible with RAG corpus.

2. You want to ensure that the location of thePineconeindex is thesame as or close to where you use Vertex AI RAG Engine for the following reasons:

   • You want to maintain reduced latencies.
   • You want to meet your data residency requirements that are set byapplicable laws.

3. During Pinecone index creation, specify the embedding dimension to use withVertex AI RAG Engine. This table provides the dimension sizes or location of thedimension sizes:

   Model	Dimension size
   First-party Gecko	768
   Fine-tuned first-party Gecko	768
   E5	SeeUse OSS embedding models.

4. Choose one of the following supported distance metrics:

   • cosine
   • dotproduct
   • euclidean

5. Optional: When you create a pod-based index, you must specify thefile_idon thepod.metadata_config.indexed field. For more information, seeSelective metadataindexing.

Create your Pinecone API key

Vertex AI RAG Engine can only connect to your Pinecone index by using your API keyfor authentication and authorization. You must follow thePinecone official guideto authentication to configure the API key-based authentication in your Pineconeproject.

Store your API key in Secret Manager

An API key holds Sensitive Personally Identifiable Information (SPII), which issubject to legal requirements. If the SPII data is compromised or misused, anindividual might experience a significant risk or harm. To minimize risks to anindividual while using Vertex AI RAG Engine, don't store and manage your API key, andavoid sharing the unencrypted API key.

To protect SPII, you must do the following:

1. Store your API key inSecret Manager.

2. Grant your Vertex AI RAG Engine service account the permissions to your secret(s),and manage the access control at the secret resource level.

   1. Navigate to yourproject's permissions.

   2. Enable the optionInclude Google-provided role grants.

   3. Find the service account, which has the format:

      service-{project number}@gcp-sa-vertex-rag.iam.gserviceaccount.com

   4. Edit the service account's principals.

   5. Add theSecret Manager Secret Accessor role to the serviceaccount.

3. During the creation or update of the RAG corpus, pass the secret resourcename to Vertex AI RAG Engine, and store the secret resource name.

When making API requests to your Pinecone index(es), Vertex AI RAG Engine uses eachservice account to read the API key that corresponds to your secret resources inSecret Manager from your project(s).

Provision your Vertex AI RAG Engine service account

When you create the first RAG corpus in your project, Vertex AI RAG Engine creates adedicated service account. You can find your service account from your project'sIdentity and Access Management page.

The service account follows this fixed format:

service-{project number}@gcp-sa-vertex-rag.iam.gserviceaccount.com

For example,

service-123456789@gcp-sa-vertex-rag.iam.gserviceaccount.com

Prepare your RAG corpus

To use your Pinecone index with Vertex AI RAG Engine, you must associate the indexwith a RAG corpus during its creation stage. After the association is made, thisbinding is permanent for the lifetime of the RAG corpus. The association can bedone using either theCreateRagCorpus or theUpdateRagCorpus API.

For the association to be considered complete, you must set three key fields onthe RAG corpus:

• rag_vector_db_config.pinecone: This field helps you to set the choice ofa vector database that you would like to associate with your RAG corpus, andit must be set during theCreateRagCorpus API call. If it isn't set, thenthe default vector database choiceRagManagedDb is assigned to your RAGcorpus.

• rag_vector_db_config.pinecone.index_name: This is the name used tocreate the Pinecone index that's used with the RAG corpus. You can set thename during theCreateRagCorpus call, or you can specify the name when youcall theUpdateRagCorpus API.

• rag_vector_db_config.api_auth.api_key_config.api_key_secret_version:This the full resource name of the secret that is stored inSecret Manager, which contains your Pinecone API key. You can set thename during theCreateRagCorpus call, or you can specify the name when youcall theUpdateRagCorpus API. Until you specify this field, you can't importdata into the RAG corpus.
  This field should have the format:
  projects/{PROJECT_NUMBER}/secrets/{SECRET_ID}/versions/{VERSION_ID}

Create your RAG corpus

If you have access to your Pinecone index name and the secret resource name withyour permissions set,then you can create your RAG corpus, and associate it with your Pinecone index,which is demonstrated in this sample code.

When it's your first time creating a RAG corpus, you won't have the serviceaccount information ready. However, the fields are optional and can beassociated with the RAG corpus using theUpdateRagCorpus API.

For an example on how to create the RAG corpus without providing the serviceaccount information, seeCreate RAG corpus without an index name or an APIkey.

fromvertexaiimportragimportvertexai# TODO(developer): Update and un-comment below lines# PROJECT_ID = "your-project-id"# pinecone_index_name = "pinecone-index-name"# pinecone_api_key_secret_manager_version = "projects/{PROJECT_ID}/secrets/{SECRET_NAME}/versions/latest"# display_name = "test_corpus"# description = "Corpus Description"# Initialize Vertex AI API once per sessionvertexai.init(project=PROJECT_ID,location="us-central1")# Configure embedding model (Optional)embedding_model_config=rag.RagEmbeddingModelConfig(vertex_prediction_endpoint=rag.VertexPredictionEndpoint(publisher_model="publishers/google/models/text-embedding-005"))# Configure Vector DBvector_db=rag.Pinecone(index_name=pinecone_index_name,api_key=pinecone_api_key_secret_manager_version,)corpus=rag.create_corpus(display_name=display_name,description=description,backend_config=rag.RagVectorDbConfig(rag_embedding_model_config=embedding_model_config,vector_db=vector_db,),)print(corpus)# Example response:# RagCorpus(name='projects/1234567890/locations/us-central1/ragCorpora/1234567890',# display_name='test_corpus', description='Corpus Description', embedding_model_config=...# ...

# Set your project ID under which you want to create the corpusPROJECT_ID="YOUR_PROJECT_ID"# Choose a display name for your corpusCORPUS_DISPLAY_NAME=YOUR_CORPUS_DISPLAY_NAME# Set your Pinecone index namePINECONE_INDEX_NAME=YOUR_INDEX_NAME# Set the full resource name of your secret. Follows the format# projects/{PROJECT_NUMER}/secrets/{SECRET_ID}/versions/{VERSION_ID}SECRET_RESOURCE_NAME=YOUR_SECRET_RESOURCE_NAME# Call CreateRagCorpus API with all the Vector DB information.# You can also add the embedding model choice or set other RAG corpus parameters on# this call per your choice.curl-XPOST\-H"Authorization: Bearer$(gcloudauthprint-access-token)"\-H"Content-Type: application/json"\https://us-central1-aiplatform.googleapis.com}/v1beta1/projects/${PROJECT_ID}/locations/us-central1/ragCorpora-d'{         "display_name" : '\""${CORPUS_DISPLAY_NAME}"\"',         "rag_vector_db_config" : {            "pinecone": {"index_name": '\""${PINECONE_INDEX_NAME}"\"'},            "api_auth": {"api_key_config":                  {"api_key_secret_version": '\""${SECRET_RESOURCE_NAME}"\"'}            }         }      }'# To poll the status of your RAG corpus creation, get the operation_id returned in# response of your CreateRagCorpus call.OPERATION_ID="YOUR_OPERATION_ID"# Poll Operation status until done = true in the response.# The response to this call will contain the ID for your created RAG corpuscurl-XGET\-H"Authorization: Bearer$(gcloudauthprint-access-token)"\-H"Content-Type: application/json"\https://us-central1-aiplatform.googleapis.com/v1beta1/projects/${PROJECT_ID}/locations/us-central1/operations/${OPERATION_ID}

Create RAG corpus without an index name or an API key

If this is your first RAG corpus and you don't have access to your serviceaccount details, or you haven't completed theprovisioning stepsfor your Pinecone index, you can still create your RAG corpus. You can thenassociate the RAG corpus with an empty Pinecone configuration, and add the details later.

The following must be taken into consideration:

• When you don't provide the index name and API key secret name, files can't beimported into the RAG corpus.
• If you choose Pinecone as your vector database for your RAG corpus, it can'tbe switched later to a different database.

This code example demonstrates how to create a RAG corpus with Pinecone withoutproviding a Pinecone index name or API secret name. Use theUpdateRagCorpusAPI to specify later the missing information.

importvertexaifromvertexai.previewimportrag# Set ProjectPROJECT_ID="YOUR_PROJECT_ID"vertexai.init(project=PROJECT_ID,location="us-central1")# Configure the Pinecone vector DB informationvector_db=rag.Pinecone()# Name your corpusDISPLAY_NAME="YOUR_CORPUS_NAME"rag_corpus=rag.create_corpus(display_name=DISPLAY_NAME,vector_db=vector_db)

# Set your project ID under which you want to create the corpusPROJECT_ID="YOUR_PROJECT_ID"# Choose a display name for your corpusCORPUS_DISPLAY_NAME=YOUR_CORPUS_DISPLAY_NAME# Call CreateRagCorpus API with all the Vector DB information.# You can also add the embedding model choice or set other RAG corpus parameters on# this call per your choice.curl-XPOST\-H"Authorization: Bearer$(gcloudauthprint-access-token)"\-H"Content-Type: application/json"\https://us-central1-aiplatform.googleapis.com}/v1beta1/projects/${PROJECT_ID}/locations/us-central1/ragCorpora-d'{      "display_name" : '\""${CORPUS_DISPLAY_NAME}"\"',      "rag_vector_db_config" : {         "pinecone": {}      }   }'# To poll the status of your RAG corpus creation, get the operation_id returned in# response of your CreateRagCorpus call.OPERATION_ID="YOUR_OPERATION_ID"# Poll Operation status until done = true in the response.# The response to this call will contain the ID for your created RAG corpuscurl-XGET\-H"Authorization: Bearer$(gcloudauthprint-access-token)"\-H"Content-Type: application/json"\https://us-central1-aiplatform.googleapis.com/v1beta1/projects/${PROJECT_ID}/locations/us-central1/operations/${OPERATION_ID}

Update your RAG corpus

TheUpdateRagCorpus API lets you update the vector database configuration.If the Pinecone index name and the API key secret version aren't previously set,you can use the Pinecone API to update the fields. The choice of a vectordatabase can't be updated. It's optional to provide the API key secret. However,if you don't specify the API key secret, you can import data into the RAGcorpus.

importvertexaifromvertexai.previewimportrag# Set ProjectPROJECT_ID="YOUR_PROJECT_ID"vertexai.init(project=PROJECT_ID,location="us-central1")# Configure the Pinecone vector DB informationvector_db=rag.Pinecone(index_name=)# Name your corpusDISPLAY_NAME="YOUR_CORPUS_NAME"rag_corpus=rag.create_corpus(display_name=DISPLAY_NAME,vector_db=vector_db)

# Set your project ID for the corpus that you want to create.PROJECT_ID="YOUR_PROJECT_ID"# Set your Pinecone index namePINECONE_INDEX_NAME=YOUR_INDEX_NAME# Set the full resource name of your secret. Follows the format# projects/{PROJECT_NUMER}/secrets/{SECRET_ID}/versions/{VERSION_ID}SECRET_RESOURCE_NAME=YOUR_SECRET_RESOURCE_NAME# Call UpdateRagCorpus API with the Vector DB information.curl-XPATCH\-H"Authorization: Bearer$(gcloudauthprint-access-token)"\-H"Content-Type: application/json"\https://us-central1-aiplatform.googleapis.com}/v1beta1/projects/${PROJECT_ID}/locations/us-central1/ragCorpora-d'{      "rag_vector_db_config" : {         "pinecone": {"index_name": '\""${PINECONE_INDEX_NAME}"\"'},         "api_auth": {"api_key_config":               {"api_key_secret_version": '\""${SECRET_RESOURCE_NAME}"\"'}         }      }   }'# To poll the status of your RAG corpus creation, get the operation_id returned in# response of your CreateRagCorpus call.OPERATION_ID="YOUR_OPERATION_ID"# Poll Operation status until done = true in the response.# The response to this call will contain the ID for your created RAG corpuscurl-XGET\-H"Authorization: Bearer$(gcloudauthprint-access-token)"\-H"Content-Type: application/json"\https://us-central1-aiplatform.googleapis.com/v1beta1/projects/${PROJECT_ID}/locations/us-central1/operations/${OPERATION_ID}

What's next

• Use Vertex AI Vector Search withVertex AI RAG Engine