In the Vertex AI Vector Search quickstart, learn how to createan index out of a sample dataset from a fictitious ecommerce clothing site. Forthe purpose of this quickstart, the embeddings have already been created. Thisquickstart is intended to be a way to get started creating and deploying anindex in under 30 minutes.

Prerequisites

This tutorial requires a Google Cloud project that is linked with a billingaccount. To create a new project, seeSet up a project and adevelopment environment. You need tocreate a project and set up your billing account.

Choose the runtime environment

This tutorial can be run on either Colab or Vertex AI Workbench.

• Colab: Open this tutorial inColab
• Vertex AI Workbench: Open this tutorial inVertex AI Workbench. If this is the first time you're using Vertex AI Workbenchin your Google Cloud project, go to the Vertex AI Workbench section of the Google Cloud console and clickEnable to enable the Notebooks API.

To view this notebook in GitHub, seeGitHub.

Cost to complete this quickstart

To complete this tutorial costs roughly a few US dollars. The pricingof the Google Cloud services used in this tutorial are available in thefollowing pages:

• Vertex AI Vector Search
• Cloud Storage
• Vertex AI Workbench

You can also use thepricing calculatorto generate a cost estimate based on your projected usage.

• Install the Vertex AI SDK for Python
• Set environment variables
• Authenticate (Colab only)
• Set IAM permissions
• Enable APIs

Install the Vertex AI SDK for Python

Vertex AI and Cloud Storage APIs can be accessed multiple ways,including REST API and Vertex AI SDK for Python. In this tutorial, the Vertex AI SDK for Pythonis used.

!pipinstall--upgrade--usergoogle-cloud-aiplatform>=1.29.0google-cloud-storage

To use the newly installed packages in this Jupyter runtime, you need to restartthe runtime, as shown in the following code snippet.

# Restart kernel after installs so that your environment can access the new packagesimportIPythonapp=IPython.Application.instance()app.kernel.do_shutdown(True)

Environment variables

Set the environment variables. If asked, replaceyour-project-idwith your project ID and run the cell.

# get project IDPROJECT_ID=!gcloudconfigget-valueprojectPROJECT_ID=PROJECT_ID[0]LOCATION="us-central1"ifPROJECT_ID=="(unset)":print(f"Please set the project ID manually below")

# define project informationifPROJECT_ID=="(unset)":PROJECT_ID="[your-project-id]"# generate a unique id for this sessionfromdatetimeimportdatetimeUID=datetime.now().strftime("%m%d%H%M")

Authentication (Colab only)

If you are running this notebook on Colab, you need to run the followingcell authentication. This step is not required if you are usingVertex AI Workbench as it is pre-authenticated.

importsys# if it's Colab runtime, authenticate the user with Google Cloudif'google.colab'insys.modules:fromgoogle.colabimportauthauth.authenticate_user()

Set IAM permissions

You need to add access permissions to the default service account for using theservices.

1. Go to the IAM page in the Google Cloud console.
2. Look for the principal for default compute service account.It should look like:compute@developer.gserviceaccount.com
3. Click the edit button and grant the default compute service account with thefollowing roles: Vertex AI User and Storage Admin and Service Usage Admin.

Enable APIs

Run the following command to enable APIs for Compute Engine, Vertex AI,and Cloud Storage with this Google Cloud project.

!gcloudservicesenablecompute.googleapis.comaiplatform.googleapis.comstorage.googleapis.com--project{PROJECT_ID}

Prepare the sample data

In this tutorial, we use theTheLook dataset that has a products table with about5,000 rows of synthetic product data for a fictitious ecommerce clothing site.

From this table, we have prepared theproduct-embs.json file.

This file is in JSONL format and each row has ID for the product ID, name forthe product name, and embedding for the embedding of the product name in 768dimensions which was generated previously with Vertex AI embeddings for text.

The text embeddings represent the meaning of the clothing product names. Inthis tutorial, we use Vector Search for completing a semantic search ofthe items. This sample code can be used as a basis for other quick recommendationsystems where you can quickly find "other products similar to this one".

To learn more about how to create the embeddings from the data on a BigQuery tableand store them in a JSON file, seeGetting Started with Text Embeddings + Vertex AI Vector Search.

Prepare the data on Cloud Storage

For building an index with Vertex AI, place the embedding filein a Cloud Storage bucket. The following code completes two tasks:

1. Creates a Cloud Storage bucket.
2. Copies the example file to your Cloud Storage bucket.

BUCKET_URI=f"gs://{PROJECT_ID}-vs-quickstart-{UID}"

!gcloudstoragebucketscreate$BUCKET_URI--location=$LOCATION--project=$PROJECT_ID!gcloudstoragecp"gs://github-repo/data/vs-quickstart/product-embs.json"$BUCKET_URI

For using Vector Search to run queries, you also need to copy theembedding file to local directory:

!gcloudstoragecp"gs://github-repo/data/vs-quickstart/product-embs.json".# for query tests

Build and deploy a Vector Search index

Learn how to create an index, create an index endpoint, and then deploy yourindex to the endpoint.

Create an index

Now it's time to load the embeddings to Vector Search.The APIs are available under theaiplatform package of the SDK.

# init the aiplatform packagefromgoogle.cloudimportaiplatformaiplatform.init(project=PROJECT_ID,location=LOCATION)

Create aMatchingEngineIndexwith itscreate_tree_ah_index function (Matching Engine is the previous name of Vector Search).

# create Indexmy_index=aiplatform.MatchingEngineIndex.create_tree_ah_index(display_name=f"vs-quickstart-index-{UID}",contents_delta_uri=BUCKET_URI,dimensions=768,approximate_neighbors_count=100,)

TheMatchingEngineIndex.create_tree_ah_index() method builds an index.This takes under 10 minutes if the dataset is small, otherwise about 60minutes or more depending on the size of the dataset. You can check status ofthe index creation on the Vector Search Google Cloud console

See Indexes

The parameters for creating index:

• contents_delta_uri: the URI of Cloud Storage directory where you storedthe embedding JSON files
• dimensions: dimension size of each embedding. In this case, it is 768 as youare using the embeddings from the text embeddings API.
• approximate_neighbors_count: how many similar items you want to retrieve intypical cases

To learn more about creating the index and the available parameters, seeCreate and manage your index

Create index endpoint and deploy the index

To use the index, you need to create an index endpoint. It works as a serverinstance accepting query requests for your index.

## create `IndexEndpoint`my_index_endpoint=aiplatform.MatchingEngineIndexEndpoint.create(display_name=f"vs-quickstart-index-endpoint-{UID}",public_endpoint_enabled=True)

With the index endpoint, deploy the index by specifying a unique deployed indexID.

DEPLOYED_INDEX_ID=f"vs_quickstart_deployed_{UID}"

# deploy the Index to the Index Endpointmy_index_endpoint.deploy_index(index=my_index,deployed_index_id=DEPLOYED_INDEX_ID)

If it is the first time deploying this index to an index endpoint, it can takearound 30 minutes to automatically build and initiate the backend. To see thestatus of the index deployment, in the Vertex AI section of theGoogle Cloud console, go to theDeploy and Use section. SelectIndexes.

See Indexes

Run a query with Vector Search

In the following code, it finds an embedding for a specified product name,and finds similar product names with the Vector Search.

Get an embedding to run a query

First, load the embedding JSON file to build adict of product names and embeddings.

importjson# build dicts for product names and embsproduct_names={}product_embs={}withopen('product-embs.json')asf:forlinf.readlines():p=json.loads(l)id=p['id']product_names[id]=p['name']product_embs[id]=p['embedding']

With theproduct_embs dictionary, you can specify a product ID to get an embedding for it.

# Get the embedding for ID 6523 "cloudveil women's excursion short"# You can also try with other IDs such as 12711, 18090, 19536 and 11863query_emb=product_embs['6523']

Run a query

Pass the embedding toEndpoint.find_neighbors() method to find similar product names.

# run queryresponse=my_index_endpoint.find_neighbors(deployed_index_id=DEPLOYED_INDEX_ID,queries=[query_emb],num_neighbors=10)# show the resultsforidx,neighborinenumerate(response[0]):print(f"{neighbor.distance:.2f}{product_names[neighbor.id]}")

Thefind_neighbors() method only takes milliseconds to fetch the similar itemseven when you have billions of items on the index, thanks to theScaNN algorithm.Vector Search also supports autoscaling which can automatically resizethe number of nodes based on the demands of your workloads.

Cleaning up

In case you are using your own Cloud project, not a temporary project on Qwiklabs,make sure to delete all of the indexes, index endpoints, and Cloud Storagebuckets after finishing this tutorial. Otherwise, you might incurunexpected costs from the remaining resources.

If you used Workbench, you might also need to delete the notebooks from the console.

# wait for a confirmationinput("Press Enter to delete Index Endpoint, Index and Cloud Storage bucket:")# delete Index Endpointmy_index_endpoint.undeploy_all()my_index_endpoint.delete(force=True)# delete Indexmy_index.delete()# delete Cloud Storage bucket!gcloudstoragerm{BUCKET_URI}--recursive

Utilities

It can take some time to create or deploy indexes, and in that time you mightlose connection with the Colab runtime. If you lose connection, instead ofcreating or deploying your new index again, you can check the Vector SearchGoogle Cloud console and use the existing ones to continue.

Get an existing index

To get an index object that already exists, replace the followingyour-index-idwith the index ID and run the cell. You can get the index ID by checking theVector Search Google Cloud console. In the Vertex AI sectionof the Google Cloud console, go totheDeploy and Use section. SelectIndexes.

See Indexes

my_index_id="[your-index-id]"my_index=aiplatform.MatchingEngineIndex(my_index_id)

Get an existing index endpoint

To get an index endpoint object that already exists, replace the followingyour-index-endpoint-id with the index endpoint ID and run the cell.You can get the index endpoint by checking the Vector Search Google Cloud console.In the Vertex AI section of the Google Cloud console, go totheDeploy and Use section. SelectIndex Endpoints.

See Index Endpoints

my_index_endpoint_id="[your-index-endpoint-id]"my_index_endpoint=aiplatform.MatchingEngineIndexEndpoint(my_index_endpoint_id)