This feature is subject to the "Pre-GA Offerings Terms" in the General Service Terms section of theService Specific Terms. Pre-GA features are available "as is" and might have limited support. For more information, see thelaunch stage descriptions.

This guide explains how to import index data from BigQuery intoVector Search with theImportIndex API, streamliningthe process of populating Vector Search indexes directly fromyour BigQuery tables that contain vector embeddings.

Preparing BigQuery data for import

Before importing index data, your BigQuery table must have thefollowing columns:

• Unique identifiers: This column contains unique identifiers for eachdata point. It is mapped to theid field in Vector Search.

• Vector embeddings: This column contains the vector embeddings,represented as a repeatedFLOAT field. It is mapped to the embeddingfield in Vector Search.

Optionally, you can include the following columns:

• Restricts: These are columns for string and numeric restricts, whichlets you filter your data during searches.

• Metadata: These are columns of metadata to be returned withVector Search index query results.

Preparing Vector Search index for import

Once you've prepared your BigQuery data, ensure the destinationVector Search index:

• Exists in Vector Search within your project: This indexserves as the destination for your imported data. The index must be createdwithin your project.

• Is set to overwrite or append data: During the import process, youhave the option to either overwrite the existing data within yourVector Search index or append the data imported fromBigQuery. Overwriting replaces the current data pointswith the imported data. Appending adds the new data to the existingindex.

• Matches dimensionality: The dimensionality of the embeddings stored inyour BigQuery data must be identical to the dimensionalityconfigured for your Vector Search index.

Specifying theImportIndexRequest

Before importing data from BigQuery, create anImportIndexRequest object that specifies the target index, whether tooverwrite existing data, and the configuration for connecting toBigQuery. Send this request object to theImportIndexAPI.

The following is an example of anImportIndexRequest in JSON format:

{"name":"projects/[PROJECT_ID]/locations/[LOCATION]/indexes/[INDEX_ID]","isCompleteOverwrite":true,"config":{"bigQuerySourceConfig":{"tablePath":"bq://[PROJECT_ID].[DATASET_ID].[TABLE_ID]","datapointFieldMapping":{"idColumn":"[ID_COLUMN_NAME]","embeddingColumn":"[EMBEDDING_COLUMN_NAME]","restricts":[{"namespace":"[RESTRICT_NAMESPACE]","allowColumn":["[RESTRICT_ALLOW_COLUMN_NAME]"],"denyColumn":["[RESTRICT_DENY_COLUMN_NAME]"]}],"numericRestricts":[{"namespace":"[RESTRICT_NAMESPACE]","valueColumn":"[RESTRICT_VALUE_COLUMN_NAME]","valueType":"INT"}],"metadataColumns":["METADATA_COLUMN1","METADATA_COLUMN2",...]}}}}

• name: The full resource name of the Vector Search indexwhere you want to import the data.

• isCompleteOverwrite: A boolean that indicates whether to overwrite existingdata in the index. Set totrue to replace existing data.

• config: Contains the configuration for the BigQuery source.

  • bigquerySourceConfig: Specifies the details for connecting to yourBigQuery table.

  • tablePath: The full path to your BigQuery table in theformatbq://[PROJECT_ID].[DATASET_ID].[TABLE_ID].

  • datapointFieldMapping: Maps the columns in yourBigQuery table to the fields in Vector Search.

    • idColumn: The name of the column containing unique identifiers.

    • embeddingColumn: The name of the column containing vectorembeddings.

    • restricts: (Optional) Specifies string restricts.

    • namespace: The namespace for the restrict.

    • allowColumn: The array containing column name(s) for allowedvalues for the restrict.

    • denyColumn: The array containing column name(s) for denied valuesfor the restrict.

    • numericRestricts: (Optional) Specifies numeric restricts.

    • namespace: The namespace for the numeric restrict.

    • value_column: The name of the column containing numeric values.

    • value_type: The type of the numeric value such asINT,FLOAT, orDOUBLE.

    • metadataColumns: (Optional) Metadata fields to include with thefeature embedding. These metadata fields can be retrieved from the indexsearch results, but they don't affect the search itself. For example,filtering cannot be performed on metadata fields.

Executing the import

Once you have created anImportIndexRequest, send it to theImportIndex APIendpoint. This triggers the import process, which exports data fromBigQuery and ingests it into your Vector Searchindex.ImportIndex returns a long-running operation. You can use the operationID to monitor the progress of the import operation.

After the imported data is stored, it resides within yourVector Search index and is indistinguishable from data ingestedusing other methods. The index can continue to be managed using standardVector Search APIs.

The following code sample demonstrates a query result withreturn_full_datapoint set to true and the BigQuery connectorconfiguration that specifies agenre restricts, ayear numeric restricts,and metadata columnstitle anddescription.

nearest_neighbors{neighbors{datapoint{datapoint_id:"4"feature_vector:0.7feature_vector:0.8restricts{namespace:"genre"allow_list:"Drama"}embedding_metadata{title:"A Movie"description:"The story of A Movie..."}crowding_tag{crowding_attribute:"0"}numeric_restricts{namespace:"year"value_int:1942}}distance:0.75}}