Note: This feature is a Preview offering, subject to the "Pre-GA Offerings Terms"of theGCP Service Specific Terms.Pre-GA products and features may have limited support, andchanges to pre-GA products and features may not be compatible with other pre-GAversions. For more information, see thelaunch stage descriptions.Further, by using this feature, you agree to theGenerative AI Preview terms and conditions("Preview Terms"). For this feature, you can process personal data as outlined in theCloud Data Processing Addendum,subject to applicable restrictions and obligations in the Agreement (as defined in the Preview Terms).

This page describes how to enforce data source access control for search apps inVertex AI Search.

Access control for your data sources in Vertex AI Search limits the datathat users can view in your search app's results. Google uses your identityprovider to identify the end user performing a search and determine if theyhave access to the documents that are returned as results.

For example, say that employees at your company search across Confluencedocuments using your search app. However, you need to make sure they can't viewcontent through the app that they aren't allowed to access. If you have set up aworkforce pool in Google Cloud for your organization's identity provider, thenyou can also specify that workforce pool in Vertex AI Search. Now, if anemployee uses your app, they get search results only for documents that theiraccount already has access to in Confluence.

About data source access control

Turning on access control is a one-time procedure.

Access control is available for Cloud Storage, BigQuery,Google Drive, and all third-party data sources.

To turn on data source access control for Vertex AI Search, you must haveyour organization's identity provider configured in Google Cloud. The followingauthentication frameworks are supported:

• Google Identity:

  • Case 1: If you use Google Identity, then all user identities and usergroups are present and managed through Google Cloud. For more informationabout Google Identity, see theGoogle Identitydocumentation.

  • Case 2: You use a third-party identity provider, and you have syncedidentities with Google Identity. Your end users use Google Identity toauthenticate before accessing Google resources or Google Workspace.

  • Case 3: You use a third-party identity provider, and you have syncedidentities with Google Identity. However, you are still using yourexisting third-party identity provider to perform the authentication. Youhave configured SSO with Google Identity such that your users begin theirsign in using Google Identity and then get directed to your third-partyidentity provider. (You might have already done this sync when setting upother Google Cloud resources or Google Workspace.)

• Third-party identity provider federation: If you use an external identityprovider—for example, Azure AD, Okta, or Ping, but don't want to sync youridentities into Google Cloud Identity, then you must set upworkforce identity federation in Google Cloudbefore you can turn on data source access control for Vertex AI Search.

  If you usethird-party connectors, thegoogle.subject attribute must map to the email address field in theexternal identity provider. The following are examplegoogle.subject andgoogle.groupsattribute mappings for commonly used identity providers:

  • Azure AD with OIDC protocol

    google.subject=assertion.emailgoogle.groups=assertion.groups

  • Azure AD with SAML protocol

    google.subject=assertion.attributes['http://schemas.xmlsoap.org/ws/2005/05/identity/claims/name'][0]google.groups=assertion.attributes['http://schemas.microsoft.com/ws/2008/06/identity/claims/groups']

  • Okta with OIDC protocol

    google.subject=assertion.emailgoogle.groups=assertion.groups

  • Okta with SAML protocol

    google.subject=assertion.subjectgoogle.groups=assertion.attributes['groups']

Limitations

Access control has the following limitations:

• 3000 readers are allowed per document. Each principal counts as a reader,where a principal can be a group or an individual user.
• You can select one identity provider per Vertex AI Search-supportedlocation.
• To set a data source as access-controlled, you must select this setting duringdata store creation. You can't turn this setting on or off for an existingdata store.
• TheData >Documents tab in the console doesn't show data foraccess-controlled data sources because this data should only be visible tousers that have view access.
• To preview UI results for search apps that use third-party accesscontrol, you must log into the federated console or use the web app.SeePreview results for access controlled apps.

Before you begin

This procedure assumes you have set up an identity provider in yourGoogle Cloud project.

• Google Identity: If you use Google Identity, you can proceed to theConnect to your identity provider procedure.
• Third-party identity provider: Make sure you have set up a workforceidentity pool for your third-party identity provider. Verify that you havespecified subject and group attribute mappings when setting up workforce pool.For information about attribute mappings, seeAttributemappings in the IAM documentation. Formore information about workforce identity pools, seeManage workforceidentity pool providers in the IAMdocumentation.

Connect to your identity provider

To specify an identity provider for Vertex AI Search and turn on datasource access control, follow these steps:

1. In the Google Cloud console, go to theAI Applications page.

   AI Applications

2. Go to theSettings >Authentication page.

3. Click the edit icon edit for thelocation that you want to update.

4. Select your identity provider in theAdd identity provider dialog. Ifyou select a third party identity provider, also select the workforce poolthat applies for your data sources.

5. ClickSave changes.

Configure a data source with access control

To apply access control to a data source, use the following steps depending onthe kind of data source you're setting up:

• Third-party data sources: No additional configuration is required when youcreate your app. Skip toPreview results for apps with third-party accesscontrol
• Google Drive: No additional configuration is required when youcreate your app.
• Unstructured data from Cloud Storage
• Structured data from Cloud Storage
• Unstructured data from BigQuery
• Structured data from BigQuery

Unstructured data from Cloud Storage

When setting up a data store for unstructured data fromCloud Storage, you need to also upload ACL metadata and set the datastore as access controlled:

1. When preparing your data, include ACL information in your metadata using theacl_info field. For example:

   {"id":"<your-id>","jsonData":"<JSON string>","content":{"mimeType":"<application/pdf or text/html>","uri":"gs://<your-gcs-bucket>/directory/filename.pdf"},"acl_info":{"readers":[{"principals":[{"group_id":"group_1"},{"user_id":"user_1"}]}]}}

   For more information about unstructured data with metadata, see theUnstructured data section ofPrepare data foringesting.

2. When following the steps for data store creation inCreate a search datastore, you can enable access control by doing thefollowing in either the console or using the API:

   • Console: When creating a data store, selectThis data store containsaccess control information during data store creation.
   • API: When creating data store, include the flag"aclEnabled": "true"in your JSON payload.

3. When following the steps for data import inCreate a search datastore, make sure to do the following:

   • Upload your metadata with ACL information from the same bucket as yourunstructured data
   • If using the API, setGcsSource.dataSchema todocument

Structured data from Cloud Storage

When setting up a data store for structured data fromCloud Storage, you need to also upload ACL metadata and set the datastore as access controlled:

1. When preparing your data, include ACL information in your metadata using theacl_info field. For example:

   {"id":"<your-id>","jsonData":"<JSON string>","acl_info":{"readers":[{"principals":[{"group_id":"group_1"},{"user_id":"user_1"}]}]}}

2. When following the steps for data store creation inCreate a search datastore, you can enable access control by doing thefollowing in either the console or using the API:

   • Console: When creating a data store, selectThis data store containsaccess control information during data store creation.
   • API: When creating data store, include the flag"aclEnabled": "true"in your JSON payload.

3. When following the steps for data import inCreate a search datastore, make sure to do the following:

   • Upload your metadata with ACL information from the same bucket as yourunstructured data
   • If using the API, setGcsSource.dataSchema todocument

Unstructured data from BigQuery

When setting up a data store for unstructured data from BigQuery,you need to set the data store as access controlled and provide ACL metadatausing a predefined schema for Vertex AI Search:

1. When preparing your data, specify the following schema. Don't use a customschema.

   [{"name":"id","mode":"REQUIRED","type":"STRING","fields":[]},{"name":"jsonData","mode":"NULLABLE","type":"STRING","fields":[]},{"name":"content","type":"RECORD","mode":"NULLABLE","fields":[{"name":"mimeType","type":"STRING","mode":"NULLABLE"},{"name":"uri","type":"STRING","mode":"NULLABLE"}]}{"name":"acl_info","type":"RECORD","mode":"NULLABLE","fields":[{"name":"readers","type":"RECORD","mode":"REPEATED","fields":[{"name":"principals","type":"RECORD","mode":"REPEATED","fields":[{"name":"user_id","type":"STRING","mode":"NULLABLE"},{"name":"group_id","type":"STRING","mode":"NULLABLE"}]}]}]}]

2. Include your ACL metadata as a column in your BigQuery table.

3. When following the steps inCreate a search datastore, enable access control in either the console orusing the API:

   • Console: When creating a data store, selectThis data store containsaccess control information during data store creation.
   • API: When creating data store, include the flag"aclEnabled": "true"in your JSON payload.

4. When following the steps for data import inCreate a search datastore, if using the API, setBigQuerySource.dataSchema todocument.

Structured data from BigQuery

When setting up a data store for structured data from BigQuery,you need to set the data store as access controlled and provide ACL metadatausing a predefined schema for Vertex AI Search:

1. When preparing your data, specify the following schema. Don't use a customschema.

   [{"name":"id","mode":"REQUIRED","type":"STRING","fields":[]},{"name":"jsonData","mode":"NULLABLE","type":"STRING","fields":[]},{"name":"acl_info","type":"RECORD","mode":"NULLABLE","fields":[{"name":"readers","type":"RECORD","mode":"REPEATED","fields":[{"name":"principals","type":"RECORD","mode":"REPEATED","fields":[{"name":"user_id","type":"STRING","mode":"NULLABLE"},{"name":"group_id","type":"STRING","mode":"NULLABLE"}]}]}]}]

2. Include your ACL metadata as a column in your BigQuery table.

3. When following the steps inCreate a search datastore, enable access control in either the console orusing the API:

   • Console: When creating a data store, selectThis data store containsaccess control information during data store creation.
   • API: When creating data store, include the flag"aclEnabled": "true"in your JSON payload.

4. When following the steps for data import inCreate a searchdata store, make sure to do the following:

   • If using the console, then when specifying the kind of data you'reuploading, selectJSONL for structured data with metadata
   • If using the API, setBigQuerySource.dataSchema todocument

Preview results for apps with third-party access control

Previewing results in the console for apps with third-party access controlrequires you to sign in with your organization's credentials.

You can preview UI results in two ways:

• Workforce Identity Federation console. Open the Workforce Identity Federationconsole and sign in with your third-party credentials. SeePreview results inthe Workforce Identity Federation console.
• Web app. Turn on and sign into a dedicated web app that is providedby Vertex AI Search. SeeTurn on the web app.

Preview results in the Workforce Identity Federation console

Follow these steps to use the Workforce Identity Federation console to view results:

1. In the Google Cloud console, go to theAI Applications page.

   AI Applications

2. Click the name of the search app whose results you want to preview.

3. Go to thePreview page.

4. ClickPreview with federated identity to go to theWorkforce Identity Federation console.

5. Enter your workforce pool provider and organization's credentials.

6. Preview results for your app on thePreview page that appears.

   For more information about previewing your search results, seeGet search results.

For more information about the Workforce Identity Federation console, seeAbout the console (federated).

Grant search permissions to your users

Caution: Previously, Google recommended granting the Discovery Engine Viewer role. This role can grant broader permissions to features that users might not need to access. The Discovery Engine Viewer role is suitable for initial testing, but Google recommends that you usecustom IAM roles for production usage.

To give your users the ability to search access-controlled data using your app,you need to grant access to users in your domain or workforce pool. Googlerecommends that you grant a custom IAM role to your user group.

• Google Identity: If you use Google Identity, then Google recommends thatyou create aGoogle group thatincludes all employees that need to search. If you're a Google Workspaceadministrator, you can include all users in an organization in a Google groupby following the steps inAdd all your organization's users to agroup.
• Third-party identity provider: If you use an external identity provider,for example Okta or Azure AD, then add everyone in your workforce pool to asingle group.

Google recommends that you create a custom IAM role to grant to your user group,using the following permissions:

• discoveryengine.answers.get
• discoveryengine.servingConfigs.answer
• discoveryengine.servingConfigs.search
• discoveryengine.sessions.get
• discoveryengine.widgetConfigs.get

For more information about permissions for Vertex AI Search resources usingIdentity and Access Management (IAM), seeAccess control with IAM.

For more information about custom roles, see theCustom roles inthe IAM documentation.

Authorize the search widget

If you want to deploy a search widget for an access-controlled app, follow thesesteps:

1. Grant the Discovery Engine Viewer role to users in your domain or workforcepool who need to make search API calls.

2. Generate authorization tokens to pass to your widget:

   • For Google Identity: Generate OAuth 2.0 access tokens.
   • For workforce identity federation: Follow the steps inObtain short-lived tokens for workforce identityfederation to get your token.

3. Follow the steps inAdd a widget with an authorization tokento pass the token to your widget.

Turn on the web app

The web app is a dedicated site generated by Vertex AI Search where youand any other users with sign-in credentials can use your search app.

To provide the search app to users without needing to integrate the searchwidget or the search API on your own application, you can provide the web appURL to your users.

Follow these steps to turn on the web app:

1. In the Google Cloud console, go to theAI Applications page.

   AI Applications

2. Click the name of the search app to create a web app for.

   The search app must be associated with at least one data source with accesscontrol. For more information, seeConfigure a data source with accesscontrol.

3. Go to theIntegration >UI tab.

4. ClickEnable the web app.

5. If you're using workforce identity federation, then select a workforce poolprovider.

6. Click the link to your web app.

7. Enter your workforce pool provider and organization's credentials.

8. Preview results for your app.

9. To configure results for the web app, go toConfigure results for thesearch widget. Any configurations for the widget alsoapply to the web app.

10. Optional: To provide the search app to your users through this dedicated webapp, copy the URL and send it to users who have sign-in credentials.They can bookmark the web app URL and go to it to use your search app.

For more information about getting search results, seeGet search results.