This feature is subject to the "Pre-GA Offerings Terms" in the General Service Terms section of theService Specific Terms. Pre-GA features are available "as is" and might have limited support. For more information, see thelaunch stage descriptions.

Learn how to get started with Gen AI evaluation service using the Google Google Cloud console.

Before you begin

Sign in to your Google Cloud account. If you're new to Google Cloud, create an account to evaluate how our products perform in real-world scenarios. New customers also get $300 in free credits to run, test, and deploy workloads.
In the Google Cloud console, on the project selector page, select or create a Google Cloud project.
Roles required to select or create a project
• Select a project: Selecting a project doesn't require a specific IAM role—you can select any project that you've been granted a role on.
• Create a project: To create a project, you need the Project Creator role (roles/resourcemanager.projectCreator), which contains theresourcemanager.projects.create permission.Learn how to grant roles.
Note: If you don't plan to keep the resources that you create in this procedure, create a project instead of selecting an existing project. After you finish these steps, you can delete the project, removing all resources associated with the project.
Go to project selector
Verify that billing is enabled for your Google Cloud project.
Make sure that you have the following role or roles on the project: Storage Admin
Check for the roles
1. In the Google Cloud console, go to theIAM page.

   Go to IAM
2. Select the project.

3. In thePrincipal column, find all rows that identify you or a group that you're included in. To learn which groups you're included in, contact your administrator.

4. For all rows that specify or include you, check theRole column to see whether the list of roles includes the required roles.
Grant the roles
1. In the Google Cloud console, go to theIAM page.

   Go to IAM
2. Select the project.
3. Clickperson_addGrant access.

4. In theNew principals field, enter your user identifier. This is typically the email address for a Google Account.

5. In theSelect a role list, select a role.
6. To grant additional roles, clickaddAdd another role and add each additional role.
7. ClickSave.

In the Google Cloud console, on the project selector page, select or create a Google Cloud project.

Roles required to select or create a project

• Select a project: Selecting a project doesn't require a specific IAM role—you can select any project that you've been granted a role on.
• Create a project: To create a project, you need the Project Creator role (roles/resourcemanager.projectCreator), which contains theresourcemanager.projects.create permission.Learn how to grant roles.

Note: If you don't plan to keep the resources that you create in this procedure, create a project instead of selecting an existing project. After you finish these steps, you can delete the project, removing all resources associated with the project.

Go to project selector

Verify that billing is enabled for your Google Cloud project.

Make sure that you have the following role or roles on the project: Storage Admin

Check for the roles

1. In the Google Cloud console, go to theIAM page.

   Go to IAM
2. Select the project.

3. In thePrincipal column, find all rows that identify you or a group that you're included in. To learn which groups you're included in, contact your administrator.

4. For all rows that specify or include you, check theRole column to see whether the list of roles includes the required roles.

Grant the roles

1. In the Google Cloud console, go to theIAM page.

   Go to IAM
2. Select the project.
3. Clickperson_addGrant access.

4. In theNew principals field, enter your user identifier. This is typically the email address for a Google Account.

5. In theSelect a role list, select a role.
6. To grant additional roles, clickaddAdd another role and add each additional role.
7. ClickSave.

Evaluate your model

To evaluate your model:

1. In the Google Cloud console, go to the Gen AI Evaluation page.

   Go to Evaluation

2. ClickNew evaluation to open the evaluation page.

3. ForDefine evaluation dataset, select an option:

   • Upload file: ClickUpload to upload a CSV or JSONL file. The dataset should contain either prompts or records to use in a prompt template and optionally model responses, with a maximum of 200 rows.

   • Generate data: Enter aPrompt template to guide the Gen AI evaluation service in generating a dataset. Variables you define in your prompt template are generated and populated in the dataset. For more information, seeUse prompt templates.

     1. Define variables to generate: Specify variables to generate and descriptions of the variable to guide generation. If needed, clickAdd another variable description.

     2. Enter aNumber of samples to generate.

     3. ClickGenerate and preview dataset to display a generated dataset based on your prompt template and variables. To adjust the dataset, you can add more details to the variable descriptions and clickRe-generate.

   • Use model logs: Use the snapshot of prompts and responses from the logged traffic of the selected model. You can only use this option if you have request-response logs enabled on a deployed model in Vertex AI. If you just enabled logging, allow time for sufficient samples to accumulate.

     1. Select theModel and theRegion you want to log traffic from. You must have enabled logging already on your selected model and region.

     2. Enter aSampling count.

     3. (Optional) EnableFilter by prompt template to use only logs that match yourPrompt template. This can be useful if you use your selected models for a variety of use cases and want to evaluate one specific use case.

4. ForDefine model responses to evaluate, select an option:

   • From dataset (only available if you selectedUpload file forDefine evaluation dataset): If you want to use one of the fields in the uploaded dataset as your response, select aResponse column.

   • From model (only available if you selectedUse model logs forDefine evaluation dataset): If you're using model logs as the evaluation dataset, the Gen AI evaluation service uses the model responses from the model logs.

   • Call model: Select a model. The Gen AI evaluation service runs prompts on the selected model and uses the responses for evaluation.

5. (Optional) ForAuto-generated evaluation metrics, you canSpecify custom instructions to guide the rubrics generated from each prompt. For example,Evaluate the dataset on cultural sensitivity to the countries {name}. For more information, seeDefine your evaluation metrics.

6. ForName and output directory, enter the following:

   1. Evaluation name: Enter a name for your evaluation.

   2. Output private data path: Enter the name of a Cloud Storage bucket where you want to store your evaluation, or click Browse to choose the bucket.

7. ClickEvaluate.

View your evaluation results

To view an evaluation result:

1. In the Google Cloud console, go to the Gen AI Evaluation page.

   Go to Evaluation

2. Click the evaluation name.

3. For each prompt in your evaluation dataset, the model's response displays along with the evaluation results.

What's next

• Define your evaluation metrics.