This product or feature is subject to the "Pre-GA Offerings Terms" in the General Service Terms section of theService Specific Terms. Pre-GA products and features are available "as is" and might have limited support. For more information, see thelaunch stage descriptions.

For model-based metrics, the Gen AI evaluation service evaluates your models with a foundational model, such as Gemini, that has been configured and prompted as a judge model. If you want to learn more about the judge model, theAdvanced judge model customization series describes additional tools you can use to evaluate and configure the judge model.

For the basic evaluation workflow, see theGen AI evaluation service quickstart. TheAdvanced judge model customization series includes the following pages:

1. Evaluate a judge model (current page)
2. Prompting for judge model customization
3. Configure a judge model

Overview

Using human judges to evaluate large language models (LLMs) can be expensive andtime consuming. Using a judge model is a more scalable way to evaluate LLMs. TheGen AI evaluation service uses a configured Gemini 2.0 Flashmodel by default as the judge model, with customizable prompts to evaluate yourmodel for various use cases.

The following sections show how to evaluate a customized judge model for your ideal use case.

Prepare the dataset

To evaluate model-based metrics, prepare an evaluation dataset with human ratings as the ground truth. The goal is to compare the scores from model-based metrics with human ratings and see if model-based metrics have the ideal quality for your use case.

• ForPointwiseMetric, prepare the{metric_name}/human_rating column in the dataset as the ground truth for the{metric_name}/score result generated by model-based metrics.

• ForPairwiseMetric, prepare the{metric_name}/human_pairwise_choice column in the dataset as the ground truth for the{metric_name}/pairwise_choice result generated by model-based metrics.

Use the following dataset schema:

Available metrics

For aPointwiseMetric that returns only 2 scores (such as 0 and 1), and aPairwiseMetric that only has 2 preference types (Model A or Model B), the following metrics are available:

confusion_matrix = [[20, 31, 15],    [10, 11,  3],   [ 3,  2,  2]]confusion_matrix_labels = ['BASELINE', 'CANDIDATE', 'TIE']

             BASELINE |  CANDIDATE | TIE   BASELINE.    20         31        15   CANDIDATE.   10         11         3   TIE.         3           2         2 |

For aPointwiseMetric that returns more than 2 scores (such as 1 through 5), and aPairwiseMetric that has more than 2 preference types (Model A, Model B, or Tie), the following metrics are available:

Where:

• \( f1 = 2 * precision * recall / (precision + recall) \)

  • \( precision = True Positives / (True Positives + False Positives) \)

  • \( recall = True Positives / (True Positives + False Negatives) \)

• \( n \) : number of classes

• \( cnt_i \) : number of \( class_i \) in ground truth data

• \( sum \): number of elements in ground truth data

To calculate other metrics, you can use open-source libraries.

Evaluate the model-based metric

The following example updates the model-based metric with a custom definition of fluency, then evaluates the quality of the metric.

fromvertexai.preview.evaluationimport{AutoraterConfig,PairwiseMetric,}fromvertexai.preview.evaluation.autorater_utilsimportevaluate_autorater# Step 1: Prepare the evaluation dataset with the human rating data column.human_rated_dataset=pd.DataFrame({"prompt":[PROMPT_1,PROMPT_2],"response":[RESPONSE_1,RESPONSE_2],"baseline_model_response":[BASELINE_MODEL_RESPONSE_1,BASELINE_MODEL_RESPONSE_2],"pairwise_fluency/human_pairwise_choice":["model_A","model_B"]})# Step 2: Get the results from model-based metricpairwise_fluency=PairwiseMetric(metric="pairwise_fluency",metric_prompt_template="please evaluate pairwise fluency...")eval_result=EvalTask(dataset=human_rated_dataset,metrics=[pairwise_fluency],).evaluate()# Step 3: Calibrate model-based metric result and human preferences.# eval_result contains human evaluation result from human_rated_dataset.evaluate_autorater_result=evaluate_autorater(evaluate_autorater_input=eval_result.metrics_table,eval_metrics=[pairwise_fluency])

What's next

• Prompting for judge model customization
• Configure your judge model