The first step to evaluate your generative models or applications is to identify your evaluation goal and define your evaluation metrics. This page provides an overview of concepts related to defining evaluation metrics for your use case.

Overview

Generative AI models can be used to create applications for a wide range of tasks, such as summarizing news articles, responding to customer inquiries, or assisting with code writing. The Gen AI evaluation service in Vertex AI lets you evaluate any model with explainable metrics.

For example, you might be developing an application to summarize articles. To evaluate your application's performance on that specific task, consider the criteria you would like to measure and the metrics that you would use to score them:

• Criteria: Single or multiple dimensions you would like to evaluate upon, such asconciseness,relevance,correctness, orappropriate choice of words.

• Metrics: A single score that measures the model output against criteria.

The Gen AI evaluation service provides two major types of metrics:

• Model-based metrics: Our model-based metrics assess your candidate model against a judge model. The judge model for most use cases is Gemini, but you can also use models such asMetricX orCOMET for translation use cases.

  You can measure model-based metricspairwise or pointwise:

  • Pointwise metrics: Let the judge model assess the candidate model's output based on the evaluation criteria. For example, the score could be 0~5, where 0 means the response does not fit the criteria, while 5 means the response fits the criteria well.

  • Pairwise metrics: Let the judge model compare the responses of two models and pick the better one. This is often used when comparing a candidate model with the baseline model. Pairwise metrics are only supported with Gemini as a judge model.

• Computation-based metrics: These metrics are computed using mathematical formulas to compare the model's output against a ground truth or reference. Commonly used computation-based metrics include ROUGE and BLEU.

You can use computation-based metrics standalone, or together with model-based metrics. Use the following table to decide when to use model-based or computation-based metrics:

To get started, seePrepare your dataset andRun evaluation.

Define your model-based metrics

Model-based evaluation involves using a machine learning model as a judge model to evaluate the outputs of the candidate model.

Proprietary Google judge models, such as Gemini, are calibrated with human raters to ensure their quality. They are managed and available out of the box. The process of model-based evaluation varies based on the evaluation metrics you provide.

Model-based evaluation follows this process:

1. Data preparation: You provide evaluation data in the form of input prompts. The candidate models receive the prompts and generate corresponding responses.

2. Evaluation: The evaluation metrics and generated responses are sent to the judge model. The judge model evaluates each response individually, providing a row-based assessment.

3. Aggregation and explanation: Gen AI evaluation service aggregates these individual assessments into an overall score. The output also includeschain-of-thought explanations for each judgment, outlining the rationale behind the selection.

Gen AI evaluation service offers the following options to set up your model-based metrics with the Vertex AI SDK:

As an example, you might want to develop a generative AI application that returns fluent and entertaining responses. For this application, you can define two criteria for evaluation using the templated interface:

• Fluency: Sentences flow smoothly, avoiding awkward phrasing or run-on sentences. Ideas and sentences connect logically, using transitions effectively where needed.

• Entertainment: Short, amusing text that incorporates emoji, exclamations, and questions to convey quick and spontaneous communication and diversion.

To turn those two criteria into a metric, you want an overall score ranging from -1 ~ 1 calledcustom_text_quality. You can define a metric like this:

# Define a pointwise metric with two criteria: Fluency and Entertaining.custom_text_quality=PointwiseMetric(metric="custom_text_quality",metric_prompt_template=PointwiseMetricPromptTemplate(criteria={"fluency":("Sentences flow smoothly and are easy to read, avoiding awkward"" phrasing or run-on sentences. Ideas and sentences connect"" logically, using transitions effectively where needed."),"entertaining":("Short, amusing text that incorporates emojis, exclamations and"" questions to convey quick and spontaneous communication and"" diversion."),},rating_rubric={"1":"The response performs well on both criteria.","0":"The response is somewhat aligned with both criteria","-1":"The response falls short on both criteria",},),)

For a complete list of metric prompt templates, seeMetric prompt templates for evaluation.

Evaluate translation models

This product or feature is subject to the "Pre-GA Offerings Terms" in the General Service Terms section of theService Specific Terms. Pre-GA products and features are available "as is" and might have limited support. For more information, see thelaunch stage descriptions.

TheGen AI evaluation service offers the following translation task evaluation metrics:

• MetricX

• COMET

• BLEU

MetricX and COMET are pointwise model-based metrics that have been trained for translation tasks. You can evaluate the quality and accuracy of translation model results for your content, whether they are outputs of NMT, TranslationLLM, or Gemini models.

You can also use Gemini as a judge model to evaluate your model for fluency, coherence, verbosity and text quality in combination with MetricX, COMET or BLEU.

• MetricX is an error-based metric developed by Google that predicts a floating point score between 0 and 25 representing the quality of a translation. MetricX is available both as a referenced-based and reference-free (QE) method. When you use this metric, a lower score is a better score, because it means there are fewer errors.

• COMET employs a reference-based regression approach that provides scores ranging from 0 to 1, where 1 signifies a perfect translation.

• BLEU (Bilingual Evaluation Understudy) is acomputation-based metric. The BLEU score indicates how similar the candidate text is to the reference text. A BLEU score value that is closer to one indicates that a translation is closer to the reference text.

Note that BLEU scores are not recommended for comparing across different corpora and languages. For example, an English to German BLEU score of 50 is not comparable to a Japanese to English BLEU score of 50. Many translation experts have shifted to model-based metric approaches, which have higher correlation with human ratings and are more granular in identifying error scenarios.

To learn how to run evaluations for translation models, seeEvaluate a translation model.

Choose between pointwise or pairwise evaluation

Use the following table to decide when you want to use pointwise or pairwise evaluation:

Computation-based metrics

Computation-based metrics compare whether the LLM-generated results areconsistent with a ground-truth dataset of input and output pairs. The commonlyused metrics can be categorized into the following groups:

• Lexicon-based metrics: Use math to calculate the stringsimilarities between LLM-generated results and groundtruth, such asExact Match andROUGE.
• Count-based metrics: Aggregate the number of rows that hit or miss certainground-truth labels, such asF1-score,Accuracy, andTool Name Match.
• Embedding-based metrics: Calculate the distance between the LLM-generatedresults and ground truth in the embedding space, reflecting their level ofsimilarity.

General text generation

The following metrics help you to evaluate the model's ability to ensure theresponses are useful, safe, and effective for your users.

Tool use and function calling

The following metrics help you to evaluate the model's ability to predict avalidtool (function) call.

In the generative AI evaluation service, you canuse computation-based metrics through the Vertex AI SDK for Python.

Baseline evaluation quality for generative tasks

When evaluating the output of generative AI models, note that the evaluation process is inherently subjective, and the quality of evaluation can vary depending on the specific task and evaluation criteria. This subjectivity also applies to human evaluators. For more information about the challenges of achieving consistent evaluation for generative AI models, seeJudging LLM-as-a-Judgewith MT-Bench and Chatbot Arena andLearning to summarize from human feedback.

What's next

• Find amodel-based metrics template.

• Prepare your evaluation dataset.

• Run an evaluation.