This article shows you how to evaluate a chat app's answers against a set of correct or ideal answers (known as ground truth). Whenever you change your chat application in a way that affects the answers, run an evaluation to compare the changes. This demo application offers tools that you can use today to make it easier to run evaluations.

By following the instructions in this article, you:

• Use provided sample prompts tailored to the subject domain. These prompts are already in the repository.
• Generate sample user questions and ground truth answers from your own documents.
• Run evaluations by using a sample prompt with the generated user questions.
• Review analysis of answers.

Architectural overview

Key components of the architecture include:

• Azure-hosted chat app: The chat app runs in Azure App Service.
• Microsoft AI Chat Protocol: The protocol provides standardized API contracts across AI solutions and languages. The chat app conforms to theMicrosoft AI Chat Protocol, which allows the evaluations app to run against any chat app that conforms to the protocol.
• Azure AI Search: The chat app uses Azure AI Search to store the data from your own documents.
• Sample questions generator: The tool can generate many questions for each document along with the ground truth answer. The more questions there are, the longer the evaluations.
• Evaluator: The tool runs sample questions and prompts against the chat app and returns the results.
• Review tool: The tool reviews the results of the evaluations.
• Diff tool: The tool compares the answers between evaluations.

When you deploy this evaluation to Azure, the Azure OpenAI Service endpoint is created for theGPT-4 model with its owncapacity. When you evaluate chat applications, it's important that the evaluator has its own Azure OpenAI resource by usingGPT-4 with its own capacity.

Prerequisites

• Azure subscription.Create one for free

• Deploy a chat app.

  • JavaScript
  • JavaScript with LangChain.js

• These chat apps load the data into the Azure AI Search resource. This resource is required for the evaluations app to work. Don't complete theClean up resources section of the previous procedure.

  You need the following Azure resource information from that deployment, which is referred to as thechat app in this article:

  • Chat API URI: The service backend endpoint shown at the end of theazd up process.
  • Azure AI Search. The following values are required:
    • Resource name: The name of the Azure AI Search resource name, reported asSearch service during theazd up process.
    • Index name: The name of the Azure AI Search index where your documents are stored. You can find the index name in the Azure portal for the Search service.

  The Chat API URL allows the evaluations to make requests through your backend application. The Azure AI Search information allows the evaluation scripts to use the same deployment as your backend, loaded with the documents.

  After you collect this information, you don’t need to use thechat app development environment again. This article refers to thechat app several times to show how theEvaluations app uses it. Don’t delete thechat app resources until you finish all steps in this article.

• Adevelopment container environment is available with all dependencies required to complete this article. You can run the development container in GitHub Codespaces (in a browser) or locally using Visual Studio Code.

  • Codespaces (recommended)
  • Visual Studio Code

  • GitHub account

Open a development environment

Follow these instructions to set up a preconfigured development environment with all the required dependencies to complete this article. Arrange your monitor workspace so that you can see this documentation and the development environment at the same time.

This article was tested with theswitzerlandnorth region for the evaluation deployment.

Prepare environment values and configuration information

Update the environment values and configuration information with the information you gathered duringPrerequisites for the evaluations app.

1. Create a.env file based on.env.sample.

   cp .env.sample .env

2. Run this command to get the required values forAZURE_OPENAI_EVAL_DEPLOYMENT andAZURE_OPENAI_SERVICE from your deployed resource group. Paste those values into the.env file.

   azd env get-value AZURE_OPENAI_EVAL_DEPLOYMENTazd env get-value AZURE_OPENAI_SERVICE

3. Add the following values from the chat app for its Azure AI Search instance to the.env file, which you gathered in thePrerequisites section.

   AZURE_SEARCH_SERVICE="<service-name>"AZURE_SEARCH_INDEX="<index-name>"

Use the Microsoft AI Chat Protocol for configuration information

The chat app and the evaluations app both implement the Microsoft AI Chat Protocol specification, an open-source, cloud, and language-agnostic AI endpoint API contract that's used for consumption and evaluation. When your client and middle-tier endpoints adhere to this API specification, you can consistently consume and run evaluations on your AI backends.

1. Create a new file namedmy_config.json and copy the following content into it:

   {    "testdata_path": "my_input/qa.jsonl",    "results_dir": "my_results/experiment<TIMESTAMP>",    "target_url": "http://localhost:50505/chat",    "target_parameters": {        "overrides": {            "top": 3,            "temperature": 0.3,            "retrieval_mode": "hybrid",            "semantic_ranker": false,            "prompt_template": "<READFILE>my_input/prompt_refined.txt",            "seed": 1        }    }}

   The evaluation script creates themy_results folder.

   Theoverrides object contains any configuration settings that are needed for the application. Each application defines its own set of settings properties.

2. Use the following table to understand the meaning of the settings properties that are sent to the chat app.

   Settings property	Description
   semantic_ranker	Whether to usesemantic ranker, a model that reranks search results based on semantic similarity to the user's query. We disable it for this tutorial to reduce costs.
   retrieval_mode	The retrieval mode to use. The default ishybrid.
   temperature	The temperature setting for the model. The default is0.3.
   top	The number of search results to return. The default is3.
   prompt_template	An override of the prompt used to generate the answer based on the question and search results.
   seed	The seed value for any calls to GPT models. Setting a seed results in more consistent results across evaluations.

3. Change thetarget_url value to the URI value of your chat app, which you gathered in thePrerequisites section. The chat app must conform to the chat protocol. The URI has the following format:https://CHAT-APP-URL/chat. Make sure the protocol and thechat route are part of the URI.

Generate sample data

To evaluate new answers, they must be compared to aground truth answer, which is the ideal answer for a particular question. Generate questions and answers from documents that are stored in Azure AI Search for the chat app.

1. Copy theexample_input folder into a new folder namedmy_input.

2. In a terminal, run the following command to generate the sample data:

   python -m evaltools generate --output=my_input/qa.jsonl --persource=2 --numquestions=14

The question-and-answer pairs are generated and stored inmy_input/qa.jsonl (inJSONL format) as input to the evaluator that's used in the next step. For a production evaluation, you would generate more question-and-answer pairs. More than 200 are generated for this dataset.

Run the first evaluation with a refined prompt

1. Edit themy_config.json configuration file properties.

   Property	New value
   results_dir	my_results/experiment_refined
   prompt_template	<READFILE>my_input/prompt_refined.txt

   The refined prompt is specific about the subject domain.

   If there isn't enough information below, say you don't know. Do not generate answers that don't use the sources below. If asking a clarifying question to the user would help, ask the question.Use clear and concise language and write in a confident yet friendly tone. In your answers, ensure the employee understands how your response connects to the information in the sources and include all citations necessary to help the employee validate the answer provided.For tabular information, return it as an html table. Do not return markdown format. If the question is not in English, answer in the language used in the question.Each source has a name followed by a colon and the actual information. Always include the source name for each fact you use in the response. Use square brackets to reference the source, e.g. [info1.txt]. Don't combine sources, list each source separately, e.g. [info1.txt][info2.pdf].

2. In a terminal, run the following command to run the evaluation:

   python -m evaltools evaluate --config=my_config.json --numquestions=14

   This script created a new experiment folder inmy_results/ with the evaluation. The folder contains the results of the evaluation.

   File name	Description
   config.json	A copy of the configuration file used for the evaluation.
   evaluate_parameters.json	The parameters used for the evaluation. Similar toconfig.json but includes other metadata like time stamp.
   eval_results.jsonl	Each question and answer, along with the GPT metrics for each question-and-answer pair.
   summary.json	The overall results, like the average GPT metrics.

Run the second evaluation with a weak prompt

1. Edit themy_config.json configuration file properties.

   Property	New value
   results_dir	my_results/experiment_weak
   prompt_template	<READFILE>my_input/prompt_weak.txt

   That weak prompt has no context about the subject domain.

   You are a helpful assistant.

2. In a terminal, run the following command to run the evaluation:

   python -m evaltools evaluate --config=my_config.json --numquestions=14

Run the third evaluation with a specific temperature

Use a prompt that allows for more creativity.

1. Edit themy_config.json configuration file properties.

   Existing	Property	New value
   Existing	results_dir	my_results/experiment_ignoresources_temp09
   Existing	prompt_template	<READFILE>my_input/prompt_ignoresources.txt
   New	temperature	0.9

   The defaulttemperature is 0.7. The higher the temperature, the more creative the answers.

   Theignore prompt is short.

   Your job is to answer questions to the best of your ability. You will be given sources but you should IGNORE them. Be creative!

2. The configuration object should look like the following example, except that you replacedresults_dir with your path:

   {    "testdata_path": "my_input/qa.jsonl",    "results_dir": "my_results/prompt_ignoresources_temp09",    "target_url": "https://YOUR-CHAT-APP/chat",    "target_parameters": {        "overrides": {            "temperature": 0.9,            "semantic_ranker": false,            "prompt_template": "<READFILE>my_input/prompt_ignoresources.txt"        }    }}

3. In a terminal, run the following command to run the evaluation:

   python -m evaltools evaluate --config=my_config.json --numquestions=14

Review the evaluation results

You performed three evaluations based on different prompts and app settings. The results are stored in themy_results folder. Review how the results differ based on the settings.

1. Use the review tool to see the results of the evaluations.

   python -m evaltools summary my_results

2. The results looksomething like:

   

   Each value is returned as a number and a percentage.

3. Use the following table to understand the meaning of the values.

   Value	Description
   Groundedness	Checks how well the model's responses are based on factual, verifiable information. A response is considered grounded if it's factually accurate and reflects reality.
   Relevance	Measures how closely the model's responses align with the context or the prompt. A relevant response directly addresses the user's query or statement.
   Coherence	Checks how logically consistent the model's responses are. A coherent response maintains a logical flow and doesn't contradict itself.
   Citation	Indicates if the answer was returned in the format requested in the prompt.
   Length	Measures the length of the response.

4. The results should indicate that all three evaluations had high relevance while theexperiment_ignoresources_temp09 had the lowest relevance.

5. Select the folder to see the configuration for the evaluation.

6. EnterCtrl +C to exit the app and return to the terminal.

Compare the answers

Compare the returned answers from the evaluations.

1. Select two of the evaluations to compare, and then use the same review tool to compare the answers.

   python -m evaltools diff my_results/experiment_refined my_results/experiment_ignoresources_temp09

2. Review the results. Your results might vary.

   

3. EnterCtrl +C to exit the app and return to the terminal.

Suggestions for further evaluations

• Edit the prompts inmy_input to tailor the answers such as subject domain, length, and other factors.
• Edit themy_config.json file to change the parameters such astemperature, andsemantic_ranker and rerun experiments.
• Compare different answers to understand how the prompt and question affect the answer quality.
• Generate a separate set of questions and ground truth answers for each document in the Azure AI Search index. Then rerun the evaluations to see how the answers differ.
• Alter the prompts to indicate shorter or longer answers by adding the requirement to the end of the prompt. An example isPlease answer in about 3 sentences.

Clean up resources and dependencies

The following steps walk you through the process of cleaning up the resources you used.

Clean up Azure resources

The Azure resources created in this article are billed to your Azure subscription. If you don't expect to need these resources in the future, delete them to avoid incurring more charges.

To delete the Azure resources and remove the source code, run the following Azure Developer CLI command:

azd down --purge

Clean up GitHub Codespaces and Visual Studio Code

Return to the chat app article to clean up those resources.

• JavaScript
• Python

Related content

• See theevaluations repository.
• See theenterprise chat app GitHub repository.
• Build achat app with Azure OpenAI best-practices solution architecture.
• Learn aboutaccess control in generative AI apps with Azure AI Search.
• Build anenterprise-ready Azure OpenAI solution with Azure API Management.
• SeeAzure AI Search: Outperforming vector search with hybrid retrieval and ranking capabilities.