Evaluate Agent Performance with Airia

Evaluations run your agents against a set of test cases and score every answer with a judge model you choose. Use them to check an agent before you publish a change, compare agents or versions on the same inputs, and track how answers shift between runs.

Before you start

• To create, run, and stop evaluations, you need the Manage permission for Evaluations. The Read permission for Evaluations lets you open evaluations and view their results. See the Permissions Reference.
• Prepare your test cases as a CSV file, or plan to type them in on the form.

Prepare your test cases

Click Download CSV Template on the create form to get a sample file. The file follows these rules:

• The first row is a header row. Airia skips it when it reads your test cases.
• Every other row is one test case with 2 or 3 columns, in this order: the input sent to the agent, the expected output, and an optional context column. The Airia evaluator scores the answer against the input and expected output. The context column is passed to Azure AI Foundry evaluators that use context.
• The input and expected output can't be empty.
• Columns can be separated by commas, semicolons, or tabs. Airia detects the separator from the header row.

When you upload the file, Airia checks every row. If a row breaks a rule, the file is flagged on the form. Click View Error to see each problem by row number (for example, Row 4: Expected 2 or 3 columns but found 1.), then fix the file and upload it again.

You can also click Add test case to type an Input and Expected Output directly, alone or on top of an uploaded file.

Create and run an evaluation

1. In the left navigation, go to Build → Evaluations and click Create Evaluation.
2. Under Details, enter an Evaluation name and, optionally, a Description.
3. Under Agents, keep Agent Source set to Airia Agent, then choose a Project and an Agent and click Add. You can add up to three agents. Each added agent uses its active version; change the version on the agent's row if you want a different one. To compare two versions of the same agent, add it twice and pick a different version on each row.
4. Under Dataset, click Upload CSV (or drop the file onto the form), click Add test case, or both.
5. Under Evaluators, select Airia (Airia Default Evaluator), then pick the Judge model. The list shows each model with its provider and, for models added to a project, the project name. Type in the search box to find a model. A smaller, lower-cost model keeps the cost of large datasets down. A stronger model gives more reliable scores on nuanced answers.
6. Click Create and run evaluation.

The evaluation appears in the Evaluations list and starts running. The list shows each evaluation's Status (Created, Pending, Running, Success, Failed, or Cancelled) and its Execution Status: the number of completed runs out of the total. The total is test cases × agents × evaluators, so 10 test cases against two agents with the Airia evaluator shows 0 / 20 when it starts.

How many runs an evaluation makes, and what it costs

Each test case runs once against each selected agent version, and the judge model scores each of those answers for accuracy and bias (and, on a run started with Run Again, for drift). For example, 200 test cases against two agents make 400 agent runs, and every one of those answers is scored by the judge model.

The cost of an evaluation is the model usage of those agent runs plus the judge model's usage. Once results are in:

• The evaluation's detail page shows Evaluation Cost for the whole run.
• Each agent's card shows its average Latency, Tokens, and Cost per test case, with a separate Evaluator row for the judge model's total usage.
• Each row of the results table shows the Tokens and Cost for that test case.

Stop a running evaluation

Open the evaluation and click Stop. Stop is available while the status is Running. The evaluation stops processing the remaining test cases and keeps the results for the test cases that already ran.

To run the full set again, click Run Again. This starts a new run with the same agent versions, test cases, and evaluators.

Review the results

When the status is Success, open the evaluation. Each agent version has a card with its scores and a View Results link.

The results page has these tabs:

• Accuracy & Bias Results: for each test case, the Input, Expected Output, Actual Output, Latency, Accuracy, Bias, Bias Type, an Evaluation Summary in which the judge explains the score, and the Tokens and Cost of the run. Switch between Condensed view and Expanded view to show full answers.
• Drift Results: appears on a run started with Run Again, and compares it with the run you started it from. It shows the Previous Output and Current Output for each test case with a Drift score, a Drift Type, and an Evaluation Summary.

Click Export Report to download the results for that agent as a CSV file.

On the Accuracy, Bias, and Drift scores, a higher Accuracy is better, and a lower Bias or Drift is better.

How the Accuracy score is calculated

The Accuracy score comes from the judge model. Airia instructs the judge to assess each answer on four dimensions: hallucination (whether claims are grounded), relevance (whether the answer addresses the input), faithfulness (whether the claims are accurate), and completeness (whether every part of the request is covered). The judge then combines them into one score, shown as a percentage, using a weighted harmonic mean, with hallucination weighted most heavily and faithfulness next, so a weak dimension pulls the overall score down. The judge's reasoning for each score appears in the Evaluation Summary column.

Track drift between runs

To see whether an agent version's answers stay consistent, open the evaluation and click Run Again. Airia runs the same agent versions against the same test cases and compares the new answers with the previous run on the Drift Results tab.

Run Again re-tests the versions from the original run, not the agent's latest version. To test a newly published version, create a new evaluation, or add the agent again and pick the new version on its row.

Compare agents

On the evaluation's detail page, click Compare Results to open Compare Agents. It lines up each test case's Input and Expected Output with every agent's answer and its Latency, Tokens, Cost, and Accuracy. Use Agents and Metrics to choose what to show, and switch between Condensed view and Expanded view to show full answers.

To try one prompt against several agents or versions interactively instead, use the Compare Playground in the agent builder, which shows up to five side by side. See Compare versions side by side.

Click View Dataset on the evaluation's detail page to see the test cases the evaluation used.

Score with Azure AI Foundry evaluators

The Evaluators section also offers AzureFoundry Evaluators, which run Azure AI Foundry's own evaluators. To use them, you need an Azure AI Foundry connection with Enable Azure AI Foundry evaluations turned on. Select AzureFoundry Evaluators, choose the Foundry connection, and pick the Foundry evaluators to run alongside or instead of the Airia evaluator. AI-assisted Foundry evaluators also need a Judge model deployment from your Foundry workspace. To set up the connection, see Azure AI Foundry.