Skip to main content
Contact Kolena to enable this feature for your Organization.
Agent Validation lets you compare your Agent’s output to known correct answers (Ground Truths). Kolena will score how well the Agent matches these Ground Truths. This allows you to:
  • Estimate the overall accuracy of the Agent at a given task
  • Perform change management by testing accuracy before and after changes to your Prompts

​
Overview

Validation has three steps:
  1. Create a Ground Truth for your Agent
  2. Perform Validation (evaluation between Ground Truth and Agent output)
  3. Review results of the Validation

​
Prerequisites

To perform validation on an Agent, you must ensure:
  • The Agent has Prompts created
  • Runs are uploaded to the Agent using inputs for which you have known correct outputs
For example, if you wish to validate a Lease Abstraction Agent:
  • Upload the lease documents you wish to abstract into Runs
  • Define all Prompts necessary to perform the abstraction
  • Ensure you have known answers to compare against for those input documents

​
Creating a Ground Truth

Using the UI, navigate to the “Validation” tab for your Agent. Click “Create New Ground Truth”. Provide a Name for your Ground Truth and click “Continue”.

​
Associate Ground Truth to Runs

In order to compare Agent outputs to Ground Truths, you need a way to associate the two. Kolena uses a label on each Run called user_defined_id to achieve this. Both your ground truths and the Agent Runs must use this label to join the two. Click “Bulk Label Runs” and provide natural language instructions on how to assign this label to each Run on your Agent. Kolena will take your instruction and define the label for each Run. You can check progress and then continue when complete.
For example, if you have a Loan Review Agent, use an identifier like the loan ID as the label. Assign labels to each run using instructions like “Grab the loan ID from the cover sheet”.

​
Upload Ground Truth

There are two ways to upload Ground Truth:
  1. Convert from an existing format like excel spreadsheet
  2. Directly upload in the expected schema
If your correct answers are in a spreadsheet (.xlsx or .csv), use option 1. Kolena will map your spreadsheet columns to the correct Prompt names and generate the Ground Truth JSON for you.

​
Ground Truth Schema

If uploading Ground Truth directly as JSON, the following structure is expected:
Fields:
  • evaluation_instructions — Natural language guidance for how Kolena should compare outputs to ground truths. Useful for specifying acceptable formats, equivalences (e.g. “$4,500” vs 4500), or what counts as a correct partial match.
  • runs — A list of expected results, one per Agent Run.
    • user_defined_id — A stable identifier that matches the ground truth to a Run in the Agent.
    • data — A map from Prompt name to expected value.

​
Run a Validation

  1. Open a Ground Truth from the Validation tab.
  2. Click Run Validation.
  3. Results stream in real-time as each cell is evaluated. When complete, you’ll see:
    • An overall accuracy score (0–100) for the sheet
    • Per-Prompt scores showing which Prompts are most/least accurate
    • Per-Run scores showing which Runs had the most errors
    • Cell-level reasoning explaining why a value was marked incorrect

​
Editing Ground Truth

You can update individual expected values without re-uploading the entire file:
  1. Open a Validation and click on a cell.
  2. Click “Edit Ground Truth” and Edit the expected value in the side pane.
  3. Click Save. Kolena creates a new version of the Ground Truth, preserving history.

​
How Scoring Works

Outputs that match Ground Truth exactly are matched immediately and skip evaluation. For the rest, an AI coding agent inspects your data and writes and runs a script to compare outputs field-by-field, producing deterministic match / no-match results. Your evaluation_instructions guide this script, including how to normalize formats and how to align table rows. A natural-language second pass runs only on the mismatches for fields you explicitly ask to compare by meaning (see When the second pass runs). Field-level results roll up into per-cell, per-Prompt, per-Run, and overall accuracy scores (0–100).

​
When the second pass runs

The natural-language second pass is not automatic. It only re-judges mismatches on fields you explicitly ask to be compared by meaning — using language like “semantically”, “same meaning”, or “paraphrase” in your evaluation_instructions. Everything else (date, currency, number, and boolean equivalence, synonym lists, casing, whitespace) is handled deterministically by the script. If you never request semantic comparison, a script mismatch stays a mismatch.

​
Custom instructions

By default comparison is as literal as possible. Evaluation Instructions can be set to customize comparison logic. Use these to tell Kolena how to handle acceptable variations — for example, equivalent date formats, optional whitespace, or acceptable abbreviations. This prevents the evaluator from penalizing correct-but-differently-formatted outputs.
Evaluation Instructions can be used to provide custom logic per Prompt or to ignore differences in specific outputs.For example,
When comparing percentages, convert to numeric and match to 2 decimal places (treat “50%” and 50 as equal) When comparing currency, convert to numeric and treat equal to the nearest dollar (“$1,000” == 1000) When comparing dates/datetimes, treat as equal if equivalent to the nearest hour (2025-01-08 02:45PM == 2025-01-08 14:45:00) When comparing addresses, ignore casing and abbreviations (“123 Main St” == “123 Main Street”) Treat null and None as equal; treat a null ground truth and a false output as equivalent When comparing the “Summary” field, treat as correct if it conveys the same meaning # triggers the second pass Skip any “Comments” prompts in the comparison
Most rules are applied deterministically by the script. Only rules that ask for meaning-based comparison trigger the natural-language second pass:

​
Using the API

Validation can also be performed programmatically using Kolena’s API.