Skip to main content
Contact Kolena to enable this feature for your Organization.
Agent Validation lets you compare your Agent’s output to known correct answers (Ground Truths). Kolena will score how well the Agent matches these Ground Truths. This allows you to:
  • Estimate the overall accuracy of the Agent at a given task
  • Perform change management by testing accuracy before and after changes to your Prompts

​
Overview

Validation has three steps:
  1. Create a Ground Truth for your Agent
  2. Perform Validation (evaluation between Ground Truth and Agent output)
  3. Review results of the Validation

​
Prerequisites

To perform validation on an Agent, you must ensure:
  • The Agent has Prompts created
  • Runs are uploaded to the Agent using inputs for which you have known correct outputs
For example, if you wish to validate a Lease Abstraction Agent:
  • Upload the lease documents you wish to abstract into Runs
  • Define all Prompts necessary to perform the abstraction
  • Ensure you have known answers to compare against for those input documents

​
Creating a Ground Truth

Using the UI, navigate to the “Validation” tab for your Agent. Click “Create New Ground Truth”. Provide a Name for your Ground Truth and click “Continue”.

​
Associate Ground Truth to Runs

In order to compare Agent outputs to Ground Truths, you need a way to associate the two. Kolena uses a label on each Run called user_defined_id to achieve this. Both your ground truths and the Agent Runs must use this label to join the two. Click “Bulk Label Runs” and provide natural language instructions on how to assign this label to each Run on your Agent. Kolena will take your instruction and define the label for each Run. You can check progress and then continue when complete.
For example, if you have a Loan Review Agent, use an identifier like the loan ID as the label. Assign labels to each run using instructions like “Grab the loan ID from the cover sheet”.

​
Upload Ground Truth

Upload a single file containing your correct answers. Kolena checks whether it already matches the expected schema:
  • If it matches exactly, the file is saved directly as your Ground Truth.
  • Otherwise, Kolena converts it for you, mapping your data to the correct Prompt names and generating the Ground Truth JSON. You can optionally add instructions to guide the conversion (for example, how to associate ground truth data to particular runs).
Supported file types are .xlsx, .xls, .xlsm, .csv, .json, and .txt.

​
Ground Truth Schema

The expected schema is below. A file that already matches this structure is uploaded directly; anything else is converted into it.
Fields:
  • evaluation_instructions — Natural language guidance for how Kolena should compare outputs to ground truths. Useful for specifying acceptable formats, equivalences (e.g. “$4,500” vs 4500), or what counts as a correct partial match.
  • runs — A list of expected results, one per Agent Run.
    • user_defined_id — A stable identifier that matches the ground truth to a Run in the Agent.
    • data — A map from Prompt name to expected value.

​
Run a Validation

  1. Open a Ground Truth from the Validation tab.
  2. Click Run Validation.
  3. Results stream in real-time as each cell is evaluated. When complete, you’ll see:
    • An overall Score (0–100) for the Agent, with its matched, mismatched and skipped field counts
    • A breakdown by field, ranked worst-first, so the weakest fields surface immediately
    • Per-Prompt and per-Run scores showing where the errors concentrate
    • Cell-level reasoning explaining why a value was marked incorrect

​
Editing Ground Truth

You can update individual expected values without re-uploading the entire file:
  1. Open a Validation and click on a cell.
  2. Click “Edit Ground Truth” and Edit the expected value in the side pane.
  3. Click Save. Kolena creates a new version of the Ground Truth, preserving history.

​
How Scoring Works

Outputs that match Ground Truth exactly are matched immediately and skip evaluation. For the rest, an AI coding agent inspects your data and writes and runs a script to compare outputs field-by-field, producing deterministic match / no-match results. Your evaluation_instructions guide this script, including how to normalize formats and how to align table rows. A natural-language second pass runs only on the mismatches for fields you explicitly ask to compare by meaning (see When the second pass runs). Field-level results roll up into per-cell, per-Prompt, per-Run, and overall Scores (0–100).

​
How the Score is calculated

The Score is the share of compared fields that match:
Fields that your Evaluation Instructions exclude are skipped: they count towards neither side, so excluding a field never helps or hurts the Score. A Validation where every field is skipped has no Score at all. Every field carries the same weight, whichever Prompt it belongs to. A Prompt extracting twenty fields therefore moves the Score ten times as much as one extracting two — the Score tracks how much of your data is right, not how many Prompts are doing well. Watch the per-Prompt scores alongside it to see whether one Prompt is responsible for most of the errors. The score returned by the API on a completed Validation is this same number.

​
When the second pass runs

The natural-language second pass is not automatic. It only re-judges mismatches on fields you explicitly ask to be compared by meaning — using language like “semantically”, “same meaning”, or “paraphrase” in your evaluation_instructions. Everything else (date, currency, number, and boolean equivalence, synonym lists, casing, whitespace) is handled deterministically by the script. If you never request semantic comparison, a script mismatch stays a mismatch.

​
Custom instructions

By default comparison is as literal as possible. Evaluation Instructions can be set to customize comparison logic. Use these to tell Kolena how to handle acceptable variations — for example, equivalent date formats, optional whitespace, or acceptable abbreviations. This prevents the evaluator from penalizing correct-but-differently-formatted outputs.
Evaluation Instructions can be used to provide custom logic per Prompt or to ignore differences in specific outputs.For example,
When comparing percentages, convert to numeric and match to 2 decimal places (treat “50%” and 50 as equal) When comparing currency, convert to numeric and treat equal to the nearest dollar (“$1,000” == 1000) When comparing dates/datetimes, treat as equal if equivalent to the nearest hour (2025-01-08 02:45PM == 2025-01-08 14:45:00) When comparing addresses, ignore casing and abbreviations (“123 Main St” == “123 Main Street”) Treat null and None as equal; treat a null ground truth and a false output as equivalent When comparing the “Summary” field, treat as correct if it conveys the same meaning # triggers the second pass Skip any “Comments” prompts in the comparison
Most rules are applied deterministically by the script. Only rules that ask for meaning-based comparison trigger the natural-language second pass:

​
Using the API

Validation can also be performed programmatically using Kolena’s API.