Evals

Evals measure how accurately Omni’s AI answers natural-language questions about your data. You define a prompt set (a reusable collection of questions scoped to a single Omni model), then execute an eval run that scores each AI response with a built-in accuracy judge. Because a run targets a specific model branch, evals let you measure the impact of data modeling and curation changes - such as new fields and joins, or ai_context and other context adjustments - before promoting them to your shared model. Use evals to catch regressions before they reach users and to compare results across branches. A run can also target a specific LLM and thinking level, so you can compare LLMs before you release one to your users.

Requirements

At least Querier access on the shared model you want to evaluate
At least one topic optimized for AI — see Optimizing models for Omni AI

Accessing evals

Evals are in the AI Hub, under the Prompt sets and Eval runs tabs.

Prompt sets

A prompt set is a reusable list of up to 25 natural-language prompts that you run against the AI together. You can create as many prompt sets as you need — for example, one per topic, one per release, or one for regression coverage.

Manage prompts

From the Prompt sets tab in the AI Hub, you can:

Create a prompt set and give it a descriptive name
Add or edit prompts in an existing set, up to the 25-prompt limit
Delete a prompt you no longer need
Save changes to persist edits
Archive a prompt set to remove it from the active list without losing it. Archived sets can be restored.

What makes an effective prompt set

Strong prompt sets are representative, broad, and unambiguous:

Mirror real user questions — use the AI usage analytics dashboard to find what users actually ask
Favor breadth over depth — covering several topics with one prompt each yields more signal than ten prompts on the same topic
Phrase prompts naturally — write the way a user would ask, not in language that mirrors field names
Add regression prompts when an AI answer turns out to be wrong, so the same failure is caught next time
Start small and expand toward the 25-prompt cap — begin with 10–15 prompts that cover your core scenarios, then iterate on ai_context and sample_queries and add more prompts as you go

Expected behavior

By default, the accuracy judge decides whether the AI’s response is a correct, reasonable answer to the prompt on its own terms. You can optionally describe the expected behavior for any prompt — a reference answer describing the result you expect — and the judge checks the AI’s analysis against it. Expected behavior is useful when a prompt has a known correct answer you want to pin down, such as a specific value or ranking (“The top product by revenue should be Aniseed Syrup.”), a direction (“Revenue should be up year over year.”), or a breakdown the answer must include. When a prompt defines expected behavior, the judge:

Treats a material divergence as a failing result — wrong numbers, the wrong direction, or a required result that’s missing. Differences in wording, formatting, or extra detail don’t fail the prompt.
Scores the response against its best interpretation, rather than failing the prompt outright, when the expected behavior is vague or only partially addressed.
Restates the expected behavior in its rationale, so opening the chat shows exactly what was expected next to the verdict.

The expected behavior is shown only to the judge — the AI being evaluated never sees it, so it can’t steer toward the expected answer. Each run snapshots the expected behavior in effect when it started, so editing a prompt’s expected behavior later doesn’t change how earlier runs were scored.

Expected behavior is set per prompt through the prompt sets API (POST and PATCH /api/v1/ai/eval/prompt-sets) — pass an optional expectation string of up to 16,000 characters on each prompt. It isn’t yet editable from the Prompt sets tab in the AI Hub.

Running an eval

When you start an eval run, Omni runs each prompt as an asynchronous agentic AI job (the same engine behind the Create AI job API), captures the AI’s response, and then scores it with Omni’s built-in accuracy judge. The judge model is fixed; it isn’t configurable per run. When a prompt defines expected behavior, the judge also scores the response against it.

Select a prompt set

Open the Prompt sets tab in the AI Hub and pick the prompt set you want to evaluate.

Choose a model branch

Select the model branch the AI should query against, or pick main to evaluate the live shared model. Running on a branch lets you measure the impact of in-progress model changes before promotion.

Choose an LLM and thinking level

Optional. By default, every prompt runs on your organization’s default LLM and thinking level. To use different options, select them in the LLM and Thinking dropdowns.The LLM dropdown lists every LLM your organization approved, including LLMs marked as test only. See Evaluating an LLM for more information.

Add a description

Add a short description for the run — for example, “Baseline before topic restructure” or “After adding ai_context to order_items.” Descriptions appear in the run list and the comparison view, so write something you’ll recognize later.

Start the run

Submit the run. Each prompt executes asynchronously as an AI job and is then scored by the judge. You can leave the page — results stream in as prompts complete.

You can have up to two eval runs in progress at once. Wait for a run to finish (or cancel it) before starting a third.

Evaluating an LLM

Use the LLM and Thinking dropdowns to measure an LLM or thinking level before your users run on it:

If you leave both dropdowns unchanged, the run uses the organization default.
If you change either dropdown, every prompt in the run uses the LLM and thinking level you selected.

Eval runs can use LLMs marked as test only. This lets an Organization Admin approve a new LLM as test only, run the same prompt set on the current default and on the new LLM, and then compare the two runs before releasing the new LLM to users. Each run records the LLM and thinking level it ran on. To set the LLM through the Start an eval run API, pass the optional run_config.llm object when you create a run. The LLM must be one that your organization approved or the request will return a 422 error:

{
  "prompt_set_id": "550e8400-e29b-41d4-a716-446655440000",
  "run_config": {
    "llm": {
      "provider": "bedrock",
      "model": "claude-sonnet-5",
      "thinking": "medium"
    }
  }
}

How the accuracy judge works

The accuracy judge is a separate AI model that reviews each completed AI response and returns a pass/fail verdict. It’s tuned to catch the analysis mistakes that most often make an answer wrong, not to grade wording, formatting, or style.

What the judge sees

The judge reads the evaluated AI’s own work for that prompt — the same internal message history the AI produced and saw while answering:

The user prompt
Every tool call the AI made and its result — including the generated queries, the query results, and any data the AI sampled or retrieved
The AI’s final written response

The judge evaluates only what the AI did and saw, including the state of the model and data as it existed when the prompt ran. It doesn’t independently re-run queries or inspect the live semantic model. A verdict therefore reflects the AI’s reasoning against the model the AI actually worked with at the time.

What the judge checks for

The judge’s instructions focus on common, high-impact analysis errors. It looks for problems like:

Problem	Description
Hallucinations	Claims in the written response that the query results don’t actually support. The judge verifies numbers and statements against the underlying query and results rather than trusting the AI’s own summary.
Date and time filtering	Wrong or off-by-one date filters, missing “complete” period handling, and incorrect reasoning about fiscal periods.
Row-limit handling	Treating a result that hit the row limit as if it were complete, such as summing a truncated column and reporting it as a total.
Mental math	Figures the AI computed in its written response instead of in the query, which can be wrong or hallucinated.
Period-over-period errors	Comparison filters that cover the wrong window, like filtering two years of data and then applying a one-year comparison on top.
Ignoring the request	Not honoring an explicitly requested query structure, such as “pivot it by month.”
Query and calculation problems	Using the wrong topic, degenerate results (all zero, null, or identical), or defining a calculated field the query never uses.

When a prompt defines expected behavior, the judge also checks the AI’s findings against it.

How the judge scores

The judge returns a binary verdict for each prompt:

Pass when the response clears every check above
Fail when one or more critical errors are present — a single critical error fails the prompt

If the AI’s response comes back empty — no written answer and no tool call — the prompt is automatically scored as a fail and never reaches the judge. This isn’t the same as a judge failing to score a response, which could be due to an outage or a rate limit issue. In these cases, the prompt is recorded as errored, not failed, and doesn’t count against accurracy. The judge also reports a rationale for each verdict, anchored in evidence from the conversation. Open the chat for any prompt to read both.

The accuracy judge is a single pass/fail correctness check, not a multi-criteria rubric. A passing verdict means the AI avoided the analysis errors above — not that the answer is the single best possible response.

Reviewing results

Once a run completes, open it to see per-prompt and aggregate results. For each prompt, the run shows:

Name	Description
Accuracy	A pass/fail verdict from Omni’s built-in accuracy judge on whether the AI’s response correctly answered the prompt, scored against the prompt’s expected behavior when it’s set
Prompt credits	The credits consumed generating the AI response
Judge credits	The credits consumed scoring the response
Duration	How long the prompt took to complete
Open chat	A link that opens the underlying AI conversation. Beyond the generated query, fields, and answer, this is where you can see the accuracy judge’s full verdict and rationale for that prompt — including how the response compared against the expected behavior, when it was set. This doesn’t apply to prompts where the AI didn’t give a response. In these cases, the prompts fails automatically and never reaches the judge, so there won’t be a rationale to open.

Aggregate metrics for the run include total credits, total duration, and overall accuracy, so you can summarize a run at a glance. Errored results, including judge failures, are left out of accuracy and of credit cost per correct answer, but still count toward total credits. You can archive a completed run to remove it from the active list. Archived runs are preserved and can be restored later.

Comparing two runs

To measure the impact of a model change, run an eval on a branch, then compare it to a baseline run on main. To measure the impact of an LLM change, run the same prompt set on each LLM and compare the runs. Comparison is limited to two runs of the same prompt set, so both runs evaluate the same questions. The comparison view shows side by side, per prompt:

Name	Description
Accuracy	The accuracy judge’s pass/fail verdict on each run, so you can see exactly which prompts regressed or improved. Open the chat for any prompt to read the judge’s rationale behind a verdict.
Credits	The combined prompt + judge credits for each run, so you can spot changes that hurt efficiency
Duration	Run-to-run latency differences

Use the comparison to validate a change before promoting it: if accuracy improves and credits hold steady, the change is a clear win; if accuracy is flat but credits spike, you may want to investigate before merging.

Next steps

Optimize models for Omni AI — Add ai_context, sample queries, and field metadata to improve AI accuracy
Learn from conversation — Let the AI capture business context from your conversations
Model AI settings — Configure AI query scope, validation, and context management
LLM settings — Approve LLMs and set your organization’s default LLM and thinking level
Create AI job API — The endpoint that powers each prompt in an eval run

Getting started

Manage content

Analyze & build

Visualize & present

Deliver content

Model data

Manage integrations

Requirements

Accessing evals

Prompt sets

Manage prompts

What makes an effective prompt set

Expected behavior

Running an eval

Evaluating an LLM

How the accuracy judge works

What the judge sees

What the judge checks for

How the judge scores

Reviewing results

Comparing two runs

Next steps

​Requirements

​Accessing evals

​Prompt sets

​Manage prompts

​What makes an effective prompt set

​Expected behavior

​Running an eval

​Evaluating an LLM

​How the accuracy judge works

​What the judge sees

​What the judge checks for

​How the judge scores

​Reviewing results

​Comparing two runs

​Next steps

Requirements

Accessing evals

Prompt sets

Manage prompts

What makes an effective prompt set

Expected behavior

Running an eval

Evaluating an LLM

How the accuracy judge works

What the judge sees

What the judge checks for

How the judge scores

Reviewing results

Comparing two runs

Next steps