Consistency Is Not Correctness. Test Both Before an AI Workflow Goes Live.

The pilot review is going well. The AI workflow reads incoming claims, pulls out the key details and suggests a decision. Someone on the risk team has done what the vendor never did: run the same fifty cases through it twice.

Every answer came back the same. The room relaxes. One more box ticked.

Then a claims handler with twenty years on the desk reads ten of the suggestions and stops at the fourth. It is wrong. It was wrong the first time, and it was wrong in exactly the same way the second time.

The paradox: The test that should have built confidence only proved the workflow was reliably repeating its mistake.

What We Found

Before we recommend anything about an AI workflow, we study what the public record says about how such systems behave. For this article we went through published research on run-to-run variance in language models and decisions by a court and a tribunal on AI-generated answers, from 2023 to 2025.

Four findings stand out.

Settings meant to make a model repeatable often do not. A 2024 study (revised in 2025) ran five language models, each configured to be deterministic, on eight common tasks, ten times each. The authors saw “accuracy variations up to 15% across naturally occurring runs”, with a gap between best and worst possible performance of up to 70%. Their summary: “none of the LLMs consistently delivers repeatable accuracy across all tasks, much less identical output strings.”

Variance can be engineered away, and that settles only one question. In September 2025 one AI research lab asked a large open model the same question 1,000 times at temperature 0, the setting meant to remove randomness. It got 80 different completions. They were identical for the first 102 tokens and split at the 103rd. With changes to how the computation is batched, all 1,000 completions came out identical. The work was about reproducibility. It made no claim that the repeated answer was correct.

A tool can repeat a wrong answer with confidence. In June 2023 a US federal court in New York, in Mata v. Avianca, sanctioned lawyers who had filed “non-existent judicial opinions with fake quotes and citations created by the artificial intelligence tool ChatGPT”. One of the lawyers had asked the tool whether one of the cases was real. In his own words, quoted in the court’s opinion, “ChatGPT again responded that Varghese ‘does indeed exist’”. The court imposed a penalty of 5,000 US dollars and reminded lawyers of their “gatekeeping role” to ensure the accuracy of their filings.

The business owns the wrong answer, however it was produced. In February 2024 a Canadian tribunal, in Moffatt v. Air Canada, rejected an airline’s argument that its website chatbot was responsible for its own answers. “It makes no difference whether the information comes from a static page or a chatbot,” the tribunal held. The airline was ordered to pay CA$812.02 in total.

What our review does not show: The study measured public models on public tasks, not a firm’s own workflow. The lab’s fix was shown on one model and one question. The two decisions concern a chatbot and a drafting tool, not a system that makes credit, claims or eligibility decisions. We found no public record that measures both the consistency and the accuracy of a production business workflow. We use these as lessons about how AI answers behave, not as evidence of what any particular firm’s system does.

Two Questions That Get Treated as One

Most AI pilots are judged on one number: how often the answer was right on a test set. Our reading of the record is that this leaves out two different questions, and that firms who hear about variance often swap one gap for the other.

The first question is consistency: does the same case get the same answer every time? A pilot scored once on a test set has not answered it. The study above shows the score itself can move from one run to the next.

The second question is correctness: is that answer the one your own experts would give? Making a workflow repeatable does not answer it. A repeatable workflow that is wrong is wrong at scale, and it looks more trustworthy while it does it. The lawyer who asked twice got the same false answer twice.

A business that relies on an AI answer needs both. The tribunal in the airline case did not ask whether the chatbot was consistent. It asked whether the customer was told the truth, and it held the business responsible.

The gap we see: many firms test one of these two things and believe they have tested both. Firms that test both, step by step, can put AI into the parts of a workflow where it has earned that place and keep the rest with code or people. Firms that test one cannot say, when a customer or an auditor asks, why a decision came out the way it did and whether it was right.

This is our reading of the evidence. It is not a measured market figure.

Four Ways to Test a Pilot, Weighed

A firm deciding whether an AI workflow is ready has four real options. Each one is right for somebody.

Option	When it fits	What it risks
Judge it on a test-set accuracy score	Early exploration, where a person checks every output anyway	Says nothing about the next run; the same score may not repeat
Make the model repeatable (fixed settings, controlled infrastructure)	You need to reproduce runs for audit or debugging	Repeats a wrong answer as faithfully as a right one
Keep a person checking every answer	Volumes are low and each decision carries high stakes	Cost grows with volume, and checkers tire of agreeing with a machine
Run two tests, step by step: rerun each case many times for consistency, and compare answers with your own experts for correctness	Answers are acted on without a person checking them first	Needs expert time to build a set of decided cases

Our recommendation: run both tests on every step where someone acts on the answer without a check. A step that passes both can use a model at that level. A step that fails either stays with a person, or its decision moves into rules your team has approved. Prices, approvals, refunds and policy statements stay with code or people whatever the tests say.

Where You Are Now, and Where You Should Be

A recommendation is only useful if you can tell later whether it worked. So before any work begins we set a benchmark: a few measures taken from your own workflow, each with a target.

For this topic, the benchmark would look like this.

Measure	Typical starting point	Where it should be	How it is checked
Consistency of the parts someone acts on	Never measured, or one rerun of a sample	Each decision-bearing step rerun many times on the same cases, and the parts someone acts on compared	A rerun test on the client’s own cases
Correctness	A single accuracy score on a test set	Answers compared with cases your own experts have decided, set beside the consistency result	A set of expert-decided cases
Where answers are acted on without a check	Not listed	Every such point in the workflow is listed and owned	A map of the workflow
What happens when the model or its provider changes	Found out by users	Both tests rerun after any model or prompt change	A record of each re-test
Who makes the commitments (prices, approvals, refunds)	In places, the model	Code or people, with the rule written down	A trace from each commitment to its rule

The starting points above are what we expect to find. In an engagement they are replaced by your own figures, taken in the first review. The targets are what we would work towards with you. They are not results we have already achieved for a client.

How We Work With You

1. We listen: a free Reliability Review. A first conversation about the workflow, how it has been tested, and where its answers are acted on. You leave with a short written note of what we found.
2. We research. We research your sector, the regulator’s position and what the public record says about firms in your situation. This article is an example of that work.
3. We diagnose: the Workflow Diagnostic. It sorts each step of the workflow into compile, keep on an AI model, or don’t automate, and includes a 100-rerun consistency test of your current pilot, a policy-to-rules map, the evidence gaps, the cost per case today against projected, and a change plan. It may recommend another firm or tool, or not automating at all. We take no referral fees on model or platform recommendations.
4. If you approve, we prove it: a Proof Sprint. One workflow, on your own cases, measured for consistency and for accuracy against your own experts. You keep the code.
5. We review against the benchmark. Before work starts we set the benchmark. At each stage we show you where each measure stands, and the filled-in benchmark is yours at the end.

The Review is free. Every paid step is fixed price, agreed at scoping. Stop after any step. The findings are yours. The Diagnostic fee is credited against a Proof Sprint within 90 days.

What changes for the business:

Role	What they gain
CTO or CIO	A go-live decision that rests on two measured results, not on a demo or a single score.
Head of AI or Data	A pilot that can show, step by step, both that it repeats and that it is right, so leadership can fund the next step on evidence.
Risk and compliance	An answer to “why did the system decide this, and was it right?” for every step where an answer is acted on.
COO	Teams know which answers they can act on, and which steps still need a person, before customers meet the workflow.

The cases in this article, and what IOanyT does about that kind of problem:

The case	What IOanyT does
Researchers found accuracy moving between runs of models configured to be deterministic.	The Workflow Diagnostic includes a 100-rerun consistency test of the current pilot, on the client’s own cases.
A lab made a model’s answers identical, with no claim that they were correct.	We measure accuracy against the client’s own experts alongside consistency, never one in place of the other.
In Mata v. Avianca, a tool repeated a false answer when asked again.	We list every point where an answer is acted on without a check, so the client can decide who checks it.
In Moffatt v. Air Canada, the business was held responsible for its chatbot’s answer.	We map policy into rules the client approves, so prices, refunds and policy statements come from those rules, not from the model.

IOanyT is a consultancy with its own engineering team: we study the problem, recommend, and prove it by building only what you approve. We will tell you when a step should not run on a model at all.

Book a free Reliability Review →

Get a fixed quote →

See the Workflow Diagnostic →