Evaluation for reliability

Lesson 4 said every step moves up the ladder only after it has proven itself on your data. This lesson is about what “proven” means in practice. None of it is specific to IOanyT; it is the general discipline any team should apply before trusting an AI-assisted workflow.

Step 1: build a golden dataset

A golden dataset is a set of real cases from the workflow, each with an answer your own experts agree is correct.

• Use real cases, not invented ones. Pull them from past tickets, applications, documents or calls, removing personal data where needed.
• Include the hard ones. Ask the people who do the work which cases trip them up. Those matter more than another hundred easy ones.
• Agree the answers. Where experts disagree, settle it before testing. If your own team can’t agree what’s correct, no system can be measured against it, and you’ve found a policy question, not a technology one.

Step 2: measure accuracy and consistency

Run the system over every case and compare its output with the agreed answers. That’s accuracy.

Then run every case again, ideally several times, and check whether the outputs match each other. That’s consistency. A system can score well on accuracy in one run and still give a different answer to the same case on the next. For any decision people rely on, you need both numbers.

A compiled (deterministic) component will show perfect consistency by construction. The interesting measurements are accuracy, and the consistency of any steps that still involve a model at run time.

Step 3: analyse the errors

A single score hides what to fix. Group the mistakes by kind, for example:

• misread input (a date, an amount, a name)
• missing rule (a case the logic never anticipated)
• wrong rule (the logic disagrees with policy)
• ambiguous case (experts would hesitate too)
• refusal or hand-off that should, or shouldn’t, have happened

Each group has a different fix: better input checks, a new rule, a corrected rule, a hand-off to a person, or a policy decision. The biggest group is usually the best next fix.

Step 4: fix, then re-run everything

After every change (a new rule, a different model, a prompt edit, a provider update), re-run the whole golden dataset. This is regression evaluation. Fixes in one place often break behaviour somewhere else, and the only way to know is to look.

Grow the dataset over time. Every time production surfaces a new kind of mistake, add the case. The test set becomes a record of everything the system has learned not to get wrong.

Agree the bar before you test

Decide in advance what result counts as good enough, based on what an error costs in that workflow. A drafting assistant with a reviewer behind it can tolerate far more than a step that sets prices or approves applications. Setting the bar after seeing the results is how weak systems get approved.

Try it yourself: a 20-case starter set

1. Pick one workflow and collect 20 real past cases, including at least 5 that experts call “tricky”.
2. Have two experts record the correct answer for each, independently, then reconcile disagreements.
3. Run your current tool or process over all 20, twice.
4. Count: how many were right, how many changed between runs, and what kinds of mistakes appeared.

You now have a baseline, which is more than most AI pilots ever measure.

What comes next

Reliability is one side of the case for compiled AI; cost is the other. Lesson 6 works through the cost math of compiled versus runtime models.