The Stalled Claims-Triage Pilot: Which Steps Should Never Have Used a Language Model

The claims-triage pilot was meant to take three months. It is now in its ninth.

The demo was strong. A first notice of loss came in by email, and the model read it, decided whether the policy covered it, scored how serious it was, flagged anything odd and drafted a letter to the customer. One model, one prompt, every step.

Then the questions started. Two near-identical claims got different coverage answers. Nobody could say why one was routed to a senior adjuster and the other was not. Compliance asked for a written description of how the AI was governed. The pilot has been “nearly ready” ever since.

The paradox: The pilot stalled because the model was asked to do everything, including the steps it should never have touched.

The pilot in this article is a composite, written to illustrate how such projects stall. It describes no client and no real insurer.

What We Found

Before we recommend anything about a claims workflow, we study what the public record says about AI in insurance and about how AI answers behave. For this article we went through an insurance regulators’ model bulletin and its adoption record, a study of run-to-run variance in language models, a tribunal decision on an AI answer given to a customer, and analyst predictions about AI projects, from 2023 to 2026.

Four findings stand out.

Insurers are expected to govern their AI in writing. In December 2023 the US National Association of Insurance Commissioners adopted a model bulletin on the use of AI systems by insurers. It expects insurers to keep a written AI systems program covering governance, risk management and internal controls, including oversight of third-party AI systems and data. As of 8 October 2026 it had been adopted by 29 jurisdictions (27 states plus DC and Puerto Rico). It binds insurers where it is adopted; some large states have their own insurance AI rules instead.

A model’s answer to the same claim can change between runs. A 2024 study (revised in 2025) ran five language models, each configured to be deterministic, ten times on the same tasks. The authors saw “accuracy variations up to 15% across naturally occurring runs”. A coverage answer that changes between runs cannot be explained to a policyholder or to a regulator.

The business owns what its AI tells a customer. In February 2024 a Canadian tribunal, in Moffatt v. Air Canada, rejected the argument that a website chatbot was responsible for its own answers. “It makes no difference whether the information comes from a static page or a chatbot,” the tribunal held. A letter drafted by a model and sent to a policyholder is the insurer’s letter.

Many AI projects are predicted to stop before they pay. One analyst firm predicts that over 40% of agentic AI projects will be cancelled by the end of 2027, citing escalating costs, unclear business value or inadequate risk controls.

What our review does not show: We found no public record of a specific claims-triage pilot stalling, and the pilot above is a composite. The bulletin describes what insurers in adopting jurisdictions are expected to have; it does not say which steps may use a model, and this article is not legal advice. The variance study measured public models on public tasks, not claims. The step-by-step verdicts below are our reasoning from this record and from how the workflow behaves.

The Pilot, Taken Apart

Our reading is that the pilot did not fail because language models are bad at claims. It failed because one model was given jobs that belong to code, to rules the insurer approves, and to people. Taken apart, a typical triage workflow has eight steps, and they need different answers.

Step	What it does	Our verdict
1. Read the first notice of loss	Pulls the facts from an email, a form, photos or a call note	A model fits, filling a fixed form that code checks; unclear cases go to a person
2. Find the policy	Is it in force, who holds it, what are the limits and dates	Never a model. A lookup in the policy system
3. Decide coverage	Does the policy wording cover this event	Never a model generating the answer. Rules written from the policy wording, approved by the insurer and tested; ambiguous wording goes to a person
4. Score severity and route	How serious, and which team or adjuster	Approved rules or a tested scoring model, never a fresh generation; the score must repeat
5. Flag possible fraud	Patterns that deserve a closer look	Rules or a tested model may flag; only a person decides
6. Set the first reserve	The money set aside for the claim	Never a model. The insurer’s own reserving rules
7. Write to the customer	Acknowledge, request documents, explain next steps	A model may draft wording; every statement of cover, amount or deadline comes from the rules, and a template or a person sends it
8. Handle exceptions and complaints	Disputes, vulnerable customers, anything unusual	A person, always

Two steps use a model as the core of the work (1 and 7), and both inside limits. Five steps should never have used one to make the decision (2, 3, 4, 5 and 6). One belongs to people (8). The pilot gave all eight to one prompt.

The gap we see: many insurers have a claims-triage pilot that works in the demo and cannot be approved. Insurers that take the workflow apart step by step can ship the steps where a model helps (reading notices and drafting letters) sooner, while coverage, reserves and decisions on people stay where they can be explained. They also have the written description of their AI that the bulletin expects.

This is our reading of the evidence. It is not a measured market figure.

Four Ways to Restart the Pilot, Weighed

An insurer with a stalled pilot has four real options. Each one is right for somebody.

Option	When it fits	What it risks
Keep one model doing everything, with more prompt work	Very low volume, a person checks every output	Coverage answers that vary; no explanation a regulator accepts; the stall continues
Keep one model, and have an adjuster review every claim	High stakes, low volume, and adjusters to spare	Review becomes a rubber stamp; the cost per claim does not fall
Split the workflow: models read and draft, approved rules decide, people own exceptions	Volume is meaningful and the policy wording can be written as rules	Needs the policy wording turned into rules and tested, step by step
Pause the pilot and fix the process or data first	Policy wording is ambiguous, or the data the steps need is not reliable	Delay, which should be stated in the plan rather than discovered

Our recommendation: split the workflow. Use a model where it reads and drafts, inside fixed forms and with a person for unclear cases. Turn coverage, routing and reserving into rules the insurer approves and tests. Keep fraud decisions, exceptions and complaints with people. Where the policy wording itself is ambiguous, settle it with the people who own it before any build.

Where You Are Now, and Where You Should Be

A recommendation is only useful if you can tell later whether it worked. So before any work begins we set a benchmark: a few measures taken from your own workflow, each with a target.

Measure	Typical starting point	Where it should be	How it is checked
Steps decided by a model generating an answer	All of them	Only reading and drafting, inside fixed forms	The step-by-step triage
Coverage answers for the same claim	Can differ between runs	The same every time, traced to the approved rule	A rerun test on the insurer’s own past claims
Coverage answers that are right	Judged on a demo	Compared with claims the insurer’s own adjusters have decided	A set of decided claims
A written description of how the AI is governed	Not written	Written, step by step, with who owns each decision	The insurer’s AI documentation
Statements of cover, amounts and deadlines in letters	Drafted by the model	Taken from the rules; the model only words them	Review of sent letters

The starting points above are what we expect to find. In an engagement they are replaced by your own figures, taken in the first review. The targets are what we would work towards with you. They are not results we have already achieved for a client.

How We Work With You

1. We listen: a free Reliability Review. A first conversation about the pilot, where it stalled and what compliance has asked for. You leave with a short written note of what we found.
2. We research. We research your sector, the regulator’s position and what the public record says about firms in your situation. This article is an example of that work.
3. We diagnose: the Workflow Diagnostic. It sorts each step of the workflow into compile, keep on an AI model, or don’t automate, and includes a 100-rerun consistency test of your current pilot, a policy-to-rules map, the evidence gaps, the cost per case today against projected, and a change plan. It may recommend another firm or tool, or not automating at all. We take no referral fees on model or platform recommendations.
4. If you approve, we prove it: a Proof Sprint. One workflow, on your own cases, measured for consistency and for accuracy against your own experts. You keep the code.
5. We review against the benchmark. Before work starts we set the benchmark. At each stage we show you where each measure stands, and the filled-in benchmark is yours at the end.

The Review is free. Every paid step is fixed price, agreed at scoping. Stop after any step. The findings are yours. The Diagnostic fee is credited against a Proof Sprint within 90 days.

What changes for the business:

Role	What they gain
COO	The steps where a model helps go live sooner, and claims handlers keep the decisions that need them.
CEO	A claims programme that moves again, with an answer ready when the regulator asks how the AI is governed.
CTO or CIO	A workflow built from parts that can each be tested, instead of one prompt nobody can debug.
Head of AI or Data	A pilot unblocked by narrowing it, so the model is judged on the work it is suited to.

The cases in this article, and what IOanyT does about that kind of problem:

The case	What IOanyT does
The NAIC model bulletin expects insurers to keep a written AI systems program.	The Workflow Diagnostic sets out, step by step, what each part of the workflow does, who owns it and what evidence exists, which the insurer can use in its own documentation.
A study found accuracy moving between runs of models configured to be deterministic.	The Diagnostic includes a 100-rerun consistency test of the current pilot, on the insurer’s own claims.
In Moffatt v. Air Canada, the business was held responsible for its chatbot’s answer.	Statements of cover, amounts and deadlines come from rules the insurer approves; a model only words them.
An analyst firm predicts many agentic AI projects will be cancelled over cost, value and risk.	We will recommend narrowing a pilot, or not automating a step, when that is better, and we take no referral fees.

IOanyT is a consultancy with its own engineering team: we study the problem, recommend, and prove it by building only what you approve. A stalled pilot is often closer to live than it looks, once it is asked to do less.

Book a free Reliability Review →

Get a fixed quote →

See the Workflow Diagnostic →