"We Documented It" Is Not the Same as Having the Rules. A Benchmark Shows the Difference.

Last year the technology team documented the loan system. It took three months. The result was a thick, careful description of what the application does.

This year the firm began replacing one module, using that document. In testing, penalties came out a few paise different on every account. Then holiday due dates behaved differently. Then a month-end report had no figures in it.

None of those rules had been in the application. One was inside the database. One was in a job that runs at night. One was in a spreadsheet that a finance officer uses on the last day of each month.

What We Found

We looked at where the rules of an old lending system are kept, and at what the public record says about customised systems and the moment a replacement goes live.

A document of the application is not a record of the rules. This is our reasoning, not a finding from a named case. In an old system, one rule is often spread across several places: the application; the database, in small programs called stored procedures; scheduled jobs that run overnight or at month end; tables of codes and rates; and spreadsheets and manual steps outside the system. A document organised by screen describes the first of these well and says nothing of the rest.

A customised system has more connections than anyone counts. When a group of credit unions replaced its heavily customised core system, the work meant rebuilding hundreds of screens and hundreds of interfaces to other systems. An interface sends or receives data, often with no screen at all.

Go-live is when a difference reaches customers. When TSB Bank in the UK moved its customers to a new IT platform in April 2018, it received 225,492 complaints over the following year. The regulators’ finding, in December 2022, was about how it managed its outsourcing arrangement with a critical supplier. The record does not say unwritten rules played a part, and we do not claim they did.

What a Benchmark Opens Up

“We documented it” is usually treated as the end of the job. Our reading is that it is a claim nobody can check until it is turned into a few counted measures.

• The board can see how complete the record is. How many rules have a written line, how many are traced to the code, and how many are waiting for a decision.
• A change can start from the record. A fee change or a new product variant starts from the written rule.
• Figures can be explained. A question about a charge is answered from a page, not a week of searching.

This is our reading. It is not a measured market figure.

Three Ways to Know Your Record Is Complete, Weighed

Option	When it fits	What it risks
Sign the document off once reviewed	The system is small, and its rules sit in the application	Gaps elsewhere stay invisible, and the signature gives unearned confidence
Find the gaps by testing old against new at replacement	A small system, with a replacement already planned	Gaps are found late and all at once, close to go-live
Measure the record against a benchmark, rule by rule, before anything changes	The system is old, customised and poorly documented	Counting takes time, and the first count will show how much is missing

Where You Are Now, and Where You Should Be

We would set this benchmark before the first change, and fill it in as the work goes on.

Measure	Typical starting point	Where it should be	How it is checked
Rules with a written line	A document of the application, organised by screen	Every rule in scope has its own numbered line, in plain words	The specification, confirmed by your own experts
Rules traced to the code	Not traced	Each line points to where the code carries it out	Our senior engineers’ review, line by line
Places outside the application where rules may sit	Not considered	Named in the record, each marked as covered or not yet covered	Your own experts confirm the list
Rules nobody can explain	Carried across unseen, or dropped unseen	Each one listed, with the business’s decision recorded on its line	The decision, recorded against the line
Figures that can be explained	”The system calculated it”	A charge, fee or interest figure can be explained from a written rule	Figures from recent statements, explained from the record
Old functions linked to their replacement	No link is kept	Each one linked and compared before its module goes live	The traceability matrix for each module

The starting points above are what we expect to find. In an engagement they are replaced by your own position, taken in the first review. The targets are what we would work towards with you. They are not results we have already achieved for a lender.

A small part of the finished record might look like this.

Line	The rule, in plain words	Traced to	Decision
14	Penalty starts after a grace period on an overdue instalment	Penalty module, application code	Keep
15	No penalty for borrowers under listed salary schemes	Penalty module and a table of scheme codes	Scheme ended: the business decided to retire it
16	Penalty is rounded before posting	Not yet traced: thought to sit in the database	Not yet covered; your experts to confirm

The rows above are an illustration, not taken from a real system.

How We Work With You, and What You Are Left With

1. We listen: a free Reliability Review. A first conversation about the system, the document you hold, and what it has missed. You leave with a short written note of what we found. You can start with a conversation about the system; we ask for read-only code access only when the assessment begins.
2. We research. We research your sector, the regulator’s position and what the public record says about firms in your situation.
3. We read the system and propose: the Legacy System Specification Assessment. It produces a written specification of what the system does, a risk register, a cost comparison between keeping and replacing the system, and a staged plan with a fixed quote. AI does a first pass, our senior engineers review it and trace it to the code, and your own experts confirm it. Code access is read-only and logged, never used for training, and deleted within 30 days of the evaluation ending unless we engage. Each rule nobody can explain is listed for the business to decide.
4. If you approve, we build. We replace a system one module at a time while the old one keeps running. We never do a single cutover. A traceability matrix links every old function to its replacement. The work is billed by milestone, each milestone tied to a module you accept, and you can pause after any accepted module.
5. We review against the benchmark. Before work starts we set the benchmark above. As the work goes on we show you where each measure stands, and the filled-in benchmark is yours at the end.

The Review is free. Every paid step is fixed price, agreed at scoping. Stop after any step. The findings are yours. The assessment fee is credited in full against the first build stage if it starts within 90 days of the report.

This has a price. The first count will show gaps nobody expected, and rules that are plainly wrong, each needing a decision. What to do about past charges is for your legal and compliance advisers. And the record must be kept current, or the firm drifts back to where it started.

At the end you hold three things: the specification, the matrix for each module replaced, and the benchmark with every line filled in.

What changes for the business:

Role	What they gain
CEO	One document that is the firm’s own account of how its system works.
CTO or CIO	Nothing is hidden in the database or the overnight jobs; a change or replacement starts from the full picture.
CFO	Any future replacement is priced on the whole set of rules, including month-end steps.
COO	The overnight jobs and reports operations relies on are known and will not be lost in a change.
Risk and compliance	A record in which each statement can be followed back to its source.

The cases in this article, and what IOanyT does about that kind of problem:

The case	What IOanyT does
TSB Bank received 225,492 complaints after its new platform went live. The regulators’ finding was about how it managed a critical supplier.	Our part is narrower than that case. When we replace a module, a traceability matrix links each old function to its replacement before that module goes live.
A group of credit unions rebuilt hundreds of screens and hundreds of interfaces when their customised core was replaced.	Our assessment takes the specification from the system itself and traces each rule to the code, so the record can be counted before anyone rebuilds.

IOanyT is a consultancy with its own engineering team: we study the problem, recommend, and prove it by building only what you approve. The next time someone says the system is documented, you can show them how much of it is.

Book a free Reliability Review →

Get a fixed quote →

See the Legacy System Specification Assessment →