Software Releases Are Checked Against the Faults Already Known. Customers Find the Rest.

The release meeting is short. The list of known issues has forty-one lines, and every one is marked closed.

Someone asks whether anything else needs checking. There is a pause. Two people have clicked through the main screens. Nothing looked wrong. The release goes out at six.

The first ticket arrives at nine the next morning. A customer’s weekly export, which has run every Monday for five years, now leaves out one column. It was not on the list. Nobody at the firm knew that a customer depended on it.

What We Found

For this article we went through the public record on how the releases of a software product are tested and checked: one trade publication’s account of a large firm’s rewritten app, one survey of developers who work in PHP, and one developer’s account of changing old code.

Three findings stand out.

A list of known faults can be worked through and still fall short. One large US consumer electronics firm also makes the mobile app that controls its products. By one published account, the existing app ran on outdated code and infrastructure that had been left largely untouched for many years. The rewritten app was launched in 2024 missing many basic features its customers relied on. The firm reportedly had a list of essential faults to deal with before launch, and by the same account the list was not complete. The account does not say what testing was done. It suggests reasons for how the launch went; we did not examine why the app was released as it was, and we draw no cause from it. We take one narrow point: a list of faults already found is not a record of what customers rely on.

Many products have few tests, or none, to check a release against. In one survey of more than a thousand developers who work mainly in PHP, a widely used web language, about a third said they still write no tests at all. That is about developers in general, not about software firms alone. Where there are no tests, the check before a release is whatever people have time to click.

Without tests, change itself becomes the thing to fear. One developer, writing about years of work on old code, put it plainly: nothing is more frightening than changing a critical old system that has no tests. It is one person’s account. It names something many product teams will know: the product still sells, so it must keep changing, and each change is made without a way to see what it disturbs.

Our reading is that releases do not miss what customers rely on because engineers are careless. They miss it because what the product does today has never been written down or recorded as tests, so the only thing a release can be checked against is the list of faults found last time.

What Waits While Customers Do the Testing

Weak testing is usually discussed as an engineering matter. Our reading is that it is first a matter of how often, and how calmly, a firm can change what it sells.

The first cost is that customers become the last check. What a release missed is found on live data, by someone who pays for the product, and reported in a ticket.

The second cost is pace. A team that cannot see what a change disturbs releases less often, bundles more into each release, and so makes each one riskier than the last.

The third is that nothing improves. Each fault a customer finds becomes one more line on next month’s list, and the list still says nothing about what the product is meant to go on doing.

This is our reading of the evidence. It is not a measured market figure.

Four Ways to Check a Release, Weighed

There are four real options. Each one is right for somebody.

Option	When it fits	What it risks
More checking by hand before each release	The product is small, and the people checking know every customer’s habits	It reaches only the screens people remember, it costs the same every time, and the clock decides when it stops
Write tests for the whole product before changing anything	The firm can pause new work for a long time	Months of effort before the first release is safer, much of it on parts that may never change
Record what the product does today as tests around the parts that change, run them in an automated release that makes the same checks every time, and write and rehearse the way back	The product is large and old, sells every day, and is changed every week	It takes effort before releases get calmer, and the tests and release process must be kept up by the firm’s own team afterwards
Rebuild the product in stages, with each old function linked to its replacement	The product must be replaced anyway, for other reasons	It is a much larger decision than the testing problem alone calls for

Where You Are Now, and Where You Should Be

A recommendation is only useful if you can tell later whether it worked. So before any work begins we set a benchmark: a few measures taken from your own product, each with a target.

For this issue, the benchmark would look like this.

Measure	Typical starting point	Where it should be	How it is checked
What the product does, written down	Known to the engineers and support staff who have been there longest	Every function in plain words, confirmed by your own people	The specification
Key functions covered by tests	Few or none; a list of known faults instead	Each function customers rely on covered, starting with the parts releases change	The test run, shown against the specification
How a release is judged ready	”The known issues are closed”	The tests pass, and any difference is explained before release	The record of each release
Releases run the same way every time	Steps done from memory, by whoever is releasing	The same scripted steps and checks on every release	The release process and its record
Faults first found by customers after a release	Not counted	Counted, and falling release by release	Your own support records

The starting points above are what the public record suggests and what we expect to find. In an engagement they are replaced by your own figures, taken in the first review. The targets are what we would work towards with you. They are not results we have already achieved for a software firm.

How We Work With You

1. We listen: a free Reliability Review. A first conversation about your product, how a release is checked today and what customers found after the last one. You leave with a short written note of what we found. You can start with a conversation about the system; we ask for read-only code access only when the assessment begins.
2. We research. We research your sector, the regulator’s position and what the public record says about firms in your situation. This article is an example of that work.
3. We read the system and propose: the Legacy System Specification Assessment. Here the system is your own product’s code and the way it is released today, and the assessment records what the product does now and which parts your releases change most. It produces a map of the system as it stands, a written specification of what it does, a risk register, a cost comparison between keeping and replacing the system, and a staged plan with a fixed quote. AI does a first pass, our senior engineers review it and trace it to the code, and your own experts confirm it. Code access is read-only and logged, never used for training, and deleted within 30 days of the evaluation ending unless we engage. You get the options and our recommendation.
4. If you approve, we build. Here that means tests that record what the product does today around the parts that change, and an automated release that runs them and the same checks every time, with a written way back we rehearse with your team, all handed back to your own engineers. Where a module has to be replaced, we replace a system one module at a time while the old one keeps running. We never do a single cutover. Each rebuilt module has a traceability matrix. The work is billed by milestone, each milestone tied to a module you accept, and you can pause after any accepted module.
5. We review against the benchmark. Before work starts we set the benchmark. After each stage we show you where each measure stands, and the filled-in benchmark is yours at the end.

The Review is free. Every paid step is fixed price, agreed at scoping. Stop after any step. The findings are yours. The assessment fee is credited in full against the first build stage if it starts within 90 days of the report.

This approach has limits. We do not run, start or monitor your releases, and we are not on call when one goes out. We do not test by hand as a service, and we do not supply staff. We do not back up, restore or recover data, and a way back for code is not a way back for data a release has changed or deleted. We do not build mobile apps, which is what the case above involved, and we do not run load tests. We do not maintain the tests or the release process afterwards; your own team does. Tests show that the product behaves as it did; they do not show that it was right, and they cover only the cases that were recorded. No test or check means a release cannot fail. The older technologies we read include VB6, Delphi, AngularJS, PHP 4 to 8, Java 8, .NET Framework and classic ASP. We have no published case of this work for a software firm.

What changes for the business:

Role	What they gain
CEO	Customers stop being the ones who find what a release missed.
CTO or CIO	”Ready” means the product still does what customers rely on, and not only that the known faults are closed.
Head of product	A feature customers use cannot quietly disappear in a release, because its behaviour is one of the things checked.
Head of operations	Support hears of a fault from the release check, and not from the ticket queue the morning after.
CISO	The behaviour the firm cares most about, such as who can see which customer’s records, can be among the things recorded as tests, so a change to a recorded case shows before a release goes out. This is not a test of security.

The cases in this article, and what IOanyT does about that kind of problem:

The case	What IOanyT does
A large US consumer electronics firm launched a rewritten app in 2024 missing many basic features, and its list of essential faults was, by one account, not complete.	Our part is narrower. We do not build mobile apps, and we make no comment on that release or say it would have gone differently. For a product in a technology we read, we record what it does today as tests and run them in an automated release, so a release is checked against what the product did and not only against a list of faults.

IOanyT is a consultancy with its own engineering team: we study the problem, recommend, and prove it by building only what you approve. We start by recording what your product does today, because that is the list every release has been missing.

Book a free Reliability Review →

Get a fixed quote →

See the Legacy System Specification Assessment →