When a Release Goes Out Thinly Checked

1. At a Glance

Issue study · From the public record. None of the organisations described here was an IOanyT client, and IOanyT had no part in the events described. Everything about them comes from the public record.

Organisation type	Software firms whose product changes every week and reaches every customer at once: two large firms as the record describes them, set beside surveys of developers and IT professionals
Size band	The firms in the record: both large. Our reader: a small or mid-size software firm whose product has grown for years
Region	The two firms: not stated here. The surveys: one of developers who work mainly in PHP (region not stated here), one of IT professionals in the US
The decision at stake	How to test, check and, when needed, undo a release, so the firm can change its product often without fear
The public outcome	One firm’s rewritten app was launched in 2024 missing many basic features its customers relied on. At the other, a script deleted whole customer sites and several hundred customers were affected by the outage
Why it matters to a small or mid-size software firm	Customers find what a release missed, and a firm that cannot check or undo a release changes its product less often than its customers need

2. The Business Problem, in the Buyer’s Words

“We release every week, and every week I hold my breath. Our checks are the list of known issues and two people clicking through the main screens. Last month a customer found a missing column in an export before we did. The month before, a change two engineers had reviewed switched off a feature for the wrong customers. When something does go wrong, someone says ‘roll it back’, and nobody on the call has ever done it. I want to release more often, not less. At the moment I cannot see how.”

These words are imagined. They describe the decision as the research shows it, not a real conversation.

The firm is caught between a product that must keep changing and releases it cannot check or undo with confidence. Each bad release makes the next one more frightening, and changes queue up behind the fear.

3. What We Found

Method. We went through the public record on how the releases of a software product are tested, checked and undone: one trade publication’s account of a large firm’s rewritten app, one large software firm’s own report of an outage, one survey of developers who work in PHP, one developer’s account of changing old code, and one survey of IT professionals on why old systems stay.

A rewritten app, launched without basic features. From one trade publication’s account, 2024.

What	Public record
Organisation	A large US consumer electronics firm, which also makes the mobile app that controls its products
The old app	It ran on outdated code and infrastructure that had been left largely untouched for many years
The release	The rewritten app was launched in 2024 missing many basic features its customers relied on
What the account says of the fault list	The firm reportedly had a list of essential faults to deal with before launch; by the same account the list was not complete
What testing was done, and why the app went out as it did	The account does not say what testing was done. It suggests reasons for how the launch went; we did not examine why the app was released as it was, and we draw no cause from it

This is a mobile app, which IOanyT does not build. The case is used for one narrow point: a list of faults already found can be worked through and still miss what customers rely on.

A reviewed script, and customer sites deleted. An older case, from the firm’s own post-incident review, 2022.

What	Public record
Organisation	A large software company
When	2022
What was being done	A script was run to retire an old app that the firm had acquired and integrated
What the review says of the checks	The script had passed the firm’s standard peer review, which looked at which endpoint was called and how, but did not cross-check the site IDs it was given. The IDs given were those of whole sites, not of the app
What happened	Whole customer cloud sites were deleted. Several hundred customers were affected by the outage, which lasted up to two weeks for some of them
What the review says of recovery	The firm could recover a single site, but had not built the means to recover a large batch of sites

Two narrow points are taken from it: a change can pass review and still act on the wrong thing, and the means of getting back were not in place for the case that arrived. Recovery of deleted data is not something IOanyT offers. What the firm built afterwards is its own remedy and is not described here.

What developers and IT professionals report. In one survey of more than a thousand developers who work mainly in PHP, a widely used web language, about a third said they still write no tests at all. One developer wrote that nothing is more frightening than changing a critical old system that has no tests. In one survey of about five hundred IT professionals in the US, run by a firm that sells software services and covering every sector, more than a third named fear of disrupting operations among their reasons for not modernising old systems. None of these is about software firms alone.

Our reading of why releases go out thinly checked. These four are our reasoning from the record above. None is a finding of any source, and neither named case is said to have come about for these reasons.

1. Nobody has recorded what the product does. So a release can be checked only against the faults already found.
2. Release checks are made by people, from memory. Which checks happen depends on who is releasing and how late it is.
3. Review looks at the code and not at what the code will act on. The real inputs arrive last, after the checking.
4. Going back has never been written or tried. So the first attempt is made in an emergency, and what cannot be undone is discovered then.

4. The Options a Firm in This Position Had

Option	When it fits	What it risks	Our view
More checking by hand, and another approval	Releases are rare and small, and the same careful people always do them	It reaches only the screens people remember, and nothing looks at what a change will act on	Right only for a small product with few releases
Write tests for the whole product before changing anything	The firm can pause new work for a long time	Months before the first calmer release, much of it on parts that may never change	Right for a small product; for a large one, start with the parts that change
Record what the product does today as tests around the parts that change; run them in an automated release that makes the same checks every time, including what a change will act on; write and rehearse the way back, stating what it cannot undo	The product is large and old, sells every day, and is changed every week	Effort before releases get calmer; a check catches only what someone thought to check; the firm’s own team must keep it up	Our recommendation
Rebuild the product in stages, with a traceability matrix linking each old function to its replacement	The product must be replaced anyway, for other reasons	A far larger decision than the release problem calls for	Right when replacement is already decided; not as a cure for weak releases

5. Our Recommendation, and the Decision It Asks of the Board

Record what the product does today as tests, starting with the parts releases change, the results customers rely on and the cases support staff hear about. Put the release into an automated process that runs those tests and the same checks, in the same order, every time, including a check of what a change is about to act on, and a pause for a named person where the firm decides one is needed. Keep a record of what went out, what was checked and who approved it. Keep the previous version, write the step that returns to it, rehearse that step on a copy, and state for each release what a return would not undo. Hand it all to the firm’s own team, who run and keep it from then on.

6. Benchmark: Where You Are Now, and Where You Should Be

These are targets, not results. In an engagement the starting points are replaced by the firm’s own figures, taken in the first review.

Measure	Typical starting point	Where it should be	How it is checked
Key functions covered by tests	Few or none; a list of known faults instead	Each function customers rely on covered, starting with the parts releases change	The test run, shown against the specification
Releases run the same way every time	Steps done from memory, by whoever is releasing	The same scripted steps and checks on every release	The release process and its record
Checks of what a change will act on	None; the code is reviewed, its inputs are not	Every change that acts on customers or records shows what it will act on before it runs	The release record
The way back written and rehearsed	Never tried	Written as a step, rehearsed on a copy, with what it cannot undo stated for each release	The rehearsal record
Who approved what, on record	In a chat thread and in people’s memories	For every release: what went out, what was checked and who approved it	The release record
Faults first found by customers after a release	Not counted	Counted, and falling release by release	The firm’s own support records

7. How IOanyT Would Work

1. We listen: a free Reliability Review. A first conversation about your product, how a release is checked today and what happened the last time one went wrong. You leave with a short written note of what we found. You can start with a conversation about the system; we ask for read-only code access only when the assessment begins.
2. We research. We research your sector, the regulator’s position and what the public record says about firms in your situation. This study is an example of that work.
3. We read the system and propose: the Legacy System Specification Assessment. Here the system is your own product’s code and the way it is released today, and the assessment records what the product does now and each step of a release. It produces a map of the system as it stands, a written specification of what it does, a risk register, a cost comparison between keeping and replacing the system, and a staged plan with a fixed quote. AI does a first pass, our senior engineers review it and trace it to the code, and your own experts confirm it. Code access is read-only and logged, never used for training, and deleted within 30 days of the evaluation ending unless we engage.
4. If you approve, we build. Here that means tests that record what the product does today around the parts that change, and an automated release that runs them and the same checks every time, with a written way back we rehearse with your team, all handed back to your own engineers. Where a module has to be replaced, we replace a system one module at a time while the old one keeps running. We never do a single cutover. A traceability matrix links every old function of a rebuilt module to its replacement. The work is billed by milestone, each milestone tied to a module you accept, and you can pause after any accepted module.
5. We review against the benchmark. Before work starts we set the benchmark. After each stage we show you where each measure stands, and the filled-in benchmark is yours at the end.

The Review is free. Every paid step is fixed price, agreed at scoping. Stop after any step. The findings are yours. The assessment fee is credited in full against the first build stage if it starts within 90 days of the report.

8. What the Client Would Own

• The map and the specification: what the product does and each step of a release, confirmed by the firm’s own people.
• The risk register: what the assessment found, listed before any price is set.
• The tests: a record of what the product does today, around the parts that change, handed to the firm’s engineers with instructions.
• The release process: the automated release, its checks and its record, in the firm’s own account.
• The way back: the written return step, its rehearsal record, and the form for stating what a release cannot undo.
• The traceability matrix and the code: only where a module is rebuilt, every old function linked to its replacement.
• The filled-in benchmark: each measure, where it started and where it stands.

9. What This Approach Does Not Cover

• IOanyT does not run, start, approve or monitor a firm’s releases, does not decide when to go back, and is not on call for them.
• IOanyT does not back up, restore or recover data. Returning the code to an earlier version does not bring back data a release has changed or deleted. Some releases cannot be reversed at all.
• IOanyT does not build mobile apps, which is what one of the two public cases involved.
• IOanyT does not run load tests. Nothing here is a test of security, and IOanyT’s other services are not presented here as one.
• IOanyT does not test by hand as a service and does not supply staff.
• Tests show that a product behaves as it did. They do not show that it was right, and they cover only the cases that were recorded. A check catches only what someone thought to check, and an automated release puts out a faulty change as readily as a good one if its checks miss it.
• No test, check or way back means a release cannot fail.
• IOanyT does not maintain the tests, the release process or the way back afterwards. The firm’s own team does.
• This study does not say that IOanyT has done this work for a software firm. IOanyT has no published case of it.

About this study: IOanyT did not work on this case. It is drawn from published findings, cited below.

10. Sources

Source	Used for
Android Police, 2024	The rewritten app and the list of essential faults
The firm’s own post-incident review, 2022	The script, the review, the outage and the firm’s sentence on recovery
A tool maker’s developer survey, 2025	Developers who do not write tests
A developer’s post, 2025	One developer’s words on changing old code without tests
A vendor’s survey of US IT professionals, 2025	Fear of disrupting operations

IOanyT is a consultancy with its own engineering team: we study the problem, recommend, and prove it by building only what you approve. We start by recording what the product does today, so every release has something to be checked against and a way back that has been tried.

Book a free Reliability Review →

Get a fixed quote →

See the Legacy System Specification Assessment →