A Way Back Nobody Has Tried Is Not a Way Back. The Benchmark for a Release You Can Check.

The release goes out at nine in the evening. By half past, something is plainly wrong. Customers’ dashboards load, and the figures on them are blank.

Someone on the call says the obvious thing. Roll it back. There is a silence. The team has released more than a hundred times. Nobody on the call has ever gone back.

Where is the previous version kept? Does it still match the database, which the release also changed? Nobody is sure. They work it out, carefully, with customers waiting. Service is as it was at a quarter to one.

What We Found

For this article we went through the public record on what happens when a change to a software product goes wrong and has to be undone: one large software firm’s own report of an outage, one developer’s account of changing old code, and one survey of IT professionals on why old systems stay as they are.

Three findings stand out.

The means of getting back were not built before they were needed. At a large software company in 2022, a script run to retire an old app deleted whole customer cloud sites. Several hundred customers were affected, some of them for up to two weeks. The firm’s own report says it could recover a single site, but had not built the means or the processes to recover a large batch of sites at once. This is an older case, from a large firm, and it is about recovering deleted customer data, which we do not offer. We report what the firm wrote and add nothing about why. We take one narrow point: a way back that covers one case does not cover the case that arrives.

Without tests, every change is frightening, going back included. One developer, writing about years of work on old code, said nothing is more frightening than changing a critical old system that has no tests. It is one person’s account. A return to the previous version is a change too, made under pressure, and without tests nobody can show afterwards that the product behaves as it did.

Fear of disruption is the common reason to leave things as they are. In one survey of about 500 IT professionals in the US, run by a firm that sells software services and not about software firms, nearly four in ten named fear of disrupting operations among their reasons for not modernising old systems.

Our reading is that teams do not neglect the way back because they are careless. Going back is needed rarely, so it is never rehearsed, and the hard parts of it (the data the release changed, the things it deleted) are discovered only on the night.

What Waits While Going Back Is Worked Out on the Night

A firm without a tried way back does not stop releasing. It releases with held breath, and each release that goes wrong costs more than it needs to.

The first cost is time with customers waiting. Every minute spent finding the previous version, or working out whether it still runs, is a minute customers cannot use the product.

The second cost is the answer support can give. “We are working on it” is all there is to say, because nobody knows how long going back will take.

The third is the decision made blind. Some releases change or delete data, and going back to the old code does not bring that data back. Without a statement made beforehand, the people approving a release cannot tell which kind it is.

This is our reading of the evidence. It is not a measured market figure.

Four Ways to Make a Release Safe to Undo, Weighed

There are four real options. Each one is right for somebody.

Option	When it fits	What it risks
More checking by hand before each release, and work out the way back if needed	Releases are rare and change only screens, never data	The first return is attempted in an emergency, and what cannot be undone is found then
Write tests for the whole product first, and leave the release as it is	The firm can pause new work for a long time	Months before anything changes on release night, and going back is still untried
Record what the product does today as tests around the parts that change, run them in an automated release that makes the same checks every time, and write and rehearse the way back, stating for each release what it cannot undo	The product is large and old, sells every day, and is changed every week	Rehearsing takes time that produces nothing visible, a return that worked on a copy can still fail on the live product, and the firm’s own team must keep it all up
Rebuild the product in stages, with each old function linked to its replacement	The product must be replaced anyway, for other reasons	A far larger decision than the release problem calls for

Where You Are Now, and Where You Should Be

A recommendation is only useful if you can tell later whether it worked. So before any work begins we set a benchmark: a few measures taken from your own product, each with a target.

For this issue, the benchmark would look like this.

Measure	Typical starting point	Where it should be	How it is checked
Key functions covered by tests	Few or none; a list of known faults instead	Each function customers rely on covered, starting with the parts releases change	The test run, shown against the specification
Releases run the same way every time	Steps done from memory, by whoever is releasing	The same scripted steps and checks on every release	The release process and its record
Checks of what a change will act on	None; the code is reviewed, its inputs are not	Every change that acts on customers or records shows what it will act on before it runs	The release record
The way back written and rehearsed	Never tried	Written as a step, rehearsed on a copy, and rehearsed again when the release process changes	The rehearsal record
What a release cannot undo, stated beforehand	Discovered on the night	Stated for each release before it is approved	The release record
Who approved what, on record	In a chat thread and in people’s memories	For every release: what went out, what was checked and who approved it	The release record

The starting points above are what the public record suggests and what we expect to find. In an engagement they are replaced by your own figures, taken in the first review. The targets are what we would work towards with you. They are not results we have already achieved for a software firm.

How We Work With You

1. We listen: a free Reliability Review. A first conversation about your product, how a release is done today and what happened the last time one had to be undone. You leave with a short written note of what we found. You can start with a conversation about the system; we ask for read-only code access only when the assessment begins.
2. We research. We research your sector, the regulator’s position and what the public record says about firms in your situation. This article is an example of that work.
3. We read the system and propose: the Legacy System Specification Assessment. Here the system is your own product’s code and the way it is released today, and the assessment records what the product does now and what each kind of release changes. It produces a map of the system as it stands, a written specification of what it does, a risk register, a cost comparison between keeping and replacing the system, and a staged plan with a fixed quote. AI does a first pass, our senior engineers review it and trace it to the code, and your own experts confirm it. Code access is read-only and logged, never used for training, and deleted within 30 days of the evaluation ending unless we engage. You get the options and our recommendation.
4. If you approve, we build. Here that means tests that record what the product does today around the parts that change, and an automated release that runs them and the same checks every time, with a written way back we rehearse with your team, all handed back to your own engineers. Where a module has to be replaced, we replace a system one module at a time while the old one keeps running. We never do a single cutover. Each rebuilt module has a traceability matrix. The work is billed by milestone, each milestone tied to a module you accept, and you can pause after any accepted module.
5. We review against the benchmark. Before work starts we set the benchmark. After each stage we show you where each measure stands, and the filled-in benchmark is yours at the end.

The Review is free. Every paid step is fixed price, agreed at scoping. Stop after any step. The findings are yours. The assessment fee is credited in full against the first build stage if it starts within 90 days of the report.

This approach has limits. We do not back up, restore or recover data. Going back to an earlier version of the code does not bring back data a release has changed or deleted, and nothing we build claims to; your backups are yours. We do not run, start or monitor your releases, we do not decide when to go back, and we are not on call. A return that worked on a copy can still fail on the live product. We do not test by hand as a service, supply staff, build mobile apps or run load tests. We do not maintain the tests, the release process or the way back afterwards; your own team does. Tests show that the product behaves as it did, not that it was right. No test, check or way back means a release cannot fail. The older technologies we read include VB6, Delphi, AngularJS, PHP 4 to 8, Java 8, .NET Framework and classic ASP. We have no published case of this work for a software firm.

What changes for the business:

Role	What they gain
CEO	A bad release becomes a short interruption the firm can explain, and not days of improvised repair.
CTO or CIO	Going back is a written step the team has rehearsed, and not something worked out at midnight.
Head of product	Product can agree to a release knowing what happens if it goes wrong, and what could not be undone.
Head of operations	Support can tell customers what will happen next, because going back has been tried before it was needed.
CISO	The firm knows before a release what could not be put back, and can decide with that in view.

The cases in this article, and what IOanyT does about that kind of problem:

The case	What IOanyT does
A large software company wrote that it had not built the means to recover a large batch of sites at once, and some of its several hundred affected customers were without service for up to two weeks.	Our part is far narrower. We do not back up, restore or recover data, and we do not say the event would have gone differently. For a product in a technology we read, we set up an automated release with a written way back for the code, rehearsed on a copy, and a statement for each release of what that way back cannot undo.

IOanyT is a consultancy with its own engineering team: we study the problem, recommend, and prove it by building only what you approve. We set the benchmark before the work, so you can see the way back tried before the night it is needed.

Book a free Reliability Review →

Get a fixed quote →

See the Legacy System Specification Assessment →