Releasing Often Without Fear Is the Opportunity. Four Ways to Get There, Weighed.

The change was small. An old feature was to be switched off for customers on a plan the firm no longer sells.

An engineer wrote the script. Two colleagues reviewed it, line by line, and approved it. It did exactly what it said. On Thursday evening it was run against a list of the customers on that plan.

The list had been exported three weeks earlier. Since then forty customers had moved to a newer plan that includes the feature. On Friday morning forty paying customers found it gone.

What We Found

For this article we went through the public record on how a change to a software product is checked before it reaches customers: one large software firm’s own report of an outage, one survey of developers who work in PHP, and one survey of IT professionals on why old systems stay as they are.

Three findings stand out.

A change can pass review and still act on the wrong thing. At a large software company, in 2022, a script run to retire an old app deleted whole customer cloud sites. The firm’s own report says the script had gone through its usual peer review, which looked at what the script called and how, but did not check the site identifiers it had been given. The identifiers were those of whole sites, not of the app. Several hundred customers were affected, some of them for up to two weeks. This is an older case, from a large firm. We report only what the firm’s own report says, and draw no further cause from it. We take one narrow point: a review can be sound about the code and silent about what the code is about to act on.

Many products have few tests for a release to run. In one survey of more than 1,700 developers who work mainly in PHP, a widely used web language, about a third said they still write no tests at all. That is about developers in general, not about software firms alone. Where there are no tests, the check on release day is a person’s approval and whatever else that person remembers.

Fear of disruption holds change back. In one survey of about 500 IT professionals in the US, run by a firm that sells software services and not about software firms, nearly four in ten named fear of disrupting operations among their reasons for not modernising old systems. The fear is reasonable. A firm that has no way to check a change will be slow to make one.

Our reading is that a firm which fears its own releases does not fear them because its people are careless. It fears them because the checks depend on who is releasing and how late it is, and because nothing looks at what a change will act on before it acts.

What a Firm That Fears Its Own Releases Cannot Do

A software firm sells change. Customers pay for a product that keeps improving, and every improvement reaches every customer through a release.

While releases are feared, three things wait.

The first is pace. Changes are held back and bundled, so each release carries more and is harder to check, and the next one is feared more.

The second is the small, useful change: the fix for one customer, the setting for one plan. Each needs a release, and each release is a risk nobody can measure, so small changes queue behind large ones.

The third is the answer to a careful buyer. When a customer’s auditor asks what went out last Thursday, what was checked and who approved it, the honest answer is in a chat thread and in people’s memories.

This is our reading of the evidence. It is not a measured market figure.

Four Ways to Check a Release, Weighed

There are four real options. Each one is right for somebody.

Option	When it fits	What it risks
More checking by hand, and another approval	Releases are rare and small, and the same careful people always do them	Each new signature makes the others read less closely, and nothing looks at what the change will act on
Write tests for the whole product before changing the release	The firm can pause new work for a long time	Months before the first calmer release, and the steps of the release are still done from memory
Record what the product does today as tests around the parts that change, run them in an automated release that makes the same checks every time, including what a change will act on, and write and rehearse the way back	The product is large and old, sells every day, and is changed every week	It takes time from the people who release today, a check catches only what someone thought to check, and the firm’s own team must keep it up afterwards
Rebuild the product in stages, with each old function linked to its replacement	The product must be replaced anyway, for other reasons	A far larger decision than the release problem calls for, and old releases go on as before until it is done

Where You Are Now, and Where You Should Be

A recommendation is only useful if you can tell later whether it worked. So before any work begins we set a benchmark: a few measures taken from your own product, each with a target.

For this issue, the benchmark would look like this.

Measure	Typical starting point	Where it should be	How it is checked
Releases run the same way every time	Steps done from memory, by whoever is releasing	The same scripted steps and checks on every release	The release process and its record
Checks of what a change will act on	None; the code is reviewed, its inputs are not	Every change that acts on customers or records shows what it will act on, as of the moment of release	The release record
Key functions covered by tests	Few or none	Each function customers rely on covered, starting with the parts releases change	The test run, shown against the specification
Who approved what, on record	In a chat thread and in people’s memories	For every release: what went out, what was checked and who approved it	The release record
The way back written and rehearsed	Never tried	Written as a step, rehearsed on a copy, with what it cannot undo stated	The rehearsal record

The starting points above are what the public record suggests and what we expect to find. In an engagement they are replaced by your own figures, taken in the first review. The targets are what we would work towards with you. They are not results we have already achieved for a software firm.

How We Work With You

1. We listen: a free Reliability Review. A first conversation about your product, how a release is done today and the changes that wait because releases are feared. You leave with a short written note of what we found. You can start with a conversation about the system; we ask for read-only code access only when the assessment begins.
2. We research. We research your sector, the regulator’s position and what the public record says about firms in your situation. This article is an example of that work.
3. We read the system and propose: the Legacy System Specification Assessment. Here the system is your own product’s code and the way it is released today, and the assessment records what the product does now and each step of a release. It produces a map of the system as it stands, a written specification of what it does, a risk register, a cost comparison between keeping and replacing the system, and a staged plan with a fixed quote. AI does a first pass, our senior engineers review it and trace it to the code, and your own experts confirm it. Code access is read-only and logged, never used for training, and deleted within 30 days of the evaluation ending unless we engage. You get the options and our recommendation.
4. If you approve, we build. Here that means tests that record what the product does today around the parts that change, and an automated release that runs them and the same checks every time, with a written way back we rehearse with your team, all handed back to your own engineers. Where a module has to be replaced, we replace a system one module at a time while the old one keeps running. We never do a single cutover. Each rebuilt module has a traceability matrix. The work is billed by milestone, each milestone tied to a module you accept, and you can pause after any accepted module.
5. We review against the benchmark. Before work starts we set the benchmark. After each stage we show you where each measure stands, and the filled-in benchmark is yours at the end.

The Review is free. Every paid step is fixed price, agreed at scoping. Stop after any step. The findings are yours. The assessment fee is credited in full against the first build stage if it starts within 90 days of the report.

This approach has limits. We build the release process and hand it over; we do not run, start, approve or monitor your releases, and we are not on call when one goes out. We do not test by hand as a service, and we do not supply staff. We do not back up, restore or recover data, which is what the case above turned on, and a way back for code is not a way back for data a release has changed or deleted. We do not build mobile apps, and we do not run load tests. A check catches only what someone thought to check, and an automated release puts out a bad change as readily as a good one if its checks miss it. We do not maintain the tests or the release process afterwards; your own team does. Tests show that the product behaves as it did, not that it was right. No check means a release cannot fail. The older technologies we read include VB6, Delphi, AngularJS, PHP 4 to 8, Java 8, .NET Framework and classic ASP. We have no published case of this work for a software firm.

What changes for the business:

Role	What they gain
CEO	A release no longer depends on the right person noticing the right thing on the day.
CTO or CIO	The same steps and checks run on every release, so a change that was reviewed is also a change that was checked.
Head of product	A change meant for some customers reaches those customers and no others.
Head of operations	Release night follows the same written steps each time, and operations knows what was checked before anything reached a customer.
CISO	A record, kept for every release, of what went out, what was checked and who approved it.

The cases in this article, and what IOanyT does about that kind of problem:

The case	What IOanyT does
At a large software company in 2022, a script that had passed the firm’s usual peer review did not check the site identifiers it was given, and several hundred customers were affected.	Our part is far narrower. We do not back up, restore or recover data, and we make no comment on that firm’s practice or say the event would have gone differently. For a product in a technology we read, we set up an automated release that runs the same checks every time, including what a change is about to act on, and hand it to your team.

IOanyT is a consultancy with its own engineering team: we study the problem, recommend, and prove it by building only what you approve. We build the checks into the release itself, so a change that was approved is also a change that was checked.

Book a free Reliability Review →

Get a fixed quote →

See the Legacy System Specification Assessment →