"It Seems Fine" Is Not a Test. An Upgrade Your Own Engineers Can Check.

Two engineers spent three weeks moving the product to the supported version of its framework. On a test copy, every screen they opened worked.

In the release meeting someone asks whether it is safe to ship. One of the engineers answers honestly. It seems fine.

It ships. Nine days later a support ticket arrives. A customer’s month-end report shows a different total from last month’s, for the same data. Nobody changed the report. Something underneath it changed, and nothing reported an error.

What We Found

For this article we went through the public record on how old code is checked when what lies beneath it changes: one survey of developers who work in PHP, one developer’s own account of old code, one large software firm’s engineering account, and published support dates.

Three findings stand out.

Many products have no tests to check an upgrade against. In one survey of more than 1,700 developers who work mainly in PHP, a widely used web language, about a third said they still write no tests at all. One developer, writing about old code, said nothing is more frightening than changing a critical old system that has no tests. Both are about developers in general, not about software firms alone.

Code that looks right can behave differently. One large software firm, writing about its own move of old code, found that translating individual files without the code they depend on gave partial rewrites that looked correct as code but behaved differently when they ran. Nothing failed to run. It simply did something else.

The dates keep coming. Among the versions of PHP, 8.1 reached its end on 31 December 2025, and 8.2 gets security fixes only until 31 December 2026. A product that moves once will have to move again. Each move asks the same question as the last: does the product still do what it did?

Our reading is that most upgrades of a product’s language or framework do not go wrong because the engineers are careless. They go wrong silently, in a total or a rule that nobody thought to check, because there was nothing to check it against.

What Waits While Every Upgrade Is a Guess

A firm that cannot check an upgrade does not stop upgrading. It upgrades less often, later, and with more held breath each time.

The first cost is delay. Each end-of-support date is met late, because the last upgrade was frightening.

The second cost is the support desk. What the upgrade changed is found by customers, one ticket at a time, and each ticket needs an engineer who knows the old behaviour.

The third is that the next upgrade starts from scratch. Nothing from the last one was kept that would make the next one safer.

This is our reading of the evidence. It is not a measured market figure.

Four Ways to Check an Upgrade, Weighed

There are four real options. Each one is right for somebody.

Option	When it fits	What it risks
Click through the screens people remember	The product is small and few customers rely on its edge cases	What nobody remembers is not checked, and customers find it
Upgrade, ship, and fix what customers report	Customers are few, close and forgiving	Faults reach customers first, and support pays for the testing
Write tests for the whole product before any upgrade	The firm can pause other work for a long time	Months of work before the first upgrade, on parts that may never change
Record what the product does today, starting with the parts the upgrade touches, and check each upgrade against it	The product is large and old, and its maker’s support dates keep coming	It costs effort before the upgrade, and the tests must be kept up by the firm’s own team afterwards

Where You Are Now, and Where You Should Be

A recommendation is only useful if you can tell later whether it worked. So before any work begins we set a benchmark: a few measures taken from your own product, each with a target.

For this issue, the benchmark would look like this.

Measure	Typical starting point	Where it should be	How it is checked
What the product does, written down	Known to the engineers who wrote it	Every function in plain words, confirmed by your own people	The specification
Behaviour recorded as tests, for the parts an upgrade touches	Few or none	Each such part covered, including the totals customers rely on	The test run, shown against the specification
How an upgrade is judged ready	”It seems fine”	The tests pass, and any difference is explained before release	The record of each upgrade
Differences found by customers after an upgrade	Not counted	Counted, and falling with each upgrade	Your own support records
Who can run the checks	The engineers who did the upgrade	Any engineer on your team	The tests and their instructions, handed over

The starting points above are what the public record suggests and what we expect to find. In an engagement they are replaced by your own figures, taken in the first review. The targets are what we would work towards with you. They are not results we have already achieved for a software firm.

How We Work With You

1. We listen: a free Reliability Review. A first conversation about your product, the upgrades ahead of it and how the last one went. You leave with a short written note of what we found. You can start with a conversation about the system; we ask for read-only code access only when the assessment begins.
2. We research. We research your sector, the regulator’s position and what the public record says about firms in your situation. This article is an example of that work.
3. We read the system and propose: the Legacy System Specification Assessment. Here the system is your own product’s code. It produces a map of the system as it stands, a written specification of what it does, a risk register, a cost comparison between keeping and replacing the system, and a staged plan with a fixed quote. AI does a first pass, our senior engineers review it and trace it to the code, and your own experts confirm it. Code access is read-only and logged, never used for training, and deleted within 30 days of the evaluation ending unless we engage. You get the options and our recommendation.
4. If you approve, we build. Here that means tests that record what the product does today, for the parts an upgrade touches, handed to your engineers with the specification so they can check each upgrade. Where a part has no supported version to move to, we replace a system one module at a time while the old one keeps running. We never do a single cutover. Each rebuilt module has a traceability matrix. The work is billed by milestone, each milestone tied to a module you accept, and you can pause after any accepted module.
5. We review against the benchmark. Before work starts we set the benchmark. After each stage we show you where each measure stands, and the filled-in benchmark is yours at the end.

The Review is free. Every paid step is fixed price, agreed at scoping. Stop after any step. The findings are yours. The assessment fee is credited in full against the first build stage if it starts within 90 days of the report.

This approach has limits. We do not upgrade your product from one version of its language, framework or runtime to the next, and we do not apply security fixes; your own engineers do. Tests show that the product behaves as it did; they do not show that it was right, and they do not make it secure. We do not maintain the tests afterwards; your own team keeps them current. We do not run your releases or host your product. The older technologies we read include VB6, Delphi, AngularJS, PHP 4 to 8, Java 8, .NET Framework and classic ASP. We have no published case of this work for a software firm.

What changes for the business:

Role	What they gain
CEO	Keeping the product on supported technology becomes ordinary work for the firm’s own team, and stops being a crisis every few years.
CTO or CIO	A set of tests that records what the product does today, so the team can take the next version and see what changed before customers do.
Head of product	A release that changes what is underneath the product no longer changes what customers see by accident.
CFO	Faults from an upgrade are found in a check before release, and not afterwards in support time and unhappy customers.
CISO	The next end-of-support date can be met when it comes, because the firm has a way to check an upgrade.

The cases in this article, and what IOanyT does about that kind of problem:

The case	What IOanyT does
This article names no organisation. Its figures come from a survey of developers, one developer’s account, one large firm’s engineering account and published support dates.	We record what your product does today as tests your own engineers can run before each upgrade. We do not carry out the upgrade, apply security fixes or keep the tests current afterwards.

IOanyT is a consultancy with its own engineering team: we study the problem, recommend, and prove it by building only what you approve. We hand your engineers a record of what the product does, so “it seems fine” can become “it passed”.

Book a free Reliability Review →

Get a fixed quote →

See the Legacy System Specification Assessment →