Your Releases Happen at Midnight, by Hand. Four Ways Out, Weighed.
A release done by hand makes a firm release less often, and each release larger. Here is what that holds back, the four ways out, and the benchmark for a release that is the same every time.
It is eleven o’clock on a Saturday night. One engineer is at a desk with a checklist of forty-three steps.
Stop the application. Take a backup. Copy these files to that server. Run this script. Restart. Each step is done by hand, on the live system, from a document last updated by someone who has since left.
At step twenty-seven the engineer is unsure whether the instruction still applies. There is nobody to ask. The step is skipped.
On Monday the branches find out what step twenty-seven was for.
The paradox: The more careful the firm tries to be about releases, the rarer and riskier each one becomes.
What We Found
We looked at two things: what the public record says about how changes reach live systems in financial services, and how a release done by hand develops over the years.
The regulator looks at the change process. In April 2024 the RBI barred Kotak Mahindra Bank from taking new customers through its online and mobile channels and from issuing new credit cards, for about ten months. Patch and change management was among the areas it cited.
The bigger the change, the bigger the harm when it goes wrong. TSB Bank’s move to a new platform in April 2018 was one very large change to live systems, made at once. The bank received 225,492 complaints in the year that followed and was fined £48.65 million in December 2022.
Releasing less often makes each release larger. A firm hurt by a bad release does fewer of them. Changes pile up between windows, so each release carries more, takes longer and is harder to undo. When it fails, nobody can tell which change caused it.
The slow route gets bypassed. When the normal route means waiting weeks for a window, urgent work goes through as an “emergency” release, with fewer checks and approval signed after the event. In time the emergency route becomes the normal one.
What our review does not show: The first two findings come from the regulators’ records. The record does not say how Kotak or TSB released software, or what testing TSB carried out, and we do not say TSB’s change was untested. The last two findings are our own reasoning about how manual release processes develop. We found no published figure for how often mid-size lenders release, so we give none.
What Waits Behind the Release Window
The cost of a release done by hand is usually counted in weekend hours. Our reading is that the larger cost is what waits.
Finished work waits. A change completed in the first week of the month sits until the window in the fourth. A new product that is ready on Thursday goes live three weeks from Saturday.
Security fixes wait. A fix is a release like any other. With one window a month, the fix queues behind the business change that has a sponsor.
The process stops meaning what it says. When half the year’s changes go through the emergency route, the change log records the exceptions, not the process.
What opens up. A firm whose release is run by a program, the same way every time, can release one small change when it is ready. Products go out when they are finished, fixes the week they arrive, and the change record is produced by the release itself.
The gap we see: most firms treat the release as a risk to be reduced by doing it less. The opportunity is the other way round: make each release small, tested and the same every time, and doing it more often becomes the safer choice.
This is our reading of the evidence. It is not a measured market figure.
Four Ways to Fix the Release Process, Weighed
Each of these is right for somebody.
| Option | When it fits | What it risks |
|---|---|---|
| More checking by hand, more sign-offs, a second engineer on release night | Releases are rare and the system changes little | Each release takes longer and still depends on people doing forty-three steps the same way |
| A change freeze, with only emergency releases | A short, known period of high risk, such as a year-end or an audit | Work piles up, emergencies go through with fewer checks, and the release after the freeze is the largest of the year |
| Rebuild the whole system on a modern platform | The system has to be replaced for other reasons | It answers a release problem with the biggest change of all, and releases to the old system still have to go in while it runs |
| Establish what the system does today, add automated tests and an automated release, then hand back to the firm’s own team | The system is still needed and changes often | Turning a checklist into a program uncovers steps nobody can explain, and the tests must come first, because an automated release of untested code only fails faster |
Our recommendation: the fourth option, in that order. First a written record of what the system does today. Then automated tests around the parts that matter most. Then a release that a program runs, the same way every time, with a tested way back. A person still approves each release; only the hand-work goes. Then we hand the system and its release process back. We do not run your releases as a service. Your own team runs them, and applies the fixes, after hand-back.
Some old systems cannot be changed without a short stop, so releases in working hours may not be possible for every part. That is a limit to plan for, not a reason to keep the checklist.
Where You Are Now, and Where You Should Be
Before the first release is automated, we would set a benchmark for the release process itself.
| Measure | Typical starting point | Where it should be | How it is checked |
|---|---|---|---|
| Release steps done by hand | A long checklist, followed at night | None, apart from a person’s approval | The release record |
| Key functions covered by automated tests | None; people check screens by hand | Every function on the money paths, run before each release | The test run, shown against the list of key functions |
| Changes rolled back | Undone by hand, under pressure, if at all | Every release can be rolled back by the release itself | Rehearsed on a copy, then recorded at each release |
| How long a security fix waits | Measured in the first review | Released as it arrives, after the tests pass | The patch record |
| Who can start a release | One or two trusted engineers | Anyone authorised on your team | The release record |
| A written record of what the system does | In people’s heads | Written down, and confirmed by your own experts | The specification |
The starting points above are what we expect to find. In an engagement they are replaced by your own position, taken in the first review. The targets are what we would work towards with you. They are not results we have already achieved for a lender.
How We Work With You
- We listen: a free Reliability Review. A first conversation about the system, how releases are done today, and the work that waits for each window. You leave with a short written note of what we found.
- We research. We research your sector, the regulator’s position and what the public record says about firms in your situation. This article is an example of that work.
- We read the system and propose: the Legacy System Specification Assessment. It establishes what the system does today: a written specification of what it does, a risk register, a cost comparison between keeping and replacing the system, and a staged plan with a fixed quote. AI does a first pass, our senior engineers review it and trace it to the code, and your own experts confirm it. Code access is read-only and logged, never used for training, and deleted within 30 days of the evaluation ending unless we engage. You get the options and our recommendation.
- If you approve, we build. We add automated tests and an automated, repeatable release, strengthen the weak points, and hand the system back to your team. Where a module has to be replaced, we replace a system one module at a time while the old one keeps running. We never do a single cutover. The work is billed by milestone, each milestone tied to a module you accept, and you can pause after any accepted module.
- We review against the benchmark. Before work starts we set the benchmark. As the work goes on we show you where each measure stands, and the filled-in benchmark is yours at the end.
The Review is free. Every paid step is fixed price, agreed at scoping. Stop after any step. The findings are yours. The assessment fee is credited in full against the first build stage if it starts within 90 days of the report.
What changes for the business:
| Role | What they gain |
|---|---|
| CEO | Releases stop being events the firm holds its breath for, and finished work no longer waits weeks for a window. |
| CTO or CIO | A release no longer depends on one person and a checklist, and it can be undone. |
| CISO | Who released what, and when, is recorded automatically, and fewer people need hands on the live system. |
| COO | Changes arrive in small pieces, with less weekend downtime and fewer Monday surprises. |
| Risk and compliance | A change process that is consistent and recorded for every release, emergency or not. |
The cases in this article, and what IOanyT does about that kind of problem:
| The case | What IOanyT does |
|---|---|
| The RBI cited patch and change management at Kotak Mahindra Bank among its reasons for restricting the bank (April 2024). | We set up tests and an automated release, so every release runs the same way and records itself. Answering the regulator is the firm’s own job. |
| TSB Bank’s single very large change to live systems, in April 2018, was followed by 225,492 complaints. | Our part is narrower: we make releases to the systems around the core small, tested and reversible, so a problem affects one change and can be undone. |
IOanyT is a consultancy with its own engineering team: we study the problem, recommend, and prove it by building only what you approve. The checklist becomes a program your own team runs, in daylight.
Sources
In this issue
Weak patch and change process
Research note 1
The Regulator Now Looks at How You Change Your Systems. Most Old Systems Cannot Show a Change Was Safe.
Regulators now look at how firms change their systems, yet most old systems cannot show a change was safe. What we found, the options, our benchmark.
Research note 3
Your Security Fixes Are Waiting in a Queue. Here Is the Benchmark We Would Set.
Security fixes wait on old systems because nobody can see what they break. The benchmark we would set so your own team applies them as they arrive.
Issue study
The Change Nobody Could Test
What the RBI cited on patch and change management, what one very large change cost a UK bank, the options a lender has and the benchmark we would set.
More from Banking
Regulator restrictions and penalties
A Replacement Plan Needs Stages Someone Can Check. Here Is the Benchmark We Would Set.
Dependence on one vendor or shared provider
A Supplier Runs Your System. Here Is the Benchmark for Overseeing It.
Business rules buried in code