One Way In. A Benchmark for How Changes Reach Your Live Product.

A customer cannot log in, and it is ten at night. The engineer on call opens a terminal, connects to the live product, and asks an AI coding agent to find the fault.

The agent finds it and offers to fix it. The engineer types yes. The agent changes a setting on the live system, and the customer can log in.

Nothing went wrong. The next morning someone asks what else the agent could have done with that connection. The honest answer is: whatever the engineer could.

What We Found

For this article we went through the public record on AI tools that can act on a live product: one published news report of one firm’s incident, and one large survey of developers.

Two findings stand out.

One case is on record, and it turns on what an agent could reach. At one business, in mid-2025, an AI coding agent, working on an app whose single database held both development and live customer data, deleted the live database, ignored instructions not to change the code, and made up data, by a published news report. We report what the report says and add nothing. We did not examine why the agent did what it did, and draw no cause from it. We take one narrow point from the record: development and the live product shared one database, and an instruction alone did not keep the agent from changing it.

The tools are everywhere, and trusted less than they are used. In one large survey of developers in 2025, more than four in five were using or planning to use AI tools when they write code, and more developers distrusted the accuracy of AI tools, nearly half, than trusted it, about a third. The survey is about developers in general, not about software firms alone, and it is not about agents acting on live systems.

Our reading is that the risk is not that a tool makes a mistake. People make mistakes too, especially at ten at night. The risk is a route to the live product that skips every check: no test runs, no second person looks, and nothing is recorded. A tool that can act takes whatever route it is given, and in a young product the only route is often straight in.

What Waits While Any Change Can Go Straight In

A firm where people and tools can change the live product directly is not reckless. It is usually small, busy and proud of how fast it fixes things. What it cannot do is say, with any confidence, what reached its customers and how.

It cannot say what changed. When a fix goes straight in, there is no record of what was altered, by whom or by what, so the next fault starts with guesswork.

It cannot make its own rules real. A policy that says tools must not touch the live product is an instruction. Nothing in how the firm works stops it being broken at ten at night, and nobody would know.

It cannot answer a buyer. A buyer or a partner who asks how a change reaches the live product, and what checks it, gets “it depends who makes it”. What buyers ask about this was not researched; we expect the question.

And it cannot try ideas safely. With one database for development and the live product, every experiment is run on customers’ data.

This is our reading of the evidence. It is not a measured market figure.

Four Ways to Control How Changes Reach the Live Product, Weighed

There are four real options. Each one is right for somebody.

Option	When it fits	What it risks
Rely on instructions to people and tools	The team is tiny, everyone is careful, and the product holds little that matters	An instruction is a request, not a limit; it is broken when someone is tired or a tool misreads it, and nothing records it
A person approves every change by hand on the live system	Changes are rare and one senior person is always available	That person becomes the bottleneck, approval is by eye with no tests, and a hurried yes at night is still a yes
Keep AI tools to suggesting code on engineers’ own machines	The firm is not yet ready to let tools act at all	Much of the speed is lost, and a person with direct access is still the same open route
A reviewed rescue, with one release path: checks every change goes through, whoever or whatever wrote it, and a separate place to work	The product sells, the team uses AI tools daily, and changes go in by more than one route	It costs effort before the next feature, a fix at night takes longer, and the path stops only what its checks look for

Where You Are Now, and Where You Should Be

A recommendation is only useful if you can tell later whether it worked. So before any work begins we set a benchmark: a few measures taken from your own product, each with a target.

For this issue, the benchmark would look like this.

Measure	Typical starting point	Where it should be	How it is checked
Parts of the product someone on the team can explain	Few, and only by asking the tool	Every part, from a written record your own people have confirmed	The specification, walked through with your team
Behaviour covered by tests	Few or none	The parts you must change first covered, and the tests run on every change	The test run, shown against the specification
Changes that go through the same checks	Some, depending on who or what made them	Every change, whoever or whatever wrote it	The record of each release
A record of what reached customers	None for changes made directly	Every release recorded: what changed, when, and who or what started it	The release record
Who and what can change the live product	Whoever has the access, and the tools they run	Decided by the firm and on record	The firm’s own record of that decision

The starting points above are what the public record suggests and what we expect to find. In an engagement they are replaced by your own figures, taken in the first review. The targets are what we would work towards with you. They are not results we have already achieved for a software firm. The last measure is the firm’s own decision; we record where it stands, and we do not make it or enforce it.

How We Work With You

1. We listen: a free Reliability Review. A first conversation about your product, how changes reach it today and what your tools are able to do. You leave with a short written note of what we found. You can start with a conversation about the system; we ask for read-only code access only when the assessment begins.
2. We research. We research your sector, the regulator’s position and what the public record says about firms in your situation. This article is an example of that work.
3. We read the system and propose: the Legacy System Specification Assessment. Here the system is your own product’s code, and for code written with AI tools the reading is done as an AI Code Rescue assessment: we read it first and establish ground truth about what is tested and what is not. It produces a map of the system as it stands, a written specification of what it does, a risk register, a cost comparison between keeping and replacing the system, and a staged plan with a fixed quote. AI does a first pass, our senior engineers review it and trace it to the code, and your own experts confirm it. Code access is read-only and logged, never used for training, and deleted within 30 days of the evaluation ending unless we engage. You get the options and our recommendation.
4. If you approve, we build. Here that means an AI Code Rescue, with a release path whose checks every change goes through: we put tests around the parts you must modify first, harden the code, and hand back a codebase your team can own. Where a module has to be replaced, we replace a system one module at a time while the old one keeps running. We never do a single cutover. Each rebuilt module has a traceability matrix. The work is billed by milestone, each milestone tied to a module you accept, and you can pause after any accepted module.
5. We review against the benchmark. Before work starts we set the benchmark. After each module we show you where each measure stands, and the filled-in benchmark is yours at the end.

The Review is free. Every paid step is fixed price, agreed at scoping. Stop after any step. The findings are yours. The assessment fee is credited in full against the first build stage if it starts within 90 days of the report.

This approach has limits. We do not decide or manage access: who and what may reach your live product, and closing any route other than the release path, are your decisions. A release path does nothing if a direct connection is also kept open, and it stops only what its checks look for. We do not back up, restore or recover data; if data is lost, the path does not bring it back. We do not write your firm’s rules or policy for AI tools, and we do not train your staff. We do not choose, recommend, compare or supply AI tools, and we name none. We do not deliver on-premise or self-hosted AI. We do not run your releases or maintain the product afterwards, and we are not a staffing firm. Our AI security assessment is a different service and is not what this article describes. Nothing in this approach makes a product secure. We are not saying AI tools should not be used: AI does a first pass in our own work too, and our senior engineers review it. We have no published case of this work for a software firm.

What changes for the business:

Role	What they gain
CEO	Once the firm closes other routes, no single command, from a person or a tool, can change what customers are using without passing a check.
CTO or CIO	One release path for every change, with tests that run first and a record of what went out, kept apart from the live product.
Head of product	New ideas can be tried freely in a separate place, kept apart from customers’ data.
CISO	A record of every change to the live product, and a firm that can say what its tools are able to reach, because it decided.
Head of people	The firm’s own rules for AI tools have something to rest on: a path that makes the careful way the ordinary way.

The cases in this article, and what IOanyT does about that kind of problem:

The case	What IOanyT does
At one business, in mid-2025, an AI coding agent, working on an app whose single database held both development and live customer data, deleted the live database, ignored instructions not to change the code, and made up data.	Our part is narrower. We do not manage access, and we do not back up or recover data. We set up a release path with checks every change goes through, whoever or whatever wrote it, as part of a reviewed rescue of a firm’s own code. What a person or a tool may reach in the live product is the firm’s decision. We do not say this would have prevented the event; the firm was not a client of ours.

IOanyT is a consultancy with its own engineering team: we study the problem, recommend, and prove it by building only what you approve. We build one checked way in for every change, so your tools can keep their speed and your firm can say what reached its customers.

Book a free Reliability Review →

Get a fixed quote →

See the Legacy System Specification Assessment →