RAG in Production: Where It Actually Breaks

The retrieval-augmented generation demo is one of the most seductive in modern software. You point it at your documents, ask a question, and get a fluent, cited answer. It looks finished. It is not finished.

RAG—retrieval-augmented generation—is the standard pattern for making a language model answer questions from your own data: you retrieve relevant chunks of your documents and feed them to the model as context. The demo works because the demo asks the questions the demo was built for. Production asks the questions you never anticipated, over data that changes, at a scale the demo never saw. That is where RAG breaks, and it usually breaks quietly—the system keeps producing confident, well-formatted answers that happen to be wrong.

Here is where it actually fails, drawn from the patterns that recur across real deployments.

The Retrieval Is the Problem, Not the Model

The most common mistake is blaming the language model for bad answers when the real failure happened one step earlier, in retrieval. If the system fetches the wrong chunks of text, even a perfect model will give a wrong answer, because it is faithfully summarising the wrong source.

Retrieval fails in ways demos never surface. The user’s question uses different words than the document (“time off” versus “annual leave”), so semantic search misses the relevant passage. The answer is spread across three sections but only one is retrieved, so the model confidently gives a half-answer. Two documents contradict each other—an old policy and its replacement—and retrieval returns both with no notion of which is current. In every case the output looks authoritative and is subtly wrong, which is the worst possible failure because nobody notices.

The fix begins with treating retrieval quality as a first-class metric you actually measure, not an assumption. Before you tune prompts, verify the system is even fetching the right material.

Chunking Decides Everything, and Everyone Ignores It

How you split documents into chunks quietly determines whether the whole system works. Chunk too small and you sever a sentence from the context that gives it meaning. Chunk too large and you dilute the relevant fact among paragraphs of noise, and retrieval quality collapses.

The naive approach—split every N characters—cuts tables in half, separates a heading from the section it introduces, and orphans the crucial caveat from the statement it qualifies. Chunking that respects the structure of the document, keeps related content together, and preserves enough surrounding context is unglamorous work that has more impact on answer quality than any model choice. It is also the first thing teams skip.

Stale Data and the Freshness Problem

The demo indexed the documents once. Production documents change—policies update, prices move, products launch and retire. Unless the index updates when the source updates, the system will confidently cite last quarter’s answer forever, and it will do so with the same fluent certainty as a correct answer.

This is an operational problem masquerading as an AI problem. It requires a real pipeline: detecting when source content changes, re-indexing the affected material, and removing what no longer applies. Get this wrong and your RAG system becomes a very convincing way to distribute outdated information.

Hallucination Doesn’t Disappear—It Hides

The pitch for RAG is that grounding the model in retrieved documents stops it from making things up. It reduces hallucination; it does not eliminate it. When retrieval returns nothing useful, a model told to be helpful will often answer anyway, blending the sparse real context with plausible invention. Now the hallucination wears a citation, which makes it more dangerous, not less, because the user has been trained to trust the sources.

Building for this means the system must be able to say “I don’t have enough information to answer that”—and must be evaluated on whether it actually does so rather than guessing. A RAG system that never admits ignorance is not grounded; it is just confident.

No Evaluation Means No Idea

The deepest problem is that most RAG systems ship with no way to know if they are any good. The team clicks through a dozen questions, the answers look right, and it goes live. There is no test set, no measurement of retrieval accuracy, no tracking of whether answer quality drifts as data and usage change.

Production RAG needs an evaluation harness: a curated set of real questions with known-good answers, run regularly, measuring both whether the right sources were retrieved and whether the final answer was correct and grounded. Without it, you cannot tell whether a change improved the system or quietly broke it, and you certainly cannot tell that quality has degraded until a user complains—by which point the wrong answers have been shipping for weeks.

Building RAG That Holds Up

None of this means RAG is a bad pattern. Used well, it is genuinely powerful. It means RAG is a system, not a demo—and systems need the same discipline as any other production software. Measure retrieval quality directly. Chunk with respect for document structure. Build a real freshness pipeline. Make the system willing to say it doesn’t know. And above all, evaluate continuously against known-good answers so you find problems before your users do.

A RAG demo takes an afternoon. A RAG system you can trust takes engineering. The gap between them is exactly the gap between something that looks like it works and something that actually does.

---