Why Your RAG Demo Breaks in Production

Every RAG demo looks production-ready right up until it meets production. You pick ten representative questions, the retriever finds the right chunk, the model answers correctly, everyone nods. Three weeks after launch, a support lead asks something slightly off-script, the system answers with total confidence, and the answer is wrong — not vague-wrong, but specific-sounding, cited-sounding, wrong.

That gap between demo and production isn't a model problem. It's an architecture problem, and it shows up in the same handful of places every time, long before generation ever runs.

RAG reduces hallucination — it doesn't remove it

The pitch for RAG is that grounding the model in retrieved documents stops it from making things up. That's true, with a large asterisk: it only holds if the right documents actually get retrieved, and if the model is constrained to use only what it retrieved. Neither of those is automatic. A naive implementation — embed the query, grab the top five chunks by cosine similarity, stuff them into the prompt, generate — introduces failure points at every one of those steps, and the model will paper over each one with a fluent, confident sentence.

Where naive retrieval actually fails

Three failure modes account for most of what looks like "the model hallucinated" in a production RAG system — and none of them originate in the model.

• Retrieval misses. The correct passage exists in the corpus but isn't retrieved — because it's chunked badly (split mid-sentence, context stripped from its heading), because the embedding model doesn't represent domain-specific or acronym-heavy terms well, or because the query uses different words than the document. A pure dense-vector search is especially bad at this: it's excellent at semantic similarity and weak at exact terms — part numbers, error codes, product SKUs, legal citations — that a keyword search would catch instantly.
• Fusion and reranking gaps. Even when the right chunk is somewhere in the retrieved set, a plain top-k-by-similarity ranking often buries it below three chunks that are topically related but factually beside the point. Without a reranking step that actually scores relevance against the specific query, the model generates from whatever landed in the first few positions — which may not be the chunk that actually answers the question.
• No abstention path. This is the one that does the most damage. A model asked a question, handed a stack of context, and told to answer will answer — even when the context doesn't actually contain the answer. There's no built-in mechanism that says "I don't have enough here," unless the system explicitly builds one and checks it before generation runs.

A RAG system that always answers is a RAG system that will eventually answer confidently from nothing.

Hybrid search: the fix for most retrieval misses

Dense vector search and keyword full-text search fail in different, complementary ways. Vector search generalizes across phrasing but misses exact terms; full-text search nails exact terms but misses paraphrase and synonymy. Running both and fusing the results — commonly with Reciprocal Rank Fusion (RRF), which combines two ranked lists without needing either one's raw scores to be comparable — catches a meaningfully larger share of the genuinely relevant chunks than either method alone. This is the single highest-leverage change for a RAG system that's missing obvious answers: it's not a smarter model, it's a second retrieval path the first one doesn't have.

Reranking, and the evidence gate most demos skip entirely

After hybrid retrieval, a cross-encoder reranker scores the candidate chunks directly against the query — slower than vector similarity, but far more precise, because it looks at the query and each chunk together instead of comparing pre-computed embeddings in isolation. That gets the right chunk near the top of the stack. It still doesn't answer the harder question: is what made it to the top of the stack actually enough to answer the question at all?

That's the piece we think almost every production RAG system is missing: an explicit evidence gate that runs after retrieval and before generation, decides whether the retrieved context is sufficient to support an answer, and refuses — out loud, with an explanation — when it isn't. Not a lower temperature. Not a system-prompt instruction to "only use the provided context," which models follow inconsistently under pressure. A separate, deliberate decision point the pipeline cannot skip.

Citations that trace back to a real chunk

The other half of trustworthy output is verifiability. An answer with no citations asks the reader to take it on faith; an answer with citations that don't actually trace back to the retrieved chunk is worse, because it looks verified when it isn't. Every claim in a generated answer should be attributable to a specific retrieved passage — and that mapping should be mechanical, built from the retrieval step itself, not generated by the model describing its own sources from memory.

What this looks like in Ragix

This is the architecture behind Ragix, our production RAG platform: hybrid search fusing dense vectors and full-text with RRF, a cross-encoder reranker on top of that, and an evidence gate that runs before a single token of generation — deciding whether the retrieved context actually supports an answer, and abstaining with an explanation when it doesn't. Every response carries inline citations tied to the chunks that were actually retrieved. A retrieval debugger exposes the full trace — query, candidates, scores, the gate's decision — for every request, so "why did it say that" has an answer that isn't a guess.

Combined with the tenant isolation enforced at the database layer (see our post on multi-tenant RAG row-level security), the goal is a system that fails safe in both directions: it won't retrieve across a boundary it shouldn't, and it won't answer when it doesn't actually know.

If your RAG system passed its demo and you haven't specifically tested what it does when the answer genuinely isn't in the retrieved context, that's the test worth running before a user runs it for you.