Why your RAG demo works and your production system doesn't

A RAG demo is easy. Load ten clean PDFs, embed them, retrieve the top three chunks, and put them in a prompt. It looks impressive. Then you point it at the real corpus — a decade of scanned contracts, inconsistent headings, tables, amendments — and it falls apart.

This is not a model problem. In almost every case I've investigated, the failure is upstream of generation. Here are the four failure modes that show up at scale, and the order I fix them in.

1. The right passage never reaches the model

This is the most common cause of "hallucination." The model isn't inventing an answer from nothing; it's inventing one because the evidence was missing from the context.

Three specific causes:

• Chunks ignore document structure. Fixed 512-token windows split a clause from its exception, or a table from its header. The retrieved text is topically related but semantically incomplete.
• Embeddings blur exact terms. Section numbers, statute names, product codes, and names are where dense retrieval is weakest. A query for "Section 12(3)" can retrieve a dozen semantically similar sections.
• No reranking. The retriever returns ten candidates; the generator sees the top three. If the correct passage ranked fourth, it never had a chance.

The fix is not a better model. It's metadata-aware chunking, hybrid retrieval, and a reranker. I wrote about the retrieval side separately in hybrid search.

2. Nobody defined "correct"

This is the failure that hides all the others. If you never built a labelled question set, every change is judged by whether a few sample answers "look better." That is not measurement; it's vibes with extra steps.

Without evaluation you cannot answer basic questions:

• Did chunking change X improve retrieval, or just shift which questions fail?
• Is the system retrieving the right document but citing the wrong section?
• Did the new embedding model help overall while hurting the exact-match queries that matter most?

A minimum viable evaluation is small: 25–50 real questions, each with the document (and ideally the passage) that contains the answer. Then measure retrieval separately from generation. The metrics are covered in this post.

3. Context assembly is treated as an afterthought

Retrieval returns chunks. Generation needs a context window. The step between them is usually a naive concatenation, and it quietly destroys quality.

What goes wrong:

• Redundant chunks crowd out diverse evidence. Three near-identical passages consume the budget that could have held the exception clause.
• Ordering matters. Models attend more reliably to the beginning and end of long contexts than to the middle. Put the strongest evidence first, not in the middle of a wall of text.
• Sources are not labelled. If chunks arrive without document, section, and page metadata, the model cannot cite them — and you cannot audit the answer.

Treat context construction as a ranking and budgeting problem, not string concatenation.

4. The system has no concept of "I don't know"

Ask a demo a question whose answer isn't in the corpus and it will often answer anyway. In a chat toy that's amusing. In legal, compliance, or finance it's a liability.

Two features fix this, and they are not optional:

1. Citations. Every claim maps to a retrieved passage, and the claim is checked against that passage — not just linked to it.
2. Refusal. When retrieval confidence is low or the claim isn't supported, the system says so instead of guessing.

Both are covered in citations and refusal.

The order to fix things

Teams usually start with the model because it's the most visible component. Do it in this order instead:

1. Build the evaluation set. 25 real questions with known answers.
2. Measure retrieval alone. If recall@k is low, generation is irrelevant.
3. Fix chunking and metadata. Structure-aware chunks with filters.
4. Add hybrid retrieval and reranking. Re-measure after each change.
5. Then improve generation, citations, and refusal.
6. Only then consider fine-tuning.

Why this matters commercially

Every week spent tuning prompts against a broken retrieval layer is a week of budget that produces no measurable improvement. The teams that ship reliable RAG systems are not using better models — they built a measurement loop and fixed the pipeline in order.

You can see the measurement side implemented in the open-source RAG Eval Kit.