Your RAG problem is a retrieval problem wearing a trench coat

A team once showed me three weeks of prompt iterations for a support bot that kept inventing refund policies. Version 41 of the prompt said "ONLY use the provided context" in caps, bold, and what I can only describe as a threatening tone. Nobody had printed what the retriever returned. It was returning the 2019 policy. The model did exactly what it was told, with garbage.

Generation gets the blame because generation writes the sentence. But in every RAG autopsy I have done, the corpse is upstream: chunking that split a table from its header, embeddings that think "cancel subscription" and "cancel order" are the same concept, or a top-k that buried the right passage at rank 14.

debug in this order

1. Print the retrieved chunks for ten failing queries. Actually read them. This step alone solves half of cases and costs nothing.
2. Score retrieval separately from generation. Recall at k against a small labeled set tells you which component is lying to you.
3. Only then touch the prompt.

the fixes that actually move recall

Chunking by document structure instead of fixed token counts. Attaching context to chunks before embedding them, which Anthropic wrote up as contextual retrieval with big reductions in retrieval failure rates. Hybrid search, because BM25 still catches the exact-match queries embeddings whiff on. And reranking the top 50 down to 5 instead of trusting the ANN index's ordering. Check the MTEB leaderboard before marrying an embedding model; the spread between neighbors on that table is bigger than any prompt tweak you will ever write.

the eval that takes an afternoon, not a quarter

"Score retrieval separately" sounds like a project, so here is the afternoon version that has embarrassed every pipeline I have pointed it at. Collect thirty real user queries from logs. For each, find the passage that SHOULD have been retrieved; yes, by hand, with coffee. That is your labeled set. Now run retrieval and record the rank of the right passage. You get three numbers: how often it appears at all, how often in the top five, and the median rank when it shows up. No framework, one script, and the numbers survive arguments the way "the bot feels dumber" never does.

Rerun the same thirty queries after every retrieval change and you have regression testing. The afternoon set caught my own favorite self-inflicted wound: a chunking "improvement" that lifted average recall while dropping exactly the six queries about tables to rank 40. Averages hide murders. Keep the per-query view.

the corpus is a production system, act like it

Documents change, and most RAG failures I get called about after launch are freshness failures wearing a costume. The 2019 policy in my opening story did not sneak in; it was never evicted. Treat ingestion like any other pipeline: versioned documents, deletion propagated to the index within a deadline someone wrote down, and a canary query per critical document ("what is the current refund window") that fails loudly when the answer goes stale. If your index cannot tell you when a chunk was last verified against its source, you do not have a knowledge base, you have a landfill with an API.

A closing confession for symmetry with the opening story: my own support bot once cited a policy that had never existed in any document. Everyone assumed hallucination, three days went into prompt guardrails, and the actual cause was a chunker that had welded two adjacent policies into one chunk across a page boundary. The model faithfully summarized a document we accidentally wrote. Retrieval does not just fail by missing; it fails by inventing source material, and no prompt on earth defends against a corpus that lies.

The prompt was fine. The prompt was always fine. Go look at your chunks.