Why your RAG pilot stalled at 70% accuracy

The pattern is consistent enough to be predictable. A team builds a retrieval-augmented assistant, it works impressively in a demo, it reaches roughly seventy per cent accuracy on real questions, and then it stops improving no matter which model they swap in. The instinct is to blame the model. The model is almost never the problem.

The diagnostic question

Before changing anything, answer this: when the system gives a wrong answer, was the correct information in the retrieved context?

If the right passage was retrieved and the model still answered incorrectly, you have a generation problem — and those are comparatively rare with current models. If the right passage was not retrieved, no model will save you, because the information was never in front of it.

In our experience, somewhere between seventy and eighty-five per cent of failures are retrieval failures. Which means most teams optimising the wrong half of the system.

If the answer was not in the retrieved context, the model was never the variable worth changing.

Problem one: chunking that destroys meaning

The default of splitting documents into fixed-size chunks with a fixed overlap is fine for prose and actively harmful for the structured documents most organisations actually have.

• A table split across two chunks loses its header row, so the numbers become meaningless
• A clause separated from its defined terms becomes ambiguous or misleading
• A procedure split mid-sequence returns steps 4–7 with no indication that 1–3 exist
• A heading separated from its content strands the only signal of what the content is about

The fix is structure-aware chunking: split on document structure — sections, clauses, table boundaries — and carry the heading hierarchy into each chunk as a prefix. On one engagement that change alone moved retrieval precision from 0.61 to 0.84 with no other modifications.

Problem two: an embedding model chosen by default

Most pilots use whatever embedding model appeared in the tutorial. That is usually a general-purpose model trained on general text, and it performs accordingly on domain vocabulary — clinical abbreviations, insurance product codes, internal system names.

You do not necessarily need a fine-tuned model. Two cheaper interventions usually close most of the gap:

• Hybrid search. Combine dense vector similarity with BM25 keyword matching. Exact identifiers, codes and rare terms are precisely what dense embeddings handle worst and keyword search handles best.
• A reranker. Retrieve twenty candidates cheaply, then rerank them with a cross-encoder and pass the top five. This consistently produces the largest single improvement per hour of engineering effort.

Problem three: no evaluation set, so no measurable progress

This is the one that actually keeps teams stuck, because without it every change is judged by spot-checking a handful of questions — which is how you end up making three changes that each help slightly and one that quietly makes things worse.

A useful evaluation set does not need to be large. One hundred to two hundred questions with known correct answers and known source passages is enough to detect meaningful movement. What matters is that it is built from real user questions, including the awkward ones.

Measure retrieval separately from generation — an aggregate score hides which half is failing.

A realistic fortnight

• Days 1–3: Build the evaluation set from real questions; establish a baseline
• Days 4–6: Structure-aware chunking with heading context; re-measure
• Days 7–9: Add hybrid search; re-measure
• Days 10–12: Add a reranker; re-measure
• Days 13–14: Tune the generation prompt and citation enforcement against whatever failures remain

In most engagements this sequence takes a system from around 0.70 to somewhere between 0.88 and 0.94 on groundedness, without changing the foundation model at all.

When it genuinely is the model

Occasionally it is. If retrieval recall is above 0.95 and groundedness is still poor, the generation step is the constraint — usually on tasks requiring multi-step reasoning across several retrieved passages. That is a real limitation and a legitimate reason to move to a more capable model.

But confirm the retrieval numbers first. Upgrading the model to compensate for bad retrieval is expensive, and it works just well enough to disguise the underlying problem for another quarter.