You build a RAG system. The model has access to ten thousand documents. Its answers are vague, occasionally wrong, and sometimes just invented. You spend two weeks iterating on the prompt. The answers are still vague, occasionally wrong, and sometimes invented. The prompt was never the problem.
This is how a lot of RAG projects go. The model is the visible part of the pipeline, the part that produces the answer, so it's what you optimize when the answer is bad. But in a retrieval-augmented system, the model is not the bottleneck. The retrieval is.
What RAG actually is
RAG stands for Retrieval-Augmented Generation. The idea is simple: rather than relying entirely on what the model learned during training (which has a knowledge cutoff, and which the model may misremember), you retrieve relevant documents at inference time and include them in the prompt. The model then generates an answer grounded in that retrieved context.
The pipeline looks like this: a user query comes in. A retrieval system searches a document store and returns the most relevant chunks. Those chunks get inserted into the prompt alongside the query. The model generates an answer using both.
The structure is simple enough that you can prototype a RAG system in an afternoon. The quality of the output depends almost entirely on step two, whether the right chunks come back. When they do, a mediocre model can produce a good answer. When they don't, the best model in the world will either hallucinate or produce an answer that confidently ignores the question.
The retrieval problem
If the wrong chunks come back, the model is in an impossible position. It has been given information that doesn't answer the question and asked to produce an answer anyway. Its options: admit it doesn't know (which requires explicit instruction, and many models won't without it), ignore the retrieved context and answer from training data (hallucination), or produce a vague answer that sounds grounded in the context but isn't.
All three outcomes look like model failures. They are retrieval failures. The model is not at fault. You gave it the wrong information and expected it to know the right answer anyway. Fixing the prompt does nothing because the prompt is not the broken part.
The model cannot compensate for bad retrieval. It can only work with what it's given.
Chunking is underrated
Retrieval works on chunks of text, not full documents. When you index ten thousand documents, you first split them into chunks, embed each chunk, and store the embeddings. At query time, you embed the query and find the nearest chunks.
Chunking strategy has a bigger effect on retrieval quality than most teams expect, and they usually discover this late.
Too small: a single sentence stripped of its surrounding paragraph loses context. A sentence that says "This does not apply in all cases" is meaningless without knowing what "this" refers to. The retrieved chunk is syntactically present but semantically useless to the model. It has the words, not the meaning.
Too large: chunks that are hundreds of paragraphs long dilute relevance. If the answer to the question is in paragraph 3 and the chunk runs from paragraph 1 to 50, the relevant content is buried in noise. Large chunks also push against context window limits and force the model to sift through irrelevant text to find the part that matters.
The practical heuristic is to chunk at natural semantic boundaries, paragraphs or sections, rather than at fixed character counts. A paragraph is usually a coherent unit of meaning. A 512-character slice rarely is. Overlapping chunks (where adjacent chunks share some text at their boundaries) help with cases where the answer spans a chunk boundary, at the cost of some redundancy in the index.
Embeddings are not magic
Embedding-based semantic search works by converting text to a vector of numbers that captures meaning, then finding text whose vector is close to the query's vector. This is powerful for finding paraphrases and conceptually similar text. If the document says "automobile" and the query says "car," semantic search handles that.
It is unreliable for specific factual lookups where exact keywords matter. If you need the document that contains "ISO 27001 certification date" and the document says "ISO 27001 certification date: March 2023," semantic search should find it. But if there are many documents about ISO 27001 certifications and you need the date for one specific entity, embedding similarity may return the wrong one. They're all semantically similar, even though only one has the right date.
For this kind of query, keyword search, specifically BM25, the probabilistic ranking algorithm behind most traditional search engines, often outperforms dense embeddings. BM25 is essentially a smarter version of counting how often query terms appear in candidate documents, weighted by how rare those terms are. It's exact match-friendly in a way that embedding search isn't.
Hybrid search, which combines embedding similarity with BM25 scores (typically by summing weighted ranks from both), usually beats either method alone. The two methods have complementary failure modes: embeddings fail on exact lookups, BM25 fails on paraphrases. Together they cover more ground.
Reranking
Even with a good retrieval setup, the initial ranking isn't always correct. Reranking is a second pass over retrieval results that measurably improves precision, at the cost of latency.
The two-stage setup works like this: first, retrieve 50 candidate chunks cheaply using BM25, embeddings, or hybrid search. This is fast because each candidate is scored independently using precomputed vectors. Then, rerank those 50 candidates using a cross-encoder.
A cross-encoder is a model that takes a query and a document together as input and scores their relevance jointly. Unlike embedding search, where the query and document are encoded separately and compared by vector distance, a cross-encoder can look at both at once, which lets it catch interactions and disambiguations that independent encodings miss. It's slower because there's no precomputation; you're running a forward pass for each query-document pair. But you're only running it on 50 candidates, not on your entire index.
The latency cost is real and often acceptable. The precision improvement is usually significant enough to justify it. If your retrieval is the bottleneck for answer quality (which it probably is), better retrieval is where to invest.
Evaluating retrieval separately from generation
You cannot improve what you don't measure. Most teams measure RAG quality by evaluating the final answer: is it correct? This is necessary but not sufficient. If the answer is wrong, you need to know whether retrieval failed or generation failed, because these require different fixes.
Retrieval metrics measure whether the right information reached the model:
Recall@K: does the correct document chunk appear somewhere in the top K retrieved results? This tells you whether the answer is retrievable at all from your index, before you worry about ranking.Mean Reciprocal Rank (MRR): on average, how high in the ranking does the correct chunk appear? A correct chunk ranked 1st is more useful than the same chunk ranked 10th, because the model attends more to earlier context.Precision@K: of the K retrieved chunks, what fraction are actually relevant? High recall with low precision means you're retrieving the right chunk buried in noise.
These metrics require a labeled test set: queries paired with the correct document chunks. Building this is work. It's the kind of work that pays back immediately: once you have it, you can run retrieval experiments in minutes rather than doing end-to-end evaluations that take hours and mix retrieval and generation signal.
Without measuring retrieval separately, you can't tell whether poor answers are a retrieval problem or a generation problem. You end up doing the equivalent of debugging a two-process pipeline by observing only the output of the second process.
Instrument before you iterate
Before you touch the prompt, instrument the retrieval. Log what chunks are retrieved for each query. Build a small labeled test set; even 50 queries with known correct chunks is enough to start. Compute Recall@K on that test set. If Recall@K is low, your retrieval is broken; fix it. If Recall@K is high but answers are still wrong, now you have a generation problem and prompt iteration is the right tool.
Most teams invert this order. They iterate on the prompt first because it's easy and visible, discover that it doesn't help, and eventually, weeks later, instrument retrieval and find that the relevant chunk was never coming back in the first place.
The model is not hiding the answer from you. It never had it.
Back to writing