Retrieval evaluation (RAG)

How it works

In a RAG system the model can only be as good as what it was handed. Retrieval evaluation isolates that step and scores it with the information-retrieval metrics that predate LLMs entirely: recall@k (was the answer-bearing passage retrieved at all), precision and nDCG (how much of the context window is wasted on irrelevance, and how well the useful passages rank), and hit rate across a labelled query set. Because it needs no generation, it is cheap and fully deterministic, and it separates two failures that look identical from the outside — the model ignored good context, versus the model never got the context. Chunking strategy, embedding model, index configuration and re-ranking are all tuned against these numbers rather than against vibes downstream.

When to use it

Any retrieval-augmented system: before tuning prompts, when changing chunking, embedding model, or re-ranking, and as a standing regression gate on the knowledge base as it grows — a corpus that doubles in size can quietly change recall@k even with the retriever's configuration untouched. Also the cheapest first stop when a RAG system 'hallucinates': ruling out a retrieval miss costs an offline metric run, not an investigation into the model.

Limitations

Needs a labelled query→relevant-passage set, which is real annotation work and ages as the corpus changes — new documents added to the index have no labels until someone writes them. Good retrieval scores don't guarantee a good answer — the model still has to use what it was given, which is what groundedness checking measures.

Cite this