Retrieval evaluation (RAG)

How it works

In a RAG system the model can only be as good as what it was handed. Retrieval evaluation isolates that step and scores it with the information-retrieval metrics that predate LLMs entirely: recall@k (was the answer-bearing passage retrieved at all), precision and nDCG (how much of the context window is wasted on irrelevance, and how well the useful passages rank), and hit rate across a labelled query set. Because it needs no generation, it is cheap and fully deterministic, and it separates two failures that look identical from the outside — the model ignored good context, versus the model never got the context. Chunking strategy, embedding model, index configuration and re-ranking are all tuned against these numbers rather than against vibes downstream.

When to use it

Any retrieval-augmented system: before tuning prompts, when changing chunking/embedding/re-ranking, and as a regression gate on the knowledge base as it grows.

Limitations

Needs a labelled query→relevant-passage set, which is real annotation work and ages as the corpus changes. Good retrieval scores don't guarantee a good answer — the model still has to use what it was given, which is what groundedness checking measures.

Cite this