RAG evaluation metrics that actually tell you something

Most RAG projects are evaluated by reading a few answers and deciding they look fine. That works until a change makes one question better and three worse, and nobody notices.

Evaluation has two halves: retrieval (did we find the right evidence?) and generation (did we use it correctly?). Measure them separately. A single end-to-end score tells you the system is worse but not where.

The golden set

Everything depends on this. Build 25–50 real questions from real users, each with:

• The question, phrased the way a user would ask it
• The document(s) that contain the answer
• Ideally, the exact passage or section
• The expected answer, for generation checks

Include the ugly cases: ambiguous questions, questions with no answer in the corpus, questions where two documents conflict, and questions with exact identifiers.

Retrieval metrics

Recall@k

Of all relevant documents, how many appear in the top k results?

recall@k = |relevant ∩ top_k| / |relevant|

This is the most important retrieval metric for RAG. If recall@k is low, the generator cannot possibly answer correctly. Fix this before anything else.

Pitfall: with a single relevant document, recall@k is binary — 1 or 0. That's fine, but average across many queries before drawing conclusions.

Precision@k

Of the top k results, how many are relevant?

precision@k = |relevant ∩ top_k| / k

Precision matters when context space is expensive. If you retrieve 20 chunks and only two are useful, you're paying tokens for noise and increasing distraction.

MRR (Mean Reciprocal Rank)

How high did the first relevant document rank?

RR = 1 / rank_of_first_relevant
MRR = mean(RR) across queries

MRR is useful for question answering, where one good passage is often enough. It ignores additional relevant documents, so pair it with recall.

nDCG@k

Normalized discounted cumulative gain rewards relevant documents for appearing near the top and handles multiple relevant documents with graded relevance.

DCG@k  = Σ (relevance_i / log2(i + 1))
nDCG@k = DCG@k / IDCG@k

Use nDCG when some documents are more relevant than others. Use binary relevance if you can't reliably grade — a bad graded label set is worse than none.

Hit rate

Did any relevant document appear in the top k? Useful as a sanity check, but it hides how badly you ranked the rest.

Generation metrics

Faithfulness

Is every claim in the answer supported by the retrieved context? This is the metric that catches hallucination. It is distinct from correctness: an answer can be faithful to bad context and still be wrong.

Two ways to measure it:

• LLM-as-judge: ask a strong model to mark each claim as supported or not, given the context. Cheap, scalable, imperfect.
• Human review: slower, but essential for calibrating the judge.

Answer relevance

Does the answer actually address the question? A faithful answer can still dodge the question by summarizing the context instead of answering.

Refusal accuracy

When the answer isn't in the corpus, does the system refuse? Measure this with a set of deliberately unanswerable questions. Most systems fail here, and it's usually invisible because nobody tests for it.

Operational metrics

Metric	Why it matters
p50 / p95 latency	Retrieval and reranking dominate user-perceived delay
Cost per query	Embedding + rerank + generation + vector DB reads
Index freshness	Stale documents produce confidently wrong answers
Refusal rate	A sudden drop can mean retrieval silently degraded

Common mistakes

1. Evaluating end-to-end only. You learn the system got worse, not why.
2. Using the test set to tune. It stops being a test. Hold out a final set.
3. Too few queries. Ten questions is an anecdote. Aim for 50+.
4. Only easy questions. Include the failures you already know about.
5. Ignoring refusal. Unanswerable questions are where trust is won or lost.
6. Chasing a single number. Recall, faithfulness, and latency trade off. Decide what matters.

A minimal implementation

You don't need a platform. You need labelled data and ~200 lines of code. Here's the shape:

from rag_eval import load_dataset, build_index, evaluate

dataset = load_dataset("corpus.jsonl", "queries.jsonl")
index = build_index("hybrid", dataset.documents, alpha=0.5)

rankings = {
    qid: [doc for doc, _ in index.search(query, k=10)]
    for qid, query in dataset.queries.items()
}

print(evaluate(rankings, dataset.qrels, k=5))

The RAG Eval Kit implements this with BM25, a dense baseline, and hybrid retrieval, with 31 tests covering the metric edge cases.