Reranking explained: the cheapest quality win in RAG

Bi-encoders embed the query and each document separately. That's fast, but it means the model never sees them together, so it can't reason about interaction. A reranker does see them together — and that's why it fixes so many retrieval failures.

The pattern

candidates = hybrid_search(query, k=50)     # fast, high recall
ranked     = rerank(query, candidates, k=8) # slow, high precision
context    = build_context(ranked)

Retrieve broadly, then rank narrowly. You get recall from the first stage and precision from the second.

Bi-encoder vs cross-encoder

	Bi-encoder	Cross-encoder
Input	Query and document separately	Query + document together
Precompute	Yes — embed the corpus once	No — score per pair
Speed	Fast at scale	Slow at scale, fine for 50
Accuracy	Good	Better
Use	First-stage retrieval	Second-stage reranking

Why it helps so much

Cross-encoders catch things bi-encoders miss:

• Negation ("does not apply")
• Entity binding (which party the clause refers to)
• Exact constraint matching ("within 14 days")
• Near-duplicate passages that differ in one critical word

Latency and cost

Reranking 50 passages with a cross-encoder typically adds 100–400 ms depending on model and hardware. That's usually acceptable for interactive use.

Ways to control it:

• Rerank 20–30 candidates instead of 100.
• Use a smaller reranker for first pass, a larger one for the top 10.
• Cache rerank scores for repeated queries.
• Run reranking only when the first-stage margin is small.

How to measure it

Compare retrieval metrics with and without reranking on the same golden set:

before = evaluate(rankings_hybrid, qrels, k=5)
after  = evaluate(rankings_reranked, qrels, k=5)
print(before)
print(after)

Watch precision@5 and nDCG@5. Recall at high k may not change much — reranking mostly improves ordering, not coverage.

When not to rerank

• Latency budget under ~300 ms total
• Very small corpora where the retriever is already precise
• Queries that are exact lookups (IDs, codes)
• Cost-sensitive, high-volume consumer chat