Choosing an embedding model for RAG (and measuring it)

Embedding leaderboards rank models on generic benchmarks. Your corpus is not a generic benchmark. A model that wins on average can lose badly on your documents — especially if they contain identifiers, domain jargon, or a low-resource language.

What actually matters

Factor	Why it matters
Domain fit	Legal, medical, and code corpora have different vocabularies
Language coverage	Nepali and mixed-script text are often weak spots
Dimensions	Affects storage, memory, and search speed
Max sequence length	Long chunks get truncated silently
Cost	Embedding the corpus is a one-time cost; queries are ongoing
Latency	Query embedding sits on the critical path
Hosting	API vs local changes privacy and ops

Dimensions are a tradeoff, not a quality score

Higher-dimensional embeddings can capture more nuance but cost more storage and slower search. Many modern models offer reduced dimensions with small quality loss.

For a million chunks at 1,536 dimensions in float32, the vectors alone are roughly 6 GB. At 384 dimensions, about 1.5 GB. Plan for this before you pick.

Why generic benchmarks mislead

• They optimize for retrieval over web text, not contracts or statutes.
• They rarely test exact identifiers (section numbers, SKUs).
• They may not cover your language or script mix.
• They average across query types; your users may be skewed.

How to evaluate on your corpus

1. Take 50 real queries with known relevant chunks.
2. Embed the corpus with each candidate model.
3. Compute recall@5, recall@10, and MRR.
4. Add a hybrid BM25 baseline for comparison.
5. Measure query latency and cost.
6. Pick the best recall per dollar, not the best leaderboard rank.

for model in candidates:
    index = DenseIndex(documents, embed=model)
    rankings = {qid: index.search(q, k=10) for qid, q in queries.items()}
    print(model.name, evaluate(rankings, qrels, k=5))

When to combine with BM25

Almost always. Embeddings handle paraphrase; BM25 handles exact terms. Hybrid retrieval is the default for document-heavy domains. See hybrid search.

Fine-tuning embeddings

If domain recall is weak after trying good models, fine-tuning a bi-encoder on query–passage pairs can help. Requirements:

• Several hundred to several thousand labelled pairs
• Hard negatives (near-miss passages), not random ones
• A held-out evaluation set
• Willingness to re-embed the corpus when the model changes