Fine-tuning vs RAG: a decision framework

"Should we fine-tune or use RAG?" is usually the wrong question. It frames them as alternatives when they solve different problems. The useful question is: does the model need to know something, or behave differently?

The core distinction

Facts belong in an index. Behavior belongs in weights.

If the answer changes when a document changes, it belongs in retrieval. If the style, format, or reasoning pattern needs to change, it belongs in fine-tuning.

Decision matrix

Need	RAG	Fine-tuning	Both
Answer from private documents	Yes	No	—
Cite sources	Yes	No	—
Frequent content updates	Yes	No	—
Consistent output format	Partial	Yes	Yes
Domain tone / phrasing	Partial	Yes	Yes
Complex domain reasoning	Partial	Yes	Yes
Reduce prompt length	No	Yes	Yes
Lower latency / cost per call	Varies	Yes (small model)	Yes
Access control per user	Yes	No	—

When RAG is the right answer

• Knowledge lives in documents that change.
• You need citations for legal, compliance, or audit reasons.
• Different users should see different documents (permission-aware retrieval).
• You need to add or remove knowledge without retraining.
• You want to know why an answer was produced.

In all of these, fine-tuning is structurally wrong: it bakes facts into weights that can't be updated, audited, or permission-scoped.

When fine-tuning is the right answer

• The model needs a consistent structure — JSON, a specific report format, a legal memo style.
• Domain language is underrepresented (low-resource languages, niche jargon).
• You want a smaller, cheaper model to match a larger model's behavior on a narrow task.
• You need to reduce a long prompt into learned behavior.
• Classification, extraction, or transformation tasks with labelled examples.

Why fine-tuning is not a knowledge store

A fine-tuned model can memorize training examples, but it cannot reliably retrieve a specific clause from a 400-page document, cite it, or tell you when the document changed. Worse, memorization is uneven and unauditable — exactly what regulated industries can't accept.

If your training data goes stale in a week, fine-tuning is a liability, not an asset.

The cost comparison

Cost	RAG	Fine-tuning
Upfront	Pipeline, index, eval	Dataset, training, eval
Ongoing	Embedding + retrieval + generation	Inference (smaller models can be cheaper)
Updates	Re-index a document	Retrain
Failure diagnosis	Inspect retrieved evidence	Opaque
Data needed	A corpus	Labelled input/output pairs

RAG's ongoing cost is usually higher per query. Fine-tuning's ongoing cost is retraining whenever behavior or knowledge needs to change.

The hybrid pattern

Most serious systems use both:

1. Fine-tune for format, tone, and domain reasoning.
2. Retrieve the facts at query time.
3. Verify the output against the retrieved evidence.

In E-Wakil, the legal assistant I built, fine-tuning taught the model legal phrasing and structure. Retrieval supplied the actual statutes. Facts never went into the weights.

How to decide in one hour

1. Write ten example inputs and the ideal outputs.
2. Ask: could a perfect retriever produce the output from documents alone?
3. If yes → RAG.
4. If no → what's missing? Format and style suggest fine-tuning; missing facts suggest your corpus is incomplete.
5. If both → start with RAG, add fine-tuning after retrieval is measured and healthy.

Evaluation is not optional

Fine-tuning without a held-out set is guesswork. Track:

• Task accuracy on unseen examples
• Format compliance
• Regression on general capability (catastrophic forgetting)
• Latency and cost at inference

A model that's better on your task and worse at everything else may be a net loss.