Context window budgeting for RAG: more is not better

A large context window is not an invitation to fill it. Beyond a point, extra context increases cost and latency while decreasing accuracy — models attend unevenly and get distracted by near-duplicates.

What goes into the prompt

Component	Typical share
System instructions	5–10%
Retrieved evidence	60–80%
Conversation history	5–20%
Output reserve	10–20%

Reserve output tokens first. A prompt that fills the window leaves the model no room to answer.

Deduplicate before you assemble

Hybrid retrieval often returns three near-identical chunks from overlapping windows. They consume budget and crowd out diverse evidence.

def dedupe(chunks, threshold=0.92):
    kept = []
    for c in chunks:
        if all(similarity(c.text, k.text) < threshold for k in kept):
            kept.append(c)
    return kept

Order matters

Models attend more reliably to the beginning and end of long contexts. Put the strongest evidence first. If you must include weaker context, place it in the middle.

Do not sort by document order if relevance differs — sort by relevance, then group by source for citation.

Label every chunk

[SOURCE 1] Master Services Agreement, §4.2, p.12 (effective 2024-01-01)
Either party may terminate this Agreement upon thirty (30) days written notice.

[SOURCE 2] Amendment No. 3, §1, p.1 (effective 2025-06-01)
Section 4.2 is amended to require sixty (60) days written notice.

Without labels, the model cannot cite, and you cannot audit. Labels also help the model resolve conflicts between documents.

Handle conflicting evidence explicitly

Real corpora contain contradictions: an original clause and its amendment, two policies from different years. Instruct the model:

If sources conflict, prefer the most recent effective date and state that a conflict exists. Do not silently choose one.

A token budget example

max_context      = 8_000
system_tokens    =   500
history_tokens   = 1_000
output_reserve   = 1_500
evidence_budget  = max_context - system - history - output
# 5,000 tokens for evidence → ~6–10 chunks at 500–800 tokens

Signals you're over-stuffing

• Answers cite the first chunk and ignore the rest
• Quality drops as k increases
• Latency and cost rise without accuracy gains
• The model repeats or blends conflicting sources