Chunking is a retrieval decision, not a preprocessing step

Most chunking advice is about size: "use 512 tokens with 50 overlap." That's like choosing a database schema by picking a row size. The chunk is the unit your system retrieves, ranks, cites, and bills for. Its boundaries determine what evidence can ever reach the model.

What a chunk actually does

A chunk serves four roles at once:

1. Retrieval unit: it gets embedded and compared to queries.
2. Citation unit: it's what the answer points to.
3. Context unit: it's what the model reads.
4. Cost unit: it's what you pay to embed and to put in the prompt.

A chunk that's great for retrieval can be terrible for citation, and vice versa. That tension is the whole design problem.

The four common strategies

1. Fixed-size chunking

Split every N tokens, with overlap. Simple, fast, predictable.

for i in range(0, len(tokens), size - overlap):
    yield tokens[i:i + size]

Good for: unstructured prose, quick baselines.
Bad for: anything with structure — it splits clauses, tables, and code blocks in the middle.

2. Recursive character splitting

Try paragraph breaks, then sentence breaks, then words, until the chunk fits. This preserves natural boundaries far better than fixed windows and is a reasonable default.

Good for: general documents.
Bad for: documents where the boundary that matters is semantic, not typographic.

3. Semantic chunking

Embed sentences, detect where topic shifts, and cut there. More expensive, and results vary by embedding model.

Good for: long, discursive documents with no clear structure.
Bad for: cost and reproducibility. "Semantic" boundaries are a heuristic, not a guarantee.

4. Structure- and metadata-aware chunking

Parse the document's own hierarchy — headings, sections, clauses, tables — and chunk along it. Attach metadata: document id, section number, page, effective date, document type.

{
  "id": "contract-882::section-4.2",
  "text": "Either party may terminate...",
  "metadata": {
    "doc_id": "contract-882",
    "section": "4.2",
    "page": 12,
    "type": "msa",
    "effective": "2024-01-01"
  }
}

This is what production systems use. The metadata enables filtering before similarity search, which is usually a bigger precision win than switching embedding models.

Chunk size: the real tradeoff

Chunk size	Retrieval	Generation
Small (100–200 tokens)	Precise, but may lack context	Cheap, but fragmented evidence
Medium (300–800)	Balanced	Usually the sweet spot
Large (1,000+)	Recall up, precision down	Expensive, dilutes attention

There is no universal number. The right size depends on how much context a claim needs to be interpretable. A contract clause may need 150 tokens; a policy interpretation may need 800.

Overlap: usually a band-aid

Overlap exists to avoid cutting a sentence in half. But 20% overlap on a 500-token chunk duplicates 100 tokens across neighbors, inflating the index and surfacing near-identical results that crowd out diversity.

Prefer structure-aware boundaries first. Use overlap only when you can't detect them, and keep it to 10–15%.

Parent-child retrieval

A useful pattern: embed small child chunks for precise retrieval, but return the larger parent section to the model for context.

child chunk  → used for similarity search
parent chunk → sent to the LLM, cited in the answer

This decouples the retrieval unit from the context unit — the tension I mentioned earlier.

Tables and code need their own handling

Naive chunking destroys tables. A row without its header is meaningless; a cell without its column is ambiguous.

• Serialize tables with headers repeated per row, or convert to a sentence.
• Keep code blocks intact; never split inside a function.
• Store the table title and caption as metadata for retrieval.

How to know if chunking helped

Measure recall@k on the same golden set before and after. If you changed chunking and didn't measure, you don't know — you have a feeling.

Specifically, check whether the correct passage is now retrievable as a unit. A common failure is that recall stays flat while citation accuracy collapses, because the chunk contains the answer but not the identifying context.

Summary

• Chunking decides what evidence is even reachable.
• Structure beats size. Metadata beats both.
• Decouple the retrieval unit from the context unit when you can.
• Measure with recall@k before and after — every time.