RAG architecture from scratch: components and data flow

Strip away the framework names and RAG is a simple loop: turn documents into searchable units, find the units relevant to a question, put them in a prompt, and generate an answer. Everything else is engineering around failure modes.

Two pipelines, not one

Offline (ingestion): parse → clean → chunk → embed → index.

Online (query): rewrite → retrieve → filter → rerank → assemble context → generate → verify.

Conflating them is a common design error. Ingestion is batch, idempotent, and cheap to retry. Query is latency-sensitive and must degrade gracefully.

Components

Layer	Responsibility	Typical failure
Parser	Extract text and structure	Lost tables, broken reading order
Chunker	Create retrieval units	Splits clauses from context
Embedder	Map text to vectors	Weak on identifiers and jargon
Index	Store and search vectors	Stale data, no metadata filters
Retriever	Return candidates	Low recall at the chosen k
Reranker	Reorder candidates	Too slow, or never added
Assembler	Build the context	Redundancy, wrong ordering
Generator	Write the answer	Unsupported claims
Verifier	Check claims against evidence	Missing entirely
Evaluator	Measure all of the above	Never built

A minimal data model

{
  "chunk_id": "doc-42::s3.1::c2",
  "doc_id": "doc-42",
  "text": "...",
  "metadata": {
    "title": "Master Services Agreement",
    "section": "3.1",
    "page": 7,
    "type": "msa",
    "effective": "2024-01-01",
    "acl": ["legal-team"]
  },
  "embedding": [0.0123, -0.0441, ...]
}

The acl field matters: retrieval must filter by what the user is allowed to see before ranking. Post-filtering leaks information through scores and counts.

The query path

1. Normalize the question. Resolve pronouns from conversation history.
2. Classify intent. Is this answerable from documents at all?
3. Rewrite for retrieval. Expand acronyms, add domain synonyms.
4. Retrieve broadly. Hybrid search, k = 30–50.
5. Filter. Metadata and permissions.
6. Rerank. Cross-encoder down to 5–8.
7. Assemble. Deduplicate, order strongest-first, label sources.
8. Generate. Grounded prompt with explicit refusal instruction.
9. Verify. Check claims against the exact passages.
10. Log. Store everything for audit and evaluation.

What to build first

Resist the urge to start with generation. Build in this order:

1. Parser and chunker with metadata
2. A 25-question golden set
3. BM25 baseline + recall@k
4. Dense retrieval + comparison
5. Hybrid + reranking
6. Generation with citations
7. Verification and refusal
8. Latency and cost optimization

Each step has a measurable acceptance criterion. If you can't measure it, don't build it yet.

Latency budget

For an interactive system, aim for:

• Query rewrite: < 150 ms (or skip)
• Retrieval: < 200 ms
• Rerank: < 300 ms
• Generation: 1–4 s (streamed)
• Verification: 0.3–1 s (or async)

Verification can run asynchronously for low-risk use cases, but must block for regulated ones.