Citations and refusal: the two features that make RAG safe

A RAG system that answers fluently but can't prove its answer is a demo. In legal, compliance, finance, and healthcare, the answer matters less than the evidence behind it.

Two features separate a toy from a system you can deploy: verifiable citations and calibrated refusal. Neither is a prompt trick. Both are architecture.

Citation is not attribution

Most "citation" implementations just append the source document to the answer. That tells you where the model looked. It does not tell you whether the answer is actually supported.

	Attribution	Verification
What it does	Links to a retrieved chunk	Checks the claim against the chunk
Catches hallucination	No	Yes
Implementation	Metadata passthrough	Entailment or quote matching
Cost	Near zero	One extra model call (or cheap heuristic)

Attribution is necessary but not sufficient. A confident false claim with a plausible-looking footnote is worse than no citation at all, because it borrows credibility it hasn't earned.

Level 1: span-level attribution

Require the model to return, for each claim, the exact supporting text:

{
  "claim": "Either party may terminate with 30 days notice.",
  "source_id": "contract-882::section-4.2",
  "quote": "Either party may terminate this Agreement upon thirty (30) days written notice."
}

Then verify the quote exists in the retrieved chunk. If it doesn't, the claim is unsupported — regardless of how plausible it sounds.

This catches fabricated quotes, drifted paraphrases, and claims attributed to the wrong section.

Level 2: entailment verification

For claims that don't map to a direct quote, check whether the passage entails the claim.

• Use a natural language inference (NLI) model, or
• Use an LLM judge with a strict rubric, or
• Use both and flag disagreements for human review.

prompt = """Given the passage, is the claim fully supported?
Return one of: SUPPORTED, PARTIALLY_SUPPORTED, NOT_SUPPORTED.

Passage: {passage}
Claim: {claim}
"""

Treat PARTIALLY_SUPPORTED as unsupported for regulated use. Partial support is how subtle misstatements survive.

Level 3: refuse when evidence is weak

Refusal is a product decision as much as a model decision. Define, in advance, what counts as "not enough evidence":

• No retrieved chunk above a similarity threshold
• Top chunk is relevant but doesn't contain the answer
• Sources conflict
• Claim verification fails
• The question is outside the corpus's scope or date range

Then make the refusal specific:

"I couldn't find support for this in the provided documents.
The closest section is 4.2, which covers termination but not notice periods."

A specific refusal is useful. A generic "I don't know" trains users to ignore the system.

Calibrating the threshold

You need two labelled sets: questions with answers, and questions deliberately without. Then measure:

Outcome	Meaning
True answer	Answered correctly, evidence found
False answer	Answered, but unsupported — the dangerous case
Correct refusal	Refused because evidence was absent
False refusal	Refused despite evidence — hurts trust and usefulness

Optimize for a low false-answer rate first. False refusals are annoying; false answers are liabilities.

Show your work in the UI

Citations only create trust if users can inspect them:

• Inline markers linked to the exact passage, not the whole document
• Highlight the supporting sentence in the source
• Show the document title, section, page, and effective date
• Let the user open the raw text
• Distinguish "directly stated" from "inferred"

If the user has to hunt for the evidence, the citation is decoration.

Log everything

For every answer, store:

query, retrieved_ids, retrieved_scores, context_hash,
answer, claims, citation_ids, verification_results,
model_version, prompt_version, latency, cost, timestamp

Without this, you cannot reproduce a bad answer, prove compliance, or measure whether a change helped. Audit logs are a feature, not overhead.

What to measure

• Citation precision: what fraction of citations actually support their claim?
• Citation coverage: what fraction of claims have a citation?
• False answer rate: unsupported answers that were not refused.
• Refusal accuracy: correct refusals on unanswerable questions.
• False refusal rate: refusals despite available evidence.

Why teams skip this

Because it's slower to build and it makes the demo look worse. A system that refuses confidently feels less impressive than one that always answers — until the first wrong answer reaches a client, a regulator, or a court.

The teams that ship in regulated industries build this first. It's also what makes the system defensible in a sales conversation.