Vector databases for RAG: what actually matters

Vector databases are the least interesting decision in RAG — and the one teams spend the most time on. For most systems, any competent option works. The differences that matter are filtering, hybrid support, and operational fit.

First: do you need one?

Under ~100,000 chunks, an in-process index (FAISS, NumPy, or a simple inverted index) is often enough. A dedicated vector database adds operational surface you may not need.

You need one when you have:

• Millions of vectors
• Multi-tenant access control
• Frequent updates and deletes
• Metadata filtering at scale
• Multiple services querying concurrently

The options

Option	Strength	Watch out for
pgvector	One database; SQL joins; transactional	ANN performance at very large scale
Qdrant	Strong filtering; payload indexes; self-hostable	Extra service to operate
Weaviate	Hybrid built in; schema-rich	More opinionated; heavier
Pinecone	Managed; low ops	Cost; vendor lock-in
FAISS	Fast; embeddable; free	No persistence, filtering, or auth

Filtering is the real differentiator

In document RAG you almost always filter before or during search: by tenant, document type, date, section, or access-control list.

Pre-filtering that isn't supported efficiently forces you to over-retrieve and post-filter, which destroys recall and leaks information.

# Pre-filter: only documents this user may see, of the right type
results = index.search(
    vector=query_vector,
    filter={"tenant_id": user.tenant, "type": "policy"},
    limit=50,
)

Hybrid support

Some databases support BM25 or sparse vectors natively; others require a separate lexical index. Either is fine — but know which you're building. See hybrid search.

Operations

• Backups: can you restore the index and metadata together?
• Re-indexing: how long to re-embed the corpus after a model change?
• Deletes: do deleted documents actually disappear from results?
• Multi-tenancy: is isolation enforced in the query path?
• Observability: can you see query latency and recall per tenant?

A sane default

If you're already on Postgres, start with pgvector. You get transactions, joins, backups, and one fewer service to run. Move to a dedicated engine when you hit a measured limit, not a hypothetical one.