Vector Database for RAG: When to Ditch the List (Part 4)

Series: RAG in Python: Zero to Production This is Part 4. Part 1 built RAG from four functions; Part 2 added the hit-rate eval; Part 3 broke the retrieval ceiling with re-ranking. This part answers the question Part 3 deferred: when do you need a real vector database for RAG — and what does hybrid search add on top?

Every time your Part 1 script starts, it re-embeds the entire corpus. That was a feature while you were learning — the whole index was a Python list you could print. But it means every restart costs API calls, nothing persists, and there’s no way to say “only search the 2026 docs.”

By the end of this post your index will live on disk in Chroma, with metadata filters, and retrieve() will keep the exact same contract. You’ll also have a hybrid retriever that catches the error-code and function-name queries pure embeddings fumble. Just as important: you’ll know the three concrete triggers that justify the move, because below them, the list you already have is not a prototype embarrassment — it’s the right tool.

When Do You Actually Need a Vector Database for RAG?#

Before installing anything, be clear on what the list can’t do — because raw speed at small scale usually isn’t the problem. A brute-force cosine scan over a few thousand vectors is fast enough that you won’t feel it. The real triggers are different:

• Persistence. The list dies with the process. Re-embedding a growing corpus on every run costs money and minutes, and it gets worse linearly forever.
• Metadata filtering. “Search only the API docs” or “only documents newer than March” requires storing structured fields next to each vector and filtering before similarity ranking. Bolting that onto a list means reinventing a query engine.
• Scale. Past hundreds of thousands of chunks, brute force stops being cute. A vector database builds an ANNApproximate Nearest Neighbor — an index structure that finds close vectors without comparing against every single one. index so query time stays flat as the corpus grows.

Notice what’s not on the trigger list: “everyone uses one.” The bottom line: a vector database for RAG is justified by persistence, filters, or scale — if none of the three bites you yet, keep the list and spend the effort on retrieval quality instead.

Step 1 — Swap the List for Chroma#

The move should feel anticlimactic, and that’s the point — the four functions survive intact. I default to Chroma for this series because it runs embedded in your Python process with zero infrastructure: no server, no Docker, just a folder on disk.

pip install chromadb rank_bm25

Index once, persist forever:

import chromadb
 
db = chromadb.PersistentClient(path="rag_index")
col = db.get_or_create_collection("docs")
 
# index once — survives restarts
col.add(
    ids=[str(i) for i in range(len(chunks))],
    documents=chunks,
    embeddings=[embed(c).tolist() for c in chunks],
    metadatas=[{"source": s} for s in sources],
)

PersistentClient writes everything under rag_index/, so the next run skips straight to querying. Each chunk carries a metadata dict — that’s what unlocks filtered retrieval. The new retrieve() keeps Part 1’s shape:

def retrieve(question, k=3, where=None):
    res = col.query(
        query_embeddings=[embed(question).tolist()],
        n_results=k,
        where=where,  # e.g. {"source": "api-docs"}
    )
    return res["documents"][0]

Same input, same output — but notice the one honest breaking change: the store argument is gone, because Chroma now holds the store. Downstream callers adapt in one mechanical move: delete store from the call. answer(question, store) becomes answer(question), and Step 3 shows the two-line version of the same fix for the Part 2 eval. This is the swap-the-store move: the architecture didn’t change, only the address of the vectors — and one argument that no longer needs passing.

Step 2 — Add Hybrid Search: BM25 + Vectors, Fused by Rank#

Here’s a failure you’ve probably already met. Ask your system about ERR_QUOTA_429 or a function called chunk_text, and pure vector retrieval often shrugs. Embeddings compress meaning, and rare exact identifiers are exactly what that compression throws away. Meanwhile BM25A decades-old keyword ranking algorithm that scores documents by how often and how distinctively they contain the query's terms. — boring, keyword-based, pre-LLM — nails those queries and can’t handle paraphrase at all. Neither wins alone. Analyses of production RAG failures keep landing on retrieval, not generation, as the thing to fix. So run both and fuse:

The fusion trick is reciprocal rank fusion (RRF). BM25 scores and cosine similarities live on incompatible scales — averaging them is meaningless — so RRF ignores scores entirely and rewards rank positions:

from rank_bm25 import BM25Okapi
 
bm25 = BM25Okapi([c.lower().split() for c in chunks])
 
def rrf(rankings, k=60):
    scores = {}
    for ranking in rankings:
        for rank, doc in enumerate(ranking):
            scores[doc] = scores.get(doc, 0) + 1 / (k + rank + 1)
    return sorted(scores, key=scores.get, reverse=True)
 
def hybrid_retrieve(question, k=3, pool=20):
    vec_hits = retrieve(question, k=pool)
    kw_hits = bm25.get_top_n(question.lower().split(), chunks, n=pool)
    return rrf([vec_hits, kw_hits])[:k]

A chunk ranked #1 by either method gets a big boost; a chunk ranked mid-list by both still beats a chunk only one method liked. That’s the whole algorithm — ten lines, no tuning, no score normalization.

One caveat to keep the persistence story honest: the BM25 index doesn’t persist — it’s rebuilt from chunks at startup. That’s fine, because building it is a cheap local computation, not an API bill; re-embedding was the expensive part. On a fresh run, pull the chunks back out of Chroma instead of re-reading files: chunks = col.get()["documents"].

The bottom line: hybrid search isn’t an alternative to your vector store — it’s a second, cheap ranking that rescues the exact-match queries embeddings systematically miss.

Step 3 — Measure It (Same Eval, New Retriever)#

You know the Part 2 discipline: change one thing, re-run hit_rate, watch the number. The eval needs the same one-line adaptation as everything else — drop store — plus one upgrade that makes it better: take the retriever as a parameter, so the same function scores both:

def hit_rate(gold, k=3, retriever=retrieve):
    hits = 0
    for question, needle in gold:
        retrieved = " ".join(retriever(question, k=k)).lower()
        hits += needle.lower() in retrieved
    return hits / len(gold)
 
print(f"vector only: {hit_rate(gold):.0%}")
print(f"hybrid     : {hit_rate(gold, retriever=hybrid_retrieve):.0%}")

Two lines changed from Part 2: store is gone, and retriever is now an argument. Where you should expect movement: questions containing rare exact terms — error codes, config keys, product names, function names. Where you shouldn’t: purely conceptual questions, which vector search already handled.

Common mistake: benchmarking hybrid search on a gold set with no keyword-style questions in it. The fused retriever will score the same as plain vector search and you’ll conclude hybrid “does nothing.” Your gold set has to contain the failure you’re trying to fix — add a few identifier-heavy questions before you judge.

And the Part 3 reranker still applies afterwards: hybrid widens what makes the pool, the cross-encoder reorders the pool. They stack, and the eval tells you what each layer earned. As always, any specific numbers you see in posts like this — including mine — are illustrative until you’ve measured your own corpus.

Chroma vs pgvector vs Qdrant: An Honest Decision#

Sooner or later someone asks why you didn’t pick a different database, so here’s the short version of a long argument:

My take: for this series’ scale, Chroma is correct. The moment your app already has Postgres, pgvector is the pragmatic answer — it adds no new system to operate. Qdrant is what you graduate to when retrieval is a real service with real load — not before. Picking the scale-tier database on day one is how you spend your first month on infrastructure instead of retrieval quality. If your corpus is messy PDFs rather than clean text, that’s a parsing problem no database fixes — see the RAGFlow deep-dive.

Quick Recap#

• Three triggers — persistence, metadata filters, scale — justify a vector database; below them, the list is fine.
• Chroma is a drop-in: PersistentClient + a collection, and retrieve() keeps its contract.
• Pass your own embeddings to Chroma, or index and query vectors won’t share a space.
• Hybrid search = BM25 + vectors fused by RRF — rank-based, no score normalization, ten lines.
• Measure with a gold set that contains keyword-style questions, or hybrid will look useless.
• Chroma → pgvector → Qdrant is a graduation path, not a day-one menu.

Frequently Asked Questions#

When should I move to a vector database? When re-embedding on every run hurts, when you need metadata filters, or when the corpus outgrows a brute-force scan. Not because a tutorial told you to.

What is hybrid search in RAG? BM25 keyword ranking and vector ranking run in parallel, fused by reciprocal rank fusion — exact terms and semantic matches both make the pool.

Do I still need the Part 3 reranker? Yes — hybrid changes what gets retrieved, the reranker changes the order. They fix different failures and stack cleanly.

Is a Python list ever the right store? Absolutely — for learning, prototypes, and small stable corpora it’s simpler, debuggable, and has no moving parts.

Conclusion#

Part 4 closes the storage question the series opened deliberately naive. The list taught you what the database does, so you can now adopt a vector database for RAG for reasons instead of fashion. The index persists, filters work, exact-term queries stop falling through — and the four functions from Part 1 are still recognizably the same system.

Which trigger pushed you off the list — persistence, filters, or scale — or are you happily still on it? Tell me in the comments.

Read next: Semantic Chunking & Re-Ranking (Part 3) — the retrieval-quality layer this store now serves.