RAG Chunking & Retrieval Quality: Fix Bad Answers (Part 2)

Series: RAG in Python: Zero to Production This is Part 2. In Part 1 we built a working RAG system from four functions — load, chunk, embed, retrieve, answer — that split documents on blank lines. That naive RAG chunking is exactly what we fix here. New here? You only need Part 1’s retrieve() and a folder of text to follow along.

Your RAG system gives a confidently wrong answer, and you blame the model. Nine times out of ten, the model is fine — it answered faithfully from the chunk it was handed, and the chunk was the problem. Retrieval quality, set by how you split your documents, decides whether the right text ever reaches the model. This post fixes the two weakest links from Part 1: the RAG chunking, and your ability to tell whether it improved.

Why Your RAG Chunking Returns Bad Answers#

Part 1 split documents on blank lines — one paragraph, one chunk. It’s the simplest thing that works, and it fails in three specific ways once your documents are real.

• A long paragraph becomes one fuzzy vector. Cram three ideas into one chunk and its embedding averages all three, so it ranks below a shorter, sharper chunk and never gets retrieved.
• A fact splits across a boundary. “Annual plans are non-refundable after 30 days” lands half in one paragraph and half in the next, so no single chunk answers the question.
• A tiny chunk loses its context. A one-line paragraph embeds with no surrounding meaning, matching on keywords instead of intent.

The bottom line: paragraphs are how humans format text, not how retrieval wants it. The fix is to chunk on size, not on punctuation.

Fix 1: Fixed-Size Chunks With Overlap#

Instead of splitting on blank lines, fixed-size RAG chunking slides a window across the raw text and lets each window repeat the tail of the one before it. That repeated tail — the overlap — is what keeps a boundary-straddling fact whole.

def chunk_text(text, size=1500, overlap=200):
    """Split text into overlapping windows of ~size characters."""
    chunks, start = [], 0
    while start < len(text):
        chunks.append(text[start:start + size])
        start += size - overlap        # step back by `overlap` so boundary facts survive
    return chunks

Drop it straight into Part 1’s loader — it’s a one-line change to where the splitting happens:

def load_and_chunk(folder, size=1500, overlap=200):
    chunks = []
    for path in glob.glob(f"{folder}/*.txt"):
        text = open(path, encoding="utf-8").read()
        chunks += chunk_text(text, size, overlap)   # was: text.split("\n\n")
    return chunks

size and overlap are in characters here. A token is roughly four characters, so the size=1500 default is about 375 tokens — right inside the 256–512 sweet spot. Drop toward 800 characters (~200 tokens) for short factual snippets, or push to 2,000 for dense, analytical docs.

Pick a Chunk Size — and Question the Overlap#

There’s no universal best size for RAG chunking, but there is a sane starting range. The 2026 consensus is a 256–512 token chunk (Pinecone’s chunking guide is a good reference): drop toward 128–256 tokens when your questions are short factual lookups, and rise toward 512–1,024 when they need analysis across more context.

Overlap is where I’ll push back on the standard advice. The usual rule is 10–20% overlap, and it does prevent boundary cuts — but it isn’t free. Every overlapping window re-embeds text you already stored, inflating both index size and cost. Some 2026 retrieval benchmarks found overlap gave no measurable accuracy gain on their datasets — so I treat overlap as a hypothesis to test, not a setting to assume. Which brings us to the part most tutorials skip.

Fix 2: Measure Retrieval, Don’t Guess#

You cannot tune what you don’t measure, and “the answers feel better” is not a measurement. The cheapest honest signal is hit rate: build a tiny gold set of questions, each paired with a string that has to appear in a correct answer, and check how often retrieval actually returns it.

# A tiny gold set: (question, a string that must appear in a good chunk)
gold = [
    ("What is the refund window for annual plans?", "30 days"),
    ("Can I cancel mid-term?", "cancel at any time"),
]
 
def hit_rate(gold, store, k=3):
    hits = 0
    for question, needle in gold:
        retrieved = " ".join(retrieve(question, store, k)).lower()
        hits += needle.lower() in retrieved
    return hits / len(gold)
 
print(f"hit rate@3: {hit_rate(gold, store):.0%}")

Now changing a setting becomes an experiment, not a guess. The jump can be dramatic: on a small gold set, paragraph chunking might score hit rate@3 = 45% while fixed-size chunk_text(size=1500, overlap=200) reaches 85% on the same questions and model. Treat those figures as illustrative — your numbers will differ, and that’s the point. Re-chunk, re-embed, and watch the number move — that loop is how you actually improve retrieval.

One honest limit: in is exact substring matching, so a needle of “30 days” won’t match “thirty days” — hit rate is a smoke test, not ground truth. Its stricter cousins, recall@k and precision@k, measure how much of the relevant text you retrieved and how much junk came with it. For a first pass hit rate catches the big regressions, but I keep a proper eval harness for anything headed to production.

What Still Breaks — and Part 3#

Fixed-size chunking with a measured size and overlap will fix most bad answers. It won’t fix all of them. Splitting by size still ignores meaning, so a window can end mid-thought. Tables and PDFs parse into garbage that no chunk size rescues. And when two documents each hold half an answer, top-k retrieval over flat chunks struggles.

The bottom line: fixed-size chunking is the floor, not the ceiling. The upgrades — semantic chunking that splits on meaning, hierarchical chunking that retrieves small but generates from larger parents, and re-ranking the retrieved set — are Part 3. Reach for them only after your hit-rate eval proves fixed-size has run out of room.

Quick Recap#

• Paragraph chunking fails three ways: fuzzy long chunks, split facts, contextless tiny chunks.
• Fixed-size windows with overlap keep boundary facts whole.
• Start at 256–512 tokens; size down for facts, up for analysis.
• Overlap is a hypothesis — add it, then confirm it helps.
• Hit rate turns tuning into measurement instead of guesswork.

Frequently Asked Questions#

What is the best chunk size for RAG? A 256–512 token chunk (~1,000–2,000 characters) is the common starting point. Go smaller for factual lookups, larger for analytical questions, and tune against an eval.

Should RAG chunks overlap? Usually start with 10–20% to protect boundary facts, but it adds cost and some 2026 benchmarks show no gain — confirm it helps with a hit-rate eval.

Why does my RAG give wrong answers with a good model? The answer is decided at retrieval. Hand the model the wrong chunk and it answers from the wrong text — it’s a chunking problem, not a model one.

How do I measure retrieval quality? Hit rate against a small gold set: the fraction of questions whose retrieved chunks contain a known answer string. Recall@k and precision@k go deeper.

What’s better than fixed-size chunking? Semantic and hierarchical chunking score higher but cost more — graduate to them in Part 3 once your eval proves you need it.

Conclusion#

Part 1 got RAG working; Part 2 makes it correct. Fixed-size chunks with overlap stop facts from being split, a sensible chunk size keeps each vector sharp, and a ten-line hit-rate eval turns “feels better” into a number you can move. The pattern that matters most in RAG chunking isn’t a setting — it’s the loop: change one thing, measure, repeat.

What’s your hit rate right now — and which knob moved it most, size, overlap, or k? Tell me in the comments. If you haven’t built the base system yet, start with Part 1.

Read next: Semantic Chunking & Re-Ranking (Part 3) — the upgrades that break the fixed-size ceiling.