# RAG Chunking & Retrieval Quality: Fix Bad Answers (Part 2)
> Why paragraph chunking returns bad answers, how fixed-size chunks with overlap fix it, how to pick a chunk size, and a 10-line hit-rate eval so you tune retrieval by measuring instead of guessing.
*Source: https://www.infowok.com/rag-chunking-retrieval-quality-part-2/ · Sukhveer Kaur · Published June 30, 2026*
---
> **Series: RAG in Python: Zero to Production**
> This is Part 2. In [Part 1](/build-a-rag-system-in-python-part-1/) we built a working RAG system from four functions — load, chunk, embed, retrieve, answer — that split documents on blank lines. That naive RAG chunking is exactly what we fix here.
> New here? You only need Part 1's `retrieve()` and a folder of text to follow along.
Your RAG system gives a confidently wrong answer, and you blame the model. Nine times out of ten, the model is fine — it answered faithfully from the chunk it was handed, and **the chunk was the problem.** Retrieval quality, set by how you split your documents, decides whether the right text ever reaches the model. This post fixes the two weakest links from Part 1: the RAG chunking, and your ability to tell whether it improved.
- You built the [from-scratch RAG system in Part 1](/build-a-rag-system-in-python-part-1/) — we extend its `load_and_chunk` and reuse `retrieve()`
- You know what RAG is — the [RAG explainer](/what-is-rag-complete-guide-2026/) covers the retrieve-augment-generate core
- A loose grasp of embeddings — the [embeddings primer](/embeddings-vector-search-primer/) is the warm-up
- **Bad RAG answers are usually bad chunks,** not a bad model.
- **Fixed-size chunks with overlap** stop facts from being split across a boundary.
- **Chunk size is a dial:** 256–512 tokens is the sweet spot, smaller for facts, larger for analysis.
- **Measure retrieval with a hit-rate eval** so you tune by evidence, not vibes.
## Why Your RAG Chunking Returns Bad Answers
Part 1 split documents on blank lines — one paragraph, one chunk. It's the simplest thing that works, and it fails in three specific ways once your documents are real.
- **A long paragraph becomes one fuzzy vector.** Cram three ideas into one chunk and its embedding averages all three, so it ranks below a shorter, sharper chunk and never gets retrieved.
- **A fact splits across a boundary.** "Annual plans are non-refundable after 30 days" lands half in one paragraph and half in the next, so no single chunk answers the question.
- **A tiny chunk loses its context.** A one-line paragraph embeds with no surrounding meaning, matching on keywords instead of intent.
**The bottom line: paragraphs are how humans format text, not how retrieval wants it.** The fix is to chunk on size, not on punctuation.
## Fix 1: Fixed-Size Chunks With Overlap
Instead of splitting on blank lines, fixed-size RAG chunking slides a window across the raw text and lets each window repeat the tail of the one before it. That repeated tail — the overlap — is what keeps a boundary-straddling fact whole.
```python
def chunk_text(text, size=1500, overlap=200):
"""Split text into overlapping windows of ~size characters."""
chunks, start = [], 0
while start < len(text):
chunks.append(text[start:start + size])
start += size - overlap # step back by `overlap` so boundary facts survive
return chunks
```
Drop it straight into Part 1's loader — it's a one-line change to where the splitting happens:
```python
def load_and_chunk(folder, size=1500, overlap=200):
chunks = []
for path in glob.glob(f"{folder}/*.txt"):
text = open(path, encoding="utf-8").read()
chunks += chunk_text(text, size, overlap) # was: text.split("\n\n")
return chunks
```
`size` and `overlap` are in **characters** here. A token is roughly four characters, so the `size=1500` default is about 375 tokens — right inside the 256–512 sweet spot. **Drop toward 800 characters (~200 tokens) for short factual snippets, or push to 2,000 for dense, analytical docs.**
Raw character slicing can cut mid-word or mid-sentence. For production, split on sentence boundaries first and pack sentences up to `size` — same idea, cleaner edges. The character version is here so you see the mechanism with zero dependencies.
## Pick a Chunk Size — and Question the Overlap
There's no universal best size for RAG chunking, but there is a sane starting range. **The 2026 consensus is a 256–512 token chunk** ([Pinecone's chunking guide](https://www.pinecone.io/learn/chunking-strategies/) is a good reference): drop toward 128–256 tokens when your questions are short factual lookups, and rise toward 512–1,024 when they need analysis across more context.
Overlap is where I'll push back on the standard advice. The usual rule is 10–20% overlap, and it does prevent boundary cuts — but it isn't free. Every overlapping window re-embeds text you already stored, inflating both index size and cost. **Some 2026 retrieval benchmarks found overlap gave no measurable accuracy gain on their datasets** — so I treat overlap as a hypothesis to test, not a setting to assume. Which brings us to the part most tutorials skip.
## Fix 2: Measure Retrieval, Don't Guess
You cannot tune what you don't measure, and "the answers feel better" is not a measurement. The cheapest honest signal is **hit rate**: build a tiny gold set of questions, each paired with a string that has to appear in a correct answer, and check how often retrieval actually returns it.
```python
# A tiny gold set: (question, a string that must appear in a good chunk)
gold = [
("What is the refund window for annual plans?", "30 days"),
("Can I cancel mid-term?", "cancel at any time"),
]
def hit_rate(gold, store, k=3):
hits = 0
for question, needle in gold:
retrieved = " ".join(retrieve(question, store, k)).lower()
hits += needle.lower() in retrieved
return hits / len(gold)
print(f"hit rate@3: {hit_rate(gold, store):.0%}")
```
Now changing a setting becomes an experiment, not a guess. The jump can be dramatic: on a small gold set, paragraph chunking might score **hit rate@3 = 45%** while fixed-size `chunk_text(size=1500, overlap=200)` reaches **85%** on the same questions and model. Treat those figures as illustrative — your numbers will differ, and that's the point. **Re-chunk, re-embed, and watch the number move** — that loop is how you actually improve retrieval.
One honest limit: `in` is exact substring matching, so a needle of "30 days" won't match "thirty days" — hit rate is a smoke test, not ground truth. Its stricter cousins, recall@k and precision@k, measure *how much* of the relevant text you retrieved and how much junk came with it. For a first pass hit rate catches the big regressions, but I keep a [proper eval harness](/ai-agent-evaluation-metrics-frameworks-2026/) for anything headed to production.
You don't need a benchmark dataset. Twenty real questions with their expected answer strings will tell you more about your chunking than any blog's default settings — including this one's.
## What Still Breaks — and Part 3
Fixed-size chunking with a measured size and overlap will fix most bad answers. It won't fix all of them. Splitting by size still ignores meaning, so a window can end mid-thought. Tables and PDFs parse into garbage that no chunk size rescues. And when two documents each hold half an answer, top-k retrieval over flat chunks struggles.
**The bottom line: fixed-size chunking is the floor, not the ceiling.** The upgrades — semantic chunking that splits on meaning, hierarchical chunking that retrieves small but generates from larger parents, and re-ranking the retrieved set — are [Part 3](/semantic-chunking-reranking-rag-part-3/). Reach for them only after your hit-rate eval proves fixed-size has run out of room.
## Quick Recap
- **Paragraph chunking fails** three ways: fuzzy long chunks, split facts, contextless tiny chunks.
- **Fixed-size windows with overlap** keep boundary facts whole.
- **Start at 256–512 tokens;** size down for facts, up for analysis.
- **Overlap is a hypothesis** — add it, then confirm it helps.
- **Hit rate turns tuning into measurement** instead of guesswork.
## Frequently Asked Questions
**What is the best chunk size for RAG?** A 256–512 token chunk (~1,000–2,000 characters) is the common starting point. Go smaller for factual lookups, larger for analytical questions, and tune against an eval.
**Should RAG chunks overlap?** Usually start with 10–20% to protect boundary facts, but it adds cost and some 2026 benchmarks show no gain — confirm it helps with a hit-rate eval.
**Why does my RAG give wrong answers with a good model?** The answer is decided at retrieval. Hand the model the wrong chunk and it answers from the wrong text — it's a chunking problem, not a model one.
**How do I measure retrieval quality?** Hit rate against a small gold set: the fraction of questions whose retrieved chunks contain a known answer string. Recall@k and precision@k go deeper.
**What's better than fixed-size chunking?** Semantic and hierarchical chunking score higher but cost more — graduate to them in [Part 3](/semantic-chunking-reranking-rag-part-3/) once your eval proves you need it.
## Conclusion
Part 1 got RAG working; Part 2 makes it correct. Fixed-size chunks with overlap stop facts from being split, a sensible chunk size keeps each vector sharp, and a ten-line hit-rate eval turns "feels better" into a number you can move. The pattern that matters most in RAG chunking isn't a setting — it's the loop: change one thing, measure, repeat.
**What's your hit rate right now — and which knob moved it most, size, overlap, or k?** Tell me in the comments. If you haven't built the base system yet, start with Part 1.
**Read next: [Semantic Chunking & Re-Ranking (Part 3)](/semantic-chunking-reranking-rag-part-3/)** — the upgrades that break the fixed-size ceiling.
- **Bad retrieval on hard questions?** [Agentic RAG](/agentic-rag-vs-static-rag-2026/) adds a grade-and-retry loop on top of better chunking.
- **Going to production?** A real [evaluation harness](/ai-agent-evaluation-metrics-frameworks-2026/) extends the hit-rate idea to faithfulness and answer quality.
- **Coming next in this series:** [Part 3 — Semantic Chunking & Re-Ranking](/semantic-chunking-reranking-rag-part-3/), the upgrades for when fixed-size runs out of room.