Chunking

Chunking is the process of splitting a document into smaller, roughly self-contained passages before embedding and indexing it, so a retrieval system can return a focused excerpt instead of handing over an entire file.

Embedding models and language models both have a practical breaking point: cram a 50-page manual into one vector and its topic sentence, its appendix, and its legal boilerplate all blur into one averaged meaning that matches nothing well. Chunking exists to keep each stored unit small enough to have one coherent topic, and large enough to still make sense out of context.

How it works

Most pipelines slide a fixed-size window across the document — commonly 200 to 800 tokens — and step forward by less than the window size, so consecutive chunks overlap. The overlap exists because a fact split exactly at a chunk boundary would otherwise vanish from both halves; a sentence about "the retry limit is 3" might get orphaned from the sentence naming which service it applies to.

Better pipelines split on semantic boundaries — headings, paragraphs, list items — rather than raw character counts, so a chunk ends at a natural idea boundary instead of mid-sentence. The trade-off is unavoidable either way: chunk too large and retrieval gets fuzzy, matching on the wrong sub-topic buried inside; chunk too small and each fragment loses the surrounding context a reader — human or model — needs to make sense of it.

Why it matters

Chunk size is one of the highest-leverage, cheapest-to-tune dials in an entire RAG system — changing it requires no retraining, just re-indexing, and it routinely moves retrieval quality more than swapping embedding models does. Get it wrong and every downstream stage inherits the damage: a hybrid search or reranker can only reorder the chunks it is given, and a language model can only ground its answer in the passages retrieval handed it.

A worked example

Take a 12,000-token policy document, chunked at 500 tokens with a 50-token (10 percent) overlap. The stride between chunk starts is 500 − 50 = 450 tokens, so covering the full document takes ceil((12,000 − 50) / 450) = 27 chunks, the last one starting at token 11,700 and running to the document's end. Shrink the chunk size to 200 tokens at the same 10 percent overlap and the same document explodes to roughly 67 chunks — more precise retrieval targeting, but nearly two and a half times the embeddings to compute, store, and search, and more fragments that might individually lack context.

How miatz teaches it

Chunking is a hands-on lab, not a slide: learners take one real document, chunk it three different ways — small, large, and semantic-boundary — then run the same queries against all three indexes and watch precision and recall move in opposite directions. It sits directly upstream of the RAG lab, because no amount of clever retrieval logic fixes a badly chunked index.