Small Chunks vs Large Chunks
Chunk size is one of the cheapest, highest-leverage dials in a RAG pipeline — small chunks retrieve precisely but lose context; large chunks preserve context but retrieve fuzzily.
| Dimension | Small Chunks | Large Chunks |
|---|---|---|
| Retrieval precision | Wins here A tightly scoped chunk matches a specific query without dragging in unrelated content from the same passage. | A broad chunk can match on the wrong sub-topic buried somewhere inside it. |
| Preserving surrounding context | A fact split too finely can lose the sentence explaining which system or condition it applies to. | Wins here Keeps enough surrounding text that a reader — human or model — can make sense of the excerpt alone. |
| Number of chunks to embed, store, and search | Shrinking chunk size from 500 to 200 tokens (at the same 10 percent overlap) takes a 12,000-token document from 27 chunks to 67 — well over double the embeddings to compute and store. | Wins here Fewer, larger chunks mean fewer embeddings and a smaller index for the same document. |
| Risk of one chunk blurring multiple topics | Wins here A narrow chunk stays about one thing. | Cram enough content into one chunk and its embedding becomes an average that matches nothing well. |
| Coverage within a fixed context-window budget after retrieval | Wins here More, smaller chunks can be retrieved within a fixed token budget, covering more distinct pieces of evidence. | Fewer large chunks fit in the same budget, narrowing how many distinct sources make it into context. |
| Facts that span multiple sentences or paragraphs | Can shatter a multi-sentence explanation across two chunks, orphaning the fact from its context. | Wins here Keeps a multi-step procedure or a qualified claim in one coherent piece. |
| What actually performs best | Depends on the corpus and query style, not a fixed rule. | Depends on the corpus and query style, not a fixed rule; splitting on semantic boundaries (headings, paragraphs) usually beats either fixed small or fixed large chunking of raw text. |
When to choose Small Chunks
Chunk small when queries target specific facts or short answers, and the corpus is dense with independently meaningful sentences.
When to choose Large Chunks
Chunk large when content only makes sense with fuller context — legal text, narrative, multi-step procedures — where losing surrounding text would strand the answer.
The verdict
There is no universal right size. Measure recall@k and precision@k for your own corpus at a few chunk sizes, and try semantic-boundary chunking, rather than assuming a number from someone else's benchmark.