Chunking is the step in a retrieval-augmented generation (RAG) pipeline that splits large documents into smaller passages before they are embedded and…
See why splitting documents well is what makes RAG retrieval accurate.
In retrieval-augmented generation (RAG), each chunk is embedded into one vector and retrieved as a unit. So the chunk is the smallest thing your system can find — if the answer is split across two chunks, or buried in a chunk full of unrelated text, retrieval suffers.
Good chunking gives each passage a single, clear topic. That makes its embedding a faithful summary of its meaning, so a matching query retrieves it, and the model receives tight, on-topic context.
Compare fixed-size, recursive, and semantic chunking and when each fits.
Fixed-size chunking splits text every N tokens or characters, usually with some overlap. It is simple, fast, and predictable, and it is a fine default for uniform prose. Its weakness is that it ignores structure, so it can cut a sentence or a table in half.
Recursive chunking splits on a priority list of separators — paragraphs, then sentences, then words — trying larger boundaries first and falling back only when a piece is still too big. This respects natural structure and is the common default in libraries like LangChain.
Semantic chunking places boundaries where the meaning shifts: it embeds sentences and starts a new chunk when the topic changes enough. It produces the most coherent chunks but costs more to compute, so it suits high-value or heterogeneous corpora.
Pick concrete numbers for size and overlap and know the trade-offs.
A common starting point is chunks of roughly 200 to 1,000 tokens with 10 to 20 percent overlap. Smaller chunks give sharper retrieval but can lose surrounding context; larger chunks carry more context but dilute the embedding and may exceed what you want to pass to the model.
Overlap means adjacent chunks share some text, so a sentence sitting on a boundary still appears whole in at least one chunk. Tune size to your content: dense technical text often wants smaller chunks, narrative text larger ones.
There is no universal best size. Pick a default, then evaluate retrieval on real questions and adjust — chunking is one of the highest-impact knobs in a RAG system, so it is worth a small experiment.
Avoid the chunking errors that quietly wreck retrieval quality.
The frequent mistakes are: chunks too large (the embedding blurs several topics into one vague vector), chunks too small (single sentences lose the context needed to answer), zero overlap (ideas get severed at boundaries), and ignoring structure (splitting tables, code, or lists mid-item).
Fixes follow directly: right-size for your content, keep modest overlap, prefer structure-aware splitting, and keep metadata like the source and section with each chunk so answers stay traceable.
Chunking splits documents into passages before embedding them for retrieval-augmented generation (RAG), and it strongly shapes retrieval quality because each chunk is the smallest retrievable unit. Fixed-size chunking is a simple default, recursive chunking respects structure, and semantic chunking cuts on meaning. Start around 200-1,000 tokens with 10-20% overlap, avoid over- and under-sized chunks, and tune by measuring retrieval on real questions.
Take a document type you work with — a product manual, say. Decide which chunking strategy fits, pick a starting size and overlap, and write down one test question whose answer sits at a chunk boundary to check that your overlap keeps it intact.
What is chunking in RAG?
Chunking divides documents into retrievable passages; each chunk becomes one embedding and is the smallest unit retrieval can return.
What chunk size should I use for RAG?
There is no universal size; start in the 200-1,000 token range with modest overlap and adjust based on measured retrieval quality for your content.
What is chunk overlap and why does it matter?
Overlap prevents information at a chunk boundary from being severed; a sentence spanning the cut appears intact in one of the neighboring chunks.
What is semantic chunking?
Semantic chunking creates the most coherent, single-topic chunks by cutting on meaning rather than a fixed length, at higher compute cost.