Split documents so retrieval finds the exact evidence an answer needs.
A chunk is not just a storage unit; it is the candidate evidence the model will cite.
Learners identify the evidence unit before choosing a splitter.
Why this matters: This prevents token windows from slicing off the caveat or exception that makes an answer true.
Problem anchor: a support assistant cites the refund page but misses the paragraph saying enterprise contracts require finance approval. Retrieval “worked” by keyword, yet the answer was wrong because the returned chunk lacked the governing caveat.
The practical question is not “450 tokens or 800 tokens?” It is “what unit would let a reviewer verify this answer without opening the original document?” Headings, definitions, examples, table captions, and version dates are part of that unit.
For the refund policy page, the chunk contract says each returned unit must include the heading path, product tier, effective date, and any exception paragraph adjacent to the rule. The body target is 350-600 tokens, but the policy exception outranks the target size.
The resulting chunk text starts with a breadcrumb: “Billing > Refunds > Enterprise exceptions.” That breadcrumb gives lexical search exact words, vector search semantic context, and the generator enough provenance to cite the correct page.
The best initial splitter is usually the one that preserves the author’s organization.
Learners choose splitters from document shape instead of copying one default.
Why this matters: Structure-aware splits reduce broken tables, orphaned headings, and meaningless code fragments.
A Markdown handbook, an API reference, a spreadsheet export, and a call transcript should not all become identical 500-token windows. Each source has boundaries that already signal meaning.
Recursive splitting is useful because it tries larger natural units first, then falls back to smaller ones. Parent-child retrieval is useful when small children rank well but the answer needs the parent section.
A developer-assistant KB ingests onboarding docs. The first version chunks every 400 characters and returns code snippets without the prerequisite environment variable. The second version keeps code fences whole, summarizes tables with column names, and stores heading breadcrumbs.
Retrieval improves not because embeddings changed, but because each candidate now contains a complete procedural step. The model can answer “How do I create the API token?” without mixing token creation and token rotation.
Chunk text carries meaning; metadata carries control.
Learners plan metadata as part of chunking, not an afterthought.
Why this matters: Metadata supports tenant filters, freshness checks, exact source links, and debugging.
A chunk without source URL, heading path, version, permission scope, and freshness date is hard to trust even when the text is relevant. Metadata lets retrieval ask “relevant for whom, from which source, and as of when?”
For Agentic Learning Studio, the category and lesson slug matter because a learner asking about “agents” should not receive a vector-near chunk about “agency” in a governance note. Filters narrow the candidate pool before ranking.
| Option | Field | Why it matters | Failure if absent | When to choose | Cost | Complexity |
|---|---|---|---|---|---|---|
| Heading path | Document title plus nested headings. | Ranks vague sections and gives citations readable labels. | A “Limitations” chunk has no topic context. | Use for all docs with structure. | Low | Low |
| Permission scope | Tenant, role, source ACL, or visibility flag. | Prevents unauthorized evidence from entering prompts. | Correct retrieval can still leak private content. | Mandatory for internal or customer data. | Medium | Medium |
| Freshness/version | Effective date, source hash, product version. | Detects stale indexes and version-specific answers. | Old facts look just as authoritative as current ones. | Use whenever facts change. | Low | Medium |
The first implementation test is a chunk preview, not a model answer.
Learners implement a small chunking routine and inspect output.
Why this matters: Inspecting chunks catches broken evidence before expensive embedding and indexing.
A chunker that only reports counts hides the important failure: what text actually reaches retrieval. Print chunk IDs, headings, token estimates, and first/last sentences.
Spaced recall: earlier you named the answer evidence. Now verify the split produces that evidence as one returned unit or as a child linked to the right parent.
def split_sections(markdown): sections, current = [], {"path": [], "lines": []} for line in markdown.splitlines(): if line.startswith("## "): if current["lines"]: sections.append(current) current = {"path": [line[3:].strip()], "lines": [line]} else: current["lines"].append(line) if current["lines"]: sections.append(current) return sections def window_words(text, size=90, overlap=18): words = text.split() step = max(1, size - overlap) for start in range(0, len(words), step): yield " ".join(words[start:start + size]) policy = """## Refunds Standard plans can be refunded within 30 days. Enterprise contracts require finance approval after 30 days. Refunds are denied when usage violates the acceptable-use policy. """ for section in split_sections(policy): for i, chunk in enumerate(window_words(" ".join(section["lines"]))): print({"heading": section["path"], "chunk": i, "text": chunk})
With this small policy it stays together; on a longer section, the printed preview tells you whether to shrink, expand, or return the parent section.
A chunking strategy is not done until retrieval eval says the evidence survives.
Learners turn chunking assumptions into repeatable retrieval checks.
Why this matters: Chunk changes otherwise look harmless while breaking citations or answer completeness.
Chunking changes should be evaluated like code changes. A good test asks the exact question that used to fail and checks whether the expected source or fact appears high enough to fit in the answer prompt.
Failure modes are topic-specific: refund exceptions split away from rules, code snippets separated from prerequisites, tables indexed without column names, or stale pages kept after source refresh.
When the model gives a better-looking answer, verify the retrieval layer before celebrating.
Retrieval prompt: rebuild the lesson from memory by naming the source structure, chunk size, overlap, metadata, evaluation queries, and the failure signal each choice is meant to catch.
Take one real policy or product-doc page. Produce a chunking spec with heading-aware splits, target token range, overlap rule, metadata fields, parent-child return behavior, and six eval questions that prove the answer context survives splitting. Next rung: add hybrid retrieval when exact identifiers matter.
In your own words: what is a chunk, and why does making chunks too small hurt retrieval quality even if the right keywords are present?
A chunk must carry enough surrounding context to support a complete answer — keyword presence alone is not sufficient if the meaning depends on adjacent sentences.
A document contains a Markdown table comparing pricing tiers. You apply a standard recursive character splitter with a 512-token limit. What is the most likely failure?
Recursive character splitters are unaware of table structure and will break rows at arbitrary character boundaries, destroying the row-column relationship that gives the data meaning.
Which metadata field most directly improves BOTH lexical (keyword) search AND vector (semantic) search for a retrieved chunk?
A breadcrumb injects structured, human-readable path tokens that boost keyword matches while also anchoring the chunk's vector representation in its document hierarchy, benefiting both retrieval modes.
During evaluation, a user query returns a chunk that contains the correct answer, but the LLM's final response is still wrong. What does this most likely indicate?
When the correct chunk IS retrieved but the answer is still wrong, the failure is in generation (the LLM), not retrieval — distinguishing these two failure modes is essential for knowing what to fix.
You are writing eval queries for a document split at section boundaries. Which query type BEST targets a boundary failure specific to chunking?
Boundary-spanning queries directly stress-test whether the splitter or overlap strategy preserves cross-section evidence — the exact failure mode unique to chunking decisions.