Load docs, index chunks, query with sources, and evaluate retrieval.
Point SimpleDirectoryReader at a folder of PDFs and text files, inspect the Document objects it returns, and confirm that file-path metadata is attached before any chunking happens. The running project uses a local `./docs` folder containing three markdown files about a fictional product.
How to load a folder of files into LlamaIndex Document objects and attach metadata before any chunking happens.
Why this matters: Every downstream step — chunking, indexing, retrieval — depends on Documents being loaded correctly with the right metadata; getting this wrong silently corrupts the whole pipeline.
Decision this forces: Which file types and metadata fields to include at load time, and whether to enrich metadata before or after loading.
Your agent can only answer from what it can see. If your files never make it into memory as structured objects, the rest of the pipeline has nothing to work with.
A is LlamaIndex's envelope for a single source file: it holds the raw text and a dictionary that travels with that text forever.
The metadata dictionary is populated at load time — fields like file_path, file_name, and file_type are attached automatically by .
Anything you add or change on the Document before chunking will propagate to every derived from it — so load time is the highest-leverage moment to get metadata right.
Your project has a ./docs folder with three markdown files about a fictional product: overview.md, pricing.md, and changelog.md.
You point at the folder and call load_data(). It returns a list of three objects — one per file.
Each Document already carries file_path in its metadata, so downstream you can always trace an answer back to its source file.
Before passing the list to the chunker, you add a custom field — product_version — directly on each Document's metadata dict. Every chunk from that file will inherit it.
from llama_index.core import SimpleDirectoryReader # Attempt: load only .md files by passing a glob reader = SimpleDirectoryReader("./docs", recursive=False) documents = reader.load_data() for doc in documents: print(doc.metadata["product_version"]) # <-- will this work?
SimpleDirectoryReader("./docs", recursive=False)reader.load_data()doc.metadata["product_version"]This is the first instinct: load the folder and immediately read a custom metadata field. Predict what happens before revealing the answer.
KeyError: 'product_version'
SimpleDirectoryReader auto-populates file_path, file_name, and file_type — but nothing else. Any custom field you expect to exist must be added explicitly after loading. The reader never invents fields you didn't set.
from llama_index.core import SimpleDirectoryReader reader = SimpleDirectoryReader( input_dir="./docs", required_exts=[".md"], # only markdown files recursive=False, ) documents = reader.load_data() for doc in documents: print(doc.metadata) # inspect what the reader attached
required_exts=[".md"]doc.metadataAdding required_exts filters to only the file types you want. Predict the shape of the printed metadata before revealing.
{'file_path': './docs/overview.md', 'file_name': 'overview.md', 'file_type': 'text/markdown', 'file_size': <int bytes>, 'creation_date': '<date>', 'last_modified_date': '<date>'}
LlamaIndex auto-populates these six fields. No custom fields yet — that's Stage 3.
VERSION_MAP = {
"overview.md": "2.1",
"pricing.md": "2.1",
"changelog.md": "2.0",
}
for doc in documents:
fname = doc.metadata["file_name"]
doc.metadata["product_version"] = VERSION_MAP.get(fname, "unknown")
doc.metadata["source_type"] = "product-docs"
# Confirm propagation target
print(documents[0].metadata["product_version"]) # → "2.1"VERSION_MAP.get(fname, "unknown")doc.metadata["product_version"] = ...doc.metadata["source_type"] = "product-docs"Mutating doc.metadata in-place before any chunking step means every Node split from this Document inherits both new fields — no second pass needed.
All 10. Metadata is copied from the parent Document to every Node at split time. Enriching the Document before chunking is the only place you need to set it once.
from llama_index.core import SimpleDirectoryReader reader = SimpleDirectoryReader( input_dir="./docs", required_exts=[".pdf"], # switch to PDFs recursive=True, # include sub-folders ) documents = reader.load_data() for doc in documents: doc.metadata["doc_type"] = "pdf-reference" doc.metadata["reviewed"] = # TODO: set to True for files whose # file_name starts with "approved_", # False otherwise
recursive=Truestr.startswith("approved_")Stop — attempt the TODO before revealing. The gap is the conditional tagging logic, which is the core skill from Stage 3 applied to a new rule.
doc.metadata["file_name"] gives you the filename string. (2) Python's str.startswith() returns a bool directly.doc.metadata["reviewed"] = doc.metadata["file_name"].startswith("approved_")
Changed lines vs Stage 3:
If you omit required_exts and the folder contains a .DS_Store or .gitignore, LlamaIndex tries to parse them as text. You get a Document with garbled content and no error. Check len(documents) and spot-check doc.metadata["file_type"] to catch this.
You add a custom field after passing documents to the chunker — the Nodes are already built, so they never see it. The symptom is a KeyError on a Node's metadata at query time, not at load time. Always enrich before chunking.
A legacy PDF or Windows-encoded text file raises UnicodeDecodeError mid-load, and the entire load_data() call fails — not just that one file. Use required_exts to whitelist known-good formats, or pre-convert files before loading.
required_exts? An unconstrained reader ingests junk files silently.doc.metadata before — not after — any chunking or index call?len(documents) and at least one doc.metadata dict so you can confirm the right files loaded?Configure a SentenceSplitter with chunk size and overlap, transform your three markdown documents into Node objects, and compare how chunk size affects the granularity of retrieved evidence. You'll complete a partially written transformation pipeline by supplying the missing overlap and metadata-injection step.
Configure a SentenceSplitter and IngestionPipeline to slice your three product-doc markdown files into metadata-carrying Node objects, and understand how chunk size and overlap shift retrieval precision versus context completeness.
Why this matters: The quality of every retrieval result downstream depends entirely on how well your chunks are sized — too coarse and the LLM drowns in noise; too fine and answers get split across boundaries.
Each holds the raw text of one file plus a dict. It records the file path and any extra fields you injected. That metadata is the thread you pull through every downstream step. Chunks inherit it. Retrieved evidence always knows where it came from.
This module picks up right where that loading step ended. You have three Document objects in memory. Now you need to slice them into retrieval-sized pieces called . The question is simple: how big should each slice be, and how much should adjacent slices overlap?
Chunk size controls the granularity of the evidence your retriever can return. A small chunk, about 128 tokens, pinpoints a single fact. It may drop the surrounding context that makes it meaningful. A large chunk, about 512 tokens, preserves context. It also dilutes the relevance signal, so the retriever may rank it lower than a tighter match.
Overlap is the buffer between adjacent chunks. It is typically 10–20 % of chunk size. Without overlap, a sentence that straddles a boundary gets split in two. Neither half is complete. With too much overlap, you store redundant tokens and slow down retrieval.
For product-doc Q&A, a chunk of 256 tokens with 32–50 tokens of overlap is a common starting point. It is specific enough to rank precisely. It is wide enough to keep a full procedure step intact.
Imagine your three markdown files cover installation, configuration, and troubleshooting for a software product. A user asks: 'What port does the service listen on by default?' The answer is one sentence buried in the configuration doc.
With 512-token chunks, the retriever returns the entire configuration section. The answer is in there. So are 400 tokens of unrelated settings. The LLM has to sift through noise. It can latch onto the wrong number.
With 128-token chunks, the retriever returns the exact paragraph. But the port number may be explained across two sentences that straddle a chunk boundary. One chunk says 'the service listens on'. The next says 'port 8080 by default'. Neither chunk alone answers the question.
A 256-token chunk with 32-token overlap keeps both sentences together. It still ranks tightly. That is the sweet spot for this scenario. It is what you'll configure next.
from llama_index.core.node_parser import SentenceSplitter from llama_index.core import SimpleDirectoryReader # Load the three product-doc markdown files documents = SimpleDirectoryReader("./docs").load_data() # Configure the splitter: 256-token chunks, 32-token overlap splitter = SentenceSplitter( chunk_size=256, chunk_overlap=32, )
SentenceSplitterchunk_size=256chunk_overlap=32This sets up the splitter but doesn't transform anything yet — you're just declaring the cutting rules. chunk_size=256 targets one procedure step per chunk; chunk_overlap=32 (~12 %) prevents boundary splits from losing a sentence.
Each Node holds a text slice (≤256 tokens) plus a metadata dict inherited from its parent Document — including the file_path key that SimpleDirectoryReader attached. Metadata inheritance is automatic; you don't need to copy it manually.
from llama_index.core.ingestion import IngestionPipeline from llama_index.core.node_parser import SentenceSplitter pipeline = IngestionPipeline( transformations=[ SentenceSplitter(chunk_size=256, chunk_overlap=32), ] ) nodes = pipeline.run(documents=documents) print(f"Nodes produced: {len(nodes)}") print(nodes[0].metadata) # should show file_path + any injected keys
IngestionPipeline(transformations=[...])pipeline.run(documents=documents)nodes[0].metadataThe runs each transformation in order and returns a flat list of . The print statements let you verify the count and confirm metadata survived the split.
800 tokens ÷ 256 per chunk ≈ 4 chunks per file × 3 files = ~12 Nodes (exact count varies with sentence boundaries). nodes[0].metadata prints something like {'file_path': './docs/installation.md', 'file_name': 'installation.md'} — the keys SimpleDirectoryReader injected, now present on every Node.
from llama_index.core.ingestion import IngestionPipeline from llama_index.core.node_parser import SentenceSplitter from llama_index.core.extractors import TitleExtractor pipeline = IngestionPipeline( transformations=[ SentenceSplitter(chunk_size=256, chunk_overlap=???), # TODO 1 # TODO 2: add a TitleExtractor() so each Node gets a 'doc_title' key ] ) nodes = pipeline.run(documents=documents)
TitleExtractor()chunk_overlap=???Stop — attempt both TODOs before revealing the answer. TODO 1 asks for the overlap value that prevents boundary splits in 256-token chunks. TODO 2 is the crux: adding a second transformation that injects a document-title key into every Node's metadata.
CHANGED LINES:
chunk_overlap=32 # 32/256 ≈ 12 % — prevents boundary sentence loss
TitleExtractor(), # placed after SentenceSplitter so it runs on Nodes
WHY ORDER MATTERS: if TitleExtractor ran first it would see the full Document text, not individual Nodes, and the title would not propagate per-chunk. Transformations execute in list order, so the splitter must produce Nodes before the extractor can annotate them.
Drag to see how chunk size shifts the precision–completeness balance for your product-doc scenario.
Once your Nodes pass these checks, they're ready for the next step: embedding each chunk and loading it into a — where each Node becomes a searchable vector and the index learns which chunks sit close together in meaning.
Pass your Node list into VectorStoreIndex.from_documents, understand what happens under the hood (embed → store → build HNSW-style index), and persist the index to disk so you don't re-embed on every run. The worked example embeds the three product-doc files using OpenAI's text-embedding-3-small model.
Build a VectorStoreIndex from your Node list, choose an embedding model, and persist the index to disk so you skip re-embedding on future runs.
Why this matters: This is the indexing step that makes your product docs searchable by meaning — without it, your RAG app has no retrieval layer.
Decision this forces: Which embedding model to use (OpenAI vs. local) and whether to persist the index or rebuild on each run.
Each holds the chunk text and the metadata inherited from its parent Document — including the file path. That metadata is what lets the retriever trace an answer back to a source file later. Now those Nodes are ready to be embedded and indexed.
When you call VectorStoreIndex.from_documents, LlamaIndex runs three steps in sequence: it calls the model once per , stores each resulting vector alongside the Node's text and metadata, then builds an HNSW-style approximate-nearest-neighbour index over those vectors.
HNSW (Hierarchical Navigable Small World) is a graph structure that lets the retriever find the closest vectors in milliseconds without scanning every stored chunk.
The default embedding model is OpenAI's text-embedding-3-small — 1 536 dimensions, cheap per token, and strong enough for most product-doc use cases. You can swap it for a local model with no change to the indexing API.
from llama_index.core import VectorStoreIndex, Settings from llama_index.embeddings.openai import OpenAIEmbedding # Tell LlamaIndex which embedding model to use globally Settings.embed_model = OpenAIEmbedding(model="text-embedding-3-small") # nodes = list of Node objects from module 2 index = VectorStoreIndex(nodes) print(f"Indexed {len(index.docstore.docs)} nodes") # → Indexed 47 nodes
Settings.embed_modelOpenAIEmbedding(model=...)VectorStoreIndex(nodes)Passing your Node list directly to VectorStoreIndex(nodes) triggers one embedding API call per Node. The print confirms how many chunks were stored — a quick sanity check that no Nodes were silently dropped.
LlamaIndex will chunk each Document with its default splitter (chunk_size=1024) and embed those chunks — ignoring the SentenceSplitter settings you configured in module 2. You'll get fewer, larger chunks and lose your carefully tuned overlap. Always pass nodes, not documents, when you want to control chunking.
PERSIST_DIR = "./storage" # Save to disk (writes docstore.json, index_store.json, vector_store.json) index.storage_context.persist(persist_dir=PERSIST_DIR) # Reload on the next run — no embedding calls made from llama_index.core import StorageContext, load_index_from_storage storage_context = StorageContext.from_defaults(persist_dir=PERSIST_DIR) index = load_index_from_storage(storage_context) print(f"Reloaded {len(index.docstore.docs)} nodes") # → Reloaded 47 nodes
index.storage_context.persist(persist_dir=...)StorageContext.from_defaults(persist_dir=...)load_index_from_storage(storage_context)Persisting writes three JSON files to ./storage; reloading reads them back and reconstructs the index in memory. The node count must match — if it doesn't, the persist was incomplete.
No. load_index_from_storage reads the vectors that were stored during the original build — it never re-embeds. To pick up new chunk sizes you must rebuild the index from scratch and persist again.
from llama_index.core import VectorStoreIndex, Settings from llama_index.embeddings.huggingface import HuggingFaceEmbedding # TODO: set Settings.embed_model to HuggingFaceEmbedding # using model_name="BAAI/bge-small-en-v1.5" # (hint: same pattern as OpenAIEmbedding in Stage 1) local_index = VectorStoreIndex(nodes) local_index.storage_context.persist(persist_dir="./storage_local") print(f"Local index nodes: {len(local_index.docstore.docs)}")
HuggingFaceEmbedding(model_name=...)"BAAI/bge-small-en-v1.5"Stop — attempt the TODO before revealing. The missing line is the crux: swapping the embed model is the only change needed to go fully local. The node count should match Stage 1's output exactly.
Settings.embed_model = HuggingFaceEmbedding(model_name="BAAI/bge-small-en-v1.5")
Changed lines vs Stage 1: the import switches from openai to huggingface, and the model constructor changes — everything else (VectorStoreIndex call, persist call) is identical. The first run downloads ~130 MB; subsequent runs use the cached model.
| Option | Retrieval quality | Privacy / data leaves network | When to choose | Cost | Complexity |
|---|---|---|---|---|---|
| OpenAI text-embedding-3-small | Strong out-of-the-box; beats most local models under 1B params | Chunks are sent to OpenAI's API — not suitable for confidential data | Default choice for most RAG projects: fast setup, strong quality, low cost per token, and no GPU needed. | ~$0.02 / 1M tokens | Zero setup — pass your API key |
| Local model (e.g. BAAI/bge-small-en) | Competitive for English; may lag on domain-specific jargon without fine-tuning | All embedding runs locally — no data leaves your machine | Use when data must stay on-premise, you have a GPU, or you need zero per-call cost at scale. | Free at inference; GPU/CPU time only | Install sentence-transformers; first run downloads the model (~130 MB) |
Three failure patterns show up repeatedly when building and persisting a VectorStoreIndex.
./storage — but the index still uses the old chunks. The symptom: retrieved snippets are the wrong length and the node count doesn't match your new split. Fix: delete ./storage and rebuild.text-embedding-3-small (1 536 dims) but reload and query with a different model. Queries return nonsense rankings because the query vector lives in a different space than the stored vectors. Always set Settings.embed_model to the same model before calling load_index_from_storage.len(index.docstore.docs) right after building and compare it to len(nodes) from your pipeline.Verifying AI-generated index code: check that the generated code (1) sets Settings.embed_model before building, (2) calls .persist() with an explicit directory, and (3) prints the node count after both build and reload so you can confirm they match.
Click a query to see which product-doc chunks land nearest in embedding space. Proximity = semantic similarity, not keyword overlap.
Build a QueryEngine from the index, run a natural-language question against the product docs, and extract the source_nodes from the response to display file name, page, and passage text alongside the answer. You'll complete the source-display loop, which is left partially written.
Build a QueryEngine from your persisted index, run a natural-language query, and display the file name, similarity score, and text snippet for each source node that backed the answer.
Why this matters: Source attribution is what makes a RAG answer trustworthy — without it you can't tell whether the model retrieved real evidence or fabricated a confident-sounding response.
Decision this forces: similarity_top_k value — how many chunks to retrieve per query, trading recall breadth against answer focus.
Answer: it stores the vector (floats representing meaning) and the original text plus . At query time the vector finds the right ; the text and metadata are what you show the user.
Module 3 left you with a persisted over three product-doc markdown files. This module wires a to that index, runs a real question against it, and surfaces — the exact passages that backed the answer.
The driving question: how do you prove to a user that the answer didn't come from thin air?
Calling index.as_query_engine() wraps the in a pipeline: embed the question → retrieve top-k chunks by → pack them into a prompt → generate an answer.
The single most consequential parameter is : it controls how many are retrieved per query. Higher values broaden recall but dilute focus. Lower values keep context tight but risk missing evidence.
Every response object carries a list — one entry per retrieved chunk. Each holds the text, similarity score, and node metadata (file name, page number).
Source attribution separates trustworthy RAG answers from confident-sounding : you can show users exactly which passage the model drew from.
Imagine your team ships a support bot backed by three product-doc files: installation.md, api_reference.md, and troubleshooting.md. A user asks: "How do I reset my API key?"
The QueryEngine retrieves the top 3 chunks by similarity. Two come from api_reference.md (page 4 and page 7) and one from troubleshooting.md (page 2). The model synthesises an answer from all three.
Without source attribution, the user has to trust the bot blindly. With it, you display: "Source: api_reference.md, page 4 — 'Navigate to Settings → API Keys and click Regenerate.'" The user can verify in seconds.
This is the loop you'll complete below: query → retrieve → display file, score, and snippet for each .
from llama_index.core import StorageContext, load_index_from_storage # Reload the persisted index from Module 3 storage_context = StorageContext.from_defaults(persist_dir="./storage") index = load_index_from_storage(storage_context) # Build a QueryEngine that retrieves the top 3 chunks query_engine = index.as_query_engine(similarity_top_k=3) response = query_engine.query("How do I reset my API key?") print(response)
StorageContext.from_defaults(persist_dir=...)load_index_from_storage(storage_context)index.as_query_engine(similarity_top_k=3)query_engine.query(...)This reloads the index you persisted in Module 3 and wraps it in a with .
Running this prints the synthesised answer — but not yet which passages backed it. Stage 2 adds that.
It prints the LLM's synthesised answer as a plain string (e.g. "To reset your API key, navigate to Settings → API Keys and click Regenerate."). The source nodes are attached to the response object but are NOT printed by default — you have to access response.source_nodes explicitly.
response = query_engine.query("How do I reset my API key?") print("Answer:", response) print("\n--- Sources ---") for node in response.source_nodes: meta = node.metadata print(f"File : {meta.get('file_name', 'unknown')}") print(f"Score: {node.score:.3f}") # TODO: print a 200-character snippet of node.node.text print("---")
response.source_nodesnode.metadatanode.scorenode.node.textmeta.get('file_name', 'unknown')This is the completion rung — most of the source-display loop is written; your job is the TODO line.
Stop — attempt the TODO before revealing. Hints: (1) the full text lives at node.node.text; (2) Python string slicing gives you the first N characters.
print(f"Snippet: {node.node.text[:200]}")
--- What changed and why ---
The TODO line is the crux of this module: accessing node.node.text surfaces the actual passage the model used, and slicing [:200] keeps the output readable. Everything else in the loop was already provided. With this line in place, each iteration prints the file name, similarity score, and a 200-character excerpt — the full source-attribution display.
Drag to see how similarity_top_k shifts the precision-recall tradeoff for your product-doc queries.
Three failure patterns to watch for in your product-doc pipeline:
meta.get('file_name') returns None silently — your attribution display shows "unknown" for every source. Fix: confirm doc.metadata['file_name'] is populated before indexing (Module 1 covered this).similarity_top_k=1, a question whose answer spans two sections retrieves only one chunk. The model either hallucinates the gap or says "I don't know" — and the source list looks complete, so you won't notice without checking scores.node.node.text is accessed (not node.text — that attribute doesn't exist on NodeWithScore and raises AttributeError silently in some versions).response.source_nodes, not response.nodes (a common AI hallucination of the attribute name).Run three deliberately bad queries against the product-doc pipeline — a vague query, an out-of-scope query, and a query whose answer spans multiple chunks — and trace exactly which component fails in each case. You'll also learn how to spot hallucinated citations in the response object and apply a targeted fix (chunk size, top-k, or metadata filter) for each failure pattern.
A hands-on failure-mode lab: run three bad queries against the product-doc pipeline, read source_nodes to pinpoint which layer broke, and apply targeted fixes for each pattern.
Why this matters: Bad answers are inevitable in any RAG system; knowing whether to fix retrieval, chunking, or the prompt saves hours of blind tuning and prevents shipping hallucinated responses.
Answer: each holds the raw passage text and a dict. It includes the file name and page number.
You built the source-display loop in module 4. It surfaces those fields alongside every answer. In this module you'll stress-test that pipeline with three bad queries. Use the same to diagnose exactly which layer broke.
The driving question: when your product-doc pipeline returns a wrong or empty answer, is the fault in retrieval, chunking, or generation? How do you tell?
Every bad RAG answer traces back to one of three layers: retrieval, chunking, or generation. Knowing which layer failed tells you exactly which knob to turn.
source_nodes is empty or contains irrelevant passages. Cause: vague query, too-small , or missing .Always inspect source_nodes first — it tells you whether the problem is upstream (retrieval/chunking) or downstream (generation) before you change anything.
Run these three queries against the product-doc and observe what source_nodes returns each time.
Query: "Tell me about the product." The of this query sits near the centroid of the entire corpus, so scores are uniformly low. Result: source_nodes returns 2 weakly-matched chunks from unrelated sections; the answer is a vague paraphrase of those chunks. Fix: rewrite the query to be specific, or raise and add a to scope the search.
Query: "What is the product's stock price?" No chunk in the index covers this topic, so source_nodes returns nodes with similarity scores below 0.4. The model, lacking grounded context, may fabricate a plausible-sounding figure — a textbook . Fix: add a similarity-score threshold guard; if max score < 0.5, return a "not in docs" message instead of generating.
Query: "Summarise the installation steps and the post-install configuration." These steps live in two separate sections of the docs, split across chunk boundaries. Result: source_nodes returns only the installation chunk; the config steps are missing. The answer is half-complete, and no citation points to the config section. Fix: increase chunk overlap or raise so both sections are retrieved.
response = query_engine.query("Tell me about the product.") for i, node in enumerate(response.source_nodes): score = node.score fname = node.metadata.get("file_name", "unknown") text_preview = node.text[:120] print(f"[{i}] score={score:.3f} | file={fname}") print(f" preview: {text_preview}\n") if not response.source_nodes: print("No nodes retrieved — likely a retrieval failure.")
response.source_nodesnode.scorenode.metadata.get("file_name")node.text[:120]This loop surfaces the three signals you need to diagnose any bad answer: similarity score, source file, and the actual passage text. A score below ~0.5 or an empty list is your first sign the retrieval layer failed, not the model.
You'll see low scores (likely 0.35–0.55) and previews from loosely related sections — the query embedding sits near the corpus centroid, so no chunk is a strong match. This is the retrieval failure signature: nodes exist but none are genuinely relevant.
from llama_index.core.vector_stores import MetadataFilter, MetadataFilters filters = MetadataFilters(filters=[ MetadataFilter(key="file_name", value="installation_guide.md") ]) filtered_engine = index.as_query_engine( similarity_top_k=5, filters=filters, ) response = filtered_engine.query( "Summarise the installation steps and the post-install configuration." )
MetadataFilter(key="file_name", value=...)MetadataFilters(filters=[...])similarity_top_k=5index.as_query_engine(...)This targets the chunking failure from Query 3 by scoping retrieval to one file and raising to 5, so both the installation and config chunks can be returned together. The eliminates noise from unrelated docs before similarity scoring even runs.
More chunks (up to 5 instead of the default 2) and higher average scores — because the filter removes off-topic documents, so all candidates come from the relevant file and rank more tightly against the query.
response = query_engine.query("What is the product's stock price?") max_score = max( (node.score for node in response.source_nodes), default=0.0 ) if max_score < 0.5: answer = "This question is outside the product documentation." else: # TODO: assign `answer` from the response when retrieval looks good pass print(f"max_score={max_score:.3f} | answer: {answer}")
max(..., default=0.0)node.score for node in response.source_nodesresponse.responseThis is the completion rung for the out-of-scope failure pattern. The guard is in place; your job is to fill in the TODO so the pipeline returns the model's answer when retrieval actually succeeds.
Changed line (else block): answer = response.response
For the stock-price query, max_score will print something like 0.31–0.42 — well below 0.5 — so the guard fires and the model is never asked to generate, preventing the hallucination. The threshold branch is the key addition; everything else was already in place.
Drag to see how chunk overlap affects whether a multi-section answer is fully retrieved. Low overlap splits answers across boundaries; high overlap risks redundant context and slower retrieval.
When an AI tool generates a retrieval fix for you — a new filter, a revised top-k, a changed overlap — check these four things before merging it.
"filename" silently returns zero results if your loader stored it as "file_name".The next module, Evaluating Retrieval Quality with Ragas, gives you a systematic way to run these checks across a full query dataset. It measures , , and so you're not relying on spot-checks alone.
Build a small evaluation dataset of five query/expected-source pairs from the product docs, run the Ragas evaluator against your QueryEngine, and interpret context_precision, context_recall, and faithfulness scores to decide whether the pipeline is production-ready. This is the solo build: you supply the dataset and interpret the scores without scaffolding.
Run a five-query Ragas evaluation against your LlamaIndex QueryEngine and interpret context_precision, context_recall, and faithfulness scores to decide if the pipeline is production-ready.
Why this matters: Gives you a repeatable, numeric gate for shipping — replacing gut-feel with scores that point directly to which pipeline parameter to tune.
The LLM itself — it when the retrieved chunks didn't contain the answer. Module 5 showed you how to spot that by eye. This module gives you numbers instead of intuition: a Ragas evaluation run that scores every query systematically.
You've built the full pipeline — load, chunk, embed, query. The last question is: how do you know it's good enough to ship?
separates retrieval quality from answer quality using three scores, each answering a distinct question about your pipeline.
A faithful answer can still have low recall: the LLM only used the chunks it got, but those chunks were incomplete. That's a retriever problem, not a generation problem.
Your eval dataset needs five rows: query, expected answer, and source document(s). Pull these directly from product docs—don't invent them.
Five queries surface systematic problems. A score pattern across all five is more diagnostic than any single result.
from llama_index.core import VectorStoreIndex, StorageContext, load_index_from_storage storage = StorageContext.from_defaults(persist_dir="./storage") index = load_index_from_storage(storage) query_engine = index.as_query_engine(similarity_top_k=3) queries = ["What is the refund window?", "How do I reset 2FA?", "List all supported export formats.", "What is the SLA uptime?", "Does the product support HIPAA?"] results = [] for q in queries: resp = query_engine.query(q) results.append({ "question": q, "answer": resp.response, "contexts": [n.text for n in resp.source_nodes], })
load_index_from_storage(storage)as_query_engine(similarity_top_k=3)resp.source_nodes[n.text for n in resp.source_nodes]This stage loads the persisted index from module 3 and runs each eval query, collecting the generated answer and the raw chunk texts from . The contexts list is what Ragas will score for precision and recall.
Context recall for that row will be 0 (or very low). The two chunks containing the answer were ranked 4th and 5th, so top_k=3 never fetched them. Ragas compares the retrieved contexts against the ground-truth answer and finds no overlap — the retriever missed the evidence entirely. Fix: raise similarity_top_k to 5 and re-run.
from datasets import Dataset from ragas import evaluate from ragas.metrics import context_precision, context_recall, faithfulness ground_truths = [ ["Refunds are accepted within 30 days."], ["Go to Settings > Security > Reset 2FA."], ["CSV, JSON, and PDF are supported."], ["99.9% uptime SLA."], [""], # out-of-scope: no answer in docs ] dataset = Dataset.from_list([ {"question": r["question"], "answer": r["answer"], "contexts": r["contexts"], "ground_truth": gt[0]} for r, gt in zip(results, ground_truths) ]) scores = evaluate(dataset, metrics=[context_precision, context_recall, faithfulness]) print(scores.to_pandas()[["context_precision","context_recall","faithfulness"]])
Dataset.from_list([...])evaluate(dataset, metrics=[...])ground_truthscores.to_pandas()This stage wraps the collected results in a Hugging Face Dataset and passes it to Ragas's evaluate function, which scores each row independently and returns a DataFrame you can inspect per-query.
The empty string ground truth for the out-of-scope query is intentional: any non-zero recall score there means the retriever surfaced irrelevant chunks that could mislead the LLM.
Faithfulness will be low (near 0) because the LLM's claim 'HIPAA-compliant' can't be traced to any retrieved chunk — the chunks either don't exist or are off-topic. This exposes a hallucination risk: the model is generating from parametric memory, not from evidence. Fix: add a metadata filter or a no-answer guard when retrieved context is empty.
Drag to see how increasing top-k shifts the precision/recall balance for your product-doc QueryEngine.
Each score pattern points to a different lever. Match the symptom to the fix before touching any parameter.
With a clean eval harness and honest ground truths, your scores become a reliable gate for production readiness and the foundation for solo capstone tuning.
Before looking at the summary: reconstruct the six pipeline stages from memory — what does each stage produce, and what does the next stage consume? Then identify which two parameters you tuned most often across the build and what each one controls.
Apply what you learned to Building a RAG Application with LlamaIndex.
You load a directory of PDF and Markdown files with SimpleDirectoryReader and immediately add a custom metadata field — say, 'product_line' — to each Document object. Later, the IngestionPipeline splits those Documents into Nodes. Which statement best describes what happens to 'product_line' on the resulting Nodes?
LlamaIndex copies a Document's metadata dict to every Node the splitter produces from it, so 'product_line' appears on all Nodes regardless of how many chunks are created. The 'first Node only' option is a common misconception — there is no such limit. SentenceSplitter does preserve metadata; it only controls text boundaries. And the timing of metadata enrichment (before vs. after loading) does not affect propagation — what matters is that the field is present on the Document before the pipeline runs.
A teammate sets chunk_size=128 and chunk_overlap=0 in SentenceSplitter for a set of dense technical product docs. You notice the QueryEngine often returns answers that cut off mid-explanation. Which diagnosis and fix is most appropriate?
A chunk_size of 128 tokens is very small for dense technical text — explanations that span several sentences get split across multiple Nodes, and if those Nodes are not all retrieved, the answer is incomplete. Increasing chunk_size and adding overlap keeps more context together. Reducing chunk_size further would worsen the problem. The embedding model affects retrieval quality but not whether a single explanation fits inside a chunk. Raising similarity_top_k is a band-aid that increases noise; the root cause is the chunking strategy.
Consider this short code sequence:
index = VectorStoreIndex(nodes, embed_model=local_model)
index.storage_context.persist(persist_dir="./store")
index2 = load_index_from_storage(StorageContext.from_defaults(persist_dir="./store"))
What is the primary reason to prefer this pattern over rebuilding the index on every application start?
The main benefit of persisting a VectorStoreIndex is that embeddings are computed once and saved to disk; reloading skips the embedding step entirely, which saves both API cost (for hosted models) and wall-clock time. Persistence does not compress Nodes or reduce memory at query time — the index is loaded fully into memory either way. A persisted index is static; it does not watch source files for changes. load_index_from_storage simply deserializes the saved index — it performs no re-ranking.
After running a query, you inspect response.source_nodes and find that all returned Nodes are from the correct document but none of them contain the specific paragraph that answers the question. Which failure mode does this indicate, and what is the best first fix?
When source_nodes point to the right document but the key paragraph is absent from all of them, the retriever found the right area but the paragraph was fragmented across chunk boundaries — a chunking failure. The fix is to increase chunk_size or chunk_overlap so the paragraph lands intact in at least one Node. A generation failure would show the LLM producing an answer not grounded in any returned Node. A retrieval failure would surface wrong documents entirely, not the right document with missing content. A metadata filter issue would cause relevant Nodes to be excluded before retrieval even scores them.
In Ragas, a QueryEngine returns a response that is completely faithful to the two Nodes it retrieved — every claim in the answer is supported by those Nodes. However, the Ragas evaluation reports a low context_recall score. Explain in 1–2 sentences why this is possible and what it tells you about the pipeline.
Faithfulness and context_recall measure different things. Faithfulness checks that the generated answer does not contradict the retrieved Nodes. Context_recall checks that the retrieved Nodes collectively contain all the information a complete reference answer would require. You can have perfect faithfulness (the LLM only says what the Nodes say) alongside low context_recall (the Nodes themselves were missing key content). This tells you the retrieval step — not the generation step — is the bottleneck, and you should tune similarity_top_k, chunk_size, or the embedding model.