Assemble ingestion, retrieval, prompting, citations, and answer validation.
You load PDF, Markdown, or plain-text files and split them into short, self-contained passages called chunks. Every later step depends on chunk quality, so you'll see a worked example that loads a PDF and produces clean 400-token chunks with 50-token overlaps.
Load a PDF, extract its text, and slice it into short, labelled passages called chunks.
Why this matters: Every answer your Q&A bot gives is only as good as the chunks it searches — this step sets the quality ceiling for the whole system.
Imagine your bot needs to answer a question about page 47 of a 200-page PDF. If you hand the whole file to the model, it can't fit — and even if it could, the answer would be buried in noise.
The fix is and : you load the file, pull out its plain text, and slice it into short, self-contained passages.
Each passage — called a — is small enough to retrieve precisely and large enough to hold one complete idea.
Every later step in your Q&A bot (search, answer generation, citations) depends on chunk quality, so this is the most important foundation to get right.
Drag to see how chunk size shifts the tradeoff between precision and completeness. A token is roughly ¾ of an English word.
When you slice text at a fixed size, a sentence that starts near the end of one chunk often finishes at the start of the next.
Chunk solves this by repeating the last N tokens of each chunk at the start of the next one, so no idea is stranded alone.
A 50-token overlap on 400-token chunks means about 12% of each chunk is shared with its neighbor — enough safety margin without wasting space.
# Naive approach: split on every newline with open("handbook.pdf", "r") as f: # ❌ PDFs are binary text = f.read() chunks = text.split("\n") print(len(chunks), chunks[0])
open("handbook.pdf", "r")text.split("\n")This is the first thing most people try — and it fails immediately. Before revealing the output, predict what happens when Python tries to open a PDF as plain text.
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x25 in position 0: invalid start byte
PDFs store binary data, not plain text. You must use a PDF-parsing library to extract the text layer first. Even if you swap to 'rb' (binary mode), you get raw bytes — not readable sentences.
import pdfplumber # pip install pdfplumber pages = [] with pdfplumber.open("handbook.pdf") as pdf: for page_num, page in enumerate(pdf.pages): text = page.extract_text() or "" pages.append({"page": page_num + 1, "text": text}) print(f"Loaded {len(pages)} pages") print(pages[0]["text"][:120]) # preview first 120 chars
pdfplumber.open(...)page.extract_text() or ""enumerate(pdf.pages)Now we use a real PDF parser to pull out the text layer page by page, and we keep the page number alongside the text — that's the start of our .
Loaded 47 pages
Employee Handbook — Acme Corp
Section 1: Welcome
This handbook describes the policies and benefits available to all full-time employees hired after Jan 2023.
The text is clean prose. Notice pdfplumber preserves line breaks and headings, which we'll use as natural split points in the next stage.
def split_into_chunks(pages, size=400, overlap=50): chunks = [] chunk_id = 0 for page in pages: words = page["text"].split() start = 0 while start < len(words): end = start + size chunk_text = " ".join(words[start:end]) chunks.append({ "id": chunk_id, "source": "handbook.pdf", "page": page["page"], "text": chunk_text, }) chunk_id += 1 start += size - overlap # slide forward, keeping overlap return chunks chunks = split_into_chunks(pages) print(f"{len(chunks)} chunks — first chunk preview:") print(chunks[0])
start += size - overlap" ".join(words[start:end])chunks.append({...})This function slides a window of 400 words across each page, stepping forward by 350 words each time (400 − 50 overlap), and stamps every chunk with its source file, page number, and a unique ID.
That ID + filename + page number is the that lets your bot cite exactly where an answer came from.
3 chunks:
Chunk 1: words 0–399
Chunk 2: words 350–749 ← starts 50 words before chunk 1 ended (the overlap)
Chunk 3: words 700–799 ← shorter final chunk
The overlap means chunk 2 repeats the last 50 words of chunk 1, so any sentence that straddles the boundary appears in full in at least one chunk.
After running the code above on a 47-page handbook, you have a list of about 380 chunk dictionaries. Each has an ID, a source filename, a page number, and 400 words of clean text.
Here's what one chunk looks like at this point:
id: 12source: "handbook.pdf"page: 3text: "All employees are entitled to 20 days of annual leave…"This chunk is complete and traceable. But it's still just text. Your bot can't search it yet.
In the next module you'll pass each chunk through an model. It converts text into a list of numbers, a , that captures meaning. That makes it possible to find the most relevant chunk for any question.
Three failure modes appear in almost every first build. Know them before you trust your output.
len(chunks) == 0 or chunks full of whitespace. Fix: run OCR, such as pytesseract, before parsing. Or check chunk["text"].strip() == "" and skip those chunks.len(tokenizer.encode(text)) to measure, not word count.all("source" in c and "page" in c for c in chunks) before moving on.Verify AI-generated chunking code by checking three things. First, print chunks[0] and confirm the text is readable prose, not garbled bytes. Second, confirm chunks[-1]["text"] is non-empty. Third, spot-check that chunks[1]["text"][:50] matches the last ~50 words of chunks[0]["text"]. That's your overlap sanity check.
You pass each chunk through an embedding model to get a list of numbers (a vector) that captures its meaning, then upsert those vectors plus their metadata into a vector store such as Chroma or Qdrant. You'll see a worked example that embeds the same PDF chunks from Module 1 and stores them with a content-hash key so re-runs are safe.
This module shows how to turn text chunks into numeric vectors using an embedding model and store them in a vector store with a safe deduplication key.
Why this matters: Without this step your bot has no way to measure meaning — it can't find the right passages to answer a question.
Decision this forces: Which embedding model and vector store fit your scale — local (Chroma) vs. managed (Qdrant, Pinecone)?
Answer: a is a short, self-contained passage cut from your document, sized so it fits inside the model's working memory without losing its meaning. Overlap between chunks keeps context from being cut off at the edges.
You now have a list of clean chunks from your PDF. The bot still can't search them — it can only read text, not measure meaning. This module answers the question: how do you turn words into something a computer can compare?
An model reads text and outputs a — a list of numbers (often 384 or 1536) encoding meaning.
Similar topics get similar numbers, landing close together. Unrelated topics land far apart.
You never read these numbers directly. The vector store uses them to find chunks closest to your question via (0 to 1, where 1 means identical).
Click a query to see which PDF chunks land nearest in vector space. Points that share a topic sit close together; unrelated chunks sit far away.
A holds each chunk's vector and (source, page, ID) for retrieval.
When adding or updating, you — insert if new, replace if existing. A content-hash key prevents duplicates on re-run.
Chroma runs locally, no account needed. Qdrant and Pinecone are managed cloud services scaling to millions of vectors.
import hashlib # Pretend embed_model.encode() calls an embedding API def fake_embed(text): # Returns a short demo vector (real models return 384–1536 floats) return [0.12, 0.87, 0.34, 0.56] chunk_text = "Refunds are processed within 5 business days." vector = fake_embed(chunk_text) print("Vector length:", len(vector)) print("First 4 values:", vector)
fake_embed(text)len(vector)This stage shows the single job of an call: text goes in, a list of numbers comes out. Real models return hundreds of floats — the demo uses four so you can read them.
It prints 4 — the length of our demo vector. In a real model it would be 384 or 1536. The numbers encode the chunk's meaning; you never interpret them directly.
# Continuing from Stage 1 — chunk_text and vector are already defined def content_hash(text): return hashlib.md5(text.encode()).hexdigest()[:12] # Simulate a vector store as a plain dict vector_store = {} def upsert(store, text, vec, meta): key = content_hash(text) store[key] = {"vector": vec, "text": text, "meta": meta} return key key = upsert(vector_store, chunk_text, vector, {"source": "policy.pdf", "page": 3}) print("Stored key:", key) print("Keys in store:", list(vector_store.keys()))
hashlib.md5(text.encode()).hexdigest()[:12]store[key] = {...}"meta": metaThe content-hash key is the safety net: if you re-run ingestion on the same PDF, the same chunk produces the same key and overwrites itself — no duplicates pile up in the .
Still just one. The MD5 hash of the same text is always the same string, so the second call overwrites the first entry — the dict key doesn't change.
| Option | Setup effort | Scale ceiling | When to choose | Cost | Complexity |
|---|---|---|---|---|---|
| Chroma (local) | One pip install; runs in-process | Single machine RAM/disk; not for production at scale | Prototyping on your laptop or a small corpus (< ~100k chunks). | Free | Low — pip install, no account |
| Qdrant (managed) | Cloud account + API key; Docker option for local | Millions of vectors; built-in filtering and hybrid search | Production apps or corpora that outgrow a laptop; need filtering + hybrid search. | Free tier; paid by usage | Medium — cloud account + API key |
| Pinecone (managed) | Cloud account + API key; no self-hosting option | Serverless; scales automatically; no infra to manage | Fully managed, serverless scale with minimal ops overhead. | Free tier; paid by usage | Medium — cloud account + API key |
With embeddings stored and verified, you're ready for Module 3. You'll embed the user's question and ask the vector store for the top-k closest chunks.
You embed the user's question the same way you embedded the chunks, then ask the vector store for the top-k closest matches by cosine similarity. You'll complete a partially written retriever function — filling in the query-embedding call and the similarity search — and then add a keyword-search fallback (hybrid retrieval) to catch exact terms the vector search might miss.
How to embed a user query and retrieve the most relevant chunks from a vector store using cosine similarity, plus a keyword fallback for exact-term misses.
Why this matters: This is the search engine of your Q&A bot — without a working retriever, the language model never sees the right evidence and can't give accurate answers.
Each chunk became a — numbers capturing meaning — and was into a with (source file, page number, etc).
Now the store holds chunk vectors. The question: when a user asks, how do you find the right chunks fast?
Embed the user's question with the same model used for documents. This puts the question and chunks in the same coordinate space.
The vector store measures (0 to 1 score showing how closely two vectors align) between query and chunk vectors.
It returns the closest chunks — the k with the highest similarity scores.
Those chunks become evidence for the language model in the next step.
Your Q&A bot uses a company support handbook. User asks: "What is the return window for electronics?"
Using a different embedding model for the query breaks the match. Coordinate spaces don't align and similarity scores become meaningless.
def embed_text(text, model): # Returns a list of floats (the vector) return model.encode(text) query = "What is the return window for electronics?" query_vector = embed_text(query, embedding_model) print(type(query_vector)) # <class 'list'> print(len(query_vector)) # e.g. 384
model.encode(text)len(query_vector)This stage turns the raw question string into a using the same model used in module 2 — the model must match or similarity scores are garbage.
The query vector would live in a different coordinate space than the stored chunk vectors. Cosine similarity scores would be essentially random — high scores wouldn't mean semantic closeness — so the retrieved chunks would be irrelevant.
def retrieve_chunks(query_vector, store, k=3): results = store.similarity_search( query_vector=query_vector, top_k=k ) # results is a list of dicts: {"text": ..., "metadata": ...} return results chunks = retrieve_chunks(query_vector, vector_store, k=3) for c in chunks: print(c["metadata"]["source"], "|", c["text"][:60])
store.similarity_search(...)top_k=kc["metadata"]["source"]This stage asks the for the chunks closest to the query vector, then prints the source and a preview of each result.
You'd see something like:
handbook.pdf | Electronics must be returned within 30 days of pur
handbook.pdf | Refunds are issued to the original payment method
handbook.pdf | Damaged goods claims must be filed within 7 days
The three chunks with the highest cosine similarity to the query are returned in ranked order.
Click a query to see which chunks land closest in embedding space. Chunks near the query dot score high cosine similarity; distant ones score low and won't be retrieved.
Vector search finds paraphrases well but struggles with exact terms: model numbers, legal IDs, rare proper nouns.
runs keyword search (BM25) and vector search in parallel, then merges the result lists.
RRF (Reciprocal Rank Fusion) re-scores chunks by rank in both lists — no manual weight-tuning needed.
Add a for a third pass: it reads the query and each candidate chunk together, reordering by true relevance.
def hybrid_retrieve(query, store, bm25_index, k=3): query_vector = embed_text(query, embedding_model) # Vector results vec_results = store.similarity_search(query_vector, top_k=k) # TODO: call bm25_index.search(query, top_k=k) # and assign the result to kw_results kw_results = ??? merged = reciprocal_rank_fusion([vec_results, kw_results]) return merged[:k]
bm25_index.search(query, top_k=k)reciprocal_rank_fusion([...])merged[:k]The vector half is done for you. Your job: fill in the keyword search call so the bot catches exact terms (like "SKU-4821") that the vector search might rank low.
Also watch for these real failure modes before you trust any retriever — AI-generated or hand-written:
kw_results = bm25_index.search(query, top_k=k)
Changed lines vs. the worked example: only the ??? line. The key insight: BM25 takes the raw text string, not the embedding vector — it does its own term matching internally. The merged list now catches both semantic matches (vector) and exact-term matches (BM25).
You build a system prompt that instructs the model to answer only from the provided context and to cite each claim with a source tag like [Source 1]. You'll see a worked prompt template, then complete a version that injects the top-k chunks from Module 3 and enforces citation format — revisiting the chunk metadata you attached in Module 1.
You write a system prompt that forces the model to answer only from retrieved chunks and tag every claim with a [Source N] citation.
Why this matters: This is the step that turns raw retrieval into a trustworthy Q&A bot — without it, the model ignores your documents and invents answers.
Module 3's retriever embeds the user's question and asks the for the closest chunks by . It returns text plus metadata — source filename, page number, and more.
That metadata is essential. Without it, you can't label chunks as "Source 1" or "Source 2". The model has no way to cite sources.
This module's job: take labeled chunks and write a that forces the model to answer only from them and tag every claim with the source number.
A language model's default behavior is to answer from everything it learned during training. This means it can confidently state things not in your documents. This is called a — the main risk in a Q&A bot.
A is the standing instruction you send before the user's question. It sets the rules for every reply. When you paste retrieved chunks into that prompt, you give the model a closed set of evidence. This is called .
The (the model's working memory for one turn) holds both your instructions and the chunks. Everything the model sees lives there.
A well-written grounded prompt does three things. It pastes chunks as labeled evidence blocks. It forbids answers beyond those blocks. It requires a tag like [Source 1] after every claim.
Imagine your bot answers questions about a company handbook. Module 3 returned two chunks for the question "What is the parental leave policy?"
Here is what the fully assembled system prompt looks like before it reaches the model:
The model's reply would look like: "Primary caregivers get 16 weeks of paid leave [Source 1]. Secondary caregivers receive 4 weeks [Source 2]." Every claim is traceable to a specific chunk.
def format_chunks_as_evidence(chunks): blocks = [] for i, chunk in enumerate(chunks, start=1): label = f"[Source {i}]" file_info = chunk["metadata"]["source"] text = chunk["text"] block = f"{label}\nFile: {file_info}\n{text}" blocks.append(block) return "\n\n".join(blocks)
enumerate(chunks, start=1)chunk["metadata"]["source"]f"{label}\nFile: {file_info}\n{text}""\n\n".join(blocks)This function turns the list of retrieved chunks into labeled evidence blocks ready to paste into the prompt. Each chunk gets a [Source N] label built from its position in the list, and its filename comes from the you attached in Module 1.
"[Source 1]" — the label for the first chunk. The full first block would be "[Source 1]\nFile: <filename>\n<chunk text>".
def build_system_prompt(chunks, question): evidence = format_chunks_as_evidence(chunks) system_prompt = ( "You are a Q&A assistant for Acme Corp's employee handbook.\n" "Answer ONLY using the sources below. " "If the answer is not in the sources, say 'I don't know.'\n\n" f"{evidence}\n\n" "After every sentence, add [Source N] " "where N is the source number you used.\n\n" f"Question: {question}" ) return system_prompt
format_chunks_as_evidence(chunks)"Answer ONLY using the sources below."f"{evidence}\n\n"f"Question: {question}"This function wires Stage 1's evidence blocks into the full prompt in the correct order: role → hard rule → evidence → citation rule → question. The model receives one string that contains everything it needs to answer safely and traceably.
The model reads the task before it sees the rules and evidence. Some models will start generating an answer from training memory before processing the grounding constraint — increasing the chance of hallucination. Evidence should come before the question.
def build_system_prompt_strict(chunks, question): evidence = format_chunks_as_evidence(chunks) system_prompt = ( "You are a Q&A assistant for Acme Corp's employee handbook.\n" # TODO: Add the hard grounding rule AND the fallback instruction # Hint 1: forbid answers from outside the sources # Hint 2: tell the model what to say when no source covers the question f"{evidence}\n\n" "After every sentence, add [Source N] " "where N is the source number you used.\n\n" f"Question: {question}" ) return system_prompt
# TODO: Add the hard grounding rule AND the fallback instructionStop — attempt the TODO before revealing the answer. The two missing lines are the most important lines in the whole prompt: they are what separates a grounded bot from one that still hallucinates.
"Answer ONLY using the sources below. If the answer is not in the sources, say 'I don't know.'\n"
Changed lines vs. Stage 2: identical wording — the point is that REMOVING these two lines is what breaks grounding. Without them, the model treats the evidence as optional context rather than a hard constraint, and hallucinations return. The TODO forces you to write them from recall, not copy-paste.
Once your prompt reliably produces cited answers, the next module shows you how to automatically verify those citations. It checks whether every claim in the reply actually appears in the chunk it cites.
You apply two checks: a faithfulness check (does every claim in the answer appear in the retrieved chunks?) and a retrieval-relevance check (did the retrieved chunks actually address the question?). You'll fill in a validation function that flags low-confidence answers — and you'll revisit the chunking decision from Module 1 to see how bad chunks cause validation failures.
You add two automated checks — faithfulness and retrieval relevance — that catch wrong or hallucinated answers before they reach the user.
Why this matters: Without these checks, your Q&A bot can confidently return wrong answers; this module gives you the safety net that makes the bot trustworthy.
Answer: you injected the top-k chunks into the prompt and required a tag like [Source 1] after each claim. That tag is the link between an answer sentence and the chunk it came from. Without it, you have no way to check whether the model invented a fact or actually found it in your documents.
Module 4 told the model what to do. This module adds the check that confirms it actually did it — and flags the answer when it didn't.
Your bot can fail in two distinct ways, and each needs its own check.
You run both checks after the model responds. If either score falls below a threshold, you return a fallback like "I don't have enough information to answer that" instead of a low-confidence answer.
Drag to see what happens at each faithfulness threshold. A score of 1.0 means every claim traces back to a chunk; 0.0 means none do.
Three root causes account for almost every validation failure. Each one produces a different symptom.
Imagine your Q&A bot is answering questions about a software product's refund policy. A user asks: "Can I get a refund after 30 days?"
The retriever returns three chunks. Chunk 1 says "Refunds are available within 14 days of purchase." Chunk 2 is about shipping times. Chunk 3 is about account cancellation.
The model answers: "Yes, refunds are available up to 30 days after purchase." Let's run both checks:
The faithfulness check catches the hallucination. The bot returns: "I don't have enough information to answer that confidently" — protecting the user from a wrong answer.
def answer_question(query, chunks, model): context = "\n".join(chunks) prompt = f"Answer using only this context:\n{context}\n\nQuestion: {query}" response = model.complete(prompt) return response.text # no check — returned as-is
"\n".join(chunks)f"Answer using only this context:..."model.complete(prompt)response.textThis is the obvious first attempt: build a prompt, call the model, return whatever it says.
The problem is that the model can return a hallucinated answer and you'd never know. There's no check between the model's output and the user.
It returns the hallucinated answer unchanged. The user sees '30-day refund' with no warning. There is no faithfulness check, so the wrong claim passes straight through.
def check_faithfulness(answer, chunks, threshold=0.6): answer_sentences = answer.split(". ") matched = sum( 1 for s in answer_sentences if any(s.lower() in c.lower() for c in chunks) ) score = matched / max(len(answer_sentences), 1) return score >= threshold, score def check_retrieval_relevance(query, chunks, threshold=0.6): query_words = set(query.lower().split()) scores = [ len(query_words & set(c.lower().split())) / len(query_words) for c in chunks ] avg_score = sum(scores) / max(len(scores), 1) return avg_score >= threshold, avg_score
answer.split(". ")any(s.lower() in c.lower() for c in chunks)query_words & set(c.lower().split())max(len(answer_sentences), 1)These two functions implement lightweight versions of both checks using word overlap — no external library needed.
Word overlap is a simple proxy: a real production system would use an embedding similarity score or a dedicated evaluation model, but this version is enough to learn the pattern and catch obvious failures.
score = 1 / 3 = 0.33. That is below 0.6, so the function returns (False, 0.33) — the answer fails the faithfulness check and should trigger the fallback response.
FALLBACK = "I don't have enough information to answer that confidently." def answer_with_validation(query, chunks, model, faith_thresh=0.6, rel_thresh=0.6): rel_ok, rel_score = check_retrieval_relevance(query, chunks, rel_thresh) if not rel_ok: return FALLBACK, {"retrieval_relevance": rel_score} context = "\n".join(chunks) prompt = f"Answer using only this context:\n{context}\n\nQuestion: {query}" answer = model.complete(prompt).text faith_ok, faith_score = # TODO: call check_faithfulness here if not faith_ok: return FALLBACK, {"faithfulness": faith_score} return answer, {"faithfulness": faith_score, "retrieval_relevance": rel_score}
FALLBACK = "..."faith_thresh=0.6, rel_thresh=0.6return answer, {"faithfulness": faith_score, ...}This is the full validation wrapper. It runs the retrieval-relevance check first (before even calling the model), then runs the faithfulness check on the model's answer.
One line is missing — the call to check_faithfulness. Stop and attempt it before revealing the answer.
faith_ok, faith_score = check_faithfulness(answer, chunks, faith_thresh)
--- What changed and why ---
You connect ingest → embed → retrieve → prompt → generate → validate into a single `answer_question(query, docs)` function, run it against three test questions, and confirm that each answer carries valid citations and passes the faithfulness check from Module 5. This is your solo build: no scaffolding, just the components you wrote.
You wire all six pipeline stages — ingest, embed, retrieve, prompt, generate, and validate — into a single answer_question function and run it against test questions.
Why this matters: This is the capstone build: you leave with a working, end-to-end Document Q&A bot you can test, debug, and extend.
Decision this forces: How do you handle a query where retrieval returns no relevant chunks — silent failure, fallback message, or escalation?
Module 5 runs two checks on every answer the model produces.
When a check fails, the validator flags the answer rather than silently passing it through. You'll wire that validator as the final stage of the pipeline you build in this module.
A is a chain of steps where each step's output becomes the next step's input. Your Document Q&A bot has exactly six: ingest, embed, retrieve, prompt, generate, and validate.
Wrapping all six inside one function — answer_question(query, docs) — means a caller only needs to pass a question and a list of documents. Everything else happens inside, in order, every time.
Each stage can fail independently, which is why you'll trace failures back to their source stage rather than treating the whole pipeline as a black box.
def answer_question(query, docs): # Stage 1: Ingest — chunk every document chunks = [] for doc in docs: chunks.extend(chunk_document(doc, size=400, overlap=50)) # Stage 2: Embed — turn each chunk into a vector chunk_vectors = [embed(c.text) for c in chunks] store_vectors(chunks, chunk_vectors) # upsert into vector store # Stage 3: Retrieve — find the top-3 closest chunks query_vector = embed(query) top_chunks = vector_search(query_vector, top_k=3)
chunk_document(doc, size=400, overlap=50)embed(c.text)store_vectors(chunks, chunk_vectors)vector_search(query_vector, top_k=3)This first stage wires ingest → embed → retrieve inside one function. After this block runs, top_chunks holds the three passages most relevant to the query — the raw material for the answer.
top_chunks is a list of up to 3 Chunk objects whose vectors are closest to the query vector. If the documents have nothing relevant, the store still returns the 3 least-distant chunks — they just won't be useful. That's why the next stage must check relevance before trusting them.
# Stage 3b: Guard against empty retrieval if not top_chunks: return {"answer": "I couldn't find relevant information.", "citations": [], "passed_validation": False} # Stage 4: Prompt — inject chunks + citation rules prompt = build_prompt(query, top_chunks) # from Module 4 # Stage 5: Generate — call the language model answer = generate(prompt) # Stage 6: Validate — faithfulness + retrieval relevance result = validate(answer, top_chunks) # from Module 5 return result
if not top_chunks:build_prompt(query, top_chunks)generate(prompt)validate(answer, top_chunks)This second stage completes the function: it guards against empty retrieval, builds the grounded prompt, generates the answer, and validates it. The function always returns a dict with answer, citations, and passed_validation — so the caller always knows what happened.
validate() returns passed_validation: False. The faithfulness check catches it — it compares every claim in the answer against the retrieved chunks and flags any claim that has no supporting passage.
test_questions = [
"What is the refund policy?",
"How do I reset my password?",
"What are the support hours?",
]
for q in test_questions:
result = answer_question(q, docs=loaded_docs)
passed = result["passed_validation"]
cites = result["citations"]
# TODO: print a one-line summary for each question
# showing q, passed, and the number of citations
print(___)result["passed_validation"]result["citations"]print(___)This is your solo rung: the loop and result-unpacking are done for you — you supply the one missing print() call that reports what happened for each question.
print(f"Q: {q} | passed={passed} | citations={len(cites)}")
# Changed line: the TODO is replaced with this f-string.
# Why: it surfaces the three things you care about — the question text,
# whether validation passed, and how many citations backed the answer.
# If passed=False or citations=0, you know which stage to debug next.
citations=["[Source 1]"] every time, and the answer is vague. Fix: lower size in chunk_document().top_k=1 means one missed chunk kills the answer. Symptom: passed_validation: False on questions that span two passages. Fix: raise top_k to 3–5.validate() doesn't check len(citations) > 0, an answer with no source tags slips through as passed_validation: True. Always assert citations are non-empty.validate() receives the same chunks that were passed to build_prompt() — not a different list.citations is non-empty for each passing answer — a model that never cites has a broken prompt template.top_chunks to confirm retrieval found something, then print the raw answer before validation to see whether the model cited at all.Before you look at the build order below, try to reconstruct it from memory: what are the six stages in sequence, what does each stage hand to the next, and which stage is most responsible for answer quality? Write it out, then compare.
Apply what you learned to Document Q&A Bot.
You set chunk_size=1000 and chunk_overlap=200 when splitting a 50-page PDF. What does the overlap value actually do?
chunks = split(text, size=1000, overlap=200)
Overlap copies a tail of the previous chunk onto the head of the next one. This prevents a sentence or key phrase that straddles a boundary from being lost entirely — it survives intact in at least one chunk. The other options describe things overlap does not do: it has nothing to do with word counts, padding, or compression.
Your retriever embeds the user's question and searches the vector store, but every returned chunk is about a completely different topic. Which root cause from the three common failure modes best explains this?
When the query and the documents are embedded with different models, their vector spaces do not align — similar meaning no longer lands close together, so retrieval returns irrelevant chunks. Bad chunks could cause poor coverage but would not consistently return off-topic results. Model overreach and prompt issues happen after retrieval, not during it.
When would you choose hybrid retrieval (vector + keyword) over vector search alone?
Vector search finds semantically similar text but can miss an exact string like a model number or a person's name if the embedding model generalizes it. Adding keyword search catches those exact matches. Collection size and managed vs. local store choice are separate decisions unrelated to hybrid retrieval, and retrieval method has no effect on whether you need a system prompt.
Look at this system prompt snippet:
"Answer using ONLY the evidence blocks below.
Cite each claim as [Source N].
If the answer is not in the blocks, say so."
A user asks a question and the model responds with a correct-sounding answer but no [Source N] citations anywhere. What is the most likely pipeline stage that broke?
The system prompt tells the model to cite [Source N], but if the chunks were never inserted into the prompt as labeled evidence blocks (e.g., [Source 1]: ... [Source 2]: ...), the model has no labels to reference and cannot produce citations. Missing metadata at ingestion would affect traceability later but would not prevent citation labels from appearing. Validation runs after the answer is generated and does not fix a missing label. A duplicate hash key is an upsert issue unrelated to citation format.
A user submits a question and your retriever returns zero chunks above the relevance threshold. Name the three options for handling this situation and briefly state the trade-off of choosing a silent failure.
When retrieval finds nothing useful, the pipeline must make an explicit choice. Silent failure is the most dangerous option because it gives the user no signal — they cannot tell if the gap is in the document or in the system. A fallback message is honest and preserves trust. Escalation is appropriate when the stakes are high enough to involve a human or a broader data source.