Learn the open-book pattern for grounding model answers in retrieved sources.
You trace exactly why a language model's frozen training knowledge leads to hallucination, and you see the open-book analogy that frames every step that follows. The running scenario throughout this lesson is a customer-support bot answering questions about a software product's changelog.
This module explains why a language model's frozen training knowledge goes stale and causes hallucination, and introduces RAG as the open-book fix.
Why this matters: Before you can build a reliable customer-support bot, you need to understand exactly why a plain LLM will confidently give wrong answers — and what RAG does differently.
Your support bot tells a user that version 3.2 fixed a login bug. Your team actually shipped that fix in version 3.4, two months later. The user follows the wrong steps and files another ticket. How did the bot get it so wrong?
A learns from text collected up to a fixed date. Everything it knows is baked into its weights during training. This built-in knowledge is called : facts stored in the model's parameters.
Once training ends, that memory freezes. The model can't read your latest changelog, pricing page, or anything after its cutoff date.
When the model doesn't know the real answer, it doesn't say "I don't know." Instead it generates a plausible-sounding response from patterns in its training data. This is called : the model produces text that sounds confident but is factually wrong or made up.
In the changelog scenario, the bot might invent a version number, misattribute a bug fix, or describe a feature that was never shipped. The answer looks polished, so users trust it — and that's exactly what makes hallucination dangerous in a support context.
Here's the naive approach: a user asks the bot a question. The bot answers purely from frozen training knowledge — no changelog, no docs, nothing live.
User: "Which version fixed the broken CSV export?"
Bot (no retrieval): "The CSV export issue was resolved in version 3.1.0, released in March."
This fails every time you ask a plain LLM about post-training changes. The model has no way to say "I don't know" — it just generates the most likely-sounding answer.
solves stale memory by giving the model a live reference before it answers. The system first the relevant passage from your documents, then hands it to the model as context.
Feeding real evidence into the model's input is called . The model's answer is now in your changelog — not in frozen training memory.
RAG lets an LLM answer from your documents instead of frozen memory. The answer stays accurate as your product changes.
You know the problem: frozen parametric memory causes hallucination. You know the fix: RAG retrieves real evidence before the model writes. But how does retrieval actually work?
Two paths work together. The indexing path loads your changelog, splits it into searchable pieces, and stores them in an . The query path takes a user question, searches the index for relevant pieces, and injects them into the model's before answering.
Each point is a type of question your support bot might face. Click a query to see which approach handles it best. Points near the top-right are answered accurately with fresh knowledge; bottom-left means stale or hallucinated answers.
Even after you understand the problem, three failure patterns catch teams off guard:
You build a single mental map of the full RAG pipeline — indexing path and query path — and name each stage before diving into any one of them. Using the changelog-bot scenario, you trace a user question all the way to a cited answer at the overview level.
A full map of the RAG pipeline — the offline indexing path and the live query path — traced through a changelog-bot example.
Why this matters: Before you build any part of RAG, you need to see how all seven stages connect so you know where your work fits.
Module 1 showed that an 's knowledge is frozen at training time — it can't look anything up.
When a customer asks your changelog bot about a release from last week, the model has no record of it. It fills the gap by guessing, which is exactly how happens.
fixes this by giving the model an open book to read before it answers. This module maps out every page of that book — the full pipeline from your docs to a cited answer.
system has two separate paths that run at different times.
The indexing path runs once or on schedule. It reads documents, splits them into small pieces called , and stores them in a searchable .
The query path runs every time a user asks a question. It searches the index, pulls the best chunks, and hands them to the model.
A is a short passage cut from a larger document — typically a few sentences to a paragraph.
The model can only read a limited amount of text at once (its context window, or working memory). The pipeline needs chunks because of this constraint.
Smaller chunks also make search more precise. You retrieve the one paragraph that answers the question, not an entire 50-page manual.
Each chunk gets converted into an — a list of numbers that captures its meaning. This lets the system find it by meaning, not just by keyword.
A customer types: "What changed in version 2.4?" Here is exactly what happens at each stage.
The answer is — every fact came from the retrieved chunk, not from the model's frozen memory. That grounding is what makes the answer to your actual changelog.
Click a path to see which stages belong to it. Stages on the left run offline (once); stages on the right run live (every query).
Three failure patterns show up most often at this overview level. Know them before you build.
You now have a mental map of the full pipeline. Both paths, all seven stages, and the role of a chunk are clear.
The next module zooms into the step — the first stage of the live query path. It shows exactly how it works.
You'll follow the question "What changed in version 2.4?" as it gets converted into an and matched against stored chunk vectors to find the right evidence.
You follow a fully worked example — a user asks 'What changed in version 2.4?' — and watch the retrieval step convert that question into an embedding, search the index, and return the top matching changelog chunks. You also see what bad retrieval looks like and why chunk quality matters more than model choice.
This module shows how a user question is converted into an embedding and matched against stored chunks to retrieve the most relevant evidence.
Why this matters: Retrieval is the step that decides which facts the model sees — get it wrong and even the best model gives a bad answer.
Answer: the pieces are called , and each chunk is stored as an — a list of numbers that captures the chunk's meaning. The is the searchable store of those number-lists. Module 3 picks up exactly here: a user question arrives, and the pipeline must find the right chunks fast.
When a user types a question, the converts that question into an — the same kind of number-list used for the chunks.
It then measures how close the question's number-list is to every chunk's number-list in the . Chunks with similar meaning land close together in this number space, so "close" means "relevant."
The retriever picks the top-k closest chunks — say, the best 3 — and hands them forward. Those chunks become the evidence the model will read before writing its answer.
Follow the retrieval step end-to-end for the question "What changed in version 2.4?" — the running example for this lesson.
The passes "What changed in version 2.4?" to the embedding model. Out comes a number-list — something like [0.12, -0.87, 0.34, …] with hundreds of values. This is the embedding.
The pipeline compares that number-list to every chunk in the . Changelog chunks about v2.4 — auth fix, rate-limit change, UI update — score high because their meaning is close. The install guide and billing FAQ score low.
The retriever surfaces the three highest-scoring chunks and passes them forward. The next stage — augmentation — will slot these chunks into the prompt alongside the user's question.
# The user's question question = "What changed in version 2.4?" # Step 1: embed the question (same model used for chunks) query_vector = embed(question) # Step 2: search the index for the closest chunks results = index.search(query_vector, top_k=3) # Step 3: inspect what came back for chunk in results: print(chunk.score, chunk.text)
embed(question)index.search(query_vector, top_k=3)chunk.scorechunk.textThis pseudocode shows the three retrieval steps in plain Python-style code. Notice that embed() is called on the question — not just on documents — because both sides must live in the same number space for the comparison to work.
Each item in results holds (1) a similarity score — a number showing how close the chunk is to the question — and (2) the chunk text itself. The loop on the last two lines prints both: chunk.score and chunk.text.
question = "What changed in version 2.4?" query_vector = embed(question) # TODO: search the index but return only chunks # where chunk.metadata["version"] == "2.4" results = index.search( query_vector, top_k=3, filter=??? # <-- your line goes here ) for chunk in results: print(chunk.score, chunk.text)
filter={"version": "2.4"}chunk.metadata["version"]Stop — attempt this before revealing. The embed-and-search logic is already filled in; your job is the one missing line that narrows results to v2.4 chunks only.
Hint 1: each chunk has a metadata dict with a "version" key. Hint 2: the filter should be a dict that matches that key to the string "2.4".
Replace ??? with {"version": "2.4"}. CHANGED LINE: filter={"version": "2.4"}. Why it matters: without the filter, a question about v2.4 might pull in v2.3 or v2.1 chunks that score reasonably well — the model would then mix up version details. Metadata filtering is the crux of this variation: it shows that embedding similarity alone isn't always enough; structured metadata narrows the candidate set before scoring.
Click a query to see which changelog chunks land closest to it in meaning. Chunks that share the query's topic cluster nearby; unrelated chunks sit far away.
Poor retrieval is the most common reason a RAG answer is wrong. It's almost always a chunking problem, not a model problem.
A doesn't write anything — it only selects and passes. Its output is a ranked list of with scores, ready for the next stage.
In the next module you'll see how those chunks get assembled into a with the user's question. The model then turns that combined input into a grounded, cited answer.
You see how the retrieved changelog chunks are assembled into a prompt alongside the user's question, then watch the model write a grounded, cited answer. A completion exercise asks you to spot what's missing when the evidence is left out of the prompt.
This module shows how retrieved changelog chunks are assembled into an augmented prompt and how the model uses that evidence to write a grounded, cited answer.
Why this matters: Understanding augmentation tells you exactly why adding evidence to the prompt changes the model's output — the core mechanic behind every RAG system you'll build.
The retriever handed back a small set of — short, search-sized passages from the changelog that scored closest to the user's question.
Those chunks are now sitting in memory, waiting to be used. This module is about what happens next: you take those chunks and build the prompt that the model will actually read.
means adding retrieved evidence to the before the model sees it. Without this, the model relies only on frozen training knowledge.
An augmented has three parts: the evidence (retrieved chunks), the question (user query), and the instructions (rules like 'cite sources').
The model reads all three and writes an answer in your evidence.
A customer asks the changelog bot: "What changed in version 2.4?" Retrieval returned two chunks from the changelog. Here is the full augmented prompt the system assembles before calling the model.
"You are a helpful support assistant. Answer using ONLY the evidence below. Cite the chunk number for every fact you state."
[Chunk 1] v2.4 release notes: 'Dark mode added to the dashboard. Users can toggle it in Settings > Display.'
[Chunk 2] v2.4 release notes: 'Export to CSV now supports Unicode characters. Previous versions dropped non-ASCII text.'
"What changed in version 2.4?"
The model reads all three parts as one block of text. Because the evidence is right there, it can write a specific, cited answer instead of guessing.
The (Large Language Model — the AI that writes text) reads the full augmented prompt and generates a response. Because the evidence is in the prompt, the model can quote specific facts instead of relying on memory.
A well-grounded answer for the v2.4 question looks like this:
"Version 2.4 added dark mode to the dashboard (Chunk 1) and fixed CSV export to support Unicode characters (Chunk 2)."
This is called — every claim in the answer can be traced back to a chunk. When the evidence is missing, the model falls back on parametric memory and risk rises sharply.
Here is the prompt the bot is about to send:
"You are a helpful support assistant. Answer using ONLY the evidence below. Cite the chunk number for every fact.
What changed in version 2.4?"
Hint 1: Count the three parts. Hint 2: What does 'use ONLY the evidence below' mean when there is no evidence?
Slide to see how the number of evidence chunks in the prompt changes what the model can say.
Three things go wrong most often at augment-and-generate.
You've seen the full query path: retrieve → augment → generate grounded, cited answers. Evidence reduces but doesn't eliminate it.
The next module — Grounding, Hallucination, and Where RAG Can Still Fail — explains why evidence helps and covers three main RAG failure modes. Knowing these separates a demo from a trustworthy system.
You revisit the hallucination concept from Module 1 (spaced retrieval) and see precisely why evidence in the prompt reduces it — then work through the three main ways RAG can still go wrong: wrong chunks retrieved, chunks ignored by the model, and stale index. You finish by checking a sample answer for groundedness.
This module explains why RAG reduces hallucination and maps the three specific ways it can still fail — wrong retrieval, ignored evidence, and a stale index.
Why this matters: Knowing where RAG breaks helps you build a bot you can actually trust and audit, rather than one that sounds confident but quietly gets things wrong.
Decision this forces: Is a plain LLM sufficient, or does this use case require grounded, citable answers from a specific corpus?
The answer: the retrieved evidence placed inside the (the full text sent to the model). Without it, the model can only draw on its frozen training knowledge — what Module 1 called . With it, the model has a passage to quote, so it can write a answer instead of a confident guess.
That confident-but-wrong guess is a : the model produces text that sounds correct but isn't supported by any real source. Putting evidence in the prompt gives the model something concrete to anchor its words to, which is exactly what is designed to do.
But RAG doesn't make hallucination impossible — it just changes where things can go wrong. This module maps those new failure points so you know what to watch for.
A language model () generates each word by predicting the most likely next word based on the prompt. When the prompt contains a relevant passage, the model's prediction steers toward that text.
Without evidence, the model relies on — facts from training that may be outdated or wrong. With evidence, the model can copy, paraphrase, or cite the passage directly.
This is why works: facts come from retrieved text, not memory. The risk of drops because there is less gap to fill with guesses.
RAG adds a retrieval step with its own failure points. A perfect model still fails if evidence is wrong, missing, or ignored.
The fetches (short passages) most similar to the question. Vague questions or poor chunk splits return irrelevant passages. The model writes a confident answer based on wrong evidence.
Result: the answer sounds plausible but references the wrong version or topic.
Sometimes the right chunks are retrieved but the model doesn't use them. Long prompts bury relevant passages in the middle. The model drifts back to parametric memory and loses .
Result: the answer contradicts retrieved passages or adds unretrieved details.
The is built at a point in time. If documents update but the index doesn't, the retriever fetches old chunks.
Result: the bot quotes outdated policies or version numbers.
Your changelog bot handles customer questions about software releases. Here is how each failure mode appears in that scenario.
Checking means asking: does every claim in the answer appear in the retrieved chunks? Here is a worked example using the changelog bot.
"Version 2.4 replaces the login screen with a single-sign-on (SSO) flow. Users no longer enter a password directly. The change applies to web and mobile clients."
"In v2.4 we introduced SSO so users don't type a password. This affects both the web app and the mobile app."
"In v2.4 we introduced SSO. The new flow uses OAuth 2.0 and requires users to have a Google account."
If question 1 is 'no' for any claim, or questions 2–3 are 'yes', the answer is not fully grounded.
Slide to see how the number of retrieved chunks in the prompt shifts the hallucination risk and the 'chunks ignored' risk. There's a sweet spot — too few and the model guesses; too many and it gets lost.
You've traced the full RAG arc: why frozen training knowledge causes , how the pipeline turns questions into cited answers, and where it breaks.
The capstone challenge asks you to build and audit a RAG pipeline end-to-end. Ingest a document, retrieve evidence, generate an answer, and run the groundedness check. Everything you've practiced comes together there.
Before you see the summary, try to reconstruct the pipeline from memory: what problem does RAG solve, what are the three stages in order, and what makes an answer 'grounded'? Sketch it out, then check your version against the build order below.
Apply what you learned to RAG (Retrieval-Augmented Generation).
What is the core reason a plain LLM's knowledge goes stale over time?
A plain LLM stores everything it knows in its weights — called parametric memory — which are frozen after training. Nothing that happens after the training cutoff gets in. The 'forgets after each conversation' option confuses session memory with parametric memory. The context-window option describes a real limit but not why knowledge goes stale. The retriever option describes a RAG component, not a plain LLM.
A RAG pipeline has three stages. Which sequence is correct?
The pipeline always retrieves relevant chunks first, then augments the prompt by inserting that evidence, and finally generates the answer. Augmenting before retrieving makes no sense — you need the evidence before you can add it. Generating before augmenting would give you a plain LLM answer with no grounding. The last option would require re-running generation, which is not the standard flow.
Consider this simplified augmented prompt sent to the LLM:
Evidence: 'Refunds must be requested within 30 days of purchase.'
Question: 'Can I get a refund after 45 days?'
Instruction: 'Answer using only the evidence above.'
Which part of this prompt is the augmentation?
Augmentation specifically refers to inserting the retrieved evidence into the prompt — that is what changes a plain prompt into an augmented one. The instruction is a system directive, not the augmentation itself. The user's question is the input that triggered retrieval. The model's final answer is the output of the Generate stage, not part of the prompt structure.
Name three failure modes that are specific to RAG (not just plain LLM failures). Give one sentence on each explaining why it causes a bad answer.
RAG introduces its own failure points on top of plain LLM risks. Chunking errors happen before retrieval even runs. Retrieval misses mean good evidence never reaches the prompt. Faithfulness failures mean the model had the evidence but did not use it. A stale index means the offline indexing path was not maintained. A correct answer names three of these with a clear causal link to a bad output.
A company wants to build a chatbot that answers questions strictly from its internal legal contracts, and every answer must be traceable to a specific document. When would a plain LLM be sufficient instead of RAG?
The requirement for answers traceable to specific internal documents is exactly the signal that RAG is needed. A plain LLM cannot cite proprietary contracts it was never trained on, regardless of when they were written — private documents are not in public training data. Fitting documents in the context window is a workaround for very small corpora but does not scale and still does not provide a retrieval-and-citation mechanism. Larger models still hallucinate and cannot access private documents they have not seen.