Wire loaders, retrievers, prompts, and model calls into a grounded Q&A flow.
Ingest PDFs, plain text, and web pages using LangChain loaders, then split them into search-sized passages with a text splitter. You'll configure chunk size and overlap for the chatbot's document corpus and inspect the resulting Document objects.
How to load PDFs, text files, and web pages into Document objects, then split them into search-sized chunks using a text splitter.
Why this matters: Every answer your RAG chatbot gives is only as good as the chunks it retrieves — this step defines the evidence shape the entire pipeline depends on.
Your chatbot can't search a 200-page PDF like a human skims it. Feeding the whole file into a language model at once exceeds its context window (maximum text it holds in memory). Retrieval becomes meaningless. The fix: load each source into a object, then split it into search-sized .
A carries two things: the raw page_content string and a dict. Metadata includes source path, page number, and URL. Loaders handle format-specific parsing. You get a consistent object regardless of source type.
Splitting turns each Document into smaller using a . Two parameters control the result: chunk_size (maximum characters per chunk) and chunk_overlap (characters shared between adjacent chunks).
You're building a chatbot corpus from three sources: a product handbook PDF, a plain-text changelog, and a public docs page. Each needs a different loader.
PyPDFLoader — parses a PDF page-by-page. Each page becomes one Document with metadata["page"] set automatically.TextLoader — reads a plain .txt or .md file as a single Document. Fast and zero-config.WebBaseLoader — fetches a URL, strips HTML tags, and returns visible text as a Document with the URL in metadata.All three loaders expose the same .load() method. Your pipeline doesn't care which one ran. The each loader attaches travels with every chunk downstream and surfaces in citations.
Use PyPDFLoader for the handbook, TextLoader for the changelog, and WebBaseLoader for the docs page. Then concatenate all returned lists before splitting.
from langchain_community.document_loaders import PyPDFLoader loader = PyPDFLoader("handbook.pdf") docs = loader.load() # Inspect the first document print(len(docs)) # number of pages print(docs[0].page_content[:120]) print(docs[0].metadata)
PyPDFLoader("handbook.pdf").load()docs[0].page_contentdocs[0].metadataThis loads the PDF and prints the first page's text and metadata — but notice you get one Document per page, not one per topic. Before revealing, predict: what happens if page 1 is just a cover image with no text?
page_content is an empty string '' — PyPDFLoader extracts text layer only; scanned images yield nothing. metadata still shows {"source": "handbook.pdf", "page": 0}. Downstream, an empty chunk wastes an embedding slot and can surface as a blank retrieval result.
from langchain_community.document_loaders import ( PyPDFLoader, TextLoader, WebBaseLoader ) pdf_docs = PyPDFLoader("handbook.pdf").load() txt_docs = TextLoader("changelog.txt").load() web_docs = WebBaseLoader("https://docs.example.com/api").load() all_docs = pdf_docs + txt_docs + web_docs print(f"{len(all_docs)} documents loaded")
PyPDFLoader / TextLoader / WebBaseLoaderpdf_docs + txt_docs + web_docsf"{len(all_docs)} documents loaded"Three loaders, one combined list — the pipeline downstream sees identical Document objects regardless of source format. Predict the output before revealing.
14 — PyPDFLoader yields 12 Documents (one per page), TextLoader yields 1, WebBaseLoader yields 1. Total: 14. Each Document's metadata["source"] identifies its origin.
from langchain.text_splitter import RecursiveCharacterTextSplitter splitter = RecursiveCharacterTextSplitter( chunk_size=500, chunk_overlap=75, # ~15% overlap separators=["\n\n", "\n", " ", ""], ) chunks = splitter.split_documents(all_docs) # TODO: print the total chunk count AND the metadata of chunks[0] # Hint 1: len(chunks) gives the count. # Hint 2: chunks[0].metadata carries source + page from the original Document.
RecursiveCharacterTextSplitterchunk_size=500chunk_overlap=75separators=["\n\n", "\n", " ", ""].split_documents(all_docs)Stop — attempt the TODO before revealing. The splitter is fully configured; you only need to add two print statements that confirm the split worked and that metadata survived.
Changed lines:
print(len(chunks)) # e.g. 47
print(chunks[0].metadata) # {'source': 'handbook.pdf', 'page': 0}
Why it matters: confirming metadata survived the split is the key check — if metadata is empty here, citations downstream will be broken. The chunk count tells you whether chunk_size is reasonable (too few = chunks too large; too many = chunks too small).
Drag to see how chunk_size shifts the precision–coverage tradeoff for your chatbot corpus.
Three failure patterns appear repeatedly in production RAG pipelines:
page_content = ''. You'll see len(chunks[i].page_content) == 0 — filter these before embedding or they waste vector-store slots and surface as blank retrieval hits.splitter.split_text(doc.page_content) instead of split_documents(docs), you get plain strings — no metadata. Every chunk's dict is empty, so citations show "source": None at answer time. Always use split_documents.chunk_overlap >= chunk_size causes an infinite loop or raises ValueError: chunk_overlap must be less than chunk_size. A safe rule: keep overlap at 10–20% of chunk_size.Verify AI-generated loader/splitter code by checking three things: (1) chunks[0].metadata is non-empty, (2) no chunk has len(page_content) == 0, and (3) chunk_overlap < chunk_size. Run these assertions before wiring the chunks into any downstream step.
You now have a list of chunks, each carrying text and metadata. These are raw material for the next step: turning each chunk's text into a dense numeric vector called an .
The chunk boundaries you configured directly shape retrieval quality. A chunk that's too large buries relevant sentences in noise. One that's too small loses surrounding context a language model needs.
In the next module you'll pass this chunks list to an embedding model. Convert each chunk into a vector. Store those vectors in a — making your corpus searchable by meaning, not keywords.
Convert your Document chunks into dense vector embeddings using an OpenAI or open-source embedding model, then persist them in a vector store (Chroma or FAISS). You'll build the indexing path of the chatbot so the corpus is ready for similarity search.
Convert document chunks into vector embeddings and load them into a persistent Chroma vector store ready for similarity search.
Why this matters: This is the indexing backbone of your RAG chatbot — without it, the retriever has nothing to search and the chatbot can't ground its answers in your documents.
Decision this forces: Which embedding model and vector store backend fit the chatbot's scale and latency budget?
Document object contain after text splitting? Name its two main fields.Answer: each Document page_content (raw text) and metadata (source, page number, etc.). Module 1 produced a list. This module indexes it for search.
Your chatbot has clean chunks, but a text list can't search by meaning — only exact keywords. To answer "what is our refund policy?" it needs semantic similarity, not literal matches.
The fix: convert each chunk into a numeric (dense vector encoding meaning). Load vectors into a that finds nearest ones in milliseconds.
An embedding model maps text to a fixed-length number list — a vector — so related passages cluster nearby in high-dimensional space.
During indexing, every chunk is embedded and stored with its vector. At query time, the user's question is embedded with the same model. The store returns chunks whose vectors are closest — that's .
Key constraint: query and documents must use the same model family. Mixing models produces meaningless distances and silent retrieval failure.
Two families dominate: OpenAI's text-embedding-3-small / text-embedding-3-large (API, pay-per-token) and open-source BGE or Sentence Transformers (self-hosted, free).
from openai import OpenAI import chromadb client = OpenAI() # uses OPENAI_API_KEY env var db = chromadb.Client() collection = db.get_or_create_collection("docs") texts = [doc.page_content for doc in chunks] # chunks from module 1 ids = [str(i) for i in range(len(texts))] response = client.embeddings.create( model="text-embedding-3-small", input=texts ) vectors = [r.embedding for r in response.data] collection.upsert(documents=texts, embeddings=vectors, ids=ids)
client.embeddings.create(model=..., input=texts)collection.upsert(documents=, embeddings=, ids=)[r.embedding for r in response.data]This stage embeds the full chunk list in one batched API call and upserts the results into a Chroma collection. Notice that upsert is idempotent — re-running won't duplicate records as long as the ids stay stable.
42 — one record per chunk. Chroma stores the raw text and the vector together, so count() equals the number of upserted documents.
# --- persist (run once, after upsert) --- db = chromadb.PersistentClient(path="./chroma_store") collection = db.get_or_create_collection("docs") collection.upsert(documents=texts, embeddings=vectors, ids=ids) # --- reload (every app restart) --- db2 = chromadb.PersistentClient(path="./chroma_store") collection2 = db2.get_collection("docs") print(collection2.count()) # → 42, no re-embedding needed
chromadb.PersistentClient(path=...)db2.get_collection('docs')collection2.count()Switching from chromadb.Client() (in-memory) to PersistentClient(path=...) is the only change needed to survive restarts. On reload, use get_collection (not get_or_create_collection) so a missing store raises an error rather than silently creating an empty one.
Chroma raises an exception — the collection isn't found. This is intentional: it forces you to run the indexing step first rather than silently querying an empty store.
db = chromadb.PersistentClient(path="./chroma_store") collection = db.get_or_create_collection("docs") texts = [doc.page_content for doc in chunks] ids = [str(i) for i in range(len(chunks))] metas = [doc.metadata for doc in chunks] # e.g. {"source": "handbook.pdf", "page": 3} response = client.embeddings.create( model="text-embedding-3-small", input=texts ) vectors = [r.embedding for r in response.data] # TODO: call collection.upsert() passing texts, vectors, ids, AND metadatas # Hint: the parameter name is `metadatas` (plural)
metadatas=metasdoc.metadataStop — attempt the TODO before revealing the answer. The missing line is the crux: without metadatas, you lose the source and page info needed for citation in the final chatbot.
collection.upsert(documents=texts, embeddings=vectors, ids=ids, metadatas=metas)
Changed lines vs. Stage 1: added metadatas=metas.
Why it matters: Chroma stores the dict alongside the vector. Later, when the retriever returns a chunk, you can read chunk['metadatas'][0]['source'] to cite the original file — without this, the chatbot can't tell the user where the answer came from.
| Option | Retrieval quality | Latency (indexing) | When to choose | Cost | Complexity |
|---|---|---|---|---|---|
| OpenAI text-embedding-3-small | Strong for English; competitive on MTEB benchmarks | Network round-trip per batch; parallelisable | Prototypes and small corpora (< 1 M tokens/month) where API simplicity beats cost. | Pay-per-token (~$0.02 / 1M tokens) | Low — one API key, no infra |
| OpenAI text-embedding-3-large | Best-in-class on MTEB; noticeable lift for long or technical docs | Same network latency; slower per token due to larger model | High-stakes or multilingual corpora where top retrieval quality justifies 5× the cost. | ~$0.13 / 1M tokens | Low — same API, larger bill |
| BGE / Sentence Transformers (local) | BGE-large matches or beats ada-002; smaller models lag on domain-specific text | Fast on GPU; CPU-only can bottleneck large re-indexes | Large corpora, data-privacy requirements, or when you need zero marginal cost at scale. | Free at inference; GPU/CPU infra cost | Medium — model download, hardware sizing |
Click a query to highlight the chunks nearest to it. Chunks on similar topics cluster together — that's what the vector store exploits.
text-embedding-3-large but query with text-embedding-3-small. No error raised — distances become meaningless. Symptom: top-k results look random.get_or_create_collection on reload creates a fresh empty collection. Symptom: collection.count() returns 0, every query returns nothing — no exception.openai.BadRequestError: max batch size exceeded. Fix: batch into ≤ 2048 texts before embeddings.create.collection.count() after upsert. Assert it equals len(chunks).collection.get(ids=['0']). Confirm documents, embeddings, and metadatas are non-null.— where you'll tune top-k, switch to MMR search, and add metadata filters.
Wrap the vector store as a LangChain retriever and tune top-k, search type (similarity vs. MMR), and metadata filters so the chatbot pulls the most relevant chunks for any query. You'll also revisit chunk size from Module 1 to see how it interacts with retrieval precision.
Configure a LangChain VectorStoreRetriever with top-k, similarity vs. MMR search, and metadata filters to pull the most relevant chunks for any query.
Why this matters: The retriever is the precision dial of your RAG chatbot — misconfigure it and the model either misses evidence or drowns in noise, directly hurting answer quality.
Answer: similarity_search() is a raw store method. It has no standard interface. It lacks configurable search type and filter support that a chain can wire up automatically. A wraps the store behind a single get_relevant_documents() contract. Any chain can call it the same way, regardless of what's underneath.
This module configures that wrapper. You'll tune , search type, and filters. The chatbot pulls the right chunks — not just the nearest ones.
Every VectorStoreRetriever exposes three levers that control what comes back for a query.
source == 'handbook_2024.pdf').These three interact: a tight filter with low k is precise but brittle; a wide filter with high k is forgiving but noisy. The right balance depends on your corpus size and query diversity.
Your chatbot serves an HR team. The vector store holds three PDFs: an employee handbook, a benefits guide, and a 2023 policy archive. A user asks: "What's the parental leave policy?"
Without a filter, the retriever might return chunks from the 2023 archive — outdated policy that contradicts the current handbook. With a filter on source == 'handbook_2024.pdf', only current chunks compete.
The handbook has several sections that all mention leave. Setting search_type='mmr' with k=4 returns four chunks that cover different angles — eligibility, duration, pay, and process — instead of four near-identical eligibility paragraphs.
# Assume `vector_store` was built in Module 2 retriever = vector_store.as_retriever( search_type="similarity", search_kwargs={"k": 4}, ) # Test it docs = retriever.get_relevant_documents("What is the parental leave policy?") print(len(docs)) # → 4 print(docs[0].page_content[:120]) # first chunk preview
as_retriever(...)search_type="similarity"search_kwargs={"k": 4}get_relevant_documents(query)as_retriever() converts your vector store into a standard interface. The search_kwargs dict is passed straight through to the underlying store, so k controls how many chunks come back.
It returns all 6 chunks — the store silently caps at the corpus size. No error is raised, but you'll see len(docs) == 6, not 10. This is a common surprise when testing on small corpora.
retriever = vector_store.as_retriever( search_type="mmr", search_kwargs={ "k": 4, "fetch_k": 20, # candidate pool MMR re-ranks "lambda_mult": 0.5, # 0 = max diversity, 1 = max similarity "filter": {"source": "handbook_2024.pdf"}, }, ) docs = retriever.get_relevant_documents("parental leave policy") print([d.metadata["source"] for d in docs]) # all from handbook
search_type="mmr"fetch_k: 20lambda_mult: 0.5filter: {"source": "handbook_2024.pdf"}This stage adds two things to Stage 1: re-ranking for diversity and a filter to scope results to one document. The fetch_k pool (20 here) is the set MMR scores before picking the final k=4 — it must be ≥ k.
MMR can only return as many chunks as it fetched — you'd get 3 chunks, not 4. Always set fetch_k > k (a ratio of 4–5× is typical). Some stores raise a ValueError; others silently return fewer results.
# New scenario: corpus has chunks tagged with metadata["category"] # Categories: "benefits", "conduct", "leave" # Goal: retrieve diverse chunks ONLY from the "leave" category retriever = vector_store.as_retriever( search_type="mmr", search_kwargs={ "k": 4, "fetch_k": 16, "lambda_mult": 0.6, "filter": # ← YOUR LINE: filter to category == "leave" }, ) docs = retriever.get_relevant_documents("How do I apply for parental leave?") assert all(d.metadata["category"] == "leave" for d in docs), "Filter failed!"
assert all(...)lambda_mult: 0.6Stop — attempt the missing filter before revealing. The corpus now uses a category metadata field instead of source. Your task: write the filter dict that scopes retrieval to the "leave" category. Hint: the filter syntax mirrors Stage 2 exactly — only the key and value change.
"filter": {"category": "leave"}
Changed lines vs Stage 2: only the filter key ("source" → "category") and value ("handbook_2024.pdf" → "leave"). Everything else stays the same. The assert will pass if the vector store correctly applies the pre-filter before MMR scoring.
MMR's lambda_mult parameter controls the relevance-diversity tradeoff. Drag to see what each setting prioritises.
Three failure patterns account for most retrieval misses in production chatbots.
len(docs) > 0 after retrieval in dev.fetch_k > k when search_type is mmr, (2) every filter key exactly matches a real metadata field name in your chunks. Print docs[0].metadata to confirm. (3) len(docs) equals k on a corpus large enough to satisfy it. These three checks catch ~90% of silent retriever bugs.With a working retriever in place, the next module — "Engineer the RAG Prompt" — takes the chunks it returns. It injects them into a ChatPromptTemplate so the model answers only from that evidence, not from its parametric memory.
Build a ChatPromptTemplate that injects retrieved context into a system message and instructs the model to answer only from that evidence. You'll write grounding instructions, handle the context-injection slot, and add a citation rule so the chatbot's answers are traceable.
Build a ChatPromptTemplate that injects retrieved passages into a system message and instructs the model to answer only from that evidence, with a citation rule.
Why this matters: This is the layer that turns raw retrieved chunks into honest, traceable answers — without it, your RAG chatbot hallucinates even when the right documents are in the index.
Answer: the returns a list of objects, each holding page_content (the raw passage text) and (source file, page number, etc.).
Module 3 tuned and search type so the right arrive. Now the question is: how do you hand those chunks to the model in a way that keeps its answer honest?
A structures every model call.
A sets the rules.
A human message carries the user's question.
In RAG, the system message injects retrieved passages via a {context} slot ().
It instructs the model to answer only from that evidence ().
It adds a citation rule so every claim is traceable.
Without explicit grounding instructions, the model blends retrieved facts with training-data memory.
This is the main source of in RAG systems.
Design tension: strict "answer only from context" rules kill helpfulness when evidence is partial.
You need a graceful fallback phrase baked into the prompt itself.
Imagine your RAG chatbot answers questions about a software product's refund policy. The retriever pulls two relevant passages from the policy PDF.
Without a grounding instruction, the model might answer "Refunds take 3–5 business days" from training memory — even if your policy says 10 days. The customer gets wrong information, and you can't trace where it came from.
With a grounded prompt, the system message reads: "Use only the passages below. Cite the source filename after each claim. If the answer isn't in the passages, say: 'I don't have enough information to answer that.'"
Now the model either quotes the policy (with a citation) or admits the gap — both outcomes are safe and auditable.
from langchain.prompts import ChatPromptTemplate prompt = ChatPromptTemplate.from_messages([ ("system", "You are a helpful assistant."), ("human", "{question}"), ]) # Invoke with a retrieved context — but where does it go? formatted = prompt.format_messages( question="What is the refund window?" )
ChatPromptTemplate.from_messages([...])("system", "..."){question}prompt.format_messages(...)This prompt has no {context} slot, so retrieved passages are never injected.
The model answers from training memory alone — exactly the risk RAG is supposed to eliminate.
No. The retrieved passages are never passed to the model. The prompt has no {context} variable, so format_messages() ignores any documents the retriever returned. The model answers purely from its training weights — the retrieved evidence is silently discarded.
SYSTEM_TEMPLATE = """ You are a helpful assistant. Answer using ONLY the passages below. After each claim, cite the source like this: [source: filename]. If the answer is not in the passages, say: "I don't have enough information to answer that." Passages: {context} """ prompt = ChatPromptTemplate.from_messages([ ("system", SYSTEM_TEMPLATE), ("human", "{question}"), ])
SYSTEM_TEMPLATE = """..."""{context}"ONLY the passages below""cite the source like this: [source: filename]""I don't have enough information"The {context} slot is where retrieved passages are injected at runtime — this is in practice.
Three instructions do the heavy lifting: the "ONLY" rule grounds the answer, the citation format makes claims traceable, and the fallback phrase handles evidence gaps gracefully.
You join each document's page_content into a single string, typically with a separator like "\n\n" and optionally prefixed with its metadata source. Example:
context = "\n\n".join(
f"[source: {doc.metadata['source']}]\n{doc.page_content}"
for doc in docs
)
This string is then passed as context= when calling prompt.format_messages().
SYSTEM_TEMPLATE = """ You are a helpful assistant. Answer using ONLY the passages below. After each claim, cite the source like this: [source: filename]. If the answer is not in the passages, say: "I don't have enough information to answer that." # TODO: add one instruction that tells the model to reply # in the same language the user asked in. Passages: {context} """ prompt = ChatPromptTemplate.from_messages([ ("system", SYSTEM_TEMPLATE), ("human", "{question}"), ])
# TODO: ...This is a small variation of Stage 2 — the grounding and citation rules are already in place.
Your task: fill in the TODO with one sentence that instructs the model to match the user's language. This is the crux — a multilingual chatbot needs this rule or it defaults to English regardless of the query.
Replace the TODO with:
Always reply in the same language the user used to ask their question.
--- What changed and why ---
This single sentence is the only addition (the rest of the template is identical to Stage 2). It anchors language choice to the user's input rather than the model's default, which is critical for multilingual corpora. No other lines change.
Slide to see how tightening the grounding instruction changes model behaviour and the risk of unhelpful refusals.
{context} variable.KeyError: 'context' at format time.prompt.format_messages(context="TEST_CTX", question="TEST_Q") and read the rendered system message.{context} was replaced, not left as a literal brace.With a verified prompt template, wire it into a full LCEL .
Connect the .
Connect this prompt and the .
Use the pipe operator to link them end-to-end.
Invoke the whole pipeline on a real question.
Wire retriever → prompt → LLM into a LCEL chain using the pipe operator, then invoke it end-to-end on a real question against the chatbot's corpus. You'll complete a partially-built chain, trace the data flow, and confirm grounded output before adding conversation memory.
Wire your retriever, prompt template, and LLM into a single runnable LCEL chain using the pipe operator, then invoke and stream it end-to-end.
Why this matters: This is the assembly step that turns your four separate RAG components into a working chatbot — without it, nothing runs.
Decision this forces: Should the chain use a simple sequential pipe or a more stateful graph (LangGraph) for multi-turn conversation?
Before reading on, recall from memory: in Module 4, what two slots does your ChatPromptTemplate expose, and what fills each one at query time?
context slot is filled by retrieved chunks from your , and the question slot is filled by the user's raw query. Module 5 is where those two feeds connect.You have a retriever that pulls relevant , a that injects them as context, and an LLM ready to generate. The missing step is wiring them into one runnable so one call drives the whole pipeline.
That's exactly what (LangChain Expression Language) does. It lets you compose steps with the | pipe operator. Data flows left to right through retriever, prompt, and LLM without glue code.
In , every component — retriever, prompt, LLM, output parser — implements a common Runnable interface with invoke(), stream(), and batch() methods.
The | operator chains two Runnables. The output of the left becomes the input of the right. LangChain resolves type mismatches automatically. For example, it converts a list of objects from the retriever into a formatted string before handing it to the prompt.
A minimal RAG chain looks like: retriever | prompt | llm | output_parser. Each step receives exactly what the previous step emitted. No manual unpacking is needed.
Because every step shares the same interface, you can swap any component. For example, change the LLM or the retriever without touching the rest of the chain.
# Naive attempt — piping retriever directly into prompt chain = retriever | prompt | llm | StrOutputParser() result = chain.invoke("What is the refund policy?") print(result)
retriever | prompt | llm | StrOutputParser()StrOutputParser()chain.invoke("...")This looks right but fails at runtime because the retriever returns a list of Document objects, while the prompt expects a dict with keys {'context': str, 'question': str}.
LangChain raises a KeyError or ValidationError because the prompt template receives a list of Documents instead of the dict {'context': '...', 'question': '...'}. The {question} key is also missing entirely — the chain never threads the original query through to the prompt.
from langchain_core.runnables import RunnablePassthrough def format_docs(docs): return "\n\n".join(d.page_content for d in docs) chain = ( {"context": retriever | format_docs, "question": RunnablePassthrough()} | prompt | llm | StrOutputParser() ) result = chain.invoke("What is the refund policy?") print(result)
RunnablePassthrough()retriever | format_docs{"context": ..., "question": ...}d.page_contentThe dict literal {"context": retriever | format_docs, "question": RunnablePassthrough()} is a RunnableMap — it runs both branches in parallel and merges their outputs into the dict the prompt expects.
Running this against a corpus with a refund-policy document should produce a grounded answer citing the retrieved passage, not a hallucinated one.
RunnablePassthrough() forwards whatever the chain received as input (the user's question string) unchanged into the 'question' key. Hardcoding the string would make the chain ignore the actual user input — every call would answer the same fixed question regardless of what the user typed.
from langchain_core.runnables import RunnablePassthrough from langchain_core.messages import HumanMessage, AIMessage chat_history = [] def run_turn(question: str) -> str: result = "" for token in chain.stream({ # TODO: fill in the dict keys ??? }): print(token, end="", flush=True) result += token chat_history.append(HumanMessage(content=question)) chat_history.append(AIMessage(content=result)) return result
chain.stream({...})print(token, end="", flush=True)HumanMessage / AIMessagechat_history.append(...)This function streams the chain's output token-by-token and appends each turn to chat_history so the next call can pass prior context to the model.
Stop — attempt the TODO before revealing. Your from Stage 2 expects a dict with two keys; stream() takes the same input shape as invoke().
Replace ??? with:
"question": question,
"chat_history": chat_history
Changed lines vs Stage 2: (1) invoke() → chain.stream() so tokens arrive incrementally instead of all at once; (2) the dict now includes "chat_history" so a memory-aware prompt template can inject prior turns. Your ChatPromptTemplate must also expose a {chat_history} slot (added in Module 4's system message) for this to take effect.
| Option | Multi-turn memory handling | Branching / conditional logic | When to choose | Cost | Complexity |
|---|---|---|---|---|---|
| LCEL Pipe Chain | Manual: you append messages to a list and pass it each turn | Not supported natively; requires custom Python logic outside the chain | Single-turn Q&A or simple multi-turn where you manage chat_history manually in a list. | Minimal overhead; no extra dependencies | Low — a few lines of pipe composition |
| LangGraph Stateful Graph | Built-in: state is persisted across nodes and turns automatically | First-class: edges can be conditional, enabling routing and fallback nodes | Multi-turn chatbots that need persistent state, conditional routing, or human-in-the-loop checkpoints. | Extra dependency (langgraph); more setup code | Higher — define nodes, edges, and a state schema |
context_length_exceeded error from the API. Fix: lower top-k or reduce chunk size in Module 1's splitter.format_docs returns an empty string. The model an answer from parametric memory. There is no error, just wrong output. Fix: log len(docs) before the prompt and add a guard that returns a 'no information found' message when docs is empty.RunnablePassthrough() is in the 'question' branch, not hardcoded. Run two different questions and verify you get two different answers.chain.invoke() with a question whose answer is in your corpus. Manually confirm the answer matches the source passage, not the model's training data.The next module — 'Handle Errors, Guard Against Hallucination, and Optimize' — turns these manual checks into systematic logic and grounding guards. Your chatbot then fails safely at scale.
Identify and fix the three main RAG failure modes — retrieval misses, hallucinated answers, and latency spikes — by adding fallback logic, grounding checks, and caching. You'll also learn how to verify AI-generated chain code for correctness and security before shipping to production.
How to detect and fix the three main RAG failure modes — retrieval misses, hallucinated answers, and latency spikes — and audit AI-generated chain code before shipping.
Why this matters: Your chatbot chain is built; this module makes it reliable enough to put in front of real users by adding fallback logic, grounding checks, and caching.
Decision this forces: Which failure mode poses the highest risk for this chatbot's use case, and which mitigation ships first?
Answer: → → , chained with 's pipe operator.
That chain is now running. But it can fail in three distinct ways before reaching a user.
Every production RAG chain fails in one of three ways, and each leaves a different fingerprint.
Each failure mode has a targeted mitigation: a strategy for misses, a check for hallucinations, and caching plus async retrieval for latency.
Imagine your chatbot answers questions about an employee handbook. Here's how each failure mode surfaces.
A user asks "What is the parental leave policy for contractors?" The handbook uses "non-permanent staff," not "contractors."
The results all score below 0.5 and miss the right passage. The LLM fills the gap with generic training data.
Observable symptom: the answer sounds plausible but cites no handbook section. A manual search finds the correct passage easily.
The retriever returns three chunks about leave policy. The LLM correctly quotes the 12-week figure.
It adds "as extended by the 2023 amendment" — a detail absent from the retrieved chunks.
Observable symptom: the answer contains a specific claim (date, number, policy name) you cannot find in any retrieved chunk.
Embedding takes 300 ms. Vector search takes 200 ms. LLM call takes 1.5 s. All sequential: ~2 s per query.
Observable symptom: response time degrades as corpus size grows. Repeated identical queries are just as slow as new ones.
def rag_with_guards(query, retriever, llm, threshold=0.5): docs = retriever.get_relevant_documents(query) # returns [(doc, score)] grounded = [d for d, score in docs if score >= threshold] if not grounded: return "I don't have enough information in the handbook to answer that." context = "\n\n".join(d.page_content for d in grounded) answer = llm.invoke(build_prompt(context, query)) # Grounding check: at least one retrieved sentence must appear in the answer supported = any(sent in answer for sent in context.split(".") if len(sent) > 20) if not supported: return "I found relevant passages but couldn't form a grounded answer." return answer
retriever.get_relevant_documents(query)[d for d, score in docs if score >= threshold]context.split(".")build_prompt(context, query)This function wraps the chain with two guards: a score-threshold that fires when retrieval misses, and a lightweight check that rejects answers not anchored to the retrieved context.
The grounding check here is a heuristic — a production system would use an LLM-as-judge or a dedicated faithfulness scorer (e.g. Ragas) for higher accuracy.
It returns "I don't have enough information in the handbook to answer that." — because no chunk clears the 0.5 threshold, grounded is empty, and the early return fires before the LLM is ever called.
import asyncio, hashlib _embed_cache = {} def cached_embed(text, embed_fn): key = hashlib.md5(text.encode()).hexdigest() if key not in _embed_cache: _embed_cache[key] = embed_fn(text) # only calls API on cache miss return _embed_cache[key] async def async_retrieve(query, retriever): # TODO: await the retriever's async method here pass async def fast_rag(query, retriever, llm): docs = await async_retrieve(query, retriever) # … rest of rag_with_guards logic …
hashlib.md5(text.encode()).hexdigest()_embed_cache[key] = embed_fn(text)async def / awaitasync_retrieveThis stage adds two latency cuts: an in-process embedding cache that skips the API call for repeated queries, and an async retrieval wrapper so the event loop isn't blocked while waiting for the .
The TODO on line 11 is the crux — fill it in before revealing the answer.
Line 11 becomes: return await retriever.aget_relevant_documents(query)
Changed line: the pass is replaced with a real await call on the retriever's async method. This is the crux — without await, the coroutine is created but never executed, and docs would be a coroutine object instead of a list of Documents.
You've closed the loop: documents load and chunk (Module 1), embeddings index them (Module 2), retriever selects evidence (Module 3), prompt grounds the answer (Module 4), LCEL chain runs end-to-end (Module 5).
This module hardens it against the three failure modes.
Before the capstone, audit your chatbot chain: does it return a fallback on retrieval miss? Does it reject unsupported answers? Does it cache repeated embeddings?
The guards you added here stand between a demo and a production system.
| Option | User-visible harm if untreated | Detection difficulty | When to choose | Cost | Complexity |
|---|---|---|---|---|---|
| Retrieval miss fallback | High — user gets a confident wrong answer drawn from model memory. | Medium — low scores are measurable; you can log and alert on them. | Ship first when your corpus uses domain-specific vocabulary that diverges from user query phrasing — e.g. legal, medical, or internal jargon. | Negligible — no extra API calls; just filters existing results. | Low — a score threshold + 'I don't know' response is a one-function change. |
| Grounding / hallucination check | Very high — hallucinated facts look authoritative and are hard for users to challenge. | High — fluent, plausible-sounding answers require a faithfulness scorer or manual review to catch. | Ship first when your chatbot answers high-stakes questions (compliance, medical, financial) where a fabricated detail causes real harm. | Low (heuristic) to High (LLM judge — doubles LLM calls). | Medium — a heuristic check is fast; an LLM-as-judge scorer adds latency and cost. |
| Latency optimization (cache + async) | Medium — slow responses hurt UX but don't produce wrong answers. | Low — latency is directly measurable with any tracing tool. | Ship first when p95 response time exceeds your SLA or when repeated queries dominate traffic (e.g. an FAQ bot). | Low — saves embedding API costs on cache hits. | Low (in-process cache) to Medium (async refactor of the chain). |
AI-generated LangChain code for this module tends to fail in four specific ways — check each before shipping.
docs. If it prints <coroutine object ...> instead of a list, the await is missing.Before looking at the summary: from memory, name the six build steps in order and state what each step produces that the next step consumes. Then identify which step is most likely to cause a hallucinated answer and which setting you'd change first to fix it.
Apply what you learned to A production-ready RAG chatbot that loads documents, retrieves relevant context, and answers questions grounded in that context.
You set chunk_size=200 and chunk_overlap=0 in RecursiveCharacterTextSplitter. A user asks a question whose answer spans two adjacent paragraphs. What is the most likely retrieval outcome?
Zero overlap means adjacent chunks share no tokens. If the answer bridges a boundary, neither chunk is self-contained, so even if both are retrieved the LLM lacks the connective text. Adding overlap (e.g., 20-10% of chunk_size) lets boundary-spanning content appear in at least one chunk. The retriever does not merge chunks at query time, and the embedding model scores each chunk independently — it does not average vectors across chunks.
Your chatbot serves a legal firm with 50 GB of case documents. Retrieval latency must stay under 200 ms and re-indexing must not happen on every restart. Which combination best fits these constraints?
Persisting the Chroma store to disk and reloading it eliminates re-indexing on restart, directly satisfying that constraint. A smaller, cost-efficient embedding model keeps per-query latency within the 200 ms budget for a large corpus. Rebuilding FAISS from raw documents on every startup violates the no-re-indexing requirement. Skipping embeddings entirely removes semantic search capability, which is the core value of RAG.
A user reports that the chatbot keeps returning the same three passages even when their questions vary widely. Which retriever change directly addresses this?
MMR penalizes redundancy by balancing relevance against diversity, so it actively avoids returning near-duplicate passages. Similarity search ranks purely by vector closeness, which causes repetitive results when several chunks are semantically similar to the query. Increasing overlap affects chunk boundaries, not result diversity. A metadata filter narrows the corpus but does not diversify results within it. Reducing top-k makes the problem worse by returning fewer, not more varied, passages.
Read this chain definition:
chain = retriever | prompt | llm
response = chain.invoke({"question": user_input})
The prompt template has a {context} slot and a {question} slot. What is wrong with this chain?
A VectorStoreRetriever returns a list of Document objects. The prompt template expects a string in {context} and a string in {question}. Without an intermediate step that formats the documents into a string and packages both inputs into the correct dict, the prompt receives a malformed input and the chain raises an error or silently passes raw objects. LCEL does not auto-map retriever output to named prompt slots. chain.invoke is the correct method for dict inputs in LCEL.
Name two distinct failure modes your RAG chatbot can exhibit at runtime, and for each one state the mitigation strategy covered in the lesson.
The lesson's final module covers four risk areas: retrieval misses (fixed by fallback strategies), hallucination (fixed by grounding checks and prompt instructions), prompt injection (fixed by auditing retrieved content), and silent retrieval failures (fixed by logging and inspecting context). A strong answer names the failure clearly and pairs it with a concrete, lesson-specific mitigation — not a generic 'add error handling' response.