LangChain's document loaders, vector stores, LCEL chains, and LLM integrations give you a composable pipeline to build a production-ready RAG system…
Load a folder of PDFs and Markdown files with LangChain's DirectoryLoader, then split them into ~500-token chunks with RecursiveCharacterTextSplitter — the same corpus you'll index and query throughout the lesson. You'll see a worked split, then adjust chunk_size and chunk_overlap yourself.
Load PDFs and Markdown files with DirectoryLoader, then split them into retrievable chunks with RecursiveCharacterTextSplitter.
Why this matters: The chunks you produce here are the exact corpus you'll embed and query in every later module — getting size and overlap right now prevents silent retrieval failures later.
Decision this forces: What chunk size and overlap suit your document type and expected query length?
Documents are too long for retrieval systems whole. Cut them into : search-sized passages ranked and fed to models as evidence.
Chunk size controls precision vs. context. Small chunks (≤200 tokens) rank precisely but lack context. Large chunks (≥1000 tokens) carry meaning but dilute relevance and consume faster.
Overlap lets sentences straddling boundaries appear in both adjacent chunks. Retrieval won't silently drop them. Start with 10–20% of chunk_size — e.g., 100 tokens on a 500-token chunk.
You're building a RAG pipeline over a product knowledge base. You have 40 PDFs (release notes, specs) and 20 Markdown files (how-to guides) in one folder.
A single DirectoryLoader points at that folder with two glob patterns. It returns a flat list of LangChain Document objects — one per file, each with text and (source path, page number).
Pass that list to RecursiveCharacterTextSplitter. It splits on paragraph breaks first, then sentences, then words. Character cuts come last, preserving natural boundaries.
Each output chunk inherits the parent document's metadata. You always know which file a passage came from — critical for citation and debugging.
from langchain.document_loaders import DirectoryLoader # Naive: one glob, no loader specified loader = DirectoryLoader("./docs", glob="**/*") docs = loader.load() print(len(docs))
This looks right but silently skips or errors on file types it can't parse.
Predict: what does len(docs) return when the folder has 40 PDFs and 20 Markdown files?
It prints 20 (or raises UnstructuredFileException on the first PDF). Without specifying loader_cls, DirectoryLoader defaults to UnstructuredFileLoader, which may not have the 'unstructured' package installed for PDFs — so PDF files either raise an ImportError or are silently skipped, depending on version. You see fewer docs than expected with no warning.
from langchain.document_loaders import DirectoryLoader from langchain.document_loaders import PyPDFLoader, TextLoader pdf_loader = DirectoryLoader("./docs", glob="**/*.pdf", loader_cls=PyPDFLoader) md_loader = DirectoryLoader("./docs", glob="**/*.md", loader_cls=TextLoader) docs = pdf_loader.load() + md_loader.load() print(len(docs), docs[0].metadata)
Two loaders, each with an explicit loader_cls, are combined into one flat list — the fix for the silent-skip problem in Stage 1.
Predict: what does docs[0].metadata contain for a PDF?
{'source': './docs/release-notes-v2.pdf', 'page': 0} — PyPDFLoader emits one Document per page, so 'page' is the 0-based page index and 'source' is the file path. TextLoader emits one Document per file with only 'source'.
from langchain.text_splitter import RecursiveCharacterTextSplitter splitter = RecursiveCharacterTextSplitter( chunk_size=500, chunk_overlap=100, length_function=len, # character count; swap for tiktoken for token count ) chunks = splitter.split_documents(docs) print(len(chunks)) print(chunks[0].page_content[:120]) print(chunks[0].metadata)
RecursiveCharacterTextSplitter tries paragraph breaks (\n\n), then newlines, then spaces before cutting mid-word — so boundaries stay natural.
Predict: if the original docs list has 60 Documents averaging 2000 characters each, roughly how many chunks does this produce?
About 300 chunks. Effective stride per chunk = 500 - 100 = 400 chars. 120 000 / 400 ≈ 300. Each chunk also inherits its parent's metadata keys (source, page), so chunks[0].metadata still shows the originating file.
from langchain.text_splitter import RecursiveCharacterTextSplitter # Scenario: your queries average 3-4 sentences; answers often # span paragraph boundaries. Increase overlap to 20% of chunk_size. splitter = RecursiveCharacterTextSplitter( chunk_size=800, chunk_overlap=___, # TODO: fill in 20% of chunk_size length_function=len, ) chunks = splitter.split_documents(docs) print(f"{len(chunks)} chunks, first 80 chars: {chunks[0].page_content[:80]}")
Stop — attempt this before revealing. The scenario changes chunk_size to 800 and asks for 20% overlap. What integer goes in the blank, and what effect does it have on chunk count vs. Stage 3?
Hint 1: 20% of 800 is a round number. Hint 2: larger overlap → smaller effective stride → more chunks total.
chunk_overlap=160 (20% of 800). Changed lines: chunk_size=800 → wider context per chunk; chunk_overlap=160 → stride = 640. 120 000 / 640 ≈ 188 chunks — fewer than Stage 3's 300 because the larger window covers more text per chunk, even though overlap is proportionally bigger. Trade-off: each chunk is richer in context but less precise for short, specific queries.
Cause: no loader_cls specified, so unsupported types are dropped silently. Fix: set loader_cls explicitly. Assert len(docs) matches your expected file count.
Cause: call splitter.create_documents(texts) instead of split_documents(docs). The former takes raw strings and produces chunks with empty metadata. Result: chunk.metadata == {} for every chunk; source attribution is lost.
chunk_size=500 with length_function=len measures characters, not tokens. A 500-char chunk is roughly 100–150 tokens for English prose. Use a tiktoken-based length function for token-accurate splitting.
| Option | Retrieval precision | Context per chunk | When to choose | Cost | Complexity |
|---|---|---|---|---|---|
| Small (200–300 chars) | High — narrow match | Low — may miss surrounding context | FAQ-style docs, short factual queries where a single sentence answers the question | More chunks → more embeddings to store and search | Low |
| Medium (400–600 chars) | Good balance | Enough for most answers | General prose docs (specs, guides, release notes) with multi-sentence queries — the default starting point | Balanced chunk count | Low |
| Large (800–1200 chars) | Lower — broad match | Rich — full paragraphs | Long-form narrative docs or queries that require multi-paragraph reasoning; only when your context window can afford it | Fewer chunks but each embedding covers more noise | Low |
Embed the chunks from Module 1 with OpenAIEmbeddings (text-embedding-3-small, 1536 dimensions) and persist them in a Chroma collection — the vector store the retriever will query in Module 3. A worked example shows the full embed-and-persist call; you'll swap in a different embedding model to complete the pattern.
Embed Document chunks with OpenAI's text-embedding-3-small and persist them in a Chroma vector store — the index Module 3 will query.
Why this matters: Your SEO corpus is useless until it's searchable; this module converts raw text chunks into a queryable vector store that powers every retrieval call downstream.
Decision this forces: Which embedding model and vector store fit your latency, cost, and scale requirements?
Before reading on, predict: what does Module 1's RecursiveCharacterTextSplitter actually produce — what Python type, and what fields does each object carry?
Document objects, each with a page_content string (the chunk text) and a metadata dict (source path, page number, etc.). Those are exactly the objects you'll hand to the embedder now.Module 1 left you with a list of — text slices your retriever will eventually search. But a retriever can't compare raw text at scale; it needs numbers.
This module converts those chunks into and stores them in a so Module 3 can query them in milliseconds.
An maps any text string to a fixed-length list of numbers — a vector — where semantic closeness becomes geometric closeness.
During indexing, every chunk is embedded and stored. At query time, the user's question is embedded with the same model, and the store returns chunks whose vectors are nearest — that's .
The key constraint: query and documents must use a compatible model family. Mixing models (e.g. indexing with text-embedding-3-small but querying with a different one) silently returns garbage rankings.
OpenAI's text-embedding-3-small outputs 1536-dimensional vectors and is the default in this lesson — fast, cheap, and accurate enough for most RAG workloads.
Imagine you've loaded 200 SEO landing pages and split them into ~500-token chunks in Module 1. You now have roughly 800 Document objects, each carrying the chunk text and metadata like source URL and page_title.
You pass the whole list to Chroma.from_documents(). Under the hood, Chroma calls the embedding model once per chunk, stores each vector alongside its text and metadata, and writes the collection to disk.
The next day you restart your script. Instead of re-embedding (and paying again), you call Chroma(persist_directory=..., embedding_function=...) to reload the existing collection — the vectors are already on disk.
Module 3 will then call .as_retriever() on that reloaded store to fetch the top-k chunks for any query — but that's the next step.
from openai import OpenAI import chromadb client = chromadb.Client() # in-memory, no persist collection = client.create_collection("seo_pages") # Index with text-embedding-3-small collection.add(ids=["c1"], documents=["best meta tags for SEO"]) # Query — but forget to embed the query first results = collection.query(query_texts=["meta description tips"], n_results=1) print(results) # What happens?
This shows the raw Chroma client API — not the LangChain wrapper — to expose what the wrapper hides.
The collection was created with no explicit embedding function, so Chroma falls back to its default (all-MiniLM-L6-v2). If you later swap to text-embedding-3-small for queries, the vector spaces don't match and rankings are meaningless — no error is raised.
The query returns a result (distance score looks plausible), so no exception fires. The silent failure: if you reload the collection and pass OpenAIEmbeddings as the embedding_function, query vectors are now in a 1536-dim OpenAI space while stored vectors are in a 384-dim MiniLM space. Chroma will either raise a dimension mismatch error or — if you somehow force it — return completely wrong nearest neighbors. Always set the embedding_function at collection creation and never change it.
from langchain_openai import OpenAIEmbeddings from langchain_chroma import Chroma # chunks = list[Document] from Module 1 embedding_fn = OpenAIEmbeddings(model="text-embedding-3-small") vectorstore = Chroma.from_documents( documents=chunks, # your ~800 Document objects embedding=embedding_fn, persist_directory="./chroma_seo", collection_name="seo_pages", ) print(vectorstore._collection.count()) # e.g. 800
One call — Chroma.from_documents() — embeds every chunk and writes the collection to ./chroma_seo in one shot.
The persist_directory argument is what makes the store survive restarts — without it, the collection lives only in memory and is lost when the process exits.
count() prints the number of documents you passed in (e.g. 800 if you had 800 chunks). In ./chroma_seo/ you'll find a chroma.sqlite3 file (the metadata + vector index) and one or more binary segment files. These are what Chroma reads on reload — no re-embedding needed.
from langchain_openai import OpenAIEmbeddings from langchain_chroma import Chroma embedding_fn = OpenAIEmbeddings(model="text-embedding-3-small") # TODO: reload the persisted collection — no from_documents() call here. # Hint: use the Chroma constructor directly with the same # persist_directory and collection_name you used in Stage 2. vectorstore = TODO print(vectorstore._collection.count()) # should still print 800
Stop — attempt the TODO before revealing the answer. You need to reload the store you persisted in Stage 2 without calling from_documents() again (that would re-embed and re-pay).
The crux: which Chroma constructor signature loads an existing collection from disk?
Changed line:
vectorstore = Chroma(
persist_directory="./chroma_seo",
embedding_function=embedding_fn,
collection_name="seo_pages",
)
Why the model must match: Chroma stores raw vectors, not the model name. If you reload with a different embedding model, query vectors live in a different space — cosine distances become meaningless and retrieval silently degrades. Always pin the model string (here 'text-embedding-3-small') in a config constant shared by both indexing and query code.
| Option | Retrieval quality | Dimensions / index size | When to choose | Cost | Complexity |
|---|---|---|---|---|---|
| text-embedding-3-small (OpenAI) | Strong out of the box; MTEB top-tier for its size | 1536 dims — moderate index size | Default for most RAG prototypes and production apps where managed cost and strong quality matter; no GPU needed. | Pay-per-token via OpenAI API (~$0.02/1M tokens) | Low — one API key, no infra |
| text-embedding-3-large (OpenAI) | Best OpenAI quality; meaningful gains on long-tail queries | 3072 dims — doubles storage and RAM vs 3-small | When retrieval quality is the top priority and you can absorb higher cost and larger index size. | ~5× more expensive than 3-small | Low — same API, larger vectors |
| BGE-small / all-MiniLM (open-source) | Good for focused domains; lags on diverse open-domain queries | 384–768 dims — smallest indexes | When you need zero API cost, offline operation, or full data-privacy control; acceptable quality for focused corpora. | Free — runs locally (CPU or GPU) | Medium — must manage model download and inference |
| Chroma (local vector store) | Exact cosine search; no ANN approximation at small scale | Scales to ~millions of vectors locally; no sharding | Prototypes, notebooks, and internal tools where minimal setup beats operational maturity. | Free / open-source | Low — pip install, no server needed |
| Qdrant (dedicated vector DB) | ANN with HNSW; excellent at scale with filtering | Handles billions of vectors with quantization options | Production systems needing payload filtering, snapshots, and a separate operational boundary from your app. | Self-hosted free; cloud pricing by node | Medium-high — Docker or cloud cluster |
text-embedding-3-small but reload with a different model. No exception — retrieval just returns wrong chunks. Fix: store the model name in a config constant and assert it matches on load.from_documents(). On reload, _collection.count() returns fewer docs than expected — or Chroma raises sqlite3.DatabaseError: database disk image is malformed. Fix: delete the directory and re-run from scratch.persist_directory in the original call, so nothing is written to disk. The next session creates a fresh empty collection and count() returns 0. Fix: always pass persist_directory and confirm the directory exists after the first run.OpenAIEmbeddings(model=...) is identical in both the indexing script and the reload script.from_documents(), assert vectorstore._collection.count() == len(chunks) — a mismatch means some chunks were dropped.vectorstore.similarity_search('test query', k=1) should return a Document whose page_content is recognisably relevant, not random text../chroma_seo/chroma.sqlite3 exists on disk before calling the reload path — if it's missing, the AI-generated code silently creates a new empty store.With a verified, persisted store in hand, Module 3 turns it into a with .as_retriever(search_kwargs={'k': 4}) and confirms it returns the right chunks for a real query — that's where the pipeline becomes queryable.
Turn the Chroma store from Module 2 into a retriever with as_retriever(search_kwargs={'k': 4}), then confirm it returns the right chunks for a test query before any LLM is involved. You'll complete a partially written retriever config that adds a score threshold filter.
Configure a VectorStoreRetriever from your Chroma store, tune k and score_threshold, and choose between similarity and MMR search.
Why this matters: The retriever is the bridge between your indexed corpus and the LLM — getting k and search type right determines whether your SEO page cites accurate, specific evidence or hallucinates from noise.
Decision this forces: How many chunks (k) should the retriever return, and should you use similarity or MMR search?
Answer: vectorstore.as_retriever(search_kwargs={'k': N}) you can test before any LLM is involved.
The driving question: how do you configure that retriever to return the right number of relevant chunks? Not too few to miss evidence. Not so many that noise crowds the context window.
objects out.
sets chunk count; score_threshold drops chunks below a cosine similarity cutoff (0–1). Together they trade recall for precision.
The retriever runs independently of the LLM. Call it, inspect chunks, confirm relevance before wiring into a chain. That isolation makes debugging fast.
(the model's working memory). So k directly controls how much budget you spend on evidence.
You're building an SEO/GEO explainer page whose H1 is the exact search query "langchain for rag". The page must cite specific numbers and defaults — vague prose won't rank or get lifted by an LLM.
from noise.
k=4 is a practical starting point: enough coverage to catch complementary evidence, small enough to keep the context tight. Add a score_threshold of 0.75 to drop low-confidence chunks — especially useful when your corpus mixes on-topic and off-topic docs.
Before the chain goes live, call the retriever directly on your target query and read the returned chunks. If any chunk is clearly off-topic, tighten the threshold; if you're missing key facts, lower it or raise k.
# Assumes: vectorstore = Chroma loaded from Module 2 retriever = vectorstore.as_retriever( search_type="similarity", search_kwargs={"k": 4} ) # Test it — no LLM involved yet docs = retriever.invoke("langchain for rag") for d in docs: print(d.metadata["source"], "|", d.page_content[:80])
as_retriever() in one call. The invoke() and content before trusting the retriever in a chain.
docs/langchain_intro.md | LangChain is a framework for building LLM-powered applications with composable
Each line shows the source file path and the first 80 characters of that chunk's content — confirming which document the retriever pulled from and what text it selected.
retriever = vectorstore.as_retriever( search_type="similarity_score_threshold", search_kwargs={ "k": 4, "score_threshold": 0.75 } ) docs = retriever.invoke("langchain for rag") print(f"{len(docs)} chunk(s) passed the threshold")
Switching search_type to "similarity_score_threshold" tells the retriever to drop any chunk scoring below score_threshold — so you may get fewer than k results when the corpus has no strong matches. The count printed here is your first signal that the threshold is calibrated correctly.
2 chunk(s) passed the threshold
You asked for k=4 but only 2 chunks cleared the bar — the threshold is filtering aggressively. If those 2 chunks contain all the evidence you need, that's fine. If you're missing key facts, lower score_threshold (e.g. to 0.65) or raise k.
retriever = vectorstore.as_retriever( search_type="mmr", search_kwargs={ "k": 4, # TODO: add the MMR diversity parameter here # fetch_k controls the candidate pool before re-ranking } ) docs = retriever.invoke("langchain for rag") for d in docs: print(d.metadata["source"])
This is a completion problem — the structure is given, but the key MMR parameter is missing. Stop and supply it before revealing the answer.
first fetches a larger pool of candidates, then re-ranks for diversity. (2) The parameter name starts with fetch_ and its value should be larger than k — a common default is 20.
"fetch_k": 20 # ← the changed line
Full search_kwargs: {"k": 4, "fetch_k": 20}
Why fetch_k > k: MMR retrieves fetch_k candidates by similarity first, then selects k of them by maximising relevance while penalising redundancy. If fetch_k == k there's nothing to re-rank — you'd get the same result as plain similarity search.
| Option | Handles redundant chunks | Latency | When to choose | Cost | Complexity |
|---|---|---|---|---|---|
| similarity | Returns the k nearest vectors — duplicates or near-duplicates all come back if they score high. | Single ANN search; fastest option. | Your corpus chunks are already diverse (e.g. one chunk per doc section) and you want the fastest, most predictable retrieval. | 1× embedding lookup | Low |
| mmr | Penalises chunks too similar to already-selected ones, so the k results cover more ground. | Slightly slower — fetches a larger candidate set then re-ranks for diversity. | Your corpus has many near-duplicate chunks (e.g. repeated boilerplate, overlapping sliding-window splits) and you need diverse evidence in the context. | 1× embedding lookup + re-ranking pass | Medium |
len(docs) == 0. Fix: lower the threshold or remove it while debugging.InvalidRequestError: maximum context length exceeded. Fix: keep k × avg_chunk_tokens under the model's limit.search_type matches intent — AI tools often default to "similarity" even when you asked for MMR.score_threshold is inside search_kwargs, not a top-level argument — misplaced keys are silently ignored.retriever.invoke("your target query") and assert len(docs) > 0 before wiring the retriever into any chain.Once your retriever returns the right chunks, shape how they reach the model. Module 4 covers writing a that constrains the LLM to answer only from retrieved context, slotting in {context} and {question} variables your chain will populate.
Write a ChatPromptTemplate with a system message that constrains the model to answer only from the provided context, then slot in the {context} and {question} variables the chain will fill at runtime. You'll fix a broken prompt template that causes the model to ignore retrieved passages.
How to write a ChatPromptTemplate system message that forces the LLM to answer only from retrieved chunks, not its training data.
Why this matters: A poorly phrased system prompt is the most common reason a RAG pipeline halluccinates — fixing it is the highest-leverage step before wiring the full chain.
as_retriever(search_kwargs={'k': 4}). What does the k=4 parameter control, and what Python type does the retriever return for each result?Answer: kDocument object with a page_content string and a metadata dict.
. The question this module answers: how do you write the prompt so the model actually uses those passages instead of ignoring them?
has two memory sources: retrieved passages and parametric knowledge from training.
that sound authoritative.
is the highest-authority slot — the model weights it above user turns. Placing your grounding rule there is the single most effective lever for anchoring to retrieved evidence.
How you phrase that rule determines whether the model cites context or overrides it.
These failure patterns appear in practice. Each has a distinct symptom telling you which line to fix.
"Use the context below to help answer". The word help lets the model supplement with training knowledge. Symptom: plausible answers citing facts not in retrieved chunks — quietly wrong, no error.{context} but the chain passes context_str. Symptom: KeyError: 'context' at runtime — the chain never reaches the LLM.You're building an SEO/GEO page generator. The retriever pulls four chunks from your corpus of ranking-factor guides and algorithm-update notes.
The user asks: "Does page speed affect Google's ranking in 2024?" Your corpus has a chunk confirming Core Web Vitals as a ranking signal.
Without a strict system message, the model might answer from outdated training data. With a well-anchored prompt, it quotes the retrieved chunk and flags if the corpus is silent.
The chain fills two variables at runtime: {context} (joined chunk texts) and {question} (user query). Everything else is fixed in the template.
from langchain.prompts import ChatPromptTemplate system_msg = "You are a helpful SEO assistant. Use the context to help answer." prompt = ChatPromptTemplate.from_messages([ ("system", system_msg), ("human", "Context: {ctx}\n\nQuestion: {question}"), ]) # chain passes: {"context": chunks_text, "question": user_q} chain_output = prompt.invoke({"context": chunks_text, "question": user_q})
This prompt has two bugs that let the model ignore retrieved evidence. Spot them before reading the fix.
Bug 1 — variable mismatch: the template uses {ctx} but the chain passes 'context', so invoke() raises KeyError: 'ctx' before the LLM is ever called.
Bug 2 — weak anchor: 'help answer' permits the model to blend training knowledge with the retrieved text, so even after fixing the key error, the model may silently ignore the chunks and answer from parametric memory with no error or warning.
from langchain.prompts import ChatPromptTemplate SYSTEM = """ You are an SEO research assistant. Answer ONLY from the context provided below. If the context does not contain the answer, say: "I don't have enough information in the provided sources." Do not use outside knowledge. """ prompt = ChatPromptTemplate.from_messages([ ("system", SYSTEM), ("human", "Context:\n{context}\n\nQuestion: {question}"), ])
Three changes from Stage 1 do the anchoring work: ONLY from the context removes the model's permission to supplement; the explicit fallback phrase handles the no-answer case; and {context} now matches the key the chain will pass.
The anchor weakens slightly but usually holds — 'ONLY from the context' is the load-bearing phrase. However, 'Do not use outside knowledge' acts as a reinforcing constraint that reduces edge-case drift (e.g. when the model is highly confident in its training data). Keeping both is the safer production pattern; removing one is a measurable regression on adversarial queries.
from langchain.prompts import ChatPromptTemplate SYSTEM = """ You are an SEO research assistant. # TODO: add the two-sentence anchor that (a) restricts the model # to the provided context only, and (b) gives it a fallback # phrase when the context is silent. """ prompt = ChatPromptTemplate.from_messages([ ("system", SYSTEM), ("human", "Context:\n{context}\n\nQuestion: {question}"), ]) print(prompt.messages) # inspect the rendered template
Stop — attempt the TODO before revealing the answer. The gap is the crux of this module: the two sentences that prevent hallucination.
Hints: (1) use the word ONLY to close the permission gap; (2) give the model an exact fallback string so it doesn't invent one.
Changed lines (replace the TODO block):
Answer ONLY from the context provided below.
If the context does not contain the answer, say:
"I don't have enough information in the provided sources."
Do not use outside knowledge.
Why these lines: 'ONLY' removes the implicit permission to supplement with training data. The explicit fallback string prevents the model from inventing a polite-sounding non-answer. 'Do not use outside knowledge' is a second-layer reinforcement — models respond to explicit negation as well as positive constraints.
Run these four checks before trusting an AI-generated system prompt in production.
{context} and {question} in the template must match the keys the chain passes.chain — connecting retriever, template, and ChatOpenAI into a single pipeline you invoke end-to-end with one call.
Compose the retriever, prompt template, and ChatOpenAI model into a single LCEL chain using the pipe operator (|), then invoke it end-to-end on a real question about your document corpus. You'll build the final chain yourself from the individual components assembled in Modules 1–4.
Compose a retriever, prompt template, and LLM into a single LCEL chain using the pipe operator, then invoke or stream it end-to-end.
Why this matters: This is the assembly step that turns all the individual RAG components you built into a working, callable pipeline for your SEO page.
Decision this forces: Should you use a prebuilt chain helper (create_retrieval_chain) or compose with LCEL directly — and when does each pay off?
Answer: the two variables were {context} (the retrieved chunks) and {question} (the user's query). The system message told the model to answer only from the provided context, guarding against .
Those two slots are exactly what the chain you're about to build will fill at runtime. This module wires the , the prompt, and the together so that filling them happens automatically.
lets you compose any two objects with the | operator. The output of the left side becomes the input of the right side automatically.
Every LangChain component — retrievers, prompt templates, chat models, output parsers — implements the interface. They all snap together with | without glue code.
The chain you'll build follows this shape: retriever | prompt | llm | output_parser. Each step transforms the data. Chunks become a formatted prompt. The prompt becomes a model response. The response becomes a plain string.
Adding StrOutputParser at the end strips the AIMessage wrapper. It returns a plain Python string. That is the format most downstream code expects.
You've loaded a corpus of product documentation PDFs. You chunked them in Module 1. You embedded them in Module 2. You built a returning the top = 4 chunks in Module 3. A user asks: "What is the return policy for defective items?"
Without LCEL you'd write three separate calls: retrieve chunks, format the prompt manually, then call the model. With LCEL you call chain.invoke({"question": "..."}) and the pipeline runs end-to-end in one line.
The retriever fetches the four most relevant chunks from the . The prompt template slots them into {context} and the question into {question}. The model generates an answer grounded in those chunks. StrOutputParser returns it as a plain string.
For a streaming UI — say, a live chat widget on your SEO page — you swap .invoke() for .stream(). Tokens arrive incrementally. That cuts perceived latency without changing the chain definition at all.
from langchain_core.runnables import RunnablePassthrough # Attempt: pipe retriever directly into prompt chain = retriever | prompt | llm result = chain.invoke({"question": "What is the return policy?"}) print(result)
This looks right but crashes at runtime because the retriever outputs a list of Document objects, not the {"context": ..., "question": ...} dict the prompt template expects.
The prompt template receives the raw list and raises a KeyError: 'context' — it never got a string to slot in.
It fails at the prompt step with KeyError: 'context'. The retriever hands a list of Document objects to the prompt template, but the template expects a dict with keys 'context' and 'question'. You need a formatting step between the retriever and the prompt to convert the document list into a plain string and pair it with the question.
from langchain_core.runnables import RunnablePassthrough from langchain_core.output_parsers import StrOutputParser def format_docs(docs): return "\n\n".join(d.page_content for d in docs) chain = ( {"context": retriever | format_docs, "question": RunnablePassthrough()} | prompt | llm | StrOutputParser() ) print(chain.invoke("What is the return policy for defective items?"))
The dict literal {"context": retriever | format_docs, "question": RunnablePassthrough()} is itself a Runnable: it routes the input string to both branches in parallel, producing the dict the prompt needs.
RunnablePassthrough() passes the raw question string through unchanged; format_docs joins the retrieved into a single context string. StrOutputParser unwraps the AIMessage to a plain string.
A plain string. StrOutputParser() is the last step; it extracts the .content attribute from the AIMessage the LLM returns, so the chain's final output is a regular Python str, not a message object.
# chain is already defined above — same object, different call # TODO: replace the blank so tokens print as they arrive for token in chain.____________("What is the return policy for defective items?"): print(token, end="", flush=True) # Hint 1: the method name is one word and rhymes with "dream". # Hint 2: the chain definition does NOT change — only the call site does.
Stop — attempt this before revealing. Fill in the blank to make the chain stream tokens incrementally instead of waiting for the full response.
Every LCEL chain supports .stream() out of the box — no chain restructuring needed, which is one of LCEL's key advantages over manual orchestration.
The blank is 'stream'. Changed line: for token in chain.stream("What is the return policy for defective items?"):. The rest of the snippet is identical. .stream() returns a generator that yields string chunks as the LLM produces them; .invoke() would block until the full response is ready.
| Option | Customisability | Streaming support | Code transparency | When to choose | Cost | Complexity |
|---|---|---|---|---|---|---|
| LCEL direct (| operator) | Full control — insert any Runnable, reorder steps, add branches | Native — .stream() works on any LCEL chain without changes | Every step is explicit in the chain expression | When you need custom formatting, multi-step logic, streaming, or want full visibility into every transformation in the chain. | No extra abstraction overhead; latency is identical | Medium — you wire the dict routing and format_docs yourself |
| create_retrieval_chain | Limited — fixed input/output keys; harder to inject custom steps | Partial — streams the final answer but not intermediate steps | Opaque — internals are hidden inside the helper | When you want a working RAG chain in 2 lines for a prototype and don't need custom formatting, streaming, or intermediate step access. | Slightly more opaque; harder to customise or debug | Low — one helper call wraps retriever + combine_docs_chain |
{"context": retriever | format_docs, "question": RunnablePassthrough()} dict routing layer..invoke() internally), streaming breaks silently. The generator blocks until the full response is ready. Verify by printing each token in a loop. Check that output appears before the full response completes.format_docs (or equivalent) joins on "\n\n". A bare list join loses chunk boundaries and confuses the model.RunnablePassthrough() is present for the question key. Generated code often omits it and passes None instead.chain.invoke("test") and assert the return type is str, not AIMessage. If it's a message object, StrOutputParser is missing.With a working, verified chain in place, the next module — Failure Modes, Verification, and Next Steps — shows you how to diagnose wrong chunks, context overflow, and hallucinated answers using LangSmith traces and a groundedness check.
Diagnose the three most common RAG failures — wrong chunks retrieved, context overflow, and hallucinated answers — using LangSmith traces and a simple groundedness check. You'll also revisit the chunk_size decision from Module 1 to see how it cascades into retrieval quality.
Diagnose and fix the three most common RAG failures — wrong chunks, context overflow, and hallucinated answers — using LangSmith traces and a groundedness heuristic.
Why this matters: A RAG pipeline that returns wrong answers without a clear diagnosis path is unusable in production; this module gives you the triage and verification skills to ship with confidence.
Decision this forces: When a RAG answer is wrong, is the root cause retrieval quality, prompt design, or LLM behaviour — and how do you tell?
The answer: → → . The retriever fetches evidence, the template formats it into a grounded prompt, and the LLM generates the answer. That pipeline is now running — and this module is about what to do when it gives a wrong answer.
When a answer is wrong, the root cause is almost always one of three things — and the fix is different for each.
Each failure lives at a different stage of the pipeline, so you need to isolate the stage before you can fix it.
A LangSmith trace records every step of your chain.
It captures retrieval inputs and outputs, the assembled prompt, and the model's response.
This lets you pinpoint the failure stage without guessing.
Imagine a user asks: "What is the refund window for enterprise customers?"
The chain returns: "Enterprise customers have a 14-day refund window."
Your policy doc says 30 days.
In the refund example: the retriever returned four chunks about "standard refund policy."
None mentioned enterprise. Root cause is retrieval, not generation.
def is_grounded(answer: str, docs: list) -> bool: context = " ".join(d.page_content for d in docs) key_phrases = [s.strip() for s in answer.split(",") if len(s.strip()) > 6] hits = sum(1 for p in key_phrases if p.lower() in context.lower()) return hits / max(len(key_phrases), 1) >= 0.5 # Usage after invoking your LCEL chain: # result = chain.invoke({"question": query}) # grounded = is_grounded(result["answer"], result["source_documents"])
This heuristic checks whether at least half of the answer's key phrases appear verbatim in the retrieved — a fast, zero-cost signal that the model stayed grounded.
It won't catch paraphrased , but it catches the most common case: the model inventing facts that share no words with the context.
False. key_phrases splits on commas to get ["30-day window", "enterprise tier", "full refund"]. None appear in context, so hits = 0, and 0 / 3 = 0.0 < 0.5 — the function returns False, flagging a likely hallucination.
The chunk_size you set in Module 1 determines how many tokens each occupies in the . With k=4 and chunk_size=500, you're already committing ~2,000 tokens before the question and system prompt are added.
If you see a context-overflow error, the fastest fix is to reduce k (fewer chunks) or chunk_size (smaller chunks). Reducing k risks missing the answer; reducing chunk_size risks splitting a key sentence across two chunks.
A practical starting point: keep chunk_size × k under 60% of your model's context window, leaving room for the prompt and the generated answer.
from langchain_core.runnables import RunnablePassthrough def run_and_verify(chain, query: str, docs_key="source_documents"): result = chain.invoke({"question": query}) docs = result.get(docs_key, []) answer = result.get("answer", "") # TODO: call is_grounded(answer, docs) and print a warning if False # Hint 1: is_grounded is defined in the previous block. # Hint 2: the warning should name the query so you know which one failed. return result
Stop — attempt the TODO before revealing. The missing lines are the crux: you must call the groundedness heuristic and surface a warning when it fails, so bad answers don't silently reach users.
This is a small variation of the earlier example: the chain invocation is already done; your job is to add the verification layer that a production pipeline needs.
CHANGED LINES (the crux — everything else stays the same):
if not is_grounded(answer, docs):
print(f"[GROUNDEDNESS WARNING] Possible hallucination for query: '{query}'")
Why: is_grounded returns False when fewer than half the answer's key phrases appear in the retrieved docs. Printing the query lets you correlate warnings with LangSmith traces to decide whether the root cause is retrieval or generation.
| Option | Root cause it fixes | Implementation effort | When to choose | Cost | Complexity |
|---|---|---|---|---|---|
| Reduce k or chunk_size | Context overflow | One-line config change | When the trace shows a token-limit error or a truncated prompt (context overflow). | None | Low — one parameter change |
| Reranking | Wrong-chunk ranking | Add reranker to chain | When the right chunk is retrieved but ranked below the top-k cutoff — the answer exists in the index but doesn't surface. | Extra inference call per query | Medium — add a cross-encoder reranker step |
| Hybrid search | Semantic miss on exact terms | Dual index setup | When the query uses exact product names or codes that semantic search misses — BM25 + vector covers both. | Dual index storage | Medium — requires a BM25 index alongside the vector store |
| Query rewriting | Vocabulary mismatch | One extra prompt step | When the user's question is ambiguous or uses different vocabulary than the indexed documents. | Extra LLM call per query | Low-medium — one extra LLM call to rephrase the query |
| Tighten system prompt | Hallucination (generation) | Prompt edit only | When the trace shows the right chunks were retrieved but the LLM answered from memory — a stricter "answer only from context" instruction reduces hallucination. | None | Low — prompt edit only |
When an AI tool generates a retrieval fix or groundedness check, verify four things before trusting it:
You've now diagnosed, fixed, and verified a full pipeline.
The capstone challenge asks you to build the complete system from scratch.
Load, chunk, embed, retrieve, prompt, and verify — without scaffolding.
Everything in this lesson is fair game.
Before scrolling down, sketch the six-step pipeline from memory: what happens to a raw PDF before the LLM ever sees it, which LangChain object connects the vector store to the chain, and what the | operator is actually chaining together. Then check your sketch against the buildOrder below.
Apply what you learned to SEO/GEO page — the page title is the H1 and the exact search query. OPEN with a 40-60 word self-contained, quotable answer capsule that names the entity explicitly in sentence one (no "in this lesson…" preamble) — this is the sentence an LLM lifts. Name the entity by its full name every time (not pronoun-only). Structure each section as ONE sub-question (How it works · X vs Y · When to use it · Example · Common mistakes), each opening with its own 1-2 sentence direct answer then depth. End with a genuine 3-5 question FAQ phrased as real user questions ending in "?" (these become FAQPage schema + the knowledge check). Prefer quotable specifics — concrete numbers, defaults, version names, one short snippet — over vague prose. Self-contained for a reader who arrived cold from a search engine or an LLM. Intent = explainer: mechanism + example + when-to-use. Target query: "langchain for rag"..
You are chunking a collection of long legal contracts. Queries will ask about specific clauses that often span 3–4 sentences. Which chunk_size / chunk_overlap pair is the safest starting point?
A. chunk_size=200, chunk_overlap=0
B. chunk_size=512, chunk_overlap=50
C. chunk_size=1024, chunk_overlap=200
D. chunk_size=4096, chunk_overlap=0
Legal clauses spanning 3–4 sentences need a larger window so the full clause lands in one chunk — 1024 tokens covers that comfortably. An overlap of 200 prevents a clause from being split across a boundary with no shared context. 200-token chunks are too small and will cut clauses mid-sentence. 512/50 is a reasonable general default but undershoots multi-sentence legal language. 4096 with zero overlap risks losing boundary context entirely and inflates retrieval cost.
Without looking anything up: what is the minimum code change that lets you reload a persisted Chroma collection without re-embedding all your documents? Describe what you pass to Chroma and why it is enough.
Chroma serialises vectors to the persist_directory on first creation. On reload, supplying the same path and embedding function is sufficient — Chroma reads from disk rather than calling the embedding model. Omitting the embedding function causes query-time errors because Chroma cannot embed the query string for similarity search. Calling add_documents again would duplicate every document in the collection.
Consider this LCEL chain:
chain = retriever | prompt | llm | StrOutputParser()
result = chain.stream("What is indemnification?")
What does iterating over result give you?
Calling .stream() on an LCEL chain returns a generator that yields each token string as the LLM produces it, enabling progressive display. .invoke() (not .stream()) blocks until the full answer is ready and returns a single string. The chain does not surface raw Document objects to the caller — the retriever's output is consumed internally by the prompt step. LangSmith traces are a separate observability layer, not the return value of .stream().
When would you choose MMR (max-marginal relevance) retrieval over plain similarity search in a VectorStoreRetriever?
MMR balances relevance and diversity: it penalises chunks that are too similar to already-selected chunks, so the final k results cover more distinct facets of the answer. Plain similarity search is fine when your corpus has naturally varied chunks and redundancy is not a problem. MMR does not reduce embedding cost — every candidate chunk is still embedded; the difference is in the selection algorithm. Fitting the corpus in the context window is a reason to skip retrieval entirely, not a reason to prefer MMR.
A RAG chain returns a confidently wrong answer. You open the LangSmith trace and see that the retrieved Documents are all highly relevant and contain the correct information. Where is the most likely root cause?
If the LangSmith trace confirms the retrieved Documents contain the correct answer, retrieval is not the problem. The failure is in generation: either the prompt does not anchor the model firmly to the context (allowing parametric memory to override it), or the LLM is ignoring the instruction. The fix is prompt redesign — add an explicit instruction such as 'Answer only from the provided context' — or switch to a model that follows instructions more reliably. High chunk_overlap and embedding miscalibration are retrieval-side issues, ruled out when the trace shows correct chunks were returned.