LangGraph, CrewAI, AutoGen, LangChain, LlamaIndex, and the OpenAI Agents SDK each solve a different slice of the agentic problem — picking the right…
Trace the reason–act–observe loop, tool abstraction, and state object that all six frameworks implement differently. You'll use a minimal 'research assistant' scenario — one agent, one search tool, one memory dict — as the running example throughout the lesson.
The reason–act–observe loop, state object, and tool abstraction that every open-source agent framework shares.
Why this matters: Understanding this loop lets you read and debug any framework's agent code — LangGraph, CrewAI, smolagents, and the rest all implement these same three parts.
Your research assistant returned a stale Wikipedia summary. Why? It answered in one shot—no loop, no tool call, no check. Every capable agent runs a reason–act–observe cycle until done.
Each turn: the model reads the (goal + history), picks an action (often a call), and reads the result. That observation updates state. The cycle repeats.
Three parts make this work: the state object (what the agent remembers), the tool abstraction (what it can do), and the loop controller (when to stop). All six frameworks wire these three parts differently. But all have all three.
The state object is the agent's working memory — a dict, a typed class, or a message list that carries everything the model needs across turns. Its shape matters because the model can only reason about what's in it.
A is a Python function with a name, a description, and a typed signature. The model reads the description to decide whether to call it; the framework handles the actual execution and returns the result as an observation.
The loop controller checks after each observation whether the model's output is a tool call or a final answer. If it's a tool call, the loop continues; if it's a final answer, the loop exits. This is the logic every framework encodes differently.
A user asks: "What are the top open-source AI agent frameworks in 2025?" The research assistant runs the like this:
search tool with the query "open source AI agent frameworks 2025".The state dict after step 3 looks like: {"goal": "...", "search_results": [...], "answer": None}. After step 5: {"goal": "...", "search_results": [...], "answer": "LangGraph, CrewAI, smolagents..."}. The shape of the state dict is what the model reads — missing a key means the model can't use that information.
def run_agent_naive(goal: str) -> str: state = {"goal": goal} while True: response = model.chat(state["goal"]) # no history passed if is_tool_call(response): result = search(response.query) # BUG: result is never stored in state else: return response.text
This loop looks right but has two silent bugs that cause wrong answers in production.
Bug 1 — the search result is never written into state, so the model never sees it; it reasons from the goal alone every turn and loops forever or halts on the first non-tool response regardless of quality. Bug 2 — only the raw goal string is passed to model.chat(), so conversation history is lost between turns; the model has no memory of what it already tried.
def run_agent(goal: str, max_turns: int = 5) -> str: state = {"goal": goal, "history": [], "answer": None} for _ in range(max_turns): response = model.chat(state["goal"], history=state["history"]) if is_tool_call(response): result = search(response.query) # Act state["history"].append({"tool": "search", "result": result}) # Observe else: state["answer"] = response.text # final answer break return state["answer"] or "max turns reached"
Each observation is now written into state["history"] and passed back to the model on the next turn. The max_turns guard prevents infinite loops — a critical safety rail.
[{"tool": "search", "result": <first result>}, {"tool": "search", "result": <second result>}] — both observations are visible, so the model can synthesize across them rather than repeating the same search.
def search(query: str) -> str: """Search the web and return a summary. Use when you need current facts.""" return web_api.get(query) # real implementation swapped in here TOOLS = [search] # register: name + docstring become the model's menu # Completion task — add a second tool below: # def summarize(text: str) -> str: # """TODO: one-line description the model will read to decide when to call this""" # pass
The docstring is the model's only signal for when to call this tool — it must say what the tool does AND when to use it. The TOOLS list is the extension point: swap in a database lookup, a calculator, or a file reader without touching the loop.
def summarize(text: str) -> str:
"""Condense a long text into key points. Use after search returns more than needed."""
return summarizer_api(text)
The crux is the docstring — without 'Use after search returns more than needed', the model has no signal to distinguish this tool from search and may call the wrong one or ignore it entirely.
Three failure patterns appear in almost every first implementation:
"results" but the prompt template reads "search_results". The model sees an empty field and hallucinates. Observable symptom: confident answers that contradict the tool's actual output. Fix: validate state keys at the loop entry point.max_turns or equivalent guard exists and returns a non-None fallback.The plain-Python loop you built is real—it runs. But it has no persistence, no branching, and no pause-and-resume. That's the gap the six frameworks fill, each with a different design.
The core loop stays the same: reason → act → observe → update state → check stop. What differs is state storage, tool registration, and how the loop handles branching and recovery.
When reading any framework's code, look for these three anchors: state shape, tool registry, loop exit condition. The rest of the design will follow immediately.
while loop with an explicit —a graph of nodes and edges. The assistant can pause, save a , and resume exactly where it left off.Build the research-assistant scenario as a LangGraph StateGraph: define nodes, edges, and a MemorySaver checkpoint so the flow can pause and resume. You'll see exactly where LangGraph's determinism wins and where its verbosity costs you.
Build a LangGraph StateGraph with typed state, conditional edges, and a MemorySaver checkpoint so a research-assistant flow can pause and resume.
Why this matters: LangGraph gives you the deterministic control and durable checkpointing that a plain agent loop can't provide — essential when your SEO/GEO pipeline needs human approval steps or crash recovery.
Decision this forces: Choose LangGraph when deterministic transitions, human-in-the-loop pauses, or durable checkpointing are non-negotiable.
Every framework from module 1 runs the same reason–act–observe cycle. Each turn reads from and writes back to a shared object. LangGraph makes that state object explicit and typed. It makes every transition a named edge you can inspect — that's the upgrade.
The driving question: when your research assistant needs to pause mid-run, wait for human approval, and resume exactly where it stopped — how do you build that without re-running everything?
A is LangGraph's core primitive: a directed graph where each node is a Python function that reads the current state and returns a partial update, and each edge is an explicit route to the next node.
State is a typed dict (or Pydantic model) shared across all nodes. Every node sees the same keys and writes only what it changes. Edges are either direct (always go to node B after node A) or (a router function inspects state and returns the next node's name).
You compile the graph before running it. Compilation validates that every edge target exists and the state schema is consistent. A compiled graph is deterministic: given the same state, the same path fires every time.
Your research assistant needs three steps: fetch search results, decide if they're good enough, and either draft an answer or loop back for a better query.
In LangGraph you model this as three nodes — search, evaluate, draft — connected by a conditional edge on evaluate. The router checks state["quality"]: if "ok" it routes to draft; if "retry" it loops back to search.
The state dict carries query, results, quality, and draft. Every node reads what it needs and writes only its own keys. The graph wires it all — no function arguments needed.
from typing import TypedDict, Literal class ResearchState(TypedDict): query: str; results: list[str]; quality: Literal["ok", "retry", ""]; draft: str def search_node(state: ResearchState) -> dict: hits = run_search(state["query"]) # your search tool return {"results": hits, "quality": ""} def evaluate_node(state: ResearchState) -> dict: q = "ok" if len(state["results"]) >= 3 else "retry" return {"quality": q} # e.g. 1 result → "retry" def draft_node(state: ResearchState) -> dict: return {"draft": llm_draft(state["results"])}
Each node is a plain function: it receives the full state dict and returns only the keys it changes — LangGraph merges the partial update back into shared state.
The Literal type on quality is intentional: it makes the conditional router's valid return values explicit and catches typos at compile time.
{"quality": "retry"} — because len([one_result]) < 3. The graph will then route back to search_node for another attempt.
# Continuing from Stage 1 — add graph wiring + MemorySaver from langgraph.graph import StateGraph, START, END from langgraph.checkpoint.memory import MemorySaver builder = StateGraph(ResearchState) builder.add_node("search", search_node) builder.add_node("evaluate", evaluate_node) builder.add_node("draft", draft_node) builder.add_edge(START, "search") builder.add_edge("search", "evaluate") builder.add_conditional_edges( "evaluate", lambda s: s["quality"], # router: returns "ok" or "retry" {"ok": "draft", "retry": "search"} ) builder.add_edge("draft", END) graph = builder.compile(checkpointer=MemorySaver())
The MemorySaver checkpointer (glossary: ) snapshots the full state after every node completes — keyed by a thread ID you supply at invoke time. If the process crashes or pauses for human approval, you resume by passing the same thread ID; LangGraph replays from the last saved node, not from the start.
The conditional edge's third argument is a routing map: the router returns a string key, and the map resolves it to a node name. This keeps routing logic separate from node logic.
LangGraph reloads the checkpoint saved after evaluate_node. The state already has quality="ok", so the graph routes directly to draft_node — it does NOT re-run search_node or evaluate_node.
# Same graph — add a human-approval node before draft def approve_node(state: ResearchState) -> dict: # Pauses here; a human reviews state["results"] out-of-band approved = get_human_approval(state["results"]) quality = "ok" if approved else "retry" return {"quality": quality} builder.add_node("approve", approve_node) # TODO: rewire the conditional edge so "ok" now routes to # "approve" instead of "draft", and "approve" routes to "draft". # Hint: you need one add_conditional_edges call and one add_edge call.
Stop — attempt the two missing wiring calls before revealing the answer. The checkpoint means the graph can sit idle at approve_node indefinitely while a human reviews, then resume when they respond.
Changed lines:
builder.add_conditional_edges(
"evaluate",
lambda s: s["quality"],
{"ok": "approve", "retry": "search"} # "ok" now → approve, not draft
)
builder.add_edge("approve", "draft") # approve always proceeds to draft
Why: the conditional map is the only place routing lives, so changing one dict value re-routes the whole flow. The direct edge from approve to draft reflects that once a human approves (or rejects back to retry), the next step is always deterministic.
KeyError at runtime. This happens when a node's output grows a new quality value you forgot to add to the map.Extend the research-assistant scenario to three agents (researcher, analyst, writer) and implement it twice: once as a CrewAI Crew with role prompts and task handoffs, once as an AutoGen group chat with code-execution. You'll surface the readability vs. flexibility tradeoff and revisit the state concept from Module 1.
Compares CrewAI's declarative role-and-task pipeline with AutoGen's conversational group-chat model, building the research-assistant scenario in both frameworks.
Why this matters: Choosing the right multi-agent framework directly affects how readable, debuggable, and cost-efficient your SEO content pipeline will be.
Decision this forces: Choose CrewAI when role clarity and readable task pipelines matter most; choose AutoGen when agents need to debate, write code, and self-correct in a shared conversation.
Answer: passes a typed dict through every node; a serialises that dict to disk so the graph can resume exactly where it paused.
This module introduces two frameworks — CrewAI and AutoGen — that handle state very differently. CrewAI stores task outputs in a structured result object; AutoGen's state IS the conversation history itself. Keeping that contrast in mind will make the tradeoffs concrete.
Your research-assistant scenario now needs three agents — researcher, analyst, writer — and you have two very different ways to wire them together.
CrewAI organises agents around and explicit tasks. You declare who does what and in what order; the framework enforces the pipeline. The object owns the — you don't write a loop.
AutoGen organises agents around a shared . Agents take turns sending messages; any agent can respond, push back, or write and run code. The conversation continues until a termination condition fires.
The core tradeoff: CrewAI is readable and predictable; AutoGen is flexible and self-correcting. Neither is universally better — the right choice depends on whether your task needs a fixed pipeline or an open debate.
Imagine building an SEO page about "open source AI agents". You define three agents with distinct and chain them with tasks.
Each task's output feeds the next task's context automatically — that is what makes the pipeline sequential. You never write a for-loop; the Crew drives execution.
The result object at the end holds each task's output keyed by task name, so you can inspect intermediate work without re-running the whole crew.
# Stage 1 — define agents with role prompts researcher = Agent( role="Senior Web Researcher", goal="Find authoritative facts about {topic}", backstory="You specialise in primary sources and cite every claim.", ) writer = Agent( role="SEO Content Writer", goal="Turn the outline into a 600-word draft for '{topic}'", backstory="You write clear, quotable prose optimised for search.", )
Stage 1 shows the agent definition pattern: role names the persona, goal is the system-level objective, and backstory shapes the model's behaviour without extra prompt engineering.
Tasks run in the order they appear in the tasks list. The writer receives the analyst's output string injected into its task context automatically — you don't pass it manually. The Crew object handles the handoff.
# Stage 2 — tasks + crew assembly (analyst agent omitted for brevity) research_task = Task( description="Search for facts about {topic}. Return a bullet list.", agent=researcher, ) write_task = Task( description="Write a 600-word SEO draft for '{topic}' using the research.", agent=writer, context=[research_task], # <-- explicit handoff ) crew = Crew( agents=[researcher, writer], tasks=[research_task, write_task], process=Process.sequential, ) result = crew.kickoff(inputs={"topic": "open source AI agents"})
Stage 2 wires the tasks into a with Process.sequential. The context=[research_task] line is the key: it tells CrewAI to inject the researcher's output into the writer's prompt automatically.
# AutoGen pseudocode — three agents share one GroupChat researcher = ConversableAgent("Researcher", system_message="Find facts.") analyst = ConversableAgent("Analyst", system_message="Score relevance.") writer = ConversableAgent("Writer", system_message="Draft the page.") chat = GroupChat( agents=[researcher, analyst, writer], max_round=12, # termination guard speaker_selection_method="auto", # LLM picks who speaks next ) manager = GroupChatManager(groupchat=chat) researcher.initiate_chat(manager, message="Research 'open source AI agents'.")
The object holds all agents and the full message history — that history IS the shared . max_round is your primary safety valve against runaway loops; without it the conversation can spin indefinitely.
Pass speaker_selection_method="round_robin". The GroupChatManager stops querying the LLM to pick the next speaker and instead cycles through the agents list in order. Changed line: speaker_selection_method="round_robin". This trades flexibility for predictability — useful when you want CrewAI-style ordering inside AutoGen.
A that is too vague lets the researcher start writing prose instead of returning facts. The symptom: the writer's task receives a polished paragraph instead of bullet points, so the final draft is redundant and over-long. The fix: make the task description specify the exact output format ("Return a numbered list of ≤10 facts").
A subtler trap: adding agents for social reasons rather than functional ones. Each extra agent multiplies cost and latency without improving quality.
Without a tight max_round limit, agents can keep refining indefinitely — you'll see the token counter climb with no new information added. Observable sign: the last five messages are paraphrases of each other.
The opposite failure is silent agreement: with speaker_selection_method="auto", the LLM sometimes picks the same agent repeatedly, so the analyst never speaks. Check chat.messages after a run and count how many times each agent name appears.
agent= assignment — generated code often leaves it implicit. (2) Check that max_round is set in every GroupChat. (3) Confirm context= links are present on downstream tasks, not just assumed. (4) Run a single-round smoke test and inspect intermediate outputs before a full run.| Option | Role clarity | Self-correction / debate | Code execution | When to choose | Cost | Complexity |
|---|---|---|---|---|---|---|
| CrewAI | Explicit role + backstory per agent | Not supported in sequential mode | Possible via tools, not native | Your task has a clear pipeline (research → analyse → write) and you want readable, auditable role definitions. Prefer it when stakeholders need to understand the agent structure at a glance. | Moderate — one LLM call per task; cost scales linearly with task count. | Low — declarative config, no loop logic to write. |
| AutoGen GroupChat | System messages only; roles can blur | Native — agents reply, critique, revise | Built-in via UserProxyAgent executor | Agents need to critique each other, write and run code, or the right answer requires back-and-forth. Prefer it when the task is open-ended and self-correction matters more than a clean pipeline. | Higher — conversation rounds multiply LLM calls; set max_round carefully. | Medium — must design termination conditions and speaker-selection strategy. |
Add a private document store to the research-assistant scenario: implement it with LangChain's create_react_agent (broad tool integrations) and then with a LlamaIndex QueryEngine-as-tool (retrieval-first). You'll compare indexing control, source-connector richness, and how each framework's agent middleware handles retrieval failures.
Compares LangChain and LlamaIndex for adding a private document store to an agent, covering integration breadth versus retrieval depth.
Why this matters: Choosing the right retrieval framework directly affects the answer quality and debuggability of your research-assistant agent.
Decision this forces: Choose LangChain when you need broad model/tool portability; choose LlamaIndex when retrieval quality, parsing, and indexing choices dominate the product.
Answer: CrewAI uses a of role-prompted agents. Each agent's output becomes the next agent's input through explicit task handoffs. There is no shared message bus. It is structured task chaining.
That model works well when roles are stable and the workflow is mostly linear. But what happens when your research assistant must answer questions from a private document store? The store changes daily and must be searched, not scripted. That's the gap this module closes.
LangChain's strength is integration breadth: one consistent interface across 50+ model providers. It also covers hundreds of tool connectors. Its retriever abstraction wraps any vector store.
LlamaIndex's strength is retrieval depth: fine-grained control over chunking, parsing, and indexing strategies. It also provides a that synthesizes an answer from retrieved nodes before the agent ever sees it.
returns a synthesized answer with citations. The agent gets a cleaner signal.
The decision is not about quality in the abstract. It is about where your product's complexity lives: in the tool ecosystem, or in the retrieval pipeline.
# Stage 1 — wire a retriever as a LangChain tool docs = load_documents("./research_docs") # your doc loader chunks = split(docs, chunk_size=512, overlap=64) vector_store = embed_and_store(chunks) # any vector store retriever = vector_store.as_retriever(k=4) retriever_tool = create_retriever_tool( retriever, name="doc_search", description="Search internal research documents for facts.", ) agent = create_react_agent(llm, tools=[retriever_tool, web_search_tool])
This wires a vector-store retriever into create_react_agent alongside an existing web-search tool — one config line swaps the LLM.
The retriever returns raw chunks (k=4 passages) directly into the agent's context. The agent prompt must synthesize them — there's no intermediate answer step.
Raw text passages — up to 4 chunks from the vector store, concatenated into the agent's observation. The agent's own reasoning step must extract and synthesize the relevant facts. If the chunks are too large or off-topic, the agent's answer degrades silently.
# Stage 2 — same docs, LlamaIndex retrieval-first approach from llama_index.core import VectorStoreIndex, SimpleDirectoryReader from llama_index.core.tools import QueryEngineTool documents = SimpleDirectoryReader("./research_docs").load_data() index = VectorStoreIndex.from_documents( documents, chunk_size=512, chunk_overlap=64 ) query_engine = index.as_query_engine(similarity_top_k=4) doc_tool = QueryEngineTool.from_defaults( query_engine, name="doc_search", description="Search internal research documents for facts.", ) agent = ReActAgent.from_tools([doc_tool, web_search_tool], llm=llm)
The VectorStoreIndex builds the node pipeline; as_query_engine() wraps retrieval + synthesis into one call — the agent receives a complete answer string, not raw chunks.
Swapping chunk_size or adding a reranker happens inside the index, invisible to the agent — that's the retrieval-depth advantage.
Stage 1 (LangChain): the agent receives 4 raw text chunks — it must synthesize them itself in its reasoning step. Stage 2 (LlamaIndex): the agent receives one synthesized answer string produced by the QueryEngine's response synthesizer — the retrieval and synthesis happened before the agent saw anything. This means retrieval failures in Stage 2 are easier to isolate (wrong answer from the tool vs. wrong reasoning by the agent).
| Option | Retrieval pipeline control (chunk size, reranking, parsing) | Model & tool connector breadth | Agent middleware maturity | When to choose | Cost | Complexity |
|---|---|---|---|---|---|---|
| LangChain create_react_agent | Retriever abstraction; limited built-in reranking | 50+ model providers, hundreds of integrations | Stable create_react_agent harness with callbacks | Your agent needs many heterogeneous tools (search, SQL, APIs) and you want to swap models with one config change. | Minimal overhead; retriever is a thin wrapper | Low–medium; wiring is declarative |
| LlamaIndex QueryEngineTool | Full node pipeline: chunk size, metadata, rerankers | Fewer native integrations than LangChain | ReActAgent wraps QueryEngineTool cleanly | Retrieval quality, source parsing, and indexing choices dominate — e.g. PDFs, mixed formats, or chunk-size sensitivity. | Higher at index-build time; pays off in answer quality | Medium; index build and node pipeline need tuning |
The agent answers confidently from its weights, ignoring the tool entirely. You'll see no tool call in the trace and no source citations. Cause: the tool description doesn't match the query's phrasing, so the model skips it. Fix: rewrite the description to match real user queries; add a few example queries.
The QueryEngine returns an answer like "The document does not contain information about X." even though the fact is present. Cause: chunk_size is too small (e.g. 128 tokens) and the relevant sentence spans a chunk boundary. Fix: raise chunk_size to 512 and add chunk_overlap=64.
New documents are added to the folder but the agent keeps returning old answers. The vector store was built once and never refreshed — no error is raised; the retriever just queries the stale index silently. Fix: rebuild or incrementally update the index on a schedule, or check document count at startup.
# Running example extended — add a second index for competitor docs competitor_docs = SimpleDirectoryReader("./competitor_docs").load_data() competitor_index = VectorStoreIndex.from_documents( competitor_docs, chunk_size=512, chunk_overlap=64 ) # TODO: create a QueryEngineTool from competitor_index # name it "competitor_search" # description: "Search competitor product documents." competitor_tool = ??? agent = ReActAgent.from_tools( [doc_tool, competitor_tool, web_search_tool], llm=llm )
Stop — attempt the TODO before revealing. The pattern mirrors Stage 2 exactly, applied to a new index.
Hints: (1) competitor_index.as_query_engine(similarity_top_k=4) gives you the engine. (2) QueryEngineTool.from_defaults(...) takes the engine, name, and description — same call as Stage 2.
Changed lines:
competitor_tool = QueryEngineTool.from_defaults(
competitor_index.as_query_engine(similarity_top_k=4),
name="competitor_search",
description="Search competitor product documents.",
)
Why the description matters: the ReActAgent reads tool descriptions to decide WHICH tool to call. A vague description (e.g. 'search documents') causes the agent to pick the wrong tool or skip both. Make it specific to the corpus so the model can route correctly.
len(index.docstore.docs) before running the agent. A count of 0 means the loader silently found nothing.query_engine.query("your test question") directly. Inspect the response before attaching it to the agent.Next up: Module 5 rebuilds this same research assistant with the OpenAI Agents SDK. You'll define a native Agent object, attach handoffs to specialist sub-agents, and add an input guardrail. You trade framework portability for tight provider-native primitives.
Rebuild the research assistant with the OpenAI Agents SDK: define an Agent, attach handoffs to specialist sub-agents, add an input guardrail, and read the built-in trace. You'll weigh the productivity gains against provider lock-in and revisit the checkpointing concept from Module 2 to contrast it with SDK sessions.
Build a research assistant with the OpenAI Agents SDK: define agents, wire handoffs to specialists, attach input guardrails, and read the built-in trace.
Why this matters: The SDK's provider-native primitives cut boilerplate for OpenAI-committed stacks — understanding their tradeoffs directly shapes your framework choice for the SEO/GEO page agent.
Decision this forces: Choose the OpenAI Agents SDK when you're already committed to OpenAI models and want built-in tracing, guardrails, and handoffs without extra wiring.
A LangGraph checkpoint serialises the full snapshot — every node's output. A run can pause, survive a crash, and resume from the exact step that stopped.
The OpenAI Agents SDK uses a different approach: a server-managed — a conversation ID — to continue a run. It does not checkpoint graph nodes.
The contrast matters: LangGraph wins for deterministic mid-graph resumability. The SDK wins for fast iteration inside OpenAI's ecosystem.
The OpenAI Agents SDK wraps the with four provider-native primitives: Agent, , , and . Almost no boilerplate required.
An Agent is a model, instructions, and optional tools. A lets one agent delegate to a specialist. The SDK keeps the conversation thread intact.
A runs validation logic before the model sees input. Bad or off-topic requests never reach the LLM.
Built-in records every model call, tool invocation, handoff, and guardrail decision. No extra instrumentation needed.
Your research assistant now needs to route queries: general questions go to a triage agent, deep-dive research goes to a specialist sub-agent.
With the OpenAI Agents SDK, define the specialist first. Hand it to the triage agent as a target. The SDK wires the delegation automatically.
Before the triage agent runs, an input checks if the query is research-related. Off-topic requests are rejected before any model tokens are spent.
Every step appears in the SDK trace: the guardrail decision, the triage model call, the handoff, and the specialist's tool use. You can pinpoint exactly which step failed.
# Stage 1 — define specialist, then triage agent with handoff research_agent = Agent( name="ResearchSpecialist", model="gpt-4o", instructions="Deep-dive research. Cite sources.", tools=[search_tool], ) triage_agent = Agent( name="Triage", model="gpt-4o-mini", instructions="Route research queries to the specialist.", handoffs=[research_agent], )
The specialist is declared first so the triage agent can reference it in — the SDK resolves delegation at runtime, not at definition time.
Notice the triage agent uses a cheaper model (gpt-4o-mini) for routing; the specialist uses gpt-4o only when needed — a deliberate cost split.
The triage agent handles it. Because the query isn't research-related, the triage agent answers directly and never triggers the handoff — the specialist stays idle. The guardrail (added in Stage 2) would catch it even earlier.
# Stage 2 — attach input guardrail and run with tracing def topic_guardrail(input_text: str) -> GuardrailResult: if "research" not in input_text.lower(): return GuardrailResult(blocked=True, reason="Off-topic") return GuardrailResult(blocked=False) triage_agent.input_guardrails = [topic_guardrail] result = Runner.run(triage_agent, "Research quantum error correction") for step in result.trace: print(step.kind, step.status) # guardrail OK → model → handoff → tool
The guardrail fires before the model call; if blocked=True the run stops immediately and the trace records a guardrail_blocked step — no tokens consumed.
Iterating result.trace gives you every step in order: step.kind is the step type and step.status is ok or failed — scan for the first failed to find the broken step.
Exactly one step appears: kind='guardrail', status='blocked'. Zero model-call steps follow — the SDK short-circuits before any LLM inference runs.
# Variation: add a CitationAgent that formats sources citation_agent = Agent( name="CitationFormatter", model="gpt-4o-mini", instructions="Format raw sources into APA citations.", ) research_agent = Agent( name="ResearchSpecialist", model="gpt-4o", instructions="Deep-dive research. Pass sources to citation agent.", tools=[search_tool], handoffs=[citation_agent], # TODO: what goes here instead to also keep triage_agent's guardrail active? )
This variation extends the running example: the research specialist now hands off to a citation formatter after gathering sources.
The TODO asks you to think about guardrail scope — the crux of this module's core idea.
Move the guardrail attachment to research_agent (or whichever agent Runner.run() calls first): research_agent.input_guardrails = [topic_guardrail]. Changed line: the guardrail is now on research_agent, not triage_agent, because the SDK only fires input guardrails on the agent passed to Runner.run() — downstream handoff targets don't re-run the entry guardrail automatically.
The SDK's default model can change between releases.
Symptom: costs jump or reasoning quality shifts between deploys with no code change.
Fix: always pin model="gpt-4o" explicitly on every Agent — never rely on the default.
Input guardrails only fire on the agent passed to Runner.run().
Symptom: off-topic queries reach the specialist with no error — the guardrail silently does nothing.
Fix: attach the guardrail to the entry-point agent, and confirm in the trace that a guardrail step appears as the first trace entry.
Traces include full prompts, tool inputs, and model outputs by default.
Symptom: PII or API keys appear in your trace store — no runtime error, just a compliance breach.
Fix: treat traces as production telemetry; scrub or redact sensitive fields before storing.
Score all six frameworks across four axes — control granularity, multi-agent style, data/retrieval story, and provider lock-in — using the research-assistant scenario as the shared benchmark. You'll leave with a decision matrix and a concrete recommendation rule for each of the five most common use-case patterns.
A four-axis decision matrix and five use-case recommendation rules that let you pick the right open-source AI agent framework for any scenario.
Why this matters: After six modules of building the same research-assistant scenario six different ways, this module turns that hands-on experience into a reusable selection framework you can apply immediately to your own SEO/GEO page agent build.
Decision this forces: Final framework selection: match control needs, team/conversation style, retrieval requirements, and provider constraints to one of the six frameworks (or a deliberate combination).
Answer: provider lock-in are tightly coupled to OpenAI's models and APIs. Module 5 surfaced that tradeoff — but it left the bigger question open: given all six frameworks, which one fits your specific use case?
This module scores every framework on four axes and gives you a concrete rule for each of the five most common use-case patterns.
Every open-source AI agent framework makes four structural bets. They determine where it wins and where it costs you.
The research-assistant scenario — one agent, a search tool, a document store, and optional sub-agents — is the shared benchmark across all six modules. You already have intuition for how each framework handled it.
Five patterns cover most production agent use cases. Each maps to a clear first choice — and flags where a second framework is often layered in.
The most common production combination is LlamaIndex + LangGraph: LlamaIndex owns the retrieval pipeline; LangGraph owns the stateful orchestration around it.
# Use case: a legal-document review agent # - Must pause for human sign-off before filing # - Searches 50k internal PDFs (private corpus) # - Three roles: extractor, risk-analyst, drafter # - Team uses Azure OpenAI (no OpenAI.com) scores = { "control_granularity": "high", # pause/resume required "multi_agent_style": "role", # three named roles "retrieval_depth": "deep", # 50k private PDFs "provider_lock_in": "azure", # Azure OpenAI only } # TODO: Given these four scores, which framework (or pair) # should you recommend? Fill in the variable below. recommendation = ___
This snippet encodes the four-axis scores for a legal-review agent — your job is to name the best-fit framework before revealing the answer.
The scores are already filled in. Only the recommendation line is missing — that's the crux this module teaches.
recommendation = "LlamaIndex (retrieval) + LangGraph (orchestration)"
# Changed lines vs. the worked example:
# - retrieval_depth='deep' → rules out AutoGen, CrewAI alone, and OpenAI Agents SDK
# - control_granularity='high' + pause/resume → rules out LangChain's ReAct loop
# - multi_agent_style='role' is satisfied by LangGraph nodes named per role
# LlamaIndex QueryEngine handles the 50k-PDF corpus;
# LangGraph StateGraph adds checkpointed pause-for-human-sign-off.
# Azure OpenAI works with both frameworks — no lock-in conflict.
| Option | Control granularity | Retrieval depth | Provider flexibility | When to choose | Cost | Complexity |
|---|---|---|---|---|---|---|
| LangGraph | Highest — explicit nodes, conditional edges, typed state | Retrieval is a tool node — no first-class RAG primitives | Model-agnostic via LangChain integrations | When you need deterministic, resumable workflows with explicit state machines — e.g. multi-step approval flows or human-in-the-loop pipelines. | Free / OSS; compute cost scales with model calls, not the framework. | High — graph definition, node wiring, and checkpoint setup add boilerplate. |
| CrewAI | Medium — sequential or hierarchical process; Flow adds event routing | Knowledge sources supported; retrieval via tool or knowledge object | Model-agnostic; broad connector list | When the problem maps naturally to named roles collaborating on tasks — content pipelines, research-then-write, or any workflow a human team would recognise. | Free / OSS; enterprise control-plane is paid. | Low-medium — role prompts and task definitions read like a job description. |
| AutoGen | Low-medium — conversation-driven; less explicit state | Retrieval via tool only; no native RAG layer | Model-agnostic; strong Azure OpenAI support | When agents need to converse, debate, or iteratively refine output — code review loops, adversarial critique, or multi-model negotiation. | Free / OSS (Microsoft Research). | Medium — GroupChat setup is straightforward; custom speaker selection adds complexity. |
| LangChain | Medium — LCEL chains and ReAct loop; less explicit than LangGraph | Good RAG support; broad retriever options | Model-agnostic; largest integration library | When integration breadth matters most — connecting many APIs, databases, or third-party tools with minimal glue code. | Free / OSS; LangSmith observability is paid. | Low-medium — create_react_agent is a one-liner; custom chains add complexity. |
| LlamaIndex | Medium — agent loop is standard; retrieval pipeline is highly configurable | Highest — native QueryEngine, hybrid index, re-ranking | Model-agnostic; broad embedding/LLM support | When the core value is deep retrieval over private documents — complex indexing strategies, hybrid search, or retrieval-first agent design. | Free / OSS; LlamaCloud managed service is paid. | Medium — QueryEngine and index setup require understanding the data pipeline. |
| OpenAI Agents SDK | Medium — handoffs and guardrails are structured; less graph-level control | File search tool available; no deep RAG pipeline | Tightly coupled to OpenAI APIs | When you're all-in on OpenAI models and want built-in tracing, guardrails, and handoffs with minimal boilerplate — prototyping or production on GPT-4o. | Free / OSS framework; locked to OpenAI API pricing. | Low — fewest lines to a working agent; guardrails and tracing are on by default. |
Three failure patterns account for most bad framework choices — and two of them are silent until you're deep in production.
If you ask an LLM to recommend a framework, run these four checks before trusting the output.
You now have everything you need to answer the most common framework question cold.
That's the lesson's spine in two sentences. It is also the answer an LLM or a search engine will surface from a well-written explainer page.
across six frameworks. You built the research-assistant scenario six ways. Now you hold a decision matrix you can apply to any new use case. The solo capstone challenge asks you to pick a framework for a scenario you haven't seen before, justify it on all four axes, and sketch the key code structure. That's where the pattern becomes yours.
Before scrolling down: from memory, name the six frameworks in the order you'd evaluate them for a new project, state the one axis each framework wins on, and sketch the four-column decision matrix. Then check your answer against the module summaries.
Apply what you learned to SEO/GEO page — the page title is the H1 and the exact search query. OPEN with a 40-60 word self-contained, quotable answer capsule that names the entity explicitly in sentence one (no "in this lesson…" preamble) — this is the sentence an LLM lifts. Name the entity by its full name every time (not pronoun-only). Structure each section as ONE sub-question (How it works · X vs Y · When to use it · Example · Common mistakes), each opening with its own 1-2 sentence direct answer then depth. End with a genuine 3-5 question FAQ phrased as real user questions ending in "?" (these become FAQPage schema + the knowledge check). Prefer quotable specifics — concrete numbers, defaults, version names, one short snippet — over vague prose. Self-contained for a reader who arrived cold from a search engine or an LLM. Intent = explainer: mechanism + example + when-to-use. Target query: "open source ai agents"..
In the reason-act-observe cycle every open-source AI agent framework shares, what is the correct role of 'state'?
State accumulates everything the agent knows so far — prior reasoning, tool results, and conversation history — and is threaded through each loop iteration so the model can decide what to do next. The system prompt is a static config, not state. The final answer is the output after the loop, not state itself. An API key is infrastructure, not agent state.
A LangGraph StateGraph is defined with a MemorySaver checkpoint. A user's multi-step workflow is interrupted mid-run. What does the MemorySaver allow the framework to do that a plain in-memory agent cannot?
MemorySaver persists the graph's state at each checkpoint, so an interrupted run can be resumed from the last saved node rather than restarted from scratch. Retry logic is a separate concern not provided by MemorySaver. Model-provider swapping is a LangChain portability feature, not a LangGraph checkpoint feature. Branch merging is a graph topology concern, not a persistence concern.
Read this short CrewAI setup:
crew = Crew(agents=[researcher, writer], tasks=[research_task, write_task], process=Process.sequential)
A teammate says the writer agent keeps ignoring its role prompt after several tasks. Which failure mode does this describe, and which framework would avoid it by design?
Role-prompt drift is the CrewAI-specific failure where an agent's behavior diverges from its declared role over a long task sequence. AutoGen's message-passing model does not rely on static role prompts — agents adapt through conversation, so drift of this kind is less likely. Runaway loops are AutoGen's failure mode, not CrewAI's. State explosion and empty context belong to LangGraph and retrieval frameworks respectively.
When would you choose LlamaIndex over LangChain for building an open-source AI agent?
LlamaIndex's retrieval pipeline — with its VectorStoreIndex, chunk-size controls, and QueryEngineTool — is purpose-built for use cases where getting the right context into the model is the hardest problem. LangChain's strength is broad model and tool portability, not retrieval depth. Built-in guardrails and handoffs describe the OpenAI Agents SDK. Deterministic graph transitions describe LangGraph.
You are evaluating two production stacks. Stack A uses the OpenAI Agents SDK exclusively. Stack B uses LangGraph for orchestration and LangChain for tool/model wiring. Name ONE concrete advantage Stack B has over Stack A, and ONE concrete advantage Stack A has over Stack B.
The OpenAI Agents SDK's lock-in cost is the key tradeoff: you gain native tracing, guardrails, and handoffs for free, but you are tightly coupled to OpenAI's model tier. LangGraph plus LangChain is more verbose to set up but lets you swap providers, add durable checkpointing, and combine frameworks — a common production pattern precisely because neither framework alone covers every need.