Use durable execution when agent work must survive retries and crashes.
Trace how an event-history log turns volatile in-memory state into a replayable record, and pin down exactly what 'durable' promises versus what it doesn't. The running scenario is a multi-step research agent (fetch → summarize → store) that must survive a mid-run worker crash.
Explains how an event-history log makes workflow execution durable and what the replay guarantee actually promises versus what it doesn't.
Why this matters: Every subsequent module builds on this contract — checkpointing, state persistence, and failure recovery only make sense once you know what replay guarantees and where it breaks.
Your research agent fetches three papers and crashes mid-summarization. Without durable execution, all progress in heap memory is lost. The question isn't whether this will happen—it's how much work you'll repeat.
writes every decision and result to an log before acting. A replacement worker replays that log to reconstruct exactly, then continues from the next unfinished step.
This differs from a job queue that retries the whole job. Replay is surgical: completed steps are skipped, not re-executed.
During , the re-executes workflow code top-to-bottom but short-circuits every step whose result is already in the . The code runs; the side effects don't.
This means the contract has two halves. First: completed are never re-invoked — their recorded outputs are injected directly. Second: the workflow code itself must be — given the same history, it must make the same branching decisions every time.
What durable execution does NOT promise: exactly-once side effects on external systems. Activities run with semantics. If an activity completes but the result acknowledgment is lost before it's written to history, the activity runs again on the next worker.
Consider the three-step research agent: fetch_papers(), summarize(), store_results(). Each is an — a unit of work with its own retry policy and result recorded in the .
The workflow function itself — the code that calls these three activities in sequence — belongs in the orchestrator. It must contain no I/O, no randomness, no wall-clock reads. Its only job is to express the decision graph.
Now the crash scenario: the worker dies after summarize() completes but before store_results() is called. On replay, the engine injects the recorded summary directly — the LLM is never called again. Only store_results() actually executes.
store_results() had already written to the database before the crash, but its completion event wasn't flushed to history? The activity runs again on replay. Without an , you get a duplicate row — silently.Non-deterministic workflows don't fail loudly—they silently diverge. The replaying worker makes a different branching decision. The engine detects a history mismatch and aborts with a cryptic error.
datetime.now() inside workflow code. On replay, the clock returns a different value. Any branch conditioned on time takes a different path. Use the engine's workflow-safe timer API instead.random.random() or UUID generation outside an activity. Each replay seeds a fresh value, breaking any logic that depends on it.import datetime, random def research_workflow(topic: str): run_id = random.randint(1000, 9999) # ❌ non-deterministic papers = fetch_papers(topic) # external I/O inline if datetime.datetime.now().hour < 12: # ❌ wall-clock branch summary = summarize_fast(papers) else: summary = summarize_thorough(papers) store_results(run_id, summary) # ❌ run_id differs on replay
random.randint(1000, 9999)datetime.datetime.now().hourfetch_papers(topic)This workflow has three determinism violations — each one causes a different failure mode on replay. Identify all three before revealing the answer.
Line 3: random.randint produces a different run_id on replay — store_results writes a duplicate row under a new key. Line 5: datetime.now().hour can flip the branch on replay, causing the wrong summarizer to run and a history-command mismatch error. Line 4: fetch_papers is called inline without being wrapped as an activity — its result isn't in history, so the network call fires again on replay (at-least-once re-execution, possible duplicate fetch charges or rate-limit hits). All three are silent until replay actually occurs.
def research_workflow(topic: str, run_id: str): # id passed in, not generated papers = yield activity(fetch_papers, topic, retry=RetryPolicy(max_attempts=3)) summary = yield activity(summarize, papers) # LLM call inside activity yield activity(store_results, summary, idempotency_key=run_id) # safe on re-execution return summary
yield activity(...)RetryPolicy(max_attempts=3)idempotency_key=run_idEvery external call is now wrapped as an — its result is recorded in the before the workflow advances. The workflow function itself is pure control flow: no I/O, no randomness, no clock reads.
The on store_results means a duplicate call from at-least-once replay writes nothing new — the database sees the same key and skips the insert.
The branch lives in the workflow function (not inside an activity), because it's pure control flow with no side effects. The constraint: it must be deterministic — derive the condition from data already in the event history (e.g., len(papers) from the recorded fetch result), never from a live clock, random value, or inline model call. Changed lines: add if len(papers) > 5: before the summarize activity call, selecting the activity function accordingly. The branch itself is never recorded — only the activity result is — so the condition must always evaluate identically on replay.
Durable execution fails in three distinct ways, each with a different signature.
You ship a code change that reorders two activity calls. In-flight workflows replay against old history but encounter a different command sequence. The engine throws a non-determinism error and blocks the workflow. Observable: workflows stuck in "running" state with a history-mismatch exception in worker logs. Fix: use versioning APIs to branch old vs. new logic, or drain in-flight runs before deploying.
An activity completes and writes to a database, but the worker crashes before the completion event is persisted. On replay, the activity runs again. Observable: duplicate rows or double-charged API calls—no error, no log line. Fix: every mutating activity must carry an .
An agent that loops—polling, retrying, accumulating results—appends an event for every iteration. After thousands of loops, the grows large enough that replay becomes slow or hits engine size limits. Observable: replay latency climbs linearly with loop count; some engines hard-cap history size and terminate the workflow. Fix: use continue-as-new to reset history while preserving state.
datetime/random/UUID calls appear in workflow-function scope, (3) any loop has a continue-as-new escape hatch, and (4) all mutating activities carry idempotency keys.Slide to see how activity granularity trades replay cost against re-execution risk. Coarser activities mean less history overhead but more repeated work on failure; finer activities mean cheaper replay but a larger event log.
Examine how workflow engines checkpoint state at activity boundaries, what granularity of checkpoint is safe, and where the tradeoff between checkpoint frequency and storage cost bites. Apply this to the research agent: decide which step boundaries become checkpoint seams and why skipping one creates a re-execution hazard.
Examines how workflow engines checkpoint state at activity boundaries and where the tradeoff between checkpoint frequency and storage cost bites.
Why this matters: Choosing the wrong checkpoint granularity for a multi-step agent causes either duplicate side effects after a crash or ballooning history storage — both silent and expensive.
The log records every command issued and every result returned — not in-memory variables, but the decisions and their outcomes. re-drives workflow code against that log, rebuilding state deterministically without re-executing completed .
That guarantee is only as strong as what gets written to the log — and when. This module is about that 'when': checkpoint granularity is the decision that determines whether a crash costs you a retry or a duplicate side effect.
A is a durable record of at a specific boundary.
It is typically written after an completes and its result reaches the event log.
Coarser granularity (one checkpoint per phase) minimises write amplification. But it widens the re-execution window: a crash mid-phase re-runs every activity in that phase.
Finer granularity (one checkpoint per activity) shrinks the re-execution window. But it inflates history storage and increases orchestrator round-trips.
The non-obvious pressure: re-executing a activity produces a duplicate side effect.
A second API call, a second DB write, or a second email often causes more harm than a crash.
Safe checkpoint seams are placed after any activity with an observable external effect.
This ensures a replay never re-issues that activity.
The research agent runs three steps: fetch (HTTP call), summarize (LLM inference), and store (DB write).
Each is an activity with a distinct failure profile.
Consider two checkpoint layouts. Predict what happens after a worker crash mid-summarize:
Layout B is correct because store is non-idempotent without an explicit .
Summarize is an LLM call — expensive and non-deterministic. Checkpointing its result avoids costly re-inference on replay.
The fetch step is idempotent (read-only HTTP). A checkpoint there is optional.
But skipping it means a crash before summarize re-fetches, which is cheap and safe.
The seam after fetch is a cost-vs-safety call, not a correctness requirement.
Slide to see how checkpoint frequency shifts the re-execution window and history storage pressure for the research agent.
Workflow engines like Temporal and Restate persist inside their own event store.
The owns the source of truth and replay is automatic.
External state stores (Redis, DynamoDB, Postgres) give you control over schema and query patterns.
But they introduce a consistency gap: the engine's event log and external store can diverge.
This happens if a crash lands between the activity write and the engine's acknowledgement.
For the research agent's store step, writing to an external DB is correct.
But the activity must be wrapped so the engine records completion after the DB acknowledges, not before.
Reversing that order is the most common consistency bug in hybrid architectures.
Three failure patterns dominate — each with a distinct signature:
def research_workflow(query: str): # Activity 1: fetch raw = run_activity(fetch_source, query) # Activity 2: summarize summary = run_activity(call_llm, raw) # Activity 3: store — result written to external DB run_activity(write_to_db, summary) # Checkpoint written here (end of workflow) checkpoint()
run_activity(fn, *args)checkpoint()This workflow has one checkpoint at the end — a classic Layout A mistake. Before revealing, predict: which activity creates a duplicate side-effect risk, and what is the exact failure sequence?
Changed lines: move checkpoint() calls to immediately after each run_activity() call, or rely on the engine's per-activity event write if the framework supports it natively.
You now know where to place checkpoint seams and why ordering relative to external effects is critical.
But 'the engine records the result' is still a black box.
Who actually writes that record? How does work get routed to the right worker after a crash?
The next module opens that box: you'll map the internal components of a workflow engine.
These are the service, the , processes, and activity executors.
You'll trace exactly how a command issued by the orchestrator travels through those components.
This produces the durable record you just relied on.
Map the internal components of a workflow engine (orchestrator service, task queue, worker processes, activity executors) and trace how a command issued by workflow code travels to an activity and back. Use Temporal's architecture as the concrete reference, noting where Restate's service-first model diverges.
Maps the internal components of a workflow engine and traces how a command travels from orchestrator through task queue to worker and back.
Why this matters: Understanding this round-trip is the prerequisite for reasoning about failure recovery, retry placement, and the operational cost of choosing Temporal over Restate.
Decision this forces: Temporal's explicit workflow/activity semantics vs. Restate's service-first model — which fits the team's operational posture.
A workflow engine has four distinct runtime roles: the orchestrator service (Temporal's server), the , , and . Removing any one breaks the durability contract.
The orchestrator is stateless compute backed by a durable store. It never executes business logic. It records every command and result into the . Then it schedules tasks onto named queues. Workers long-poll those queues and pull tasks. They run either workflow replay or an activity function.
The split between workflow code and activity code is the engine's core invariant. Workflow code must be — no I/O, no randomness, no wall-clock reads. Activities are the only sanctioned place for side effects. Violating this boundary causes replay divergence bugs.
The orchestrator never received RespondActivityTaskCompleted. The ActivityTask remains in the queue past its scheduleToCloseTimeout. The engine reschedules it. But with one worker down, no process is polling. The task sits until the worker restarts or a second worker comes online.
The mitigating feature is the decoupled polling model. The orchestrator doesn't push to a specific worker address. Any worker registered on the same queue can pick up the orphaned task. Running at least two worker replicas on each task queue converts a single-process crash from a stall into a brief delay.
The non-obvious corollary: if all workers share one task queue but have different activity registrations, a task can be picked up by a worker that doesn't have the required activity function registered. The worker will fail the task immediately with a 'not registered' error. Not a timeout — an instant failure. Partition task queues by capability set, not just by service name.
# Research agent — summarize step # Stage 1: workflow issues the command def research_workflow(ctx, query): raw = yield ctx.schedule_activity( fetch_page, args=[query], task_queue="research-workers", ) # TODO: schedule summarize_text activity with raw as input # and store the result in `summary` yield ctx.schedule_activity(store_result, args=[summary])
yield ctx.schedule_activity(...)task_queue="research-workers"args=[raw]This is the research agent's workflow skeleton — fetch is wired; the summarize command is intentionally missing. Supplying the TODO is the crux: it must issue a new ActivityTask on the same task queue, passing the upstream result as input, before store_result can run.
# Changed lines (the crux):
summary = yield ctx.schedule_activity(
summarize_text,
args=[raw],
task_queue="research-workers",
)
# Event history between fetch result and store scheduling:
#
#
# (replay runs workflow code; coroutine resumes with raw)
#
# The workflow coroutine suspends again at the new yield.
| Option | Deployment surface | Workflow/activity boundary | State visibility | When to choose | Cost | Complexity |
|---|---|---|---|---|---|---|
| Temporal | Separate orchestrator service (server) + worker processes; two distinct deployment units | Hard enforced split: workflow code must be deterministic; activities own all I/O | Full event history queryable via Web UI and tctl; rich replay debuggability | Teams that can operate (or use Temporal Cloud for) a separate orchestrator cluster and want explicit workflow/activity separation with rich history tooling. | Self-hosted: infra + ops cost; Temporal Cloud: per-action pricing | Higher — orchestrator cluster + worker fleet + namespace management |
| Restate | Service-first: durable handlers live inside your service; Restate server is a thin sidecar/proxy | No enforced split — any handler can be durable; developer discipline required to avoid side-effect leaks | Journal visible but tooling is less mature than Temporal's history UI; fewer built-in observability hooks | Teams that want durable execution embedded in their existing service mesh without operating a separate orchestrator cluster, and prefer a handler-per-endpoint model over explicit workflow/activity types. | Lower operational overhead; hosted option available | Lower — single Restate server sidecar or hosted; no separate worker fleet |
Temporal caps event history at 50,000 events (default); hitting it terminates the workflow with a 'history size exceeded' error. Long-running agents that loop without using ContinueAsNew accumulate events silently until this wall. Observable symptom: workflow replay time grows linearly with history depth — you'll see worker poll latency climb before the hard cap hits.
Deploying new workflow code while in-flight workflows replay against old history causes a NonDeterministicWorkflowError if the command sequence changes. This is silent during development (unit tests don't replay history) and explodes in production on the first post-deploy replay. Fix: use workflow versioning (GetVersion / patched) before changing any command-issuing line.
In Restate's handler model, a developer who performs I/O directly in a handler body (outside a ctx.run() journaled block) gets no durability guarantee for that call. The handler re-executes on retry and the I/O fires again — classic at-least-once without idempotency. Unlike Temporal's enforced boundary, Restate's model relies on developer discipline; there is no compile-time check.
The command path you've traced assumes activities eventually succeed. But the research agent's fetch and LLM-summarize steps fail transiently. Rate limits, timeouts, and flaky upstreams cause failures. The engine reschedules, but on what interval? With what backoff? When does it give up?
The next module covers retry policy configuration. It covers initial interval, coefficient, max attempts, and non-retryable error classes. Then it tackles what happens when retries are exhausted: saga-style to undo partial work the engine can't roll back on its own.
Work through retry policy configuration (initial interval, backoff coefficient, max attempts, non-retryable errors), saga-style compensation, and the distinction between transient and permanent failures. The research agent's 'summarize' step calls an unreliable LLM endpoint — decide the retry envelope and the compensation path when all retries exhaust.
How to configure retry policies that distinguish transient from permanent failures, and how to design saga-style compensation when retries exhaust.
Why this matters: Retry and compensation are the two levers that keep a multi-step agent workflow from leaving partial state behind — getting them wrong either masks real errors or leaves orphaned artifacts.
Decision this forces: Retry-to-exhaustion with compensation vs. fail-fast with manual intervention — depends on downstream idempotency guarantees.
Your research agent's summarize step calls an unreliable LLM endpoint. Failures are either transient (429, 503, network blip) or permanent (400, auth failure, schema mismatch). Retrying permanent errors wastes quota.
A well-formed splits this explicitly. Non-retryable error types short-circuit immediately. Everything else enters the envelope. The non-retryable list is critical — omitting it burns all retry budget before you notice.
Four knobs define the envelope: initial interval (first wait), backoff coefficient (multiplier per attempt), max interval (ceiling), and max attempts. Jitter prevents thundering-herd re-entrancy.
SUMMARIZE_RETRY = RetryPolicy( initial_interval = timedelta(seconds=2), backoff_coefficient = 2.0, max_interval = timedelta(seconds=30), max_attempts = 4, non_retryable_errors = [ InvalidPromptError, # 400 — bad request, permanent AuthenticationError, # 401 — fix credentials, not time ], ) # Pass to the activity call, not the workflow-level default.
RetryPolicy(...)backoff_coefficientnon_retryable_errorsmax_attemptsThis policy gives the LLM endpoint three retries (four total attempts) with a 2 s → 4 s → 8 s → 16 s (capped at 30 s) wait curve. The non-retryable list is the load-bearing line: without it, a bad prompt would exhaust all four attempts before the workflow sees a real error.
Two attempts remain (3 and 4). The next wait is 4 s (initial 2 s × 2.0^1 = 4 s, still under the 30 s cap). The 429 is not in non_retryable_errors, so the policy continues.
Backoff coefficient = 2, initial interval = 1 s, max interval = 30 s. Drag to see cumulative wait before each attempt.
When all four attempts fail, the summarize activity raises a terminal error. At that point the workflow has already completed fetch and checkpointed its output — partial state that must be cleaned up or explicitly abandoned.
A handles this by running in reverse order: each completed step has a paired undo that the workflow invokes on the failure path. The key constraint is that compensations must themselves be idempotent — the engine may replay them.
The decision between retry-to-exhaustion + compensation and fail-fast + manual intervention hinges on one question: are the downstream steps ? If re-running a compensation could double-charge or double-publish, fail-fast and page a human.
async def research_workflow(query: str): compensations = [] try: doc = await run_activity(fetch, query, retry=FETCH_RETRY) compensations.append(lambda: run_activity(delete_draft, doc.id)) summary = await run_activity( summarize, doc, retry=SUMMARIZE_RETRY # 4 attempts ) compensations.append(lambda: run_activity(archive_summary, summary.id)) await run_activity(store, summary) except TerminalActivityError: for undo in reversed(compensations): await undo() # run in reverse; each must be idempotent raise
compensations.append(lambda: ...)reversed(compensations)TerminalActivityErrorawait undo()Each successful activity registers its undo before the next step runs, so the compensation list always reflects exactly what has completed. On terminal failure the workflow unwinds in reverse — fetch's draft is deleted before any summary artifact is touched.
Both compensations are registered by the time store fails: [delete_draft, archive_summary]. They fire in reverse: archive_summary first, then delete_draft. The store activity has no compensation because it never completed — there is nothing to undo. Changed lines vs. Stage 1: the compensations list now has two entries, and the TerminalActivityError catch covers store as well as summarize.
If delete_draft is not idempotent and the engine replays it, you get a 404 on the second attempt. Or worse: a different record deleted if IDs are reused. Observable: the workflow history shows two ActivityTaskScheduled events for the same compensation with unexpected errors.
A 400 from the LLM (bad prompt template) retried four times burns ~15 s and quota before surfacing. The symptom: a workflow exhausting retry budget in under a minute. Check the error code on the first attempt in history before assuming the endpoint is flaky.
Full jitter spreads retries across the window. But if hundreds of workflows hit the same rate limit simultaneously, even jittered retries cluster at the reset boundary. The fix is a token-bucket or queue-based admission layer upstream — backoff alone cannot solve this at scale.
If you append to compensations after the next await, a crash between activity completion and append leaves the undo unregistered. Always append the compensation lambda immediately after await returns — before any other logic.
| Option | Downstream idempotency | Compensation feasibility | When to choose | Cost | Complexity |
|---|---|---|---|---|---|
| Retry-to-exhaustion + saga compensation | Required — compensations replay under at-least-once delivery. | Explicit undo per step; works when side effects are reversible (delete draft, archive record). | Downstream steps are idempotent and each completed step has a well-defined, testable undo activity. | Quota consumed across all retry attempts; compensation activities add latency on the failure path. | High — compensation logic must be authored, tested, and kept in sync with the forward path. |
| Fail-fast + manual intervention | Not required — the workflow halts before any non-idempotent step is retried. | Bypassed entirely — a human inspects state and decides the recovery action. | Side effects are non-idempotent (e.g. charge a card, send an email) or compensation logic is ambiguous — surface the failure immediately and page a human. | No wasted retry quota; human time is the cost. | Low in code; high in ops — requires runbook, alerting, and human SLA. |
Dissect why idempotency is the hardest correctness problem in durable agents: LLM calls are non-deterministic, tool calls mutate external state, and the engine's at-least-once delivery guarantee means duplicates are the default. Design idempotency keys, deduplication windows, and activity-level fencing for the research agent's 'store' step.
Designs idempotency key schemes and activity-level fencing so the research agent's 'store' step is safe under at-least-once delivery.
Why this matters: Without explicit idempotency, every worker restart or retry in a durable workflow risks duplicate writes, double-charges, or silent data loss — this module gives you the tools to prevent all three.
Decision this forces: Client-side idempotency keys vs. server-side deduplication — tradeoffs in key lifetime, storage, and failure window coverage.
delivery. will eventually succeed. But it may execute more than once. This happens on worker crash, network timeout, or task-queue redelivery.
: every activity that mutates external state must tolerate duplicate execution. It must not produce duplicate effects. Module 5 is entirely about designing that guarantee into the research agent's 'store' step.
Three forces collide in durable agents. Each alone would be manageable. Together they make idempotency the hardest correctness problem.
Your research agent's workflow calls the LLM directly — not inside an activity. It decides whether the fetched document is worth storing.
A worker crashes after 'fetch' completes but before 'store' runs.. The LLM call re-executes and returns a different answer. The engine detects a non-determinism violation and panics the workflow.
This is the replay contract: workflow code is a pure function of the event history. All side effects — including inference — belong in activities.
must be stable across worker restarts, clock skew, and retries. It cannot be a timestamp, random UUID, or anything derived from mutable runtime state.
The right source is the workflow's own identity. Combine the workflow run ID with the activity name and a deterministic input hash. This gives a key unique per logical operation. It is identical on every retry of that same operation.
Key = f"{workflow_run_id}:{activity_name}:{hash(canonical_input)}"import hashlib def make_store_key(run_id: str, doc_url: str) -> str: digest = hashlib.sha256(doc_url.encode()).hexdigest()[:12] return f"{run_id}:store:{digest}" def store_result(db, idem_key: str, payload: dict) -> str: existing = db.get_by_key(idem_key) if existing: return existing["record_id"] # duplicate — return prior result record_id = db.insert(payload, idempotency_key=idem_key) return record_id
hashlib.sha256(...).hexdigest()[:12]db.get_by_key(idem_key)db.insert(payload, idempotency_key=idem_key)The activity checks for a prior write before inserting — the classic check-then-act fence. The key is derived from stable inputs (run ID + document URL), so it is identical on every retry of this activity.
Notice the key is built in make_store_key (called from workflow code, recorded in history) and passed in — the activity never generates its own key.
On retry, db.get_by_key returns None (the insert never committed), so the activity proceeds to insert. The result is correct: exactly one row is written. This is the classic TOCTOU gap — it is safe here only because the database enforces a unique constraint on idempotency_key; without that constraint, a concurrent retry could produce a duplicate row.
| Option | Failure window coverage | Key lifetime control | Downstream coupling | When to choose | Cost | Complexity |
|---|---|---|---|---|---|---|
| Client-side idempotency keys | Full control — key lifetime is whatever your store supports. | You set expiry; can match workflow pause duration. | Service-agnostic; works with any backend. | When the downstream service has no native deduplication, or when you need keys to survive beyond the service's dedup window (e.g. multi-day workflows). | Extra read per activity call; key store must be durable and low-latency. | Medium — you own key generation, storage, and expiry logic. |
| Server-side deduplication | Limited by service TTL — long-paused workflows may fall outside the window. | Service-defined; cannot extend without client-side augmentation. | Requires the service to support idempotency keys; not universal. | When the downstream service natively deduplicates (e.g. Stripe, Twilio, many cloud APIs) and your workflow completes within the service's dedup window. | No extra store, but you're bound by the service's key TTL (often 24 h). | Low — pass the key in the request header; the service handles storage. |
If you use only the activity name and input hash — omitting the run ID — two different workflow runs share a key. The second run's store call finds the first run's record and silently returns it. Nothing is written. Observable symptom: the second run's result record is missing from the database, but the workflow reports success.
A workflow pauses for human approval (48 h). The downstream service's dedup window is 24 h. On resume, the retry is treated as a fresh request. The service inserts a duplicate row. No error is raised. The workflow succeeds. You discover the duplicate only in a downstream audit.
Generating the key inside the activity body produces a different key on each retry. Use time.time() or uuid4() as examples. Every retry looks like a new operation. The fence never fires. Duplicates accumulate. records the first key. Subsequent retries generate new keys that bypass the check.
def store_result_v2( db, idem_key: str, payload: dict, quota_api ) -> str: existing = db.get_by_key(idem_key) if existing: return existing["record_id"] # TODO: deduct one quota unit from quota_api, # then insert payload — in the right order so a # crash between the two steps is safe on retry. ...
quota_api.deduct(idem_key)db.insert(payload, idempotency_key=idem_key)The 'store' step now also deducts a quota unit from an external API before writing the record. Stop — attempt the TODO before revealing: decide the correct operation order and explain why a crash between the two steps is still safe.
Correct body:
quota_api.deduct(idem_key) # idempotent if quota_api accepts keys
record_id = db.insert(payload, idempotency_key=idem_key)
return record_id
Order matters: insert first, then deduct — NO. Deduct first: if the worker crashes after deduct but before insert, the retry re-enters, finds no existing record, and calls quota_api.deduct again. That's a double-charge UNLESS quota_api itself accepts the idem_key and deduplicates. The safe design is: (a) pass idem_key to quota_api.deduct so it is idempotent, then (b) insert. A crash between them is safe because the retry re-deducts (no-op via key) and then inserts. Changed lines vs. Stage 1: added quota_api.deduct(idem_key) before db.insert, and both calls use the same key.
Assemble the complete pattern: wrapping LLM reasoning loops, tool calls, human-in-the-loop approvals, and long-running waits inside durable workflow boundaries. Diagnose the three most common integration failure modes — non-deterministic workflow code, unbounded history growth, and signal/approval deadlocks — using the full research agent as the test case.
Assembles the complete durable-agent pattern — LLM loops, tool calls, human approvals, and long waits inside workflow boundaries — and diagnoses the three integration failure modes.
Why this matters: This is the synthesis module: it shows how every prior concept (checkpoints, retries, idempotency, signals) locks together into a production-ready agent, and forces the engine-selection decision.
Decision this forces: Which workflow engine fits the agent's topology — Temporal for complex durable workflows, Restate for service-first stateful handlers, Inngest for event-driven lighter-weight pipelines.
You've built the pieces in isolation — checkpoints, retries, idempotency keys, signals. The final assembly question is: where exactly do the boundaries go when an agent's reasoning loop, tool calls, human approvals, and long waits all live inside one workflow?
The rule is structural: every call that touches the outside world is an . The code that sequences them must be : no random calls, no datetime.now(), no direct HTTP — only activity invocations and control flow over their results.
Human-in-the-loop approval maps to a durable wait. The workflow suspends, the worker releases its thread, and the engine persists the blocked state in the . No polling loop, no held connection — just a recorded wait that survives any crash.
Long-running waits are safe only because the engine's mechanism rebuilds workflow state from history on resume. The cost is that every event you emit grows that history — which is the first place the pattern breaks.
These are the three failure modes that appear in production integrations of the research agent. Each has a distinct symptom.
Calling an LLM directly inside workflow code (not wrapped as an activity) means the produces a different decision tree than the original run. The engine detects a history mismatch and throws a non-determinism error. Or worse, it silently takes a different branch and corrupts state. Symptom: workflow replay fails with 'history mismatch' or the agent loops on a step it already completed.
A tight reasoning loop — plan → call tool → observe → plan again — emits one history event per iteration. Without activity boundaries that batch or paginate, a 200-iteration research loop produces thousands of events. Replay time grows linearly; eventually the engine rejects the workflow or replay times out. Symptom: replay latency climbs with run length; long workflows time out on resume.
A workflow waiting on a human-approval with no timeout will block forever if the signal never arrives. A reviewer is on leave, a webhook misconfigures, or a queue drains. Conversely, a timeout that fires before the reviewer acts cancels a legitimate approval. Symptom: workflow stuck in 'waiting for signal' indefinitely, or approval silently skipped when the timer fires first.
The research agent runs: fetch sources → LLM summarize → human approval → store result. A colleague hands you this design and asks you to find what breaks before it hits production.
Stop — before reading on: identify which failure mode each bullet triggers. What would the observable symptom be in each case?
# All LLM and tool calls are activities; workflow code is pure control flow. def research_workflow(sources: list[str]) -> str: summaries = [] for batch in chunked(sources, size=10): # batch to cap history growth texts = run_activity(fetch_batch, batch) # activity: HTTP fetch summary = run_activity(llm_summarize, texts) # activity: LLM call summaries.append(summary) approved = wait_for_signal( # durable wait — no thread held signal="human_approval", timeout_seconds=86400, # 24-hour guard on_timeout="escalate", # explicit branch, not silent skip ) if not approved: run_activity(escalate_to_manager, summaries) return "escalated" return run_activity(store_result, summaries) # activity: DB write
run_activity(fn, args)chunked(sources, size=10)wait_for_signal(...)on_timeout="escalate"This pseudocode shows the three integration fixes applied together to the research agent.
Batching with chunked(sources, size=10) caps the number of history events per run — 50 sources become 5 activity pairs instead of 50. The wait_for_signal call suspends durably; the on_timeout="escalate" branch is explicit, so a late signal after timeout is handled by checking approved — it will be False, routing to escalation rather than double-executing.
The engine replays the event history up to the signal-wait command and re-enters the blocked state. The timeout clock does NOT reset — it was recorded in the event history when the wait started, so the remaining time is preserved exactly. This is the durable execution guarantee: wall-clock time is tracked in history, not in the worker process.
| Option | History model & replay fidelity | Operational footprint | Agent topology fit | When to choose | Cost | Complexity |
|---|---|---|---|---|---|---|
| Temporal | Full event-history log; determinism enforced at replay; history compaction via ContinueAsNew | Requires Temporal server (or Cloud); worker processes; namespace management | Best for complex multi-step agents, long-running plans, hierarchical multi-agent sagas | Complex durable workflows with long waits, multi-step sagas, or high reliability SLAs — and your team can operate (or pay for cloud) Temporal's server. | High (self-hosted) / usage-based (Temporal Cloud) | High — separate server, namespace config, worker fleet |
| Restate | Journal-based; durable promises per handler invocation; less granular than Temporal's full history | Lighter than Temporal; handlers live inside your services; Restate server is the only external dep | Strong for service-first agents; less suited to long multi-agent sagas or complex branching plans | Service-mesh or microservice architectures where agents are stateful handlers co-located with services — lower ops overhead than Temporal, stronger consistency than Inngest. | Medium (self-hosted) / Restate Cloud pricing | Medium — Restate server or cloud; handler-per-service model |
| Inngest | Step-level checkpoints only; no full event-history replay; limited mid-workflow branching | Near-zero ops; runs on serverless functions; Inngest cloud handles scheduling | Best for linear or lightly branching pipelines; tight reasoning loops or long signal waits push its limits | Event-driven pipelines where agents react to webhooks or queued events, steps are coarse-grained, and you want minimal infrastructure — not for workflows requiring fine-grained history or long blocking waits. | Low to medium; usage-based; generous free tier | Low — serverless-friendly; step functions over HTTP |
You can now structure a full agent — LLM reasoning loop, tool calls, human approval, long waits — inside a durable workflow boundary. The three integration failure modes are diagnosed and fixed.
You can choose between Temporal, Restate, and Inngest based on replay fidelity, operational footprint, and agent topology. Not just familiarity.
The capstone challenge ahead asks you to apply the complete pattern to a novel agent topology. You haven't seen it in the lesson — one where the failure modes interact. You'll need to identify the integration boundaries, choose the right engine, and defend the tradeoffs. Everything from modules 1–6 is in play.
Before reviewing the summary: reconstruct from memory the path from event-history contract → checkpoint placement → engine architecture → retry/compensation → idempotency fencing → agent integration. For each step, name the failure mode it prevents and the tradeoff it introduces.
Apply what you learned to Durable Agents with Workflow Engines.
Before choosing, recall the durable execution contract from memory: what guarantee does a workflow engine replay provide that a simple job queue retry does not?
The correct answer is right because durable replay rebuilds the workflow’s logical state from event history and resumes deterministic decisions. Rerunning the whole job is just retry behavior and can duplicate side effects; storing every external database change is not what the engine does; preventing worker crashes is an availability problem, not the replay guarantee.
A workflow charges a customer, sends a confirmation email, and updates an internal ledger. The worker crashes after the charge succeeds but before the activity result is recorded. Which missing checkpoint most directly risks a duplicate charge on recovery?
The correct answer is right because recovery needs to know that the charge already happened or must use the same idempotency key so retrying cannot create a second charge. The email checkpoint happens too late to protect the payment; an in-memory call stack is lost on crash and not a durable checkpoint; the prompt checkpoint may help reproduce content but does not fence the payment side effect.
When would you choose Temporal’s explicit workflow/activity model over Restate’s service-first handler model for an agent system?
The correct answer is right because Temporal fits teams ready to operate explicit workflow/activity boundaries for complex durable orchestration. Putting LLM calls directly in orchestration breaks deterministic replay; a purely stateless endpoint does not need a durable workflow engine; no engine removes the need to model retry, timeout, and compensation behavior.
What is wrong with this workflow-code snippet?
result = llm.generate(prompt)
if result.score > 0.8: approve()
The correct answer is right because workflow code must replay the same decisions, while a live LLM call may return different text or scores on replay; the call belongs in an activity with its result captured. The numeric threshold is not the issue; deterministic branching is allowed when based on recorded data; running approval first would not fix nondeterminism and could create the wrong side effect.
A travel-booking agent has reserved a hotel and then fails permanently while booking a flight. The hotel API supports cancellation, but the flight API does not guarantee idempotency. In one or two sentences, explain whether you would retry-to-exhaustion with compensation or fail-fast with manual intervention, and why.
A strong answer chooses fail-fast/manual intervention for the unsafe non-idempotent operation and uses compensation for the already-completed hotel reservation. Retrying to exhaustion is flawed here because the flight API lacks idempotency guarantees, so repeated attempts may create duplicate side effects; doing nothing about the hotel is also incomplete because the workflow has already partially completed.