Understand pixel, browser, and DOM action loops plus their safety limits.
Trace the full observe → reason → act → observe cycle that underlies every browser and computer-use agent, using a concrete file-upload task as the running scenario. Identify where each loop variant diverges from this shared skeleton.
Traces the observe-reason-act-observe cycle that underlies every browser and computer-use agent, using a file-upload task as the running scenario.
Why this matters: This loop is the shared skeleton every subsequent module builds on — understanding where grounding strategy changes the data flowing through it is the prerequisite for every design and debugging decision ahead.
Your agent must upload a CSV to a third-party portal. The upload button appears only after login. The confirmation modal has a dynamic token. How does one control pattern handle all of that without a custom script for each step?
Every browser and computer-use agent runs the same four-beat observe → reason → act → observe cycle. On each beat the agent reads the current UI state (the observation. It reasons about what action moves it closer to the goal. It emits that action. It reads the resulting state as the next observation.
The loop terminates on success, an explicit stop action, or a budget limit — not by exhausting a fixed script. What varies across pixel, browser, and DOM agents is the : the encoding of the UI state that flows into the observation slot and the action vocabulary that flows out.
The four beats of the shared control pattern, mapped to the file-upload running scenario.
strategy determines what data occupies each slot.
style the model emits a thought and an action on every tick. Reasoning is interleaved with execution. The agent can react to surprises mid-task. In plan-then-act style the model first produces a full sub-task list. Then it executes each step with minimal per-step reasoning.
Goal: upload report.csv to a third-party portal. The portal requires login, a file-picker interaction, and a confirmation step.
decision point. A pixel agent reads the filename from a screenshot. A DOM agent reads the value problem surfaces between Act₃ and Observe₄. The modal may animate in over 300 ms. Reading the DOM before it settles yields a stale or empty observation.
observation shows the pre-action state. The model reasons as if the action never happened. It re-issues the action — producing duplicate clicks or double-submits. Fix: poll for a stable-state signal (network idle, element present, attribute changed) rather than a fixed sleep.
attack unique to agents that ingest raw page content. The symptom is the agent taking an action the user never requested — often irreversible (form submit, file delete).
strategy consistent within a single task execution.
# ReAct-style: reason and act interleaved each tick obs = capture_ui_state(grounding="dom") # observe₁ while not goal_reached(obs): thought, action = llm_react_step(goal, obs) # reason if action.is_destructive: action = await confirmation_gate(action) # guard irreversibility obs = dispatch_and_wait(action) # act + observe # Plan-then-act: full plan upfront, thin per-step reasoning plan = llm_plan(goal, capture_ui_state(grounding="dom")) for step in plan.steps: obs = dispatch_and_wait(step.action) # act + observe if not step.postcondition(obs): # verify raise PlanStaleError(step, obs) # no inline recovery
capture_ui_state(grounding=...)llm_react_step(goal, obs)confirmation_gate(action)dispatch_and_wait(action)step.postcondition(obs)Both variants share the same act + observe primitive ( dispatch_and_wait); the difference is where reasoning lives. on every tick; plan-then-act has no per-tick escape hatch — a stale plan raises PlanStaleError rather than recovering gracefully.
Step 4 (click Confirm) is the first irreversible action — the file is submitted to the server. If the plan was built from an earlier DOM snapshot where the filename was correct but the portal silently swapped the target folder, the agent clicks Confirm, the upload completes to the wrong destination, and Observe₅ still shows a success banner. PlanStaleError is never raised because the postcondition (banner present) is satisfied — the corruption is silent.
. The only observation is a raw screenshot. Every action is an (x, y) coordinate.
problem is acutest here. It introduces failure modes that don't exist in DOM-grounded loops.
Examine how pixel agents encode the screen as a vision model input, emit (x, y) click/type actions, and handle the settle-wait problem — all using the running file-upload scenario on a desktop app with no accessible DOM. Focus on coordinate drift, resolution sensitivity, and OCR error propagation.
Decision this forces: When is pixel-based control the only viable option, and what compensating controls are required?
Grounding maps high-level intent ("click Upload") to concrete, executable UI actions.
With no DOM, the only observation is a raw screenshot — a pixel grid the vision model must interpret.
This module examines pixel-based grounding: how it works, where it breaks, and when it's your only option.
A captures a screenshot, encodes it as a vision-model input, and asks the model to emit a — an (x, y) pair plus action type.
The model grounds intent to pixels by identifying a visual target and estimating its center in image coordinates.
After each action, the loop must solve the problem: the UI may animate or load before the next screenshot is meaningful. The agent must wait for visual stability before capturing the next .
Settle-wait is typically a fixed sleep, a pixel-diff threshold, or both — each carries its own latency vs. correctness tradeoff.
Drag to see how DPI scaling shifts the effective pixel position of a UI target trained at 96 DPI. A model that learned 'Upload button ≈ (420, 310)' at 96 DPI will miss by increasing margins as DPI rises.
These are the three failure modes that matter in production — each has a distinct symptom that distinguishes it from the others.
The agent's task: open a legacy desktop app, locate the "Attach File" button, click it, navigate the OS file picker, and confirm the upload — all via screenshot grounding.
The third failure produces a false success signal — the most dangerous outcome, because downstream steps proceed on a lie.
import time def run_pixel_loop(agent, desktop): while True: screenshot = desktop.capture() # raw PNG bytes action = agent.next_action(screenshot) # returns {type, x, y, text?} if action["type"] == "done": break desktop.execute(action) # delivers OS-level input event time.sleep(0.2) # fixed settle-wait: 200 ms
desktop.capture()agent.next_action(screenshot)desktop.execute(action)time.sleep(0.2)This is the minimal pixel loop: capture → reason → act → sleep. The fixed 200 ms settle-wait is the critical flaw — it assumes all UI transitions complete within that window.
Predict the failure before reading Stage 2, which replaces the fixed sleep with a pixel-diff guard.
At 200 ms the picker is still mid-animation or not yet visible. The agent sees the pre-click screen, infers the click had no effect, and re-emits the same 'click Attach File' action — opening a second picker on top of the first. The fixed sleep is shorter than the UI's actual settle time, so the loop acts on a stale frame.
import numpy as np SCALE = desktop.dpi() / 96.0 # e.g. 2.0 on HiDPI def wait_for_settle(desktop, threshold=0.005, timeout=3.0): prev = desktop.capture() deadline = time.time() + timeout while time.time() < deadline: time.sleep(0.05) curr = desktop.capture() diff = np.mean(np.abs(curr.astype(float) - prev.astype(float))) / 255 if diff < threshold: return curr # stable frame prev = curr return desktop.capture() # timeout: return best available def scale_action(action): # TODO: apply SCALE to action["x"] and action["y"] before executing
desktop.dpi() / 96.0np.mean(np.abs(...))threshold=0.005return desktop.capture()Two mitigations in one stage: pixel-diff settle-wait replaces the fixed sleep, and a DPI scale factor corrects coordinate drift before the action is delivered.
The completion task is the DPI correction — the crux of the coordinate-drift failure mode. Fill in the TODO before revealing the answer.
action["x"] = int(action["x"] SCALE); action["y"] = int(action["y"] SCALE). Both x and y must be scaled because desktop.execute() delivers events in physical pixels, but the vision model emits coordinates in the logical (96-DPI) space of the screenshot it was given. Without this correction, every click drifts by the scale factor — silently, with no error raised. The int() cast is required because OS input APIs expect integer pixel positions.
Examine how Playwright-style agents expose page state (accessibility snapshots, network events, console logs) as structured observations, and how locator-based actions eliminate coordinate fragility — continuing the file-upload scenario now running inside a web app. Cover the async-settle problem, iframe isolation, and shadow-DOM gaps.
How Playwright-style agents expose structured browser state as observations and use locator-based actions to drive a web UI without coordinate fragility.
Why this matters: Gives you the tools to build browser agents that survive UI changes — and the judgment to know when a deterministic script is the better call.
Decision this forces: When does structured browser-state observation justify the added latency over a deterministic script?
Answer: gives the agent a rasterised frame — it has no signal beyond pixel change to know whether a spinner is still running or a panel has fully mounted. The agent must heuristically delay or re-screenshot, with no structural confirmation. That fragility is exactly what Playwright-style are designed to replace.
This module asks: when the app has a DOM, does structured observation actually solve the problem — and what new failure modes does it introduce?
A raw HTML dump of a modern SPA can exceed 200 KB of nested divs, inline styles, and script tags — most invisible to the user and useless to an LLM. An serialises only the browser's accessibility tree: labelled roles, names, states, and relationships. This typically uses 5–15% of the raw HTML size.
That compression matters for two reasons: it fits more page state into the context window (the model's working memory), and it removes token-level noise that causes hallucination of selectors or misidentification of controls.
The tradeoff: the accessibility tree reflects what the browser's accessibility layer exposes, not what is visually present. Custom components built without ARIA semantics — common in design-system-heavy apps — appear as generic containers or vanish entirely.
The file-upload web app renders a drag-and-drop zone and a hidden <input type="file">. A pixel agent clicked (412, 308) — the visual centre of the drop zone. After a UI refresh, the zone shifted 40 px down and the click landed on a tooltip, silently doing nothing.
A -based agent targets the control by its and accessible name: role=button, name='Upload files'. The locator resolves at action time, so layout changes are irrelevant.
But the file input is hidden — it has no accessible name and no role in the snapshot. The agent must fall back to a CSS selector or a JavaScript handle, which reintroduces fragility. This is the boundary where locator stability ends and DOM-level targeting begins.
The structural advantage holds only as far as the accessibility tree reaches. Beyond that boundary, the agent is back to heuristics.
Playwright's locator.click() auto-waits for the element to be visible, enabled, and stable — but "stable" means pixel-stable for 500 ms, not "all async data has loaded." A file list that populates via a debounced fetch will appear stable before it is complete.
Iframes introduce a frame-context boundary: a locator scoped to the top frame cannot resolve elements inside a child frame. The agent must explicitly switch context to the frame's FrameLocator before targeting controls inside it. Cross-origin iframes block this entirely.
Shadow DOM compounds the problem: elements inside a shadow root are invisible to standard CSS selectors and absent from the accessibility snapshot unless the browser's accessibility implementation pierces the shadow boundary. Behaviour varies by browser and component library.
pierce: selector engine or expose the element via a test ID attribute.async def run_agent_step(page, goal: str, history: list) -> str: snapshot = await page.accessibility.snapshot() # structured observation network_log = drain_network_events(page) # recent XHR/fetch events observation = {"a11y": snapshot, "network": network_log} action = llm_decide(goal, history, observation) # model call history.append({"obs": observation, "act": action}) if action["type"] == "click": # TODO: resolve the locator and execute the click, then assert state changed pass return await page.accessibility.snapshot() # post-action observation
page.accessibility.snapshot()drain_network_events(page)llm_decide(goal, history, observation)page.get_by_role(role, name=...)This fragment shows the single-step skeleton of an LLM-driven — the TODO is the crux: you must resolve the locator from action's target descriptor, execute the click, and assert that the post-action snapshot differs from the pre-action one.
Stop — attempt the TODO before revealing. Hints: (1) the action dict carries a role and name from the snapshot; (2) a no-delta snapshot should raise, not silently continue.
Changed lines (the crux):
locator = page.get_by_role(action['role'], name=action['name'])
await locator.click()
# assert state changed — without this, a silent no-op looks like success
post = await page.accessibility.snapshot()
assert post != snapshot, 'No state change — possible shadow-DOM or async-settle miss'
Skipping the assertion means a no-op click propagates as a successful step; the agent loops indefinitely on the same state, burning tokens and never completing the upload.
Click a query to find the nearest observation strategy. Axes are approximate and relative — use them to reason about tradeoffs, not as benchmarks.
Three failure patterns account for most production breakages — each with a distinct signature:
Error: locator.click: Target closed or strict mode violation. Fix: re-query the locator from the snapshot taken after the navigation event, never cache locator handles across observations.Examine how DOM-grounded agents serialize the live document tree into the context window, target elements by CSS selector or ARIA role, and handle the stale-element and SPA re-render problems — extending the running scenario to a React SPA where the DOM mutates after every interaction. Revisit the grounding concept from Module 1 to contrast DOM serialization with screenshot encoding.
How DOM-based agents serialize the live document tree, target elements by selector or ARIA role, and handle stale references after SPA re-renders.
Why this matters: If your agent interacts with a modern web app, understanding DOM serialization strategies and the stale-element failure mode is what separates a loop that silently misfires from one that reliably acts on the right node.
A -style resolves target elements at action-time using CSS selectors or ARIA roles. This sidesteps pixel-drift that breaks during layout shifts. On static pages, this works cleanly. But React SPAs mutate the DOM after every interaction. An element a locator resolved moments ago may no longer exist in the tree. That's the problem this module addresses.
A must serialize the live document tree into the context window before the model can reason about it. Three strategies exist, each with different context-size and fidelity tradeoffs.
<div> markup without ARIA annotations.The ARIA snapshot aligns with the Playwright exposes. But it is generated from the live DOM, not the browser's accessibility tree. This subtle difference matters when ARIA attributes are set dynamically by JavaScript.
Each point is a serialization strategy. Click a query to see which strategy best fits that constraint. Axes: x = context tokens (lower = cheaper), y = semantic fidelity (higher = more complete).
A reference does not throw a "not found" exception in most DOM automation runtimes. If the selector still matches a node — even a different, newly mounted one — the action fires on the wrong target. The agent receives no error signal. It sees a successful action and moves on.
This mirrors a pixel agent clicking the right coordinates after a layout shift. The action completes. The looks plausible. The error propagates silently into downstream steps. The failure is detectable only by comparing post-action DOM state against expected outcomes — not by inspecting the action itself.
The running scenario: the agent uploads a file through a React SPA. After dispatching the upload action, React unmounts the file-input and mounts a progress bar. The agent's next step is to read the upload status.
Module 2 introduced for pixel loops: wait until the screen stops changing before re-observing. The DOM loop needs a stricter condition. Waiting for visual stability is insufficient. React may commit a new fiber tree to the DOM while pixels appear unchanged (e.g., a spinner already spinning).
The loop must wait for DOM mutation quiescence — no MutationObserver events for a configurable quiet window (typically 150–300 ms) — before re-serializing the tree. Re-serializing too early captures a partially committed React fiber tree. This produces an mixing old and new nodes.
aria-valuenow attribute every 100 ms during upload. Does that keep the MutationObserver firing indefinitely, blocking re-observation? What condition should the loop add to handle this?def observe(page) -> str: wait_for_dom_quiescence(page, quiet_ms=200, timeout_ms=3000) return serialize_aria_snapshot(page) # role/name/value only def act(page, selector: str, action: str, value: str = ""): wait_for_dom_quiescence(page, quiet_ms=200, timeout_ms=3000) node = page.query_selector(selector) # re-resolved after quiescence if node is None: raise StaleElementError(f"{selector} not found post-render") dispatch(node, action, value) def agent_loop(page, goal: str): obs = observe(page) while True: action, selector, value = model_step(goal, obs) act(page, selector, action, value) obs = observe(page) # re-serialize after quiescence
wait_for_dom_quiescence(page, quiet_ms, timeout_ms)serialize_aria_snapshot(page)page.query_selector(selector)StaleElementErrorThis loop re-resolves the selector inside act() after waiting for DOM quiescence, so the reference is always fresh at dispatch time. The key non-obvious point: observe() and act() both call the quiescence guard independently — an action can trigger a re-render that must settle before the next observation.
wait_for_dom_quiescence never sees a 200 ms quiet window, so it hits the 3000 ms timeout and raises. Fix: add an ignore_attrs=['aria-valuenow', 'aria-valuetext'] parameter so attribute-only mutations on progress elements don't reset the quiet timer — only structural DOM changes (node additions/removals) count toward quiescence.
Module 1 introduced as the mechanism mapping model output to concrete UI targets. encodes pixels into vision model input; the model emits (x, y) coordinates. DOM encodes semantic structure; the model emits a selector or ARIA role path.
The critical difference is what survives a re-render. Pixel coordinates are invalidated by any layout change. A selector resolved post-quiescence is always structurally valid at dispatch time. But DOM serialization is blind to purely visual state. A button styled as disabled via CSS without a disabled attribute or aria-disabled is invisible to an ARIA snapshot.
Context cost also diverges. A 1080p screenshot encodes to roughly 1 000–2 000 vision tokens regardless of page complexity. A full DOM tree for a complex SPA can exceed 50 k text tokens. The ARIA snapshot is the only practical option for long-horizon tasks.
Three failure patterns to watch for in production DOM loops:
When verifying AI-generated DOM loop code, check: does it re-resolve selectors after quiescence (not before), does it distinguish structural from attribute mutations, and does it raise on a no-match rather than silently no-op?
The next module compares pixel, browser, and DOM loops head-to-head. Grounding fidelity, action precision, latency, fragility surface, and context cost are all weighed. You can then pick the right loop for the file-upload scenario and beyond.
Compare pixel, browser, and DOM loops across five dimensions — grounding fidelity, action precision, latency, fragility surface, and context cost — using the file-upload scenario run across all three to surface non-obvious tradeoffs. Pose a hybrid-loop design problem as a completion exercise.
A head-to-head comparison of pixel, browser, and DOM action loops across five dimensions — grounding fidelity, action precision, latency, fragility surface, and context cost — with a hybrid-loop design exercise.
Why this matters: Choosing the wrong loop for your target surface is the single most common source of brittle agents; this module gives you the decision framework to get it right before you build.
Decision this forces: Which loop type — or hybrid — fits a given target surface, reliability requirement, and latency budget?
Answer: the agent holds a reference. The CSS selector or node it resolved before the re-render no longer maps to a live DOM node. So the next action throws, or silently targets nothing.
That fragility is the sharpest edge of the . Now the question is: across all three loops, which breaks first under which conditions, and how do you pick the right one before you build?
Every loop trades off across five dimensions: grounding fidelity, action precision, latency, fragility surface, and context cost. No loop wins on all five — the right choice depends on which two or three matter most for your target surface.
The file-upload scenario exposes each loop's non-obvious tradeoff. The same task runs on three different surfaces.
The agent screenshots the screen. It asks the vision model to locate the "Upload" button. It emits an (x, y) click, then waits for the heuristic to confirm the dialog opened. If the OS scales the window at 125% DPI, the coordinate is off by ~15 px. The click lands on the border. The dialog never opens, and the agent retries indefinitely. There is no error. The loop just stalls.
The -style agent resolves the file input via its ARIA label ("Attach file"). It calls set_input_files(), and reads the to confirm the filename appeared. This survives a CSS redesign. It breaks only if the label is renamed or the input is replaced with a custom drag-drop widget that has no ARIA role.
The agent serializes the DOM. It targets [data-testid='file-input'], and dispatches a change event. After the upload, React re-renders the progress bar. The original node is unmounted. If the agent tries to read the progress percentage from the stale reference, it gets null silently, not an exception. The loop reports success while the file is still uploading.
def observe(page): snapshot = page.accessibility_snapshot() # browser loop path if snapshot and snapshot.has_interactive_nodes(): return {"mode": "browser", "obs": snapshot} screenshot = page.screenshot() # pixel fallback return {"mode": "pixel", "obs": screenshot} def act(observation, action_spec): if observation["mode"] == "browser": return locator_action(observation["obs"], action_spec) return coordinate_action(observation["mode"], action_spec) # TODO
page.accessibility_snapshot()snapshot.has_interactive_nodes()locator_action(obs, spec)coordinate_action(obs, spec)This hybrid loop tries the first and falls back to only when the tree is sparse — the correct pattern for a surface that mixes a rich web app with occasional native dialogs.
The deliberate bug on the last line is the kind of copy-paste error that passes static analysis but silently corrupts every pixel-path action.
observation["mode"] is passed instead of observation["obs"] — it sends the string "pixel" to the vision model instead of the screenshot bytes. The vision model receives a nonsensical input and either raises a type error or hallucinates coordinates, causing misclicks with no obvious traceback pointing to this line. Fix: coordinate_action(observation["obs"], action_spec).
| Option | Grounding fidelity | Action precision | Fragility surface | When to choose | Cost | Complexity |
|---|---|---|---|---|---|---|
| Pixel / Screenshot loop | Low — pixel grid; no semantic labels | Low — (x,y) drifts with layout | Resolution, zoom, theme, animation timing | Native desktop apps, legacy UIs, or any surface with no accessible DOM — where structured observation is simply unavailable. | Highest — screenshot capture + vision inference on every step | High — requires vision model, coordinate calibration, settle-wait tuning |
| Browser automation loop (Playwright-style) | Good — accessibility snapshot captures roles and labels | Good — locator targets named controls | Label renames, ARIA attribute churn | Standard web apps where the accessibility tree is populated and locators are stable — the default choice for most web automation. | Low — no vision model; structured observations are cheap tokens | Medium — locator strategy requires care; accessibility tree must be populated |
| DOM-based loop | Highest — full semantic tree with attributes | Highest — CSS selector or ARIA role | Framework upgrades, selector churn, SPA re-renders | React/Vue SPAs where you control the codebase and can instrument stable selectors or ARIA roles — maximises semantic richness. | Medium-high — large DOMs inflate context tokens significantly | High — serialization pipeline, stale-element handling, context-window budget management |
Click a UI-change scenario to see which loop it hits hardest. Points closer together share similar risk profiles. X = grounding richness (higher = richer semantic signal); Y = fragility under UI change (higher = breaks more easily).
Each loop has a characteristic failure signature — knowing it tells you which degradation to instrument for.
After a SPA re-render, a resolved node is unmounted. Reading its property returns null rather than throwing, so the agent records a successful observation and advances the task. The file-upload scenario above is the canonical example: the loop reports completion while the upload is still in flight.Examine irreversibility risk, permission scoping, confirmation gates, and human-in-the-loop insertion points that apply across all three loop types — using the file-upload scenario extended to a destructive action (bulk delete) to stress-test each safety layer. Cover prompt-injection via page content, action-replay attacks, and the over-broad-permission failure mode.
A safety-constraint framework that classifies agent actions by reversibility, scopes permissions to limit blast radius, and inserts human confirmation gates at the right points across all three loop types.
Why this matters: Every browser or computer-use agent you deploy can take destructive, irreversible actions — this module gives you the audit checklist and the structural patterns to prevent unrecoverable mistakes before they reach production.
The five were: grounding fidelity, action precision, latency, fragility surface, and context cost. Each dimension exposed a different blast-radius profile. A pixel agent that misclicks a coordinate can trigger any visible button. A DOM agent scoped to a single ARIA role can only reach elements it can name.
That blast-radius difference is exactly what this module is about. Now that you've chosen your loop, how do you constrain what it can destroy?
Every action an agent can take falls into one of three classes, and the class determines the minimum confirmation overhead you're allowed to skip.
The non-obvious trap: an action that is reversible in isolation becomes irreversible at scale.
Bulk-deleting 10,000 files via a loop of individually 'recoverable' moves can exhaust the trash quota, making the aggregate irreversible even though each step was not.
The file-upload agent from earlier modules has been extended. It now accepts a natural-language instruction to 'clean up the project folder' and has write access to the storage API.
Walk through each safety layer and identify where it holds and where it fails.
of 'storage:read,write' on the entire bucket. It can enumerate, read, overwrite, and delete any file.
A tighter scope — 'storage:write' scoped to '/uploads/staging/*' only — would have contained the blast radius to the staging prefix.
The delete tool has no confirmation gate. The agent classifies 'clean up' as a recoverable move-to-trash. But the storage API performs a hard delete, which mismatches the agent's mental model and the API's actual behavior.
The fix: the tool wrapper must inspect the API's own documentation or response schema to determine whether a delete is soft or hard. gate whenever the answer is ambiguous.
Even with a gate on individual deletes, the agent can loop 10,000 times and approve each one autonomously if the gate logic only checks per-action count.
A session-level counter that triggers a human review after N destructive actions in one run closes this gap. The gate must be stateful, not stateless.
surface. fed to the model.
A malicious page can embed hidden text. It can use white-on-white text, zero-font-size text, or off-screen elements. The accessibility snapshot or DOM serialization may faithfully include it.
that prevents the agent from acting outside its declared task domain. Even a successful injection then cannot reach unrelated tools. (2) a separate, non-injectable validation layer that re-checks the action plan against the original user intent before execution. The model that plans is not the model that approves.
Drag to see how widening the agent's permission scope shifts the blast radius and the required confirmation overhead.
These are the three failure modes most likely to survive code review. Each looks like a performance optimization until it causes an incident.
The agent loop has no session-level action budget or destructive-action counter. Observable outcome: the agent completes a 'clean up' task by deleting 40,000 files across all prefixes before any human sees the plan.
Fix: enforce a per-session ceiling on irreversible actions, such as max_destructive=10. Pause for human review when the ceiling is reached.
The agent reads the UI state immediately after an action, before the interface has updated. still shows the pre-action state. The agent concludes the action failed and retries, doubling the destructive effect.
problem from Module 2, now with safety consequences. A retry loop on a hard-delete is not recoverable.
. reference still resolves but points to a different record after a sort or filter.
Observable outcome: the agent deletes the wrong file with no error. The locator matched a different row after the table re-rendered. The fix is to re-fetch and re-validate the target element's identity, such as by data-id attribute, immediately before any destructive action.
def execute_action(action, session_state, user_intent): risk = classify_reversibility(action) # → 'reversible'|'recoverable'|'irreversible' if risk == "irreversible": approved = request_human_approval(action, user_intent) if not approved: return {"status": "aborted", "reason": "human_rejected"} if session_state["destructive_count"] >= MAX_DESTRUCTIVE_PER_SESSION: raise SafetyLimitError("Session destructive ceiling reached") result = run_action(action) # actual tool call session_state["destructive_count"] += (1 if risk != "reversible" else 0) return result
classify_reversibility(action)request_human_approval(action, user_intent)session_state["destructive_count"]SafetyLimitErrorThis wrapper enforces both the per-action reversibility gate and the session-level ceiling in one place — the two controls that the bulk-delete scenario showed are independently necessary.
Notice that 'recoverable' actions increment the counter but don't require human approval: this is the deliberate tradeoff between autonomy and safety for non-permanent operations.
On the 10th iteration, destructive_count reaches 10 before run_action is called, so SafetyLimitError is raised and the loop halts mid-batch. The user sees an abrupt stop with 9 files moved and 491 untouched — a partial operation. The key insight: the ceiling is checked BEFORE execution, so the 10th action never runs. If the ceiling were checked AFTER, one extra destructive action would always slip through. Changed lines vs. the naive version: the >= check before run_action, and the conditional increment that skips reversible actions.
Before you hand any agent loop to a production environment, run it against these four questions. Each maps to a failure mode this module covered.
You've now traced the full arc. It runs from the shared observe-reason-act skeleton in Module 1, through the three loop variants and their tradeoffs, to the safety constraints that make any of them deployable.
The capstone challenge asks you to design and audit a complete agent for a novel scenario. Every concept from all six modules is in play. The safety layer you've just built is the last line of defense.
Before reviewing the summary: reconstruct from memory the three loop types, the single dimension on which each is most fragile, and the one safety constraint that applies to all three regardless of grounding strategy. Write it out, then check.
Apply what you learned to Browser and Computer-use Agents.
A ReAct-style agent and a plan-then-act agent are both given the task of booking a flight across 6 steps. Midway through, the airline site shows an unexpected "seats unavailable" banner. Which agent handles this more gracefully, and why?
ReAct-style agents reason inline at every step, so an unexpected UI state triggers an immediate reasoning update before the next action. Plan-then-act agents commit to a plan upfront; mid-task surprises require replanning, which is a heavier recovery path. The two architectures are not equivalent — plan-then-act trades adaptability for efficiency on predictable, stable tasks. The claim that upfront planning covers all branches is false for open-ended web environments.
An agent uses screenshot grounding to click a "Submit" button. The user's OS display scaling is changed from 100% to 150% between runs. What is the most direct cause of the resulting action failure?
Coordinate-targeted actions are anchored to absolute pixel positions. When DPI scaling changes, the logical-to-physical pixel mapping shifts, so a stored coordinate no longer points at the intended element — this is the core fragility of pixel-based grounding. OCR error is a separate failure mode unrelated to scaling. Stale-frame action is about timing, not resolution. Viewport shrinkage is not a direct consequence of DPI scaling in the way described.
Consider this Playwright-style agent snippet:
await page.locator('button[data-id="confirm"]').click();
await page.locator('#result-message').textContent();
The second line intermittently returns an empty string on a React SPA. What is the most likely cause and the correct remediation?
On a SPA, clicking a button often triggers an async re-render. Reading the result element immediately after the click races against React committing new DOM content — the element exists but is empty until the update lands. The fix is a settle-wait that blocks until the expected state is present. Switching to XPath does not address timing. Playwright works fine with React-rendered text. Accessibility snapshots are a separate observation mechanism and not what textContent() uses.
You are auditing a browser agent loop and find these three behaviors: (1) it submits a purchase order without any confirmation gate, (2) it acts on a DOM snapshot taken before a SPA route transition completes, and (3) it reads page content injected by a third-party ad iframe and passes it directly into the agent's system prompt. Name the safety anti-pattern each behavior represents.
The three classic safety anti-patterns for agent loops are unbounded autonomy (irreversible actions without gates), stale-observation action (acting on outdated state), and prompt injection via untrusted page content. Each maps directly to one of the behaviors described. Recognizing all three in a real audit requires knowing both the action-reversibility classification and the unique injection surface that browser agents expose compared to API-only agents.
A team needs to automate a task that spans a native desktop PDF viewer (no DOM, no browser) and a web form in Chrome. Which loop architecture fits best, and what drives the choice for the PDF viewer portion specifically?
Pixel-based control is the only viable option when the target surface exposes no DOM, accessibility tree, or browser API — a native desktop PDF viewer is exactly this case. The web form in Chrome has structured browser state available, so browser or DOM grounding is preferable there for reliability. DOM serialization via accessibility APIs does not extend to arbitrary native apps in the way described. Playwright targets browser contexts, not native OS windows. The last option inverts the loop choices without justification.