agentic-mobile-control/openspec/changes/host-agent-console-visual-evidence/design.md

Context

Every planning step already computes a fused Scene via perception/scene_builder.py::build_scene(). Confirmed by reading the fusion logic: every UI-tree element passed in as ui_elements survives into Scene.elements with source == "ui" unchanged (whether or not it matched an OCR box — matching only overwrites text/confidence, never source or drops the element). Only OCR elements that never matched a UI-tree element are additionally kept, separately, in Scene.ocr_elements.

runtime/task.py::_append_timeline() today only ever extracts source == "ocr" elements (via scene.ocr_results_to_dict(), with a fallback scan of scene.elements) into the Timeline. No equivalent extraction of source == "ui" elements exists, and TimelineRecord has no field to hold them even if extracted.

host_agent/web/app.py::_ui_tree_nodes() was written against a different, incorrect assumption: that UI-tree data only exists when a step's tool call is get_ui_tree/ui_tree. That action has never existed in runtime/tool_specs.py::ALL_TOOL_SPECS (tap, swipe, input_text, launch_app, terminate_app, finish_task only) — so _ui_tree_nodes() returns [] for every real task. task_detail.html already has working Jinja2 markup for step.ui_tree_nodes (a <details> block, same shape as the OCR one), so once the data is wired through it needs no template rewrite.

tap/swipe tool arguments (runtime/tool_specs.py) are specified in "Scene pixel coordinates" — the same coordinate space as SceneElement.bounds and Scene.width/Scene.height. This is what makes drawing both the perception boxes and the action-effect marker from the same coordinate space consistent.

Goals / Non-Goals

Goals:

• Make UI-tree elements flow from the fused Scene into the Timeline and render as a list on the task-detail page, replacing dead code.
• Let an operator toggle an overlay of OCR/UI-tree bounding boxes directly on the screenshot that scene data was actually captured from.
• Visualize the actual spatial effect of tap/swipe actions on that same screenshot.
• Do this without any new backend endpoint or additional persisted fields beyond one new Timeline field (ui_tree_results), since bounds/text/tool args are already computed and already serialized to the page today.

Non-Goals:

• Not changing perception/scene_builder.py's fusion algorithm.
• Not adding overlay/animation to the after-screenshot — the persisted Scene and its bounds describe the state the action was planned against (the before-screenshot), not the resulting state. Overlaying boxes on the after-image would misleadingly imply they describe post-action element positions.
• Not persisting rendered overlay images; overlay/animation are computed client-side from data already in the page.
• Not adding overlay/animation for non-spatial actions (input_text, launch_app, terminate_app, finish_task).

Decisions

D1: UI-tree evidence sourced from Scene.elements where source == "ui", not a new Scene field

Decision: _append_timeline() computes ui_tree_results = [element.to_dict() for element in scene.elements if element.source == "ui"] and passes it to Timeline.append(..., ui_tree_results=ui_tree_results). TimelineRecord gains ui_tree_results: list[dict[str, Any]] = field(default_factory=list), mirroring the existing ocr_results field exactly.

Why not add a raw ui_elements field to Scene: scene_builder.build_scene() already preserves every UI-tree element in Scene.elements untouched aside from an OCR-provided text/confidence merge; filtering by source at the Timeline layer needs no change to the perception layer, and matches the existing pattern for how ocr_results gets a fallback scan of scene.elements by source.

Why not reuse the old get_ui_tree/ui_tree tool-name check: There is no such tool; it was never callable. Removing the dead branch entirely rather than keeping it alongside the new path avoids two divergent, only one of which is reachable, code paths for the same concept.

D2: host_agent/web/app.py::_ui_tree_nodes() reads the new field directly

Decision: _ui_tree_nodes(record) becomes a direct, unconditional read of record.get("ui_tree_results", []) filtered to dicts — the same shape as the existing _ocr_results(record). The tool-name filter is removed.

Why: task_detail.html already renders step.ui_tree_nodes as a collapsible list identical in structure to step.ocr_results; no template change is needed once the data source is correct.

D3: Overlay is rendered entirely client-side from data already in the page

Decision: The task-detail page already embeds each step's OCR/UI-tree results and screenshot as inline data (base64 image src, and the per-step context dict rendered into the template). Add a <template> or inline JSON blob per step with {elements: [...], scene: {width, height}} already available from _timeline_step_context(), and a toggle checkbox (one global control, applying to all steps) that adds/removes a CSS class showing/hiding an absolutely-positioned overlay <div> layer per bounding box. Box position/size are computed in JS as left = box.x / scene.width * renderedImageWidth, and analogously for top/width/height, recomputed on image load and on window resize.

Why client-side rather than server-rendered overlay images: The boxes and coordinates are already present in the page; server-side image compositing would require a new image-processing dependency and a new per-step render step for a purely presentational feature. Client-side positioning also lets the operator toggle without a page reload.

Why one global toggle rather than per-step: Operators reviewing a task want a consistent view across all steps while scanning; a per-step toggle adds UI clutter for no clear benefit. (A follow-on can add per-source toggles — OCR only vs. UI-tree only — if requested; out of scope here.)

D4: Action-effect marker/path derived from tool_call.args, drawn on the before-screenshot

Decision: For a step whose tool_call.action is "tap", render a marker (e.g. a small pulsing ring) centered at (args.x, args.y) in the same Scene-pixel-to-rendered-pixel scaling as D3. For "swipe", render a line/arrow from (args.start_x, args.start_y) to (args.end_x, args.end_y) with a small dot animated along the path on load, indicating direction. For any other action, no marker is rendered.

Why the before-screenshot and not the after-screenshot: tap/swipe coordinates are the input to the action, computed against the Scene that was captured before the action executed — i.e., the before-screenshot. Drawing them there is literally accurate ("this is where the AI aimed"); drawing them on the after-screenshot would incorrectly suggest the coordinates describe the resulting frame.

Why derive from tool_call.args rather than a new persisted field: _append_timeline() already persists tool_call={"action":..., "args":...} for every step; args already contains exactly the coordinates needed. No new Timeline field is needed for this part.

D5: Legacy/incomplete records degrade to no overlay/animation, not an error

Decision: If a step's record has no scene (legacy timeline record — see runtime-task-evidence's existing legacy-screenshot handling) or the tool_call.args lacks the expected coordinate keys, the overlay toggle and action marker simply render nothing for that step; the rest of the page (screenshots, OCR/UI-tree lists, result) renders exactly as it does today.

Why: Consistent with the repo's existing degrade-gracefully pattern for legacy Timeline records (single-screenshot records, missing OCR) — a presentational enhancement must never make an old record fail to render.

Risks / Trade-offs

• Coordinate scaling drift: if the actual screenshot's pixel dimensions ever differ from scene.screen.width/height (should not normally happen — both originate from the same captured screenshot), overlay boxes and the action marker would be positioned slightly off. Accepted: this mirrors an existing assumption already relied upon by the AI planner itself (tool args are specified "in Scene pixel coordinates"). Rendered-size fallback re-measures on image load, so a slow-loading image doesn't produce zero-sized boxes.
• Overlay only on before-screenshot may surprise operators expecting it on both: accepted per D3/D4 — showing it on the after-screenshot would be actively misleading, not just unhelpful, since element positions may have moved once the action executed.
• No per-source (OCR vs. UI-tree) toggle in this iteration: operators who only want one or the other must visually distinguish by box style (e.g. color/label per source). Follow-on work can split the toggle if requested.

Migration Plan

1. Ship TimelineRecord.ui_tree_results (additive, defaults to [] — existing serialized records without the field remain readable).
2. Deploy Host Agent with the updated _append_timeline(), _ui_tree_nodes(), and task_detail.html/static JS.
3. Rollback: revert Host Agent to prior version; no data migration to reverse since the field is additive and unused by older code.

Open Questions

• None blocking implementation.