agentic-mobile-control/openspec/changes/host-agent-console-visual-evidence/tasks.md

1. Timeline — persist UI-tree elements per step

• 1.1 Add ui_tree_results: list[dict[str, Any]] = field(default_factory=list) to TimelineRecord in storage/timeline.py, mirroring ocr_results
• 1.2 Add ui_tree_results: list[dict[str, Any]] | None = None parameter to Timeline.append(), stored the same way as ocr_results
• 1.3 Add unit tests: Timeline.append() with ui_tree_results → record round-trips it; without it → record defaults to []; a record serialized before this field existed → reads as [] without error

2. Runtime — extract UI-tree elements into the Timeline

• 2.1 In runtime/task.py::_append_timeline(), compute ui_tree_results = [element.to_dict() for element in scene.elements if element.source == "ui"] and pass it to self.timeline.append(..., ui_tree_results=ui_tree_results)
• 2.2 Add unit tests: a scene with source == "ui" elements → ui_tree_results populated in the appended record; a scene with no UI elements → ui_tree_results is []; a scene with mixed ui/ocr/merged elements → only source == "ui" elements appear in ui_tree_results and only source == "ocr" (or fallback) elements appear in ocr_results

3. Host Agent console — render real UI-tree data

• 3.1 Rewrite _ui_tree_nodes() in apps/device-host-agent/host_agent/web/app.py to read record.get("ui_tree_results", []) directly (list of dicts), removing the dead tool_call.get("action") in {"get_ui_tree", "ui_tree"} check
• 3.2 Confirm _timeline_step_context() passes the corrected ui_tree_nodes through unchanged; no task_detail.html markup changes needed for the list view itself (existing step.ui_tree_nodes block already renders it)
• 3.3 Add/update unit tests for _ui_tree_nodes(): a record with ui_tree_results → returns those dicts; a record without the field (legacy) → returns []; a record with non-dict entries → those entries are filtered out

4. Overlay toggle — OCR/UI-tree bounding boxes on the before-screenshot

• 4.1 Add _overlay_payload(record) to app.py exposing each step's scene screen dimensions (width/height) and its fused element list (bounds/text/source, straight from scene.elements — already the combined OCR+UI-tree view) as step.overlay; embedded per-step as an inline <script type="application/json"> blob in task_detail.html
• 4.2 Add a single page-level toggle control (checkbox #overlay-boxes-toggle) in task_detail.html that adds/removes a show-boxes class on every step's overlay <svg>
• 4.3 Render each step's overlay as an inline <svg viewBox="0 0 {scene.width} {scene.height}"> absolutely positioned over the before-screenshot <img>; boxes are drawn as <rect> elements directly in scene-pixel coordinates, so the SVG viewBox scaling handles image-size scaling natively — no manual JS scale computation or resize listener needed (simpler than the originally planned separate static JS file with manual scaling)
• 4.4 OCR boxes (source="ocr") and UI-tree boxes (source="ui") get distinct stroke colors (orange vs. blue) via CSS classes
• 4.5 Verified: _overlay_payload() degrades to {"width": 0, "height": 0, "elements": []} for a record with no/malformed scene; the template only emits the <svg> when overlay.width/overlay.height are truthy, so legacy records render the screenshot with no overlay markup at all

5. Action-effect visualization — tap marker / swipe path

• 5.1 Inline JS in task_detail.html reads each step's already-embedded tool_call.action/args; draws an SVG <circle> marker for tap, or an SVG <line> plus an <animateMotion>-animated dot for swipe; any other action draws nothing
• 5.2 Action markers are drawn in the same <svg viewBox> as the overlay boxes (task 4.3), so no separate scaling logic was needed
• 5.3 Verified: buildActionGroup() checks isFinite() on every required coordinate before drawing; missing/non-numeric args (or no <svg> at all, for a legacy record) render nothing, no error
• 5.4 Rendered a real task-detail page (via the FastAPI test client with fabricated tap/swipe Timeline records) and inspected the generated HTML/JSON: viewBox, box coordinates, tap marker coordinates, and swipe line/path coordinates all match the source data; opened the page in a browser for visual confirmation

6. Spec corrections

• 6.1 openspec/specs/runtime-task-evidence/spec.md (via this change's spec delta): UI-tree requirement no longer references a get_ui_tree/ui_tree tool call
• 6.2 openspec/specs/host-agent-console-task-pages/spec.md (via this change's spec delta): UI-tree scenario corrected; overlay toggle and action-effect requirements added

7. End-to-end verification

• 7.1 Run full non-integration test suite (uv run --all-packages pytest -m "not integration") and confirm no regressions
• 7.2 Run ruff check and ruff format --check on all modified Python files
• 7.3 Run python -m compileall on modified packages
• 7.4 Run openspec validate --strict host-agent-console-visual-evidence and confirm all artifacts pass validation
• 7.5 Manual smoke test (requires Host Agent + Appium + a device): run a live task, open its task-detail page, confirm UI-tree elements render, the overlay toggle shows/hides boxes correctly on the before-screenshot, and the tap/swipe action-effect marker/path renders correctly