agentic-mobile-control/openspec/changes/host-agent-console-visual-evidence/specs/runtime-task-evidence/spec.md at 557c8a25baf83daf4457ec00987d5ef4ce4554b1

q792602257/agentic-mobile-control

Fork 0

Files

T

q792602257 8162509158

Tests / Test passed: 863

Details

feat(host-agent): persist UI-tree evidence and add overlay/action visualization

Fixes issue 3: the host-agent console showed OCR results but never real
UI-tree data, because _ui_tree_nodes() checked for a get_ui_tree/ui_tree
tool action that has never existed anywhere in the codebase.

- storage/timeline.py: add a ui_tree_results field to TimelineRecord and
  Timeline.append(), mirroring the existing ocr_results field.
- runtime/task.py: _append_timeline() now extracts scene.elements with
  source == "ui" into ui_tree_results (scene_builder.build_scene() already
  preserved these; they were just never persisted).
- host_agent/web/app.py: _ui_tree_nodes() reads the new field directly
  instead of the dead tool-action check. New _overlay_payload() exposes
  each step's scene dimensions and fused element list for client-side
  rendering.
- task_detail.html: adds a toggle to overlay OCR (orange) and UI-tree
  (blue) bounding boxes on the before-action screenshot, plus a visual
  marker for the actually executed action (tap circle, or an animated
  swipe path) using an SVG viewBox so no manual coordinate-scaling JS is
  needed. Legacy/incomplete records degrade to no overlay, never an error.

Also corrects openspec/specs/runtime-task-evidence and
host-agent-console-task-pages, which had encoded the same nonexistent-tool
assumption, via the new host-agent-console-visual-evidence change.

600 tests passing; ruff/compileall/openspec validate all clean.

2026-07-15 14:39:28 +08:00

1.5 KiB

Raw Blame History

MODIFIED Requirements

Requirement: Runtime task evidence retains UI-tree inspection results

The Runtime SHALL retain, for each step, the UI-tree elements present in that step's fused Scene (the elements produced by perception/scene_builder.py::build_scene() with source == "ui") as a ui_tree_results list on the Timeline record. The Host Agent task-detail UI SHALL render those elements in a structured, collapsible view. The Runtime SHALL NOT require a dedicated UI-tree tool call to capture this data, and SHALL NOT change the tool response contract or duplicate the result in a separate persistence field.

Scenario: A step's scene contains UI-tree elements

WHEN a task step's planning Scene contains one or more elements with source == "ui"
THEN the Timeline record's ui_tree_results includes each such element's type, visible text or identifier, bounds, and available confidence, and the Host Agent task-detail page displays them

Scenario: A step's scene has no UI-tree elements

WHEN a task step's planning Scene contains no elements with source == "ui" (e.g. UI-tree parsing failed and degraded to an empty list)
THEN the Timeline record's ui_tree_results is an empty list and the Host Agent task-detail page does not render an empty UI-tree section

Scenario: A legacy Timeline record has no `ui_tree_results` field

WHEN a Timeline record was persisted before this field existed
THEN the Runtime exposes it as an empty ui_tree_results list without failing to render the record

1.5 KiB Raw Blame History