feat(host-agent): persist UI-tree evidence and add overlay/action visualization
Tests / Test passed: 863

Fixes issue 3: the host-agent console showed OCR results but never real
UI-tree data, because _ui_tree_nodes() checked for a get_ui_tree/ui_tree
tool action that has never existed anywhere in the codebase.

- storage/timeline.py: add a ui_tree_results field to TimelineRecord and
  Timeline.append(), mirroring the existing ocr_results field.
- runtime/task.py: _append_timeline() now extracts scene.elements with
  source == "ui" into ui_tree_results (scene_builder.build_scene() already
  preserved these; they were just never persisted).
- host_agent/web/app.py: _ui_tree_nodes() reads the new field directly
  instead of the dead tool-action check. New _overlay_payload() exposes
  each step's scene dimensions and fused element list for client-side
  rendering.
- task_detail.html: adds a toggle to overlay OCR (orange) and UI-tree
  (blue) bounding boxes on the before-action screenshot, plus a visual
  marker for the actually executed action (tap circle, or an animated
  swipe path) using an SVG viewBox so no manual coordinate-scaling JS is
  needed. Legacy/incomplete records degrade to no overlay, never an error.

Also corrects openspec/specs/runtime-task-evidence and
host-agent-console-task-pages, which had encoded the same nonexistent-tool
assumption, via the new host-agent-console-visual-evidence change.

600 tests passing; ruff/compileall/openspec validate all clean.
This commit is contained in:
2026-07-15 14:39:28 +08:00
parent 367fd0d412
commit 8162509158
12 changed files with 767 additions and 23 deletions
+4
View File
@@ -364,6 +364,9 @@ class TaskRunner:
for element in scene.elements
if element.source == "ocr"
]
ui_tree_results = [
element.to_dict() for element in scene.elements if element.source == "ui"
]
self.timeline.append(
task_id=task.id,
scene=scene.to_dict(),
@@ -379,6 +382,7 @@ class TaskRunner:
before_screenshot=before_screenshot,
after_screenshot=after_screenshot,
ocr_results=ocr_results,
ui_tree_results=ui_tree_results,
)
def _step_for_device(self, step: PlannedStep, device_id: str) -> PlannedStep: