feat(host-agent): persist UI-tree evidence and add overlay/action visualization
Tests / Test passed: 863
Tests / Test passed: 863
Fixes issue 3: the host-agent console showed OCR results but never real UI-tree data, because _ui_tree_nodes() checked for a get_ui_tree/ui_tree tool action that has never existed anywhere in the codebase. - storage/timeline.py: add a ui_tree_results field to TimelineRecord and Timeline.append(), mirroring the existing ocr_results field. - runtime/task.py: _append_timeline() now extracts scene.elements with source == "ui" into ui_tree_results (scene_builder.build_scene() already preserved these; they were just never persisted). - host_agent/web/app.py: _ui_tree_nodes() reads the new field directly instead of the dead tool-action check. New _overlay_payload() exposes each step's scene dimensions and fused element list for client-side rendering. - task_detail.html: adds a toggle to overlay OCR (orange) and UI-tree (blue) bounding boxes on the before-action screenshot, plus a visual marker for the actually executed action (tap circle, or an animated swipe path) using an SVG viewBox so no manual coordinate-scaling JS is needed. Legacy/incomplete records degrade to no overlay, never an error. Also corrects openspec/specs/runtime-task-evidence and host-agent-console-task-pages, which had encoded the same nonexistent-tool assumption, via the new host-agent-console-visual-evidence change. 600 tests passing; ruff/compileall/openspec validate all clean.
This commit is contained in:
@@ -364,6 +364,9 @@ class TaskRunner:
|
||||
for element in scene.elements
|
||||
if element.source == "ocr"
|
||||
]
|
||||
ui_tree_results = [
|
||||
element.to_dict() for element in scene.elements if element.source == "ui"
|
||||
]
|
||||
self.timeline.append(
|
||||
task_id=task.id,
|
||||
scene=scene.to_dict(),
|
||||
@@ -379,6 +382,7 @@ class TaskRunner:
|
||||
before_screenshot=before_screenshot,
|
||||
after_screenshot=after_screenshot,
|
||||
ocr_results=ocr_results,
|
||||
ui_tree_results=ui_tree_results,
|
||||
)
|
||||
|
||||
def _step_for_device(self, step: PlannedStep, device_id: str) -> PlannedStep:
|
||||
|
||||
Reference in New Issue
Block a user