feat(host-agent): persist UI-tree evidence and add overlay/action visualization
Tests / Test passed: 863

Fixes issue 3: the host-agent console showed OCR results but never real
UI-tree data, because _ui_tree_nodes() checked for a get_ui_tree/ui_tree
tool action that has never existed anywhere in the codebase.

- storage/timeline.py: add a ui_tree_results field to TimelineRecord and
  Timeline.append(), mirroring the existing ocr_results field.
- runtime/task.py: _append_timeline() now extracts scene.elements with
  source == "ui" into ui_tree_results (scene_builder.build_scene() already
  preserved these; they were just never persisted).
- host_agent/web/app.py: _ui_tree_nodes() reads the new field directly
  instead of the dead tool-action check. New _overlay_payload() exposes
  each step's scene dimensions and fused element list for client-side
  rendering.
- task_detail.html: adds a toggle to overlay OCR (orange) and UI-tree
  (blue) bounding boxes on the before-action screenshot, plus a visual
  marker for the actually executed action (tap circle, or an animated
  swipe path) using an SVG viewBox so no manual coordinate-scaling JS is
  needed. Legacy/incomplete records degrade to no overlay, never an error.

Also corrects openspec/specs/runtime-task-evidence and
host-agent-console-task-pages, which had encoded the same nonexistent-tool
assumption, via the new host-agent-console-visual-evidence change.

600 tests passing; ruff/compileall/openspec validate all clean.
This commit is contained in:
2026-07-15 14:39:28 +08:00
parent 367fd0d412
commit 8162509158
12 changed files with 767 additions and 23 deletions
+66
View File
@@ -162,3 +162,69 @@ def test_task_runner_persists_action_evidence_and_raw_ocr(tmp_path) -> None:
assert Path(record["after_screenshot_path"]).read_bytes() == b"after-action"
assert record["tool_call"]["description"] == "tap search"
assert record["ocr_results"][0]["text"] == "Search"
def test_task_runner_persists_ui_tree_elements_from_fused_scene(tmp_path) -> None:
scene = Scene(
width=10,
height=20,
elements=[
SceneElement(
id="ui-000",
type="button",
text="Search",
bounds=Bounds(1, 2, 3, 4),
source="ui",
),
SceneElement(
id="ocr-000",
type="text",
text="Unrelated label",
bounds=Bounds(5, 6, 3, 4),
source="ocr",
),
],
)
timeline = Timeline(ArtifactStore(tmp_path / "history"))
runner = TaskRunner(
planner=ScriptedPlanner(
[PlannedStep(action="tap", description="tap search", args={})]
),
executor=Executor(
tools={"tap": lambda **kwargs: {"ok": True}},
config=ExecutorConfig(max_retries=1, backoff_seconds=0),
),
timeline=timeline,
config=TaskRunnerConfig(max_steps=2),
observer=lambda device_id: scene,
screenshot_provider=lambda device_id: PNG_10X20,
)
result = runner.run(Task(goal="tap search", device_id="iphone-1"))
record = timeline.read(result.id)[0]
assert [element["text"] for element in record["ui_tree_results"]] == ["Search"]
assert [element["text"] for element in record["ocr_results"]] == ["Unrelated label"]
def test_task_runner_persists_empty_ui_tree_results_when_scene_has_no_ui_elements(
tmp_path,
) -> None:
scene = Scene(width=10, height=20, elements=[])
timeline = Timeline(ArtifactStore(tmp_path / "history"))
runner = TaskRunner(
planner=ScriptedPlanner(
[PlannedStep(action="tap", description="tap search", args={})]
),
executor=Executor(
tools={"tap": lambda **kwargs: {"ok": True}},
config=ExecutorConfig(max_retries=1, backoff_seconds=0),
),
timeline=timeline,
config=TaskRunnerConfig(max_steps=2),
observer=lambda device_id: scene,
screenshot_provider=lambda device_id: PNG_10X20,
)
result = runner.run(Task(goal="tap search", device_id="iphone-1"))
record = timeline.read(result.id)[0]
assert record["ui_tree_results"] == []