feat(host-agent): persist UI-tree evidence and add overlay/action visualization
Tests / Test passed: 863
Tests / Test passed: 863
Fixes issue 3: the host-agent console showed OCR results but never real UI-tree data, because _ui_tree_nodes() checked for a get_ui_tree/ui_tree tool action that has never existed anywhere in the codebase. - storage/timeline.py: add a ui_tree_results field to TimelineRecord and Timeline.append(), mirroring the existing ocr_results field. - runtime/task.py: _append_timeline() now extracts scene.elements with source == "ui" into ui_tree_results (scene_builder.build_scene() already preserved these; they were just never persisted). - host_agent/web/app.py: _ui_tree_nodes() reads the new field directly instead of the dead tool-action check. New _overlay_payload() exposes each step's scene dimensions and fused element list for client-side rendering. - task_detail.html: adds a toggle to overlay OCR (orange) and UI-tree (blue) bounding boxes on the before-action screenshot, plus a visual marker for the actually executed action (tap circle, or an animated swipe path) using an SVG viewBox so no manual coordinate-scaling JS is needed. Legacy/incomplete records degrade to no overlay, never an error. Also corrects openspec/specs/runtime-task-evidence and host-agent-console-task-pages, which had encoded the same nonexistent-tool assumption, via the new host-agent-console-visual-evidence change. 600 tests passing; ruff/compileall/openspec validate all clean.
This commit is contained in:
+124
@@ -0,0 +1,124 @@
|
||||
## MODIFIED Requirements
|
||||
|
||||
### Requirement: Host Agent local console exposes read-only task history with per-step detail and screenshots
|
||||
|
||||
The Host Agent's local console SHALL provide authenticated, read-only pages
|
||||
listing recently executed local tasks and, for a selected task, its full
|
||||
per-step history from the Host-local metadata store and Timeline. The task
|
||||
detail SHALL show available before and after screenshots, operation details and
|
||||
arguments, execution result, OCR observations, and normalized UI-tree results.
|
||||
It SHALL render legacy Timeline records that only have a single screenshot as
|
||||
a post-action image.
|
||||
|
||||
#### Scenario: Operator lists recent Host executions
|
||||
|
||||
- **WHEN** an authenticated operator opens the Host Agent local console's task
|
||||
list page
|
||||
- **THEN** it shows local executions most recent first, including terminal
|
||||
tasks and any available Cloud task ID and attempt correlation
|
||||
|
||||
#### Scenario: Operator inspects a completed task's step history
|
||||
|
||||
- **WHEN** an authenticated operator opens the detail page for a completed
|
||||
Host execution
|
||||
- **THEN** the page shows each recorded step in order with its tool call,
|
||||
result, and available before/after screenshots
|
||||
|
||||
#### Scenario: OCR was captured for a step
|
||||
|
||||
- **WHEN** the selected Timeline record contains OCR observations
|
||||
- **THEN** the detail page shows each observation's text, confidence, and
|
||||
bounds
|
||||
|
||||
#### Scenario: A step's fused scene contains UI-tree elements
|
||||
|
||||
- **WHEN** the selected Timeline record's `ui_tree_results` is non-empty
|
||||
- **THEN** the detail page exposes a structured, collapsible node view
|
||||
(type, text/identifier, bounds, confidence) while retaining the
|
||||
persisted result JSON
|
||||
|
||||
#### Scenario: A legacy Timeline record is displayed
|
||||
|
||||
- **WHEN** a Timeline record has only `screenshot_path`
|
||||
- **THEN** the detail page renders it as the post-action image without failing
|
||||
|
||||
#### Scenario: Unauthenticated request
|
||||
|
||||
- **WHEN** a request to the task list or task detail pages is made without a
|
||||
valid Host Agent console session
|
||||
- **THEN** the Host Agent rejects the request the same way it rejects
|
||||
unauthenticated requests to its other console pages
|
||||
|
||||
## ADDED Requirements
|
||||
|
||||
### Requirement: Task-detail page can overlay OCR/UI-tree bounding boxes on the before-screenshot
|
||||
|
||||
The Host Agent local console's task-detail page SHALL provide an operator
|
||||
toggle that overlays each step's OCR and UI-tree element bounding boxes
|
||||
directly on that step's before-screenshot, computed client-side from the
|
||||
bounds and screen dimensions already present in the rendered page, without
|
||||
a new backend endpoint or additional persisted data.
|
||||
|
||||
#### Scenario: Operator enables the overlay toggle
|
||||
|
||||
- **WHEN** an authenticated operator turns on the bounding-box overlay
|
||||
toggle on a task-detail page
|
||||
- **THEN** every step's before-screenshot shows a box for each of that
|
||||
step's OCR and UI-tree elements, positioned and sized proportionally to
|
||||
the element's bounds and the scene's screen dimensions
|
||||
|
||||
#### Scenario: Operator disables the overlay toggle
|
||||
|
||||
- **WHEN** an authenticated operator turns off the bounding-box overlay
|
||||
toggle
|
||||
- **THEN** the before-screenshots render without any bounding-box overlay
|
||||
|
||||
#### Scenario: A step has no OCR or UI-tree elements
|
||||
|
||||
- **WHEN** the overlay toggle is on and a step's `ocr_results` and
|
||||
`ui_tree_results` are both empty
|
||||
- **THEN** that step's before-screenshot renders with no overlay boxes and
|
||||
without error
|
||||
|
||||
#### Scenario: A legacy step has no scene dimensions
|
||||
|
||||
- **WHEN** the overlay toggle is on and a step's Timeline record predates
|
||||
scene/screen-dimension persistence
|
||||
- **THEN** that step's before-screenshot renders unmodified, with no
|
||||
overlay boxes, without error
|
||||
|
||||
### Requirement: Task-detail page visualizes the executed action's spatial effect
|
||||
|
||||
The Host Agent local console's task-detail page SHALL render a visual
|
||||
indicator of a `tap` or `swipe` step's target coordinates on that step's
|
||||
before-screenshot, derived from the step's persisted `tool_call.args`. A
|
||||
`tap` step SHALL show a marker at the tapped point. A `swipe` step SHALL
|
||||
show a path from the start point to the end point. Steps for other actions
|
||||
SHALL render no such indicator.
|
||||
|
||||
#### Scenario: A tap step is displayed
|
||||
|
||||
- **WHEN** a task step's tool call is `tap` with `x`/`y` arguments
|
||||
- **THEN** the step's before-screenshot shows a marker at the point
|
||||
corresponding to those coordinates
|
||||
|
||||
#### Scenario: A swipe step is displayed
|
||||
|
||||
- **WHEN** a task step's tool call is `swipe` with `start_x`/`start_y`/
|
||||
`end_x`/`end_y` arguments
|
||||
- **THEN** the step's before-screenshot shows a path from the start point
|
||||
to the end point corresponding to those coordinates
|
||||
|
||||
#### Scenario: A non-spatial step is displayed
|
||||
|
||||
- **WHEN** a task step's tool call is `input_text`, `launch_app`,
|
||||
`terminate_app`, or `finish_task`
|
||||
- **THEN** the step's before-screenshot renders with no action-effect
|
||||
marker or path
|
||||
|
||||
#### Scenario: A tap/swipe step is missing expected coordinate arguments
|
||||
|
||||
- **WHEN** a task step's tool call is `tap` or `swipe` but its persisted
|
||||
`args` lacks the expected coordinate keys
|
||||
- **THEN** the step's before-screenshot renders unmodified, with no
|
||||
action-effect marker or path, without error
|
||||
+34
@@ -0,0 +1,34 @@
|
||||
## MODIFIED Requirements
|
||||
|
||||
### Requirement: Runtime task evidence retains UI-tree inspection results
|
||||
|
||||
The Runtime SHALL retain, for each step, the UI-tree elements present in
|
||||
that step's fused `Scene` (the elements produced by
|
||||
`perception/scene_builder.py::build_scene()` with `source == "ui"`) as a
|
||||
`ui_tree_results` list on the Timeline record. The Host Agent task-detail
|
||||
UI SHALL render those elements in a structured, collapsible view. The
|
||||
Runtime SHALL NOT require a dedicated UI-tree tool call to capture this
|
||||
data, and SHALL NOT change the tool response contract or duplicate the
|
||||
result in a separate persistence field.
|
||||
|
||||
#### Scenario: A step's scene contains UI-tree elements
|
||||
|
||||
- **WHEN** a task step's planning `Scene` contains one or more elements
|
||||
with `source == "ui"`
|
||||
- **THEN** the Timeline record's `ui_tree_results` includes each such
|
||||
element's type, visible text or identifier, bounds, and available
|
||||
confidence, and the Host Agent task-detail page displays them
|
||||
|
||||
#### Scenario: A step's scene has no UI-tree elements
|
||||
|
||||
- **WHEN** a task step's planning `Scene` contains no elements with
|
||||
`source == "ui"` (e.g. UI-tree parsing failed and degraded to an empty
|
||||
list)
|
||||
- **THEN** the Timeline record's `ui_tree_results` is an empty list and the
|
||||
Host Agent task-detail page does not render an empty UI-tree section
|
||||
|
||||
#### Scenario: A legacy Timeline record has no `ui_tree_results` field
|
||||
|
||||
- **WHEN** a Timeline record was persisted before this field existed
|
||||
- **THEN** the Runtime exposes it as an empty `ui_tree_results` list
|
||||
without failing to render the record
|
||||
Reference in New Issue
Block a user