q792602257
8162509158
Tests / Test passed: 863
Fixes issue 3: the host-agent console showed OCR results but never real UI-tree data, because _ui_tree_nodes() checked for a get_ui_tree/ui_tree tool action that has never existed anywhere in the codebase. - storage/timeline.py: add a ui_tree_results field to TimelineRecord and Timeline.append(), mirroring the existing ocr_results field. - runtime/task.py: _append_timeline() now extracts scene.elements with source == "ui" into ui_tree_results (scene_builder.build_scene() already preserved these; they were just never persisted). - host_agent/web/app.py: _ui_tree_nodes() reads the new field directly instead of the dead tool-action check. New _overlay_payload() exposes each step's scene dimensions and fused element list for client-side rendering. - task_detail.html: adds a toggle to overlay OCR (orange) and UI-tree (blue) bounding boxes on the before-action screenshot, plus a visual marker for the actually executed action (tap circle, or an animated swipe path) using an SVG viewBox so no manual coordinate-scaling JS is needed. Legacy/incomplete records degrade to no overlay, never an error. Also corrects openspec/specs/runtime-task-evidence and host-agent-console-task-pages, which had encoded the same nonexistent-tool assumption, via the new host-agent-console-visual-evidence change. 600 tests passing; ruff/compileall/openspec validate all clean.
35 lines
1.5 KiB
Markdown
35 lines
1.5 KiB
Markdown
## MODIFIED Requirements
|
|
|
|
### Requirement: Runtime task evidence retains UI-tree inspection results
|
|
|
|
The Runtime SHALL retain, for each step, the UI-tree elements present in
|
|
that step's fused `Scene` (the elements produced by
|
|
`perception/scene_builder.py::build_scene()` with `source == "ui"`) as a
|
|
`ui_tree_results` list on the Timeline record. The Host Agent task-detail
|
|
UI SHALL render those elements in a structured, collapsible view. The
|
|
Runtime SHALL NOT require a dedicated UI-tree tool call to capture this
|
|
data, and SHALL NOT change the tool response contract or duplicate the
|
|
result in a separate persistence field.
|
|
|
|
#### Scenario: A step's scene contains UI-tree elements
|
|
|
|
- **WHEN** a task step's planning `Scene` contains one or more elements
|
|
with `source == "ui"`
|
|
- **THEN** the Timeline record's `ui_tree_results` includes each such
|
|
element's type, visible text or identifier, bounds, and available
|
|
confidence, and the Host Agent task-detail page displays them
|
|
|
|
#### Scenario: A step's scene has no UI-tree elements
|
|
|
|
- **WHEN** a task step's planning `Scene` contains no elements with
|
|
`source == "ui"` (e.g. UI-tree parsing failed and degraded to an empty
|
|
list)
|
|
- **THEN** the Timeline record's `ui_tree_results` is an empty list and the
|
|
Host Agent task-detail page does not render an empty UI-tree section
|
|
|
|
#### Scenario: A legacy Timeline record has no `ui_tree_results` field
|
|
|
|
- **WHEN** a Timeline record was persisted before this field existed
|
|
- **THEN** the Runtime exposes it as an empty `ui_tree_results` list
|
|
without failing to render the record
|