q792602257
8162509158
Tests / Test passed: 863
Fixes issue 3: the host-agent console showed OCR results but never real UI-tree data, because _ui_tree_nodes() checked for a get_ui_tree/ui_tree tool action that has never existed anywhere in the codebase. - storage/timeline.py: add a ui_tree_results field to TimelineRecord and Timeline.append(), mirroring the existing ocr_results field. - runtime/task.py: _append_timeline() now extracts scene.elements with source == "ui" into ui_tree_results (scene_builder.build_scene() already preserved these; they were just never persisted). - host_agent/web/app.py: _ui_tree_nodes() reads the new field directly instead of the dead tool-action check. New _overlay_payload() exposes each step's scene dimensions and fused element list for client-side rendering. - task_detail.html: adds a toggle to overlay OCR (orange) and UI-tree (blue) bounding boxes on the before-action screenshot, plus a visual marker for the actually executed action (tap circle, or an animated swipe path) using an SVG viewBox so no manual coordinate-scaling JS is needed. Legacy/incomplete records degrade to no overlay, never an error. Also corrects openspec/specs/runtime-task-evidence and host-agent-console-task-pages, which had encoded the same nonexistent-tool assumption, via the new host-agent-console-visual-evidence change. 600 tests passing; ruff/compileall/openspec validate all clean.
125 lines
5.1 KiB
Markdown
125 lines
5.1 KiB
Markdown
## MODIFIED Requirements
|
|
|
|
### Requirement: Host Agent local console exposes read-only task history with per-step detail and screenshots
|
|
|
|
The Host Agent's local console SHALL provide authenticated, read-only pages
|
|
listing recently executed local tasks and, for a selected task, its full
|
|
per-step history from the Host-local metadata store and Timeline. The task
|
|
detail SHALL show available before and after screenshots, operation details and
|
|
arguments, execution result, OCR observations, and normalized UI-tree results.
|
|
It SHALL render legacy Timeline records that only have a single screenshot as
|
|
a post-action image.
|
|
|
|
#### Scenario: Operator lists recent Host executions
|
|
|
|
- **WHEN** an authenticated operator opens the Host Agent local console's task
|
|
list page
|
|
- **THEN** it shows local executions most recent first, including terminal
|
|
tasks and any available Cloud task ID and attempt correlation
|
|
|
|
#### Scenario: Operator inspects a completed task's step history
|
|
|
|
- **WHEN** an authenticated operator opens the detail page for a completed
|
|
Host execution
|
|
- **THEN** the page shows each recorded step in order with its tool call,
|
|
result, and available before/after screenshots
|
|
|
|
#### Scenario: OCR was captured for a step
|
|
|
|
- **WHEN** the selected Timeline record contains OCR observations
|
|
- **THEN** the detail page shows each observation's text, confidence, and
|
|
bounds
|
|
|
|
#### Scenario: A step's fused scene contains UI-tree elements
|
|
|
|
- **WHEN** the selected Timeline record's `ui_tree_results` is non-empty
|
|
- **THEN** the detail page exposes a structured, collapsible node view
|
|
(type, text/identifier, bounds, confidence) while retaining the
|
|
persisted result JSON
|
|
|
|
#### Scenario: A legacy Timeline record is displayed
|
|
|
|
- **WHEN** a Timeline record has only `screenshot_path`
|
|
- **THEN** the detail page renders it as the post-action image without failing
|
|
|
|
#### Scenario: Unauthenticated request
|
|
|
|
- **WHEN** a request to the task list or task detail pages is made without a
|
|
valid Host Agent console session
|
|
- **THEN** the Host Agent rejects the request the same way it rejects
|
|
unauthenticated requests to its other console pages
|
|
|
|
## ADDED Requirements
|
|
|
|
### Requirement: Task-detail page can overlay OCR/UI-tree bounding boxes on the before-screenshot
|
|
|
|
The Host Agent local console's task-detail page SHALL provide an operator
|
|
toggle that overlays each step's OCR and UI-tree element bounding boxes
|
|
directly on that step's before-screenshot, computed client-side from the
|
|
bounds and screen dimensions already present in the rendered page, without
|
|
a new backend endpoint or additional persisted data.
|
|
|
|
#### Scenario: Operator enables the overlay toggle
|
|
|
|
- **WHEN** an authenticated operator turns on the bounding-box overlay
|
|
toggle on a task-detail page
|
|
- **THEN** every step's before-screenshot shows a box for each of that
|
|
step's OCR and UI-tree elements, positioned and sized proportionally to
|
|
the element's bounds and the scene's screen dimensions
|
|
|
|
#### Scenario: Operator disables the overlay toggle
|
|
|
|
- **WHEN** an authenticated operator turns off the bounding-box overlay
|
|
toggle
|
|
- **THEN** the before-screenshots render without any bounding-box overlay
|
|
|
|
#### Scenario: A step has no OCR or UI-tree elements
|
|
|
|
- **WHEN** the overlay toggle is on and a step's `ocr_results` and
|
|
`ui_tree_results` are both empty
|
|
- **THEN** that step's before-screenshot renders with no overlay boxes and
|
|
without error
|
|
|
|
#### Scenario: A legacy step has no scene dimensions
|
|
|
|
- **WHEN** the overlay toggle is on and a step's Timeline record predates
|
|
scene/screen-dimension persistence
|
|
- **THEN** that step's before-screenshot renders unmodified, with no
|
|
overlay boxes, without error
|
|
|
|
### Requirement: Task-detail page visualizes the executed action's spatial effect
|
|
|
|
The Host Agent local console's task-detail page SHALL render a visual
|
|
indicator of a `tap` or `swipe` step's target coordinates on that step's
|
|
before-screenshot, derived from the step's persisted `tool_call.args`. A
|
|
`tap` step SHALL show a marker at the tapped point. A `swipe` step SHALL
|
|
show a path from the start point to the end point. Steps for other actions
|
|
SHALL render no such indicator.
|
|
|
|
#### Scenario: A tap step is displayed
|
|
|
|
- **WHEN** a task step's tool call is `tap` with `x`/`y` arguments
|
|
- **THEN** the step's before-screenshot shows a marker at the point
|
|
corresponding to those coordinates
|
|
|
|
#### Scenario: A swipe step is displayed
|
|
|
|
- **WHEN** a task step's tool call is `swipe` with `start_x`/`start_y`/
|
|
`end_x`/`end_y` arguments
|
|
- **THEN** the step's before-screenshot shows a path from the start point
|
|
to the end point corresponding to those coordinates
|
|
|
|
#### Scenario: A non-spatial step is displayed
|
|
|
|
- **WHEN** a task step's tool call is `input_text`, `launch_app`,
|
|
`terminate_app`, or `finish_task`
|
|
- **THEN** the step's before-screenshot renders with no action-effect
|
|
marker or path
|
|
|
|
#### Scenario: A tap/swipe step is missing expected coordinate arguments
|
|
|
|
- **WHEN** a task step's tool call is `tap` or `swipe` but its persisted
|
|
`args` lacks the expected coordinate keys
|
|
- **THEN** the step's before-screenshot renders unmodified, with no
|
|
action-effect marker or path, without error
|