agentic-mobile-control/openspec/changes/host-agent-console-visual-evidence/specs/host-agent-console-task-pages/spec.md

MODIFIED Requirements

Requirement: Host Agent local console exposes read-only task history with per-step detail and screenshots

The Host Agent's local console SHALL provide authenticated, read-only pages listing recently executed local tasks and, for a selected task, its full per-step history from the Host-local metadata store and Timeline. The task detail SHALL show available before and after screenshots, operation details and arguments, execution result, OCR observations, and normalized UI-tree results. It SHALL render legacy Timeline records that only have a single screenshot as a post-action image.

Scenario: Operator lists recent Host executions

• WHEN an authenticated operator opens the Host Agent local console's task list page
• THEN it shows local executions most recent first, including terminal tasks and any available Cloud task ID and attempt correlation

Scenario: Operator inspects a completed task's step history

• WHEN an authenticated operator opens the detail page for a completed Host execution
• THEN the page shows each recorded step in order with its tool call, result, and available before/after screenshots

Scenario: OCR was captured for a step

• WHEN the selected Timeline record contains OCR observations
• THEN the detail page shows each observation's text, confidence, and bounds

Scenario: A step's fused scene contains UI-tree elements

• WHEN the selected Timeline record's ui_tree_results is non-empty
• THEN the detail page exposes a structured, collapsible node view (type, text/identifier, bounds, confidence) while retaining the persisted result JSON

Scenario: A legacy Timeline record is displayed

• WHEN a Timeline record has only screenshot_path
• THEN the detail page renders it as the post-action image without failing

Scenario: Unauthenticated request

• WHEN a request to the task list or task detail pages is made without a valid Host Agent console session
• THEN the Host Agent rejects the request the same way it rejects unauthenticated requests to its other console pages

ADDED Requirements

Requirement: Task-detail page can overlay OCR/UI-tree bounding boxes on the before-screenshot

The Host Agent local console's task-detail page SHALL provide an operator toggle that overlays each step's OCR and UI-tree element bounding boxes directly on that step's before-screenshot, computed client-side from the bounds and screen dimensions already present in the rendered page, without a new backend endpoint or additional persisted data.

Scenario: Operator enables the overlay toggle

• WHEN an authenticated operator turns on the bounding-box overlay toggle on a task-detail page
• THEN every step's before-screenshot shows a box for each of that step's OCR and UI-tree elements, positioned and sized proportionally to the element's bounds and the scene's screen dimensions

Scenario: Operator disables the overlay toggle

• WHEN an authenticated operator turns off the bounding-box overlay toggle
• THEN the before-screenshots render without any bounding-box overlay

Scenario: A step has no OCR or UI-tree elements

• WHEN the overlay toggle is on and a step's ocr_results and ui_tree_results are both empty
• THEN that step's before-screenshot renders with no overlay boxes and without error

Scenario: A legacy step has no scene dimensions

• WHEN the overlay toggle is on and a step's Timeline record predates scene/screen-dimension persistence
• THEN that step's before-screenshot renders unmodified, with no overlay boxes, without error

Requirement: Task-detail page visualizes the executed action's spatial effect

The Host Agent local console's task-detail page SHALL render a visual indicator of a tap or swipe step's target coordinates on that step's before-screenshot, derived from the step's persisted tool_call.args. A tap step SHALL show a marker at the tapped point. A swipe step SHALL show a path from the start point to the end point. Steps for other actions SHALL render no such indicator.

Scenario: A tap step is displayed

• WHEN a task step's tool call is tap with x/y arguments
• THEN the step's before-screenshot shows a marker at the point corresponding to those coordinates

Scenario: A swipe step is displayed

• WHEN a task step's tool call is swipe with start_x/start_y/ end_x/end_y arguments
• THEN the step's before-screenshot shows a path from the start point to the end point corresponding to those coordinates

Scenario: A non-spatial step is displayed

• WHEN a task step's tool call is input_text, launch_app, terminate_app, or finish_task
• THEN the step's before-screenshot renders with no action-effect marker or path

Scenario: A tap/swipe step is missing expected coordinate arguments

• WHEN a task step's tool call is tap or swipe but its persisted args lacks the expected coordinate keys
• THEN the step's before-screenshot renders unmodified, with no action-effect marker or path, without error