agentic-mobile-control/openspec/changes/task-execution-progress-visibility/specs/runtime-task-evidence/spec.md

ADDED Requirements

Requirement: Runtime persists complete evidence for each executed action

The shared Runtime SHALL persist, for each action it attempts, a screenshot captured immediately before the executor call, the action description and arguments, the execution result, and a screenshot captured immediately after the executor call. Existing Timeline records that contain only the legacy single screenshot SHALL remain readable, with that screenshot treated as the post-action image.

Scenario: An action succeeds

• WHEN the Runtime executes an action for a task
• THEN its Timeline record includes distinct before and after screenshots, action detail, and execution result

Scenario: An action fails

• WHEN the Runtime executor exhausts its retries for an action
• THEN the Timeline record still includes any captured screenshots and the failure result before the task is marked failed

Scenario: A legacy Timeline record is read

• WHEN a Timeline record has only the prior screenshot_path field
• THEN the Runtime exposes it as the post-action screenshot without failing to render the record

Requirement: Runtime task evidence retains available OCR observations

The shared Runtime SHALL persist raw OCR observations associated with the scene used to plan an action when available, without adding duplicate OCR data to the LLM-facing normalized Scene payload. The Host Agent task-detail UI SHALL render available OCR text, confidence, and bounds, and SHALL render normally when no OCR result exists.

Scenario: OCR found text while planning an action

• WHEN perception produced one or more OCR observations for the action's planning scene
• THEN the corresponding Timeline record includes those observations and the Host Agent task-detail page displays them

Scenario: OCR was unavailable or found no text

• WHEN perception yields no OCR observations
• THEN the Runtime records the action evidence and the Host Agent task detail renders without an OCR result list

Requirement: Runtime task evidence retains UI-tree inspection results

The Runtime SHALL retain a UI-tree inspection result when a step invokes the existing get_ui_tree or ui_tree tool and the result contains normalized nodes. The Host Agent task-detail UI SHALL render those nodes in a structured, collapsible view while retaining the recorded JSON result. The Runtime SHALL NOT change the tool response contract or duplicate the result in a separate persistence field.

Scenario: UI-tree inspection succeeds

• WHEN a task step uses get_ui_tree or ui_tree and returns one or more normalized nodes
• THEN the Host Agent task-detail page displays each node's type, visible text or identifier, bounds, and available confidence

Scenario: A non-UI-tree step is displayed

• WHEN a task step did not invoke a UI-tree tool
• THEN the Host Agent task-detail page does not render an empty UI-tree section