runtime-task-evidence Specification

Purpose

Define the per-action evidence retained by shared Runtime Timeline records and rendered by the Host Agent execution-history console.

Requirements

Requirement: Runtime persists complete evidence for each executed action

The shared Runtime SHALL persist, for each action it attempts, a screenshot captured immediately before the executor call, the action description and arguments, the execution result, and a screenshot captured immediately after the executor call. Existing Timeline records that contain only the legacy single screenshot SHALL remain readable, with that screenshot treated as the post-action image.

Scenario: An action succeeds

WHEN the Runtime executes an action for a task
THEN its Timeline record includes distinct before and after screenshots, action detail, and execution result

Scenario: An action fails

WHEN the Runtime executor exhausts its retries for an action
THEN the Timeline record still includes any captured screenshots and the failure result before the task is marked failed

Scenario: A legacy Timeline record is read

WHEN a Timeline record has only the prior screenshot_path field
THEN the Runtime exposes it as the post-action screenshot without failing to render the record

Requirement: Runtime task evidence retains available OCR observations

The shared Runtime SHALL persist raw OCR observations associated with the scene used to plan an action when available, without adding duplicate OCR data to the LLM-facing normalized Scene payload. The Host Agent task-detail UI SHALL render available OCR text, confidence, and bounds, and SHALL render normally when no OCR result exists.

Scenario: OCR found text while planning an action

WHEN perception produced one or more OCR observations for the action's planning scene
THEN the corresponding Timeline record includes those observations and the Host Agent task-detail page displays them

Scenario: OCR was unavailable or found no text

WHEN perception yields no OCR observations
THEN the Runtime records the action evidence and the Host Agent task detail renders without an OCR result list

Requirement: Runtime task evidence retains UI-tree inspection results

The Runtime SHALL retain a UI-tree inspection result when a step invokes the existing get_ui_tree or ui_tree tool and the result contains normalized nodes. The Host Agent task-detail UI SHALL render those nodes in a structured, collapsible view while retaining the recorded JSON result. The Runtime SHALL NOT change the tool response contract or duplicate the result in a separate persistence field.

Scenario: UI-tree inspection succeeds

WHEN a task step uses get_ui_tree or ui_tree and returns one or more normalized nodes
THEN the Host Agent task-detail page displays each node's type, visible text or identifier, bounds, and available confidence

Scenario: A non-UI-tree step is displayed

WHEN a task step did not invoke a UI-tree tool
THEN the Host Agent task-detail page does not render an empty UI-tree section

3.1 KiB Raw Blame History