2.9 KiB
ADDED Requirements
Requirement: Runtime persists complete evidence for each executed action
The Runtime SHALL persist, for each action it attempts, a screenshot captured immediately before the executor call, the action description and arguments, the execution result, and a screenshot captured immediately after the executor call. Existing timeline records that contain only the legacy single screenshot SHALL remain readable, with that screenshot treated as the post-action image.
Scenario: An action succeeds
- WHEN the Runtime executes an action for a task
- THEN its timeline record includes distinct before and after screenshots, the action detail, and the execution result
Scenario: An action fails
- WHEN the Runtime executor exhausts its retries for an action
- THEN the action's timeline record still includes any screenshots that were captured and the failure result before the task is marked failed
Scenario: A legacy timeline record is read
- WHEN a timeline record has only the prior
screenshot_pathfield - THEN the Runtime exposes it as the post-action screenshot without failing to render the record
Requirement: Runtime task evidence exposes available OCR observations
The Runtime SHALL persist raw OCR observations associated with the scene used to plan an action when available, without adding duplicate OCR data to the LLM-facing normalized Scene payload. The Runtime task-detail UI SHALL render available OCR text, confidence, and bounds, and SHALL render normally when no OCR result exists.
Scenario: OCR found text while planning an action
- WHEN perception produced one or more OCR observations for the action's planning scene
- THEN the corresponding timeline record includes those observations and the Runtime task-detail UI displays them
Scenario: OCR was unavailable or found no text
- WHEN perception yields no OCR observations
- THEN the Runtime records the action evidence and renders the task detail without an OCR result list
Requirement: Runtime task evidence renders UI-tree inspection results
When a Runtime step invokes the existing get_ui_tree or ui_tree tool and the persisted result contains normalized UI nodes, the Runtime task-detail UI SHALL render those nodes in a structured, collapsible view while retaining the recorded JSON result. The Runtime SHALL NOT change the tool's response contract or duplicate the result in a separate persistence field.
Scenario: UI-tree inspection succeeds
- WHEN a task step uses
get_ui_treeorui_treeand returns one or more normalized nodes - THEN the task-detail UI displays each node's type, visible text or identifier, bounds, and available confidence
Scenario: A non-UI-tree step is displayed
- WHEN a task step did not invoke a UI-tree tool
- THEN the task-detail UI does not render an empty UI-tree section