39 lines
2.9 KiB
Markdown
39 lines
2.9 KiB
Markdown
## ADDED Requirements
|
|
|
|
### Requirement: Runtime persists complete evidence for each executed action
|
|
The Runtime SHALL persist, for each action it attempts, a screenshot captured immediately before the executor call, the action description and arguments, the execution result, and a screenshot captured immediately after the executor call. Existing timeline records that contain only the legacy single screenshot SHALL remain readable, with that screenshot treated as the post-action image.
|
|
|
|
#### Scenario: An action succeeds
|
|
- **WHEN** the Runtime executes an action for a task
|
|
- **THEN** its timeline record includes distinct before and after screenshots, the action detail, and the execution result
|
|
|
|
#### Scenario: An action fails
|
|
- **WHEN** the Runtime executor exhausts its retries for an action
|
|
- **THEN** the action's timeline record still includes any screenshots that were captured and the failure result before the task is marked failed
|
|
|
|
#### Scenario: A legacy timeline record is read
|
|
- **WHEN** a timeline record has only the prior `screenshot_path` field
|
|
- **THEN** the Runtime exposes it as the post-action screenshot without failing to render the record
|
|
|
|
### Requirement: Runtime task evidence exposes available OCR observations
|
|
The Runtime SHALL persist raw OCR observations associated with the scene used to plan an action when available, without adding duplicate OCR data to the LLM-facing normalized Scene payload. The Runtime task-detail UI SHALL render available OCR text, confidence, and bounds, and SHALL render normally when no OCR result exists.
|
|
|
|
#### Scenario: OCR found text while planning an action
|
|
- **WHEN** perception produced one or more OCR observations for the action's planning scene
|
|
- **THEN** the corresponding timeline record includes those observations and the Runtime task-detail UI displays them
|
|
|
|
#### Scenario: OCR was unavailable or found no text
|
|
- **WHEN** perception yields no OCR observations
|
|
- **THEN** the Runtime records the action evidence and renders the task detail without an OCR result list
|
|
|
|
### Requirement: Runtime task evidence renders UI-tree inspection results
|
|
When a Runtime step invokes the existing `get_ui_tree` or `ui_tree` tool and the persisted result contains normalized UI nodes, the Runtime task-detail UI SHALL render those nodes in a structured, collapsible view while retaining the recorded JSON result. The Runtime SHALL NOT change the tool's response contract or duplicate the result in a separate persistence field.
|
|
|
|
#### Scenario: UI-tree inspection succeeds
|
|
- **WHEN** a task step uses `get_ui_tree` or `ui_tree` and returns one or more normalized nodes
|
|
- **THEN** the task-detail UI displays each node's type, visible text or identifier, bounds, and available confidence
|
|
|
|
#### Scenario: A non-UI-tree step is displayed
|
|
- **WHEN** a task step did not invoke a UI-tree tool
|
|
- **THEN** the task-detail UI does not render an empty UI-tree section
|