This commit is contained in:
+39
-16
@@ -1,38 +1,61 @@
|
||||
## ADDED Requirements
|
||||
|
||||
### Requirement: Runtime persists complete evidence for each executed action
|
||||
The Runtime SHALL persist, for each action it attempts, a screenshot captured immediately before the executor call, the action description and arguments, the execution result, and a screenshot captured immediately after the executor call. Existing timeline records that contain only the legacy single screenshot SHALL remain readable, with that screenshot treated as the post-action image.
|
||||
The shared Runtime SHALL persist, for each action it attempts, a screenshot
|
||||
captured immediately before the executor call, the action description and
|
||||
arguments, the execution result, and a screenshot captured immediately after
|
||||
the executor call. Existing Timeline records that contain only the legacy
|
||||
single screenshot SHALL remain readable, with that screenshot treated as the
|
||||
post-action image.
|
||||
|
||||
#### Scenario: An action succeeds
|
||||
- **WHEN** the Runtime executes an action for a task
|
||||
- **THEN** its timeline record includes distinct before and after screenshots, the action detail, and the execution result
|
||||
- **THEN** its Timeline record includes distinct before and after screenshots,
|
||||
action detail, and execution result
|
||||
|
||||
#### Scenario: An action fails
|
||||
- **WHEN** the Runtime executor exhausts its retries for an action
|
||||
- **THEN** the action's timeline record still includes any screenshots that were captured and the failure result before the task is marked failed
|
||||
- **THEN** the Timeline record still includes any captured screenshots and the
|
||||
failure result before the task is marked failed
|
||||
|
||||
#### Scenario: A legacy timeline record is read
|
||||
- **WHEN** a timeline record has only the prior `screenshot_path` field
|
||||
- **THEN** the Runtime exposes it as the post-action screenshot without failing to render the record
|
||||
#### Scenario: A legacy Timeline record is read
|
||||
- **WHEN** a Timeline record has only the prior `screenshot_path` field
|
||||
- **THEN** the Runtime exposes it as the post-action screenshot without
|
||||
failing to render the record
|
||||
|
||||
### Requirement: Runtime task evidence exposes available OCR observations
|
||||
The Runtime SHALL persist raw OCR observations associated with the scene used to plan an action when available, without adding duplicate OCR data to the LLM-facing normalized Scene payload. The Runtime task-detail UI SHALL render available OCR text, confidence, and bounds, and SHALL render normally when no OCR result exists.
|
||||
### Requirement: Runtime task evidence retains available OCR observations
|
||||
The shared Runtime SHALL persist raw OCR observations associated with the scene
|
||||
used to plan an action when available, without adding duplicate OCR data to the
|
||||
LLM-facing normalized Scene payload. The Host Agent task-detail UI SHALL render
|
||||
available OCR text, confidence, and bounds, and SHALL render normally when no
|
||||
OCR result exists.
|
||||
|
||||
#### Scenario: OCR found text while planning an action
|
||||
- **WHEN** perception produced one or more OCR observations for the action's planning scene
|
||||
- **THEN** the corresponding timeline record includes those observations and the Runtime task-detail UI displays them
|
||||
- **WHEN** perception produced one or more OCR observations for the action's
|
||||
planning scene
|
||||
- **THEN** the corresponding Timeline record includes those observations and
|
||||
the Host Agent task-detail page displays them
|
||||
|
||||
#### Scenario: OCR was unavailable or found no text
|
||||
- **WHEN** perception yields no OCR observations
|
||||
- **THEN** the Runtime records the action evidence and renders the task detail without an OCR result list
|
||||
- **THEN** the Runtime records the action evidence and the Host Agent task
|
||||
detail renders without an OCR result list
|
||||
|
||||
### Requirement: Runtime task evidence renders UI-tree inspection results
|
||||
When a Runtime step invokes the existing `get_ui_tree` or `ui_tree` tool and the persisted result contains normalized UI nodes, the Runtime task-detail UI SHALL render those nodes in a structured, collapsible view while retaining the recorded JSON result. The Runtime SHALL NOT change the tool's response contract or duplicate the result in a separate persistence field.
|
||||
### Requirement: Runtime task evidence retains UI-tree inspection results
|
||||
The Runtime SHALL retain a UI-tree inspection result when a step invokes the
|
||||
existing `get_ui_tree` or `ui_tree` tool and the result contains normalized
|
||||
nodes. The Host Agent task-detail UI SHALL render those nodes in a structured,
|
||||
collapsible view while retaining the recorded JSON result. The Runtime SHALL
|
||||
NOT change the tool response contract or duplicate the result in a separate
|
||||
persistence field.
|
||||
|
||||
#### Scenario: UI-tree inspection succeeds
|
||||
- **WHEN** a task step uses `get_ui_tree` or `ui_tree` and returns one or more normalized nodes
|
||||
- **THEN** the task-detail UI displays each node's type, visible text or identifier, bounds, and available confidence
|
||||
- **WHEN** a task step uses `get_ui_tree` or `ui_tree` and returns one or
|
||||
more normalized nodes
|
||||
- **THEN** the Host Agent task-detail page displays each node's type, visible
|
||||
text or identifier, bounds, and available confidence
|
||||
|
||||
#### Scenario: A non-UI-tree step is displayed
|
||||
- **WHEN** a task step did not invoke a UI-tree tool
|
||||
- **THEN** the task-detail UI does not render an empty UI-tree section
|
||||
- **THEN** the Host Agent task-detail page does not render an empty UI-tree
|
||||
section
|
||||
|
||||
Reference in New Issue
Block a user