79 lines
3.1 KiB
Markdown
79 lines
3.1 KiB
Markdown
# runtime-task-evidence Specification
|
|
|
|
## Purpose
|
|
|
|
Define the per-action evidence retained by shared Runtime Timeline records and
|
|
rendered by the Host Agent execution-history console.
|
|
|
|
## Requirements
|
|
|
|
### Requirement: Runtime persists complete evidence for each executed action
|
|
|
|
The shared Runtime SHALL persist, for each action it attempts, a screenshot
|
|
captured immediately before the executor call, the action description and
|
|
arguments, the execution result, and a screenshot captured immediately after
|
|
the executor call. Existing Timeline records that contain only the legacy
|
|
single screenshot SHALL remain readable, with that screenshot treated as the
|
|
post-action image.
|
|
|
|
#### Scenario: An action succeeds
|
|
|
|
- **WHEN** the Runtime executes an action for a task
|
|
- **THEN** its Timeline record includes distinct before and after screenshots,
|
|
action detail, and execution result
|
|
|
|
#### Scenario: An action fails
|
|
|
|
- **WHEN** the Runtime executor exhausts its retries for an action
|
|
- **THEN** the Timeline record still includes any captured screenshots and the
|
|
failure result before the task is marked failed
|
|
|
|
#### Scenario: A legacy Timeline record is read
|
|
|
|
- **WHEN** a Timeline record has only the prior `screenshot_path` field
|
|
- **THEN** the Runtime exposes it as the post-action screenshot without
|
|
failing to render the record
|
|
|
|
### Requirement: Runtime task evidence retains available OCR observations
|
|
|
|
The shared Runtime SHALL persist raw OCR observations associated with the scene
|
|
used to plan an action when available, without adding duplicate OCR data to the
|
|
LLM-facing normalized Scene payload. The Host Agent task-detail UI SHALL render
|
|
available OCR text, confidence, and bounds, and SHALL render normally when no
|
|
OCR result exists.
|
|
|
|
#### Scenario: OCR found text while planning an action
|
|
|
|
- **WHEN** perception produced one or more OCR observations for the action's
|
|
planning scene
|
|
- **THEN** the corresponding Timeline record includes those observations and
|
|
the Host Agent task-detail page displays them
|
|
|
|
#### Scenario: OCR was unavailable or found no text
|
|
|
|
- **WHEN** perception yields no OCR observations
|
|
- **THEN** the Runtime records the action evidence and the Host Agent task
|
|
detail renders without an OCR result list
|
|
|
|
### Requirement: Runtime task evidence retains UI-tree inspection results
|
|
|
|
The Runtime SHALL retain a UI-tree inspection result when a step invokes the
|
|
existing `get_ui_tree` or `ui_tree` tool and the result contains normalized
|
|
nodes. The Host Agent task-detail UI SHALL render those nodes in a structured,
|
|
collapsible view while retaining the recorded JSON result. The Runtime SHALL
|
|
NOT change the tool response contract or duplicate the result in a separate
|
|
persistence field.
|
|
|
|
#### Scenario: UI-tree inspection succeeds
|
|
|
|
- **WHEN** a task step uses `get_ui_tree` or `ui_tree` and returns one or
|
|
more normalized nodes
|
|
- **THEN** the Host Agent task-detail page displays each node's type, visible
|
|
text or identifier, bounds, and available confidence
|
|
|
|
#### Scenario: A non-UI-tree step is displayed
|
|
|
|
- **WHEN** a task step did not invoke a UI-tree tool
|
|
- **THEN** the Host Agent task-detail page does not render an empty UI-tree
|
|
section
|