feat(host-agent): make execution history authoritative
Tests / Test failed: 2, passed: 830

This commit is contained in:
2026-07-15 11:46:27 +08:00
parent ccde30e378
commit 77d4813bb2
46 changed files with 890 additions and 3132 deletions
@@ -1,38 +1,61 @@
## ADDED Requirements
### Requirement: Runtime persists complete evidence for each executed action
The Runtime SHALL persist, for each action it attempts, a screenshot captured immediately before the executor call, the action description and arguments, the execution result, and a screenshot captured immediately after the executor call. Existing timeline records that contain only the legacy single screenshot SHALL remain readable, with that screenshot treated as the post-action image.
The shared Runtime SHALL persist, for each action it attempts, a screenshot
captured immediately before the executor call, the action description and
arguments, the execution result, and a screenshot captured immediately after
the executor call. Existing Timeline records that contain only the legacy
single screenshot SHALL remain readable, with that screenshot treated as the
post-action image.
#### Scenario: An action succeeds
- **WHEN** the Runtime executes an action for a task
- **THEN** its timeline record includes distinct before and after screenshots, the action detail, and the execution result
- **THEN** its Timeline record includes distinct before and after screenshots,
action detail, and execution result
#### Scenario: An action fails
- **WHEN** the Runtime executor exhausts its retries for an action
- **THEN** the action's timeline record still includes any screenshots that were captured and the failure result before the task is marked failed
- **THEN** the Timeline record still includes any captured screenshots and the
failure result before the task is marked failed
#### Scenario: A legacy timeline record is read
- **WHEN** a timeline record has only the prior `screenshot_path` field
- **THEN** the Runtime exposes it as the post-action screenshot without failing to render the record
#### Scenario: A legacy Timeline record is read
- **WHEN** a Timeline record has only the prior `screenshot_path` field
- **THEN** the Runtime exposes it as the post-action screenshot without
failing to render the record
### Requirement: Runtime task evidence exposes available OCR observations
The Runtime SHALL persist raw OCR observations associated with the scene used to plan an action when available, without adding duplicate OCR data to the LLM-facing normalized Scene payload. The Runtime task-detail UI SHALL render available OCR text, confidence, and bounds, and SHALL render normally when no OCR result exists.
### Requirement: Runtime task evidence retains available OCR observations
The shared Runtime SHALL persist raw OCR observations associated with the scene
used to plan an action when available, without adding duplicate OCR data to the
LLM-facing normalized Scene payload. The Host Agent task-detail UI SHALL render
available OCR text, confidence, and bounds, and SHALL render normally when no
OCR result exists.
#### Scenario: OCR found text while planning an action
- **WHEN** perception produced one or more OCR observations for the action's planning scene
- **THEN** the corresponding timeline record includes those observations and the Runtime task-detail UI displays them
- **WHEN** perception produced one or more OCR observations for the action's
planning scene
- **THEN** the corresponding Timeline record includes those observations and
the Host Agent task-detail page displays them
#### Scenario: OCR was unavailable or found no text
- **WHEN** perception yields no OCR observations
- **THEN** the Runtime records the action evidence and renders the task detail without an OCR result list
- **THEN** the Runtime records the action evidence and the Host Agent task
detail renders without an OCR result list
### Requirement: Runtime task evidence renders UI-tree inspection results
When a Runtime step invokes the existing `get_ui_tree` or `ui_tree` tool and the persisted result contains normalized UI nodes, the Runtime task-detail UI SHALL render those nodes in a structured, collapsible view while retaining the recorded JSON result. The Runtime SHALL NOT change the tool's response contract or duplicate the result in a separate persistence field.
### Requirement: Runtime task evidence retains UI-tree inspection results
The Runtime SHALL retain a UI-tree inspection result when a step invokes the
existing `get_ui_tree` or `ui_tree` tool and the result contains normalized
nodes. The Host Agent task-detail UI SHALL render those nodes in a structured,
collapsible view while retaining the recorded JSON result. The Runtime SHALL
NOT change the tool response contract or duplicate the result in a separate
persistence field.
#### Scenario: UI-tree inspection succeeds
- **WHEN** a task step uses `get_ui_tree` or `ui_tree` and returns one or more normalized nodes
- **THEN** the task-detail UI displays each node's type, visible text or identifier, bounds, and available confidence
- **WHEN** a task step uses `get_ui_tree` or `ui_tree` and returns one or
more normalized nodes
- **THEN** the Host Agent task-detail page displays each node's type, visible
text or identifier, bounds, and available confidence
#### Scenario: A non-UI-tree step is displayed
- **WHEN** a task step did not invoke a UI-tree tool
- **THEN** the task-detail UI does not render an empty UI-tree section
- **THEN** the Host Agent task-detail page does not render an empty UI-tree
section