feat(runtime): capture step evidence in console

2026-07-15 10:12:09 +08:00
parent 8d5b02e37f
commit ccde30e378
20 changed files with 594 additions and 53 deletions
@@ -0,0 +1,38 @@
+## ADDED Requirements
+
+### Requirement: Runtime persists complete evidence for each executed action
+The Runtime SHALL persist, for each action it attempts, a screenshot captured immediately before the executor call, the action description and arguments, the execution result, and a screenshot captured immediately after the executor call. Existing timeline records that contain only the legacy single screenshot SHALL remain readable, with that screenshot treated as the post-action image.
+
+#### Scenario: An action succeeds
+- **WHEN** the Runtime executes an action for a task
+- **THEN** its timeline record includes distinct before and after screenshots, the action detail, and the execution result
+
+#### Scenario: An action fails
+- **WHEN** the Runtime executor exhausts its retries for an action
+- **THEN** the action's timeline record still includes any screenshots that were captured and the failure result before the task is marked failed
+
+#### Scenario: A legacy timeline record is read
+- **WHEN** a timeline record has only the prior `screenshot_path` field
+- **THEN** the Runtime exposes it as the post-action screenshot without failing to render the record
+
+### Requirement: Runtime task evidence exposes available OCR observations
+The Runtime SHALL persist raw OCR observations associated with the scene used to plan an action when available, without adding duplicate OCR data to the LLM-facing normalized Scene payload. The Runtime task-detail UI SHALL render available OCR text, confidence, and bounds, and SHALL render normally when no OCR result exists.
+
+#### Scenario: OCR found text while planning an action
+- **WHEN** perception produced one or more OCR observations for the action's planning scene
+- **THEN** the corresponding timeline record includes those observations and the Runtime task-detail UI displays them
+
+#### Scenario: OCR was unavailable or found no text
+- **WHEN** perception yields no OCR observations
+- **THEN** the Runtime records the action evidence and renders the task detail without an OCR result list
+
+### Requirement: Runtime task evidence renders UI-tree inspection results
+When a Runtime step invokes the existing `get_ui_tree` or `ui_tree` tool and the persisted result contains normalized UI nodes, the Runtime task-detail UI SHALL render those nodes in a structured, collapsible view while retaining the recorded JSON result. The Runtime SHALL NOT change the tool's response contract or duplicate the result in a separate persistence field.
+
+#### Scenario: UI-tree inspection succeeds
+- **WHEN** a task step uses `get_ui_tree` or `ui_tree` and returns one or more normalized nodes
+- **THEN** the task-detail UI displays each node's type, visible text or identifier, bounds, and available confidence
+
+#### Scenario: A non-UI-tree step is displayed
+- **WHEN** a task step did not invoke a UI-tree tool
+- **THEN** the task-detail UI does not render an empty UI-tree section