feat(runtime): capture step evidence in console

openspec/changes/task-execution-progress-visibility/design.md

@@ -19,7 +19,7 @@ Three surfaces currently cannot show a task's in-progress status:
 - No SSE/WebSocket push. All three surfaces keep polling (Host Agent console 5s, Cloud Console/`cloud-console` and `console/` at their existing intervals). Nothing here requires push infra, and adding it would be a bigger, separate change.
 - The coarse live step index/status/summary reported via lease renewal (D4/D5) stays *latest-snapshot-only* on the Cloud side (overwrite semantics) — it is a "what's happening right now" indicator, not a history mechanism, and is unaffected by the LLM-content decisions below.
 - Full step-by-step **LLM interaction** history *is* now synced to and durably persisted by Cloud (see D7/D8) — this is a deliberate reversal of this change's earlier draft, which had scoped Cloud to latest-only. The motivation: Host Agent deployments already route every AI Planner decision through Cloud API's existing `cloud-planner-proxy` endpoint when configured for the `cloud` transport, so Cloud is already on the natural path for this data and centralizing it there (rather than only on whichever edge host happens to still be running) is the more useful place for an operator doing centralized troubleshooting. This durable log is scoped strictly to LLM prompt/response text — it is not a general-purpose duplicate of the Host Agent's full `Timeline` (no screenshots, no scene JSON dumps beyond what's embedded in the prompt text itself).
-- No change to the Runtime `console/` SPA or `runtime/`-owned packages. The multi-backend-SPA alternative was considered and rejected (see Decisions).
+- No multi-backend Runtime console or cross-origin Host Agent integration. Runtime changes remain local to its existing same-origin API/UI and shared timeline evidence model.
 - No change to `driver/`/`device/`/`core/` device-control internals.
 - No screenshot or full-scene data leaves the Host Agent process as part of progress reporting, and no screenshot bytes are ever persisted by the Cloud Control Plane. (Full LLM prompt/response *text* is, by contrast, an explicit Goal below — see D7/D8 — which partially supersedes the original `cloud-planner-proxy` proposal's "does not durably persist ... full prompt text" statement. Screenshots remain excluded; prompts no longer are.)
 
@@ -96,6 +96,14 @@ This is a pre-existing gap in a component shared by Runtime and Host Agent alike
 
 **Decision**: extend `ToolCallDecision` with the actual `user_prompt`/`system_prompt` it was given (or have `AIPlanner.plan()` return a small side-channel result instead of changing the `Planner` interface's return type) so `TaskRunner._append_timeline()` can pass the real per-step prompt into `Timeline.append()`. Rename `Timeline`/`TimelineRecord`'s `prompt` field's meaning (or add a new field) to unambiguously mean "the prompt actually sent to the LLM for this step." This fix lands in the shared `runtime`/`storage` packages, so both Host Agent's local console (D3) and the existing Runtime `console/` automatically benefit — it is not Host-Agent-specific plumbing.
 
+### D10: Persist and render before/after action evidence in the Runtime timeline
+
+The existing Timeline captures only one screenshot after an action, while the planning screenshot used before the action is transient. Extend each Timeline record with distinct before/after screenshot paths, preserving the existing `screenshot_path` as a backward-compatible alias for the after screenshot. `TaskRunner` captures each image immediately before and after invoking the executor, including failed action attempts. The Runtime console API inlines both images and the existing task-detail page renders them beside the recorded action and result.
+
+OCR is already available during perception but fused into the normalized Scene, where an OCR value that overlaps a UI-tree node can lose its raw provenance. Preserve raw OCR observations on `Scene` for local timeline capture only; keep them out of `Scene.to_dict()` so the LLM-facing planner payload does not grow with duplicated text. The Runtime task-detail page renders the persisted OCR list when it is available and handles legacy records without it.
+
+The existing `get_ui_tree`/`ui_tree` tool returns a normalized flat list of UI nodes, not the driver-specific raw XML hierarchy. Its StepResult is already persisted in the Timeline. Detect those action names in the Runtime UI and render the returned nodes in a collapsible structured view while retaining the full JSON result below it. This makes the inspection result readable without changing the tool response contract or storing a second tree copy.
+
 ## Risks / Trade-offs
 
 - [Risk] Per-step SQLite writes in a long-lived Host Agent process add I/O overhead → Mitigation: this is the same write pattern Runtime's own console has always used per step; no new proof of acceptability needed. If profiling later shows it matters for very high step-rate tasks, batching/debouncing is a follow-up, not a blocker here.

openspec/changes/task-execution-progress-visibility/proposal.md

@@ -12,6 +12,7 @@ Nobody can see a task while it is running. Host Agent's local console only shows
 - All three surfaces continue to use polling (matching current behavior); this change does not introduce SSE/WebSocket infrastructure unless design.md finds a compelling reason to.
 - Fix the shared `Timeline`/`TaskRunner`/`AIPlanner` recording path so a persisted step's "prompt" is the *actual* prompt sent to the LLM for that step (not the task's overall goal) and the model's resulting decision is captured too — this pre-existing gap affects Runtime and Host Agent alike and undermines the step-level detail this change otherwise adds.
 - Persist a durable, per-step log of full LLM prompt/response content on the Cloud Control Plane, for centralized troubleshooting — reusing the already-existing `cloud-planner-proxy` decide endpoint as the capture point (no new protocol/endpoint) rather than the coarse lease-renewal piggyback used for live index/status. This durable log is populated only for hosts using the `cloud` planner transport; hosts on the `direct` transport still get only the coarse index/status via lease renewal. Cloud Console gains a view to browse a task's full LLM interaction history.
+- Extend the local Runtime timeline so every executed action retains a before screenshot, operation detail, after screenshot, and any available raw OCR observations; render that evidence in the Runtime task-detail UI and expose it from the existing Runtime console API. When a step uses the existing UI-tree tool, render its normalized node result as a structured, collapsible view as well.
 
 ## Capabilities
 
@@ -19,6 +20,7 @@ Nobody can see a task while it is running. Host Agent's local console only shows
 - `host-agent-task-progress`: Host Agent captures step-level execution progress for its in-flight assignment (via a wired `metadata_store`/`timeline`) and exposes it through its local console/API.
 - `cloud-task-progress-visibility`: Cloud Control Plane receives, persists, and exposes near-real-time step-level progress for assignments it has dispatched to a Host Agent, and Cloud Console renders it.
 - `host-agent-console-task-pages`: Host Agent's local server-rendered console gains read-only task list/detail/timeline pages (mirroring `api/console.py`'s task query shape) so an operator can inspect a Host-Agent-executed task's progress and history without needing the separate Runtime `console/` SPA or violating the `runtime/`-package host/cloud isolation boundary.
+- `runtime-task-evidence`: Runtime task history retains and renders pre/post action evidence, raw OCR observations, and existing UI-tree inspection results.
 
 ### Modified Capabilities
 - `host-agent-protocol`: add a requirement that the Host Agent reports in-progress step-level status updates to the control plane (in addition to the existing heartbeat/claim/renewal/result operations), and that the control plane accepts and stores them per active assignment.
@@ -27,8 +29,8 @@ Nobody can see a task while it is running. Host Agent's local console only shows
 ## Impact
 
 - `apps/device-host-agent/host_agent/app.py`, `execution.py`, `assignment.py`, `status.py`, `web/app.py` — wire a real metadata/timeline store into the in-process `TaskRunner`, extend status tracking and the local console UI/API.
-- `runtime/task.py`, `runtime/ai_planner.py`, `runtime/tool_calling_client.py`, `storage/timeline.py` — fix the shared step-recording path so the real per-step prompt and the model's response are captured, not just the task goal and parsed tool call (a correctness fix in already-existing, shared code, not new behavior scope).
+- `runtime/task.py`, `runtime/ai_planner.py`, `runtime/tool_calling_client.py`, `storage/timeline.py`, `storage/artifact_store.py`, `core/models.py`, `perception/scene_builder.py` — fix the shared step-recording path so the real per-step prompt and the model's response are captured, retain pre/post action screenshots and raw OCR observations, not just the task goal and parsed tool call.
 - `packages/cloud-platform/cloud/internal_api/models.py`, `packages/cloud-platform/cloud/internal_api/api.py` (`decide_planner_call`), `repository.py`/`sql_repository.py`, a new Alembic migration — new progress-reporting request/response models and a persistence + query path for latest per-assignment coarse progress, *and* a new durable per-step `planner_decision_log` table (with its own retention job) populated from the existing planner-decision endpoint.
 - `cloud-console/` (Vue3 SPA) — new UI to render live per-assignment progress, and a new view to browse a task's full LLM interaction history.
-- `apps/device-host-agent/host_agent/web/app.py` — new read-only task list/detail/timeline pages backed by the Host-Agent-local `TaskMetadataStore`/`Timeline`; no changes anticipated to `console/` (Vue3 SPA).
+- `api/console.py`, `api/console_web.py`, `api/templates/runtime_console/*` — expose and render Runtime-local step evidence; Host Agent task pages receive the data through the shared timeline shape.
 - No changes anticipated to `driver/`, `device/`, `core/` device-control internals.

openspec/changes/task-execution-progress-visibility/specs/runtime-task-evidence/spec.md

@@ -0,0 +1,38 @@
+## ADDED Requirements
+
+### Requirement: Runtime persists complete evidence for each executed action
+The Runtime SHALL persist, for each action it attempts, a screenshot captured immediately before the executor call, the action description and arguments, the execution result, and a screenshot captured immediately after the executor call. Existing timeline records that contain only the legacy single screenshot SHALL remain readable, with that screenshot treated as the post-action image.
+
+#### Scenario: An action succeeds
+- **WHEN** the Runtime executes an action for a task
+- **THEN** its timeline record includes distinct before and after screenshots, the action detail, and the execution result
+
+#### Scenario: An action fails
+- **WHEN** the Runtime executor exhausts its retries for an action
+- **THEN** the action's timeline record still includes any screenshots that were captured and the failure result before the task is marked failed
+
+#### Scenario: A legacy timeline record is read
+- **WHEN** a timeline record has only the prior `screenshot_path` field
+- **THEN** the Runtime exposes it as the post-action screenshot without failing to render the record
+
+### Requirement: Runtime task evidence exposes available OCR observations
+The Runtime SHALL persist raw OCR observations associated with the scene used to plan an action when available, without adding duplicate OCR data to the LLM-facing normalized Scene payload. The Runtime task-detail UI SHALL render available OCR text, confidence, and bounds, and SHALL render normally when no OCR result exists.
+
+#### Scenario: OCR found text while planning an action
+- **WHEN** perception produced one or more OCR observations for the action's planning scene
+- **THEN** the corresponding timeline record includes those observations and the Runtime task-detail UI displays them
+
+#### Scenario: OCR was unavailable or found no text
+- **WHEN** perception yields no OCR observations
+- **THEN** the Runtime records the action evidence and renders the task detail without an OCR result list
+
+### Requirement: Runtime task evidence renders UI-tree inspection results
+When a Runtime step invokes the existing `get_ui_tree` or `ui_tree` tool and the persisted result contains normalized UI nodes, the Runtime task-detail UI SHALL render those nodes in a structured, collapsible view while retaining the recorded JSON result. The Runtime SHALL NOT change the tool's response contract or duplicate the result in a separate persistence field.
+
+#### Scenario: UI-tree inspection succeeds
+- **WHEN** a task step uses `get_ui_tree` or `ui_tree` and returns one or more normalized nodes
+- **THEN** the task-detail UI displays each node's type, visible text or identifier, bounds, and available confidence
+
+#### Scenario: A non-UI-tree step is displayed
+- **WHEN** a task step did not invoke a UI-tree tool
+- **THEN** the task-detail UI does not render an empty UI-tree section

openspec/changes/task-execution-progress-visibility/tasks.md

@@ -66,3 +66,11 @@
 - [x] 9.2 Add a `cloud-console/` view rendering a task's full LLM interaction history (prompt + resulting decision per step).
 - [x] 9.3 When a task's host used the `direct` transport (no persisted decisions and the host's configured transport is known to be `direct`), show an explicit "not reported by this host's transport" state rather than an empty list.
 - [x] 9.4 Add Vitest coverage for populated history, empty-but-cloud-transport (task hasn't produced any decisions yet), and direct-transport-hidden cases.
+
+## 10. Runtime per-step evidence (D10)
+
+- [x] 10.1 Extend the shared Scene/Timeline/ArtifactStore model to retain raw OCR observations and separate before/after screenshots while preserving compatibility with existing single-screenshot records.
+- [x] 10.2 Capture before and after screenshots around every TaskRunner executor call, then persist the action, result, and available OCR observations in the same timeline record.
+- [x] 10.3 Extend the Runtime console JSON API and task-detail UI to render before/after screenshots, operation details, and available OCR results.
+- [x] 10.4 Render persisted normalized UI-tree tool results in the Runtime task-detail UI without changing the tool contract or duplicating stored data.
+- [x] 10.5 Add focused Timeline, TaskRunner, perception, JSON API, Runtime UI, and UI-tree regression coverage; run the relevant format, lint, test, and strict OpenSpec validation commands.