agentic-mobile-control/openspec/changes/task-execution-progress-visibility/proposal.md

Why

Nobody can see a task while it is running. Host Agent's local console only shows a coarse "current assignment" snapshot (task id, device id, goal, started-at) because the TaskRunner it drives is wired with metadata_store=None, timeline=None — every step transition happens in memory and is discarded the instant the assignment finishes. Cloud Control Plane only learns about an assignment at claim, heartbeat, and terminal-result time, so Cloud Console has nothing better to show. The local Runtime (api/rest.py + console/) is a separate process with its own independent TaskMetadataStore/Timeline, so it never sees a task that a Host Agent executed at all. An operator debugging a stuck or misbehaving task currently has no live signal anywhere in the system until the task finishes or times out.

What Changes

• Wire Host Agent's AssignmentExecutor / create_execution_factories() with a real metadata_store and timeline (or Host-Agent-local equivalents) so the in-process TaskRunner actually records step-by-step status instead of discarding it.
• Expose that step-level detail through Host Agent's local console: extend AgentStatusTracker//api/status (or add a focused endpoint) with current step index, step status, and a short in-progress step log; render it in the dashboard's "Current assignment" section instead of the static started-at-only view.
• Add a progress-reporting path from Host Agent to Cloud Control Plane so the control plane learns step-level state near-real-time rather than only at claim/heartbeat/terminal-result. Extend the existing cloud.internal_api.models Pydantic schema (the single source of truth Host Agent already imports directly) rather than introducing a parallel schema.
• Persist and expose the latest per-assignment progress on the Cloud Control Plane side, and surface it in Cloud Console so an operator watching a remote task sees live step progress, not just "dispatched" / "succeeded" / "failed".
• Give an operator a web console view of Host-Agent-executed task progress without making the runtime/-owned packages import host or cloud concerns (enforced by test_runtime_owned_packages_do_not_import_host_or_cloud_concerns). After evaluating the alternative of making the Runtime console/ SPA multi-backend (pointing its existing JS bundle at a Host Agent's console origin), this change instead extends Host Agent's own server-rendered local console with read-only task list/detail/timeline pages, reusing the same query shape api/console.py already exposes to the Runtime SPA. This avoids new cross-origin/session-cookie surface between the SPA and Host Agent, and keeps runtime/ untouched. See design.md for the full trade-off analysis.
• All three surfaces continue to use polling (matching current behavior); this change does not introduce SSE/WebSocket infrastructure unless design.md finds a compelling reason to.
• Fix the shared Timeline/TaskRunner/AIPlanner recording path so a persisted step's "prompt" is the actual prompt sent to the LLM for that step (not the task's overall goal) and the model's resulting decision is captured too — this pre-existing gap affects Runtime and Host Agent alike and undermines the step-level detail this change otherwise adds.
• Persist a durable, per-step log of full LLM prompt/response content on the Cloud Control Plane, for centralized troubleshooting — reusing the already-existing cloud-planner-proxy decide endpoint as the capture point (no new protocol/endpoint) rather than the coarse lease-renewal piggyback used for live index/status. This durable log is populated only for hosts using the cloud planner transport; hosts on the direct transport still get only the coarse index/status via lease renewal. Cloud Console gains a view to browse a task's full LLM interaction history.
• Extend the local Runtime timeline so every executed action retains a before screenshot, operation detail, after screenshot, and any available raw OCR observations; render that evidence in the Runtime task-detail UI and expose it from the existing Runtime console API. When a step uses the existing UI-tree tool, render its normalized node result as a structured, collapsible view as well.

Capabilities

New Capabilities

• host-agent-task-progress: Host Agent captures step-level execution progress for its in-flight assignment (via a wired metadata_store/timeline) and exposes it through its local console/API.
• cloud-task-progress-visibility: Cloud Control Plane receives, persists, and exposes near-real-time step-level progress for assignments it has dispatched to a Host Agent, and Cloud Console renders it.
• host-agent-console-task-pages: Host Agent's local server-rendered console gains read-only task list/detail/timeline pages (mirroring api/console.py's task query shape) so an operator can inspect a Host-Agent-executed task's progress and history without needing the separate Runtime console/ SPA or violating the runtime/-package host/cloud isolation boundary.
• runtime-task-evidence: Runtime task history retains and renders pre/post action evidence, raw OCR observations, and existing UI-tree inspection results.

Modified Capabilities

• host-agent-protocol: add a requirement that the Host Agent reports in-progress step-level status updates to the control plane (in addition to the existing heartbeat/claim/renewal/result operations), and that the control plane accepts and stores them per active assignment.
• cloud-planner-proxy: the existing planner-decision endpoint additionally persists each resolved decision's prompt/response into a durable, bounded-retention per-step log, instead of discarding it after the response is returned.

Impact

• apps/device-host-agent/host_agent/app.py, execution.py, assignment.py, status.py, web/app.py — wire a real metadata/timeline store into the in-process TaskRunner, extend status tracking and the local console UI/API.
• runtime/task.py, runtime/ai_planner.py, runtime/tool_calling_client.py, storage/timeline.py, storage/artifact_store.py, core/models.py, perception/scene_builder.py — fix the shared step-recording path so the real per-step prompt and the model's response are captured, retain pre/post action screenshots and raw OCR observations, not just the task goal and parsed tool call.
• packages/cloud-platform/cloud/internal_api/models.py, packages/cloud-platform/cloud/internal_api/api.py (decide_planner_call), repository.py/sql_repository.py, a new Alembic migration — new progress-reporting request/response models and a persistence + query path for latest per-assignment coarse progress, and a new durable per-step planner_decision_log table (with its own retention job) populated from the existing planner-decision endpoint.
• cloud-console/ (Vue3 SPA) — new UI to render live per-assignment progress, and a new view to browse a task's full LLM interaction history.
• api/console.py, api/console_web.py, api/templates/runtime_console/* — expose and render Runtime-local step evidence; Host Agent task pages receive the data through the shared timeline shape.
• No changes anticipated to driver/, device/, core/ device-control internals.