agentic-mobile-control/openspec/changes/task-execution-progress-visibility/design.md

Context

Three surfaces currently cannot show a task's in-progress status:

1. Host Agent local console (apps/device-host-agent/host_agent/web/app.py): HostAgentApplication's builder calls create_execution_factories(resolved_manager, host_agent_config=resolved_config) without metadata_store/timeline (app.py:218-223), so the in-process TaskRunner (execution.py:39-46) runs with metadata_store=None, timeline=None. runtime/task.py::TaskRunner._update_task() only persists when metadata_store is truthy, so step transitions vanish. The console's only signal is AgentStatusTracker.snapshot() (status.py:14-88), a single opaque span: task_id/device_id/goal/started_at, cleared on mark_assignment_finished().
2. Cloud Control Plane / Cloud Console: the internal protocol (packages/cloud-platform/cloud/internal_api/models.py) only has claim, heartbeat, lease-renewal, and terminal-result messages. Lease renewal (ActiveAssignmentRunner._renew_while_running, apps/device-host-agent/host_agent/lease.py:79-106) already fires periodically (~1/3 of remaining lease) for every in-flight assignment, but carries no task-progress payload.
3. Runtime console/ SPA: api/rest.py::create_app() builds its own TaskMetadataStore/Timeline/TaskRunner (api/rest.py:24-41), queried by api/console.py::create_console_router()'s /console/tasks* routes, which console/src/api.ts calls against API_BASE_URL (default http://127.0.0.1:8000). This is a wholly separate process from Host Agent, with no shared store or IPC. runtime/-owned packages are forbidden from importing host/cloud concerns (test_runtime_owned_packages_do_not_import_host_or_cloud_concerns), and host-agent-local-console/design.md already treats Host Agent's local history as "not a system of record," distinct from Cloud's.

Goals / Non-Goals

Goals:

• Host Agent's local console shows live step-level progress for its current assignment (step index, step status, short summary), not just a static started-at snapshot.
• Cloud Control Plane learns step-level progress near-real-time (piggybacked on existing periodic traffic, not a new polling loop) and Cloud Console renders it for an operator watching a remote task.
• An operator can inspect a Host-Agent-executed task's progress/history from a web console, without coupling runtime/ to host/cloud concerns.
• Reuse existing, already-tested building blocks (TaskMetadataStore, Timeline, api/console.py's query shape, the lease-renewal cadence) instead of inventing new storage or transport primitives.
• Both the Host Agent's local task history and the Cloud Control Plane persist the actual per-step content sent to and received from the LLM (the real prompt text, not just the task's overall goal; the model's resulting decision, not just the parsed tool call) — so a step's record answers "what did we actually ask the model and what did it say," not just "what call did we make." Cloud's copy is the durable, centrally-searchable record for troubleshooting; the Host Agent's local copy remains the on-device record of what actually ran.

Non-Goals:

• No SSE/WebSocket push. All three surfaces keep polling (Host Agent console 5s, Cloud Console/cloud-console and console/ at their existing intervals). Nothing here requires push infra, and adding it would be a bigger, separate change.
• The coarse live step index/status/summary reported via lease renewal (D4/D5) stays latest-snapshot-only on the Cloud side (overwrite semantics) — it is a "what's happening right now" indicator, not a history mechanism, and is unaffected by the LLM-content decisions below.
• Full step-by-step LLM interaction history is now synced to and durably persisted by Cloud (see D7/D8) — this is a deliberate reversal of this change's earlier draft, which had scoped Cloud to latest-only. The motivation: Host Agent deployments already route every AI Planner decision through Cloud API's existing cloud-planner-proxy endpoint when configured for the cloud transport, so Cloud is already on the natural path for this data and centralizing it there (rather than only on whichever edge host happens to still be running) is the more useful place for an operator doing centralized troubleshooting. This durable log is scoped strictly to LLM prompt/response text — it is not a general-purpose duplicate of the Host Agent's full Timeline (no screenshots, no scene JSON dumps beyond what's embedded in the prompt text itself).
• No multi-backend Runtime console or cross-origin Host Agent integration. Runtime changes remain local to its existing same-origin API/UI and shared timeline evidence model.
• No change to driver//device//core/ device-control internals.
• No screenshot or full-scene data leaves the Host Agent process as part of progress reporting, and no screenshot bytes are ever persisted by the Cloud Control Plane. (Full LLM prompt/response text is, by contrast, an explicit Goal below — see D7/D8 — which partially supersedes the original cloud-planner-proxy proposal's "does not durably persist ... full prompt text" statement. Screenshots remain excluded; prompts no longer are.)

Decisions

D1: Wire a real metadata_store/timeline into Host Agent's TaskRunner, reusing the existing classes

create_execution_factories() already accepts metadata_store: TaskMetadataStore | None and timeline: Timeline | None (execution.py:28-34) — these are the same classes api/rest.py uses for the Runtime's own console. Reuse them as-is rather than inventing a Host-Agent-specific store: HostAgentApplication's builder constructs a TaskMetadataStore(db_path=<host-agent-local path>) and Timeline(ArtifactStore(<host-agent-local path>)) and passes them into create_execution_factories().

Alternative considered: a bespoke in-memory-only step recorder (cheaper, no disk I/O). Rejected because TaskMetadataStore/Timeline are already proven in production (Runtime's own console has run on them since the original web-console change) and reusing them means Host Agent's task/timeline data has the exact same shape as api/console.py already serializes — a prerequisite for D3 below. A bespoke format would need its own (de)serialization and its own console rendering code for no real benefit.

New config: HOST_AGENT_TASK_PROGRESS_DB_PATH (default e.g. host_agent_data/task_progress.sqlite3) and a matching artifact directory, kept separate from any local Runtime tasks/ directory that might exist on the same machine, to avoid two unrelated processes silently sharing or colliding on a path.

D2: Bounded retention for the Host-Agent-local store

Unlike a Runtime dev session (short-lived, manually cleared), a Host Agent process runs indefinitely and executes many assignments over its lifetime. Unbounded TaskMetadataStore rows and Timeline screenshot artifacts would grow without limit.

Decision: add a lightweight retention pass (e.g. on a timer, or opportunistically after each assignment finishes) that prunes tasks older than a configurable window or beyond a configurable count, deleting both the tasks row and its Timeline/ArtifactStore files. This is new logic — neither TaskMetadataStore nor Timeline currently supports deletion — scoped as a small addition to storage/task_metadata.py / storage/timeline.py (or a Host-Agent-side wrapper if adding delete methods to the shared storage package feels too broad; prefer extending storage since both Runtime and Host Agent benefit from bounded retention).

D3: Give Host Agent's own console read-only task/timeline pages instead of making console/ multi-backend

Two options were evaluated for capability host-agent-console-task-pages:

• (a) Multi-backend Runtime console/ SPA: let the existing Vue SPA point at a Host Agent's console origin (a saved/selectable base URL) in addition to the local Runtime. Since D1 makes Host Agent's data shape identical to what api/console.py already serializes, the SPA's existing fetch/render code would work unmodified against a Host Agent origin.
• (b) Extend Host Agent's own server-rendered console (host_agent/web/app.py) with new read-only pages for task list/detail/timeline, reusing TaskMetadataStore/Timeline query calls directly (the same calls api/console.py's handlers make), rendered as server-side HTML via the existing f-string + html.escape() convention (no SPA, no frontend build) established by host-agent-local-console.

Decision: (b). (a) requires Host Agent's console to serve cross-origin, credentialed requests from whatever origin console/ is running on (Vite dev server, or a different deployed origin) — meaning either relaxing Host Agent's CORS posture for its session-cookie+CSRF-protected endpoints, or reworking its auth model to tolerate cross-origin fetches. That is real new attack surface on a console whose entire local-only threat model (console_bind_host defaulting to loopback, explicit opt-in for non-loopback) was deliberately conservative. (b) reuses an already-accepted pattern (server-rendered, same-origin, same auth) and fully satisfies the actual operator need — a web page showing what a Host Agent is doing — without touching runtime/, console/, or Host Agent's CORS/auth posture at all. The cost is a small amount of view duplication (Host Agent's console and Runtime's console/ render similar-shaped data with different templates/frameworks), judged acceptable given they serve different operational contexts (local direct-connect Runtime vs. remote Host Agent fleet).

Any future desire for a unified SPA across both surfaces is left as a follow-up, not blocked by this decision (D1's shared data shape keeps that door open).

D4: Piggyback progress reporting on the existing lease-renewal call

ActiveAssignmentRunner._renew_while_running (lease.py:79-106) already makes an authenticated, per-assignment, periodic call (client.renew(assignment), roughly every 1/3 of remaining lease) while a task executes, validated server-side against host_id/task_id/attempt/lease_id (internal_api/api.py:291-320). This is the natural piggyback point for progress: extend LeaseRenewalRequest with an optional progress: TaskProgressModel | None field (new small model: step_index: int, step_status: Literal[...], summary: str bounded length — no screenshot/scene payload) in cloud/internal_api/models.py, the single schema both sides already import directly (no parallel schema, matching existing convention noted for this protocol).

A small thread-safe "latest progress" holder (similar in spirit to LeaseGuard) is written by the execution thread (via a hook on TaskRunner/AssignmentExecutor, updated once per step) and read by _renew_while_running just before each renewal call, so no new timer/cadence is introduced.

Alternative considered: a dedicated POST /internal/v1/hosts/{host_id}/assignments/{task_id}/progress endpoint called on its own cadence. Rejected — it would duplicate the auth/validation renew_assignment already does, and introduce a second periodic call where one already exists and fires at a reasonable frequency for this purpose.

D5: Cloud persists only the latest progress snapshot, overwritten alongside lease renewal

renew_lease() (repository.py:407, sql_repository.py:1480) already takes a row lock and writes a lease-expiry update on every renewal; extend it to also accept and store the optional progress fields (few scalar columns — progress_step_index, progress_step_status, progress_summary, progress_updated_at — rather than a schema-flexible JSON blob, keeping it queryable and consistent with the rest of the row's typed columns). This requires a new Alembic migration (head is currently 0007_llm_provider_management) plus the equivalent SQLite schema change, applied to both repository.py and sql_repository.py to keep the dual-backend contract (cloud-control-plane spec's "Deployment and local persistence modes share one contract" requirement).

Progress columns are cleared (or simply superseded and ignored) once a terminal result is recorded — they represent "what's happening right now," not history; Cloud's durable attempt/result history is unaffected and unduplicated.

D6: Cloud Console reads progress from the existing task/attempt query path, not a new endpoint

Extend whatever response model Cloud Console's task list/detail already uses (Cloud API public router) with the optional latest-progress fields from D5, rather than adding a new endpoint. Cloud Console renders it as a small inline badge/line ("step 4: tapping login button") next to the existing status, refreshed on the SPA's existing polling interval.

D7: Capture full LLM interaction history for free via the existing cloud-planner-proxy decide endpoint, not a new reporting channel

Investigated the Host Agent → Cloud call path in detail: when a Host Agent is configured with AI_PLANNER_TRANSPORT=cloud, AIPlanner.plan() (runtime/ai_planner.py) calls CloudProxyToolCallingClient.decide() (apps/device-host-agent/host_agent/cloud_planner_client.py:46-95), which POSTs the complete system_prompt/user_prompt (plus task_id/attempt/lease_id from current_planner_execution_context()) to Cloud API's /hosts/{host_id}/planner/decide (packages/cloud-platform/cloud/internal_api/api.py::decide_planner_call, line ~367). That handler already resolves and returns a ToolCallDecision (tool_name/arguments/usage) and already does per-call bookkeeping (pool.store.settle_host_token_reservation(...), line ~463) using the same repository object this change's D5 already touches for lease renewal.

This means every planning step's actual prompt and resulting decision already flows through a Cloud-owned request handler when the cloud transport is used — no new endpoint, no new protocol field, no queue/batching scheme is needed to get full LLM content to Cloud. The only change needed is to make that handler persist what it currently discards.

Decision: extend decide_planner_call to, immediately after computing decision (success path only — a ToolCallUnavailable/502 path persists nothing), insert one row into a new log table (D8) keyed by (task_id, attempt, step_index), where step_index is assigned by the Cloud side itself (an auto-incrementing counter scoped to task_id+attempt, e.g. select count(*) + 1 under the same row lock, or a DB sequence/identity column) — Host Agent does not need to track or send a step counter for this.

Explicit limitation, called out rather than papered over: this only captures LLM content for hosts using the cloud transport. A host on the (still-default) direct transport never sends its prompts to Cloud at all — Cloud has zero LLM content for that host's tasks, and only ever sees the coarse index/status/summary from D4/D5's lease-renewal piggyback (which is transport-agnostic, since it's driven by TaskRunner step completion, not by the planner's transport choice). This is a real operational dependency: centralized LLM-interaction troubleshooting via Cloud Console requires the fleet (or the hosts an operator cares about) to run with AI_PLANNER_TRANSPORT=cloud. This change does not make cloud the new default transport — that remains a separate, already-existing configuration decision outside this change's scope.

Alternative considered: extend the D4 lease-renewal piggyback to also carry full prompt/response text (queued, not overwritten, so no step is lost between renewals). Rejected: it would duplicate a transport that already exists for exactly this payload (the decide call itself) whenever cloud transport is active, and would still need a separate new channel for the direct-transport case where Cloud never sees the prompt anyway — i.e., it does not actually solve the direct-transport gap, so it only adds complexity without expanding coverage.

D8: New bounded-retention table for the full per-step LLM decision log, extended on both repository backends

Add a new table (e.g. planner_decision_log): id, host_id, task_id, attempt, step_index, system_prompt (text), user_prompt (text), tool_name, arguments_json (text), created_at. No screenshot column — screenshots are never sent to this endpoint's persistence path (the request's screenshot_base64 is used only to call the LLM provider and is never written to this log, consistent with the Non-Goals screenshot exclusion).

Like D5, this needs a new Alembic migration and the equivalent SQLite path, implemented on both repository.py's SQLite-backed implementation and sql_repository.py::SQLAlchemyCloudRepository to preserve the existing dual-backend contract. Unlike D5's few-nullable-columns-on-an-existing-row approach, this is an independent append-only table (one row per decide call, not an overwrite), since the whole point is durable per-step history rather than a live snapshot.

Retention: this table grows once per planning step across the whole fleet, indefinitely, on a shared multi-tenant Cloud database — unbounded growth is a real risk here in a way D5's single-row-per-assignment overwrite never was. Decision: a scheduled prune job (mirrors D2's Host-Agent-local retention) deletes rows whose owning task reached a terminal state more than a configurable window ago (default: prune 7 days after task terminal, or once the task itself is pruned/archived by whatever existing Cloud task-retention policy applies — reuse that cadence rather than inventing a second one if cloud-control-plane already has one; otherwise default to a simple time-based prune).

D9: Fix the shared Timeline/TaskRunner/AIPlanner path so the actual per-step prompt and response are recorded locally, not just the task goal and the parsed tool call

Independent of Cloud persistence, the existing local recording is itself wrong today: TaskRunner._append_timeline() (runtime/task.py:296-319) calls Timeline.append(prompt=task.goal, tool_call={"action": ..., "description": ..., "args": ...}, ...). task.goal is the overall task goal, not the per-step prompt actually sent to the LLM — the real per-step prompt (planner_user_prompt(goal, scene_json, history_summary), built in ai_planner.py::plan()) is constructed, sent, and discarded entirely within AIPlanner.plan(), never reaching TaskRunner. Likewise ToolCallDecision (runtime/tool_calling_client.py:23-27) carries only the parsed tool_name/arguments/usage — any raw response text/content the model returned is discarded during parsing (_decision_from_anthropic_response/_decision_from_openai_response).

This is a pre-existing gap in a component shared by Runtime and Host Agent alike (not new to this change), and it undermines the very "step-level detail" goal D1 already committed to — a persisted step whose "prompt" field is just the task's goal repeated on every row is not useful for troubleshooting.

Decision: extend ToolCallDecision with the actual user_prompt/system_prompt it was given (or have AIPlanner.plan() return a small side-channel result instead of changing the Planner interface's return type) so TaskRunner._append_timeline() can pass the real per-step prompt into Timeline.append(). Rename Timeline/TimelineRecord's prompt field's meaning (or add a new field) to unambiguously mean "the prompt actually sent to the LLM for this step." This fix lands in the shared runtime/storage packages, so both Host Agent's local console (D3) and the existing Runtime console/ automatically benefit — it is not Host-Agent-specific plumbing.

D10: Persist and render before/after action evidence in the Runtime timeline

The existing Timeline captures only one screenshot after an action, while the planning screenshot used before the action is transient. Extend each Timeline record with distinct before/after screenshot paths, preserving the existing screenshot_path as a backward-compatible alias for the after screenshot. TaskRunner captures each image immediately before and after invoking the executor, including failed action attempts. The Runtime console API inlines both images and the existing task-detail page renders them beside the recorded action and result.

OCR is already available during perception but fused into the normalized Scene, where an OCR value that overlaps a UI-tree node can lose its raw provenance. Preserve raw OCR observations on Scene for local timeline capture only; keep them out of Scene.to_dict() so the LLM-facing planner payload does not grow with duplicated text. The Runtime task-detail page renders the persisted OCR list when it is available and handles legacy records without it.

The existing get_ui_tree/ui_tree tool returns a normalized flat list of UI nodes, not the driver-specific raw XML hierarchy. Its StepResult is already persisted in the Timeline. Detect those action names in the Runtime UI and render the returned nodes in a collapsible structured view while retaining the full JSON result below it. This makes the inspection result readable without changing the tool response contract or storing a second tree copy.

Risks / Trade-offs

• [Risk] Per-step SQLite writes in a long-lived Host Agent process add I/O overhead → Mitigation: this is the same write pattern Runtime's own console has always used per step; no new proof of acceptability needed. If profiling later shows it matters for very high step-rate tasks, batching/debouncing is a follow-up, not a blocker here.
• [Risk] Unbounded local disk growth from Timeline screenshots on an indefinitely-running Host Agent → Mitigation: D2's bounded retention pass; must ship in the same change as D1, not deferred, since D1 alone would otherwise introduce an unbounded-growth regression.
• [Risk] New DB columns/migration touch both repository.py (SQLite) and sql_repository.py (Postgres) — drift between the two has been a real defect category in this codebase (see cloud-control-plane-integration archive notes) → Mitigation: contract/parity tests already exist for this dual-backend boundary; extend them to cover the new progress columns.
• [Risk] View duplication between Host Agent's server-rendered task pages (D3) and Runtime console/'s task pages (different frameworks, same data shape) → Mitigation: accepted trade-off (see D3 rationale); shared data shape keeps future unification possible without rework.
• [Risk] Progress payload could accidentally grow to include sensitive data (scene dumps, prompts) if a future change casually extends TaskProgressModel → Mitigation: summary field is explicitly bounded/plain-text only; code review for this change and future extensions should treat this the same as the existing cloud-planner-proxy no-screenshot-persistence rule (screenshots specifically remain excluded from every Cloud-side table this change introduces).
• [Risk] planner_decision_log (D8) grows unboundedly across the whole fleet, unlike D5's single-row-per-assignment overwrite → Mitigation: D8's retention/prune job must ship in the same change as D7/D8, not deferred, for the same reason D2 must ship alongside D1.
• [Risk] Full prompt text can itself contain sensitive on-screen content (whatever text was visible in the scene description embedded in the prompt) — persisting it centrally is a deliberate trade-off the user has explicitly requested for centralized troubleshooting, but it is a real expansion of what Cloud stores → Mitigation: no additional mitigation beyond what's already decided (screenshots still excluded); flagged here so it is a visible, intentional decision rather than a silent scope creep.
• [Risk] Full LLM history in Cloud is silently absent for any host on the direct transport, which could read as "it's broken" rather than "expected" → Mitigation: Cloud Console should visibly distinguish "no progress reported yet" from "this host does not report LLM content" (e.g. by also surfacing the host's configured transport), rather than just showing an empty history with no explanation.

Migration Plan

1. Add TaskProgressModel and the optional progress field to LeaseRenewalRequest/response in cloud/internal_api/models.py. Backward compatible: progress is optional, older Host Agents omit it.
2. Add progress columns + Alembic migration 0008_task_progress_columns; extend renew_lease() in both repository implementations to accept/store them.
3. Extend Cloud Console's task read model and UI to render the new fields (no-op if absent, keeping rollback trivial).
4. Wire metadata_store/timeline into Host Agent's create_execution_factories() call site, add the retention pass (D2), and add the progress-holder hook feeding D4's renewal piggyback.
5. Extend Host Agent's local console with the read-only task/timeline pages (D3).
6. Fix Timeline/TaskRunner/AIPlanner/ToolCallingClient to record the real per-step prompt and response locally (D9) — independent of Cloud, benefits both Host Agent and Runtime consoles immediately.
7. Add planner_decision_log + Alembic migration for it (D8) on both repository backends, with its retention/prune job.
8. Extend decide_planner_call (internal_api/api.py) to persist each resolved decision into planner_decision_log (D7).
9. Extend Cloud Console with a per-task LLM interaction history view reading the new table, including the "host uses direct transport, no LLM content available" distinction from the Risks section.
10. Rollback: each step is independently revertible (optional field, additive columns, additive tables, additive UI, additive Host Agent wiring) — no destructive migration is required; both 0008 and the new decision-log migration's down-revisions drop only what they added.

Open Questions

• Exact retention window/count defaults for D2 (time-based vs. count-based, or both) — left for tasks.md to pick a concrete, documented default (e.g. keep last 50 tasks or 7 days, whichever is smaller) rather than block design on it.
• Whether Cloud Console's existing task list/detail component can absorb the new fields with a small edit or needs a new sub-component — an implementation detail, not an architectural fork.
• Whether a future change should unify Host Agent's server-rendered task pages and Runtime console/'s SPA into one shared frontend, now that D1 gives them an identical underlying data shape — explicitly deferred, not part of this change.
• Exact retention default for D8's planner_decision_log (prune-after-terminal window, or reuse an existing Cloud task-retention cadence if one already exists) — left for tasks.md to pick a concrete default rather than block design on it.
• Whether step_index in D8 should be assigned via a SELECT count(*) + 1 under a row lock or a dedicated per-(task_id, attempt) counter/sequence — an implementation detail to resolve in tasks.md, not an architectural fork.