agentic-mobile-control/openspec/changes/archive/2026-07-15-task-execution-progress-visibility/design.md

Context

Host Agent executes Cloud assignments in-process through the shared runtime.TaskRunner. Its HostAgentApplication already constructs a Host-local TaskMetadataStore and Timeline and injects them into the runner factory. The missing link is task creation: AssignmentExecutor constructs a Task, calls runner.run(task), and TaskRunner only issues updates. SQLite therefore receives updates for a row that does not exist, so the Host Agent /tasks UI is empty.

The separate api.rest process was never on this execution path. It owned a different database and artifact root, so its REST endpoints and UI could only show tasks submitted directly to that unrelated process. Running it alongside a Host Agent created two competing operator entry points without transferring any history between them.

Goals / Non-Goals

Goals:

• Make durable task-row creation an invariant of TaskRunner.run() whenever a metadata store is configured.
• Correlate a Host-local goal execution with the Cloud task ID and attempt that caused it, without making shared Runtime or storage packages import Cloud models.
• Make the authenticated Host Agent console at port 8765 the authoritative web view for actual task execution: task list, detail, live status, before screenshot, operation, after screenshot, OCR observations, and normalized UI-tree results.
• Preserve the shared Runtime Timeline as the evidence model and keep its before/after screenshot, OCR, and UI-tree capture behavior.
• Retire the standalone Runtime REST service/UI and its Host Agent supervisor configuration while retaining Runtime, storage, MCP, and skill-sync library modules.
• Retain the existing Cloud latest-progress and Cloud-proxy planner-decision history behavior. Cloud remains a fleet-level view and never stores screenshots.

Non-Goals:

• No SSE/WebSocket push; Host and Cloud consoles keep their existing polling.
• No duplicate full evidence upload to Cloud. Screenshots and Timeline artifacts remain Host-local.
• No new Host/Cloud dependencies in runtime/, storage/, driver/, or device/.
• No replacement general-purpose device-control REST API. Operators use Host Agent device management and task pages for the managed execution workflow.

Decisions

D1: TaskRunner owns idempotent task metadata creation

TaskRunner.run() will call TaskMetadataStore.create_task(task) before the first running-state update. create_task() will use idempotent insert semantics so callers that already created a direct Runtime task remain compatible and workflow-created tasks are captured automatically.

This places the invariant at the shared execution boundary rather than relying on every caller to remember an out-of-band persistence call. It fixes goal assignments and prevents the same failure for WorkflowRunner planned-goal steps.

D2: Host assignment correlation stays in the Host adapter

Before executing a Cloud goal assignment, AssignmentExecutor creates the local task record with optional generic source_task_id and source_attempt metadata. Shared storage uses generic names and does not import Cloud types. The Host Agent task list/detail renders those fields as the Cloud task ID and attempt.

The local Task.id remains generated by the Runtime. Reusing the Cloud task ID as an artifact directory name would make retries overwrite each other and would admit unsafe path characters from an external identifier.

D3: Host Agent console is the authoritative evidence UI

The Host Agent already owns the device manager, assignment executor, metadata store, Timeline, local account, session, and CSRF boundary. Its same-origin /tasks and /tasks/{task_id} pages therefore render the shared Timeline directly. The task list is named for Host executions rather than "Local Runtime tasks", and submission feedback tells the operator that the submitted Cloud task appears there when this Host begins execution.

No browser needs to point a separate frontend at the Host Agent. This keeps the conservative local-account/session model and does not add CORS.

D4: Complete per-step evidence is rendered by the Host Agent

Timeline records retain distinct pre-action and post-action screenshot paths, the action description and arguments, the execution result, raw OCR observations, and the existing normalized UI-tree result. The Host Agent page creates data URIs only for available local artifacts and supports legacy records where the single screenshot_path is the post-action image.

OCR is rendered when present. UI-tree output is rendered only for get_ui_tree and ui_tree records that contain normalized nodes; it uses a collapsible structured view while retaining the JSON result. No tool contract or duplicate persistence field is introduced.

D5: Host-local retention remains bounded

The existing Host retention pass continues to remove metadata rows, Timeline records, and artifacts according to the configured count and age thresholds. The new task creation invariant must use that same store so it cannot create an unbounded second history source.

D6: Cloud progress remains a latest snapshot on lease renewal

The Host execution thread writes a bounded latest-progress holder. The lease renewal path optionally carries its step index, status, and summary; Cloud stores only the latest snapshot for an active assignment and stops exposing it after terminal completion. No screenshot or scene payload enters this protocol.

D7: Cloud-proxy planner decisions remain durable Cloud history

For AI_PLANNER_TRANSPORT=cloud, Cloud's existing planner-decision endpoint persists successful system/user prompts, tool calls, arguments, and a task/attempt-scoped step index. The direct transport intentionally produces no such Cloud history. This log has bounded terminal-task retention and never persists request screenshot bytes.

D8: Retire the standalone Runtime REST service and UI

Remove api/rest.py, api/console.py, api/console_web.py, their templates/static assets, their package-data declarations, and their service/UI tests. Preserve api/mcp.py, api/errors.py, skill-sync, and skill-catalog modules because they are independent library integrations.

Remove HOST_AGENT_RUNTIME_SUPERVISED, HOST_AGENT_RUNTIME_HOST, and HOST_AGENT_RUNTIME_PORT. The optional dependency supervisor continues to support Appium only. Configuration with a removed Runtime-supervision variable fails with an actionable migration error instead of silently doing nothing.

D9: Documentation points operators to Host Agent

Operator documentation no longer instructs users to start uvicorn api.rest:create_app or browse port 8000. It identifies the Host Agent console at http://127.0.0.1:8765/tasks as the execution-history authority, explains its local authentication, and documents that the Cloud Console is a fleet/progress and Cloud-proxy LLM-history surface rather than a screenshot store.

Risks / Trade-offs

• Per-step metadata writes add small SQLite I/O. This is the same local store already selected for Host history, and bounded retention limits growth.
• A Host-local evidence record is only available while retained on that Host. This is intentional: it reflects the actual device execution and avoids sending screenshots to Cloud.
• Removing the unauthenticated Runtime REST service is a breaking operator change. Clear configuration errors and documentation avoid a silent fallback to a nonexistent inspection surface.
• Full Cloud-proxy prompts can contain visible screen text. This is the previously accepted Cloud troubleshooting trade-off; screenshot bytes remain excluded.

Migration Plan

1. Upgrade the Host Agent code. Existing task databases gain nullable source correlation columns on startup; legacy Timeline records remain readable.
2. Remove any HOST_AGENT_RUNTIME_* environment variables and stop any api.rest process. Start or browse only the Host Agent console for local execution evidence.
3. Confirm a completed Host assignment appears at /tasks with its Cloud task/attempt correlation and complete Timeline evidence.
4. Roll back only by restoring the prior release. The retired REST/UI routes are deliberately not kept as a compatibility alias because their storage was not connected to Host execution.

Open Questions

• Manual verification still requires a real Host Agent, Appium, and device. Automated coverage verifies persistence, correlation, rendering, and service removal; real hardware validates screenshots and OCR availability.