agentic-mobile-control/openspec/changes/web-console/proposal.md

Why

Apex Agent currently exposes device control and task execution only through the REST/MCP tool surface (api/rest.py, api/mcp.py) — there is no way for a human operator to see, at a glance, which devices are connected, what a running task is doing, or to register a new device/tweak a runtime setting without editing code and restarting the process. As the platform moves past a single-developer MVP, an operator needs a lightweight status/config surface: view device and task state (including step-by-step timeline/screenshot replay for debugging), and perform simple configuration (register/remove devices, adjust runtime parameters) through a web UI instead of the Python API directly.

What Changes

• Add a console status API: GET endpoints to list devices with live status, list/filter tasks, fetch task detail, and fetch a task's per-step timeline (tool call, result, scene, screenshot) built on top of the existing TaskMetadataStore.list_tasks() and Timeline.read(), which are implemented but not yet exposed over HTTP.
• Add a console config API: endpoints to register a new device (driver_type + connection_info, e.g. WDA server_url/udid/wda_local_port), unregister a device, and view/update adjustable runtime parameters (currently just TaskRunnerConfig.max_steps).
• Add a device config store (new persistence, SQLite like TaskMetadataStore) so devices registered through the console survive process restarts — on startup the app reloads persisted device configs and re-registers them with DeviceManager, instead of devices existing only in the in-memory dict as today.
• Add an independent web console frontend (separate Vue 3 SPA project with its own build/deploy) that consumes the two APIs above to render: a device status dashboard, a task list/detail view with timeline + screenshot replay, and configuration screens for device management and runtime parameters.
• Explicitly out of scope: authentication/authorization for the console (assumed to run on a trusted network for now), editing agent/planner logic from the UI, multi-user concurrent config editing safeguards, and any change to the existing MCP tool surface or /agent/task execution semantics.

Capabilities

New Capabilities

• console-status-api: Read-only REST endpoints exposing device status, task list/detail, and per-task execution timeline (including screenshot retrieval) for the web console.
• console-config-api: REST endpoints for device registration/deregistration (backed by a persisted device config store) and viewing/updating simple runtime parameters (e.g. task runner max_steps).
• web-console-ui: Independent frontend SPA providing the status dashboard and configuration pages, consuming console-status-api and console-config-api.

Modified Capabilities

(none — this change is additive on top of device-management, task-memory, and mcp-tool-server; no existing requirements change)

Impact

• New code: api/console.py (or similar) for the new REST routes; storage/device_config.py for the persisted device config store; a new top-level frontend project (e.g. console/ or a sibling repo) for the Vue 3 SPA.
• Modified code: api/rest.py to mount the console routes and to reload persisted device configs at create_app() startup; runtime/task.py/TaskRunnerConfig to allow max_steps to be read/updated at runtime.
• Dependencies: no new backend dependencies expected (reuses FastAPI/SQLite already in pyproject.toml); the new frontend project brings its own Node/Vue 3 toolchain, separate from the Python package.
• Impact on existing behavior: DeviceManager in-memory registration behavior is unchanged; the console config store only adds a reload-on-startup convenience layer on top of it. Existing /devices, /agent/task, /task/{id} endpoints and MCP tools are unaffected.