agentic-mobile-control/openspec/changes/archive/2026-07-06-device-agent-runtime-foundation/proposal.md

Why

This project started as "Apex Agent," an iPhone-specific automation platform, and apex-agent-mvp already implements a driver-independent Driver interface, a WDADriver, a DeviceManager, a Scene perception pipeline, an Agent Runtime, task-memory timelines, and an MCP/REST surface. The actual long-term value isn't "control an iPhone" — it's giving an LLM a stable way to operate any real-world device, with iPhone/WDA as just the first driver. Nothing about the implemented behavior needs to change to realize this, but the naming, package layout, and a couple of remaining structural gaps still assume "iPhone is the platform" rather than "iPhone is one driver." Fixing this now — before Android/Chrome/Windows drivers or a skill/workflow/multi-agent layer accrete on top — is far cheaper than un-tangling it later. This change is Milestone 0 (Foundation) of a longer roadmap: establish the device-agnostic runtime shape and the planning artifacts (roadmap, ADR, research track) that later milestones (Perception, Execution, Planning, Semantic, World, Skill, Workflow, Agent, Cloud Runtime) will build on.

What Changes

• Reorganize core/ into a driver/ package (the Driver interface + concrete driver implementations, WDADriver first) and a device/ package (DeviceManager and device lifecycle), leaving only genuinely shared domain models/errors in core/. No behavior changes — this is a structural move plus import updates.
• Rename vision/ to perception/ so the screenshot → OCR → tree → Scene pipeline is named for what it produces (perception of a device's screen), not for one technique (OCR) inside it.
• Rename the ApexAgentError exception base (and its "Apex Agent" framing in docstrings/comments) to a device-agnostic name, since it is the one place the old iPhone-only brand is baked into code rather than docs.
• Relocate the driver-type-to-factory registry (currently SUPPORTED_DRIVER_TYPES/build_driver_factory inside api/console.py) into the driver/ package as a first-class Driver Registry, so adding a new driver type (Android, browser, ...) never requires touching the API/console layer. BREAKING: api/console.py's import path for driver-factory construction changes.
• Rebrand the project from "Apex Agent" to Device Agent Runtime (working name; long-term direction is a "DeviceOS"-style universal device runtime, recorded as a roadmap milestone, not implemented now): update pyproject.toml project name/description, and add a root README.md stating the new positioning.
• Add docs/ROADMAP.md capturing the milestone sequence (Foundation → Device → Perception → Execution → Planning → Semantic → World → Skill → Workflow → Agent → Cloud Runtime) and the three-phase delivery view, so future changes can be scoped against a shared plan instead of ad hoc.
• Add docs/adr/0001-device-agnostic-runtime.md recording this repositioning as a formal ADR (context, decision, alternatives, consequences).
• Add docs/CONSTITUTION.md capturing the durable architectural invariants that must hold regardless of milestone (Driver contract, tools/ boundary, Scene as the only perception artifact the LLM sees, Planner/Executor split, stateless drivers), so future AI-assisted changes have a stable reference instead of re-deriving these from apex-agent-mvp/design.md.
• Add a research/ track scaffold (001-scene-model .. 007-ui-understanding) with placeholder READMEs describing each track's purpose — no code, no benchmarks yet.
• Introduce a PerceptionProvider port in perception/: the existing OCR+tree fusion (scene_builder.py) becomes the default implementation behind this port, and a NullPerceptionProvider (returns an empty Scene, no OCR dependency required) is added alongside it — so tools/, runtime/, and api/ depend only on the port, never on a specific perception technique, matching the same extension-point pattern as the Driver Registry.
• Codify Hexagonal (Ports-and-Adapters) + DDD layering as the project's governing architecture, with an explicit dependency direction and build order for future milestones: core (domain models/errors, zero framework deps) → driver/device (adapters over external device SDKs) → tools (capability layer) → perception (Scene, mockable via the port above) → storage (timeline/task persistence) → runtime (Planner/Executor, the application layer) → real perception techniques (OCR, vision) → LLM-backed Planner behavior → api (REST/MCP, the outermost adapter). Later milestones (Semantic, World, Skill, Workflow, Agent, Cloud Runtime) must build on top of this direction, not introduce LLM or transport-layer (HTTP/MCP) dependencies into core, driver, device, or tools.
• No runtime/test behavior changes: every existing test in tests/ must continue to pass with only import-path updates, not logic changes.

Capabilities

New Capabilities

• driver-registry: A central, driver-agnostic registry mapping a driver_type string to a driver-factory builder, owned by the driver/ package, used by any config/API surface (console, future CLI, future cloud scheduler) that needs to construct a driver for a device without knowing concrete driver classes.
• perception-provider: A PerceptionProvider port in the perception/ package, with the existing OCR+tree fusion registered as its default implementation and a NullPerceptionProvider added for tests/low-dependency development, so perception techniques are swappable the same way driver types are.

Modified Capabilities

(none — device-management, scene-perception, agent-runtime, task-memory, and mcp-tool-server keep their existing requirements from apex-agent-mvp; this change moves their implementation to new package paths and renames the project around them, but does not change any of their specified behavior.)

Impact

• Moved code: core/driver.py → driver/base.py, core/wda_driver.py → driver/wda_driver.py, core/device_manager.py → device/manager.py; core/models.py and core/errors.py stay in core/ as shared domain types (with ApexAgentError renamed). vision/* → perception/*. SUPPORTED_DRIVER_TYPES/build_driver_factory move from api/console.py into the new driver/registry.py.
• Import updates: all 37 files across core/, tools/, vision/, runtime/, storage/, api/, and tests/ that import core.driver, core.device_manager, core.models, core.errors, or vision.* need updated import paths.
• Config: pyproject.toml — name, add description, and [tool.setuptools.packages.find].include list (driver*, device* replacing implicit core* scope, perception* replacing vision*).
• New docs, no code impact: README.md, docs/ROADMAP.md, docs/adr/0001-device-agnostic-runtime.md, docs/adr/0002-layered-hexagonal-architecture.md, docs/CONSTITUTION.md, research/**/README.md.
• New port: perception/provider.py (PerceptionProvider ABC, NullPerceptionProvider), with scene_builder.py's existing fusion logic wrapped as the default provider — no change to scene-perception's specified behavior.
• Out of scope: no changes to skill-catalog-subscription or web-console (the other two pending, unapplied changes) — they are left as-is and can be re-read against docs/ROADMAP.md later if needed. No second driver (Android/etc.) is implemented in this change; only the registry extension point moves to where a second driver would plug in.