3.3 KiB
Roadmap
This roadmap positions the project as a Device Agent Runtime: a stable runtime for LLM agents to operate real devices through device-agnostic contracts.
Delivery Phases
Phase 1: Local Runtime Foundation
Build and stabilize the single-process runtime that can control one real device, perceive its screen, execute tool calls, and persist task history.
Phase 2: Intelligence and Reuse
Add semantic scene understanding, world state, skill learning, workflow orchestration, and multi-agent collaboration on top of the runtime boundaries.
Phase 3: Cloud Runtime
Scale the runtime into a schedulable, multi-device platform with device pools, plugin surfaces, and operational APIs.
Milestones
Milestone 0: Foundation
Reposition the project as Device Agent Runtime, split driver/device/perception
packages, move driver selection into driver/registry.py, add
PerceptionProvider, and record the architecture invariants.
Milestone 1: Device Management
Already implemented by apex-agent-mvp through the device-management
capability: device registration, status, connection lifecycle, retries, and WDA
as the first concrete driver.
Milestone 2: Scene Perception
Already implemented by apex-agent-mvp through the scene-perception
capability: screenshot and UI tree fusion into a single Scene with OCR and
deduplication.
Milestone 3: Agent Runtime
Already implemented by apex-agent-mvp through the agent-runtime capability:
Planner/Executor structure, tool execution, retry behavior, and task runner.
Milestone 4: Task Memory
Already implemented by apex-agent-mvp through the task-memory capability:
timeline records, screenshots, prompts, tool calls, and task metadata.
Milestone 5: Semantic Scene
Add compact semantic summaries on top of Scene so prompts consume page
identity, supported intents, and widget purposes rather than raw geometry.
Milestone 6: World Model
Maintain task-relevant state across observations, including durable facts about apps, workflows, and device state transitions.
Milestone 7: Skill Learning
Represent reusable skills, version them, retrieve them by embedding or metadata, and allow new skills to be authored from successful task traces.
Milestone 8: Workflow Orchestration
Add explicit workflow definitions, branching, waiting, and reusable execution plans above the low-level tool layer.
Milestone 9: Multi-Agent Runtime
Support multiple collaborating roles, shared context, review loops, and coordinated execution while preserving the same device/runtime boundaries.
Milestone 10: Cloud Runtime
Add device pools, scheduling, plugins, and platform APIs for distributed execution across many devices.
The packaging baseline is complete: device-agent-runtime remains the root uv
workspace member, while the existing cloud modules and SDK are published by
the one-way dependent device-cloud-platform member under
packages/cloud-platform. A deployable control plane and Host Agent execution
loop remain separate follow-up work.
Long-Term Direction
The long-term v2.0 direction is a "DeviceOS" / Universal Device Runtime: one agent-facing operating surface for many real-world device types. That direction is explicitly not started by this roadmap. Current work remains focused on the local Device Agent Runtime and the milestone sequence above.