89 lines
3.0 KiB
Markdown
89 lines
3.0 KiB
Markdown
# Roadmap
|
|
|
|
This roadmap positions the project as a Device Agent Runtime: a stable runtime
|
|
for LLM agents to operate real devices through device-agnostic contracts.
|
|
|
|
## Delivery Phases
|
|
|
|
### Phase 1: Local Runtime Foundation
|
|
|
|
Build and stabilize the single-process runtime that can control one real
|
|
device, perceive its screen, execute tool calls, and persist task history.
|
|
|
|
### Phase 2: Intelligence and Reuse
|
|
|
|
Add semantic scene understanding, world state, skill learning, workflow
|
|
orchestration, and multi-agent collaboration on top of the runtime boundaries.
|
|
|
|
### Phase 3: Cloud Runtime
|
|
|
|
Scale the runtime into a schedulable, multi-device platform with device pools,
|
|
plugin surfaces, and operational APIs.
|
|
|
|
## Milestones
|
|
|
|
### Milestone 0: Foundation
|
|
|
|
Reposition the project as Device Agent Runtime, split driver/device/perception
|
|
packages, move driver selection into `driver/registry.py`, add
|
|
`PerceptionProvider`, and record the architecture invariants.
|
|
|
|
### Milestone 1: Device Management
|
|
|
|
Already implemented by `apex-agent-mvp` through the `device-management`
|
|
capability: device registration, status, connection lifecycle, retries, and WDA
|
|
as the first concrete driver.
|
|
|
|
### Milestone 2: Scene Perception
|
|
|
|
Already implemented by `apex-agent-mvp` through the `scene-perception`
|
|
capability: screenshot and UI tree fusion into a single `Scene` with OCR and
|
|
deduplication.
|
|
|
|
### Milestone 3: Agent Runtime
|
|
|
|
Already implemented by `apex-agent-mvp` through the `agent-runtime` capability:
|
|
Planner/Executor structure, tool execution, retry behavior, and task runner.
|
|
|
|
### Milestone 4: Task Memory
|
|
|
|
Already implemented by `apex-agent-mvp` through the `task-memory` capability:
|
|
timeline records, screenshots, prompts, tool calls, and task metadata.
|
|
|
|
### Milestone 5: Semantic Scene
|
|
|
|
Add compact semantic summaries on top of `Scene` so prompts consume page
|
|
identity, supported intents, and widget purposes rather than raw geometry.
|
|
|
|
### Milestone 6: World Model
|
|
|
|
Maintain task-relevant state across observations, including durable facts about
|
|
apps, workflows, and device state transitions.
|
|
|
|
### Milestone 7: Skill Learning
|
|
|
|
Represent reusable skills, version them, retrieve them by embedding or metadata,
|
|
and allow new skills to be authored from successful task traces.
|
|
|
|
### Milestone 8: Workflow Orchestration
|
|
|
|
Add explicit workflow definitions, branching, waiting, and reusable execution
|
|
plans above the low-level tool layer.
|
|
|
|
### Milestone 9: Multi-Agent Runtime
|
|
|
|
Support multiple collaborating roles, shared context, review loops, and
|
|
coordinated execution while preserving the same device/runtime boundaries.
|
|
|
|
### Milestone 10: Cloud Runtime
|
|
|
|
Add device pools, scheduling, plugins, and platform APIs for distributed
|
|
execution across many devices.
|
|
|
|
## Long-Term Direction
|
|
|
|
The long-term v2.0 direction is a "DeviceOS" / Universal Device Runtime: one
|
|
agent-facing operating surface for many real-world device types. That direction
|
|
is explicitly not started by this roadmap. Current work remains focused on the
|
|
local Device Agent Runtime and the milestone sequence above.
|