feat: checkpoint device agent runtime milestones
This commit is contained in:
@@ -0,0 +1,45 @@
|
||||
# Constitution
|
||||
|
||||
These invariants are durable project rules. Every future milestone change must
|
||||
state in its `design.md` how it preserves them.
|
||||
|
||||
## Device Boundary
|
||||
|
||||
`Driver` is the only device-capability contract. Concrete drivers translate
|
||||
external SDKs into that contract and do not own task, workflow, or planner
|
||||
state.
|
||||
|
||||
Drivers are stateless with respect to agent tasks. A driver may hold a live
|
||||
connection handle, but task memory, retries, plans, and workflow state belong
|
||||
above the driver layer.
|
||||
|
||||
## Tool Boundary
|
||||
|
||||
`tools/` exposes device capabilities to the runtime. Tools call into
|
||||
device/driver capabilities through abstractions and never import a concrete
|
||||
driver such as `WDADriver`.
|
||||
|
||||
## Perception Boundary
|
||||
|
||||
`Scene` is the only perception artifact the LLM sees. It is produced through
|
||||
`PerceptionProvider`, not by direct calls to OCR, UI tree parsing, or
|
||||
`scene_builder` from runtime and API layers.
|
||||
|
||||
## Runtime Boundary
|
||||
|
||||
The Planner produces a plan. The Executor is the only component that calls
|
||||
tools and handles retries for tool execution.
|
||||
|
||||
LLM dependencies enter at `runtime` through Planner behavior. HTTP and MCP
|
||||
dependencies enter at `api`. No LLM, HTTP, or MCP dependency may appear in
|
||||
`core`, `driver`, `device`, or `tools`.
|
||||
|
||||
## Change Discipline
|
||||
|
||||
Future changes must keep the dependency direction:
|
||||
|
||||
`core` -> `driver`/`device` -> `tools` -> `perception` -> `storage` -> `runtime`
|
||||
-> `api`.
|
||||
|
||||
When a change needs a new external integration, add it at the adapter layer
|
||||
that owns that concern, not at the domain or device boundary.
|
||||
@@ -0,0 +1,88 @@
|
||||
# Roadmap
|
||||
|
||||
This roadmap positions the project as a Device Agent Runtime: a stable runtime
|
||||
for LLM agents to operate real devices through device-agnostic contracts.
|
||||
|
||||
## Delivery Phases
|
||||
|
||||
### Phase 1: Local Runtime Foundation
|
||||
|
||||
Build and stabilize the single-process runtime that can control one real
|
||||
device, perceive its screen, execute tool calls, and persist task history.
|
||||
|
||||
### Phase 2: Intelligence and Reuse
|
||||
|
||||
Add semantic scene understanding, world state, skill learning, workflow
|
||||
orchestration, and multi-agent collaboration on top of the runtime boundaries.
|
||||
|
||||
### Phase 3: Cloud Runtime
|
||||
|
||||
Scale the runtime into a schedulable, multi-device platform with device pools,
|
||||
plugin surfaces, and operational APIs.
|
||||
|
||||
## Milestones
|
||||
|
||||
### Milestone 0: Foundation
|
||||
|
||||
Reposition the project as Device Agent Runtime, split driver/device/perception
|
||||
packages, move driver selection into `driver/registry.py`, add
|
||||
`PerceptionProvider`, and record the architecture invariants.
|
||||
|
||||
### Milestone 1: Device Management
|
||||
|
||||
Already implemented by `apex-agent-mvp` through the `device-management`
|
||||
capability: device registration, status, connection lifecycle, retries, and WDA
|
||||
as the first concrete driver.
|
||||
|
||||
### Milestone 2: Scene Perception
|
||||
|
||||
Already implemented by `apex-agent-mvp` through the `scene-perception`
|
||||
capability: screenshot and UI tree fusion into a single `Scene` with OCR and
|
||||
deduplication.
|
||||
|
||||
### Milestone 3: Agent Runtime
|
||||
|
||||
Already implemented by `apex-agent-mvp` through the `agent-runtime` capability:
|
||||
Planner/Executor structure, tool execution, retry behavior, and task runner.
|
||||
|
||||
### Milestone 4: Task Memory
|
||||
|
||||
Already implemented by `apex-agent-mvp` through the `task-memory` capability:
|
||||
timeline records, screenshots, prompts, tool calls, and task metadata.
|
||||
|
||||
### Milestone 5: Semantic Scene
|
||||
|
||||
Add compact semantic summaries on top of `Scene` so prompts consume page
|
||||
identity, supported intents, and widget purposes rather than raw geometry.
|
||||
|
||||
### Milestone 6: World Model
|
||||
|
||||
Maintain task-relevant state across observations, including durable facts about
|
||||
apps, workflows, and device state transitions.
|
||||
|
||||
### Milestone 7: Skill Learning
|
||||
|
||||
Represent reusable skills, version them, retrieve them by embedding or metadata,
|
||||
and allow new skills to be authored from successful task traces.
|
||||
|
||||
### Milestone 8: Workflow Orchestration
|
||||
|
||||
Add explicit workflow definitions, branching, waiting, and reusable execution
|
||||
plans above the low-level tool layer.
|
||||
|
||||
### Milestone 9: Multi-Agent Runtime
|
||||
|
||||
Support multiple collaborating roles, shared context, review loops, and
|
||||
coordinated execution while preserving the same device/runtime boundaries.
|
||||
|
||||
### Milestone 10: Cloud Runtime
|
||||
|
||||
Add device pools, scheduling, plugins, and platform APIs for distributed
|
||||
execution across many devices.
|
||||
|
||||
## Long-Term Direction
|
||||
|
||||
The long-term v2.0 direction is a "DeviceOS" / Universal Device Runtime: one
|
||||
agent-facing operating surface for many real-world device types. That direction
|
||||
is explicitly not started by this roadmap. Current work remains focused on the
|
||||
local Device Agent Runtime and the milestone sequence above.
|
||||
@@ -0,0 +1,49 @@
|
||||
# ADR 0001: Device-Agnostic Runtime
|
||||
|
||||
## Status
|
||||
|
||||
Accepted
|
||||
|
||||
## Context
|
||||
|
||||
The project began as Apex Agent, centered on iPhone automation. The existing
|
||||
MVP already has a driver-independent `Driver` interface, WDA as the first
|
||||
driver, device lifecycle management, scene perception, runtime execution, task
|
||||
memory, and REST/MCP surfaces. The old names and package layout still implied
|
||||
that iPhone/WDA was the platform rather than the first adapter.
|
||||
|
||||
The OpenSpec change `device-agent-runtime-foundation` records the detailed
|
||||
design decisions for this repositioning.
|
||||
|
||||
## Decision
|
||||
|
||||
Reposition the project as Device Agent Runtime:
|
||||
|
||||
- Keep `core/` for shared domain models and errors.
|
||||
- Move the driver contract and concrete drivers into `driver/`.
|
||||
- Move device lifecycle management into `device/`.
|
||||
- Rename `vision/` to `perception/`.
|
||||
- Move driver-type factory selection into `driver/registry.py`.
|
||||
- Rename the shared error base to `DeviceRuntimeError`.
|
||||
- Treat iPhone/WDA as the first driver, not the product boundary.
|
||||
|
||||
## Alternatives Considered
|
||||
|
||||
Keeping the old Apex Agent/iPhone framing was rejected because it would make
|
||||
future Android, browser, desktop, and cloud-device support look like exceptions
|
||||
instead of first-class adapters.
|
||||
|
||||
Eliminating `core/` entirely was rejected for this change because models such
|
||||
as `Scene`, `Task`, and `Device` are shared across several layers today.
|
||||
Forcing them into `perception/` or `runtime/` would add coupling without
|
||||
behavioral value.
|
||||
|
||||
Keeping a deprecated `ApexAgentError` alias was rejected because there are no
|
||||
external consumers to preserve yet.
|
||||
|
||||
## Consequences
|
||||
|
||||
Future driver support is added in the driver layer and registry. Future
|
||||
perception techniques are added behind `PerceptionProvider`. User-facing
|
||||
runtime behavior remains unchanged, while package names and docs now match the
|
||||
device-agnostic direction.
|
||||
@@ -0,0 +1,53 @@
|
||||
# ADR 0002: Layered Hexagonal Architecture
|
||||
|
||||
## Status
|
||||
|
||||
Accepted
|
||||
|
||||
## Context
|
||||
|
||||
Upcoming milestones add semantic understanding, world state, skill retrieval,
|
||||
workflow orchestration, multi-agent behavior, and cloud scheduling. Those
|
||||
features introduce LLM, embedding, HTTP, MCP, storage, and plugin concerns that
|
||||
must not leak into the device and domain contracts.
|
||||
|
||||
The `device-agent-runtime-foundation` design decision D7 records the governing
|
||||
architecture: Hexagonal / Ports-and-Adapters with DDD-style domain boundaries.
|
||||
|
||||
## Decision
|
||||
|
||||
Use this dependency direction:
|
||||
|
||||
1. `core`: domain models and errors. No framework, LLM, HTTP, MCP, or external
|
||||
device SDK dependencies.
|
||||
2. `driver`: adapters over external device SDKs, implementing the `Driver`
|
||||
contract.
|
||||
3. `device`: device lifecycle management and active driver ownership.
|
||||
4. `tools`: capability layer called by the runtime. Tools use device/driver
|
||||
capabilities and do not import concrete drivers.
|
||||
5. `perception`: produces `Scene` behind `PerceptionProvider`.
|
||||
6. `storage`: persists timeline, task metadata, artifacts, and device config.
|
||||
7. `runtime`: application layer. Planner and Executor orchestrate tools,
|
||||
perception, storage, retries, and task state. LLM-backed planning first
|
||||
enters here.
|
||||
8. `api`: outermost transport adapter for REST, MCP, and future platform APIs.
|
||||
|
||||
Future milestones must build bottom-up through these boundaries. LLM and
|
||||
transport dependencies may not be introduced into `core`, `driver`, `device`,
|
||||
or `tools`.
|
||||
|
||||
## Alternatives Considered
|
||||
|
||||
Leaving the layering implicit was rejected because upcoming AI-heavy milestones
|
||||
will otherwise tend to call LLM or transport APIs from convenient lower layers.
|
||||
|
||||
Adding lint enforcement now was deferred. The written architecture is the
|
||||
current source of truth; mechanical enforcement can be added when a concrete
|
||||
boundary violation or second-driver pressure appears.
|
||||
|
||||
## Consequences
|
||||
|
||||
The runtime can add smarter planning and richer perception without changing the
|
||||
device capability contract. API and cloud concerns stay outside domain and
|
||||
adapter layers. Future OpenSpec designs must state how they uphold this
|
||||
dependency direction.
|
||||
Reference in New Issue
Block a user