feat: checkpoint device agent runtime milestones
This commit is contained in:
@@ -0,0 +1,2 @@
|
||||
schema: spec-driven
|
||||
created: 2026-07-06
|
||||
@@ -0,0 +1,82 @@
|
||||
## Context
|
||||
|
||||
`apex-agent-mvp` is code-complete (35/38 tasks; the remaining 3 require a physical iPhone and are unaffected by this change) but was never archived — `openspec/specs/` is still empty, so its capability specs (`device-management`, `scene-perception`, `agent-runtime`, `task-memory`, `mcp-tool-server`) exist only as pending deltas under `openspec/changes/apex-agent-mvp/specs/`. The working codebase already reflects a mostly driver-agnostic design: `Driver` is an ABC with no framework types leaking through, `WDADriver` is the only implementation, `DeviceManager` tracks devices by an opaque `driver_type` string, and `Scene` (not raw XML/OCR boxes) is the only thing the LLM-facing tools ever touch. The gaps are cosmetic-but-real: the package holding `Driver`+`DeviceManager` is called `core/` (a grab-bag name), the perception package is called `vision/` (names the technique, not the capability), the shared error base is literally `ApexAgentError`, and the one true "pick a driver by type" extension point (`SUPPORTED_DRIVER_TYPES` in `api/console.py`) lives in the API layer instead of the driver layer. Two other changes (`skill-catalog-subscription`, `web-console`) are proposed but unapplied and untouched by this change.
|
||||
|
||||
Two stakeholders: the LLM/agent runtime consuming the driver/device/tools/perception layers (must see zero behavior change), and future contributors/AI coding agents who will read `docs/CONSTITUTION.md` and `docs/ROADMAP.md` before adding the next driver or milestone.
|
||||
|
||||
## Goals / Non-Goals
|
||||
|
||||
**Goals:**
|
||||
- Zero behavior change: every existing test passes after the move with only import-path edits, not logic edits.
|
||||
- Make the physical package layout match the mental model: `driver/` (interface + implementations), `device/` (lifecycle/registry of *devices*, not drivers), `core/` (truly shared models/errors only).
|
||||
- Give driver-type registration a home in `driver/` so a second driver (Android, browser, ...) never requires touching `api/console.py`.
|
||||
- Give perception-technique registration the same treatment via a `PerceptionProvider` port, so OCR is one swappable implementation, not the pipeline's identity.
|
||||
- Produce durable planning artifacts (`ROADMAP.md`, two ADRs, `CONSTITUTION.md`, `research/` scaffold) that later milestone changes can be scoped against, instead of re-deriving architecture context from `apex-agent-mvp/design.md` each time.
|
||||
- Codify a **Hexagonal (Ports-and-Adapters) + DDD layering** as the project's governing architecture, with an explicit dependency direction (domain has zero framework/LLM/HTTP dependencies; adapters depend on domain, never the reverse) and a bottom-up build order that future milestone changes are expected to respect: `core` (domain) → `driver`/`device` (adapters) → `tools` (capability/ports layer) → `perception` (Scene, mockable) → `storage` → `runtime` (application/orchestration) → real perception techniques → LLM-backed planning → `api` (outermost adapter).
|
||||
|
||||
**Non-Goals:**
|
||||
- No second driver implementation (Android/browser/Windows) — only the extension point moves.
|
||||
- No changes to `skill-catalog-subscription` or `web-console` proposals/specs.
|
||||
- No change to `agent-runtime`, `task-memory`, or `mcp-tool-server` behavior/specs — only their import paths shift if they reference `core.driver`/`core.device_manager`/`vision.*`.
|
||||
- No archiving of `apex-agent-mvp` as part of this change — that remains a separate decision for the user (real-device verification tasks 7.1–7.3 are still outstanding). Note `apex-agent-mvp` already bundled Perception (real OCR), Planning (LLM), and API/MCP into one MVP change, ahead of the layering this change now codifies — that is accepted as-is (not rebuilt or reordered); the layering/build-order constraint below governs *future* milestone changes (Semantic, World, Skill, Workflow, Agent, Cloud Runtime), not a retroactive rewrite of already-shipped code.
|
||||
- No CI/lint pipeline changes beyond what's needed to keep `pytest` green — mechanical enforcement of the layering (e.g. import-linter contracts) is explicitly deferred, not done now.
|
||||
|
||||
## Decisions
|
||||
|
||||
### D1: Split `core/` into `driver/` + `device/`, keep `core/` as a shared-models-only package
|
||||
`driver/base.py` gets the `Driver` ABC (from `core/driver.py`); `driver/wda_driver.py` gets `WDADriver` (from `core/wda_driver.py`); `device/manager.py` gets `DeviceManager`/`DriverFactory`/`DEFAULT_MANAGER` (from `core/device_manager.py`). `core/models.py` (Bounds, Device, SceneElement, Scene, Task, Step) and `core/errors.py` stay in `core/`, since they're shared across driver, device, perception, runtime, and storage — not device- or driver-specific.
|
||||
- **Alternative considered**: Eliminate `core/` entirely and push models/errors into whichever package "owns" them most (e.g. `Scene`/`SceneElement` → `perception/`, `Task`/`Step` → `runtime/`). Rejected for this change — it would force `device/` to import from `perception/` and `runtime/` to import from `perception/`, creating cross-package coupling that doesn't exist today; splitting shared models is a legitimate follow-up but is a behavior-neutral naming change's job to avoid, not force.
|
||||
|
||||
### D2: Rename `vision/` → `perception/`
|
||||
Straight package rename, same file names inside (`ocr.py`, `ui_parser.py`, `scene_builder.py`, `icon_detector.py`). No merging or splitting.
|
||||
- **Alternative considered**: Keep `vision/` and only rename the *pipeline* concept in docs. Rejected — the proposal's own roadmap explicitly calls this stage "Perception," and leaving the package named after one input signal (OCR/vision) while `scene_builder.py` already fuses tree+OCR+vision is the exact naming drift this change exists to fix.
|
||||
|
||||
### D3: Driver Registry moves into `driver/registry.py`, `api/console.py` imports from it
|
||||
New `driver/registry.py` holds `SUPPORTED_DRIVER_TYPES: dict[str, DriverFactoryBuilder]` and `build_driver_factory(driver_type, connection_info)`, moved verbatim from `api/console.py` (currently lines ~17-50). `api/console.py` imports `build_driver_factory` from `driver.registry` instead of defining it. Adding `AndroidDriver` later means adding one entry to `driver/registry.py`, not touching `api/console.py`.
|
||||
- **Alternative considered**: Leave the registry in `api/console.py` and only document the intent to move it later. Rejected — this is precisely the kind of small, cheap, high-leverage structural fix Milestone 0 exists to make before a second driver shows up and someone has to thread a device-agnostic factory builder through an API-layer dict under time pressure.
|
||||
|
||||
### D4: Rename `ApexAgentError` → `DeviceRuntimeError`
|
||||
`core/errors.py`'s base class and its subclasses (`DriverError`, `DeviceNotFoundError`, `DeviceOfflineError`, `DeviceBusyError`, `ElementNotFoundError`, `TaskFailedError`) keep their names; only the base changes from `ApexAgentError` to `DeviceRuntimeError`. All `except ApexAgentError` / `raise ApexAgentError` call sites across `api/`, `tools/`, `runtime/`, `tests/` update accordingly.
|
||||
- **Alternative considered**: Keep `ApexAgentError` as a deprecated alias (`ApexAgentError = DeviceRuntimeError`) for backward compatibility. Rejected — there are no external consumers of this exception type yet (nothing has shipped), so a compatibility shim adds dead code for a compatibility need that doesn't exist.
|
||||
|
||||
### D5: Rebrand via `pyproject.toml` + root `README.md`, not a source-wide string sweep
|
||||
`pyproject.toml`'s `name` becomes `device-agent-runtime` and a `description` field is added; a root `README.md` (currently absent) states the new positioning and links to `docs/ROADMAP.md`. `apex-agent-mvp/proposal.md` and `design.md` are left untouched as a historical record of the original framing — they describe a change that already happened, and rewriting history in a planning artifact adds no value.
|
||||
- **Alternative considered**: Grep-and-replace every "Apex Agent" / "IPA" string across existing `openspec/changes/apex-agent-mvp/**` docs. Rejected per the chosen change scope (foundation docs + restructure only, not reconciling other pending changes) — those files are a record of a past decision, not living documentation.
|
||||
|
||||
### D6: Roadmap, ADR, Constitution, and research/ track are pure additions, no code coupling
|
||||
`docs/ROADMAP.md`, `docs/adr/0001-device-agnostic-runtime.md`, `docs/CONSTITUTION.md`, and `research/00N-*/README.md` are net-new Markdown files with no imports from or references into runtime code, so they carry zero risk to the passing test suite.
|
||||
- **Alternative considered**: Encode `docs/CONSTITUTION.md`'s invariants as enforced lint rules (e.g. import-linter contracts forbidding `tools/` from importing `driver/` directly). Deferred, not rejected — worth doing once there's a second driver to actually violate the boundary against; adding enforcement machinery for a boundary nothing has crossed yet is premature for a foundation change.
|
||||
|
||||
### D7: Adopt Hexagonal + DDD layering as the governing architecture, recorded in a second ADR
|
||||
`docs/adr/0002-layered-hexagonal-architecture.md` records the dependency direction and build order as project law: `core` (domain: `Bounds`/`Device`/`Scene`/`Task`/`Step`/errors) has zero framework, LLM, or transport dependencies; `driver`/`device` are adapters translating external device SDKs (Appium/WDA today) into the domain's `Driver` contract; `tools` is the capability/port layer the Agent Runtime calls; `perception` produces `Scene` behind the `PerceptionProvider` port (D8); `storage` persists timeline/task state; `runtime` (Planner/Executor) is the application layer that orchestrates the above; only at `runtime`'s Planner does an LLM enter the picture, and only `api` (REST/MCP) is HTTP/MCP-transport-aware. This is recorded as a standing constraint for *future* milestone changes (Semantic, World, Skill, Workflow, Agent, Cloud Runtime): none of them may introduce LLM or transport dependencies into `core`, `driver`, `device`, or `tools`.
|
||||
- **Alternative considered**: Leave this as implicit convention (as it already mostly was) rather than a written ADR. Rejected — the explicit trigger for writing it down now is that `apex-agent-mvp` already shipped Perception+Planning+API bundled in one MVP change, i.e. the layering was followed loosely, not strictly by construction; upcoming AI-heavy milestones (Semantic Scene's LLM call, Skill embedding retrieval, Multi-Agent roles) are exactly where it's cheapest to keep LLM/transport concerns out of the domain/adapter layers *before* they're written, not after.
|
||||
|
||||
### D8: `PerceptionProvider` port mirrors the Driver Registry pattern (D3)
|
||||
`perception/provider.py` defines a `PerceptionProvider` ABC with one method, `build_scene(screenshot, tree) -> Scene`. The existing OCR+tree fusion in `scene_builder.py` is wrapped as the default implementation (registered, not rewritten — same behavior, same `Scene` output). A `NullPerceptionProvider` returning an empty `Scene` (correct `width`/`height`, no elements) is added for tests and any environment without OCR dependencies installed. `tools/describe_screen.py` and `runtime/` depend on the port type, not on `scene_builder` directly.
|
||||
- **Alternative considered**: Skip the port and let callers import `scene_builder.build_scene()` directly (as today), documenting only that *conceptually* it's swappable. Rejected — the whole point of D7's constraint is that swapping/mocking perception must be a real, exercised code path (useful today for fast tests without OCR installed) rather than an aspiration that nothing currently proves; this mirrors why the Driver Registry (D3) was made a real registry instead of a documented convention.
|
||||
|
||||
## Risks / Trade-offs
|
||||
|
||||
- **[Risk]** Renaming `core.driver`/`core.device_manager`/`vision.*` import paths across 37 files by hand risks missing one and breaking an import at runtime rather than at test time → **Mitigation**: run the full `pytest` suite (all of `tests/`) after the move as the acceptance gate in `tasks.md`; a missed import surfaces immediately as a collection error, not a silent behavior change.
|
||||
- **[Risk]** Moving `SUPPORTED_DRIVER_TYPES`/`build_driver_factory` out of `api/console.py` could break `tests/test_console_api.py` if it patches/imports those names directly → **Mitigation**: check `test_console_api.py`'s imports as part of the move and update them alongside the production code, in the same task.
|
||||
- **[Risk]** Wrapping `scene_builder.py` behind `PerceptionProvider` could subtly change its return value if the wrapper reshapes data → **Mitigation**: the default provider must call the existing `scene_builder` function unmodified and return its result as-is; `tests/test_scene_builder.py` passing unchanged is the acceptance check, not a new test suite.
|
||||
- **[Trade-off]** Codifying D7's layering as an ADR/constitution now, without lint enforcement, means it's a convention future contributors (human or AI) must read and follow, not something that fails a build if violated → acceptable per D6's reasoning: enforcement machinery is worth adding once there's a concrete violation to enforce against (e.g. when a Semantic Scene change is tempted to call an LLM SDK from within `core`), not preemptively.
|
||||
- **[Trade-off]** Keeping `core/` alive (rather than eliminating it) means the package boundary story is "driver / device / core (shared) / perception / runtime / storage / api" — one more package than the roadmap's illustrative "driver/device/runtime" — acceptable because this codebase already has more layers than the roadmap's Milestone-1-only sketch assumed, and forcing models into whichever package "feels right" would create the cross-package coupling described in D1.
|
||||
- **[Trade-off]** Not archiving `apex-agent-mvp` as part of this change means `openspec/specs/` stays empty and this change's own capability (`driver-registry`) has no prior baseline to diff against — acceptable since `driver-registry` is genuinely new (ADDED, not MODIFIED) and archival ordering is the user's call, not a blocker for this change's own correctness.
|
||||
|
||||
## Migration Plan
|
||||
|
||||
1. Create new packages/files (`driver/`, `device/`, `perception/`) via `git mv` (preserves history) rather than copy+delete.
|
||||
2. Update every import site (37 files) from `core.driver`/`core.device_manager`/`vision.*` to the new paths; rename `ApexAgentError` → `DeviceRuntimeError` at all call sites.
|
||||
3. Add `perception/provider.py` (`PerceptionProvider`, `NullPerceptionProvider`, default provider wrapping `scene_builder.py`); point `tools/describe_screen.py`/`runtime/` at the port.
|
||||
4. Move `SUPPORTED_DRIVER_TYPES`/`build_driver_factory` into `driver/registry.py`; update `api/console.py`'s imports.
|
||||
5. Update `pyproject.toml` (`name`, `description`, `packages.find.include`).
|
||||
6. Run `pytest` — must be 100% green with no test-content changes (only import updates inside `tests/` itself where needed).
|
||||
7. Add `README.md`, `docs/ROADMAP.md`, `docs/adr/0001-device-agnostic-runtime.md`, `docs/adr/0002-layered-hexagonal-architecture.md`, `docs/CONSTITUTION.md`, `research/00N-*/README.md`.
|
||||
8. Rollback: since every step is a rename/move plus additive docs with no data migration, reverting is `git revert` of the commit(s); no runtime state or external system is touched.
|
||||
|
||||
## Open Questions
|
||||
|
||||
- Whether `core/` should eventually be renamed too (e.g. `shared/` or `domain/`) once it's clearer which models belong to which future layer (World Model in Milestone 6 will likely want to own more of `core/models.py`) — left open, not blocking this change.
|
||||
- Whether `docs/CONSTITUTION.md` should later be enforced mechanically (import-linter, custom lint rule) once a second driver exists to validate the boundary against — deferred to a future change per D6.
|
||||
- Whether the Semantic Scene (Milestone 5), World Model (Milestone 6), and Skill Learning (Milestone 7) changes — currently being drafted in parallel — correctly keep their LLM/embedding calls confined to a `runtime`-level (or new dedicated) layer rather than `perception`/`core`, per D7's constraint; worth a follow-up read once those changes are drafted.
|
||||
@@ -0,0 +1,36 @@
|
||||
## Why
|
||||
|
||||
This project started as "Apex Agent," an iPhone-specific automation platform, and `apex-agent-mvp` already implements a driver-independent `Driver` interface, a `WDADriver`, a `DeviceManager`, a Scene perception pipeline, an Agent Runtime, task-memory timelines, and an MCP/REST surface. The actual long-term value isn't "control an iPhone" — it's giving an LLM a stable way to operate *any* real-world device, with iPhone/WDA as just the first driver. Nothing about the implemented behavior needs to change to realize this, but the naming, package layout, and a couple of remaining structural gaps still assume "iPhone is the platform" rather than "iPhone is one driver." Fixing this now — before Android/Chrome/Windows drivers or a skill/workflow/multi-agent layer accrete on top — is far cheaper than un-tangling it later. This change is Milestone 0 (Foundation) of a longer roadmap: establish the device-agnostic runtime shape and the planning artifacts (roadmap, ADR, research track) that later milestones (Perception, Execution, Planning, Semantic, World, Skill, Workflow, Agent, Cloud Runtime) will build on.
|
||||
|
||||
## What Changes
|
||||
|
||||
- Reorganize `core/` into a `driver/` package (the `Driver` interface + concrete driver implementations, `WDADriver` first) and a `device/` package (`DeviceManager` and device lifecycle), leaving only genuinely shared domain models/errors in `core/`. No behavior changes — this is a structural move plus import updates.
|
||||
- Rename `vision/` to `perception/` so the screenshot → OCR → tree → Scene pipeline is named for what it produces (perception of a device's screen), not for one technique (OCR) inside it.
|
||||
- Rename the `ApexAgentError` exception base (and its "Apex Agent" framing in docstrings/comments) to a device-agnostic name, since it is the one place the old iPhone-only brand is baked into code rather than docs.
|
||||
- Relocate the driver-type-to-factory registry (currently `SUPPORTED_DRIVER_TYPES`/`build_driver_factory` inside `api/console.py`) into the `driver/` package as a first-class **Driver Registry**, so adding a new driver type (Android, browser, ...) never requires touching the API/console layer. **BREAKING**: `api/console.py`'s import path for driver-factory construction changes.
|
||||
- Rebrand the project from "Apex Agent" to **Device Agent Runtime** (working name; long-term direction is a "DeviceOS"-style universal device runtime, recorded as a roadmap milestone, not implemented now): update `pyproject.toml` project name/description, and add a root `README.md` stating the new positioning.
|
||||
- Add `docs/ROADMAP.md` capturing the milestone sequence (Foundation → Device → Perception → Execution → Planning → Semantic → World → Skill → Workflow → Agent → Cloud Runtime) and the three-phase delivery view, so future changes can be scoped against a shared plan instead of ad hoc.
|
||||
- Add `docs/adr/0001-device-agnostic-runtime.md` recording this repositioning as a formal ADR (context, decision, alternatives, consequences).
|
||||
- Add `docs/CONSTITUTION.md` capturing the durable architectural invariants that must hold regardless of milestone (Driver contract, `tools/` boundary, Scene as the only perception artifact the LLM sees, Planner/Executor split, stateless drivers), so future AI-assisted changes have a stable reference instead of re-deriving these from `apex-agent-mvp/design.md`.
|
||||
- Add a `research/` track scaffold (`001-scene-model` .. `007-ui-understanding`) with placeholder READMEs describing each track's purpose — no code, no benchmarks yet.
|
||||
- Introduce a **`PerceptionProvider` port** in `perception/`: the existing OCR+tree fusion (`scene_builder.py`) becomes the default implementation behind this port, and a `NullPerceptionProvider` (returns an empty `Scene`, no OCR dependency required) is added alongside it — so `tools/`, `runtime/`, and `api/` depend only on the port, never on a specific perception technique, matching the same extension-point pattern as the Driver Registry.
|
||||
- Codify **Hexagonal (Ports-and-Adapters) + DDD layering as the project's governing architecture**, with an explicit dependency direction and build order for future milestones: `core` (domain models/errors, zero framework deps) → `driver`/`device` (adapters over external device SDKs) → `tools` (capability layer) → `perception` (Scene, mockable via the port above) → `storage` (timeline/task persistence) → `runtime` (Planner/Executor, the application layer) → real perception techniques (OCR, vision) → LLM-backed Planner behavior → `api` (REST/MCP, the outermost adapter). Later milestones (Semantic, World, Skill, Workflow, Agent, Cloud Runtime) must build on top of this direction, not introduce LLM or transport-layer (HTTP/MCP) dependencies into `core`, `driver`, `device`, or `tools`.
|
||||
- No runtime/test behavior changes: every existing test in `tests/` must continue to pass with only import-path updates, not logic changes.
|
||||
|
||||
## Capabilities
|
||||
|
||||
### New Capabilities
|
||||
- `driver-registry`: A central, driver-agnostic registry mapping a `driver_type` string to a driver-factory builder, owned by the `driver/` package, used by any config/API surface (console, future CLI, future cloud scheduler) that needs to construct a driver for a device without knowing concrete driver classes.
|
||||
- `perception-provider`: A `PerceptionProvider` port in the `perception/` package, with the existing OCR+tree fusion registered as its default implementation and a `NullPerceptionProvider` added for tests/low-dependency development, so perception techniques are swappable the same way driver types are.
|
||||
|
||||
### Modified Capabilities
|
||||
(none — `device-management`, `scene-perception`, `agent-runtime`, `task-memory`, and `mcp-tool-server` keep their existing requirements from `apex-agent-mvp`; this change moves their implementation to new package paths and renames the project around them, but does not change any of their specified behavior.)
|
||||
|
||||
## Impact
|
||||
|
||||
- **Moved code**: `core/driver.py` → `driver/base.py`, `core/wda_driver.py` → `driver/wda_driver.py`, `core/device_manager.py` → `device/manager.py`; `core/models.py` and `core/errors.py` stay in `core/` as shared domain types (with `ApexAgentError` renamed). `vision/*` → `perception/*`. `SUPPORTED_DRIVER_TYPES`/`build_driver_factory` move from `api/console.py` into the new `driver/registry.py`.
|
||||
- **Import updates**: all 37 files across `core/`, `tools/`, `vision/`, `runtime/`, `storage/`, `api/`, and `tests/` that import `core.driver`, `core.device_manager`, `core.models`, `core.errors`, or `vision.*` need updated import paths.
|
||||
- **Config**: `pyproject.toml` — `name`, add `description`, and `[tool.setuptools.packages.find].include` list (`driver*`, `device*` replacing implicit `core*` scope, `perception*` replacing `vision*`).
|
||||
- **New docs, no code impact**: `README.md`, `docs/ROADMAP.md`, `docs/adr/0001-device-agnostic-runtime.md`, `docs/adr/0002-layered-hexagonal-architecture.md`, `docs/CONSTITUTION.md`, `research/**/README.md`.
|
||||
- **New port**: `perception/provider.py` (`PerceptionProvider` ABC, `NullPerceptionProvider`), with `scene_builder.py`'s existing fusion logic wrapped as the default provider — no change to `scene-perception`'s specified behavior.
|
||||
- **Out of scope**: no changes to `skill-catalog-subscription` or `web-console` (the other two pending, unapplied changes) — they are left as-is and can be re-read against `docs/ROADMAP.md` later if needed. No second driver (Android/etc.) is implemented in this change; only the registry extension point moves to where a second driver would plug in.
|
||||
@@ -0,0 +1,19 @@
|
||||
## ADDED Requirements
|
||||
|
||||
### Requirement: Driver type registry lives in the driver layer
|
||||
The system SHALL provide a registry, owned by the `driver` package, that maps a `driver_type` string (e.g. `"wda"`) to a builder function producing a `DriverFactory` for that type, so that any caller needing to construct a driver for a device does so without importing a concrete driver class directly.
|
||||
|
||||
#### Scenario: Building a factory for a known driver type
|
||||
- **WHEN** a caller requests a driver factory for `driver_type="wda"` with connection info (e.g. `server_url`, `udid`)
|
||||
- **THEN** the registry returns a `DriverFactory` that, when invoked, constructs a working `WDADriver` configured with that connection info
|
||||
|
||||
#### Scenario: Building a factory for an unknown driver type
|
||||
- **WHEN** a caller requests a driver factory for a `driver_type` that is not registered
|
||||
- **THEN** the registry raises a clear error naming the unsupported `driver_type`, instead of returning `None` or a factory that fails later at connect time
|
||||
|
||||
### Requirement: Adding a driver type requires no changes outside the driver layer
|
||||
The system SHALL allow a new driver type to be added by registering it in the `driver` package's registry alone; no other package (`api/`, `device/`, `tools/`, `runtime/`) SHALL need code changes to support constructing devices of the new type.
|
||||
|
||||
#### Scenario: API layer is agnostic to registered driver types
|
||||
- **WHEN** the console config API resolves a `driver_type` string into a driver factory to register a device
|
||||
- **THEN** it does so by calling into the driver registry rather than maintaining its own mapping of `driver_type` to concrete driver classes
|
||||
@@ -0,0 +1,22 @@
|
||||
## ADDED Requirements
|
||||
|
||||
### Requirement: Perception is exposed through a swappable provider port
|
||||
The system SHALL define a `PerceptionProvider` interface (`build_scene`) in the `perception` package that any concrete perception implementation (the existing OCR+tree fusion, a future cloud vision API, a future null/mock provider) must satisfy identically, so `tools/`, `runtime/`, and `api/` depend only on the port, never on a specific perception technique.
|
||||
|
||||
#### Scenario: Building a scene through the default provider
|
||||
- **WHEN** a caller requests a Scene for a screenshot and UI tree via the default (OCR+tree fusion) provider
|
||||
- **THEN** the provider returns a `Scene` identical in shape and content to what the existing fusion logic already produces — no change to `scene-perception`'s specified behavior
|
||||
|
||||
### Requirement: A null perception provider is available for testing and low-dependency development
|
||||
The system SHALL provide a `NullPerceptionProvider` that returns an empty `Scene` (correct screen width/height, zero elements) without requiring OCR/vision dependencies to be installed or invoked.
|
||||
|
||||
#### Scenario: Running without OCR dependencies installed
|
||||
- **WHEN** the runtime is configured to use `NullPerceptionProvider` (e.g. in a test or a minimal development environment)
|
||||
- **THEN** `describe_screen`/Agent Runtime calls succeed and return an empty Scene instead of failing due to missing OCR dependencies
|
||||
|
||||
### Requirement: Adding a perception technique requires no changes outside the perception layer
|
||||
The system SHALL allow a new perception provider (e.g. a cloud vision API) to be added by registering it within the `perception` package alone; `tools/`, `runtime/`, and `api/` SHALL NOT require code changes to use a newly registered provider.
|
||||
|
||||
#### Scenario: Runtime is agnostic to which provider is active
|
||||
- **WHEN** the Agent Runtime requests a Scene for the current screenshot and UI tree
|
||||
- **THEN** it does so through the `PerceptionProvider` port without importing or knowing about `scene_builder.py`, OCR, or any other concrete technique
|
||||
@@ -0,0 +1,54 @@
|
||||
## 1. Split `core/` into `driver/` and `device/`
|
||||
|
||||
- [x] 1.1 `git mv core/driver.py driver/base.py`, add `driver/__init__.py`, update its internal imports if any
|
||||
- [x] 1.2 `git mv core/wda_driver.py driver/wda_driver.py`, update its import of `core.driver` → `driver.base`
|
||||
- [x] 1.3 `git mv core/device_manager.py device/manager.py`, add `device/__init__.py`, update its imports of `core.driver`/`core.errors`/`core.models` → `driver.base`/`core.errors`/`core.models`
|
||||
- [x] 1.4 Confirm `core/__init__.py`, `core/models.py`, `core/errors.py` remain in place unchanged (shared domain types only)
|
||||
|
||||
## 2. Rename `vision/` to `perception/`
|
||||
|
||||
- [x] 2.1 `git mv vision perception` (carries `ocr.py`, `ui_parser.py`, `scene_builder.py`, `icon_detector.py`, `__init__.py`)
|
||||
- [x] 2.2 Update internal imports within the renamed package if any reference `vision.*`
|
||||
|
||||
## 3. Introduce the `PerceptionProvider` port
|
||||
|
||||
- [x] 3.1 Create `perception/provider.py` with a `PerceptionProvider` ABC (`build_scene(screenshot, tree) -> Scene`)
|
||||
- [x] 3.2 Add a default provider that wraps the existing `scene_builder.py` fusion logic unmodified (same inputs, same `Scene` output — no behavior change)
|
||||
- [x] 3.3 Add `NullPerceptionProvider` returning an empty `Scene` (correct `width`/`height`, zero elements), usable without any OCR dependency installed
|
||||
- [x] 3.4 Point `tools/describe_screen.py` (and any other caller of `scene_builder` directly) at the `PerceptionProvider` port instead of importing `scene_builder` directly
|
||||
- [x] 3.5 Confirm `tests/test_scene_builder.py` still passes unchanged against the default provider
|
||||
|
||||
## 4. Relocate the Driver Registry
|
||||
|
||||
- [x] 4.1 Create `driver/registry.py`; move `SUPPORTED_DRIVER_TYPES`, `DriverFactoryBuilder`, `build_wda_driver_factory`, and `build_driver_factory` out of `api/console.py` into it, importing `WDADriver`/`WDADriverConfig` from `driver.wda_driver` and `DriverFactory` from `device.manager`
|
||||
- [x] 4.2 Update `api/console.py` to import `build_driver_factory` (and `SUPPORTED_DRIVER_TYPES` if referenced) from `driver.registry` instead of defining them locally
|
||||
- [x] 4.3 Update `tests/test_console_api.py` (and any other test importing these names from `api.console`) to import from `driver.registry` where appropriate
|
||||
|
||||
## 5. Rename the shared error base
|
||||
|
||||
- [x] 5.1 In `core/errors.py`, rename `ApexAgentError` → `DeviceRuntimeError` (subclasses `DriverError`, `DeviceNotFoundError`, `DeviceOfflineError`, `DeviceBusyError`, `ElementNotFoundError`, `TaskFailedError` keep their names, just re-parented)
|
||||
- [x] 5.2 Update every `ApexAgentError` reference across `api/`, `tools/`, `runtime/`, `core/`, `driver/`, `device/`, `perception/`, and `tests/` to `DeviceRuntimeError`
|
||||
|
||||
## 6. Fix up all import sites and re-point package config
|
||||
|
||||
- [x] 6.1 Grep the full tree for `core.driver`, `core.device_manager`, `core.wda_driver`, and `vision.` and update every remaining import (spans `tools/`, `runtime/`, `storage/`, `api/`, `tests/`) to the new `driver.*`/`device.*`/`perception.*` paths
|
||||
- [x] 6.2 Update `pyproject.toml`: `[tool.setuptools.packages.find].include` to list `driver*`, `device*`, `perception*` alongside the existing `api*`, `core*`, `runtime*`, `storage*`, `tools*`
|
||||
- [x] 6.3 Run the full `pytest` suite and fix any remaining import errors until it is fully green with no test-logic changes
|
||||
|
||||
## 7. Rebrand project identity
|
||||
|
||||
- [x] 7.1 Update `pyproject.toml`: `name = "device-agent-runtime"`, add a one-line `description` reflecting the new positioning
|
||||
- [x] 7.2 Create root `README.md`: project positioning as a device-agnostic Device Agent Runtime (iPhone/WDA as the first driver), link to `docs/ROADMAP.md` and `docs/CONSTITUTION.md`
|
||||
|
||||
## 8. Planning artifacts
|
||||
|
||||
- [x] 8.1 Create `docs/ROADMAP.md` with the Milestone 0–10 sequence, the three delivery phases, and the long-term "DeviceOS" / Universal Device Runtime direction (v2.0, explicitly not started); explicitly map Milestones 1–4 to already-implemented `apex-agent-mvp` capabilities (`device-management`, `scene-perception`, `agent-runtime`, `task-memory`) rather than treating them as separate future work
|
||||
- [x] 8.2 Create `docs/adr/0001-device-agnostic-runtime.md` recording this repositioning decision (context, decision, alternatives considered, consequences), referencing this change's design.md decisions
|
||||
- [x] 8.3 Create `docs/adr/0002-layered-hexagonal-architecture.md` recording the Hexagonal/Ports-and-Adapters + DDD layering decision (D7): dependency direction (`core` domain has zero framework/LLM/HTTP deps; `driver`/`device` adapt external SDKs; `tools` is the capability/port layer; `perception` sits behind `PerceptionProvider`; `runtime` is the application layer where an LLM first enters via the Planner; `api` is the outermost transport adapter) and the bottom-up build order future milestones must respect
|
||||
- [x] 8.4 Create `docs/CONSTITUTION.md` capturing durable invariants: `Driver` is the only device-capability contract; `tools/` only calls into device/driver capabilities, never a concrete driver; `Scene` is the only perception artifact the LLM ever sees, produced through `PerceptionProvider`; drivers are stateless; Planner produces a plan, Executor is the only thing that calls tools and retries; **no LLM, HTTP, or MCP dependency may appear in `core`, `driver`, `device`, or `tools`** — those enter only at `runtime` (Planner) and `api` respectively; every future milestone change must state in its design.md how it upholds this boundary
|
||||
- [x] 8.5 Create `research/001-scene-model/README.md` through `research/007-ui-understanding/README.md` (scene-model, world-model, skill, agent, memory, planner, ui-understanding), each a short placeholder stating the track's purpose and that it holds papers/experiments/benchmarks, not code
|
||||
|
||||
## 9. Final verification
|
||||
|
||||
- [x] 9.1 Run `pytest` once more after all doc additions to confirm nothing in `tests/` was accidentally affected
|
||||
- [x] 9.2 Manually skim every touched file's diff to confirm no behavior changed — only paths, names, new port, and net-new docs
|
||||
Reference in New Issue
Block a user