feat: checkpoint device agent runtime milestones
This commit is contained in:
@@ -0,0 +1,2 @@
|
||||
schema: spec-driven
|
||||
created: 2026-07-06
|
||||
@@ -0,0 +1,65 @@
|
||||
## Context
|
||||
|
||||
The repository is currently empty (only tooling config exists: `pyproject.toml`, `.gitignore`, editor/agent config dirs, and this `openspec/` planning tree). We are building the first version of **Apex Agent**, an AI-native iPhone automation platform, from scratch. The defining constraint from the proposal is that the LLM must never be coupled to a specific automation framework (Appium/WDA) — it only ever sees semantic tools and a unified `Scene` model. Everything described here is new; there is no legacy system to interoperate with.
|
||||
|
||||
Primary stakeholder/consumer: an LLM (GPT/Claude/Qwen/DeepSeek) acting as the agent "brain," calling into this platform either through MCP tool calls or REST, to drive one or more physical iPhones running WebDriverAgent.
|
||||
|
||||
## Goals / Non-Goals
|
||||
|
||||
**Goals:**
|
||||
- Ship a working end-to-end loop: Observe (screenshot+OCR+tree → Scene) → Think (LLM/Planner) → Act (tool call → Driver → WDA) → Observe again.
|
||||
- Keep the LLM-facing surface 100% driver-agnostic: no Appium/WDA/XCUIElement types ever appear in tool names, parameters, or the Scene model.
|
||||
- Make the `Driver` interface implementable by something other than WDA later (e.g. `AndroidDriver`) without touching `tools/`, `vision/`, `runtime/`, or `api/`.
|
||||
- Persist a per-task timeline (screenshot/ocr/tree/prompt/action/result) so a task can eventually be replayed or debugged.
|
||||
- Expose the capability layer as MCP tools (primary) and a thin REST API (secondary, useful for debugging/dashboards).
|
||||
|
||||
**Non-Goals:**
|
||||
- No script recorder, no visual flow/orchestration editor, no "keystroke wizard" style macro engine.
|
||||
- No second driver implementation (Android/browser/Windows) in this change — only the interface must allow it.
|
||||
- No production task queue (Redis/RabbitMQ) or horizontal scaling — synchronous, single-process execution is sufficient for the MVP.
|
||||
- No PostgreSQL/MinIO — SQLite + local filesystem is sufficient for the MVP; swapping storage later must not require changing the `runtime/` or `tools/` layers.
|
||||
- No icon-detection model beyond a basic/stubbed implementation (`icon_detector.py` may return "not found" or use simple template matching); full CV-based icon recognition is future work.
|
||||
|
||||
## Decisions
|
||||
|
||||
### D1: Capability Layer sits between Agent Runtime and Driver, and is the only thing tools/ calls
|
||||
The `tools/` package (screenshot, tap, swipe, input_text, launch_app, ui_tree, describe_screen) is a thin wrapper that calls the currently active `Driver` instance obtained from `device/` (device manager/registry), and nothing else. `runtime/` (Planner/Executor) only ever calls functions in `tools/`, never a `Driver` directly.
|
||||
- **Alternative considered**: Let the Agent Runtime call `Driver` directly and skip the `tools/` layer. Rejected — this would make MCP tool schemas and Driver method signatures the same thing, so any future driver-specific quirk (e.g. Android needing a different tap gesture) leaks into the LLM-facing tool contract.
|
||||
|
||||
### D2: `Driver` is a stateless abstract interface; `WDADriver` is the only concrete implementation for this change
|
||||
`Driver` (in `core/driver.py`) defines `connect/disconnect/screenshot/tap/swipe/input/launch/terminate/tree/home/lock/unlock`. `core/wda_driver.py` implements it using the Appium Python client (talking to WebDriverAgent over HTTP). The driver holds no business/task state — only the live connection handle to one physical device.
|
||||
- **Alternative considered**: Call WDA's HTTP API directly instead of going through Appium's Python client. Deferred, not rejected — Appium client is faster to get working for the MVP; a follow-up change can swap to raw WDA HTTP calls behind the same `Driver` interface if Appium proves too heavy, without affecting any other layer.
|
||||
|
||||
### D3: Scene is the single perception artifact the LLM ever consumes
|
||||
`vision/scene_builder.py` fuses a screenshot, the accessibility/UI tree, and an OCR pass into one `Scene` object: `{ screen: {width, height}, elements: [{id, type, text, bounds, confidence}] }`. `tools/describe_screen.py` and `tools/find_text.py`/`find_icon()`-style helpers operate only on `Scene`, never on raw XCUIElement XML or raw OCR boxes.
|
||||
- **Alternative considered**: Expose UI tree (XML) and OCR results as two separate tool calls and let the LLM merge them. Rejected — pushes fusion logic (dedup, coordinate reconciliation) onto the LLM, which is unreliable and burns context; fusing once in `scene_builder.py` is strictly cheaper and more consistent.
|
||||
|
||||
### D4: Planner produces a step plan; Executor is the only thing that calls tools and retries
|
||||
`runtime/planner.py` takes a natural-language goal + current `Scene` and returns a small ordered list of intended steps (e.g. "find search box", "tap it", "type query"). `runtime/executor.py` turns each planned step into concrete tool calls, owns retry/backoff/wait-for-element logic, and re-invokes the Planner (or asks the LLM again) when a step fails or the Scene doesn't match expectations.
|
||||
- **Alternative considered**: Let the LLM emit raw tool calls one at a time with no separate Planner ("ReAct"-style, no plan object). Rejected per the proposal's explicit design goal (P-level principle from the source discussion: "AI 不容易跑飞") — an explicit plan gives the Executor something to check progress against and reduces runaway/looping behavior; the Planner step is intentionally kept lightweight for the MVP (no long-horizon planning algorithm required).
|
||||
|
||||
### D5: Task Memory writes one (screenshot, Scene JSON, prompt, tool call, result) record per step to local disk, indexed by task id
|
||||
`storage/timeline.py` + `storage/artifact_store.py` write `tasks/history/<task_id>/NNN.png` and `NNN.json` (Scene + prompt + tool call + result) on every Executor step. SQLite holds task/device metadata (task id, device id, status, timestamps); images and per-step JSON stay on the filesystem, not in the DB.
|
||||
- **Alternative considered**: Store everything (including images) in SQLite as blobs. Rejected — bloats the DB, complicates the later MinIO migration mentioned in the proposal's deferred work, and gains nothing since images are always accessed by path, never queried.
|
||||
|
||||
### D6: MCP Server is the primary LLM-facing surface; REST is a secondary/debug surface over the same `tools/` functions
|
||||
`api/mcp.py` registers each `tools/` function as an MCP tool (name, description, JSON-schema params) with no translation layer beyond argument (de)serialization. `api/rest.py` exposes the same underlying functions via FastAPI routes (`GET /devices`, `POST /devices/{id}/tap`, etc.) mainly for manual testing/dashboards, not as the primary agent interface.
|
||||
- **Alternative considered**: Build REST first and put MCP behind/on top of it (MCP server calls the REST API internally). Rejected — adds an unnecessary network hop and serialization layer for the primary path; both surfaces calling `tools/` directly keeps latency low and keeps `tools/` as the true single source of truth.
|
||||
|
||||
## Risks / Trade-offs
|
||||
|
||||
- **[Risk]** WDA/Appium connections are flaky (device sleep, WDA process crash, USB/Wi-Fi drop) → **Mitigation**: `Driver.connect()`/`Executor` retry logic treats connection loss as a recoverable error with bounded retries and surfaces a clear `device offline` state to the Device Manager rather than silently hanging.
|
||||
- **[Risk]** OCR + UI tree fusion in `scene_builder.py` may produce duplicate/conflicting elements (same visual button reported by both tree and OCR) → **Mitigation**: dedupe by bounding-box overlap (IoU threshold) in the MVP, prefer the UI-tree entry when both exist since it has ground-truth bounds; document this as a known heuristic, not a solved problem.
|
||||
- **[Risk]** Treating the Planner as "lightweight" could let the agent loop indefinitely on a stuck screen → **Mitigation**: Executor enforces a max-step and max-retry ceiling per task, at which point the task fails loudly (recorded in the timeline) instead of looping silently.
|
||||
- **[Trade-off]** Synchronous, single-process execution (no queue) means only as many concurrent tasks as the process can handle in-loop, and one slow/stuck task blocks its device — acceptable for MVP given the proposal explicitly defers queueing to a later change.
|
||||
- **[Trade-off]** SQLite + local filesystem storage doesn't survive multi-host deployment — acceptable for MVP (single machine, small number of physical devices); the proposal already earmarks Postgres/MinIO as deferred follow-up.
|
||||
|
||||
## Migration Plan
|
||||
|
||||
Not applicable — this is the first change in a new, empty repository. There is no prior system state or data to migrate, and no rollback target other than "delete the new code."
|
||||
|
||||
## Open Questions
|
||||
|
||||
- Which OCR engine ships as the MVP default (PaddleOCR, as suggested in the source discussion, vs. a lighter-weight/cloud alternative) — deferred to implementation time in `tasks.md`, since it doesn't change any interface in this design.
|
||||
- Whether the Planner should call the same LLM the caller is using, or be allowed a separate/cheaper model for step planning — left open; MVP can start by having the caller's LLM do both planning and per-step reasoning through the MCP tools, with `runtime/planner.py` as a thin pass-through it can later replace.
|
||||
- Exact MCP SDK/library choice for `api/mcp.py` — to be resolved during Sprint 4 implementation, doesn't affect the capability contracts defined here.
|
||||
@@ -0,0 +1,32 @@
|
||||
## Why
|
||||
|
||||
Traditional "device farm" (群控) tools work by having a human write a fixed automation script that the system blindly replays. That model breaks the moment a UI changes, and it caps the system at whatever the script author anticipated. This project inverts the model: an LLM observes the real screen state, decides the next action itself, and the system's only job is to expose stable, semantic capabilities (screenshot, tap, OCR, launch, ...) that the LLM can call. We are building **Apex Agent**, an AI-native iPhone Agent Platform (working name "IPA"), starting from an empty repository. WebDriverAgent (WDA) is treated as just one interchangeable driver behind a capability layer, not the platform itself — so Android/other drivers can be added later without touching the agent or tool layers.
|
||||
|
||||
## What Changes
|
||||
|
||||
- Introduce a **Device Manager** that discovers, connects to, and tracks the lifecycle/state (idle/busy/offline/error) of physical iPhones.
|
||||
- Introduce a **driver-independent Capability Layer** (screenshot, tap, swipe, input, launch, terminate, tree, home, lock/unlock) implemented first via a stateless **WDA Driver** adapter (Appium Python Client / WDA HTTP), designed so a future `AndroidDriver` can implement the same interface.
|
||||
- Introduce a **Vision/Perception pipeline** that fuses screenshot + UI tree + OCR into a single unified **Scene** model (JSON: screen size + elements with type/text/bounds/confidence) — the only representation the AI ever sees; raw XCUIElement/XML types are never exposed.
|
||||
- Introduce an **Agent Runtime** (Planner + Executor + Memory/Context) that runs the Observe → Think → Act → Observe loop, decomposing a high-level goal (e.g. "open Taobao and search Mac mini") into tool calls, with retry/wait handling in the Executor so the LLM doesn't have to micromanage timing.
|
||||
- Introduce a **Task Memory / Timeline** store that persists each step's screenshot, OCR result, UI tree, prompt, tool call, and result to disk (per-task history), laying the groundwork for future replay.
|
||||
- Introduce an **MCP Tool Server** (and a thin REST API) that exposes all of the above as semantic, LLM-facing tools (`take_screenshot`, `tap`, `swipe`, `input_text`, `launch_app`, `find_text`, `find_icon`, `get_ui_tree`, `describe_screen`, `list_devices`, `device_status`, ...) so any MCP-compatible client (Claude Desktop, GPT function calling, etc.) can drive a real iPhone without ever knowing Appium/WDA/HTTP exist underneath.
|
||||
- Explicitly out of scope for this change (non-goals): scripted "keystroke wizard" style macros, a script recorder/IDE, a visual flow orchestrator, Android/browser/Windows drivers (interface must allow them later, but no second driver ships now), and a production task queue (Redis/RabbitMQ) — synchronous execution is enough for the MVP loop.
|
||||
|
||||
## Capabilities
|
||||
|
||||
### New Capabilities
|
||||
- `device-management`: Device discovery/connection/state tracking (DeviceManager) plus the driver-independent device capability interface (connect/disconnect/screenshot/tap/swipe/input/launch/terminate/tree/home/lock/unlock) and its first concrete implementation, the stateless WDA driver.
|
||||
- `scene-perception`: Turns a raw screenshot + UI tree + OCR pass into the unified Scene JSON model that the agent and LLM consume instead of raw XML/XCUIElement data.
|
||||
- `agent-runtime`: Planner + Executor + Memory/Context implementing the Observe → Think → Act → Observe loop, including retry and wait handling around tool calls.
|
||||
- `task-memory`: Per-task timeline persistence of each step's screenshot, OCR, tree, prompt, tool call, and result to local storage (artifact store), forming the basis for future replay.
|
||||
- `mcp-tool-server`: MCP server (plus thin REST surface) exposing device, perception, and agent capabilities as stable, semantic tools for LLM function calling, with no Appium/WDA concepts leaking through.
|
||||
|
||||
### Modified Capabilities
|
||||
(none — this is the first change in an empty project)
|
||||
|
||||
## Impact
|
||||
|
||||
- **New code**: entire initial codebase — `core/` (device manager, driver interface, WDA driver, scene model, memory, models), `tools/` (screenshot, tap, swipe, input_text, launch_app, ui_tree, describe_screen), `vision/` (ocr, ui_parser, scene_builder, icon_detector), `runtime/` (planner, executor, context, task), `api/` (rest, mcp), `storage/` (timeline, artifact_store), `tests/`.
|
||||
- **Dependencies**: Python 3.14 (per existing `pyproject.toml`), Appium Python Client / WDA HTTP client, an OCR engine (e.g. PaddleOCR), FastAPI, an MCP server SDK, SQLite for MVP metadata storage, local filesystem for screenshots/artifacts.
|
||||
- **External systems**: requires a real (or simulator) iPhone reachable via WebDriverAgent/Appium; no cloud services required for the MVP.
|
||||
- **Follow-on work explicitly deferred**: Android/browser/Windows drivers, PostgreSQL migration, MinIO/object storage, Redis/RabbitMQ task queue, icon-detection model beyond a basic stub.
|
||||
@@ -0,0 +1,41 @@
|
||||
## ADDED Requirements
|
||||
|
||||
### Requirement: Observe-Think-Act-Observe execution loop
|
||||
The system SHALL execute a task as a repeating loop: observe the current Scene, decide the next step, act via a capability tool call, then observe the resulting Scene again, continuing until the goal is met, a failure ceiling is hit, or the task is cancelled.
|
||||
|
||||
#### Scenario: Loop continues until goal is met
|
||||
- **WHEN** a task with a natural-language goal (e.g. "open Taobao and search Mac mini") is started
|
||||
- **THEN** the runtime repeats observe→think→act until the Planner/Executor determines the goal has been reached, then marks the task complete
|
||||
|
||||
#### Scenario: Loop stops after max steps
|
||||
- **WHEN** a task exceeds a configured maximum number of steps without reaching its goal
|
||||
- **THEN** the runtime stops the loop and marks the task as failed with a reason, instead of looping indefinitely
|
||||
|
||||
### Requirement: Planner produces a step plan from goal and current Scene
|
||||
The system SHALL provide a Planner that, given a goal and the current Scene, produces an ordered list of intended next steps (e.g. "find search box", "tap it", "type query").
|
||||
|
||||
#### Scenario: Planner emits steps for a new goal
|
||||
- **WHEN** the Planner is invoked with a goal and the current Scene at the start of a task
|
||||
- **THEN** it returns a non-empty ordered list of intended steps for the Executor to attempt
|
||||
|
||||
#### Scenario: Planner re-plans after unexpected Scene
|
||||
- **WHEN** the Executor reports that the Scene after an action does not match what the current step expected
|
||||
- **THEN** the Planner is invoked again with the updated Scene to produce a revised step (or remaining steps)
|
||||
|
||||
### Requirement: Executor performs retry and wait handling around tool calls
|
||||
The system SHALL provide an Executor that translates a planned step into one or more capability tool calls, and SHALL retry with backoff and/or wait-for-element behavior when a step's expected result is not immediately observed, up to a configured retry ceiling.
|
||||
|
||||
#### Scenario: Executor retries a transient failure
|
||||
- **WHEN** a tool call (e.g. `tap`) does not produce the expected Scene change on the first attempt
|
||||
- **THEN** the Executor retries the step up to a configured number of attempts before treating it as failed
|
||||
|
||||
#### Scenario: Executor gives up after retry ceiling
|
||||
- **WHEN** a step has failed for the configured maximum number of retries
|
||||
- **THEN** the Executor records the step as failed and surfaces this to the Planner/task result instead of retrying forever
|
||||
|
||||
### Requirement: Task context/memory available during a run
|
||||
The system SHALL maintain an in-run context (recent Scene history, executed steps, and their results) accessible to the Planner and Executor for the duration of a task, so re-planning decisions can reference what has already been tried.
|
||||
|
||||
#### Scenario: Re-planning uses prior step history
|
||||
- **WHEN** the Planner is re-invoked mid-task
|
||||
- **THEN** it has access to the steps already attempted in this task and their outcomes, not just the current Scene in isolation
|
||||
@@ -0,0 +1,52 @@
|
||||
## ADDED Requirements
|
||||
|
||||
### Requirement: Device discovery and listing
|
||||
The system SHALL provide a Device Manager that can list all known iPhone devices and their current status (`idle`, `busy`, `offline`, `error`).
|
||||
|
||||
#### Scenario: Listing devices returns current status
|
||||
- **WHEN** a caller requests the device list
|
||||
- **THEN** the system returns each known device's id, status, and driver connection info (e.g. WDA port) without contacting the physical device for every field
|
||||
|
||||
#### Scenario: No devices connected
|
||||
- **WHEN** a caller requests the device list and no physical devices are reachable
|
||||
- **THEN** the system returns an empty list rather than raising an error
|
||||
|
||||
### Requirement: Device connect and disconnect lifecycle
|
||||
The system SHALL allow a caller to connect to and disconnect from a specific device by id, transitioning its tracked status accordingly.
|
||||
|
||||
#### Scenario: Successful connect
|
||||
- **WHEN** a caller connects to a device id that is currently `idle` and reachable
|
||||
- **THEN** the Device Manager marks the device `busy`, and it becomes usable for capability calls
|
||||
|
||||
#### Scenario: Connect to unreachable device
|
||||
- **WHEN** a caller connects to a device id that cannot be reached (WDA not responding)
|
||||
- **THEN** the Device Manager marks the device `offline` and returns an error to the caller instead of hanging indefinitely
|
||||
|
||||
#### Scenario: Disconnect releases the device
|
||||
- **WHEN** a caller disconnects from a device it previously connected to
|
||||
- **THEN** the Device Manager releases the underlying driver connection and marks the device `idle`
|
||||
|
||||
### Requirement: Driver-independent capability interface
|
||||
The system SHALL define a single `Driver` interface (`connect`, `disconnect`, `screenshot`, `tap`, `swipe`, `input`, `launch`, `terminate`, `tree`, `home`, `lock`, `unlock`) that any concrete driver implementation (e.g. WDA, and in future Android) must satisfy identically, so callers above the driver layer never depend on a specific automation framework.
|
||||
|
||||
#### Scenario: Capability call is dispatched through the interface
|
||||
- **WHEN** a higher layer (tools/) invokes a capability such as `tap(x, y)` on a connected device
|
||||
- **THEN** the call is routed through the `Driver` interface to the concrete driver instance for that device, with no framework-specific (e.g. Appium/WDA) types or errors surfacing to the caller
|
||||
|
||||
### Requirement: WDA driver implementation
|
||||
The system SHALL provide a concrete `WDADriver` implementing the `Driver` interface using WebDriverAgent (via Appium Python client), supporting at minimum: screenshot capture, tap, swipe, text input, app launch, app terminate, UI tree retrieval, home button, and device lock/unlock.
|
||||
|
||||
#### Scenario: Screenshot via WDA driver
|
||||
- **WHEN** `screenshot()` is called on a device backed by `WDADriver`
|
||||
- **THEN** the driver returns image bytes/path representing the current physical screen contents
|
||||
|
||||
#### Scenario: Launch app via WDA driver
|
||||
- **WHEN** `launch(bundle_id_or_name)` is called on a device backed by `WDADriver`
|
||||
- **THEN** the driver starts the requested app on the physical device and the call returns once the app process is confirmed running (or raises a clear error if launch fails)
|
||||
|
||||
### Requirement: Stateless driver
|
||||
Driver implementations SHALL NOT persist task-level or business state (e.g. current task id, plan progress); any such state SHALL be owned by the Agent Runtime / Task Memory layers, not the driver.
|
||||
|
||||
#### Scenario: Driver restart does not lose task progress
|
||||
- **WHEN** a driver connection is dropped and re-established mid-task
|
||||
- **THEN** the in-progress task's plan, step history, and timeline remain intact because they were never stored in the driver, only the live device connection needs to be re-established
|
||||
@@ -0,0 +1,30 @@
|
||||
## ADDED Requirements
|
||||
|
||||
### Requirement: MCP tools expose capability layer without driver leakage
|
||||
The system SHALL expose an MCP server whose tools (at minimum: `take_screenshot`, `tap`, `swipe`, `input_text`, `launch_app`, `find_text`, `find_icon`, `get_ui_tree`, `describe_screen`, `list_devices`, `device_status`) map directly to the capability/perception layer, with tool names, parameters, and return values containing no Appium/WDA/XCUIElement-specific concepts.
|
||||
|
||||
#### Scenario: MCP client calls a tool without framework knowledge
|
||||
- **WHEN** an MCP-compatible LLM client (e.g. Claude Desktop) calls the `tap` tool with coordinates
|
||||
- **THEN** the call succeeds by being routed through the capability layer to the underlying driver, and neither the tool schema nor its response exposes Appium/WDA-specific types or errors
|
||||
|
||||
#### Scenario: Tool list is stable across driver changes
|
||||
- **WHEN** the underlying driver for a device changes (e.g. a future non-WDA driver is used) while the MCP tool set is unchanged
|
||||
- **THEN** existing MCP tool calls continue to work without any change to tool names or parameter schemas
|
||||
|
||||
### Requirement: MCP tool errors are semantic, not framework-specific
|
||||
The system SHALL translate driver/framework-level errors (e.g. WDA connection errors, element-not-found) into clear, semantic MCP tool error responses (e.g. "device offline", "element not found") rather than passing raw framework exceptions through to the LLM.
|
||||
|
||||
#### Scenario: Device offline surfaces a clear error
|
||||
- **WHEN** an MCP tool call targets a device that is currently offline
|
||||
- **THEN** the tool call returns a semantic error indicating the device is unavailable, not a raw connection-refused/stack-trace style error
|
||||
|
||||
### Requirement: REST API mirrors the same capability functions
|
||||
The system SHALL provide a REST API (`GET /devices`, `POST /devices/{id}/tap`, `POST /devices/{id}/screenshot`, `POST /devices/{id}/launch`, `POST /agent/task`, `GET /task/{id}`) that calls the same underlying capability functions as the MCP tools, for manual testing and dashboard use.
|
||||
|
||||
#### Scenario: REST and MCP produce consistent results
|
||||
- **WHEN** the same capability (e.g. `screenshot`) is invoked once via the REST API and once via the corresponding MCP tool for the same device
|
||||
- **THEN** both calls exercise the same underlying capability function and produce equivalent results
|
||||
|
||||
#### Scenario: Starting a task via REST
|
||||
- **WHEN** a caller POSTs a goal to `/agent/task`
|
||||
- **THEN** the system creates a new task, returns its task id, and the task becomes queryable via `GET /task/{id}`
|
||||
@@ -0,0 +1,37 @@
|
||||
## ADDED Requirements
|
||||
|
||||
### Requirement: Unified Scene generation
|
||||
The system SHALL generate a single `Scene` object for the current screen by fusing a screenshot, the UI accessibility tree, and an OCR pass, so that no caller above the perception layer ever needs to read raw UI-tree XML or raw OCR output directly.
|
||||
|
||||
#### Scenario: Scene built from a live screen
|
||||
- **WHEN** the perception layer is asked to describe the current screen of a connected device
|
||||
- **THEN** it captures a screenshot, retrieves the UI tree, runs OCR, and returns one `Scene` object containing `screen` (width, height) and a list of `elements`, each with `id`, `type`, `text`, `bounds`, and `confidence`
|
||||
|
||||
#### Scenario: OCR-only element when tree has no match
|
||||
- **WHEN** OCR detects a text region that has no corresponding node in the UI tree
|
||||
- **THEN** the Scene SHALL still include that text as an element (type inferred as `text` or `unknown`), rather than silently dropping it
|
||||
|
||||
### Requirement: Duplicate element reconciliation
|
||||
The system SHALL reconcile elements that are reported by both the UI tree and OCR for the same visual region into a single Scene element, preferring UI-tree-provided bounds/type when both sources overlap significantly.
|
||||
|
||||
#### Scenario: Button text detected by both tree and OCR
|
||||
- **WHEN** a button's label is present both as a UI tree node and as an OCR text box with substantially overlapping bounds
|
||||
- **THEN** the Scene contains one element for that button, using the UI tree's type and bounds, with the OCR text merged in if the tree node lacked a text value
|
||||
|
||||
### Requirement: Semantic lookup helpers over Scene
|
||||
The system SHALL provide `find_text` and `find_icon` helpers that search the current `Scene` for a matching element and return its coordinates, without requiring the caller to parse the Scene JSON manually.
|
||||
|
||||
#### Scenario: Find text present on screen
|
||||
- **WHEN** a caller invokes `find_text("搜索")` while the current Scene contains an element with that text
|
||||
- **THEN** the system returns the tappable coordinates (center point or bounds) of the matching element
|
||||
|
||||
#### Scenario: Find text not present on screen
|
||||
- **WHEN** a caller invokes `find_text(...)` for a string that has no match in the current Scene
|
||||
- **THEN** the system returns a clear "not found" result rather than raising an unhandled exception
|
||||
|
||||
### Requirement: describe_screen returns Scene only
|
||||
The `describe_screen` tool SHALL return the Scene model (or a summarized natural-language rendering of it) and SHALL NOT return raw XCUIElement type names, raw XML, or driver-specific identifiers.
|
||||
|
||||
#### Scenario: describe_screen output is driver-agnostic
|
||||
- **WHEN** `describe_screen()` is called against a device backed by the WDA driver
|
||||
- **THEN** the returned content contains only Scene-model fields (screen size, elements with type/text/bounds) and no WDA/Appium/XCUIElement-specific terms
|
||||
@@ -0,0 +1,30 @@
|
||||
## ADDED Requirements
|
||||
|
||||
### Requirement: Per-step timeline persistence
|
||||
The system SHALL persist a record for every step of a task, containing the screenshot, the Scene JSON, the prompt/goal context, the tool call issued, and its result, written to local storage under a task-specific path.
|
||||
|
||||
#### Scenario: Step record written after each action
|
||||
- **WHEN** the Executor completes a step (successful or failed)
|
||||
- **THEN** the system writes that step's screenshot and a JSON record (Scene, prompt, tool call, result) to the task's timeline directory before proceeding to the next step
|
||||
|
||||
#### Scenario: Timeline survives process restart
|
||||
- **WHEN** the process running a task restarts after a step has already been persisted
|
||||
- **THEN** the previously persisted steps for that task remain readable from disk
|
||||
|
||||
### Requirement: Task metadata storage
|
||||
The system SHALL store task-level metadata (task id, target device id, status, start/end timestamps) in a local database (SQLite for this change), separate from the per-step image/JSON artifacts on the filesystem.
|
||||
|
||||
#### Scenario: Task status queryable during execution
|
||||
- **WHEN** a caller queries a task by id while it is still running
|
||||
- **THEN** the system returns its current status (e.g. `running`) and metadata without needing to scan the filesystem timeline
|
||||
|
||||
#### Scenario: Task status reflects completion
|
||||
- **WHEN** a task finishes (successfully or with failure)
|
||||
- **THEN** its stored metadata status is updated accordingly and its end timestamp is recorded
|
||||
|
||||
### Requirement: Timeline retrievable for future replay
|
||||
The system SHALL store each task's steps in a way that preserves their order and completeness, so that a future replay feature can reconstruct the full sequence of screenshots/Scenes/actions for a task without additional data collection.
|
||||
|
||||
#### Scenario: Steps retrievable in order
|
||||
- **WHEN** a caller requests the full timeline for a completed task
|
||||
- **THEN** the system returns all persisted steps in the order they occurred, each with its screenshot and JSON record
|
||||
@@ -0,0 +1,2 @@
|
||||
schema: spec-driven
|
||||
created: 2026-07-06
|
||||
@@ -0,0 +1,94 @@
|
||||
## Context
|
||||
|
||||
Every prior milestone — `device-management`/`driver-registry` (`apex-agent-mvp`, `device-agent-runtime-foundation`), `agent-runtime`, `semantic-scene`, `world-model`, `skill-authoring`/`skill-versioning`/`skill-embedding-retrieval`, `workflow-orchestration`, `multi-agent-collaboration` — is designed and (mostly) built around one process: one `DeviceManager` instance holding an in-memory dict of devices, one `TaskRunner`/`WorkflowRunner` executing on whichever `device_id` the caller passes in, and one `driver/registry.py` extension point (`SUPPORTED_DRIVER_TYPES`) that a caller reads to build a driver by name. That single-process model is a deliberate, explicit non-goal-turned-goal in `apex-agent-mvp/design.md` ("No production task queue ... synchronous, single-process execution is sufficient for the MVP") and every later milestone has respected it rather than re-litigating it — `workflow-orchestration-runtime`'s own Non-Goals state plainly: "No distributed/multi-device workflow execution or cross-node coordination — that is Milestone 10 (Cloud Runtime)."
|
||||
|
||||
This change is that milestone. Three things are genuinely new here that no prior milestone had to solve: (1) device state now spans more than one host process, so "list devices" and "is this device idle" can no longer be answered by one in-memory dict; (2) a caller submitting a task no longer names a `device_id` up front — something has to *pick* one, from a fleet, based on constraints; (3) new drivers/tools/skills should be addable without editing this repository's packages at all, which the existing "add a dict entry to `driver/registry.py`" extension point does not yet support for a party who cannot modify the repo.
|
||||
|
||||
Two stakeholders: an **external integrator** (a system outside this repo, e.g. a customer's backend) who needs a stable, versioned API to submit work and query status without going through the LLM-facing MCP surface or the operator-facing web console; and a **fleet operator** who runs more than one host process (each with its own `DeviceManager` and set of physically-attached devices) and needs one place to see "how many devices are idle across all of them" and have tasks land on whichever is free.
|
||||
|
||||
## Goals / Non-Goals
|
||||
|
||||
**Goals:**
|
||||
- Define an internal `DevicePool` model that aggregates device state from potentially many hosts' independent `DeviceManager` instances, degrading gracefully (marking a host's devices `unreachable`, never crashing or blocking) when a host stops reporting in.
|
||||
- Define a `TaskScheduler` that accepts task submissions with optional device constraints and assigns them to an idle, matching `PooledDevice`, via a pluggable, registrable `AssignmentStrategy` — the same "string key → swappable implementation" shape as `driver/registry.py`'s `SUPPORTED_DRIVER_TYPES` and `workflow-orchestration-runtime`'s `ConditionEvaluator` registry.
|
||||
- Define a `TaskDispatcher` that actually executes a scheduler assignment by composing the existing `agent-runtime`/`workflow-orchestration` entry points — so device selection becomes automatic without duplicating any execution logic those capabilities already own.
|
||||
- Define a `PluginManifest`/`PluginRegistry` mechanism so a new driver (and, in schema if not yet in wiring, a new tool or skill) can register itself via a manifest file or Python entry point, without editing `driver/`, `device/`, `tools/`, or the skill stores.
|
||||
- Define a versioned, public `platform-sdk` (REST API + Python client) for external integrators, clearly scoped apart from `mcp-tool-server` (LLM tool-calling surface) and `console-status-api`/`console-config-api` (single-process operator UI surface).
|
||||
|
||||
**Non-Goals:**
|
||||
- No actual multi-host deployment or infrastructure tooling (no Docker/Kubernetes manifests, no service mesh, no real network transport implementation for host-to-pool sync beyond defining the `sync_host_devices(host_id, snapshot)` call signature this change's local reference implementation exercises in-process). This change defines the runtime's internal data model and control flow for a device pool and scheduler, not ops tooling to actually run one across real machines.
|
||||
- No cross-host **task execution** dispatch over the network: `TaskDispatcher` composes `TaskRunner`/`WorkflowRunner` directly, which only works when the assigned device is owned by the same process the dispatcher runs in. When `DevicePool` reports a `PooledDevice` owned by a *different* host, `TaskDispatcher` raises a clear, typed `RemoteDispatchNotSupportedError` rather than silently no-op'ing or pretending to execute — real cross-host RPC dispatch is left as an Open Question / follow-up, consistent with this change's own "no infra" non-goal.
|
||||
- No billing, multi-tenant authentication/authorization, or rate limiting on `platform-sdk` — a later change's concern; this change may add a single API-key placeholder hook (see D8) but does not implement real auth.
|
||||
- No generic dynamic tool/skill plugin *wiring* — `plugin-system`'s manifest schema accepts `tool`/`skill`-kind entries for forward compatibility, but only `driver`-kind plugins are concretely registered into an existing extension point (`driver-registry`) in this change; `tools/` and the skill stores have no comparable public registration hook yet to compose against.
|
||||
- No changes to `driver/`, `device/`, `runtime/`, `tools/`, `workflow/`, `agents/`, `storage/`, `api/console.py`, or `api/mcp.py` — every integration is by import/composition only.
|
||||
- No replacement of `mcp-tool-server` or `console-status-api`/`console-config-api` — `platform-sdk` is a third, additive surface, not a superset or a router in front of the other two.
|
||||
|
||||
## Decisions
|
||||
|
||||
### D1: New `cloud/` package, sibling to `agents/`/`workflow/`/`semantic/`/`world/`/`skills_learning/`, not folded into `runtime/`, `device/`, or `api/`
|
||||
`cloud/` (`pool.py`, `scheduler.py`, `dispatch.py`, `plugins.py`, `store.py`, `config.py`, `sdk/`) is its own top-level package. This continues the one-package-per-milestone precedent (`agents/` for Milestone 9, `workflow/` for Milestone 8, `semantic/`/`world/`/`skills_learning/` for Milestones 5–7) and keeps `runtime/`'s job (single-goal Observe-Think-Act loop) and `device/`'s job (one process's own device lifecycle) unchanged.
|
||||
- **Alternative considered**: Extend `device/manager.py`'s `DeviceManager` in place to understand multiple hosts. Rejected — `DeviceManager` is a pending, unapplied capability (`device-management`) with no applied baseline in `openspec/specs/` to safely diff a `MODIFIED` delta against in this session (the same reasoning `workflow-orchestration-runtime`'s D1 already applied to `TaskRunner`); treating it as a stable, composed-over dependency keeps "one process's device lifecycle" and "many processes' aggregated device state" as two separate, independently testable concerns, and avoids retroactively changing another change's still-pending contract.
|
||||
- **Alternative considered**: Put the four capabilities in four separate top-level packages (`device_pool/`, `task_scheduler/`, `plugins/`, `platform_sdk/`). Rejected — these four are tightly coupled (the scheduler reads the pool; the SDK exposes both; the dispatcher needs the scheduler's assignment) and the roadmap already names this single milestone "Cloud Runtime"; one package with clear submodules keeps the coupling visible in the import graph instead of scattering it across four independent top-level packages that would need to depend on each other anyway.
|
||||
|
||||
### D2: `DevicePool` aggregates by *push*, not by remotely calling each host's `DeviceManager`
|
||||
`DevicePool.sync_host_devices(host_id, snapshot: list[Device])` is called (by a periodic job, or by `platform-sdk`'s host-registration endpoint) with the *result* of that host's own `DeviceManager.list_devices()` — the pool never reaches out over the network to call a remote `DeviceManager` method itself. `HostRegistration.last_seen_at` updates on every sync call; a background staleness check (or a check performed lazily on read, see D3) marks a host's `PooledDevice`s `unreachable` once `last_seen_at` exceeds `config.stale_after_seconds`.
|
||||
- **Alternative considered**: Give `DevicePool` an RPC client that calls a remote host's `DeviceManager` over HTTP/gRPC directly (pull model). Rejected for this change — building a real inter-host RPC layer is exactly the "actual multi-host deployment" infrastructure work this change's Non-Goals defer; a push-based sync call has a stable, host-agnostic signature today (`sync_host_devices(host_id, snapshot)`) that a later change can wire to any real transport (HTTP call from a lightweight per-host agent, message queue, etc.) without changing `DevicePool`'s internal model.
|
||||
|
||||
### D3: Host staleness is checked lazily on read, not via a background timer thread
|
||||
`DevicePool.list_devices()` and `DevicePool.get_device(device_id)` compute each host's staleness (`now - last_seen_at > stale_after_seconds`) at call time and report `unreachable` status accordingly, rather than running a background thread that mutates stored status on a timer.
|
||||
- **Alternative considered**: Run a background thread/asyncio task that periodically sweeps `HostRegistration`s and flips stale ones to `unreachable` in the store. Rejected — a background thread adds lifecycle management (start/stop with the process, thread-safety with `CloudStore`'s SQLite connections) for a property (staleness) that is cheap to compute at read time from a single stored timestamp; lazy computation is simpler, always consistent with the current clock, and avoids a whole class of "the sweep thread died silently" failure modes for no loss of correctness.
|
||||
|
||||
### D4: `TaskScheduler`'s queue and `AssignmentStrategy` registry mirror the Driver Registry / `ConditionEvaluator` pattern
|
||||
`cloud/scheduler.py` defines `AssignmentStrategy` (one method, `select(task: ScheduledTask, candidates: list[PooledDevice]) -> PooledDevice | None`) and a registry mapping a strategy name (`fifo_match` default, reserved name `capability_score` for a future scoring strategy) to an implementation, the same "string key → pluggable implementation" shape already standardized on by `driver/registry.py` (`device-agent-runtime-foundation` D3), `perception/provider.py` (D8), and `workflow/conditions.py`'s `ConditionEvaluator` (`workflow-orchestration-runtime` D6). `TaskScheduler.submit(goal, constraints)` enqueues a `ScheduledTask{id, goal, constraints, status: queued|assigned|dispatched|done|failed}`; `TaskScheduler.assign()` (called by a poll loop or triggered on submission/device-freed events) pops the head of the FIFO queue, asks `DevicePool.list_devices()` for idle candidates matching `constraints.driver_type`/`constraints.capability_tags`, and calls the configured `AssignmentStrategy.select()`.
|
||||
- **Alternative considered**: Hardcode FIFO-first-idle-match as the only assignment behavior with no strategy abstraction. Rejected — this is precisely the third occurrence of the same "pluggable-by-string-key" problem (driver selection, condition evaluation, now assignment strategy); making it a registry from the start costs one extra indirection and avoids an `if/elif` chain being added later when a capability-scoring strategy is inevitably wanted.
|
||||
|
||||
### D5: `TaskDispatcher` composes `TaskRunner`/`WorkflowRunner` as black boxes; local-only in this change
|
||||
`cloud/dispatch.py`'s `TaskDispatcher.dispatch(assignment: Assignment)` builds a `core.models.Task(goal=assignment.task.goal, device_id=assignment.device_id)` (or, if the submission specified a `workflow_definition_id` instead of a bare `goal`, a `WorkflowDefinition`) and calls the existing, unmodified `runtime.task.TaskRunner(...).run(task)` or `workflow.runner.WorkflowRunner(...).run(definition)` — exactly the composition pattern `workflow-orchestration-runtime`'s D8 already established for `WorkflowRunner` composing `TaskRunner`. If `assignment.host_id` does not match the local process's own host id, `dispatch()` raises `RemoteDispatchNotSupportedError` instead of attempting anything — this change's scheduler can *decide* a remote host should take a task, but only a local dispatch is actually executed here (see Non-Goals and Open Questions).
|
||||
- **Alternative considered**: Build a lower-level dispatch loop that re-implements plan/execute/retry logic inside `cloud/` instead of calling `TaskRunner`/`WorkflowRunner`. Rejected — duplicates already-implemented, already-tested execution logic and creates two places that must stay behaviorally consistent; composing the existing public entry points automatically inherits any future improvement to either runner with zero change to `cloud/`.
|
||||
|
||||
### D6: `PluginManifest`/`PluginRegistry` discovers via `importlib.metadata` entry points and a local manifest-file scan, both feeding one validation+registration path
|
||||
`cloud/plugins.py`'s `PluginManifest` (`name: str`, `version: str`, `entry_point_kind: Literal["driver", "tool", "skill"]`, `target: str` — a dotted module:attribute path) can be produced two ways: (a) a package installed in the environment declares an entry point in group `device_agent_runtime.plugins` whose value is itself a `PluginManifest`-shaped object or a callable returning one; (b) a `plugins/<name>/plugin.json` file on a configured scan path is parsed directly into a `PluginManifest`. Both paths converge on `PluginRegistry.register(manifest)`, which validates shape/uniqueness (no duplicate `name`, `entry_point_kind` recognized) and then, only for `entry_point_kind == "driver"`, resolves `target` to a `DriverFactoryBuilder` callable and calls `driver_registry.register_driver_type(manifest.name, builder)`.
|
||||
- **Alternative considered**: Support only Python entry points (no manifest-file scan), matching a more conventional Python-plugin-ecosystem approach (e.g., `pluggy`). Rejected for this change — requiring every plugin author to publish an installable Python package with an entry point is a real adoption barrier for a first plugin mechanism; a plain `plugin.json` file scan lets a plugin be "drop a folder in `plugins/`" as well, and both paths sharing one validation+registration function means neither is a second, divergent code path to maintain.
|
||||
- **Note (dependency on a not-yet-existing function)**: `driver/registry.py` (as specified by the pending `driver-registry` capability) today only defines `SUPPORTED_DRIVER_TYPES` (a plain dict) and `build_driver_factory()`; it does not yet expose a `register_driver_type(name, builder)` function. This change's driver-plugin registration path is written against that function *by name*, on the expectation that `device-agent-runtime-foundation`'s own stated intent ("adding a new driver type is a one entry addition to `driver/registry.py`") is realized as a small public function, not only as "edit the dict inline." This is called out again in Open Questions since it is a real dependency on another pending change's still-evolving surface, not a formal spec delta this change is making to it.
|
||||
|
||||
### D7: `cloud/store.py`'s `CloudStore` is a new, independently-owned SQLite file, not new tables in `storage/task_metadata.py` or `workflow/store.py`
|
||||
`CloudStore` opens its own file (default `cloud/cloud.sqlite3`) with `host_registrations`, `pooled_devices`, `scheduled_tasks`, and `plugins` tables, using the same connect-per-call `sqlite3` pattern as `storage/task_metadata.py` and `workflow/store.py`.
|
||||
- **Alternative considered**: Add `scheduled_tasks`/`pooled_devices` tables into `workflow/store.py`'s existing `workflows.sqlite3` (since a scheduled task's eventual execution may become a `WorkflowRun`). Rejected — `workflow/` is owned by the pending, unapplied `workflow-orchestration` capability; adding schema/migration code to it without a corresponding spec delta would be a de facto modification of another change's owned artifact, the same reasoning `workflow-orchestration-runtime`'s own D9 already used to justify *not* writing into `storage/task_metadata.py`. A separate, `cloud/`-owned store composed in prose (a `scheduled_tasks` row stores a `task_id`/`workflow_run_id` string reference once dispatched, no foreign key) is fully additive.
|
||||
|
||||
### D8: `platform-sdk`'s REST API is versioned via a URL prefix (`/v1/...`), with a single pluggable `AuthProvider` hook (no-op default), not a full auth system
|
||||
`cloud/sdk/api.py` mounts all routes under `/v1/` (e.g., `POST /v1/tasks`, `GET /v1/tasks/{id}`, `GET /v1/devices`, `GET /v1/hosts`, `GET /v1/plugins`, `POST /v1/plugins`). Every route accepts an optional `AuthProvider.authenticate(request) -> Principal | None` hook (default: a `NullAuthProvider` that always returns an anonymous `Principal`, i.e., no enforcement) so a later change can add real API-key/OAuth checking without changing route signatures.
|
||||
- **Alternative considered**: Ship `platform-sdk` with no versioning prefix at all (bare `/tasks`, `/devices`, ...), deferring versioning until a breaking change is actually needed. Rejected — this is explicitly a *public-facing* surface for external integrators (unlike `console-status-api`, which is first-party/trusted-network only per `web-console`'s own scope); starting unversioned and retrofitting a prefix later would break every existing integrator's URLs on day one of the first breaking change, which a public SDK should not do.
|
||||
- **Alternative considered**: Build real API-key authentication now, since "public-facing" implies untrusted callers. Rejected for this change per its explicit Non-Goal — auth/authz is a distinct, non-trivial concern (key storage, rotation, revocation) that deserves its own change; shipping a no-op hook that is clearly the extension point (mirroring the Driver Registry/`ConditionEvaluator`/`AssignmentStrategy` pattern used everywhere else in this design) is enough to make "add real auth later" a localized change, not a rewrite.
|
||||
|
||||
### D9: `cloud/sdk/client.py` is a thin `requests`/`httpx`-based wrapper, not a code-generated client
|
||||
The Python SDK client (`CloudClient`) is hand-written, wrapping the same `/v1/...` routes `cloud/sdk/api.py` exposes, using `httpx` (already a dev dependency, promoted to a runtime dependency here since the client ships as installable code, not just test tooling).
|
||||
- **Alternative considered**: Generate the client from an OpenAPI spec (FastAPI already produces one) via `openapi-python-client` or similar. Rejected for this change — adds a codegen build step and a new dev-tool dependency for a surface with only ~6 routes; hand-writing a thin wrapper is faster to ship and easier to read/review, and nothing here forecloses generating a client later once the API surface is larger and more churn-prone.
|
||||
|
||||
## Risks / Trade-offs
|
||||
|
||||
- **[Risk]** Push-based host sync (D2) means `DevicePool`'s view of a host's devices is only as fresh as that host's last sync call; a device could go `busy`→`idle` on its host moments after a sync and the pool won't know until the next sync interval → **Mitigation**: `config.sync_interval_seconds` is deliberately short-default and documented as eventually-consistent, not real-time; `TaskScheduler.assign()` re-checks `DevicePool` state immediately before dispatch (not just at submission time) to shrink the staleness window for the assignment decision itself, though a race between "assign" and "actually dispatch" is still possible and is accepted, not solved, in this change.
|
||||
- **[Risk]** `TaskDispatcher`'s local-only execution (D5) means `TaskScheduler` can select a `PooledDevice` owned by a remote host, but `dispatch()` will then raise `RemoteDispatchNotSupportedError` rather than executing it → **Mitigation**: `AssignmentStrategy`'s default (`fifo_match`) is configured, in this change's reference wiring, to only consider `PooledDevice`s whose `host_id` equals the dispatcher's own host id, so the common single-dispatcher-process deployment never hits the error path; multi-host dispatch is explicitly flagged in Open Questions as unimplemented, not silently broken.
|
||||
- **[Risk]** `plugin-system`'s driver-plugin registration path (D6) depends on `driver/registry.py` exposing a `register_driver_type()` function that does not exist yet in the pending `driver-registry` capability's own current spec (only a dict + a lookup function) → **Mitigation**: documented explicitly as a cross-change dependency in D6 and Open Questions rather than silently assumed; `PluginRegistry.register()` for `driver`-kind manifests fails loudly (raises, does not silently no-op) if `driver_registry.register_driver_type` is not importable, so the gap surfaces as an explicit error at plugin-load time, not a silent failure to actually register the driver.
|
||||
- **[Risk]** Accepting `tool`/`skill`-kind plugin manifests in the schema (D6) without wiring them anywhere could mislead a plugin author into thinking registering a tool/skill plugin has an effect → **Mitigation**: `PluginRegistry.register()` for `tool`/`skill`-kind manifests stores the manifest (so it is listable via `platform-sdk`'s `GET /v1/plugins`) but returns a `wired: false` field in its result and logs a clear "accepted, not yet wired to any execution path" message, rather than pretending success.
|
||||
- **[Trade-off]** No real auth on `platform-sdk` (D8) means anything reachable at the API's network address can submit tasks/query status today → **Acceptable** for this change since it is not defining deployment/network topology (Non-Goals) and the `AuthProvider` hook makes adding real enforcement a scoped follow-up, not a redesign.
|
||||
- **[Trade-off]** A fourth independent SQLite file (`cloud/cloud.sqlite3`, alongside `tasks/tasks.sqlite3`, `workflows/workflows.sqlite3`, and any skill-store databases) means fleet-level state, workflow state, and task-memory state are three separate stores with only string-id cross-references, no foreign-key enforcement → **Acceptable**; this mirrors the same trade-off `workflow-orchestration-runtime`'s D9 already accepted for its own store relative to `task-memory`'s, and unifying them is explicitly out of scope for any one milestone to force on another's still-pending schema.
|
||||
|
||||
## Migration Plan
|
||||
|
||||
This is purely additive; no existing module is edited:
|
||||
1. Add the `cloud/` package: `pool.py` (`DevicePool`, `HostRegistration`, `PooledDevice`, `sync_host_devices()`), `scheduler.py` (`TaskScheduler`, `ScheduledTask`, `AssignmentStrategy` registry with `fifo_match`), `dispatch.py` (`TaskDispatcher`, `Assignment`, `RemoteDispatchNotSupportedError`), `plugins.py` (`PluginManifest`, `PluginRegistry`, entry-point + manifest-file discovery), `store.py` (`CloudStore`, schema creation for `cloud/cloud.sqlite3`), `config.py` (heartbeat/staleness/queue/strategy/API-version defaults).
|
||||
2. Wire `TaskDispatcher`'s planned-goal path to `runtime.task.TaskRunner(...).run(task)` and its workflow path to `workflow.runner.WorkflowRunner(...).run(definition)` — import only, zero edits to `runtime/` or `workflow/`.
|
||||
3. Wire `PluginRegistry.register()`'s driver-kind path to `driver_registry.register_driver_type(name, builder)` if importable; raise a clear, typed error naming the missing function if `device-agent-runtime-foundation` has not yet landed that function when this change is implemented (see D6's Note and Open Questions) — do not stub around it silently.
|
||||
4. Add `cloud/sdk/`: `models.py` (Pydantic request/response models for the `/v1/...` routes), `api.py` (`create_cloud_router(*, pool, scheduler, plugin_registry) -> APIRouter`, mountable by whatever process assembles the full FastAPI app, same shape as `api/console.py`'s `create_console_router`), `client.py` (`CloudClient`, thin `httpx` wrapper).
|
||||
5. Add `cloud*` to `pyproject.toml`'s `[tool.setuptools.packages.find].include`; promote `httpx` from `[dependency-groups].dev` to `[project].dependencies` (needed at runtime by `cloud/sdk/client.py`).
|
||||
6. Add new tests (`tests/test_device_pool.py`, `tests/test_task_scheduler.py`, `tests/test_task_dispatcher.py`, `tests/test_plugin_registry.py`, `tests/test_cloud_sdk_api.py`-style): multi-host sync + staleness degrade, FIFO assignment with constraint matching, local dispatch composing a stub `TaskRunner`, remote-assignment raising `RemoteDispatchNotSupportedError`, plugin manifest validation + driver registration (with a fake `driver_registry.register_driver_type`), and REST route round-trips using FastAPI's `TestClient`.
|
||||
7. Run `pytest` — the full existing suite stays green with zero edits to existing test files, confirming this change touches no existing behavior.
|
||||
8. Rollback: net-new package plus a net-new SQLite file with its own schema; reverting is `git revert` of the commit(s), with no data migration of any existing store and no external system involved beyond whatever device the composed `TaskRunner`/`WorkflowRunner` calls already target.
|
||||
|
||||
## Open Questions
|
||||
|
||||
- How real cross-host task dispatch should eventually work once `RemoteDispatchNotSupportedError` needs to actually go away — a lightweight per-host agent process that polls `CloudStore` for assignments targeting its own `host_id` and calls `TaskDispatcher` locally, versus a real RPC/HTTP call from the scheduler's process to a remote host's own dispatcher — left unresolved here since it requires the "no infra" Non-Goal to be revisited first.
|
||||
- Whether `driver/registry.py`'s `register_driver_type()` function (assumed by D6's driver-plugin registration path) should be proposed as an addition to `device-agent-runtime-foundation` before or alongside this change's implementation, given `openspec/specs/` is empty and that change is itself still unapplied — flagged for whoever implements `plugin-system` to resolve with that change's owner rather than silently diverging.
|
||||
- Whether `tool`/`skill`-kind plugin manifests should get a real wiring path in a follow-up change once `tools/` and the skill stores (`skill-catalog-subscription`, `skill-learning-runtime`) gain their own public registration hooks — deliberately left declared-but-unwired here (D6, mirrors `skill-learning-runtime`'s own precedent of leaving skill *execution* unbuilt until `workflow-orchestration-runtime` needed it).
|
||||
- Whether `ScheduledTask`'s `constraints` should eventually accept a `skill_id`/`workflow_definition_id` directly (submit "run this workflow on any matching device") versus only a bare `goal` string — this change's `TaskDispatcher` supports both today by inspecting which field is populated, but the scheduler's own matching logic (constraints → candidate devices) does not yet special-case workflow-shaped submissions differently from goal-shaped ones; worth revisiting once real workflow submissions are exercised through `platform-sdk`.
|
||||
- Whether `platform-sdk`'s `AuthProvider` hook should be promoted to a real auth capability in its own right (its own openspec change) once a concrete integrator/security requirement exists, rather than staying a documented no-op — intentionally left as a future decision, not prejudged here.
|
||||
@@ -0,0 +1,34 @@
|
||||
## Why
|
||||
|
||||
Every capability built through Milestone 9 — `device-management`/`driver-registry` (`apex-agent-mvp`, `device-agent-runtime-foundation`), `agent-runtime`, `workflow-orchestration`, `multi-agent-collaboration` — still assumes one process, one `DeviceManager`, and a caller who already knows which `device_id` to target. That is fine for a single developer driving a handful of physical devices from one machine, but it does not scale to "many devices across many hosts, tasks submitted by external systems without knowing which device is free, and new drivers/tools/skills added without editing this repository's core packages." Milestone 10 (Cloud Runtime) is the platform-completion step: it introduces a device pool that aggregates device state across hosts, a scheduler that assigns queued tasks to idle devices instead of requiring callers to pick one, a plugin mechanism so a second driver (or a tool/skill) can register itself without touching `driver/`, `device/`, or `tools/`, and a versioned public SDK/API for external integrators that is distinct from the LLM-facing MCP tool server. None of this replaces the existing single-process, single-goal execution path — it composes it, the same way `workflow-orchestration-runtime` composed `TaskRunner` rather than rewriting it.
|
||||
|
||||
## What Changes
|
||||
|
||||
- Add a new `cloud/` package, sibling to `agents/`, `workflow/`, `semantic/`, `world/`, `skills_learning/`, holding all four new capabilities below. No existing package (`driver/`, `device/`, `runtime/`, `tools/`, `workflow/`, `agents/`, `storage/`, `api/console.py`, `api/mcp.py`) is modified.
|
||||
- Add `cloud/pool.py`: a `DevicePool` that aggregates device state across multiple hosts, each still running its own unmodified single-process `DeviceManager` (`device-management`). A per-host sync call (`DevicePool.sync_host_devices(host_id, snapshot)`) pushes that host's `DeviceManager.list_devices()` result into the pool on a heartbeat interval; the pool tracks `HostRegistration` (host id, address, last-seen) and `PooledDevice` (device id, owning host id, driver_type, status, capability tags) records, and marks a host's devices `unreachable` (a new pool-level status, distinct from `device-management`'s own `idle`/`busy`/`offline`/`error`) once its heartbeat goes stale — never raises or blocks on a missing host.
|
||||
- Add `cloud/scheduler.py`: a `TaskScheduler` that accepts a queued task submission (goal + optional device constraints: `driver_type`, required capability tags), holds it in a bounded FIFO queue backed by `cloud/store.py`, and assigns it to the first matching `idle` `PooledDevice` the `DevicePool` reports, via a pluggable `AssignmentStrategy` registry (default: FIFO + constraint match; a priority/capability-scoring strategy can be added later without changing `TaskScheduler`'s control flow).
|
||||
- Add `cloud/dispatch.py`: a `TaskDispatcher` that takes a `TaskScheduler` assignment (task + `device_id` + `host_id`) and actually runs it by composing the existing, unmodified `agent-runtime` `TaskRunner` (single-goal tasks) or `workflow-orchestration`'s `WorkflowRunner` (multi-step `WorkflowDefinition`s), strictly through their existing public `run(task) -> Task` / `run(definition) -> WorkflowRun` entry points — closing the loop from "assigned" to "executed" without a caller ever hand-picking a `device_id` again.
|
||||
- Add `cloud/plugins.py`: a `PluginManifest` schema (name, version, `entry_point_kind: driver | tool | skill`, module/callable reference) and a `PluginRegistry` that discovers manifests via Python `entry_points` (group `device_agent_runtime.plugins`) and/or a local `plugins/*/plugin.json` directory scan, validates them, and — for `driver`-kind manifests only — registers them into `driver-registry`'s existing `driver_type -> factory` extension point. `tool`- and `skill`-kind manifests are accepted and validated by the same schema but are declared-not-yet-wired in this change (see Non-Goals in design.md), since `tools/` and the skill stores (`skill-catalog-subscription`, `skill-learning-runtime`) do not yet expose a comparable public registration hook to compose against.
|
||||
- Add `cloud/sdk/` (`api.py`, `client.py`, `models.py`): a versioned (`/v1/...`) public REST API plus a thin Python client, for external integrators to submit tasks/workflows to the pool+scheduler, poll task/device/host status, and list/register plugins — a fleet-facing surface distinct from `mcp-tool-server`'s per-device, LLM-facing tool calls and from `web-console`'s operator-facing status/config UI (`console-status-api`/`console-config-api`), which both continue to talk to one process's own `DeviceManager` directly.
|
||||
- Add `cloud/store.py`: a new, independently-owned SQLite file (`cloud/cloud.sqlite3`) with `host_registrations`, `pooled_devices`, `scheduled_tasks`, and `plugins` tables — mirrors `workflow-orchestration-runtime`'s precedent of a capability-owned store rather than adding tables to `storage/task_metadata.py`.
|
||||
- Add `cloud/config.py`: heartbeat interval, host-staleness threshold, queue depth limit, default `AssignmentStrategy` name, SDK API version prefix — all with conservative defaults so this change is inert until a caller actually registers a second host or submits through the new SDK.
|
||||
- **BREAKING**: none. `driver/`, `device/`, `runtime/`, `tools/`, `workflow/`, `agents/`, `storage/`, `api/console.py`, and `api/mcp.py` are not modified; `cloud/` is purely additive and composes all of them by import.
|
||||
|
||||
## Capabilities
|
||||
|
||||
### New Capabilities
|
||||
- `device-pool`: A multi-host device registry/discovery layer (`DevicePool`, `HostRegistration`, `PooledDevice`) aggregating the device state each host's existing single-process `DeviceManager` already tracks, with heartbeat-based staleness detection so an unreachable host degrades its devices' pool-visible status rather than blocking or crashing the pool.
|
||||
- `task-scheduler`: A queue (`TaskScheduler`) plus pluggable `AssignmentStrategy` matching submitted tasks (goal + device constraints) to an idle `PooledDevice`, and a `TaskDispatcher` that executes an assignment by composing the existing `agent-runtime`/`workflow-orchestration` execution paths as black boxes.
|
||||
- `plugin-system`: A registration mechanism (`PluginManifest` + `PluginRegistry`, entry-points or manifest-file discovery) for new drivers/tools/skills to register themselves without editing `driver/`, `device/`, `tools/`, or the skill stores; concretely wired for driver-kind plugins into `driver-registry`'s existing extension point, with tool/skill-kind plugins schema-defined for forward compatibility.
|
||||
- `platform-sdk`: A versioned, public-facing REST API and Python client for external integrators to submit tasks/workflows, query device/host/task status, and manage plugins across the device pool — separate from `mcp-tool-server` (LLM-facing) and `console-status-api`/`console-config-api` (operator-facing, single-process).
|
||||
|
||||
### Modified Capabilities
|
||||
(none — `openspec/specs/` is currently empty and none of `device-management`, `driver-registry`, `agent-runtime`, `workflow-orchestration`, `mcp-tool-server`, `console-status-api`, `console-config-api` have an applied baseline to diff against; this change composes all of them by import/prose dependency only and does not alter their specified behavior.)
|
||||
|
||||
## Impact
|
||||
|
||||
- **New package**: `cloud/` — `pool.py` (`DevicePool`, `HostRegistration`, `PooledDevice`), `scheduler.py` (`TaskScheduler`, `AssignmentStrategy` registry, `ScheduledTask`), `dispatch.py` (`TaskDispatcher`), `plugins.py` (`PluginManifest`, `PluginRegistry`), `store.py` (`CloudStore`, schema for `cloud/cloud.sqlite3`), `config.py`, and `cloud/sdk/` (`api.py`, `client.py`, `models.py`).
|
||||
- **No changes** to `core/` (or `driver/`/`device/` once `device-agent-runtime-foundation` is applied), `runtime/`, `tools/`, `storage/`, `workflow/`, `agents/`, `api/console.py`, `api/mcp.py`, `api/rest.py` — every integration point is by import/composition, matching the precedent set by `workflow-orchestration-runtime` (D1, D8) and `multi-agent-runtime` (`CollaborativeTaskRunner` composing `TaskRunner`).
|
||||
- **Reads from pending capabilities (composition only, no spec changes to them)**: `device-management`/`driver-registry` for the per-host `DeviceManager`/driver-type extension point `device-pool` aggregates and `plugin-system` registers into; `agent-runtime` and `workflow-orchestration` for the `TaskRunner`/`WorkflowRunner` entry points `task-scheduler`'s `TaskDispatcher` invokes; `mcp-tool-server` as the sibling LLM-facing surface `platform-sdk` sits alongside without replacing; `console-status-api`/`console-config-api` (`web-console`) as the sibling operator-facing surface this change does not extend or duplicate.
|
||||
- **Config**: `pyproject.toml` gains a `cloud*` entry in `[tool.setuptools.packages.find].include`; no new third-party dependency is required for the pool/scheduler/plugin data models (plugin discovery uses the standard-library `importlib.metadata`), though `cloud/sdk/api.py` reuses the already-declared `fastapi`/`uvicorn` dependencies and `cloud/sdk/client.py` will need an HTTP client (reuse `httpx`, already a dev dependency — promote to a runtime dependency in `tasks.md`).
|
||||
- **Out of scope**: no actual multi-host deployment/infra config (Docker/Kubernetes manifests, service discovery infra) — only the runtime's internal pool/scheduler/plugin/SDK model; no billing, multi-tenant auth, or rate limiting on `platform-sdk` (a later change's concern); no cross-host network dispatch mechanism for actually invoking a remote host's `DeviceManager` over the wire (`TaskDispatcher` composes `TaskRunner`/`WorkflowRunner` only when the assignment lands on the local host — see design.md's Non-Goals and Open Questions for the remote-dispatch gap); no changes to `skill-catalog-subscription`, `skill-learning-runtime`, `web-console`, or `multi-agent-runtime`.
|
||||
@@ -0,0 +1,45 @@
|
||||
## ADDED Requirements
|
||||
|
||||
### Requirement: Host registration and heartbeat sync
|
||||
The system SHALL provide a `DevicePool` that tracks a `HostRegistration` (host id, address, last-seen timestamp) for each host process that registers itself, and SHALL update a host's last-seen timestamp whenever that host pushes a device snapshot via `sync_host_devices(host_id, snapshot)`.
|
||||
|
||||
#### Scenario: New host registers and syncs devices
|
||||
- **WHEN** a previously-unknown `host_id` calls `sync_host_devices` with a list of devices
|
||||
- **THEN** the pool creates a new `HostRegistration` for that host, records the current time as its last-seen timestamp, and stores each synced device as a `PooledDevice` owned by that host
|
||||
|
||||
#### Scenario: Known host re-syncs
|
||||
- **WHEN** an already-registered `host_id` calls `sync_host_devices` again with an updated device snapshot
|
||||
- **THEN** the pool updates that host's last-seen timestamp and replaces its previously-stored `PooledDevice` records with the new snapshot, without duplicating or losing devices from other hosts
|
||||
|
||||
### Requirement: Aggregated device listing across hosts
|
||||
The system SHALL provide a way to list all `PooledDevice` records across every registered host, including each device's owning `host_id`, `driver_type`, status, and capability tags.
|
||||
|
||||
#### Scenario: Listing devices across multiple hosts
|
||||
- **WHEN** two hosts have each synced a non-empty device snapshot
|
||||
- **THEN** a caller listing pool devices sees devices from both hosts in one combined result, each tagged with its correct `host_id`
|
||||
|
||||
#### Scenario: No hosts registered
|
||||
- **WHEN** a caller lists pool devices before any host has ever synced
|
||||
- **THEN** the pool returns an empty list rather than raising an error
|
||||
|
||||
### Requirement: Stale host devices degrade to unreachable
|
||||
The system SHALL mark all `PooledDevice`s belonging to a host `unreachable` once that host's last-seen timestamp exceeds a configured staleness threshold, computed at read time, without requiring any background process and without raising an error for the stale host's absence.
|
||||
|
||||
#### Scenario: Host misses its sync interval
|
||||
- **WHEN** a host's last-seen timestamp is older than `config.stale_after_seconds` at the time of a `list_devices()`/`get_device()` call
|
||||
- **THEN** every `PooledDevice` owned by that host is reported with status `unreachable`, regardless of the status value in its last-synced snapshot
|
||||
|
||||
#### Scenario: Host resumes syncing after being stale
|
||||
- **WHEN** a host previously marked stale calls `sync_host_devices` again
|
||||
- **THEN** its devices immediately stop being reported `unreachable` and reflect the statuses in the new snapshot
|
||||
|
||||
### Requirement: Device lookup by id across the pool
|
||||
The system SHALL allow looking up a single `PooledDevice` by `device_id` regardless of which host owns it, returning a clear not-found result when no host has ever reported that device id.
|
||||
|
||||
#### Scenario: Lookup finds device on any host
|
||||
- **WHEN** a caller requests a device by id that exists in some host's synced snapshot
|
||||
- **THEN** the pool returns that `PooledDevice` including its owning `host_id`
|
||||
|
||||
#### Scenario: Lookup for unknown device id
|
||||
- **WHEN** a caller requests a device by id that no host has ever synced
|
||||
- **THEN** the pool returns a not-found result (e.g. `None`) rather than raising an unhandled exception
|
||||
@@ -0,0 +1,63 @@
|
||||
## ADDED Requirements
|
||||
|
||||
### Requirement: Versioned public API surface
|
||||
The system SHALL expose the platform SDK's REST endpoints under a versioned URL prefix (`/v1/...`), distinct from the `mcp-tool-server` and `console-status-api`/`console-config-api` surfaces, so external integrators have a stable base path that will not silently change shape.
|
||||
|
||||
#### Scenario: Routes are mounted under the version prefix
|
||||
- **WHEN** the platform SDK's router is mounted into an application
|
||||
- **THEN** every route it exposes (task submission, status queries, device/host listing, plugin listing/registration) is reachable only under the `/v1/` prefix
|
||||
|
||||
### Requirement: Task submission and status via the SDK
|
||||
The system SHALL allow an external integrator to submit a task (goal or workflow reference plus constraints) through the platform SDK's API, and to query that task's current status by id, backed by the `task-scheduler` capability.
|
||||
|
||||
#### Scenario: Submit a task via the API
|
||||
- **WHEN** an integrator calls the task-submission endpoint with a valid goal and optional constraints
|
||||
- **THEN** the API returns a task id that can be used to poll status, and the underlying `task-scheduler` records a new `queued` `ScheduledTask`
|
||||
|
||||
#### Scenario: Query status of a known task
|
||||
- **WHEN** an integrator requests status for a task id that exists
|
||||
- **THEN** the API returns that task's current status (`queued`, `assigned`, `dispatched`, `done`, or `failed`)
|
||||
|
||||
#### Scenario: Query status of an unknown task
|
||||
- **WHEN** an integrator requests status for a task id that does not exist
|
||||
- **THEN** the API returns a not-found response rather than an unhandled server error
|
||||
|
||||
### Requirement: Device and host visibility via the SDK
|
||||
The system SHALL allow an external integrator to list devices and hosts known to the `device-pool` capability through the platform SDK's API.
|
||||
|
||||
#### Scenario: List devices across the pool
|
||||
- **WHEN** an integrator calls the device-listing endpoint
|
||||
- **THEN** the API returns every `PooledDevice` known to the pool, including owning host id and current (possibly `unreachable`) status
|
||||
|
||||
#### Scenario: List registered hosts
|
||||
- **WHEN** an integrator calls the host-listing endpoint
|
||||
- **THEN** the API returns every `HostRegistration` known to the pool, including last-seen timestamp
|
||||
|
||||
### Requirement: Plugin listing and registration via the SDK
|
||||
The system SHALL allow an external integrator to list registered plugins and submit a new plugin manifest for registration through the platform SDK's API, backed by the `plugin-system` capability.
|
||||
|
||||
#### Scenario: List registered plugins
|
||||
- **WHEN** an integrator calls the plugin-listing endpoint
|
||||
- **THEN** the API returns every registered `PluginManifest`, including its `entry_point_kind` and whether it is wired to an execution path
|
||||
|
||||
#### Scenario: Register a new plugin manifest
|
||||
- **WHEN** an integrator submits a valid plugin manifest to the plugin-registration endpoint
|
||||
- **THEN** the API registers it via `plugin-system`'s `PluginRegistry` and returns the stored manifest, or a clear validation/conflict error if registration fails
|
||||
|
||||
### Requirement: Pluggable authentication hook with a safe default
|
||||
The system SHALL evaluate every platform SDK route through a configurable `AuthProvider` hook, defaulting to a no-op provider that treats every caller as an anonymous, authenticated principal, so real authentication can be added later without changing route signatures.
|
||||
|
||||
#### Scenario: Default configuration allows anonymous access
|
||||
- **WHEN** no `AuthProvider` is explicitly configured
|
||||
- **THEN** every route accepts requests without rejecting them for lack of credentials
|
||||
|
||||
#### Scenario: Custom AuthProvider is honored
|
||||
- **WHEN** a caller configures a custom `AuthProvider` that rejects a request
|
||||
- **THEN** the platform SDK's routes return an authorization error for that request instead of proceeding, without any route's own handler code needing to change
|
||||
|
||||
### Requirement: Python SDK client mirrors the REST API
|
||||
The system SHALL provide a Python client (`CloudClient`) exposing methods corresponding to each `/v1/...` route (submit task, get task status, list devices, list hosts, list plugins, register plugin), so integrators do not need to hand-construct HTTP requests.
|
||||
|
||||
#### Scenario: Client submits a task and retrieves status
|
||||
- **WHEN** a caller uses `CloudClient` to submit a task and then fetch its status by the returned id
|
||||
- **THEN** the client's methods produce the same result as calling the corresponding `/v1/...` endpoints directly over HTTP
|
||||
@@ -0,0 +1,53 @@
|
||||
## ADDED Requirements
|
||||
|
||||
### Requirement: Plugin manifest schema
|
||||
The system SHALL define a `PluginManifest` schema with a unique `name`, a `version`, an `entry_point_kind` restricted to `driver`, `tool`, or `skill`, and a `target` (a dotted module:attribute reference to the plugin's implementation), and SHALL reject a manifest missing any required field or using an unrecognized `entry_point_kind`.
|
||||
|
||||
#### Scenario: Valid manifest accepted
|
||||
- **WHEN** a manifest with all required fields and a recognized `entry_point_kind` is submitted for registration
|
||||
- **THEN** the registry accepts it and stores it as a known plugin
|
||||
|
||||
#### Scenario: Manifest with unrecognized entry_point_kind rejected
|
||||
- **WHEN** a manifest declares an `entry_point_kind` other than `driver`, `tool`, or `skill`
|
||||
- **THEN** the registry rejects it with a clear validation error and does not register it
|
||||
|
||||
#### Scenario: Duplicate plugin name rejected
|
||||
- **WHEN** a manifest is submitted whose `name` matches an already-registered plugin
|
||||
- **THEN** the registry rejects the new registration with a clear conflict error rather than silently overwriting the existing entry
|
||||
|
||||
### Requirement: Plugin discovery via entry points and manifest files
|
||||
The system SHALL discover plugin manifests both from installed Python packages declaring an entry point in the `device_agent_runtime.plugins` group and from local `plugin.json` files under a configured scan path, feeding both sources into the same validation-and-registration path.
|
||||
|
||||
#### Scenario: Discovery via installed entry point
|
||||
- **WHEN** an installed package declares an entry point in the `device_agent_runtime.plugins` group resolving to a valid manifest
|
||||
- **THEN** `PluginRegistry.discover()` finds and registers it
|
||||
|
||||
#### Scenario: Discovery via local manifest file
|
||||
- **WHEN** a `plugin.json` file exists under the configured plugin scan path and parses into a valid manifest
|
||||
- **THEN** `PluginRegistry.discover()` finds and registers it
|
||||
|
||||
#### Scenario: Malformed manifest file is skipped, not fatal
|
||||
- **WHEN** a `plugin.json` file under the scan path fails to parse or fails schema validation
|
||||
- **THEN** `PluginRegistry.discover()` skips that file, records it as a discovery error, and continues discovering remaining plugins rather than aborting the whole scan
|
||||
|
||||
### Requirement: Driver-kind plugins register into the driver registry extension point
|
||||
The system SHALL, for a manifest with `entry_point_kind == "driver"`, resolve its `target` to a driver-factory builder and register it under the manifest's `name` as a new `driver_type` in the existing driver-registry extension point, without requiring any edit to the `driver` package's own files.
|
||||
|
||||
#### Scenario: Driver plugin registered successfully
|
||||
- **WHEN** a valid `driver`-kind manifest is registered and its `target` resolves to a callable driver-factory builder
|
||||
- **THEN** the manifest's `name` becomes usable as a `driver_type` value by any caller building a driver factory, with no change to existing driver-registry code
|
||||
|
||||
#### Scenario: Driver registry extension point unavailable
|
||||
- **WHEN** a `driver`-kind manifest is registered but the driver-registry's registration function is not importable in the running environment
|
||||
- **THEN** the registry raises a clear, explicit error naming the missing integration point, rather than silently accepting the manifest without wiring it
|
||||
|
||||
### Requirement: Tool and skill plugin manifests are accepted but explicitly marked unwired
|
||||
The system SHALL accept and store `tool`- and `skill`-kind plugin manifests (listable like any other registered plugin) but SHALL report them as not wired to any execution path, rather than implying they are active.
|
||||
|
||||
#### Scenario: Tool-kind manifest registered
|
||||
- **WHEN** a valid `tool`-kind manifest is registered
|
||||
- **THEN** the registry stores it and it appears in a plugin listing with a `wired: false` indicator, and no tool dispatch path is modified as a result
|
||||
|
||||
#### Scenario: Skill-kind manifest registered
|
||||
- **WHEN** a valid `skill`-kind manifest is registered
|
||||
- **THEN** the registry stores it and it appears in a plugin listing with a `wired: false` indicator, and no skill store or execution path is modified as a result
|
||||
@@ -0,0 +1,56 @@
|
||||
## ADDED Requirements
|
||||
|
||||
### Requirement: Task submission enqueues a scheduled task
|
||||
The system SHALL allow a caller to submit a task (a goal string, or a reference to a `WorkflowDefinition`, plus optional device constraints: `driver_type`, required capability tags) and SHALL enqueue it as a `ScheduledTask` with status `queued`, returning a stable task id the caller can poll.
|
||||
|
||||
#### Scenario: Successful submission
|
||||
- **WHEN** a caller submits a task with a goal and no constraints
|
||||
- **THEN** the scheduler creates a `ScheduledTask` with status `queued`, assigns it a unique id, and returns that id to the caller without blocking for a device to become available
|
||||
|
||||
#### Scenario: Queue depth limit reached
|
||||
- **WHEN** a caller submits a task while the queue already holds `config.max_queue_depth` queued tasks
|
||||
- **THEN** the scheduler rejects the submission with a clear error rather than accepting an unbounded backlog
|
||||
|
||||
### Requirement: Assignment matches a queued task to an idle, constraint-matching device
|
||||
The system SHALL assign a queued `ScheduledTask` to an idle `PooledDevice` (as reported by the `device-pool` capability) whose `driver_type` and capability tags satisfy the task's constraints, using a named, registrable `AssignmentStrategy`.
|
||||
|
||||
#### Scenario: Matching idle device available
|
||||
- **WHEN** `assign()` runs and at least one idle `PooledDevice` matches the head-of-queue task's constraints
|
||||
- **THEN** the scheduler selects one such device via the configured `AssignmentStrategy`, transitions the task to status `assigned`, and records the chosen `device_id`/`host_id`
|
||||
|
||||
#### Scenario: No matching device available
|
||||
- **WHEN** `assign()` runs and no idle `PooledDevice` matches the head-of-queue task's constraints
|
||||
- **THEN** the task remains `queued` (not failed), and `assign()` returns without error, ready to be retried on a later call
|
||||
|
||||
#### Scenario: Unknown assignment strategy configured
|
||||
- **WHEN** `TaskScheduler` is configured with an `AssignmentStrategy` name that is not registered
|
||||
- **THEN** the scheduler raises a clear configuration error at startup/first-assign rather than silently falling back to a default strategy
|
||||
|
||||
### Requirement: Assignment strategies are pluggable by name
|
||||
The system SHALL provide an `AssignmentStrategy` registry mapping a strategy name to an implementation, with a default `fifo_match` strategy (oldest-queued matching task first, first matching idle device), and SHALL allow a new strategy to be added by registering a name without modifying `TaskScheduler`'s control flow.
|
||||
|
||||
#### Scenario: Default FIFO strategy orders by submission time
|
||||
- **WHEN** two tasks with satisfiable, overlapping constraints are queued in order A then B, and one matching idle device exists
|
||||
- **THEN** the default `fifo_match` strategy assigns the device to task A, leaving task B queued
|
||||
|
||||
#### Scenario: Adding a new strategy requires no scheduler edit
|
||||
- **WHEN** a new `AssignmentStrategy` implementation is registered under a new name
|
||||
- **THEN** `TaskScheduler` can be configured to use it by name alone, with no change to `scheduler.py`'s assignment control flow
|
||||
|
||||
### Requirement: Local dispatch executes an assignment via existing runners
|
||||
The system SHALL provide a `TaskDispatcher` that, for an assignment whose device is owned by the local process's own host, executes the assigned task by composing the existing `agent-runtime` task-execution entry point (for a goal-based submission) or the `workflow-orchestration` workflow-execution entry point (for a workflow-based submission), without reimplementing planning/execution/retry logic.
|
||||
|
||||
#### Scenario: Dispatching a goal-based assignment
|
||||
- **WHEN** `TaskDispatcher.dispatch()` is called with an assignment for a goal-based `ScheduledTask` whose device is local
|
||||
- **THEN** the dispatcher constructs and runs a `Task` through the existing task-execution entry point, and updates the `ScheduledTask`'s status to `done` or `failed` based on the resulting task's outcome
|
||||
|
||||
#### Scenario: Dispatching a workflow-based assignment
|
||||
- **WHEN** `TaskDispatcher.dispatch()` is called with an assignment referencing a `WorkflowDefinition` whose device is local
|
||||
- **THEN** the dispatcher runs the definition through the existing workflow-execution entry point and updates the `ScheduledTask`'s status based on the resulting workflow run's outcome
|
||||
|
||||
### Requirement: Remote assignments are rejected explicitly, not silently ignored
|
||||
The system SHALL raise a distinct, typed error when `TaskDispatcher.dispatch()` is called for an assignment whose device is owned by a host other than the dispatching process's own host, rather than attempting execution or silently no-op'ing.
|
||||
|
||||
#### Scenario: Assignment targets a remote host's device
|
||||
- **WHEN** `TaskDispatcher.dispatch()` is called with an assignment whose `host_id` does not match the local process's own host id
|
||||
- **THEN** the dispatcher raises a `RemoteDispatchNotSupportedError` and leaves the `ScheduledTask`'s status unchanged from `assigned`
|
||||
@@ -0,0 +1,76 @@
|
||||
## 1. Package scaffolding
|
||||
|
||||
- [ ] 1.1 Create the `cloud/` package (`__init__.py`, `pool.py`, `scheduler.py`, `dispatch.py`, `plugins.py`, `store.py`, `config.py`) and the `cloud/sdk/` sub-package (`__init__.py`, `api.py`, `client.py`, `models.py`)
|
||||
- [ ] 1.2 Add `cloud*` to `[tool.setuptools.packages.find].include` in `pyproject.toml`
|
||||
- [ ] 1.3 Promote `httpx` from `[dependency-groups].dev` to `[project].dependencies` in `pyproject.toml` (needed at runtime by `cloud/sdk/client.py`)
|
||||
- [ ] 1.4 Implement `cloud/config.py`: `CloudConfig` dataclass with `sync_interval_seconds`, `stale_after_seconds`, `max_queue_depth`, `default_assignment_strategy`, `api_version_prefix` (`"/v1"`), `db_path` (`cloud/cloud.sqlite3`), each with a conservative documented default
|
||||
- [ ] 1.5 Extend the project's smoke test (that imports every package) to import `cloud` and `cloud.sdk`
|
||||
|
||||
## 2. Device pool data model and store (capability: device-pool)
|
||||
|
||||
- [ ] 2.1 Implement `cloud/pool.py`'s data types: `HostRegistration{host_id, address, last_seen_at}`, `PooledDevice{device_id, host_id, driver_type, status, capability_tags, synced_at}`
|
||||
- [ ] 2.2 Implement `cloud/store.py`'s `CloudStore(db_path)` with schema creation for `host_registrations` and `pooled_devices` tables (connect-per-call `sqlite3` pattern, following `storage/task_metadata.py`/`workflow/store.py`)
|
||||
- [ ] 2.3 Implement `CloudStore.upsert_host(host_id, address, last_seen_at)` and `CloudStore.replace_host_devices(host_id, devices: list[PooledDevice])` (atomic replace of one host's device rows per sync)
|
||||
- [ ] 2.4 Implement `CloudStore.list_hosts()`, `CloudStore.list_devices()`, `CloudStore.get_device(device_id)`
|
||||
- [ ] 2.5 Write unit tests for `CloudStore`: host upsert + device replace round-trip; a second sync for the same host fully replaces (not appends to) its device rows; devices from two different hosts coexist without collision
|
||||
|
||||
## 3. DevicePool aggregation and staleness (capability: device-pool)
|
||||
|
||||
- [ ] 3.1 Implement `DevicePool(store: CloudStore, config: CloudConfig)` with `sync_host_devices(host_id, snapshot: list[core.models.Device])`, converting each `Device` into a `PooledDevice` and calling `CloudStore.upsert_host()`/`replace_host_devices()`
|
||||
- [ ] 3.2 Implement `DevicePool.list_devices() -> list[PooledDevice]` and `DevicePool.get_device(device_id) -> PooledDevice | None`, both computing per-host staleness lazily at call time (`now - last_seen_at > config.stale_after_seconds` implies status `unreachable`, overriding the last-synced status) rather than via a background thread
|
||||
- [ ] 3.3 Implement `DevicePool.list_hosts() -> list[HostRegistration]`
|
||||
- [ ] 3.4 Write unit tests: new host sync creates a `HostRegistration` + `PooledDevice`s; re-sync updates last-seen and replaces devices; a host whose last-seen exceeds the staleness threshold reports all its devices `unreachable` on the next `list_devices()`/`get_device()` call; a host that resyncs after being stale immediately stops being reported unreachable; lookup for an unknown `device_id` returns `None`; empty pool returns an empty list
|
||||
|
||||
## 4. TaskScheduler queue and assignment (capability: task-scheduler)
|
||||
|
||||
- [ ] 4.1 Implement `cloud/scheduler.py`'s data types: `TaskConstraints{driver_type: str | None, capability_tags: list[str]}`, `ScheduledTask{id, goal: str | None, workflow_definition_id: str | None, constraints, status, assigned_device_id, assigned_host_id, created_at}`
|
||||
- [ ] 4.2 Add `scheduled_tasks` table + `CloudStore.enqueue_task()`, `CloudStore.list_queued_tasks()`, `CloudStore.update_task()`, `CloudStore.get_task(task_id)` to `cloud/store.py`
|
||||
- [ ] 4.3 Implement `AssignmentStrategy` protocol/ABC (`select(task, candidates: list[PooledDevice]) -> PooledDevice | None`) and a registry `dict[str, AssignmentStrategy]`
|
||||
- [ ] 4.4 Implement the default `fifo_match` strategy: return the first candidate in `candidates` (oldest-synced-first is not required; caller already filters to idle+constraint-matching) or `None` if `candidates` is empty
|
||||
- [ ] 4.5 Implement `TaskScheduler(pool: DevicePool, store: CloudStore, config: CloudConfig)` with `submit(goal=None, workflow_definition_id=None, constraints=None) -> str` (returns task id), raising a clear error if `config.max_queue_depth` queued tasks already exist
|
||||
- [ ] 4.6 Implement `TaskScheduler.assign() -> list[Assignment]`: for each queued task (oldest first), filter `pool.list_devices()` to `idle` devices matching `constraints.driver_type`/`capability_tags`, call the configured `AssignmentStrategy`, and on a match transition the task to `assigned` recording `device_id`/`host_id`; leave unmatched tasks `queued`
|
||||
- [ ] 4.7 Raise a clear configuration error if `config.default_assignment_strategy` names a strategy not present in the registry
|
||||
- [ ] 4.8 Write unit tests: submission enqueues with status `queued`; queue-depth-limit rejection; assignment picks a matching idle device via `fifo_match`; assignment leaves a task queued when no device matches; two tasks queued in order are assigned in submission order when only one device is available; unregistered strategy name raises at configuration/first-assign time
|
||||
|
||||
## 5. TaskDispatcher composition (capability: task-scheduler)
|
||||
|
||||
- [ ] 5.1 Implement `cloud/dispatch.py`'s `Assignment{task_id, device_id, host_id, goal, workflow_definition_id}` and `RemoteDispatchNotSupportedError`
|
||||
- [ ] 5.2 Implement `TaskDispatcher(local_host_id: str, task_runner_factory, workflow_runner_factory, store: CloudStore)` with `dispatch(assignment: Assignment) -> None`
|
||||
- [ ] 5.3 Implement the goal-based dispatch path: construct `core.models.Task(goal=assignment.goal, device_id=assignment.device_id)`, call the existing `runtime.task.TaskRunner(...).run(task)` (import only, no edits to `runtime/`), and update the `ScheduledTask`'s status to `done`/`failed` from `task.status`/`task.failure_reason`
|
||||
- [ ] 5.4 Implement the workflow-based dispatch path: load the referenced `WorkflowDefinition` and call the existing `workflow.runner.WorkflowRunner(...).run(definition, device_id=assignment.device_id)` (import only, no edits to `workflow/`), mapping the resulting `WorkflowRun.status` to the `ScheduledTask`'s status
|
||||
- [ ] 5.5 Implement the remote-assignment guard: if `assignment.host_id != local_host_id`, raise `RemoteDispatchNotSupportedError` before constructing any `Task`/`WorkflowDefinition`, leaving the `ScheduledTask` status unchanged at `assigned`
|
||||
- [ ] 5.6 Write unit tests: local goal-based dispatch runs a stubbed `TaskRunner` and updates status to `done`/`failed` correctly; local workflow-based dispatch runs a stubbed `WorkflowRunner` and updates status correctly; remote-host assignment raises `RemoteDispatchNotSupportedError` and leaves status as `assigned`
|
||||
|
||||
## 6. Plugin manifest and registry (capability: plugin-system)
|
||||
|
||||
- [ ] 6.1 Implement `cloud/plugins.py`'s `PluginManifest{name, version, entry_point_kind: Literal["driver", "tool", "skill"], target}` with validation (all fields required, `entry_point_kind` restricted to the three literals)
|
||||
- [ ] 6.2 Add a `plugins` table + `CloudStore.save_plugin()`, `CloudStore.list_plugins()`, `CloudStore.get_plugin(name)` to `cloud/store.py`, storing `wired: bool` alongside each manifest
|
||||
- [ ] 6.3 Implement `PluginRegistry(store: CloudStore)` with `register(manifest: PluginManifest) -> PluginManifest`: reject unknown `entry_point_kind`, reject duplicate `name`, persist via `CloudStore.save_plugin()`
|
||||
- [ ] 6.4 Implement driver-kind wiring: resolve `manifest.target` (dotted `module:attribute` string) to a `DriverFactoryBuilder` callable via `importlib`, and call `driver_registry.register_driver_type(manifest.name, builder)` if importable; raise a clear, named error if the driver-registry function is not importable in the running environment
|
||||
- [ ] 6.5 Implement tool-/skill-kind handling: store the manifest with `wired=False` and do not attempt any further resolution or registration
|
||||
- [ ] 6.6 Implement `PluginRegistry.discover_entry_points() -> list[PluginManifest]` using `importlib.metadata.entry_points(group="device_agent_runtime.plugins")`, resolving each entry point and registering the resulting manifest
|
||||
- [ ] 6.7 Implement `PluginRegistry.discover_manifest_files(scan_path) -> list[PluginManifest]` globbing `plugin.json` under `scan_path`, parsing and registering each; on parse/validation failure, record the file path + error and continue (never abort the scan)
|
||||
- [ ] 6.8 Implement `PluginRegistry.discover() -> DiscoveryResult{registered: list[PluginManifest], errors: list[str]}` combining both discovery sources
|
||||
- [ ] 6.9 Write unit tests: valid manifest registers successfully; unrecognized `entry_point_kind` rejected; duplicate name rejected; driver-kind manifest registers into a fake `driver_registry.register_driver_type`; driver-kind manifest raises a named error when that function is not importable; tool-/skill-kind manifests register with `wired=False` and touch no other registry; entry-point discovery registers a fake installed plugin; manifest-file discovery registers a valid file and skips + records a malformed one without aborting
|
||||
|
||||
## 7. Platform SDK REST API (capability: platform-sdk)
|
||||
|
||||
- [ ] 7.1 Implement `cloud/sdk/models.py`: Pydantic request/response models for task submission, task status, device listing, host listing, plugin listing, and plugin registration
|
||||
- [ ] 7.2 Implement `AuthProvider` protocol (`authenticate(request) -> Principal | None`) and `NullAuthProvider` (always returns an anonymous `Principal`) in `cloud/sdk/api.py` or a small `cloud/sdk/auth.py`
|
||||
- [ ] 7.3 Implement `create_cloud_router(*, pool: DevicePool, scheduler: TaskScheduler, plugin_registry: PluginRegistry, auth_provider: AuthProvider = NullAuthProvider(), version_prefix: str = "/v1") -> APIRouter` in `cloud/sdk/api.py`, mirroring `api/console.py`'s `create_console_router` shape
|
||||
- [ ] 7.4 Implement `POST {prefix}/tasks` (submit via `TaskScheduler.submit()`), `GET {prefix}/tasks/{task_id}` (status via `CloudStore.get_task()`, 404 on unknown id)
|
||||
- [ ] 7.5 Implement `GET {prefix}/devices` (via `DevicePool.list_devices()`) and `GET {prefix}/hosts` (via `DevicePool.list_hosts()`)
|
||||
- [ ] 7.6 Implement `GET {prefix}/plugins` (via `PluginRegistry`/`CloudStore.list_plugins()`) and `POST {prefix}/plugins` (via `PluginRegistry.register()`, returning a validation/conflict error response on failure)
|
||||
- [ ] 7.7 Wire every route through `auth_provider.authenticate(request)`, returning an authorization error response when it returns `None`
|
||||
- [ ] 7.8 Write unit tests using FastAPI's `TestClient`: submit-then-status round trip; unknown task id returns 404; device/host listing reflects pool state; plugin listing/registration round trip; a custom rejecting `AuthProvider` causes every route to return an authorization error while the default `NullAuthProvider` allows all of the above through unchanged
|
||||
|
||||
## 8. Python SDK client (capability: platform-sdk)
|
||||
|
||||
- [ ] 8.1 Implement `cloud/sdk/client.py`'s `CloudClient(base_url, *, http_client=None)` using `httpx`, with `submit_task()`, `get_task_status()`, `list_devices()`, `list_hosts()`, `list_plugins()`, `register_plugin()` methods matching `cloud/sdk/api.py`'s routes
|
||||
- [ ] 8.2 Write unit tests for `CloudClient` against a live `TestClient`-backed instance of the router from task 7.3: submit + status round trip via the client returns the same result as calling the routes directly
|
||||
|
||||
## 9. Composition safety checks and full-suite validation
|
||||
|
||||
- [ ] 9.1 Confirm no existing file under `driver/`/`core/`, `device/`, `runtime/`, `tools/`, `workflow/`, `agents/`, `storage/`, `api/console.py`, or `api/mcp.py` is modified by this change (composition via import only, per design.md's D1/D5)
|
||||
- [ ] 9.2 Write a test that runs a real (non-mocked, stub-driver-backed) `runtime.task.TaskRunner` instance inside `TaskDispatcher.dispatch()`'s goal-based path, guarding against silent drift in `agent-runtime`'s public `run(task) -> Task` contract this change composes over
|
||||
- [ ] 9.3 Run the full test suite (`pytest`) and confirm every existing test in `tests/` passes unmodified, with only new `tests/test_device_pool.py`, `tests/test_task_scheduler.py`, `tests/test_task_dispatcher.py`, `tests/test_plugin_registry.py`, `tests/test_cloud_sdk_api.py`, `tests/test_cloud_client.py`-style files added
|
||||
@@ -0,0 +1,2 @@
|
||||
schema: spec-driven
|
||||
created: 2026-07-06
|
||||
@@ -0,0 +1,82 @@
|
||||
## Context
|
||||
|
||||
`apex-agent-mvp` is code-complete (35/38 tasks; the remaining 3 require a physical iPhone and are unaffected by this change) but was never archived — `openspec/specs/` is still empty, so its capability specs (`device-management`, `scene-perception`, `agent-runtime`, `task-memory`, `mcp-tool-server`) exist only as pending deltas under `openspec/changes/apex-agent-mvp/specs/`. The working codebase already reflects a mostly driver-agnostic design: `Driver` is an ABC with no framework types leaking through, `WDADriver` is the only implementation, `DeviceManager` tracks devices by an opaque `driver_type` string, and `Scene` (not raw XML/OCR boxes) is the only thing the LLM-facing tools ever touch. The gaps are cosmetic-but-real: the package holding `Driver`+`DeviceManager` is called `core/` (a grab-bag name), the perception package is called `vision/` (names the technique, not the capability), the shared error base is literally `ApexAgentError`, and the one true "pick a driver by type" extension point (`SUPPORTED_DRIVER_TYPES` in `api/console.py`) lives in the API layer instead of the driver layer. Two other changes (`skill-catalog-subscription`, `web-console`) are proposed but unapplied and untouched by this change.
|
||||
|
||||
Two stakeholders: the LLM/agent runtime consuming the driver/device/tools/perception layers (must see zero behavior change), and future contributors/AI coding agents who will read `docs/CONSTITUTION.md` and `docs/ROADMAP.md` before adding the next driver or milestone.
|
||||
|
||||
## Goals / Non-Goals
|
||||
|
||||
**Goals:**
|
||||
- Zero behavior change: every existing test passes after the move with only import-path edits, not logic edits.
|
||||
- Make the physical package layout match the mental model: `driver/` (interface + implementations), `device/` (lifecycle/registry of *devices*, not drivers), `core/` (truly shared models/errors only).
|
||||
- Give driver-type registration a home in `driver/` so a second driver (Android, browser, ...) never requires touching `api/console.py`.
|
||||
- Give perception-technique registration the same treatment via a `PerceptionProvider` port, so OCR is one swappable implementation, not the pipeline's identity.
|
||||
- Produce durable planning artifacts (`ROADMAP.md`, two ADRs, `CONSTITUTION.md`, `research/` scaffold) that later milestone changes can be scoped against, instead of re-deriving architecture context from `apex-agent-mvp/design.md` each time.
|
||||
- Codify a **Hexagonal (Ports-and-Adapters) + DDD layering** as the project's governing architecture, with an explicit dependency direction (domain has zero framework/LLM/HTTP dependencies; adapters depend on domain, never the reverse) and a bottom-up build order that future milestone changes are expected to respect: `core` (domain) → `driver`/`device` (adapters) → `tools` (capability/ports layer) → `perception` (Scene, mockable) → `storage` → `runtime` (application/orchestration) → real perception techniques → LLM-backed planning → `api` (outermost adapter).
|
||||
|
||||
**Non-Goals:**
|
||||
- No second driver implementation (Android/browser/Windows) — only the extension point moves.
|
||||
- No changes to `skill-catalog-subscription` or `web-console` proposals/specs.
|
||||
- No change to `agent-runtime`, `task-memory`, or `mcp-tool-server` behavior/specs — only their import paths shift if they reference `core.driver`/`core.device_manager`/`vision.*`.
|
||||
- No archiving of `apex-agent-mvp` as part of this change — that remains a separate decision for the user (real-device verification tasks 7.1–7.3 are still outstanding). Note `apex-agent-mvp` already bundled Perception (real OCR), Planning (LLM), and API/MCP into one MVP change, ahead of the layering this change now codifies — that is accepted as-is (not rebuilt or reordered); the layering/build-order constraint below governs *future* milestone changes (Semantic, World, Skill, Workflow, Agent, Cloud Runtime), not a retroactive rewrite of already-shipped code.
|
||||
- No CI/lint pipeline changes beyond what's needed to keep `pytest` green — mechanical enforcement of the layering (e.g. import-linter contracts) is explicitly deferred, not done now.
|
||||
|
||||
## Decisions
|
||||
|
||||
### D1: Split `core/` into `driver/` + `device/`, keep `core/` as a shared-models-only package
|
||||
`driver/base.py` gets the `Driver` ABC (from `core/driver.py`); `driver/wda_driver.py` gets `WDADriver` (from `core/wda_driver.py`); `device/manager.py` gets `DeviceManager`/`DriverFactory`/`DEFAULT_MANAGER` (from `core/device_manager.py`). `core/models.py` (Bounds, Device, SceneElement, Scene, Task, Step) and `core/errors.py` stay in `core/`, since they're shared across driver, device, perception, runtime, and storage — not device- or driver-specific.
|
||||
- **Alternative considered**: Eliminate `core/` entirely and push models/errors into whichever package "owns" them most (e.g. `Scene`/`SceneElement` → `perception/`, `Task`/`Step` → `runtime/`). Rejected for this change — it would force `device/` to import from `perception/` and `runtime/` to import from `perception/`, creating cross-package coupling that doesn't exist today; splitting shared models is a legitimate follow-up but is a behavior-neutral naming change's job to avoid, not force.
|
||||
|
||||
### D2: Rename `vision/` → `perception/`
|
||||
Straight package rename, same file names inside (`ocr.py`, `ui_parser.py`, `scene_builder.py`, `icon_detector.py`). No merging or splitting.
|
||||
- **Alternative considered**: Keep `vision/` and only rename the *pipeline* concept in docs. Rejected — the proposal's own roadmap explicitly calls this stage "Perception," and leaving the package named after one input signal (OCR/vision) while `scene_builder.py` already fuses tree+OCR+vision is the exact naming drift this change exists to fix.
|
||||
|
||||
### D3: Driver Registry moves into `driver/registry.py`, `api/console.py` imports from it
|
||||
New `driver/registry.py` holds `SUPPORTED_DRIVER_TYPES: dict[str, DriverFactoryBuilder]` and `build_driver_factory(driver_type, connection_info)`, moved verbatim from `api/console.py` (currently lines ~17-50). `api/console.py` imports `build_driver_factory` from `driver.registry` instead of defining it. Adding `AndroidDriver` later means adding one entry to `driver/registry.py`, not touching `api/console.py`.
|
||||
- **Alternative considered**: Leave the registry in `api/console.py` and only document the intent to move it later. Rejected — this is precisely the kind of small, cheap, high-leverage structural fix Milestone 0 exists to make before a second driver shows up and someone has to thread a device-agnostic factory builder through an API-layer dict under time pressure.
|
||||
|
||||
### D4: Rename `ApexAgentError` → `DeviceRuntimeError`
|
||||
`core/errors.py`'s base class and its subclasses (`DriverError`, `DeviceNotFoundError`, `DeviceOfflineError`, `DeviceBusyError`, `ElementNotFoundError`, `TaskFailedError`) keep their names; only the base changes from `ApexAgentError` to `DeviceRuntimeError`. All `except ApexAgentError` / `raise ApexAgentError` call sites across `api/`, `tools/`, `runtime/`, `tests/` update accordingly.
|
||||
- **Alternative considered**: Keep `ApexAgentError` as a deprecated alias (`ApexAgentError = DeviceRuntimeError`) for backward compatibility. Rejected — there are no external consumers of this exception type yet (nothing has shipped), so a compatibility shim adds dead code for a compatibility need that doesn't exist.
|
||||
|
||||
### D5: Rebrand via `pyproject.toml` + root `README.md`, not a source-wide string sweep
|
||||
`pyproject.toml`'s `name` becomes `device-agent-runtime` and a `description` field is added; a root `README.md` (currently absent) states the new positioning and links to `docs/ROADMAP.md`. `apex-agent-mvp/proposal.md` and `design.md` are left untouched as a historical record of the original framing — they describe a change that already happened, and rewriting history in a planning artifact adds no value.
|
||||
- **Alternative considered**: Grep-and-replace every "Apex Agent" / "IPA" string across existing `openspec/changes/apex-agent-mvp/**` docs. Rejected per the chosen change scope (foundation docs + restructure only, not reconciling other pending changes) — those files are a record of a past decision, not living documentation.
|
||||
|
||||
### D6: Roadmap, ADR, Constitution, and research/ track are pure additions, no code coupling
|
||||
`docs/ROADMAP.md`, `docs/adr/0001-device-agnostic-runtime.md`, `docs/CONSTITUTION.md`, and `research/00N-*/README.md` are net-new Markdown files with no imports from or references into runtime code, so they carry zero risk to the passing test suite.
|
||||
- **Alternative considered**: Encode `docs/CONSTITUTION.md`'s invariants as enforced lint rules (e.g. import-linter contracts forbidding `tools/` from importing `driver/` directly). Deferred, not rejected — worth doing once there's a second driver to actually violate the boundary against; adding enforcement machinery for a boundary nothing has crossed yet is premature for a foundation change.
|
||||
|
||||
### D7: Adopt Hexagonal + DDD layering as the governing architecture, recorded in a second ADR
|
||||
`docs/adr/0002-layered-hexagonal-architecture.md` records the dependency direction and build order as project law: `core` (domain: `Bounds`/`Device`/`Scene`/`Task`/`Step`/errors) has zero framework, LLM, or transport dependencies; `driver`/`device` are adapters translating external device SDKs (Appium/WDA today) into the domain's `Driver` contract; `tools` is the capability/port layer the Agent Runtime calls; `perception` produces `Scene` behind the `PerceptionProvider` port (D8); `storage` persists timeline/task state; `runtime` (Planner/Executor) is the application layer that orchestrates the above; only at `runtime`'s Planner does an LLM enter the picture, and only `api` (REST/MCP) is HTTP/MCP-transport-aware. This is recorded as a standing constraint for *future* milestone changes (Semantic, World, Skill, Workflow, Agent, Cloud Runtime): none of them may introduce LLM or transport dependencies into `core`, `driver`, `device`, or `tools`.
|
||||
- **Alternative considered**: Leave this as implicit convention (as it already mostly was) rather than a written ADR. Rejected — the explicit trigger for writing it down now is that `apex-agent-mvp` already shipped Perception+Planning+API bundled in one MVP change, i.e. the layering was followed loosely, not strictly by construction; upcoming AI-heavy milestones (Semantic Scene's LLM call, Skill embedding retrieval, Multi-Agent roles) are exactly where it's cheapest to keep LLM/transport concerns out of the domain/adapter layers *before* they're written, not after.
|
||||
|
||||
### D8: `PerceptionProvider` port mirrors the Driver Registry pattern (D3)
|
||||
`perception/provider.py` defines a `PerceptionProvider` ABC with one method, `build_scene(screenshot, tree) -> Scene`. The existing OCR+tree fusion in `scene_builder.py` is wrapped as the default implementation (registered, not rewritten — same behavior, same `Scene` output). A `NullPerceptionProvider` returning an empty `Scene` (correct `width`/`height`, no elements) is added for tests and any environment without OCR dependencies installed. `tools/describe_screen.py` and `runtime/` depend on the port type, not on `scene_builder` directly.
|
||||
- **Alternative considered**: Skip the port and let callers import `scene_builder.build_scene()` directly (as today), documenting only that *conceptually* it's swappable. Rejected — the whole point of D7's constraint is that swapping/mocking perception must be a real, exercised code path (useful today for fast tests without OCR installed) rather than an aspiration that nothing currently proves; this mirrors why the Driver Registry (D3) was made a real registry instead of a documented convention.
|
||||
|
||||
## Risks / Trade-offs
|
||||
|
||||
- **[Risk]** Renaming `core.driver`/`core.device_manager`/`vision.*` import paths across 37 files by hand risks missing one and breaking an import at runtime rather than at test time → **Mitigation**: run the full `pytest` suite (all of `tests/`) after the move as the acceptance gate in `tasks.md`; a missed import surfaces immediately as a collection error, not a silent behavior change.
|
||||
- **[Risk]** Moving `SUPPORTED_DRIVER_TYPES`/`build_driver_factory` out of `api/console.py` could break `tests/test_console_api.py` if it patches/imports those names directly → **Mitigation**: check `test_console_api.py`'s imports as part of the move and update them alongside the production code, in the same task.
|
||||
- **[Risk]** Wrapping `scene_builder.py` behind `PerceptionProvider` could subtly change its return value if the wrapper reshapes data → **Mitigation**: the default provider must call the existing `scene_builder` function unmodified and return its result as-is; `tests/test_scene_builder.py` passing unchanged is the acceptance check, not a new test suite.
|
||||
- **[Trade-off]** Codifying D7's layering as an ADR/constitution now, without lint enforcement, means it's a convention future contributors (human or AI) must read and follow, not something that fails a build if violated → acceptable per D6's reasoning: enforcement machinery is worth adding once there's a concrete violation to enforce against (e.g. when a Semantic Scene change is tempted to call an LLM SDK from within `core`), not preemptively.
|
||||
- **[Trade-off]** Keeping `core/` alive (rather than eliminating it) means the package boundary story is "driver / device / core (shared) / perception / runtime / storage / api" — one more package than the roadmap's illustrative "driver/device/runtime" — acceptable because this codebase already has more layers than the roadmap's Milestone-1-only sketch assumed, and forcing models into whichever package "feels right" would create the cross-package coupling described in D1.
|
||||
- **[Trade-off]** Not archiving `apex-agent-mvp` as part of this change means `openspec/specs/` stays empty and this change's own capability (`driver-registry`) has no prior baseline to diff against — acceptable since `driver-registry` is genuinely new (ADDED, not MODIFIED) and archival ordering is the user's call, not a blocker for this change's own correctness.
|
||||
|
||||
## Migration Plan
|
||||
|
||||
1. Create new packages/files (`driver/`, `device/`, `perception/`) via `git mv` (preserves history) rather than copy+delete.
|
||||
2. Update every import site (37 files) from `core.driver`/`core.device_manager`/`vision.*` to the new paths; rename `ApexAgentError` → `DeviceRuntimeError` at all call sites.
|
||||
3. Add `perception/provider.py` (`PerceptionProvider`, `NullPerceptionProvider`, default provider wrapping `scene_builder.py`); point `tools/describe_screen.py`/`runtime/` at the port.
|
||||
4. Move `SUPPORTED_DRIVER_TYPES`/`build_driver_factory` into `driver/registry.py`; update `api/console.py`'s imports.
|
||||
5. Update `pyproject.toml` (`name`, `description`, `packages.find.include`).
|
||||
6. Run `pytest` — must be 100% green with no test-content changes (only import updates inside `tests/` itself where needed).
|
||||
7. Add `README.md`, `docs/ROADMAP.md`, `docs/adr/0001-device-agnostic-runtime.md`, `docs/adr/0002-layered-hexagonal-architecture.md`, `docs/CONSTITUTION.md`, `research/00N-*/README.md`.
|
||||
8. Rollback: since every step is a rename/move plus additive docs with no data migration, reverting is `git revert` of the commit(s); no runtime state or external system is touched.
|
||||
|
||||
## Open Questions
|
||||
|
||||
- Whether `core/` should eventually be renamed too (e.g. `shared/` or `domain/`) once it's clearer which models belong to which future layer (World Model in Milestone 6 will likely want to own more of `core/models.py`) — left open, not blocking this change.
|
||||
- Whether `docs/CONSTITUTION.md` should later be enforced mechanically (import-linter, custom lint rule) once a second driver exists to validate the boundary against — deferred to a future change per D6.
|
||||
- Whether the Semantic Scene (Milestone 5), World Model (Milestone 6), and Skill Learning (Milestone 7) changes — currently being drafted in parallel — correctly keep their LLM/embedding calls confined to a `runtime`-level (or new dedicated) layer rather than `perception`/`core`, per D7's constraint; worth a follow-up read once those changes are drafted.
|
||||
@@ -0,0 +1,36 @@
|
||||
## Why
|
||||
|
||||
This project started as "Apex Agent," an iPhone-specific automation platform, and `apex-agent-mvp` already implements a driver-independent `Driver` interface, a `WDADriver`, a `DeviceManager`, a Scene perception pipeline, an Agent Runtime, task-memory timelines, and an MCP/REST surface. The actual long-term value isn't "control an iPhone" — it's giving an LLM a stable way to operate *any* real-world device, with iPhone/WDA as just the first driver. Nothing about the implemented behavior needs to change to realize this, but the naming, package layout, and a couple of remaining structural gaps still assume "iPhone is the platform" rather than "iPhone is one driver." Fixing this now — before Android/Chrome/Windows drivers or a skill/workflow/multi-agent layer accrete on top — is far cheaper than un-tangling it later. This change is Milestone 0 (Foundation) of a longer roadmap: establish the device-agnostic runtime shape and the planning artifacts (roadmap, ADR, research track) that later milestones (Perception, Execution, Planning, Semantic, World, Skill, Workflow, Agent, Cloud Runtime) will build on.
|
||||
|
||||
## What Changes
|
||||
|
||||
- Reorganize `core/` into a `driver/` package (the `Driver` interface + concrete driver implementations, `WDADriver` first) and a `device/` package (`DeviceManager` and device lifecycle), leaving only genuinely shared domain models/errors in `core/`. No behavior changes — this is a structural move plus import updates.
|
||||
- Rename `vision/` to `perception/` so the screenshot → OCR → tree → Scene pipeline is named for what it produces (perception of a device's screen), not for one technique (OCR) inside it.
|
||||
- Rename the `ApexAgentError` exception base (and its "Apex Agent" framing in docstrings/comments) to a device-agnostic name, since it is the one place the old iPhone-only brand is baked into code rather than docs.
|
||||
- Relocate the driver-type-to-factory registry (currently `SUPPORTED_DRIVER_TYPES`/`build_driver_factory` inside `api/console.py`) into the `driver/` package as a first-class **Driver Registry**, so adding a new driver type (Android, browser, ...) never requires touching the API/console layer. **BREAKING**: `api/console.py`'s import path for driver-factory construction changes.
|
||||
- Rebrand the project from "Apex Agent" to **Device Agent Runtime** (working name; long-term direction is a "DeviceOS"-style universal device runtime, recorded as a roadmap milestone, not implemented now): update `pyproject.toml` project name/description, and add a root `README.md` stating the new positioning.
|
||||
- Add `docs/ROADMAP.md` capturing the milestone sequence (Foundation → Device → Perception → Execution → Planning → Semantic → World → Skill → Workflow → Agent → Cloud Runtime) and the three-phase delivery view, so future changes can be scoped against a shared plan instead of ad hoc.
|
||||
- Add `docs/adr/0001-device-agnostic-runtime.md` recording this repositioning as a formal ADR (context, decision, alternatives, consequences).
|
||||
- Add `docs/CONSTITUTION.md` capturing the durable architectural invariants that must hold regardless of milestone (Driver contract, `tools/` boundary, Scene as the only perception artifact the LLM sees, Planner/Executor split, stateless drivers), so future AI-assisted changes have a stable reference instead of re-deriving these from `apex-agent-mvp/design.md`.
|
||||
- Add a `research/` track scaffold (`001-scene-model` .. `007-ui-understanding`) with placeholder READMEs describing each track's purpose — no code, no benchmarks yet.
|
||||
- Introduce a **`PerceptionProvider` port** in `perception/`: the existing OCR+tree fusion (`scene_builder.py`) becomes the default implementation behind this port, and a `NullPerceptionProvider` (returns an empty `Scene`, no OCR dependency required) is added alongside it — so `tools/`, `runtime/`, and `api/` depend only on the port, never on a specific perception technique, matching the same extension-point pattern as the Driver Registry.
|
||||
- Codify **Hexagonal (Ports-and-Adapters) + DDD layering as the project's governing architecture**, with an explicit dependency direction and build order for future milestones: `core` (domain models/errors, zero framework deps) → `driver`/`device` (adapters over external device SDKs) → `tools` (capability layer) → `perception` (Scene, mockable via the port above) → `storage` (timeline/task persistence) → `runtime` (Planner/Executor, the application layer) → real perception techniques (OCR, vision) → LLM-backed Planner behavior → `api` (REST/MCP, the outermost adapter). Later milestones (Semantic, World, Skill, Workflow, Agent, Cloud Runtime) must build on top of this direction, not introduce LLM or transport-layer (HTTP/MCP) dependencies into `core`, `driver`, `device`, or `tools`.
|
||||
- No runtime/test behavior changes: every existing test in `tests/` must continue to pass with only import-path updates, not logic changes.
|
||||
|
||||
## Capabilities
|
||||
|
||||
### New Capabilities
|
||||
- `driver-registry`: A central, driver-agnostic registry mapping a `driver_type` string to a driver-factory builder, owned by the `driver/` package, used by any config/API surface (console, future CLI, future cloud scheduler) that needs to construct a driver for a device without knowing concrete driver classes.
|
||||
- `perception-provider`: A `PerceptionProvider` port in the `perception/` package, with the existing OCR+tree fusion registered as its default implementation and a `NullPerceptionProvider` added for tests/low-dependency development, so perception techniques are swappable the same way driver types are.
|
||||
|
||||
### Modified Capabilities
|
||||
(none — `device-management`, `scene-perception`, `agent-runtime`, `task-memory`, and `mcp-tool-server` keep their existing requirements from `apex-agent-mvp`; this change moves their implementation to new package paths and renames the project around them, but does not change any of their specified behavior.)
|
||||
|
||||
## Impact
|
||||
|
||||
- **Moved code**: `core/driver.py` → `driver/base.py`, `core/wda_driver.py` → `driver/wda_driver.py`, `core/device_manager.py` → `device/manager.py`; `core/models.py` and `core/errors.py` stay in `core/` as shared domain types (with `ApexAgentError` renamed). `vision/*` → `perception/*`. `SUPPORTED_DRIVER_TYPES`/`build_driver_factory` move from `api/console.py` into the new `driver/registry.py`.
|
||||
- **Import updates**: all 37 files across `core/`, `tools/`, `vision/`, `runtime/`, `storage/`, `api/`, and `tests/` that import `core.driver`, `core.device_manager`, `core.models`, `core.errors`, or `vision.*` need updated import paths.
|
||||
- **Config**: `pyproject.toml` — `name`, add `description`, and `[tool.setuptools.packages.find].include` list (`driver*`, `device*` replacing implicit `core*` scope, `perception*` replacing `vision*`).
|
||||
- **New docs, no code impact**: `README.md`, `docs/ROADMAP.md`, `docs/adr/0001-device-agnostic-runtime.md`, `docs/adr/0002-layered-hexagonal-architecture.md`, `docs/CONSTITUTION.md`, `research/**/README.md`.
|
||||
- **New port**: `perception/provider.py` (`PerceptionProvider` ABC, `NullPerceptionProvider`), with `scene_builder.py`'s existing fusion logic wrapped as the default provider — no change to `scene-perception`'s specified behavior.
|
||||
- **Out of scope**: no changes to `skill-catalog-subscription` or `web-console` (the other two pending, unapplied changes) — they are left as-is and can be re-read against `docs/ROADMAP.md` later if needed. No second driver (Android/etc.) is implemented in this change; only the registry extension point moves to where a second driver would plug in.
|
||||
@@ -0,0 +1,19 @@
|
||||
## ADDED Requirements
|
||||
|
||||
### Requirement: Driver type registry lives in the driver layer
|
||||
The system SHALL provide a registry, owned by the `driver` package, that maps a `driver_type` string (e.g. `"wda"`) to a builder function producing a `DriverFactory` for that type, so that any caller needing to construct a driver for a device does so without importing a concrete driver class directly.
|
||||
|
||||
#### Scenario: Building a factory for a known driver type
|
||||
- **WHEN** a caller requests a driver factory for `driver_type="wda"` with connection info (e.g. `server_url`, `udid`)
|
||||
- **THEN** the registry returns a `DriverFactory` that, when invoked, constructs a working `WDADriver` configured with that connection info
|
||||
|
||||
#### Scenario: Building a factory for an unknown driver type
|
||||
- **WHEN** a caller requests a driver factory for a `driver_type` that is not registered
|
||||
- **THEN** the registry raises a clear error naming the unsupported `driver_type`, instead of returning `None` or a factory that fails later at connect time
|
||||
|
||||
### Requirement: Adding a driver type requires no changes outside the driver layer
|
||||
The system SHALL allow a new driver type to be added by registering it in the `driver` package's registry alone; no other package (`api/`, `device/`, `tools/`, `runtime/`) SHALL need code changes to support constructing devices of the new type.
|
||||
|
||||
#### Scenario: API layer is agnostic to registered driver types
|
||||
- **WHEN** the console config API resolves a `driver_type` string into a driver factory to register a device
|
||||
- **THEN** it does so by calling into the driver registry rather than maintaining its own mapping of `driver_type` to concrete driver classes
|
||||
@@ -0,0 +1,22 @@
|
||||
## ADDED Requirements
|
||||
|
||||
### Requirement: Perception is exposed through a swappable provider port
|
||||
The system SHALL define a `PerceptionProvider` interface (`build_scene`) in the `perception` package that any concrete perception implementation (the existing OCR+tree fusion, a future cloud vision API, a future null/mock provider) must satisfy identically, so `tools/`, `runtime/`, and `api/` depend only on the port, never on a specific perception technique.
|
||||
|
||||
#### Scenario: Building a scene through the default provider
|
||||
- **WHEN** a caller requests a Scene for a screenshot and UI tree via the default (OCR+tree fusion) provider
|
||||
- **THEN** the provider returns a `Scene` identical in shape and content to what the existing fusion logic already produces — no change to `scene-perception`'s specified behavior
|
||||
|
||||
### Requirement: A null perception provider is available for testing and low-dependency development
|
||||
The system SHALL provide a `NullPerceptionProvider` that returns an empty `Scene` (correct screen width/height, zero elements) without requiring OCR/vision dependencies to be installed or invoked.
|
||||
|
||||
#### Scenario: Running without OCR dependencies installed
|
||||
- **WHEN** the runtime is configured to use `NullPerceptionProvider` (e.g. in a test or a minimal development environment)
|
||||
- **THEN** `describe_screen`/Agent Runtime calls succeed and return an empty Scene instead of failing due to missing OCR dependencies
|
||||
|
||||
### Requirement: Adding a perception technique requires no changes outside the perception layer
|
||||
The system SHALL allow a new perception provider (e.g. a cloud vision API) to be added by registering it within the `perception` package alone; `tools/`, `runtime/`, and `api/` SHALL NOT require code changes to use a newly registered provider.
|
||||
|
||||
#### Scenario: Runtime is agnostic to which provider is active
|
||||
- **WHEN** the Agent Runtime requests a Scene for the current screenshot and UI tree
|
||||
- **THEN** it does so through the `PerceptionProvider` port without importing or knowing about `scene_builder.py`, OCR, or any other concrete technique
|
||||
@@ -0,0 +1,54 @@
|
||||
## 1. Split `core/` into `driver/` and `device/`
|
||||
|
||||
- [x] 1.1 `git mv core/driver.py driver/base.py`, add `driver/__init__.py`, update its internal imports if any
|
||||
- [x] 1.2 `git mv core/wda_driver.py driver/wda_driver.py`, update its import of `core.driver` → `driver.base`
|
||||
- [x] 1.3 `git mv core/device_manager.py device/manager.py`, add `device/__init__.py`, update its imports of `core.driver`/`core.errors`/`core.models` → `driver.base`/`core.errors`/`core.models`
|
||||
- [x] 1.4 Confirm `core/__init__.py`, `core/models.py`, `core/errors.py` remain in place unchanged (shared domain types only)
|
||||
|
||||
## 2. Rename `vision/` to `perception/`
|
||||
|
||||
- [x] 2.1 `git mv vision perception` (carries `ocr.py`, `ui_parser.py`, `scene_builder.py`, `icon_detector.py`, `__init__.py`)
|
||||
- [x] 2.2 Update internal imports within the renamed package if any reference `vision.*`
|
||||
|
||||
## 3. Introduce the `PerceptionProvider` port
|
||||
|
||||
- [x] 3.1 Create `perception/provider.py` with a `PerceptionProvider` ABC (`build_scene(screenshot, tree) -> Scene`)
|
||||
- [x] 3.2 Add a default provider that wraps the existing `scene_builder.py` fusion logic unmodified (same inputs, same `Scene` output — no behavior change)
|
||||
- [x] 3.3 Add `NullPerceptionProvider` returning an empty `Scene` (correct `width`/`height`, zero elements), usable without any OCR dependency installed
|
||||
- [x] 3.4 Point `tools/describe_screen.py` (and any other caller of `scene_builder` directly) at the `PerceptionProvider` port instead of importing `scene_builder` directly
|
||||
- [x] 3.5 Confirm `tests/test_scene_builder.py` still passes unchanged against the default provider
|
||||
|
||||
## 4. Relocate the Driver Registry
|
||||
|
||||
- [x] 4.1 Create `driver/registry.py`; move `SUPPORTED_DRIVER_TYPES`, `DriverFactoryBuilder`, `build_wda_driver_factory`, and `build_driver_factory` out of `api/console.py` into it, importing `WDADriver`/`WDADriverConfig` from `driver.wda_driver` and `DriverFactory` from `device.manager`
|
||||
- [x] 4.2 Update `api/console.py` to import `build_driver_factory` (and `SUPPORTED_DRIVER_TYPES` if referenced) from `driver.registry` instead of defining them locally
|
||||
- [x] 4.3 Update `tests/test_console_api.py` (and any other test importing these names from `api.console`) to import from `driver.registry` where appropriate
|
||||
|
||||
## 5. Rename the shared error base
|
||||
|
||||
- [x] 5.1 In `core/errors.py`, rename `ApexAgentError` → `DeviceRuntimeError` (subclasses `DriverError`, `DeviceNotFoundError`, `DeviceOfflineError`, `DeviceBusyError`, `ElementNotFoundError`, `TaskFailedError` keep their names, just re-parented)
|
||||
- [x] 5.2 Update every `ApexAgentError` reference across `api/`, `tools/`, `runtime/`, `core/`, `driver/`, `device/`, `perception/`, and `tests/` to `DeviceRuntimeError`
|
||||
|
||||
## 6. Fix up all import sites and re-point package config
|
||||
|
||||
- [x] 6.1 Grep the full tree for `core.driver`, `core.device_manager`, `core.wda_driver`, and `vision.` and update every remaining import (spans `tools/`, `runtime/`, `storage/`, `api/`, `tests/`) to the new `driver.*`/`device.*`/`perception.*` paths
|
||||
- [x] 6.2 Update `pyproject.toml`: `[tool.setuptools.packages.find].include` to list `driver*`, `device*`, `perception*` alongside the existing `api*`, `core*`, `runtime*`, `storage*`, `tools*`
|
||||
- [x] 6.3 Run the full `pytest` suite and fix any remaining import errors until it is fully green with no test-logic changes
|
||||
|
||||
## 7. Rebrand project identity
|
||||
|
||||
- [x] 7.1 Update `pyproject.toml`: `name = "device-agent-runtime"`, add a one-line `description` reflecting the new positioning
|
||||
- [x] 7.2 Create root `README.md`: project positioning as a device-agnostic Device Agent Runtime (iPhone/WDA as the first driver), link to `docs/ROADMAP.md` and `docs/CONSTITUTION.md`
|
||||
|
||||
## 8. Planning artifacts
|
||||
|
||||
- [x] 8.1 Create `docs/ROADMAP.md` with the Milestone 0–10 sequence, the three delivery phases, and the long-term "DeviceOS" / Universal Device Runtime direction (v2.0, explicitly not started); explicitly map Milestones 1–4 to already-implemented `apex-agent-mvp` capabilities (`device-management`, `scene-perception`, `agent-runtime`, `task-memory`) rather than treating them as separate future work
|
||||
- [x] 8.2 Create `docs/adr/0001-device-agnostic-runtime.md` recording this repositioning decision (context, decision, alternatives considered, consequences), referencing this change's design.md decisions
|
||||
- [x] 8.3 Create `docs/adr/0002-layered-hexagonal-architecture.md` recording the Hexagonal/Ports-and-Adapters + DDD layering decision (D7): dependency direction (`core` domain has zero framework/LLM/HTTP deps; `driver`/`device` adapt external SDKs; `tools` is the capability/port layer; `perception` sits behind `PerceptionProvider`; `runtime` is the application layer where an LLM first enters via the Planner; `api` is the outermost transport adapter) and the bottom-up build order future milestones must respect
|
||||
- [x] 8.4 Create `docs/CONSTITUTION.md` capturing durable invariants: `Driver` is the only device-capability contract; `tools/` only calls into device/driver capabilities, never a concrete driver; `Scene` is the only perception artifact the LLM ever sees, produced through `PerceptionProvider`; drivers are stateless; Planner produces a plan, Executor is the only thing that calls tools and retries; **no LLM, HTTP, or MCP dependency may appear in `core`, `driver`, `device`, or `tools`** — those enter only at `runtime` (Planner) and `api` respectively; every future milestone change must state in its design.md how it upholds this boundary
|
||||
- [x] 8.5 Create `research/001-scene-model/README.md` through `research/007-ui-understanding/README.md` (scene-model, world-model, skill, agent, memory, planner, ui-understanding), each a short placeholder stating the track's purpose and that it holds papers/experiments/benchmarks, not code
|
||||
|
||||
## 9. Final verification
|
||||
|
||||
- [x] 9.1 Run `pytest` once more after all doc additions to confirm nothing in `tests/` was accidentally affected
|
||||
- [x] 9.2 Manually skim every touched file's diff to confirm no behavior changed — only paths, names, new port, and net-new docs
|
||||
@@ -0,0 +1,2 @@
|
||||
schema: spec-driven
|
||||
created: 2026-07-06
|
||||
@@ -0,0 +1,69 @@
|
||||
## Context
|
||||
|
||||
`agent-runtime` (`apex-agent-mvp`, code-complete but unapplied) runs a single Observe→Think→Act loop: `TaskRunner.run()` (`runtime/task.py`) calls `Planner.plan()` once per iteration and hands every produced `PlannedStep` to `Executor.execute()`, whose only failure signal is "did the tool call raise" — retried with bounded backoff (`max_retries`). Nothing in that loop ever asks "did the intended effect actually happen," using anything richer than the tool call's own return value: a `tap` on a send button can return `success=True` from the driver while the message never actually sent. `semantic-scene-runtime` (Milestone 5) and `world-model-runtime` (Milestone 6) are adding richer, semantically-informed state (`SemanticScene`, `WorldState`) that a single Planner+Executor pair has no natural place to use for verification or recovery — today that state is planning input only. This change adds three new collaborating roles — Observer, Verifier, Reflector — around the existing, unmodified Planner/Executor, so that semantic verification and bounded recovery become first-class steps in the loop, layered above (not replacing) the Executor's existing low-level retry/backoff safety net.
|
||||
|
||||
## Goals / Non-Goals
|
||||
|
||||
**Goals:**
|
||||
- Define a handoff protocol — `Observation`, `VerificationVerdict`, `ReflectionOutcome` — as plain dataclasses passed between roles, so the protocol itself, not any one role's internal implementation, is the stable, spec'd contract.
|
||||
- Add a `CollaborativeTaskRunner` that composes the existing `Planner`, `Executor`, and `TaskRunner` (`agent-runtime`) strictly by import, in a defined loop: Observer → Planner → Executor → Verifier → (Reflector only on a Verifier-flagged failure) → back to Observer.
|
||||
- Make multi-agent collaboration opt-in per task run, defaulting to **disabled**, so applying this change never silently changes any existing task's cost, latency, or step count.
|
||||
- Bound Reflector-triggered replanning with an explicit ceiling, separate from and on top of the Executor's own `max_retries`, so a persistently-failing step cannot loop forever between Verifier and Reflector.
|
||||
- Reuse the existing `semantic/llm_client.py` LLM client abstraction (added by `semantic-scene-runtime`) for Verifier/Reflector reasoning, rather than introducing a second LLM client abstraction.
|
||||
|
||||
**Non-Goals:**
|
||||
- No change to `runtime/planner.py`'s `Planner`, `runtime/executor.py`'s `Executor`, or `runtime/task.py`'s `TaskRunner` — multi-agent collaboration is an opt-in alternate driver of the same Task/Planner/Executor contracts, not a replacement for them.
|
||||
- No new LLM provider/vendor integration — Verifier/Reflector reasoning reuses the existing client abstraction.
|
||||
- No persistence of `Observation`/`VerificationVerdict`/`ReflectionOutcome` into `storage/timeline.py` in this change (see Open Questions).
|
||||
- No change to `workflow/runner.py`'s step-handler contract — `CollaborativeTaskRunner` is usable as an alternate driver for a Workflow's planned-goal step, but wiring that composition is left to whichever change adopts it, not authored here.
|
||||
|
||||
## Decisions
|
||||
|
||||
### D1: New `agents/` package, not folded into `runtime/`
|
||||
The Observer/Verifier/Reflector roles and the handoff protocol are conceptually "one layer above" the existing Observe→Think→Act loop, with different characteristics (LLM calls, optionality, per-task opt-in). A sibling `agents/` package (`models.py`, `observer.py`, `verifier.py`, `reflector.py`, `collab_runner.py`, `config.py`) keeps `runtime/`'s existing contract unchanged and makes the new roles' composition explicit at the package boundary, mirroring `semantic-scene-runtime`'s `semantic/` package precedent.
|
||||
**Alternative considered**: add verification/reflection hooks directly onto `TaskRunner`. Rejected — it would force every `TaskRunner` caller to reason about the new roles' failure modes and config even when not using them.
|
||||
|
||||
### D2: Handoff protocol as plain dataclasses with no role-specific side effects
|
||||
`Observation`, `VerificationVerdict`, and `ReflectionOutcome` (plus `ReflectionAction`) are plain dataclasses with `to_dict()`/`from_dict()`, mirroring `core/models.py`'s style. No role reaches into another role's internals; each role's public surface is "takes typed input, returns typed output."
|
||||
**Alternative considered**: pass a single mutable shared context object between roles (similar to `TaskContext`). Rejected — a shared mutable object makes it harder to reason about which role produced which piece of state, and the whole point of this change is making each handoff an explicit, testable data contract.
|
||||
|
||||
### D3: `CollaborativeTaskRunner` composes `TaskRunner`/`Planner`/`Executor` by import, not fork
|
||||
`agents/collab_runner.py` imports and calls into `runtime/task.py`'s `TaskRunner` as the underlying step-execution engine, reusing its existing planning/execution/timeline-append logic rather than reimplementing it. This is the same composition pattern `workflow-orchestration-runtime`'s `WorkflowRunner` already uses for its planned-goal steps.
|
||||
**Alternative considered**: fork `TaskRunner`'s loop logic into `agents/` to have full control over each iteration. Rejected — forking creates two divergent copies of step-execution logic that must be kept in sync; composing by import guarantees `CollaborativeTaskRunner` and plain `TaskRunner` runs stay behaviorally identical wherever the new roles don't intervene.
|
||||
|
||||
### D4: Verifier/Reflector reuse `semantic/llm_client.py`, not a new client abstraction
|
||||
Both roles need an LLM call (Verifier to judge whether a step's intended effect occurred; Reflector to propose a recovery action or replan request). Both reuse the existing `semantic/llm_client.py` abstraction added by `semantic-scene-runtime`, rather than introducing a second wrapper around the same underlying SDK.
|
||||
**Alternative considered**: give each role its own bespoke client. Rejected — would duplicate timeout/retry/structured-output handling already solved once in `semantic/llm_client.py`, with no benefit specific to Verifier/Reflector's use case.
|
||||
|
||||
### D5: Reflector proposes a bounded recovery action or a replan request, never a blind re-issue
|
||||
`Reflector.reflect(...)` is invoked only when the Verifier flags a step as not-achieved. It analyzes the `Observation`/`PlannedStep`/`StepResult`/verdict and returns a `ReflectionOutcome` carrying either a `ReflectionAction` (a bounded, different corrective action) or a replan request back to the Planner — never simply re-issuing the identical failed step, since that is already the Executor's own retry's job and re-issuing an already-failed-at-the-semantic-level step is unlikely to succeed differently.
|
||||
**Alternative considered**: let Reflector re-issue the same `PlannedStep` up to N times. Rejected — the Executor already owns mechanical (tool-call-level) retry; the Reflector's value is specifically in proposing something *different* when the mechanical retry already reported success but the semantic effect didn't happen.
|
||||
|
||||
### D6: Collaboration is opt-in per task run, defaulting to disabled
|
||||
A configuration flag (`agents/config.py`) enables `CollaborativeTaskRunner` per task run, defaulting to **disabled**, mirroring `semantic-scene-runtime`'s default-off precedent. When disabled, a task runs exactly as `runtime/task.py`'s existing `TaskRunner.run()` today.
|
||||
**Alternative considered**: default to enabled for all new tasks. Rejected — this change must not silently change any existing task's cost, latency, or step count; requiring an explicit opt-in keeps the blast radius at zero for callers who don't ask for it.
|
||||
|
||||
### D7: Reflection-recovery ceiling is a separate, explicit counter from the Executor's `max_retries`
|
||||
A configurable max-reflection-recovery-attempts ceiling (per task) is tracked independently in `agents/collab_runner.py`, distinct from and layered on top of `Executor`'s own `max_retries` (which bounds *mechanical* retries of a single tool call). Once the ceiling is reached, the loop stops attempting further reflection-driven recovery for that task and surfaces the failure instead of looping indefinitely.
|
||||
**Alternative considered**: reuse/extend `Executor.max_retries` to also cover reflection attempts. Rejected — conflating the two counters would make it impossible to reason independently about "how many times did the tool call itself get retried" versus "how many times did we try a semantically different recovery," which are different failure classes with different appropriate bounds.
|
||||
|
||||
## Risks / Trade-offs
|
||||
|
||||
- **[Risk] Verifier/Reflector LLM calls add latency and cost to every collaboratively-run step** → Mitigation: opt-in default-off (D6) means only callers who explicitly enable collaboration pay this cost; reuse of the existing cached/structured-output client (D4) keeps per-call overhead in line with `semantic-scene-runtime`'s established pattern.
|
||||
- **[Risk] Verifier/Reflector depend on `SemanticScene`/`WorldState`, both of which may be absent (disabled config, degrade path, or not-yet-applied milestones)** → Mitigation: Observer must work when `SemanticScene`/`WorldState` are `None`, falling back to the raw `Scene`/`PlannedStep`/`StepResult`, matching the existing skip/fallback degrade path those capabilities define.
|
||||
- **[Risk] Verifier/Reflector could loop indefinitely on a step that never semantically succeeds** → Mitigation: the explicit reflection-recovery ceiling (D7), independent of and on top of `Executor.max_retries`.
|
||||
- **[Risk] Verifier judgment quality (false positive/negative on "did the effect happen") is not directly testable against real model output in unit tests** → Mitigation: the LLM client is injected/mockable (following `semantic/llm_client.py`'s existing pattern), so unit tests exercise the verdict/reflection parsing and loop-control logic against canned responses; only a small, explicitly-marked integration test exercises the real API.
|
||||
|
||||
## Migration Plan
|
||||
|
||||
This is a purely additive change with no rollback complexity beyond removing the new package and config:
|
||||
1. Add the `agents/` package (`models.py`, `observer.py`, `verifier.py`, `reflector.py`, `collab_runner.py`, `config.py`); no change to `runtime/`, `core/models.py`, or `storage/timeline.py`.
|
||||
2. Add an `agents*` entry to `pyproject.toml`'s `[tool.setuptools.packages.find].include`; no new third-party dependency (reuses `semantic/llm_client.py`).
|
||||
3. Add collaboration config (enabled/disabled per task run, defaulting to disabled; max-reflection-recovery-attempts ceiling), sourced from the same single config location pattern `semantic-scene-runtime` established.
|
||||
4. Rollback is simply not enabling collaboration in config / not constructing a `CollaborativeTaskRunner`; no existing capability needs to be reverted.
|
||||
|
||||
## Open Questions
|
||||
|
||||
- Should `Observation`/`VerificationVerdict`/`ReflectionOutcome` be recorded in the timeline (`storage/timeline.py`) alongside `Task`/`StepResult`, or are they purely in-loop, non-persisted artifacts for this milestone? Leaning toward "not persisted in this change," mirroring `world-model-runtime`'s deferred `WorldState` persistence question, to keep the boundary with future milestones clean.
|
||||
- Is the reflection-recovery ceiling scoped per task or per step? Leaning toward per-task for this milestone (simpler to reason about and configure); revisit if real usage shows a single problematic step exhausting the budget for an otherwise-healthy task.
|
||||
- Should `CollaborativeTaskRunner`'s composition with `workflow-orchestration`'s `WorkflowRunner` (using it as the driver for a planned-goal step) be wired in this change or left to a follow-up change once both capabilities are applied? Leaning toward follow-up, since `workflow-orchestration-runtime`'s step-handler contract (`run(task) -> Task`) already accommodates either runner without modification.
|
||||
@@ -0,0 +1,29 @@
|
||||
## Why
|
||||
|
||||
`agent-runtime` (`apex-agent-mvp`, code-complete but unapplied) runs a single Observe→Think→Act loop: `TaskRunner.run()` calls `Planner.plan()` once per iteration and hands every produced `PlannedStep` to `Executor.execute()`, whose only failure signal is "did the tool call raise" — retried with bounded backoff. That is a *mechanical* success signal, not a *semantic* one: a `tap` on the send button can return `success=True` from the driver while the message never actually sent (wrong element, stale screen, silent app-level rejection), and nothing in the current loop would ever notice, because no role ever asks "did the intended effect actually happen" using anything richer than the tool call's own return value. As the runtime grows richer state to reason with — `SemanticScene` (Milestone 5, page identity/intents/widget purposes) and `WorldState` (Milestone 6, current app/page/variables/bounded history) — a single Planner+Executor pair has no natural place to *use* that richer state for verification or recovery; today it is planning input only. This change splits the loop's responsibilities across five collaborating roles — Observer, Planner, Executor, Verifier, Reflector — so that "did this step actually work" and "what should we do about it if not" become first-class, semantically-informed steps in the loop, layered above (not replacing) the Executor's existing low-level retry/backoff safety net.
|
||||
|
||||
## What Changes
|
||||
|
||||
- Add a new `agents/` package implementing three new roles — **Observer** (perceives and summarizes current device state via `SemanticScene`/`WorldState`, producing an `Observation`), **Verifier** (checks whether an already-*executed* step actually achieved its intended effect, comparing pre-step and post-step `Observation`s against the `PlannedStep`'s stated intent, distinct from and running after the Executor's own low-level retry), and **Reflector** (invoked only when the Verifier flags a step as not-achieved; analyzes the likely failure cause from the `Observation`/`PlannedStep`/`StepResult`/Verifier verdict and proposes either a bounded recovery action or a replan request, never a blind re-issue of the same step).
|
||||
- Add a `CollaborativeTaskRunner` (`agents/collab_runner.py`) that composes the existing, **unmodified** `Planner` and `Executor` (`agent-runtime`) with the three new roles in a defined handoff loop: `Observer → Planner → Executor → Verifier → (Reflector only on a Verifier-flagged failure) → back to Observer`. It reuses `runtime/task.py`'s `TaskRunner` as the underlying step-execution engine (composed by import, not forked) rather than reimplementing planning/execution/timeline-append logic.
|
||||
- Define the **handoff protocol** as typed data passed between roles (`Observation`, `VerificationVerdict`, `ReflectionOutcome`), each a plain dataclass with no role-specific side effects, so the protocol itself — not any one role's internal implementation — is the stable, spec'd contract.
|
||||
- Add configuration to enable/disable multi-agent collaboration per task run, defaulting to **disabled** (mirrors `semantic-scene-runtime`'s default-off precedent) so that applying this change does not silently change any existing task's cost, latency, or step count; when disabled, a task runs exactly as `runtime/task.py`'s existing `TaskRunner.run()` today.
|
||||
- Bound Reflector-triggered replanning with an explicit ceiling (a configurable max reflection-recovery attempts per task, separate from and on top of the Executor's own `max_retries`), so a persistently-failing step cannot loop forever between Verifier and Reflector.
|
||||
- **BREAKING**: none. `runtime/planner.py`'s `Planner`, `runtime/executor.py`'s `Executor`, and `runtime/task.py`'s `TaskRunner` are not modified by this change; multi-agent collaboration is an opt-in alternate driver of the same underlying Task/Planner/Executor contracts, not a replacement.
|
||||
|
||||
## Capabilities
|
||||
|
||||
### New Capabilities
|
||||
- `multi-agent-collaboration`: Defines the Observer/Verifier/Reflector roles, their `Observation`/`VerificationVerdict`/`ReflectionOutcome` data contracts, and the Observer→Planner→Executor→Verifier→(Reflector)→Observer handoff protocol that composes with the existing Planner/Executor (`agent-runtime`) as a higher semantic-verification layer above (not a replacement for) the Executor's own bounded low-level retries, usable both for a single-goal task and for one step of a multi-step Workflow (`workflow-orchestration`).
|
||||
|
||||
### Modified Capabilities
|
||||
(none — `agent-runtime`'s `Planner`, `Executor`, and `TaskRunner` (`apex-agent-mvp`) keep their existing requirements unchanged; `CollaborativeTaskRunner` composes them strictly by import as an opt-in alternate driver, and neither `semantic-scene` nor `world-model` has their specified behavior altered by this change, see Impact.)
|
||||
|
||||
## Impact
|
||||
|
||||
- **New package**: `agents/` — `models.py` (`Observation`, `VerificationVerdict`, `ReflectionOutcome`, `ReflectionAction` dataclasses), `observer.py` (`Observer.observe(...) -> Observation`), `verifier.py` (`Verifier.verify(...) -> VerificationVerdict`), `reflector.py` (`Reflector.reflect(...) -> ReflectionOutcome`), `collab_runner.py` (`CollaborativeTaskRunner`, the handoff loop), `config.py` (enable flag, max-reflection-attempts ceiling).
|
||||
- **No change** to `runtime/planner.py`, `runtime/executor.py`, `runtime/task.py`, `core/models.py`, `storage/timeline.py`, or any existing tool in `tools/` — all existing tests and callers keep working unmodified; `CollaborativeTaskRunner` composes `TaskRunner`/`Planner`/`Executor` strictly by import, as `workflow-orchestration-runtime`'s `WorkflowRunner` already does for its planned-goal steps.
|
||||
- **Reads from pending capabilities (read-only composition, no spec changes to them)**: `semantic-scene` (Milestone 5) for `SemanticScene` as Observer/Verifier input (with the existing skip/fallback degrade path honored — Observer must work when `SemanticScene` is `None`), `world-model` (Milestone 6) for `WorldState` as additional Observer/Verifier context, and `agent-runtime` (`apex-agent-mvp`) for `Task`/`PlannedStep`/`StepResult`/`TaskContext` as the underlying vocabulary every new role's inputs/outputs are built from.
|
||||
- **Composable with `workflow-orchestration`** (Milestone 8): a `WorkflowDefinition`'s planned-goal step may be driven by `CollaborativeTaskRunner` instead of the plain `TaskRunner`, without any change to `workflow/runner.py`'s step-handler contract (`run(task) -> Task` in, `Task` out) — this change does not itself modify `workflow/`.
|
||||
- **Config**: `pyproject.toml` gains an `agents*` entry in `[tool.setuptools.packages.find].include`; no new third-party dependency (roles reuse `semantic/llm_client.py`'s existing LLM client abstraction for Verifier/Reflector reasoning, added in Milestone 5, not a new SDK).
|
||||
- **Out of scope**: no new LLM provider/vendor integration; no removal or modification of the Executor's existing retry/backoff logic; no persistence of `Observation`/`VerificationVerdict`/`ReflectionOutcome` into `storage/timeline.py` (left as an open question, mirroring `world-model-runtime`'s deferred `WorldState` persistence question); no change to `skill-catalog-subscription`, `web-console`, or `skill-learning-runtime`.
|
||||
@@ -0,0 +1,59 @@
|
||||
## ADDED Requirements
|
||||
|
||||
### Requirement: Handoff protocol data contracts
|
||||
The system SHALL define `Observation`, `VerificationVerdict`, `ReflectionOutcome`, and `ReflectionAction` as plain dataclasses with `to_dict()`/`from_dict()` methods, forming the stable, spec'd contract passed between the Observer, Verifier, and Reflector roles, independent of any single role's internal implementation.
|
||||
|
||||
#### Scenario: Handoff dataclasses round-trip through serialization
|
||||
- **WHEN** an `Observation`, `VerificationVerdict`, or `ReflectionOutcome` instance is serialized via `to_dict()` and then reconstructed via `from_dict()`
|
||||
- **THEN** the reconstructed instance is equal to the original instance
|
||||
|
||||
### Requirement: Observer produces an Observation from available device state
|
||||
The system SHALL provide an Observer role that produces an `Observation` summarizing current device state, using `SemanticScene` and `WorldState` when available and falling back to the raw `Scene`/`PlannedStep`/`StepResult` when either or both are absent.
|
||||
|
||||
#### Scenario: Observation succeeds with SemanticScene and WorldState present
|
||||
- **WHEN** `Observer.observe(...)` is called and both `SemanticScene` and `WorldState` are available
|
||||
- **THEN** it returns an `Observation` incorporating both as context
|
||||
|
||||
#### Scenario: Observation degrades gracefully when semantic state is absent
|
||||
- **WHEN** `Observer.observe(...)` is called and `SemanticScene` and/or `WorldState` is `None`
|
||||
- **THEN** it returns an `Observation` built from the raw `Scene`/`PlannedStep`/`StepResult` instead of raising an exception
|
||||
|
||||
### Requirement: Verifier checks whether an executed step achieved its intended effect
|
||||
The system SHALL provide a Verifier role that, after a `PlannedStep` has already been executed by the existing `Executor`, compares a pre-step and post-step `Observation` against the step's stated intent and produces a `VerificationVerdict` indicating whether the intended effect was actually achieved, distinct from and running after the Executor's own low-level tool-call retry.
|
||||
|
||||
#### Scenario: Verifier confirms an achieved effect
|
||||
- **WHEN** `Verifier.verify(...)` is called with a pre-step `Observation`, a post-step `Observation`, the executed `PlannedStep`, and its `StepResult`, and the post-step `Observation` reflects the step's stated intent
|
||||
- **THEN** it returns a `VerificationVerdict` marking the step as achieved
|
||||
|
||||
#### Scenario: Verifier flags a mechanically-successful but semantically-failed step
|
||||
- **WHEN** the `StepResult` reports mechanical success but the post-step `Observation` does not reflect the step's stated intent
|
||||
- **THEN** `Verifier.verify(...)` returns a `VerificationVerdict` marking the step as not achieved
|
||||
|
||||
### Requirement: Reflector proposes bounded recovery, never a blind re-issue
|
||||
The system SHALL invoke a Reflector role only when the Verifier produces a not-achieved `VerificationVerdict`, and the Reflector SHALL analyze the `Observation`/`PlannedStep`/`StepResult`/verdict to produce a `ReflectionOutcome` carrying either a bounded, distinct recovery `ReflectionAction` or a replan request back to the Planner, never an outcome that simply re-issues the identical failed step.
|
||||
|
||||
#### Scenario: Reflector proposes a distinct recovery action
|
||||
- **WHEN** `Reflector.reflect(...)` is invoked following a not-achieved `VerificationVerdict` and a distinct corrective action is identifiable
|
||||
- **THEN** it returns a `ReflectionOutcome` carrying a `ReflectionAction` that differs from the originally failed `PlannedStep`
|
||||
|
||||
#### Scenario: Reflector requests a replan when no bounded recovery action applies
|
||||
- **WHEN** `Reflector.reflect(...)` is invoked and no bounded corrective action is identifiable from the available `Observation`/`PlannedStep`/`StepResult`/verdict
|
||||
- **THEN** it returns a `ReflectionOutcome` carrying a replan request rather than re-issuing the failed step
|
||||
|
||||
### Requirement: Reflection-driven recovery is bounded by an explicit ceiling
|
||||
The system SHALL enforce a configurable maximum number of Reflector-triggered recovery attempts per task, tracked independently of and in addition to the Executor's own `max_retries`, so that a persistently-failing step cannot loop indefinitely between the Verifier and Reflector.
|
||||
|
||||
#### Scenario: Reflection loop stops once the ceiling is reached
|
||||
- **WHEN** a task's Verifier-Reflector loop reaches the configured maximum reflection-recovery attempts without a step being verified as achieved
|
||||
- **THEN** the `CollaborativeTaskRunner` stops attempting further reflection-driven recovery for that task and surfaces the failure instead of continuing the loop
|
||||
|
||||
### Requirement: Multi-agent collaboration composes existing Planner/Executor without modifying them
|
||||
The system SHALL provide a `CollaborativeTaskRunner` that composes the existing `Planner`, `Executor`, and `TaskRunner` (`agent-runtime`) by import, without modifying `runtime/planner.py`, `runtime/executor.py`, or `runtime/task.py`, and SHALL default to disabled so that applying this capability does not alter any existing task's behavior, latency, or cost unless explicitly enabled.
|
||||
|
||||
#### Scenario: Collaboration disabled by default leaves existing task behavior unchanged
|
||||
- **WHEN** a task is run without explicitly enabling multi-agent collaboration
|
||||
- **THEN** it executes exactly as `runtime/task.py`'s existing `TaskRunner.run()` today, with no Observer/Verifier/Reflector invoked
|
||||
|
||||
#### Scenario: Enabling collaboration does not require changes to Planner or Executor
|
||||
- **WHEN** a task is run with multi-agent collaboration explicitly enabled via `CollaborativeTaskRunner`
|
||||
- **THEN** the existing `Planner.plan()` and `Executor.execute()` are invoked unchanged, with the Observer/Verifier/Reflector roles composed around them
|
||||
@@ -0,0 +1,48 @@
|
||||
## 1. Package scaffolding
|
||||
|
||||
- [ ] 1.1 Create the `agents/` package (`__init__.py`, `models.py`, `observer.py`, `verifier.py`, `reflector.py`, `collab_runner.py`, `config.py`)
|
||||
- [ ] 1.2 Add an `agents*` entry to `[tool.setuptools.packages.find].include` in `pyproject.toml` (no new third-party dependency)
|
||||
- [ ] 1.3 Add collaboration configuration: enabled/disabled flag (default disabled) and max-reflection-recovery-attempts ceiling, sourced from a single place `agents/config.py` reads from
|
||||
- [ ] 1.4 Extend the project's smoke test (that imports every package) to import `agents`
|
||||
|
||||
## 2. Handoff protocol data models (capability: multi-agent-collaboration)
|
||||
|
||||
- [ ] 2.1 Implement `agents/models.py`: `Observation`, `VerificationVerdict`, `ReflectionOutcome`, and `ReflectionAction` dataclasses with `to_dict()`/`from_dict()`, mirroring the style of `core/models.py`
|
||||
- [ ] 2.2 Write unit tests for each dataclass round-tripping through `to_dict()`/`from_dict()`
|
||||
|
||||
## 3. Observer role (capability: multi-agent-collaboration)
|
||||
|
||||
- [ ] 3.1 Implement `agents/observer.py`: `Observer.observe(...) -> Observation`, perceiving current device state via `SemanticScene`/`WorldState` when available
|
||||
- [ ] 3.2 Make `Observer.observe(...)` work when `SemanticScene` and/or `WorldState` are `None`, falling back to the raw `Scene`/`PlannedStep`/`StepResult`
|
||||
- [ ] 3.3 Write unit tests for `Observer.observe(...)` covering: both `SemanticScene`/`WorldState` present, both absent, and each present individually
|
||||
|
||||
## 4. Verifier role (capability: multi-agent-collaboration)
|
||||
|
||||
- [ ] 4.1 Implement `agents/verifier.py`: `Verifier.verify(pre_observation, post_observation, planned_step, step_result) -> VerificationVerdict`, comparing pre-step and post-step `Observation`s against the `PlannedStep`'s stated intent
|
||||
- [ ] 4.2 Wire `Verifier` to reuse `semantic/llm_client.py`'s existing LLM client abstraction (injectable/mockable) rather than a new client
|
||||
- [ ] 4.3 Write unit tests for `Verifier.verify(...)` against a fake client covering: verified-achieved, verified-not-achieved, and client-failure degrade cases
|
||||
|
||||
## 5. Reflector role (capability: multi-agent-collaboration)
|
||||
|
||||
- [ ] 5.1 Implement `agents/reflector.py`: `Reflector.reflect(observation, planned_step, step_result, verdict) -> ReflectionOutcome`, invoked only on a Verifier-flagged not-achieved verdict
|
||||
- [ ] 5.2 Ensure `Reflector.reflect(...)` never returns an outcome that simply re-issues the identical failed `PlannedStep`; it returns either a distinct `ReflectionAction` or a replan request
|
||||
- [ ] 5.3 Write unit tests for `Reflector.reflect(...)` covering: recovery-action outcome, replan-request outcome, and client-failure degrade case
|
||||
|
||||
## 6. CollaborativeTaskRunner and handoff loop (capability: multi-agent-collaboration)
|
||||
|
||||
- [ ] 6.1 Implement `agents/collab_runner.py`: `CollaborativeTaskRunner`, composing the existing `Planner`, `Executor`, and `runtime/task.py`'s `TaskRunner` strictly by import
|
||||
- [ ] 6.2 Implement the handoff loop: Observer → Planner → Executor → Verifier → (Reflector only on a Verifier-flagged failure) → back to Observer
|
||||
- [ ] 6.3 Implement the reflection-recovery ceiling: a configurable max-attempts counter, independent of and layered on top of `Executor.max_retries`, that stops further reflection-driven recovery once exhausted and surfaces the failure
|
||||
- [ ] 6.4 Write unit tests for `CollaborativeTaskRunner` covering: a task that completes without any Verifier-flagged failure, a task recovered via one Reflector-proposed action, and a task that exhausts the reflection-recovery ceiling
|
||||
|
||||
## 7. Config wiring and opt-in behavior (capability: multi-agent-collaboration)
|
||||
|
||||
- [ ] 7.1 Confirm collaboration is disabled by default: constructing/running a task without explicitly enabling it behaves exactly as plain `TaskRunner.run()` today
|
||||
- [ ] 7.2 Write a unit test asserting `runtime/planner.py`'s `Planner`, `runtime/executor.py`'s `Executor`, and `runtime/task.py`'s `TaskRunner` are unmodified/unaffected by this change (no accidental coupling introduced from `agents/`)
|
||||
|
||||
## 8. End-to-end validation
|
||||
|
||||
- [ ] 8.1 Write an end-to-end test simulating a full collaborative task run against a mocked `Driver`/`Scene`/`SemanticScene`/`WorldState` and a mocked LLM client, asserting the loop completes normally whether verification succeeds or triggers reflection-driven recovery
|
||||
- [ ] 8.2 Write an integration test (skippable without network/API credentials, following the existing `apex-agent-mvp` skippable-integration-test pattern) exercising Verifier/Reflector against the real LLM client
|
||||
- [ ] 8.3 Confirm multi-agent collaboration stays disabled by default after applying this change (no existing task's behavior, latency, or cost changes unless a caller explicitly opts in)
|
||||
- [ ] 8.4 Run the full test suite (`pytest`) and confirm no existing test in `tests/` needed a behavior change, only additive new tests
|
||||
@@ -0,0 +1,2 @@
|
||||
schema: spec-driven
|
||||
created: 2026-07-06
|
||||
@@ -0,0 +1,74 @@
|
||||
## Context
|
||||
|
||||
`apex-agent-mvp` (code-complete, unapplied) already produces a `Scene` for every Observe step: `core/models.py` defines `Scene { width, height, elements: [SceneElement] }` and `SceneElement { id, type, bounds, text, confidence, source }`, built by `vision/scene_builder.py` (planned rename: `perception/scene_builder.py` per `device-agent-runtime-foundation`) by fusing OCR output and the device's UI tree via bounding-box IoU. `runtime/planner.py` (`Planner.plan()`) is currently a stub that returns one hardcoded `describe_screen` step and then nothing — there is no real LLM call anywhere in the codebase yet. This change introduces the **first real LLM integration point**: a single enrichment call per Observe step that turns a `Scene` into a compact `SemanticScene` (page identity, supported intents, per-widget purpose labels), so a real LLM-driven `Planner` (a later milestone) and other prompts can consume a stable, low-token JSON summary instead of re-deriving page semantics from raw element geometry or a screenshot every step.
|
||||
|
||||
Because this is the first LLM call in the project, this design also sets the pattern (client shape, structured-output mechanism, failure handling) that later milestones (a real `Planner`, World Runtime's cross-step memory, Skill synthesis) are expected to reuse rather than inventing their own.
|
||||
|
||||
## Goals / Non-Goals
|
||||
|
||||
**Goals:**
|
||||
- Given a `Scene`, produce a `SemanticScene` — `{page: str, intents: list[str], widgets: [{element_id, purpose}]}` — using exactly one LLM call.
|
||||
- Guarantee the enrichment call never blocks or fails the Observe→Act loop: any failure (timeout, rate limit, malformed response, disabled config) degrades to "no semantic scene," and callers fall back to the raw `Scene`.
|
||||
- Keep the LLM client abstraction narrow and swappable (mockable in tests without network access), scoped inside the new `semantic/` package rather than becoming a project-wide `llm/` dependency other capabilities must adopt.
|
||||
- Produce output that is schema-valid JSON by construction (not "usually valid, occasionally needs a repair pass"), since downstream code will parse it directly into `SemanticScene`.
|
||||
- Make enrichment optional and cheap enough to run on every Observe step without materially changing task latency or cost profile.
|
||||
|
||||
**Non-Goals:**
|
||||
- No change to `scene-perception`'s `Scene` format or fusion logic — `SemanticScene` is a separate, additive artifact layered on top, never a replacement.
|
||||
- No persistent cross-step semantic state, caching of `SemanticScene` across steps, or diffing between steps — that is Milestone 6 (World Runtime).
|
||||
- No skill synthesis or action recommendation derived from intents/purposes — that is Milestone 7 (Skill).
|
||||
- No change to `Planner`/`Executor` control flow itself — this change only makes a new artifact available; wiring a real LLM-driven `Planner` to consume `SemanticScene` is left to a later milestone (see Migration Plan).
|
||||
- No multi-provider LLM abstraction (e.g. supporting both Anthropic and OpenAI behind a common interface) — a single concrete client is enough for this milestone; a provider-agnostic port can be extracted later if a second provider is actually needed (YAGNI).
|
||||
|
||||
## Decisions
|
||||
|
||||
### D1: New `semantic/` package, not folded into `perception/` or `runtime/`
|
||||
`SemanticScene` is conceptually "one layer above Scene," but it has fundamentally different runtime characteristics: it makes a network call, it can fail/timeout, and it is optional. Mixing it into `perception/` (which is currently synchronous, local-only, and always-on) would force every `perception/` consumer to reason about network failure modes it doesn't otherwise have. A sibling `semantic/` package (`models.py`, `llm_client.py`, `enricher.py`, `prompts.py`) keeps `perception/`'s contract unchanged and makes the enrichment layer's optionality explicit at the package boundary.
|
||||
**Alternative considered**: add an `enrich()` method directly on `Scene` or inside `scene_builder.py`. Rejected — it would make `perception/` depend on an LLM SDK and network config, contradicting `device-agent-runtime-foundation`'s stated dependency direction (`perception` must stay a local, deterministic pipeline; LLM-backed behavior is explicitly scoped to sit above it, closer to `runtime`).
|
||||
|
||||
### D2: `enrich_scene()` returns `SemanticScene | None`, never raises for expected failure modes
|
||||
`enricher.enrich_scene(scene, *, context=None) -> SemanticScene | None` catches all expected LLM-client failure modes (timeout, rate limit, connection error, malformed/schema-invalid response, enrichment disabled by config) internally and returns `None` on any of them, logging the reason. Callers (tools, later a real `Planner`) treat `None` as "use the raw `Scene`," never as an exception to handle.
|
||||
**Alternative considered**: raise a typed `EnrichmentUnavailableError` and require every caller to catch it. Rejected — experience from `runtime/executor.py`'s existing retry/backoff pattern shows try/except-per-call-site is exactly the kind of boilerplate that leads to a caller eventually forgetting to catch it, silently turning a "nice-to-have" feature into a hard dependency. A `None`-returning function makes the degrade path the *only* path for callers to write, not an opt-in one.
|
||||
|
||||
### D3: Structured output via schema-constrained response, not free-text parsing or prompt-based JSON coaxing
|
||||
The enrichment call uses the Messages API's structured-output mechanism (`output_config: {format: {type: "json_schema", schema: {...}}}`, or the SDK's `messages.parse()` helper with a `SemanticScene`-shaped Pydantic model) so the response is schema-valid JSON by construction, rather than asking the model to "reply with JSON" in the prompt and parsing/repairing free text. This directly eliminates an entire class of degrade-path triggers (truncated/wrapped/commented JSON) that a prompt-only approach would otherwise have to detect and handle.
|
||||
**Alternative considered**: frame enrichment as a tool call with `strict: true` tool-input validation instead of a direct structured message reply. Rejected for this milestone — a direct structured response is simpler (no tool-loop bookkeeping) for a single, non-interactive extraction call; the tool-call pattern is more valuable when the model needs to choose between actions, which does not apply here.
|
||||
|
||||
### D4: Default model is a small/fast tier (`claude-haiku-4-5`), overridable via config
|
||||
Enrichment is a bounded, low-complexity extraction task (label a page, list a handful of intents, tag widgets already enumerated in the `Scene`) invoked once per Observe step, so latency and per-call cost dominate the choice over raw capability. `claude-haiku-4-5` is the default: cheapest/fastest tier that still supports schema-constrained structured output. The model is a config value, not hardcoded, so a later milestone can upgrade the default if enrichment quality proves insufficient in practice without a code change.
|
||||
**Alternative considered**: default to the project's most capable tier (as a generic "always use the best model" policy would suggest). Rejected specifically for this call site — the enrichment task is simple extraction over an already-structured `Scene` (not open-ended reasoning), and this call sits in the hot path of every Observe step, where added latency directly slows the whole task loop. A stronger model remains available via config for callers who need it (e.g. a harder-to-classify page).
|
||||
|
||||
### D5: No extended-thinking / effort tuning on the enrichment call
|
||||
The enrichment call omits `thinking` entirely and does not set `output_config.effort`. `claude-haiku-4-5` is a fast-extraction model; forcing extra reasoning depth on a bounded-output classification/labeling task increases latency and cost with no expected quality benefit here, and some effort/thinking parameter combinations are rejected outright on newer model tiers depending on model family. Keeping the request shape minimal (system prompt + schema + scene JSON) also maximizes what can be prompt-cached (see D6).
|
||||
**Alternative considered**: enable adaptive thinking for robustness on ambiguous screens. Rejected for the default path — if a later milestone finds enrichment quality insufficient on complex/ambiguous screens, this is a config-level model/parameter change, not a structural one.
|
||||
|
||||
### D6: Prompt caching on the fixed system/schema prefix
|
||||
The enrichment system prompt (instructions + JSON schema description) is static across every call within a task (and across tasks), while only the `Scene` JSON body varies per call. The client marks the system-prompt block with a `cache_control: {"type": "ephemeral"}` breakpoint so repeated per-step calls within a task reuse the cached prefix instead of paying full input-token price every step. This is a pure cost optimization with no behavior change; if the fixed prefix is ever short enough to fall under a given model's minimum cacheable-prefix length, caching simply has no effect (not an error) — the design must not depend on caching succeeding for correctness.
|
||||
**Alternative considered**: no caching, accept repeated system-prompt cost. Rejected — enrichment is invoked once per Observe step, so a multi-step task repeats the identical instructions/schema text on every call; caching this prefix is a low-effort, no-risk win once the per-step call pattern exists.
|
||||
|
||||
### D7: New `describe_screen_semantic` tool wraps the existing tool, existing tool untouched
|
||||
A new `tools/describe_screen_semantic.py` calls the existing `tools/describe_screen.py` to get a `Scene`, then calls `enrich_scene()`, and returns `{scene, semantic_scene}` (with `semantic_scene` possibly `None`). `tools/describe_screen.py` itself is not modified — existing callers (including the current `Planner` stub and any wiring in `runtime/executor.py`'s `default_tool_registry()`) keep working unchanged, and the new tool is additive to the registry.
|
||||
**Alternative considered**: add an `enrich: bool = False` kwarg directly to `describe_screen()`. Rejected — it would make `tools/describe_screen.py` depend on the new `semantic/` package (and transitively on LLM client config) even for callers who never pass `enrich=True`, which cuts against `scene-perception`'s existing "no LLM dependency" boundary. A separate wrapper tool keeps the dependency opt-in at the import level, not just the call-site level.
|
||||
|
||||
## Risks / Trade-offs
|
||||
|
||||
- **[Risk] Enrichment latency adds to every Observe step even when unused by the caller** → Mitigation: enrichment is only invoked through the new `describe_screen_semantic` tool / explicit `enrich_scene()` call, never automatically inside the existing `describe_screen`; callers who don't need it pay zero extra latency.
|
||||
- **[Risk] LLM cost scales with task step count (one call per Observe step)** → Mitigation: cheap default model tier (D4) + prompt caching of the fixed prefix (D6); config exposes a global enable/disable switch so cost-sensitive environments (CI, offline dev) can turn enrichment off entirely.
|
||||
- **[Risk] Enrichment quality (wrong page label, missed intent, mislabeled widget purpose) is not directly testable against real model output in unit tests** → Mitigation: `llm_client` is injected/mockable in `enricher.py`, so unit tests exercise the parsing/degrade-path logic against canned responses; only a small, explicitly-marked integration test exercises the real API (matching the existing `apex-agent-mvp` pattern of skippable hardware/network integration tests).
|
||||
- **[Risk] `element_id` values in `widgets[].element_id` could drift from the `Scene`'s actual element IDs if the model hallucinates or omits one** → Mitigation: `enricher.py` validates every returned `element_id` against the input `Scene`'s element ID set post-response; any widget referencing an unknown ID is dropped (not treated as a hard failure) rather than propagated as a dangling reference.
|
||||
- **[Risk] Introducing the project's first external LLM dependency adds a new operational failure mode (network, auth, rate limits) that didn't exist before** → Mitigation: this is exactly why D2's `None`-returning contract and the mandatory fallback-to-raw-`Scene` path exist; the entire capability is designed so this failure mode degrades gracefully rather than being a new way for tasks to fail.
|
||||
|
||||
## Migration Plan
|
||||
|
||||
This is a purely additive change with no rollback complexity beyond removing the new package and config:
|
||||
1. Add the `semantic/` package and `tools/describe_screen_semantic.py`; wire the new tool into `runtime/executor.py`'s `default_tool_registry()` under its own key (e.g. `"describe_screen_semantic"`), alongside (not replacing) the existing `describe_screen` entry.
|
||||
2. Add the `anthropic` SDK dependency and `semantic*` to `pyproject.toml`'s package-find include list; add enrichment config (enabled/disabled, model name) with enrichment defaulting to **disabled** until a caller explicitly opts in, so applying this change never silently changes existing task behavior or cost.
|
||||
3. No data migration, no changes to `storage/`'s timeline format in this change — if a later milestone wants `SemanticScene` persisted per step, that is a `task-memory` capability change to propose separately.
|
||||
4. Rollback is simply not calling the new tool / leaving enrichment disabled in config; no existing capability needs to be reverted.
|
||||
|
||||
## Open Questions
|
||||
|
||||
- Should `SemanticScene` (when present) be recorded in `TaskContext`/the timeline alongside `Scene`, or is it purely a same-step, non-persisted artifact until World Runtime (Milestone 6) defines cross-step state? Leaning toward "not persisted in this change" to keep the boundary with Milestone 6 clean, but this affects whether `runtime/task.py` needs any change at all in this milestone.
|
||||
- Should the default enrichment model become configurable per-task (e.g. a harder app might warrant a stronger tier) or is a single global default sufficient until real usage data exists? Leaning toward global-default-only for this milestone.
|
||||
- Exact timeout budget for the enrichment call (e.g. 5s vs 10s) before triggering the degrade path — needs to be tuned against real latency data once implemented; not fixed in this design.
|
||||
- Should intents be a fully open-vocabulary list (whatever the model says) or validated against a small controlled vocabulary to keep them stable/comparable across steps and pages? Leaning open-vocabulary for this milestone since no downstream consumer (yet) needs a fixed taxonomy; revisit if World/Skill milestones need one.
|
||||
@@ -0,0 +1,28 @@
|
||||
## Why
|
||||
|
||||
`scene-perception` (from `apex-agent-mvp`, code-complete but unapplied) fuses a screenshot, the device's UI tree, and OCR output into a `Scene`: screen dimensions plus a flat list of typed elements (bounds, text, confidence, source). That is still a *structural* description of the screen — it tells the LLM "there is a button-shaped element at (x, y) with text 'Send'," not "this is the WeChat chat screen and tapping that button sends the message." Every planning step, the LLM has to re-derive page identity and intent from raw element geometry, which burns context on repeated low-level reasoning and is brittle to layout drift (the same page re-derived slightly differently step to step). This change adds a **Semantic Runtime**: a single additional LLM enrichment call that turns a `Scene` into a `SemanticScene` — page identity, a short list of plain-language supported intents, and a purpose label per widget — so most subsequent prompts can rely on a compact, stable JSON summary instead of re-parsing raw geometry or screenshots. This is Milestone 5 (Semantic) of the device-agnostic runtime roadmap established by `device-agent-runtime-foundation`, sitting directly on top of the existing Perception milestone's `Scene` output.
|
||||
|
||||
## What Changes
|
||||
|
||||
- Add a new `semantic/` package that consumes an existing `Scene` (from `perception/`, per the `scene-perception` capability) and produces a `SemanticScene`: `{page: str, intents: list[str], widgets: [{element_id: str, purpose: str}]}`.
|
||||
- Introduce the **first real LLM integration point** in the codebase: a narrow, swappable LLM client abstraction used only by `semantic/` to make one structured-output call per enrichment (model, prompt, and structured-schema details are an implementation decision in `design.md`, not part of this proposal's contract).
|
||||
- Define a mandatory **degrade path**: if the enrichment LLM call is unavailable, times out, or fails for any reason, semantic enrichment is skipped and callers fall back to using the raw `Scene` directly — enrichment failure must never block or fail the Observe→Act loop.
|
||||
- Add a new tool-level entry point (e.g. `describe_screen_semantic` or an `enrich` flag on the existing describe-screen flow) so `runtime/` and `api/` callers can opt into the enriched view without every existing caller of `describe_screen` needing to change.
|
||||
- Add configuration to enable/disable semantic enrichment globally (so it can be turned off entirely in environments without LLM access, e.g. tests or offline development) without touching `scene-perception`.
|
||||
- **BREAKING**: none. This is a purely additive layer; nothing in `scene-perception`, `agent-runtime`, or any other existing capability changes shape or behavior.
|
||||
|
||||
## Capabilities
|
||||
|
||||
### New Capabilities
|
||||
- `semantic-scene`: Builds a `SemanticScene` (page identity, supported intents, per-widget purpose labels) from an existing `Scene` via one additional LLM call, with a defined skip/fallback degrade path when that call is unavailable or fails, so enrichment is strictly additive and non-blocking to the Observe→Act loop.
|
||||
|
||||
### Modified Capabilities
|
||||
(none — `scene-perception`'s `Scene` format and requirements from `apex-agent-mvp` are read-only input to this change and are not modified; `agent-runtime`'s Planner/Executor loop shape is not changed by this proposal, only optionally consumed by it, see `design.md` for how a future milestone might wire it in.)
|
||||
|
||||
## Impact
|
||||
|
||||
- **New package**: `semantic/` — `models.py` (`SemanticScene`, `SemanticWidget` dataclasses), `llm_client.py` (narrow LLM client wrapper + typed result), `enricher.py` (`enrich_scene(scene, ...) -> SemanticScene | None`), `prompts.py` (the enrichment system prompt / JSON schema description).
|
||||
- **New tool**: `tools/describe_screen_semantic.py` (or equivalent), wrapping the existing `tools/describe_screen.py` output with an enrichment pass and degrade path, following the same `manager`/`device_id` kwarg pattern as other tools.
|
||||
- **Config**: `pyproject.toml` gains an `anthropic` SDK dependency and a `semantic*` entry in `[tool.setuptools.packages.find].include`; a new settings surface (e.g. env var or config value) to enable/disable enrichment and select the model.
|
||||
- **No change** to `core/models.py`'s `Scene`/`SceneElement`, `vision`/`perception`'s fusion pipeline, or any existing tool signatures — all existing callers of `describe_screen` keep working unmodified.
|
||||
- **Out of scope**: no persistent cross-step semantic state or caching across steps (Milestone 6, World Runtime); no skill synthesis from semantic labels (Milestone 7); no change to `skill-catalog-subscription` or `web-console`.
|
||||
@@ -0,0 +1,45 @@
|
||||
## ADDED Requirements
|
||||
|
||||
### Requirement: Semantic scene enrichment from an existing Scene
|
||||
The system SHALL provide a function that, given an existing `Scene` (as produced by the `scene-perception` capability), produces a `SemanticScene` consisting of a page identity string, a list of plain-language supported intents, and a list of per-widget purpose labels referencing the `Scene`'s element IDs, using exactly one LLM call.
|
||||
|
||||
#### Scenario: Enrichment succeeds for a recognizable screen
|
||||
- **WHEN** `enrich_scene()` is called with a `Scene` describing a recognizable app screen (e.g. a chat screen with a text input and a send button)
|
||||
- **THEN** it returns a `SemanticScene` with a non-empty `page` string, a non-empty `intents` list of plain-language strings, and a `widgets` list where each entry's `element_id` matches an element ID present in the input `Scene`
|
||||
|
||||
#### Scenario: Widget purpose labels reference only known elements
|
||||
- **WHEN** the LLM response includes a widget purpose label whose `element_id` does not match any element ID in the input `Scene`
|
||||
- **THEN** `enrich_scene()` discards that widget entry from the returned `SemanticScene` rather than propagating a dangling element reference
|
||||
|
||||
### Requirement: Enrichment failure degrades to no semantic scene, never blocks the loop
|
||||
The system SHALL treat any enrichment failure (LLM call timeout, connection error, rate limit, malformed or schema-invalid response, or enrichment disabled by configuration) as a non-fatal condition, returning an absence of a semantic scene rather than raising an exception, so that callers always have a defined fallback of using the raw `Scene`.
|
||||
|
||||
#### Scenario: LLM call times out
|
||||
- **WHEN** the enrichment LLM call does not complete within the configured timeout
|
||||
- **THEN** `enrich_scene()` returns `None` and the caller proceeds using the raw `Scene` without the task step being marked as failed
|
||||
|
||||
#### Scenario: LLM response fails schema validation
|
||||
- **WHEN** the enrichment LLM call returns a response that does not conform to the expected `SemanticScene` JSON schema
|
||||
- **THEN** `enrich_scene()` returns `None` instead of raising, and no partially-parsed `SemanticScene` is returned
|
||||
|
||||
#### Scenario: Enrichment disabled by configuration
|
||||
- **WHEN** semantic enrichment is disabled in configuration
|
||||
- **THEN** `enrich_scene()` returns `None` immediately without making an LLM call
|
||||
|
||||
### Requirement: Structured, schema-constrained LLM output
|
||||
The system SHALL request the enrichment LLM call using a schema-constrained structured-output mechanism so that any successful response is guaranteed to be valid JSON matching the `SemanticScene` shape, rather than relying on free-text parsing of an unconstrained model reply.
|
||||
|
||||
#### Scenario: Successful call yields directly parseable output
|
||||
- **WHEN** the enrichment LLM call completes successfully
|
||||
- **THEN** the raw response body is valid JSON matching the declared `SemanticScene` schema without requiring text extraction, regex matching, or a JSON-repair step
|
||||
|
||||
### Requirement: Semantic enrichment is opt-in and does not alter existing tool behavior
|
||||
The system SHALL expose semantic enrichment through a new, separate tool entry point rather than modifying the existing `describe_screen` tool's signature or behavior, so that every existing caller of `describe_screen` continues to receive only a `Scene`, unchanged, unless it explicitly opts into the new semantic-enriched entry point.
|
||||
|
||||
#### Scenario: Existing describe_screen callers are unaffected
|
||||
- **WHEN** an existing caller invokes the `describe_screen` tool as it did before this change
|
||||
- **THEN** it receives the same `Scene` result as before, with no semantic enrichment attempted and no new LLM-related dependency invoked
|
||||
|
||||
#### Scenario: A caller opts into semantic enrichment
|
||||
- **WHEN** a caller invokes the new semantic-enrichment tool entry point for a given device
|
||||
- **THEN** it receives both the underlying `Scene` and, when enrichment succeeds, the corresponding `SemanticScene`; when enrichment fails or is disabled, it receives the `Scene` with an explicit absence of a `SemanticScene` rather than a partial or error result
|
||||
@@ -0,0 +1,42 @@
|
||||
## 1. Package scaffolding
|
||||
|
||||
- [x] 1.1 Create the `semantic/` package (`__init__.py`, `models.py`, `llm_client.py`, `enricher.py`, `prompts.py`)
|
||||
- [x] 1.2 Add the `anthropic` SDK dependency and a `semantic*` entry to `[tool.setuptools.packages.find].include` in `pyproject.toml`
|
||||
- [x] 1.3 Add enrichment configuration: enabled/disabled flag (default disabled), model name, and timeout, sourced from environment/config in a single place `semantic/` reads from
|
||||
- [x] 1.4 Extend the project's smoke test (that imports every package) to import `semantic`
|
||||
|
||||
## 2. SemanticScene data model (capability: semantic-scene)
|
||||
|
||||
- [x] 2.1 Implement `semantic/models.py`: `SemanticWidget` (`element_id: str`, `purpose: str`) and `SemanticScene` (`page: str`, `intents: list[str]`, `widgets: list[SemanticWidget]`) dataclasses with `to_dict()`/`from_dict()` mirroring the style of `core/models.py`
|
||||
- [x] 2.2 Define the JSON schema for `SemanticScene` used to constrain the LLM's structured output, matching the `SemanticScene` dataclass shape exactly
|
||||
- [x] 2.3 Write unit tests for `SemanticScene`/`SemanticWidget` round-tripping through `to_dict()`/`from_dict()`
|
||||
|
||||
## 3. LLM client abstraction (capability: semantic-scene)
|
||||
|
||||
- [x] 3.1 Implement `semantic/llm_client.py`: a narrow client wrapper around the Anthropic SDK exposing a single `enrich(scene_json: dict, *, timeout: float) -> dict` method that issues one structured-output (`output_config.format` / schema-constrained) call and returns the parsed JSON body
|
||||
- [x] 3.2 Configure the enrichment system prompt/schema block with a `cache_control: {"type": "ephemeral"}` breakpoint so repeated per-step calls reuse the cached prefix
|
||||
- [x] 3.3 Map the SDK's typed exceptions (timeout, connection error, rate limit, authentication error, API status error) into a single internal `EnrichmentUnavailable` signal consumed only inside `semantic/` (never re-raised past `enricher.py`)
|
||||
- [x] 3.4 Make the client injectable/mockable (constructor parameter or factory function) so tests can supply a fake client with canned responses instead of calling the network
|
||||
- [x] 3.5 Write unit tests for the client wrapper against a fake transport covering: success, timeout, rate limit, malformed/schema-invalid JSON response
|
||||
|
||||
## 4. Enrichment pass and degrade path (capability: semantic-scene)
|
||||
|
||||
- [x] 4.1 Implement `semantic/prompts.py`: the enrichment system prompt describing the page-identity/intents/widget-purpose task and referencing the input `Scene`'s elements by ID
|
||||
- [x] 4.2 Implement `semantic/enricher.py`: `enrich_scene(scene: Scene, *, client=None) -> SemanticScene | None`, serializing the `Scene` to JSON, calling the LLM client, and parsing the result into a `SemanticScene`
|
||||
- [x] 4.3 Add post-response validation in `enricher.py`: drop any `widgets[]` entry whose `element_id` does not match an element ID present in the input `Scene`
|
||||
- [x] 4.4 Make `enrich_scene()` return `None` (never raise) for every expected failure mode: disabled config, `EnrichmentUnavailable` from the client, or a `SemanticScene` that fails post-response validation entirely
|
||||
- [x] 4.5 Write unit tests for `enrich_scene()` covering: successful enrichment, disabled-by-config short-circuit (no client call made), degrade-to-`None` on client failure, and dangling-`element_id` filtering
|
||||
- [x] 4.6 Write an integration test (skippable without network/API credentials, following the existing `apex-agent-mvp` skippable-integration-test pattern) that calls the real LLM client against a fixture `Scene` and asserts a schema-valid `SemanticScene` is returned
|
||||
|
||||
## 5. Tool integration (capability: semantic-scene)
|
||||
|
||||
- [x] 5.1 Implement `tools/describe_screen_semantic.py`: calls the existing `tools/describe_screen.py` for a `Scene`, then `enrich_scene()`, returning `{"scene": Scene, "semantic_scene": SemanticScene | None}`, following the same `device_id`/`manager`/`ocr_engine` kwarg pattern as `describe_screen`
|
||||
- [x] 5.2 Register the new tool under its own key (e.g. `"describe_screen_semantic"`) in `runtime/executor.py`'s `default_tool_registry()`, additive alongside the existing `"describe_screen"` entry
|
||||
- [x] 5.3 Write a unit test asserting `tools/describe_screen.py`'s existing behavior and return type are unchanged after this change (no accidental coupling to `semantic/`)
|
||||
- [x] 5.4 Write a unit test for `describe_screen_semantic` covering both the enrichment-succeeds and enrichment-degrades-to-`None` cases, using a fake/mock LLM client
|
||||
|
||||
## 6. End-to-end validation
|
||||
|
||||
- [x] 6.1 Write an end-to-end test simulating an Observe step that calls `describe_screen_semantic` against a mocked `Driver`/`Scene` and a mocked LLM client, asserting the task loop completes normally whether enrichment succeeds or is forced to fail
|
||||
- [x] 6.2 Confirm enrichment stays disabled by default after applying this change (no existing task's behavior, latency, or cost changes unless a caller explicitly enables it and opts into the new tool)
|
||||
- [x] 6.3 Run the full test suite (`pytest`) and confirm no existing test in `tests/` needed a behavior change, only additive new tests
|
||||
@@ -0,0 +1,2 @@
|
||||
schema: spec-driven
|
||||
created: 2026-07-06
|
||||
@@ -0,0 +1,65 @@
|
||||
## Context
|
||||
|
||||
This builds on the `apex-agent-mvp` change (currently proposed, not yet applied), which establishes the Device/Driver/Perception/Agent-Runtime/MCP layers for Apex Agent. That change already defines an MCP Tool Server (`api/mcp.py`) exposing device capabilities; this change adds a parallel `skills/` package and new MCP tools onto the same server, without touching the device/perception/runtime code.
|
||||
|
||||
The key external constraint: skill **authoring, versioning, and entitlement** live in a separate, already-planned **Subscription Platform** (a distinct information-management system, out of scope here). Apex Agent is a **consumer** of that platform's catalog, not the source of truth for skill content. We only control: how Apex Agent stores what it has synced, how it exposes that to the LLM via MCP, and the contract it expects the Subscription Platform to satisfy.
|
||||
|
||||
Two kinds of Skill must be supported, per product direction:
|
||||
- **Knowledge skill**: free-form structured instructional content (markdown/text + metadata) the LLM reads to decide how to act — e.g. "tips for searching effectively on Xiaohongshu." The LLM still plans/executes via the existing `agent-runtime`/`tools` layer.
|
||||
- **Flow-template skill**: a predefined, parameterized sequence of capability calls (tap/swipe/input steps with placeholders, e.g. `{query}`), which the LLM mostly fills in and triggers rather than re-planning from scratch.
|
||||
|
||||
## Goals / Non-Goals
|
||||
|
||||
**Goals:**
|
||||
- Define a single `Skill` data model general enough to represent both knowledge and flow-template skills with shared metadata (id, name, description, version, tags, subscription/source id).
|
||||
- Expose skills to the LLM through MCP tools that are indistinguishable in style from the existing device tools (semantic, driver/platform-agnostic).
|
||||
- Define a clear, minimal contract for what Apex Agent needs from the Subscription Platform: a way to fetch "skills I'm entitled to" and a way to learn about changes (poll or push), without dictating that platform's internal design.
|
||||
- Enforce subscription-based visibility so `list_skills`/`search_skills`/`get_skill` only ever surface skills the current deployment (tenant/device/agent) is actually subscribed to.
|
||||
- Let a flow-template skill's steps execute through the *existing* `tools/` functions from `apex-agent-mvp` (no new execution primitives) — a flow template is just a data-driven script over the same capability calls the Executor already uses.
|
||||
|
||||
**Non-Goals:**
|
||||
- Designing or implementing the Subscription Platform itself (its UI, billing, authoring workflow, or storage).
|
||||
- Automatic skill generation/authoring by the LLM.
|
||||
- A full permission/RBAC system — subscription visibility is scoped to "is this skill in my entitled set," not fine-grained per-user roles.
|
||||
- Real-time push infrastructure (websockets/queues) as a hard requirement — push is supported as an *optional* transport; polling sync must work standalone.
|
||||
- Executing flow-template skills with a new engine — they are interpreted by the existing `runtime/executor.py`, not a separate workflow engine.
|
||||
|
||||
## Decisions
|
||||
|
||||
### D1: `Skill` is a single model with a `kind` discriminator (`knowledge` | `flow_template`), not two unrelated types
|
||||
Both kinds share `SkillMetadata` (id, name, description, version, tags, source/subscription id, updated_at). A `KnowledgeSkill` carries `content` (markdown/text). A `FlowTemplateSkill` carries `steps` (ordered list of `{tool, args_template}`, where `args_template` values may contain `{param}` placeholders) and a `parameters` schema (name, type, required, description) used to validate/prompt for inputs before execution.
|
||||
- **Alternative considered**: Two entirely separate catalogs/tables/tool sets for knowledge vs. flow skills. Rejected — `list_skills`/`search_skills` would need to be duplicated, and most metadata (id/name/description/version/tags) is identical; a discriminated union keeps one catalog and one set of list/search tools while `get_skill` returns kind-specific content.
|
||||
|
||||
### D2: Skill Catalog is a local, read-mostly cache; Subscription Platform is always the source of truth
|
||||
`skills/catalog.py` stores the last-synced copy of entitled skills locally (for offline availability and low-latency MCP responses) but never treats local edits as authoritative — there is no "create/update skill" MCP tool or API in this change. All content changes flow one direction: Subscription Platform → sync client → local catalog.
|
||||
- **Alternative considered**: Let Apex Agent locally override/edit synced skills. Rejected — breaks the "another system manages skills" requirement from the proposal and creates drift/merge-conflict problems with no clear owner.
|
||||
|
||||
### D3: Sync client supports pull (poll) as the required baseline, push (webhook) as an optional accelerator
|
||||
`skills/sync_client.py` defines a `fetch_entitled_skills(subscription_id, since_version)` contract against the Subscription Platform's assumed API, called on a configurable interval (baseline). If the platform can also deliver change notifications (webhook/callback), an optional receiver triggers an immediate out-of-cycle fetch instead of waiting for the next poll — but correctness never depends on push arriving.
|
||||
- **Alternative considered**: Push-only (webhook-driven) sync. Rejected — makes Apex Agent's catalog freshness dependent on network reachability of an inbound webhook, which is operationally harder (firewalls/NAT) than Apex Agent making outbound poll requests; poll-as-baseline is strictly more deployable.
|
||||
|
||||
### D4: Subscription-based visibility is enforced at the catalog query layer, not just at sync time
|
||||
Even though sync only ever pulls "entitled" skills, `skills/catalog.py`'s query functions (used by both MCP tools and any future internal caller) re-check that a skill's subscription id is still in the caller's active subscription set at query time, not just at last-sync time. This guards against a stale local cache still containing a skill whose entitlement was revoked but not yet re-synced.
|
||||
- **Alternative considered**: Trust the local cache fully between syncs (no re-check at query time). Rejected — entitlement revocation should not have to wait for the next poll interval to stop being visible if the revocation is already known locally (e.g. via a push notification that only carries "removed" without full content).
|
||||
|
||||
### D5: Flow-template skills execute via existing `tools/`/`runtime/executor.py`, with a resolution step in between
|
||||
`skills/mcp_tools.py` exposes `get_skill` (returns metadata + content/template) and a parameter-resolution helper; it does **not** expose a `run_skill` tool that silently executes on the LLM's behalf. The LLM fetches the flow template, fills placeholders itself (or asks the user for missing required parameters), and then issues the already-existing device-capability tool calls (`tap`, `input_text`, etc.) from `apex-agent-mvp` in the order/values the template specifies.
|
||||
- **Alternative considered**: Add a single `run_skill_flow(skill_id, params)` MCP tool that executes the whole sequence server-side. Rejected for this change — it would bypass the Agent Runtime's Observe→Think→Act loop and Executor retry/wait logic from `apex-agent-mvp`, silently skipping Scene verification between steps. Keeping execution inside the existing loop (LLM issues each tool call itself, informed by the template) preserves the "AI decides based on what it currently observes" principle the whole platform is built on. A batched execution helper can be revisited later as a design decision in its own change if needed.
|
||||
|
||||
## Risks / Trade-offs
|
||||
|
||||
- **[Risk]** The Subscription Platform's actual API shape is unknown/assumed (`fetch_entitled_skills`, optional webhook) → **Mitigation**: keep `skills/sync_client.py` behind a small internal interface (similar to `Driver` in `apex-agent-mvp`) so the concrete HTTP client can be adjusted once the real Subscription Platform API is finalized, without touching `catalog.py` or `mcp_tools.py`.
|
||||
- **[Risk]** Stale cache could serve outdated skill content between polls → **Mitigation**: D4's query-time entitlement re-check plus a configurable poll interval; document the staleness window explicitly rather than promising real-time freshness.
|
||||
- **[Risk]** Flow-template skills reference tools/parameters that drift from the actual `tools/` function signatures in `apex-agent-mvp` (e.g. a template calls a tool that was renamed) → **Mitigation**: validate a flow template's `tool` names against the currently registered MCP/tool set at sync time (or at least at `get_skill` time) and surface a clear "skill unavailable/invalid" error rather than letting a bad call reach the device.
|
||||
- **[Trade-off]** No local skill authoring/editing keeps this change simple and avoids ownership ambiguity, but means Apex Agent is fully dependent on the Subscription Platform being reachable at least once to have any skills at all — acceptable since the platform is a required dependency by design, not an optional enhancement.
|
||||
- **[Trade-off]** Not providing a server-side `run_skill_flow` execution tool keeps flow templates consistent with the Observe-Think-Act loop, at the cost of the LLM needing a few more tool-call round trips per flow skill than a single batched call would take — acceptable given the platform's core principle of always re-observing between actions.
|
||||
|
||||
## Migration Plan
|
||||
|
||||
Additive only: new `skills/` package, new MCP tools, and new local storage tables/files. No existing `apex-agent-mvp` code paths are modified (only composed with, per D-decisions above). If `apex-agent-mvp` has already been applied, this change's MCP tools register onto its existing MCP server instance; if not yet applied, `skills/mcp_tools.py` can stand up its own MCP server instance for independent testing and be merged once `apex-agent-mvp` lands. Rollback is simply removing the `skills/` package and its MCP tool registrations; no data migration or schema changes to existing tables are required.
|
||||
|
||||
## Open Questions
|
||||
|
||||
- Exact Subscription Platform API contract (auth mechanism, request/response shapes, whether it supports `since_version` incremental sync or only full-catalog fetch) — to be confirmed with that platform's team/spec before `skills/sync_client.py` is finalized; this design assumes an interface shape that can absorb either.
|
||||
- Whether flow-template `args_template` placeholders need a richer expression language (e.g. simple conditionals) or plain `{param}` substitution is sufficient for the MVP — left open, default to plain substitution and revisit if a real skill needs more.
|
||||
- Where the local Skill Catalog lives relative to `apex-agent-mvp`'s existing SQLite task-metadata DB (same DB file, new tables, vs. a separate DB/file) — deferred to implementation time in `tasks.md`, doesn't affect the capability contracts defined here.
|
||||
@@ -0,0 +1,28 @@
|
||||
## Why
|
||||
|
||||
Apex Agent's MCP tool server (see change `apex-agent-mvp`) gives the LLM raw device capabilities (tap/swipe/screenshot/...), but it has no notion of reusable, task-specific know-how — e.g. "how to search on Xiaohongshu," or "the tap/swipe/input sequence to place an order on Taobao." Today that knowledge would have to live entirely inside the LLM's own reasoning or be re-derived from scratch on every task. We need a **Skill** concept the AI can discover and pull on demand via MCP, and — since skill content will be authored, versioned, and entitled to specific tenants/devices by a separate, already-planned **Subscription Platform** (统一订阅平台，另一套信息管理系统) — Apex Agent needs a defined contract for consuming that platform's catalog rather than owning skill authoring itself.
|
||||
|
||||
## What Changes
|
||||
|
||||
- Introduce a **Skill Catalog** capability: a local data model and store for Skills, where a Skill is either a **knowledge skill** (structured instructional/markdown content the AI reads to decide how to act, analogous to Claude Skills) or a **flow-template skill** (a parameterized, predefined sequence of capability calls — e.g. tap/swipe/input steps with placeholders — that the AI mostly fills in parameters for and triggers, rather than re-planning from scratch). Both kinds share common metadata (id, name, description, version, tags) so they can be listed/searched uniformly.
|
||||
- Introduce **Skill MCP tools** (`list_skills`, `search_skills`, `get_skill`, `run_skill_flow`-input-resolution helper) exposed through the same MCP surface established in `apex-agent-mvp`'s `mcp-tool-server`, so an LLM can discover which skills are available and fetch their content/flow template without knowing anything about the Subscription Platform underneath.
|
||||
- Introduce a **Skill Subscription Sync** capability: a client-side contract for talking to the external Subscription Platform, covering (a) pulling/receiving the catalog of skills a given deployment is entitled to, (b) keeping the local Skill Catalog in sync (create/update/remove on change), and (c) enforcing subscription-based visibility so only skills the current tenant/device/agent is subscribed to are listed or fetchable via the MCP tools.
|
||||
- Explicitly out of scope for this change: designing or building the Subscription Platform itself (authoring UI, billing, skill publishing workflow) — it is treated as an existing/external system; this change only defines the integration contract (API shape, sync semantics, auth) Apex Agent needs from it. Also out of scope: automatic skill-authoring/generation by the LLM, and a skill marketplace UI.
|
||||
|
||||
## Capabilities
|
||||
|
||||
### New Capabilities
|
||||
- `skill-catalog`: Local Skill data model (knowledge-doc and flow-template variants), storage, and search/query functions used by both the MCP tools and the sync client.
|
||||
- `skill-mcp-tools`: MCP-facing tool surface for listing, searching, and fetching Skill content/flow templates, plus resolving flow-template parameters, without exposing any Subscription Platform or storage detail to the LLM.
|
||||
- `skill-subscription-sync`: Contract and client implementation for syncing the Skill Catalog from the external Subscription Platform (pull and/or push), and for enforcing subscription-based visibility/permission scoping per tenant/device/agent.
|
||||
|
||||
### Modified Capabilities
|
||||
(none — `mcp-tool-server` from the pending `apex-agent-mvp` change is composed with, not modified: this change adds new tools to the same MCP server process rather than changing that capability's existing requirements. If `apex-agent-mvp` has not yet been applied when this change is implemented, the Skill MCP tools should still be registrable on their own MCP server instance and merged in later.)
|
||||
|
||||
## Impact
|
||||
|
||||
- **New code**: `skills/` package — `skills/models.py` (Skill, KnowledgeSkill, FlowTemplateSkill, SkillMetadata), `skills/catalog.py` (local store + search/query), `skills/sync_client.py` (Subscription Platform client: pull/push, auth, visibility filtering), `skills/mcp_tools.py` (registers `list_skills`/`search_skills`/`get_skill`/flow-param-resolution as MCP tools).
|
||||
- **Dependencies**: depends on the `apex-agent-mvp` change for the MCP server process and `tools/`/`runtime/` capability layer that flow-template skills ultimately drive (a flow-template skill's steps still execute through existing `tools/` functions); does not depend on any new external dependency beyond an HTTP client for the sync API.
|
||||
- **External systems**: introduces a new external dependency — the Subscription Platform's API (assumed to expose an endpoint to fetch entitled skills and, optionally, a webhook/push channel for change notifications). Exact base URL/auth mechanism is a deployment-time configuration, not a code dependency.
|
||||
- **Storage**: adds a local Skill Catalog store (SQLite table(s) alongside the existing task-metadata DB from `apex-agent-mvp`, or an embedded file-based store — to be decided in design) plus a small sync-state table (last-synced version/timestamp per subscription).
|
||||
- **Follow-on work explicitly deferred**: Subscription Platform's own design/build, skill-authoring workflows, billing/entitlement logic beyond "is this skill visible to me," and any LLM-driven automatic skill generation.
|
||||
@@ -0,0 +1,53 @@
|
||||
## ADDED Requirements
|
||||
|
||||
### Requirement: Unified Skill data model
|
||||
The system SHALL represent every Skill with shared metadata (id, name, description, version, tags, source/subscription id, updated_at) and a `kind` discriminator of either `knowledge` or `flow_template`, so both kinds can be listed and searched through one catalog.
|
||||
|
||||
#### Scenario: Knowledge skill has content
|
||||
- **WHEN** a Skill with `kind = knowledge` is stored
|
||||
- **THEN** it includes a `content` field (structured instructional text/markdown) in addition to the shared metadata
|
||||
|
||||
#### Scenario: Flow-template skill has steps and parameters
|
||||
- **WHEN** a Skill with `kind = flow_template` is stored
|
||||
- **THEN** it includes an ordered `steps` list (each referencing a tool name and an args template that may contain `{param}` placeholders) and a `parameters` schema (name, type, required, description) in addition to the shared metadata
|
||||
|
||||
### Requirement: Local skill storage
|
||||
The system SHALL persist the synced Skill Catalog locally so that skills can be listed and fetched without a live round-trip to the Subscription Platform for every query.
|
||||
|
||||
#### Scenario: Skill readable after sync
|
||||
- **WHEN** a skill has been synced from the Subscription Platform into the local catalog
|
||||
- **THEN** subsequent list/search/get operations return that skill without requiring a new network call to the Subscription Platform
|
||||
|
||||
#### Scenario: Local catalog is not independently authored
|
||||
- **WHEN** a caller attempts to create or edit a skill directly in the local catalog (outside of a sync operation)
|
||||
- **THEN** the system SHALL reject or ignore the write, since the Subscription Platform is the sole source of truth for skill content
|
||||
|
||||
### Requirement: Skill search and query
|
||||
The system SHALL provide functions to list all currently visible skills, search skills by name/tag/description text, and fetch a single skill by id.
|
||||
|
||||
#### Scenario: List returns only visible skills
|
||||
- **WHEN** the catalog is queried for the list of skills visible to the current caller
|
||||
- **THEN** it returns only skills whose subscription is currently active for that caller, sorted in a stable order (e.g. by name)
|
||||
|
||||
#### Scenario: Search matches on name, tags, or description
|
||||
- **WHEN** a search query string matches a skill's name, a tag, or its description
|
||||
- **THEN** that skill is included in the search results
|
||||
|
||||
#### Scenario: Get by id returns full content
|
||||
- **WHEN** a caller fetches a skill by its id
|
||||
- **THEN** the system returns the full skill record, including `content` for knowledge skills or `steps`/`parameters` for flow-template skills
|
||||
|
||||
#### Scenario: Get by id for unknown or invisible skill
|
||||
- **WHEN** a caller fetches a skill id that does not exist, or exists but is not currently visible to them (subscription inactive)
|
||||
- **THEN** the system returns a clear "not found" result rather than leaking the skill's existence or content
|
||||
|
||||
### Requirement: Flow-template tool reference validation
|
||||
The system SHALL validate that a flow-template skill's referenced tool names correspond to currently registered device-capability tools, and SHALL mark a flow-template skill as invalid/unavailable rather than allowing it to be fetched successfully if a referenced tool does not exist.
|
||||
|
||||
#### Scenario: Valid flow template
|
||||
- **WHEN** every step in a flow-template skill references a tool name that is currently registered
|
||||
- **THEN** the skill is fetchable and returned normally
|
||||
|
||||
#### Scenario: Flow template references an unknown tool
|
||||
- **WHEN** a flow-template skill's steps reference a tool name that is not currently registered (e.g. renamed or removed)
|
||||
- **THEN** fetching that skill returns a clear "skill unavailable/invalid" result instead of a step list containing a dangling tool reference
|
||||
@@ -0,0 +1,41 @@
|
||||
## ADDED Requirements
|
||||
|
||||
### Requirement: MCP tools for skill discovery and retrieval
|
||||
The system SHALL expose `list_skills`, `search_skills`, and `get_skill` as MCP tools on the same MCP server surface used for device capabilities, so an LLM client can discover and fetch Skill content using the same tool-calling mechanism it already uses for device actions.
|
||||
|
||||
#### Scenario: LLM lists available skills
|
||||
- **WHEN** an MCP client calls `list_skills`
|
||||
- **THEN** it receives the set of skills currently visible to it (per subscription visibility), each with id, name, description, kind, and tags, without any Subscription Platform-specific fields or identifiers
|
||||
|
||||
#### Scenario: LLM searches for a relevant skill
|
||||
- **WHEN** an MCP client calls `search_skills` with a query string
|
||||
- **THEN** it receives matching skills ranked/filtered by relevance to the query, using the same visibility rules as `list_skills`
|
||||
|
||||
#### Scenario: LLM fetches a specific skill's content
|
||||
- **WHEN** an MCP client calls `get_skill` with a skill id
|
||||
- **THEN** it receives the full skill content appropriate to its kind: `content` text for a `knowledge` skill, or `steps`/`parameters` for a `flow_template` skill
|
||||
|
||||
### Requirement: Flow-template parameter resolution helper
|
||||
The system SHALL provide an MCP-facing helper that, given a flow-template skill id and a set of proposed parameter values, validates the values against the skill's declared `parameters` schema and returns the fully resolved step sequence (placeholders substituted) for the LLM to then execute step-by-step via the existing device-capability tools.
|
||||
|
||||
#### Scenario: Valid parameters resolve the template
|
||||
- **WHEN** the LLM provides values for all required parameters of a flow-template skill
|
||||
- **THEN** the system returns the ordered steps with all `{param}` placeholders substituted by the provided values
|
||||
|
||||
#### Scenario: Missing required parameter
|
||||
- **WHEN** the LLM omits a required parameter when resolving a flow-template skill
|
||||
- **THEN** the system returns a clear validation error identifying the missing parameter(s) instead of returning a partially-substituted step list
|
||||
|
||||
### Requirement: No server-side flow execution tool
|
||||
The system SHALL NOT expose an MCP tool that executes a flow-template skill's full step sequence server-side on the LLM's behalf; the LLM SHALL issue each resulting device-capability tool call itself so every step remains subject to the existing Agent Runtime's Observe-Think-Act loop and Executor retry/wait handling.
|
||||
|
||||
#### Scenario: No batch-execute tool is available
|
||||
- **WHEN** the MCP tool list is inspected
|
||||
- **THEN** it contains skill discovery/retrieval/resolution tools but no tool that both resolves and executes a flow-template skill's steps in a single call
|
||||
|
||||
### Requirement: Skill MCP errors are semantic
|
||||
The system SHALL translate catalog-level errors (skill not found, skill not visible/entitled, invalid/unavailable flow template) into clear, semantic MCP tool error responses, consistent in style with the device-capability tool error handling.
|
||||
|
||||
#### Scenario: Requesting a non-visible skill
|
||||
- **WHEN** `get_skill` is called with a skill id that exists but is not visible to the caller's current subscriptions
|
||||
- **THEN** the tool returns a semantic "not found" error rather than a raw database or internal exception
|
||||
@@ -0,0 +1,61 @@
|
||||
## ADDED Requirements
|
||||
|
||||
### Requirement: Pull-based sync from the Subscription Platform
|
||||
The system SHALL periodically fetch, from the external Subscription Platform, the set of skills the current deployment (tenant/device/agent) is entitled to, and SHALL apply creates/updates/removals to the local Skill Catalog to match that entitled set.
|
||||
|
||||
#### Scenario: New skill appears after sync
|
||||
- **WHEN** the Subscription Platform reports a new entitled skill that does not yet exist locally
|
||||
- **THEN** the next sync cycle creates it in the local Skill Catalog
|
||||
|
||||
#### Scenario: Updated skill content is refreshed
|
||||
- **WHEN** the Subscription Platform reports a newer version of a skill already present locally
|
||||
- **THEN** the next sync cycle updates the local copy to the newer version
|
||||
|
||||
#### Scenario: Revoked entitlement removes local visibility
|
||||
- **WHEN** the Subscription Platform no longer includes a previously-entitled skill in the current entitled set
|
||||
- **THEN** the next sync cycle removes or marks that skill as no longer visible in the local Skill Catalog
|
||||
|
||||
#### Scenario: Sync interval is configurable
|
||||
- **WHEN** the deployment is configured with a sync poll interval
|
||||
- **THEN** the system performs pull sync on approximately that interval without requiring a restart to take effect on the next cycle
|
||||
|
||||
### Requirement: Optional push-triggered sync
|
||||
The system MAY support receiving a change notification (e.g. webhook) from the Subscription Platform, and WHEN it does, SHALL trigger an immediate out-of-cycle pull sync rather than waiting for the next poll interval; correctness of the catalog SHALL NOT depend on push notifications being delivered.
|
||||
|
||||
#### Scenario: Push notification triggers immediate sync
|
||||
- **WHEN** a change notification is received from the Subscription Platform
|
||||
- **THEN** the system performs a pull sync immediately, independent of the regular poll schedule
|
||||
|
||||
#### Scenario: No push configured still stays eventually consistent
|
||||
- **WHEN** push notifications are not configured or not received
|
||||
- **THEN** the local Skill Catalog still reflects the Subscription Platform's entitled set within one poll interval via the baseline pull sync
|
||||
|
||||
### Requirement: Subscription-based visibility enforcement
|
||||
The system SHALL enforce, at both sync time and query time, that only skills belonging to the caller's currently active subscription(s) are stored as visible or returned by catalog queries.
|
||||
|
||||
#### Scenario: Query-time re-check catches stale entitlement
|
||||
- **WHEN** a skill's entitlement has been revoked but the local cache has not yet completed its next sync cycle
|
||||
- **AND** a catalog query for that skill is made using already-known revocation information
|
||||
- **THEN** the system does not return that skill as visible, even though its record may still exist locally pending cleanup
|
||||
|
||||
#### Scenario: Multiple subscriptions compose visibility
|
||||
- **WHEN** a deployment holds more than one active subscription, each entitling a different set of skills
|
||||
- **THEN** catalog queries return the union of skills entitled across all of that deployment's active subscriptions
|
||||
|
||||
### Requirement: Sync client contract is transport-replaceable
|
||||
The system SHALL define the Subscription Platform integration (fetch entitled skills, optional change notification receipt) behind a single internal interface, so the concrete HTTP client/auth mechanism can be adjusted to match the real Subscription Platform API without changes to the Skill Catalog or MCP tool layers.
|
||||
|
||||
#### Scenario: Sync client swap does not affect catalog/tools
|
||||
- **WHEN** the concrete Subscription Platform client implementation is replaced (e.g. different auth scheme or request/response shape)
|
||||
- **THEN** `skill-catalog` and `skill-mcp-tools` behavior and their own specs remain unaffected, as long as the new client still satisfies the sync interface
|
||||
|
||||
### Requirement: Sync failure handling
|
||||
The system SHALL treat a failed sync attempt (network error, auth failure, malformed response) as non-fatal to already-cached skills: the local catalog SHALL continue serving its last-known-good state, and the failure SHALL be recorded/observable rather than silently discarded.
|
||||
|
||||
#### Scenario: Sync failure preserves last-known catalog
|
||||
- **WHEN** a sync attempt fails due to a network or platform error
|
||||
- **THEN** the local Skill Catalog is left unchanged (not cleared or partially corrupted) and remains queryable using its last successfully synced state
|
||||
|
||||
#### Scenario: Sync failure is observable
|
||||
- **WHEN** a sync attempt fails
|
||||
- **THEN** the system records the failure (e.g. last-error timestamp/reason) so it can be surfaced to operators rather than failing silently forever
|
||||
@@ -0,0 +1,39 @@
|
||||
## 1. Package scaffolding & data model
|
||||
|
||||
- [ ] 1.1 Create `skills/` package (`__init__.py`) alongside existing `core/`, `tools/`, `vision/`, `runtime/`, `api/`, `storage/`
|
||||
- [ ] 1.2 Add `skills/models.py`: `SkillMetadata` (id, name, description, version, tags, source/subscription id, updated_at), `KnowledgeSkill` (content), `FlowTemplateSkill` (steps, parameters), and a `kind` discriminator uniting them
|
||||
- [ ] 1.3 Add dependencies if needed (HTTP client for sync, no new heavy deps expected) to `pyproject.toml`
|
||||
|
||||
## 2. Skill Catalog storage & query (capability: skill-catalog)
|
||||
|
||||
- [ ] 2.1 Implement `skills/catalog.py` local store (decide SQLite table(s) reusing the existing DB vs. separate file, per design's open question) with create/update/remove operations used only by the sync path
|
||||
- [ ] 2.2 Implement `list_skills(caller_context)` returning only skills visible under the caller's active subscriptions, stable-sorted
|
||||
- [ ] 2.3 Implement `search_skills(query, caller_context)` matching name/tags/description
|
||||
- [ ] 2.4 Implement `get_skill(skill_id, caller_context)` returning full content, or a clear "not found" for unknown/invisible ids
|
||||
- [ ] 2.5 Reject/ignore any direct create-or-edit call to the catalog that didn't come from the sync path (enforce "Subscription Platform is sole source of truth")
|
||||
- [ ] 2.6 Implement flow-template tool-reference validation: check each step's `tool` name against the currently registered device-capability tools; mark skill invalid/unavailable if any reference is dangling
|
||||
- [ ] 2.7 Write unit tests: list/search/get visibility filtering, not-found cases, and flow-template validation (valid and dangling-reference cases)
|
||||
|
||||
## 3. Subscription sync client (capability: skill-subscription-sync)
|
||||
|
||||
- [ ] 3.1 Define the internal sync interface (e.g. `SubscriptionClient.fetch_entitled_skills(subscription_id, since_version)` supporting full or incremental fetch) in `skills/sync_client.py`
|
||||
- [ ] 3.2 Implement a concrete HTTP-based `SubscriptionClient` against the assumed Subscription Platform API (configurable base URL/auth), isolated behind the interface from 3.1
|
||||
- [ ] 3.3 Implement the poll loop: fetch on a configurable interval, diff against local catalog, and apply create/update/remove to `skills/catalog.py`
|
||||
- [ ] 3.4 Implement optional push-triggered sync: a receiver (e.g. a small webhook endpoint) that, on notification, triggers an immediate out-of-cycle pull rather than being required for correctness
|
||||
- [ ] 3.5 Implement subscription-based visibility enforcement at query time (re-check active subscription set, not just last-sync membership) so revoked entitlements stop being visible without waiting for the next full sync
|
||||
- [ ] 3.6 Implement sync failure handling: on network/auth/malformed-response errors, leave the local catalog unchanged and record a last-error timestamp/reason for observability
|
||||
- [ ] 3.7 Write unit tests: create/update/remove sync scenarios, push-triggered immediate sync, query-time revocation re-check, and sync-failure-preserves-cache behavior (using a fake `SubscriptionClient`)
|
||||
|
||||
## 4. Skill MCP tools (capability: skill-mcp-tools)
|
||||
|
||||
- [ ] 4.1 Implement `skills/mcp_tools.py` registering `list_skills`, `search_skills`, `get_skill` as MCP tools on the existing MCP server surface from `apex-agent-mvp` (or a standalone MCP server instance if that change is not yet applied)
|
||||
- [ ] 4.2 Implement the flow-template parameter-resolution MCP tool: validate provided values against a skill's `parameters` schema and return the fully substituted step sequence, or a clear validation error listing missing/invalid parameters
|
||||
- [ ] 4.3 Add semantic error translation for skill MCP tools (not-found, not-visible/entitled, invalid/unavailable flow template) consistent in style with the device-capability tool error handling
|
||||
- [ ] 4.4 Confirm no MCP tool exists that both resolves and executes a flow-template skill's steps server-side (execution stays with the LLM issuing existing device-capability tool calls one at a time)
|
||||
- [ ] 4.5 Write tests: `list_skills`/`search_skills`/`get_skill` via MCP against a seeded catalog (mocked sync), parameter-resolution success/failure cases, and an explicit check that no batch-execute tool is registered
|
||||
|
||||
## 5. End-to-end validation
|
||||
|
||||
- [ ] 5.1 Seed a local catalog via a fake `SubscriptionClient` with one knowledge skill and one flow-template skill, and verify both are listable/searchable/fetchable via MCP tools
|
||||
- [ ] 5.2 Simulate an entitlement revocation (remove a skill from the fake client's entitled set) and confirm it disappears from `list_skills`/`get_skill` after both a sync cycle and, separately, via the query-time re-check before the next sync completes
|
||||
- [ ] 5.3 Resolve a flow-template skill's parameters via MCP, then manually drive the resulting step sequence through the existing `apex-agent-mvp` device-capability tools against a mocked device, confirming each step still goes through the normal Observe-Think-Act loop
|
||||
@@ -0,0 +1,2 @@
|
||||
schema: spec-driven
|
||||
created: 2026-07-06
|
||||
@@ -0,0 +1,87 @@
|
||||
## Context
|
||||
|
||||
Two capabilities already exist as pending, unapplied changes that this milestone sits between: `task-memory` (`apex-agent-mvp`) persists a per-task `Timeline` — `storage/timeline.py`'s `Timeline.append()` writes one `TimelineRecord{index, scene, prompt, tool_call, result, timestamp, screenshot_path}` per executed step, retrievable in order via `Timeline.read(task_id)` — and `skill-catalog-subscription` defines a `Skill` data model (`kind = knowledge | flow_template`, shared metadata `id/name/description/version/tags`, plus for `flow_template` an ordered `steps` list of tool-name + args-template and a `parameters` schema) synced *from* an external Subscription Platform into a local read-mostly catalog (`skill-catalog-subscription`'s own spec explicitly rejects local writes outside of sync: "a caller attempts to create or edit a skill directly in the local catalog... reject or ignore"). Neither change gives the agent a way to go from "I just executed a successful sequence of tool calls" to "that sequence is now a reusable skill" — `task-memory`'s `Timeline` is write-once/read-only historical record, and `skill-catalog`'s local store is deliberately sync-only, not authorable.
|
||||
|
||||
This creates a real tension this design must resolve explicitly: `skill-catalog-subscription`'s spec says the local catalog rejects non-sync writes, but this change's whole point is to write locally-synthesized skills into a catalog. The resolution (D6 below) is that locally-authored skills live in a **separate local store** (`skills_learning/`'s own catalog table) that composes with, rather than writes into, the `skill-catalog-subscription` store — the two are presented to a Planner/MCP-tool layer as one logical namespace in prose, but this change does not touch `skill-catalog-subscription`'s schema, sync semantics, or its "sync-only" write rule.
|
||||
|
||||
`runtime/task.py`'s `TaskRunner.run()` (code-complete, unapplied) is the Observe→Think→Act→Observe loop; it already knows the task's `goal`, calls `Timeline.append()` once per step (via whatever wiring `apex-agent-mvp` gives it — the loop has access to everything `Timeline` needs), and terminates with a final status. This change adds exactly one new integration point at the very end of that loop: a hook fired once, only on success.
|
||||
|
||||
Two stakeholders: `runtime/task.py`'s `TaskRunner` (must gain a well-defined, optional, no-op-by-default hook), and a future LLM-driven Planner (not built in this change, matching `world-model-runtime`'s and `semantic-scene-runtime`'s precedent of "expose the input, defer the smart consumer") that will eventually call `retrieve_candidate_skills()` before planning from scratch.
|
||||
|
||||
## Goals / Non-Goals
|
||||
|
||||
**Goals:**
|
||||
- Synthesize a parameterized `FlowTemplateSkill` from a completed task's `Timeline` + goal, automatically, with no manual authoring step.
|
||||
- Detect when a later execution that would synthesize into "the same" skill (same name/goal-family) has actually diverged in its step sequence, and version rather than silently overwrite.
|
||||
- Make locally-learned skills discoverable by semantic similarity to a new goal, not just exact name/id lookup, so a Planner can find "something like this" even when the new goal's wording differs from the one that produced the stored skill.
|
||||
- Compose cleanly with `skill-catalog-subscription`'s existing model and its "sync-only local writes" rule — do not require a delta against that change's spec.
|
||||
- Keep the synthesis/versioning/embedding path entirely off the hot Observe→Act loop — it runs once, after a task finishes, never mid-task.
|
||||
- Default to disabled, so applying this change does not silently add CPU/LLM cost to every successful task run.
|
||||
|
||||
**Non-Goals:**
|
||||
- No change to `skill-catalog-subscription`'s sync contract, MCP tool surface (`list_skills`/`search_skills`/`get_skill`), or subscription/visibility/entitlement model.
|
||||
- No Planner wiring that actually *consumes* `retrieve_candidate_skills()` results to skip planning — this change only makes retrieval available, matching `world-model-runtime`'s precedent of exposing a read-only input without teaching a Planner to use it.
|
||||
- No skill *execution* engine (running a `FlowTemplateSkill`'s steps against `tools/` with resolved parameters) — `skill-catalog-subscription`'s proposal already scoped a "run_skill_flow parameter-resolution helper" as its own concern; this change produces skills, it does not add a new runner for them.
|
||||
- No cross-device/cross-tenant skill sharing beyond whatever `skill-catalog`'s existing storage model already implies.
|
||||
- No UI (`web-console`'s domain, not touched here).
|
||||
- No multi-provider embedding abstraction — a single concrete embedding client is enough for this milestone, matching `semantic-scene-runtime`'s D-non-goal of not building a multi-provider LLM abstraction prematurely (YAGNI).
|
||||
- Failed or partially-completed tasks are never synthesized into skills — only `status = succeeded` timelines are eligible input.
|
||||
|
||||
## Decisions
|
||||
|
||||
### D1: New `skills_learning/` package, sibling to `semantic/` and `world/`, not folded into `runtime/` or `skills/`
|
||||
`skills_learning/` (`synthesis.py`, `versioning.py`, `embeddings.py`, `retrieval.py`, `config.py`) is its own package rather than extending `skill-catalog-subscription`'s planned `skills/` package (`skills/models.py`, `skills/catalog.py`, `skills/sync_client.py`, `skills/mcp_tools.py`) in place. Learning-from-execution has fundamentally different runtime characteristics from that package's sync-client role: it reads `task-memory`'s `Timeline`, it makes an embedding call, and it produces new skill records rather than pulling existing ones from an external platform. Keeping it separate means `skill-catalog-subscription`, when eventually implemented, does not need to anticipate a local-authoring code path it explicitly designed against (its own spec's "local catalog is not independently authored" scenario).
|
||||
**Alternative considered**: add authoring/versioning/embedding logic directly inside `skills/catalog.py` behind a feature flag. Rejected — it would make the sync-client package responsible for a write path its own spec explicitly forbids for the *synced* store, forcing every future reader of `skills/catalog.py` to reason about two different write-authority rules (sync-only vs. locally-authored) inside one module; a separate package with its own store keeps each write-authority rule enforceable at the package boundary, mirroring `semantic-scene-runtime`'s D1 and `world-model-runtime`'s D1 precedent of "new sibling package over folding into the layer below."
|
||||
|
||||
### D2: Synthesis triggered by an optional `TaskRunner.on_task_succeeded` hook, not a background scanner over `task-memory`
|
||||
`runtime/task.py`'s `TaskRunner.run()` gains an optional constructor argument `on_task_succeeded: Callable[[str, str, Timeline], None] | None = None` (task_id, goal, timeline), invoked exactly once, immediately after the loop determines final `status = succeeded`, before `run()` returns. When Skill Authoring is enabled in `skills_learning/config.py`, `TaskRunner` wires this to `synthesis.synthesize_flow_skill`; when disabled or left `None`, behavior is identical to today.
|
||||
**Alternative considered**: a separate offline batch job that periodically scans `storage/task_metadata.py`'s SQLite table for newly-succeeded tasks and synthesizes skills asynchronously. Rejected for this milestone — a batch job adds a second process/scheduling concern (when does it run, how does it avoid re-processing the same task twice) for a benefit (decoupling from the task loop's latency) that doesn't matter here, since synthesis is deliberately post-completion, not mid-loop; an in-process hook fired once at completion is simpler and has no scheduling state to get wrong. A batch/backfill mode remains a reasonable follow-up once there's a concrete need to synthesize skills from tasks that ran before this change existed.
|
||||
|
||||
### D3: Parameter abstraction via cross-goal diffing against the same tool-call skeleton, not NLP-based slot extraction from the goal string
|
||||
`synthesize_flow_skill(goal, timeline)` first extracts the ordered `(tool_name, args)` sequence from `timeline.read(task_id)`'s `tool_call` fields (dropping non-mutating/read-only calls like `describe_screen`/`screenshot`/`ui_tree` from the *template*, since those are re-derivable at replay time, while keeping `tap`/`swipe`/`input_text`/`launch_app`/similar mutating calls). It then looks up any already-stored skill whose stored steps have the same tool-name sequence (same skeleton, e.g. `launch_app → tap → input_text → tap`); if one exists, it diffs argument values position-by-position across the two executions and promotes any value that differs between the stored version and this run into a named `{param}` placeholder (e.g. a `input_text` call's literal text becomes `{search_query}`), inferring the parameter's `name` from the corresponding UI element's label/purpose when available (reusing `semantic-scene-runtime`'s `SemanticScene.widgets[].purpose` label as the naming source, when present, else falling back to a positional name like `param_2`). If no matching skeleton exists yet, the very first execution is stored as-is with zero parameters (nothing to diff against) and only gains parameters once a second, divergent-in-values execution is synthesized.
|
||||
**Alternative considered**: parse the natural-language `goal` string itself (e.g. via NER/pattern matching) to identify which words are "the variable part" (a search term, a contact name) and map those directly to template parameters. Rejected — goal phrasing is free-form and unconstrained (matching `semantic-scene-runtime`'s own open-vocabulary stance on intents), so goal-string parsing is fragile and would require its own NLP pipeline; diffing the *executed steps'* concrete argument values across two runs of "the same" flow is grounded in what the agent actually did, not what it said it wanted, and needs no parameter-worthy-token classifier of its own.
|
||||
|
||||
### D4: Versioning triggers on tool-name-sequence divergence beyond a configured tolerance, tracked as an explicit version chain
|
||||
`versioning.py`'s `diff_flow_versions(stored_steps, executed_steps) -> VersionDiff` computes: (a) whether the tool-name sequence itself differs (insertion/deletion/reorder of a step — always triggers a new version, no tolerance), and (b) for a matching skeleton, what fraction of argument positions newly qualify as "must become a parameter" per D3 (a configurable tolerance, default: any position that has ever varied is fine to keep parameterizing on the same version; only a *sequence*-level change forces a version bump). Every stored `FlowTemplateSkill` version keeps `version: int` (bumped) and `parent_version_id: str | None` (points at the version it diverged from), so history is a chain, not an overwrite; the newest version is what `skill-catalog`-style `get_skill`/list operations surface by default, but older versions remain fetchable by id.
|
||||
**Alternative considered**: version by content hash of the full step list (any byte-level difference is a new version, including argument-value-only changes that D3 would otherwise absorb into a parameter). Rejected — this would create a new version on essentially every synthesis run (since argument values differ per goal by design), defeating the entire point of parameterization; divergence has to be judged at the *tool-name-sequence* level (structural), with argument-value differences intentionally absorbed as parameters rather than treated as version-worthy changes, otherwise D3's parameter abstraction and D4's versioning contradict each other.
|
||||
|
||||
### D5: Embedding-based retrieval via a dedicated `EmbeddingIndex`, not full-text/keyword search reused from `skill-catalog`'s existing search
|
||||
`skill-embedding-retrieval` embeds `f"{name}: {description}\nOriginal goal: {goal}"` per locally-authored skill into a fixed-dimension vector using a single concrete embedding client (provider/model selected in implementation, config-overridable, mirroring `semantic-scene-runtime`'s D4 "config value, not hardcoded" stance), stores it in a new `skill_embeddings` local index (skill id → vector + model name + `updated_at`, re-embedded when a skill's description/goal changes at re-synthesis), and answers `retrieve_candidate_skills(goal, top_k)` via cosine similarity over that index, entirely in-process (no external vector-DB dependency introduced for this milestone's expected scale). `skill-catalog-subscription`'s existing `search_skills` (name/tag/description substring matching) is left untouched and is a different, complementary retrieval mode (exact/keyword vs. semantic).
|
||||
**Alternative considered**: extend `skill-catalog-subscription`'s planned `search_skills` MCP tool to also do embedding similarity internally. Rejected — that tool's contract (per its own spec) is a synchronous, no-network-round-trip local query over the synced catalog; folding an embedding-model call into it would either force every `search_skills` invocation to pay embedding-call latency, or require pre-computed embeddings for *subscription-sourced* skills too, which is `skill-catalog-subscription`'s decision to make (or not) in its own future revision, not something this change should impose on it. A standalone `retrieve_candidate_skills()` in `skills_learning/retrieval.py` keeps the new retrieval mode additive and independently callable.
|
||||
|
||||
### D6: Locally-authored skills live in `skills_learning/`'s own catalog table, tagged `source = "local-synthesis"`, composed with `skill-catalog` only in prose
|
||||
Synthesized `FlowTemplateSkill` records are persisted in a new local store owned by `skills_learning/` (same in-memory/on-disk shape as `skill-catalog-subscription`'s `Skill`/`FlowTemplateSkill` dataclasses, reused by import — not redefined — so both stores speak the same schema), with `source = "local-synthesis"` and no `subscription id`. This does not write into, nor require any schema change of, `skill-catalog-subscription`'s own store. Any future unification (one queryable namespace across "synced" and "locally-learned" skills, e.g. for a single `list_skills` MCP call to return both) is left as this change's Open Questions / a follow-on `skill-catalog-subscription` revision to make, since that store's spec today explicitly rejects non-sync writes and this change must not contradict that spec.
|
||||
**Alternative considered**: propose a `MODIFIED Requirements` delta against `skill-catalog-subscription`'s `skill-catalog` spec loosening its "reject non-sync writes" rule to allow a `source = local-synthesis` exception. Rejected per this session's explicit constraint — `openspec/specs/` has no applied baseline for `skill-catalog`, so there is nothing to diff a `MODIFIED` delta against yet, and `skill-catalog-subscription` remains its own pending, unmodified change; composing in prose (shared dataclass shapes, distinct stores) achieves the same practical reuse without touching another change's files.
|
||||
|
||||
### D7: Reuse `semantic-scene-runtime`'s LLM-integration pattern for the embedding call: narrow client, degrade-safe, never blocks
|
||||
The embedding client in `skills_learning/embeddings.py` follows the same shape `semantic-scene-runtime`'s `llm_client.py` established: a narrow, mockable client interface; `embed_skill_text(text) -> list[float] | None`, returning `None` (never raising) on timeout/rate-limit/disabled-config, exactly like `enrich_scene()`'s `SemanticScene | None` contract. Since embedding failure happens on the *post-task* synthesis path (not the live Observe→Act loop), a `None` result simply means "store the skill without an embedding, retrievable by name/id but not yet by similarity search" rather than blocking synthesis or versioning — `retrieve_candidate_skills()` skips any skill record with no stored vector.
|
||||
**Alternative considered**: make embedding a hard requirement of synthesis (skill authoring fails/rolls back entirely if embedding fails). Rejected — this would let a transient embedding-provider outage silently prevent skill learning even though the flow-template synthesis and versioning (the actually load-bearing artifact) succeeded independently; degrading to "skill stored, not yet semantically searchable" is strictly better than "skill not learned at all," and matches the project's established degrade-path precedent from `semantic-scene-runtime`.
|
||||
|
||||
## Risks / Trade-offs
|
||||
|
||||
- **[Risk]** Diffing argument values across two executions to infer parameters (D3) can misfire if the *first* two executions of a goal-family happen to share a value that later varies (e.g. always searching "coffee" twice, never generalizing to `{search_query}` until a third, different execution appears) → **Mitigation**: this is an inherent, accepted limitation of any 2-sample diff approach — document it in Open Questions as "parameterization confidence improves with more samples, not guaranteed correct after exactly one divergence"; the skill is still usable (with the shared literal baked in) until a genuinely divergent execution arrives to trigger parameterization, so nothing breaks, it just under-parameterizes early.
|
||||
- **[Risk]** Synthesizing a skill from a single successful task risks encoding an accidental/fragile path (e.g. steps that happened to work once but aren't a generally reliable flow) as if it were a validated reusable skill → **Mitigation**: out of scope to solve with a confidence/reliability score in this change (no repeated-execution signal exists yet for a single-run skill); `skills_learning/config.py` can gate authoring behind a "minimum N successful executions of this goal-family before first synthesis" threshold as a future config addition, but a real Planner retrieving and *trying* a skill (future work) is still expected to verify against the live `Scene`/`SemanticScene` before blindly trusting a one-shot-derived flow, matching `world-model-runtime`'s "advisory, not source-of-truth" stance on `WorldState`.
|
||||
- **[Risk]** The tool-name-sequence-skeleton matching in D3/D4 (used to decide "is this the same skill as before") has no formal goal-similarity threshold defined in this design — two semantically different goals that happen to execute the identical tool-name skeleton (e.g. two different searches) could be treated as "the same skill family" prematurely → **Mitigation**: skeleton-matching is scoped as a heuristic starting point, documented as an Open Question; if this proves too coarse in practice, a future revision can require skeleton match **and** a minimum goal-embedding similarity (reusing D5's embedding index) before treating two runs as the same skill family, rather than skeleton alone.
|
||||
- **[Trade-off]** Keeping locally-authored skills in a separate store from `skill-catalog-subscription`'s synced catalog (D6) means any consumer wanting "all skills, synced and learned" must query two stores and merge, not one → **Acceptable** because unifying them requires a `skill-catalog-subscription` spec change this session is explicitly scoped not to make; the shared dataclass shape (same `Skill`/`FlowTemplateSkill` types) keeps the merge trivial for whichever future change decides to do it (likely `skill-mcp-tools`' `list_skills` gaining a second source).
|
||||
- **[Trade-off]** Defaulting Skill Authoring to disabled (unlike `world-model-runtime`'s default-enabled precedent) means most existing/near-term task runs will not accumulate learned skills until explicitly turned on → **Acceptable**, matching `semantic-scene-runtime`'s reasoning: synthesis + embedding is a real CPU/LLM-adjacent cost on the success path, and applying this change must not silently change existing task cost/latency profiles for callers who haven't opted in.
|
||||
- **[Trade-off]** No skill-execution engine is built here (Non-Goal) — a synthesized `FlowTemplateSkill` is inert data until some future runner resolves its parameters and drives `tools/` — this is intentional scope discipline (per the proposal's Impact section) but means this milestone's value is not observable end-to-end (goal → learned skill → skill actually reused) until that follow-on work exists; acceptable since `skill-catalog-subscription`'s own proposal already scoped a "run_skill_flow parameter-resolution helper" as a distinct concern this change should not duplicate.
|
||||
|
||||
## Migration Plan
|
||||
|
||||
This is purely additive, matching the "safe default, optional hook" shape of `semantic-scene-runtime` and `world-model-runtime`:
|
||||
1. Add the `skills_learning/` package: `synthesis.py`, `versioning.py`, `embeddings.py`, `retrieval.py`, `config.py` (enable flag defaulting to **disabled**, divergence tolerance, embedding model name, default `top_k`).
|
||||
2. Reuse (import, do not redefine) `Skill`/`FlowTemplateSkill`/`SkillMetadata` dataclass shapes from `skill-catalog-subscription`'s planned `skills/models.py` once that change is implemented; if implemented first, `skills_learning/` depends on it for shared types only, never for its sync/storage code paths. If `skills_learning/` is implemented before `skill-catalog-subscription`, define the minimal shared dataclass shape locally and note it must be reconciled (not duplicated) once that change lands.
|
||||
3. Add a new local store for synthesized skills (`skills_learning/`-owned catalog table, `source = "local-synthesis"`) plus the `skill_embeddings` index table.
|
||||
4. Add the optional `on_task_succeeded: Callable[[str, str, Timeline], None] | None = None` constructor argument to `TaskRunner` (`runtime/task.py`); when Skill Authoring is enabled in config and no explicit hook is passed, `TaskRunner` wires a default one calling `synthesis.synthesize_flow_skill`; call it once, only after `status = succeeded` is determined, at the very end of `run()`.
|
||||
5. Add the embedding-provider SDK dependency and `skills_learning*` to `pyproject.toml`'s `[tool.setuptools.packages.find].include` list; add embedding config with the enable flag defaulting to disabled so applying this change never silently changes existing task cost/latency (matching `semantic-scene-runtime`'s D-config precedent).
|
||||
6. No changes to `storage/timeline.py`'s persisted format, `skill-catalog-subscription`'s schema/spec, or any existing tool signature.
|
||||
7. Run `pytest` — full existing suite green with zero test-content edits; new tests (`tests/test_skill_synthesis.py`-style) exercise synthesis/versioning/retrieval logic against canned `Timeline` fixtures and a mocked embedding client, not live model calls.
|
||||
8. Rollback: net-new package plus one additive, default-`None` constructor argument with a config-gated default wiring — reverting is `git revert` of the commit(s); no data migration of existing stores, no external system beyond the embedding provider (itself optional/degrade-safe per D7).
|
||||
|
||||
## Open Questions
|
||||
|
||||
- Whether the "same skill family" match (D3/D4's tool-name-skeleton matching) needs a goal-embedding-similarity floor in addition to skeleton equality, to avoid conflating two structurally-identical-but-semantically-different goals — left as a heuristic-first approach for this milestone, revisit once real synthesis data exists.
|
||||
- Whether a minimum-successful-executions threshold should gate first-time synthesis (to avoid learning from a single lucky run) — left as a future `skills_learning/config.py` addition once there's real usage data to tune it against, per the Risks section.
|
||||
- Whether/how a future `skill-catalog-subscription` revision should unify the synced and locally-authored stores behind one `list_skills`/`search_skills` surface (D6's deferred merge) — explicitly left to that change, not decided here.
|
||||
- Which embedding provider/model to default to, and whether it should be the same provider as `semantic-scene-runtime`'s enrichment client (operational simplicity: one API key/provider to manage) or an independent choice optimized purely for embedding quality/cost — left as an implementation-time decision, not fixed in this design.
|
||||
- Whether `WorldState.history` (from `world-model-runtime`, already flagged in that change's own Open Questions as a possible future Skill-synthesis consumer) should feed into synthesis here — e.g. using recent world-state transitions to help segment "which steps belong to this skill" versus incidental navigation — left open; this change synthesizes purely from `Timeline`'s tool-call sequence and does not yet consume `WorldState`.
|
||||
@@ -0,0 +1,30 @@
|
||||
## Why
|
||||
|
||||
`skill-catalog-subscription` (already proposed, unapplied) lets the agent *consume* skills that an external Subscription Platform authored, versioned, and pushed down — but it has no mechanism for the agent to *learn* a skill from its own experience. Every task the agent completes successfully today (once `apex-agent-mvp`'s `agent-runtime`/`task-memory` capabilities are applied) leaves behind a fully-recorded `Timeline` of scenes/prompts/tool-calls/results, and that recording is simply discarded once the task ends — there is no path from "the agent just did X successfully" to "the agent can do X again faster, or hand X to another task as a reusable flow." This change closes that gap: **Skill Runtime (learning half)** synthesizes a reusable, parameterized flow-template skill from a completed task's executed step sequence, detects when a later execution of "the same" skill diverges enough to warrant a new version, and makes locally-learned skills discoverable by semantic similarity to a new goal — so the Planner (present stub, future LLM-driven) has a growing, self-improving library of known-good flows to try before planning from scratch every time.
|
||||
|
||||
## What Changes
|
||||
|
||||
- Add a **Skill Authoring** capability: after a task completes with `status = succeeded`, a synthesis pass reads that task's `Timeline` (from `task-memory`'s `storage/timeline.py`) plus the goal string that produced it, extracts the ordered sequence of executed tool calls (`tap`/`swipe`/`input_text`/`launch_app`/...), and produces a `FlowTemplateSkill` (matching `skill-catalog-subscription`'s `skill-catalog` shape: an ordered `steps` list of tool name + args-template, and a `parameters` schema) with literal argument values that vary across similar goals abstracted into named `{param}` placeholders.
|
||||
- Add a **Skill Versioning** capability: when synthesis produces a step sequence for a skill that already exists in local storage (matched by name/goal-family, not by id), diff the newly executed steps against the currently-stored version's steps; if they differ beyond a configured tolerance (extra/missing/reordered steps, or a materially different parameter set), store a new version and retain version history rather than overwriting silently.
|
||||
- Add a **Skill Embedding Retrieval** capability: embed each locally-authored skill's `name` + `description` + originating goal text into a vector, persist those vectors alongside the skill record, and expose a `retrieve_candidate_skills(goal, top_k)` function returning ranked candidates by cosine similarity to a new incoming goal, so a Planner can check "has something like this already been learned" before planning from scratch.
|
||||
- Extend `TaskRunner`'s completion path with an optional **post-task synthesis hook** (`on_task_succeeded(task_id, goal, timeline)`), called once, only on success, only when Skill Authoring is enabled in config (default **disabled**, since synthesis + embedding is extra CPU/LLM cost on the success path and should not silently change task latency for existing callers/tests) — mirroring `semantic-scene-runtime`'s default-off precedent for cost-bearing additions, not `world-model-runtime`'s default-on precedent (which is pure in-memory derivation, not an LLM/embedding call).
|
||||
- Compose with, but do not modify, `skill-catalog-subscription`'s `skill-catalog` storage model: locally-authored skills are written as `FlowTemplateSkill` records tagged with a `source = "local-synthesis"` discriminator (vs. `source = "<subscription-id>"` for externally-synced skills) so both kinds can be listed/searched through the same catalog surface without the sync client ever attempting to push a locally-authored skill upstream or the synthesis pass ever overwriting a subscription-sourced skill.
|
||||
- **BREAKING**: none. `TaskRunner`'s new completion hook defaults to a no-op when Skill Authoring is disabled; nothing existing changes shape or behavior.
|
||||
|
||||
## Capabilities
|
||||
|
||||
### New Capabilities
|
||||
- `skill-authoring`: Synthesizes a parameterized `FlowTemplateSkill` from a completed task's `Timeline` (executed tool-call sequence) and the goal that produced it, abstracting goal-specific literals into named parameters.
|
||||
- `skill-versioning`: Detects divergence between a newly synthesized flow and the currently-stored version of "the same" skill, and manages version bump/history (never silent overwrite) when the executed sequence has materially changed.
|
||||
- `skill-embedding-retrieval`: Embeds locally-authored (and, read-only, subscription-sourced) skill descriptions/goals and retrieves a ranked list of candidate skills by semantic similarity to a new incoming goal, for a Planner to consider before planning from scratch.
|
||||
|
||||
### Modified Capabilities
|
||||
(none — `task-memory` (`apex-agent-mvp`) and `skill-catalog`/`skill-mcp-tools`/`skill-subscription-sync` (`skill-catalog-subscription`) are composed with in prose only; neither has an applied baseline in `openspec/specs/` to diff against, and this change does not alter either's specified behavior. `task-memory`'s `Timeline` is read as an input; `skill-catalog`'s storage shape is reused as the target format for synthesized skills, tagged with a new `source` value it already accommodates as a free-form field.)
|
||||
|
||||
## Impact
|
||||
|
||||
- **New package**: `skills_learning/` — `synthesis.py` (`synthesize_flow_skill(goal, timeline) -> FlowTemplateSkill`, tool-call-sequence extraction + parameter abstraction), `versioning.py` (`diff_flow_versions()`, `VersionStore` — bump/history logic), `embeddings.py` (`embed_skill_text()`, `EmbeddingIndex` — vector storage + cosine-similarity ranking), `retrieval.py` (`retrieve_candidate_skills(goal, top_k)`), `config.py` (enable flag, divergence tolerance, embedding model name, `top_k` default).
|
||||
- **Modified**: `runtime/task.py` (`TaskRunner` gains an optional `on_task_succeeded` hook invoked once at the end of a successful `run()`, default `None`/no-op); no change to `runtime/context.py`, `runtime/planner.py`, `runtime/executor.py` beyond the hook wiring — this change does not itself teach a Planner to call `retrieve_candidate_skills()` (that is a future LLM-driven Planner's job, matching `world-model-runtime`'s precedent of exposing a read-only input without building its consumer).
|
||||
- **Storage**: reuses `skill-catalog-subscription`'s catalog store for `FlowTemplateSkill` records (adding `source`, `version`, `parent_version_id` fields it already anticipates via free-form metadata); adds a new local `skill_embeddings` table/index (skill id → vector, model name, updated_at) — a new store, not a modification of `skill-catalog`'s schema, since `skill-catalog-subscription` does not define one today.
|
||||
- **External dependency**: introduces an embedding model call (provider TBD in design.md) as the second LLM-adjacent integration point in the codebase after `semantic-scene-runtime`'s enrichment call; reuses that change's pattern (narrow, mockable client; degrade-safe; never blocks the task loop it hooks into) rather than inventing a new one.
|
||||
- **Out of scope**: no changes to `skill-catalog-subscription`'s sync contract, MCP tool surface, or subscription/visibility model; no UI (that is `web-console`'s domain); no automatic skill *execution* triggering (a Planner choosing to run a retrieved skill is future Planner work); no cross-device or cross-tenant skill sharing beyond whatever the shared `skill-catalog` store already implies.
|
||||
@@ -0,0 +1,60 @@
|
||||
## ADDED Requirements
|
||||
|
||||
### Requirement: Synthesis triggers only on successful task completion
|
||||
The system SHALL synthesize a flow-template skill only when a task's final status is `succeeded`, and SHALL NOT attempt synthesis for a task that failed, was cancelled, or is still running.
|
||||
|
||||
#### Scenario: Successful task triggers synthesis
|
||||
- **WHEN** a task completes with status `succeeded` and Skill Authoring is enabled
|
||||
- **THEN** the system reads that task's timeline and goal and produces a flow-template skill candidate
|
||||
|
||||
#### Scenario: Failed task does not trigger synthesis
|
||||
- **WHEN** a task completes with status `failed` (or is cancelled/still running)
|
||||
- **THEN** the system does not synthesize any skill from that task's timeline
|
||||
|
||||
### Requirement: Skill Authoring defaults to disabled
|
||||
The system SHALL leave Skill Authoring disabled by default in configuration, so applying this capability does not change the cost or latency of any existing task run until a caller explicitly enables it.
|
||||
|
||||
#### Scenario: Default configuration performs no synthesis
|
||||
- **WHEN** a task completes successfully and Skill Authoring has not been explicitly enabled in configuration
|
||||
- **THEN** the system performs no synthesis work and the task's completion path behaves exactly as it would without this capability
|
||||
|
||||
#### Scenario: Explicit enable activates synthesis
|
||||
- **WHEN** an operator enables Skill Authoring in configuration
|
||||
- **THEN** subsequently completed successful tasks are eligible for synthesis
|
||||
|
||||
### Requirement: Flow-template skill synthesized from executed tool-call sequence
|
||||
The system SHALL derive a flow-template skill's ordered steps from the sequence of mutating tool calls (e.g. `tap`, `swipe`, `input_text`, `launch_app`) recorded in the completed task's timeline, in the order they were executed, and SHALL exclude read-only/observational tool calls (e.g. `describe_screen`, `screenshot`, `ui_tree`) from the synthesized step list.
|
||||
|
||||
#### Scenario: Mutating steps are included in order
|
||||
- **WHEN** a task's timeline contains a sequence of `launch_app`, `tap`, `input_text`, `tap` tool calls that all succeeded
|
||||
- **THEN** the synthesized skill's steps list contains those four steps in that same order
|
||||
|
||||
#### Scenario: Read-only observation calls are excluded
|
||||
- **WHEN** a task's timeline includes `describe_screen` or `screenshot` calls interleaved with mutating calls
|
||||
- **THEN** the synthesized skill's steps list omits those read-only calls and retains only the mutating steps
|
||||
|
||||
### Requirement: Parameter abstraction from cross-execution argument diffing
|
||||
The system SHALL abstract a synthesized skill's step arguments into named parameters by comparing the newly executed argument values against a previously stored skill with the same tool-name step sequence, promoting any argument value that differs between the two executions into a named placeholder, and SHALL leave a first-time synthesis (no prior matching skeleton) with zero parameters.
|
||||
|
||||
#### Scenario: First execution of a flow has no parameters
|
||||
- **WHEN** no previously stored skill shares the newly executed tool-name sequence
|
||||
- **THEN** the synthesized skill is stored with its literal argument values and an empty parameters list
|
||||
|
||||
#### Scenario: Second, divergent execution promotes a differing value to a parameter
|
||||
- **WHEN** a later successful task executes the same tool-name sequence as a stored skill but with a different literal value at one argument position
|
||||
- **THEN** the system promotes that argument position to a named parameter in the skill's `parameters` schema and replaces the literal in `steps` with a `{param}` placeholder referencing it
|
||||
|
||||
#### Scenario: Identical repeated execution does not spuriously add parameters
|
||||
- **WHEN** a later successful task executes the same tool-name sequence as a stored skill with identical argument values at every position
|
||||
- **THEN** the system does not introduce any new parameter for that skill
|
||||
|
||||
### Requirement: Locally-authored skills are stored separately from synced skills
|
||||
The system SHALL persist locally-synthesized flow-template skills tagged with a `source` of `local-synthesis`, in a store owned by this capability, and SHALL NOT write into or modify the externally-synced skill catalog store or its sync-only write contract.
|
||||
|
||||
#### Scenario: Synthesized skill is tagged as locally-authored
|
||||
- **WHEN** a flow-template skill is synthesized from a completed task
|
||||
- **THEN** its stored record has `source = "local-synthesis"` and no subscription identifier
|
||||
|
||||
#### Scenario: Synced skill catalog is untouched by synthesis
|
||||
- **WHEN** a skill is synthesized and stored by this capability
|
||||
- **THEN** no record in the externally-synced skill catalog store is created, modified, or removed as a result
|
||||
@@ -0,0 +1,41 @@
|
||||
## ADDED Requirements
|
||||
|
||||
### Requirement: Skill text embedded on synthesis and re-synthesis
|
||||
The system SHALL compute and persist an embedding vector for each locally-authored skill's name, description, and originating goal text whenever that skill is first synthesized or a new version is stored, associated with that skill's id and version.
|
||||
|
||||
#### Scenario: New skill gains an embedding
|
||||
- **WHEN** a flow-template skill is synthesized for the first time
|
||||
- **THEN** the system computes an embedding vector from its name, description, and originating goal, and stores it alongside the skill record
|
||||
|
||||
#### Scenario: New version gains its own embedding
|
||||
- **WHEN** a new version of an existing skill is created
|
||||
- **THEN** the system computes and stores an embedding for that version, independent of any embedding stored for prior versions
|
||||
|
||||
### Requirement: Embedding failure degrades to non-retrievable-by-similarity, never blocks synthesis
|
||||
The system SHALL NOT allow an embedding-provider failure (timeout, rate limit, disabled configuration, connection error) to prevent a skill from being synthesized or versioned; on such failure, the skill SHALL be stored without a similarity-searchable embedding.
|
||||
|
||||
#### Scenario: Embedding call fails but skill is still stored
|
||||
- **WHEN** the embedding provider call fails or times out during synthesis of an otherwise-successful skill
|
||||
- **THEN** the skill's flow-template record is still stored, and it is retrievable by exact name/id lookup but excluded from similarity-based retrieval results until a subsequent embedding attempt succeeds
|
||||
|
||||
### Requirement: Ranked retrieval of candidate skills by goal similarity
|
||||
The system SHALL provide a function that, given a new goal string and a requested result count, returns locally-authored skills that have a stored embedding, ranked by descending semantic similarity between the goal and each skill's stored embedding.
|
||||
|
||||
#### Scenario: Similar goal returns matching skill highest-ranked
|
||||
- **WHEN** a new goal is semantically similar to a previously-learned skill's originating goal
|
||||
- **THEN** that skill appears in the ranked candidate results, ordered ahead of less-similar skills
|
||||
|
||||
#### Scenario: Requested count limits results
|
||||
- **WHEN** a caller requests the top `k` candidate skills for a goal
|
||||
- **THEN** the system returns at most `k` ranked results, even if more embedded skills exist
|
||||
|
||||
#### Scenario: No embedded skills yields an empty result
|
||||
- **WHEN** no locally-authored skill currently has a stored embedding
|
||||
- **THEN** the retrieval function returns an empty ranked list rather than raising an error
|
||||
|
||||
### Requirement: Retrieval scoped to locally-authored skills unless explicitly extended
|
||||
The system SHALL restrict ranked candidate retrieval to skills stored by this capability's own local-synthesis store by default, and SHALL treat inclusion of externally-synced skills as a separate, explicit extension rather than an implicit default.
|
||||
|
||||
#### Scenario: Default retrieval excludes synced-only skills without embeddings
|
||||
- **WHEN** the skill catalog contains externally-synced skills that have never been embedded by this capability
|
||||
- **THEN** ranked candidate retrieval returns only locally-authored skills with stored embeddings, without erroring on the presence of unembedded synced skills
|
||||
@@ -0,0 +1,38 @@
|
||||
## ADDED Requirements
|
||||
|
||||
### Requirement: Structural divergence triggers a new version
|
||||
The system SHALL compare a newly synthesized flow's tool-name step sequence against the currently-stored version of the matching skill, and SHALL create a new version (rather than overwriting the stored one) whenever the tool-name sequence differs by insertion, deletion, or reordering of a step.
|
||||
|
||||
#### Scenario: Extra step triggers a new version
|
||||
- **WHEN** a newly executed flow for a matching skill contains an additional tap step not present in the currently-stored version's sequence
|
||||
- **THEN** the system stores a new version of the skill rather than overwriting the existing stored version
|
||||
|
||||
#### Scenario: Reordered steps trigger a new version
|
||||
- **WHEN** a newly executed flow for a matching skill executes the same tool names as the stored version but in a different order
|
||||
- **THEN** the system stores a new version of the skill
|
||||
|
||||
#### Scenario: Argument-value-only differences do not trigger a version bump
|
||||
- **WHEN** a newly executed flow has the identical tool-name sequence as the stored version and differs only in argument values already covered by parameter abstraction
|
||||
- **THEN** the system does not create a new version, and instead updates the existing version's parameters per the skill-authoring capability
|
||||
|
||||
### Requirement: Version history is retained, never silently overwritten
|
||||
The system SHALL retain every version of a skill it creates, each carrying an incrementing `version` number and a reference to the version it diverged from, and SHALL NOT delete or overwrite a prior version's stored record when a new version is created.
|
||||
|
||||
#### Scenario: New version references its parent
|
||||
- **WHEN** a new version of a skill is created due to structural divergence
|
||||
- **THEN** the new version's record stores a reference to the prior version's id and an incremented version number
|
||||
|
||||
#### Scenario: Prior version remains fetchable
|
||||
- **WHEN** a new version of a skill has been created
|
||||
- **THEN** the prior version's record remains retrievable by its own id, unmodified
|
||||
|
||||
### Requirement: Default retrieval surfaces the newest version
|
||||
The system SHALL treat the highest-numbered version of a skill as the default result returned by a lookup-by-name/goal-family query, while still allowing an explicit lookup of any specific prior version by its id.
|
||||
|
||||
#### Scenario: Lookup by name returns newest version
|
||||
- **WHEN** a caller looks up a skill by its name or goal-family without specifying a version
|
||||
- **THEN** the system returns the highest-numbered stored version of that skill
|
||||
|
||||
#### Scenario: Explicit id lookup returns the requested version
|
||||
- **WHEN** a caller requests a skill by a specific prior version's id
|
||||
- **THEN** the system returns that exact version's record, not the newest version
|
||||
@@ -0,0 +1,49 @@
|
||||
## 1. Package scaffolding
|
||||
|
||||
- [ ] 1.1 Create `skills_learning/` package with `__init__.py`, `models.py`, `synthesis.py`, `versioning.py`, `embeddings.py`, `retrieval.py`, `config.py`, `store.py`
|
||||
- [ ] 1.2 Add `skills_learning*` to `pyproject.toml`'s `[tool.setuptools.packages.find].include` list and add the embedding-provider SDK dependency
|
||||
- [ ] 1.3 Add `skills_learning/config.py` with `SkillAuthoringConfig` (enable flag default `False`, divergence tolerance, embedding model name, default `top_k`) and a module-level accessor mirroring `semantic-scene-runtime`'s/`world-model-runtime`'s config-module pattern
|
||||
- [ ] 1.4 Add `skills_learning/models.py` defining (or importing, if `skill-catalog-subscription` is already implemented) the shared `Skill`/`FlowTemplateSkill`/`SkillMetadata` dataclass shapes, plus this capability's own `source`, `version`, `parent_version_id` fields
|
||||
|
||||
## 2. Local skill store (skill-authoring, skill-versioning)
|
||||
|
||||
- [ ] 2.1 Implement `skills_learning/store.py`: a local store for locally-synthesized `FlowTemplateSkill` records, separate from `skill-catalog-subscription`'s synced catalog, with `create_version()`, `get_by_id()`, `get_latest_by_name()`, `list_versions(name)`
|
||||
- [ ] 2.2 Enforce `source = "local-synthesis"` tagging on every record written by this store; add a guard/test that this store never writes into or imports a write-path of `skill-catalog-subscription`'s catalog module
|
||||
- [ ] 2.3 Write unit tests for the store's version-chain semantics: creating a new version does not delete/modify prior versions, and `get_latest_by_name()` returns the highest `version`
|
||||
|
||||
## 3. Timeline extraction and parameter abstraction (skill-authoring)
|
||||
|
||||
- [ ] 3.1 Implement `skills_learning/synthesis.py`'s tool-call extraction: given a `task_id`, read `storage.timeline.Timeline.read(task_id)` and produce an ordered list of `(tool_name, args)` pairs, filtering out read-only tool names (`describe_screen`, `screenshot`, `ui_tree`, `find_text`, `find_icon`)
|
||||
- [ ] 3.2 Implement skeleton matching: given an extracted tool-name sequence, look up any stored skill (via `store.py`) whose latest version has the identical tool-name sequence
|
||||
- [ ] 3.3 Implement cross-execution argument diffing: compare extracted argument values position-by-position against a matched stored version's steps, and promote any differing value into a named `{param}` placeholder plus a corresponding entry in the skill's `parameters` schema
|
||||
- [ ] 3.4 Implement parameter naming: prefer a name derived from the corresponding `SemanticScene.widgets[].purpose` label when available (optional dependency on `semantic/`'s output, degrading gracefully when absent), else fall back to a positional name (e.g. `param_2`)
|
||||
- [ ] 3.5 Implement `synthesize_flow_skill(goal, timeline) -> FlowTemplateSkill`: orchestrates extraction → skeleton match → diffing → parameter promotion → returns a candidate skill record (not yet persisted)
|
||||
- [ ] 3.6 Write unit tests for first-time synthesis (no prior match, zero parameters), second-execution parameter promotion, and identical-repeat synthesis (no spurious new parameters), using canned `TimelineRecord` fixtures
|
||||
|
||||
## 4. Version divergence detection (skill-versioning)
|
||||
|
||||
- [ ] 4.1 Implement `skills_learning/versioning.py`'s `diff_flow_versions(stored_steps, executed_steps) -> VersionDiff`: detect tool-name-sequence insertion/deletion/reorder (structural divergence) versus argument-value-only differences
|
||||
- [ ] 4.2 Implement version-bump logic: on structural divergence, construct a new `FlowTemplateSkill` version with incremented `version` and `parent_version_id` set to the prior version's id; on argument-only divergence, update the existing version's parameters in place (no bump)
|
||||
- [ ] 4.3 Write unit tests: extra/missing/reordered step triggers a version bump; identical-sequence-different-values does not bump but does update parameters; assert prior version records remain retrievable and unmodified after a bump
|
||||
|
||||
## 5. Post-task synthesis hook wiring
|
||||
|
||||
- [ ] 5.1 Add an optional `on_task_succeeded: Callable[[str, str, Timeline], None] | None = None` constructor argument to `TaskRunner` in `runtime/task.py`, invoked exactly once at the end of `run()` when the final status is `succeeded`
|
||||
- [ ] 5.2 Wire a default hook (when `on_task_succeeded` is not explicitly passed and Skill Authoring is enabled in `skills_learning/config.py`) that calls `synthesis.synthesize_flow_skill()`, runs versioning via `versioning.py`, and persists the result via `store.py`
|
||||
- [ ] 5.3 Verify that when Skill Authoring is disabled (default) or `on_task_succeeded` is left `None` and disabled, `TaskRunner.run()`'s behavior and return value are byte-for-byte identical to before this change
|
||||
- [ ] 5.4 Write a unit test that runs a fake successful `TaskRunner` loop with Skill Authoring enabled and asserts a skill record is stored after completion, and a test that asserts no store write occurs when disabled
|
||||
|
||||
## 6. Embedding and retrieval (skill-embedding-retrieval)
|
||||
|
||||
- [ ] 6.1 Implement `skills_learning/embeddings.py`'s embedding client interface: `embed_skill_text(text) -> list[float] | None`, catching timeout/rate-limit/disabled-config/connection-error internally and returning `None` rather than raising, mirroring `semantic/llm_client.py`'s degrade-safe contract
|
||||
- [ ] 6.2 Implement a local `skill_embeddings` index (skill id + version → vector, model name, `updated_at`) in `skills_learning/store.py` or a dedicated `skills_learning/embeddings_store.py`
|
||||
- [ ] 6.3 Wire embedding computation into the post-synthesis/versioning path: call `embed_skill_text()` on `name + description + goal` for every newly stored skill version, storing the resulting vector (or leaving the skill un-embedded if the call returns `None`)
|
||||
- [ ] 6.4 Implement `skills_learning/retrieval.py`'s `retrieve_candidate_skills(goal, top_k) -> list[ScoredSkill]`: embed the incoming goal, compute cosine similarity against every stored skill embedding, and return the top `top_k` ranked results, skipping skills with no stored embedding
|
||||
- [ ] 6.5 Write unit tests: ranked ordering for a goal similar to a stored skill's originating goal (using a fake/deterministic embedding function), `top_k` truncation, empty-result case when no skill has an embedding, and a case where an embedding call returns `None` and the skill is stored but excluded from retrieval results
|
||||
|
||||
## 7. Integration tests and validation
|
||||
|
||||
- [ ] 7.1 Write an end-to-end test: run a fake successful task twice with slightly different goal text/argument values through `TaskRunner` (Skill Authoring enabled, embedding client mocked), asserting the second run produces a new skill version with a promoted parameter and its own embedding
|
||||
- [ ] 7.2 Write an end-to-end test: run a fake successful task, then call `retrieve_candidate_skills()` with a new, semantically similar goal string, asserting the synthesized skill is returned
|
||||
- [ ] 7.3 Run the full existing `pytest` suite and confirm zero existing test files require content changes (only new `tests/test_skill_*.py`-style files are added)
|
||||
- [ ] 7.4 Add a smoke test importing `skills_learning` alongside existing `tests/` smoke coverage, confirming the package has no import-time dependency on `skill-catalog-subscription`'s sync client (only, optionally, its shared model shapes if already implemented)
|
||||
@@ -0,0 +1,2 @@
|
||||
schema: spec-driven
|
||||
created: 2026-07-06
|
||||
@@ -0,0 +1,46 @@
|
||||
## Context
|
||||
|
||||
The backend (`api/rest.py`, `api/mcp.py`) currently exposes only device-control and task-execution primitives (`/devices`, `/agent/task`, `/task/{id}`), all consumed by an LLM/MCP client. Two pieces of already-implemented backend functionality are not exposed over HTTP at all: `TaskMetadataStore.list_tasks()` (storage/task_metadata.py) and `Timeline.read(task_id)` / `ArtifactStore.read_steps(task_id)` (storage/timeline.py, storage/artifact_store.py). Device registration (`DeviceManager.register_device`, core/device_manager.py) requires a Python `driver_factory` callable and is purely in-memory — there is no config file or persistence, so every device must be (re-)registered in code on each process start. There is no operator-facing UI at all today.
|
||||
|
||||
## Goals / Non-Goals
|
||||
|
||||
**Goals:**
|
||||
- Expose existing device/task/timeline state over a read-only console REST API.
|
||||
- Let an operator register/remove devices and adjust simple runtime parameters (`max_steps`) through a REST API, with changes surviving process restarts.
|
||||
- Ship an independent Vue 3 SPA that consumes these APIs for a status dashboard + configuration screens.
|
||||
|
||||
**Non-Goals:**
|
||||
- Authentication/authorization for console endpoints (assumed trusted network for now).
|
||||
- Editing planner/agent logic, prompts, or tool registration from the UI.
|
||||
- Concurrent multi-operator conflict resolution for config edits.
|
||||
- Any change to the existing MCP tool surface or `/agent/task` execution semantics.
|
||||
- Supporting driver types beyond `wda` in the config UI (Android etc. remain out of scope, consistent with `apex-agent-mvp`).
|
||||
|
||||
## Decisions
|
||||
|
||||
- **New `/console` router, shared state.** Add `api/console.py` exposing an `APIRouter` mounted under `/console` inside the existing `create_app()` in `api/rest.py`, reusing the same `device_manager`, `metadata_store`, and `task_runner` instances already constructed there. Rejected alternative: a second standalone FastAPI app — would duplicate state wiring and require its own process/port for no benefit.
|
||||
- **New persisted device-config store, additive to `DeviceManager`.** Add `storage/device_config.py` (SQLite, same pattern as `TaskMetadataStore`) storing `device_id, name, driver_type, connection_info(json)`. On `create_app()` startup, read all rows and call `device_manager.register_device(...)` with a `driver_factory` built via a small `driver_type -> factory` registry (today: `{"wda": WDADriverConfig-based factory}`). `DeviceManager` itself is not modified — it stays a pure in-memory runtime registry; the config store is the source of truth for "which devices should exist," the manager is the source of truth for "current connection state." Rejected alternative: teaching `DeviceManager` to persist itself — would blur its existing single responsibility and touch a capability (`device-management`) this change intends to leave unmodified.
|
||||
- **Runtime parameters persisted in the same store.** Add a small `settings(key, value)` key-value table to the same SQLite file for `max_steps` (and future simple scalars), read at startup to construct `TaskRunnerConfig`, and updated in-place on the live `TaskRunner.config` object when changed via `PUT /console/config` — so changes apply immediately without a restart and still survive one. Rejected alternative: in-memory-only (rejected: contradicts "simple configuration" persisting across restarts, which is the point of the config API); a separate config file (rejected: adds a second persistence mechanism for no benefit over reusing the SQLite file already needed for device configs).
|
||||
- **Screenshots returned as base64 JSON, not static files.** `GET /console/tasks/{id}/timeline` reads screenshots via `ArtifactStore`/`Timeline` and inlines them as base64, mirroring the existing `/devices/{id}/screenshot` convention. Rejected alternative: mounting `tasks/history` as a static file directory — simpler but opens a directory-listing/path-traversal surface for no strong benefit at this stage; base64 keeps the same trust boundary as the rest of the API.
|
||||
- **Driver type allow-list validated at registration time.** `POST /console/devices` rejects any `driver_type` not in the small supported set (`{"wda"}`) with `400`, rather than silently accepting arbitrary strings that would fail later at `connect()` time.
|
||||
- **Independent Vue 3 SPA, no backend template rendering.** Per explicit choice, the console frontend is its own project (own build/deploy, e.g. Vite), calling the JSON APIs over HTTP. `create_app()` enables permissive CORS for local development only (see Open Questions for production hosting).
|
||||
|
||||
## Risks / Trade-offs
|
||||
|
||||
- [No auth on console endpoints] → Mitigation: document as trusted-network-only for this change; flag auth as explicit follow-up before any non-trusted deployment.
|
||||
- [`max_steps` change applied to a running `TaskRunner` while a task is mid-execution] → Mitigation: change only affects the loop bound checked at the top of each iteration in `TaskRunner.run`; an in-flight task simply picks up the new bound on its next iteration, no partial-state corruption possible.
|
||||
- [SQLite device-config store and `TaskMetadataStore` are two separate files/schemas] → Mitigation: acceptable for this change's scope (mirrors existing pattern); revisit consolidation only if a third store is added later.
|
||||
- [Base64 screenshot inlining increases timeline payload size for long tasks] → Mitigation: acceptable for MVP console scope (tasks capped at `max_steps`, typically ≤ 20 steps); revisit pagination/lazy-loading if task length grows.
|
||||
|
||||
## Migration Plan
|
||||
|
||||
1. Add `storage/device_config.py` (device configs + settings key-value table).
|
||||
2. Add `api/console.py` with status endpoints (read-only) first; mount into `create_app()`.
|
||||
3. Add config endpoints (device register/unregister, runtime param get/update) to `api/console.py`; wire startup reload of persisted devices/settings into `create_app()`.
|
||||
4. Scaffold the independent Vue 3 SPA project consuming the above.
|
||||
5. Rollback: the change is purely additive (new router, new tables, new frontend project) — rollback is removing the router mount and the new frontend project; no migration of existing `tasks.sqlite3` or `tasks/history` data is required or affected.
|
||||
|
||||
## Open Questions
|
||||
|
||||
- Where should the Vue 3 SPA project live — a `console/` subfolder in this repo, or a separate sibling repository? Defaulting to an in-repo `console/` subfolder for now (simplest to keep in sync with the backend during early iteration); revisit if it needs independent versioning/deploy cadence.
|
||||
- Should console endpoints eventually sit behind the same auth mechanism as a future MCP-facing auth layer, or get their own? Deferred until authentication is actually scoped.
|
||||
@@ -0,0 +1,28 @@
|
||||
## Why
|
||||
|
||||
Apex Agent currently exposes device control and task execution only through the REST/MCP tool surface (`api/rest.py`, `api/mcp.py`) — there is no way for a human operator to see, at a glance, which devices are connected, what a running task is doing, or to register a new device/tweak a runtime setting without editing code and restarting the process. As the platform moves past a single-developer MVP, an operator needs a lightweight status/config surface: view device and task state (including step-by-step timeline/screenshot replay for debugging), and perform simple configuration (register/remove devices, adjust runtime parameters) through a web UI instead of the Python API directly.
|
||||
|
||||
## What Changes
|
||||
|
||||
- Add a **console status API**: `GET` endpoints to list devices with live status, list/filter tasks, fetch task detail, and fetch a task's per-step timeline (tool call, result, scene, screenshot) built on top of the existing `TaskMetadataStore.list_tasks()` and `Timeline.read()`, which are implemented but not yet exposed over HTTP.
|
||||
- Add a **console config API**: endpoints to register a new device (driver_type + connection_info, e.g. WDA `server_url`/`udid`/`wda_local_port`), unregister a device, and view/update adjustable runtime parameters (currently just `TaskRunnerConfig.max_steps`).
|
||||
- Add a **device config store** (new persistence, SQLite like `TaskMetadataStore`) so devices registered through the console survive process restarts — on startup the app reloads persisted device configs and re-registers them with `DeviceManager`, instead of devices existing only in the in-memory dict as today.
|
||||
- Add an **independent web console frontend** (separate Vue 3 SPA project with its own build/deploy) that consumes the two APIs above to render: a device status dashboard, a task list/detail view with timeline + screenshot replay, and configuration screens for device management and runtime parameters.
|
||||
- Explicitly out of scope: authentication/authorization for the console (assumed to run on a trusted network for now), editing agent/planner logic from the UI, multi-user concurrent config editing safeguards, and any change to the existing MCP tool surface or `/agent/task` execution semantics.
|
||||
|
||||
## Capabilities
|
||||
|
||||
### New Capabilities
|
||||
- `console-status-api`: Read-only REST endpoints exposing device status, task list/detail, and per-task execution timeline (including screenshot retrieval) for the web console.
|
||||
- `console-config-api`: REST endpoints for device registration/deregistration (backed by a persisted device config store) and viewing/updating simple runtime parameters (e.g. task runner `max_steps`).
|
||||
- `web-console-ui`: Independent frontend SPA providing the status dashboard and configuration pages, consuming `console-status-api` and `console-config-api`.
|
||||
|
||||
### Modified Capabilities
|
||||
(none — this change is additive on top of `device-management`, `task-memory`, and `mcp-tool-server`; no existing requirements change)
|
||||
|
||||
## Impact
|
||||
|
||||
- **New code**: `api/console.py` (or similar) for the new REST routes; `storage/device_config.py` for the persisted device config store; a new top-level frontend project (e.g. `console/` or a sibling repo) for the Vue 3 SPA.
|
||||
- **Modified code**: `api/rest.py` to mount the console routes and to reload persisted device configs at `create_app()` startup; `runtime/task.py`/`TaskRunnerConfig` to allow `max_steps` to be read/updated at runtime.
|
||||
- **Dependencies**: no new backend dependencies expected (reuses FastAPI/SQLite already in `pyproject.toml`); the new frontend project brings its own Node/Vue 3 toolchain, separate from the Python package.
|
||||
- **Impact on existing behavior**: `DeviceManager` in-memory registration behavior is unchanged; the console config store only adds a reload-on-startup convenience layer on top of it. Existing `/devices`, `/agent/task`, `/task/{id}` endpoints and MCP tools are unaffected.
|
||||
@@ -0,0 +1,45 @@
|
||||
## ADDED Requirements
|
||||
|
||||
### Requirement: Register a device
|
||||
The system SHALL expose `POST /console/devices` accepting `driver_type`, `connection_info`, and an optional `name`, persist the configuration, and register the device with the running `DeviceManager`. Unsupported `driver_type` values SHALL be rejected.
|
||||
|
||||
#### Scenario: Register a supported device
|
||||
- **WHEN** an operator posts `{"driver_type": "wda", "connection_info": {"server_url": "http://127.0.0.1:4723", "udid": "abc123"}}` to `POST /console/devices`
|
||||
- **THEN** the response is a `201` with the new device's id and status `idle`, the device appears in `GET /console/devices`, and its configuration is persisted
|
||||
|
||||
#### Scenario: Reject unsupported driver type
|
||||
- **WHEN** an operator posts a device with `driver_type` not in the supported set (currently `wda`)
|
||||
- **THEN** the response is a `400` and no device is registered or persisted
|
||||
|
||||
### Requirement: Unregister a device
|
||||
The system SHALL expose `DELETE /console/devices/{device_id}` that disconnects and removes the device from `DeviceManager` and deletes its persisted configuration, or returns a `404` if the device is unknown.
|
||||
|
||||
#### Scenario: Unregister a known device
|
||||
- **WHEN** an operator calls `DELETE /console/devices/{device_id}` for a registered device
|
||||
- **THEN** the response is a `204`, the device no longer appears in `GET /console/devices`, and its persisted configuration is removed
|
||||
|
||||
#### Scenario: Unregister an unknown device
|
||||
- **WHEN** an operator calls `DELETE /console/devices/{device_id}` for a device id that is not registered
|
||||
- **THEN** the response is a `404`
|
||||
|
||||
### Requirement: Persisted devices reload on startup
|
||||
The system SHALL re-register every persisted device configuration with `DeviceManager` automatically when the application starts, without requiring manual re-registration.
|
||||
|
||||
#### Scenario: Restart with previously registered devices
|
||||
- **WHEN** the application starts and one or more device configurations exist in the persisted device config store
|
||||
- **THEN** each persisted device appears in `GET /console/devices` immediately after startup, without any additional operator action
|
||||
|
||||
### Requirement: View and update runtime parameters
|
||||
The system SHALL expose `GET /console/config` returning current adjustable runtime parameters (currently the task runner's `max_steps`) and `PUT /console/config` to update them, applying the change to the running task runner immediately and persisting it across restarts.
|
||||
|
||||
#### Scenario: View current runtime parameters
|
||||
- **WHEN** an operator calls `GET /console/config`
|
||||
- **THEN** the response is a `200` including the current `max_steps` value
|
||||
|
||||
#### Scenario: Update max_steps
|
||||
- **WHEN** an operator calls `PUT /console/config` with `{"max_steps": 30}`
|
||||
- **THEN** the response is a `200` with the updated value, subsequently started tasks (and the next iteration of any in-flight task) use the new `max_steps`, and the value is persisted so it survives a restart
|
||||
|
||||
#### Scenario: Reject invalid runtime parameter value
|
||||
- **WHEN** an operator calls `PUT /console/config` with a non-positive `max_steps` (e.g. `0` or `-1`)
|
||||
- **THEN** the response is a `400` and the previous value remains in effect
|
||||
@@ -0,0 +1,49 @@
|
||||
## ADDED Requirements
|
||||
|
||||
### Requirement: List device status
|
||||
The system SHALL expose `GET /console/devices` returning every registered device's id, name, status (`idle`/`busy`/`offline`/`error`), and driver type, reusing `DeviceManager.list_devices()`.
|
||||
|
||||
#### Scenario: Devices are registered
|
||||
- **WHEN** an operator calls `GET /console/devices` while one or more devices are registered
|
||||
- **THEN** the response is a `200` with a JSON array containing one entry per device with its current `status`
|
||||
|
||||
#### Scenario: No devices registered
|
||||
- **WHEN** an operator calls `GET /console/devices` while no devices are registered
|
||||
- **THEN** the response is a `200` with an empty JSON array
|
||||
|
||||
### Requirement: List tasks
|
||||
The system SHALL expose `GET /console/tasks` returning all tasks known to `TaskMetadataStore`, most recently created first, with optional `device_id` and `status` query filters.
|
||||
|
||||
#### Scenario: List all tasks
|
||||
- **WHEN** an operator calls `GET /console/tasks` with no query parameters
|
||||
- **THEN** the response is a `200` with a JSON array of tasks ordered by `created_at` descending
|
||||
|
||||
#### Scenario: Filter by device
|
||||
- **WHEN** an operator calls `GET /console/tasks?device_id=<id>`
|
||||
- **THEN** the response contains only tasks whose `device_id` matches `<id>`
|
||||
|
||||
#### Scenario: Filter by status
|
||||
- **WHEN** an operator calls `GET /console/tasks?status=running`
|
||||
- **THEN** the response contains only tasks whose `status` equals `running`
|
||||
|
||||
### Requirement: Task detail
|
||||
The system SHALL expose `GET /console/tasks/{task_id}` returning the full task record, or a `404` if the task does not exist.
|
||||
|
||||
#### Scenario: Task exists
|
||||
- **WHEN** an operator calls `GET /console/tasks/{task_id}` for a known task id
|
||||
- **THEN** the response is a `200` with the task's goal, device_id, status, timestamps, and failure_reason (if any)
|
||||
|
||||
#### Scenario: Task does not exist
|
||||
- **WHEN** an operator calls `GET /console/tasks/{task_id}` for an unknown task id
|
||||
- **THEN** the response is a `404`
|
||||
|
||||
### Requirement: Task timeline replay
|
||||
The system SHALL expose `GET /console/tasks/{task_id}/timeline` returning the ordered list of execution steps recorded for the task, each including its scene, prompt, tool call, result, timestamp, and a base64-encoded screenshot when one was captured for that step.
|
||||
|
||||
#### Scenario: Task has recorded steps
|
||||
- **WHEN** an operator calls `GET /console/tasks/{task_id}/timeline` for a task that executed at least one step
|
||||
- **THEN** the response is a `200` with a JSON array ordered by step `index` ascending, each entry including `image_base64` when a screenshot was captured for that step
|
||||
|
||||
#### Scenario: Task has no recorded steps
|
||||
- **WHEN** an operator calls `GET /console/tasks/{task_id}/timeline` for a task with no timeline history (e.g. it failed before its first step)
|
||||
- **THEN** the response is a `200` with an empty JSON array
|
||||
@@ -0,0 +1,45 @@
|
||||
## ADDED Requirements
|
||||
|
||||
### Requirement: Device status dashboard
|
||||
The web console SHALL display a list of registered devices with their current status, fetched from `console-status-api`, and SHALL show an explicit empty state when no devices are registered.
|
||||
|
||||
#### Scenario: Devices registered
|
||||
- **WHEN** an operator opens the dashboard while devices are registered
|
||||
- **THEN** each device is shown with its name/id and current status, refreshed without a full page reload
|
||||
|
||||
#### Scenario: No devices registered
|
||||
- **WHEN** an operator opens the dashboard while no devices are registered
|
||||
- **THEN** an empty-state message is shown inviting the operator to add a device
|
||||
|
||||
### Requirement: Task list and timeline replay
|
||||
The web console SHALL let an operator browse the task list, open a task's detail view, and step through its recorded timeline including screenshots.
|
||||
|
||||
#### Scenario: Browse tasks
|
||||
- **WHEN** an operator opens the task list view
|
||||
- **THEN** tasks are shown most-recent-first with their goal, device, and status, and can be filtered by device or status
|
||||
|
||||
#### Scenario: Replay a task's timeline
|
||||
- **WHEN** an operator opens a completed or failed task's detail view
|
||||
- **THEN** the console renders each recorded step in order, showing its tool call, result, and screenshot (when available)
|
||||
|
||||
### Requirement: Device configuration
|
||||
The web console SHALL let an operator register a new device (choosing a supported driver type and entering its connection info) and remove an existing device, surfacing any validation error returned by `console-config-api`.
|
||||
|
||||
#### Scenario: Add a device successfully
|
||||
- **WHEN** an operator submits the add-device form with valid driver type and connection info
|
||||
- **THEN** the new device appears in the dashboard without a page reload
|
||||
|
||||
#### Scenario: Add a device with invalid input
|
||||
- **WHEN** an operator submits the add-device form with an unsupported driver type
|
||||
- **THEN** the console displays the error returned by the API and does not add the device to the list
|
||||
|
||||
#### Scenario: Remove a device
|
||||
- **WHEN** an operator confirms removal of a registered device
|
||||
- **THEN** the device disappears from the dashboard without a page reload
|
||||
|
||||
### Requirement: Runtime parameter configuration
|
||||
The web console SHALL let an operator view and edit adjustable runtime parameters (currently `max_steps`) through a settings form.
|
||||
|
||||
#### Scenario: Update a runtime parameter
|
||||
- **WHEN** an operator changes `max_steps` in the settings form and saves
|
||||
- **THEN** the console shows the updated value on success, or the validation error if the API rejects it
|
||||
@@ -0,0 +1,36 @@
|
||||
## 1. Device config & settings store
|
||||
|
||||
- [x] 1.1 Add `storage/device_config.py` with a SQLite-backed `DeviceConfigStore` (`device_id, name, driver_type, connection_info json`) following the `TaskMetadataStore` pattern, with `add`, `remove`, `list`, and `get` methods
|
||||
- [x] 1.2 Add a `settings(key, value)` table to the same store with `get_setting`/`set_setting` helpers, seeded with a default `max_steps`
|
||||
- [x] 1.3 Add unit tests for `DeviceConfigStore` (add/remove/list, settings get/set, unknown key/device handling)
|
||||
|
||||
## 2. Console status API
|
||||
|
||||
- [x] 2.1 Add `api/console.py` with an `APIRouter`; implement `GET /console/devices` (list + status via `DeviceManager.list_devices()`)
|
||||
- [x] 2.2 Implement `GET /console/tasks` (via `TaskMetadataStore.list_tasks()`) with optional `device_id`/`status` query filters
|
||||
- [x] 2.3 Implement `GET /console/tasks/{task_id}` (404 on unknown id)
|
||||
- [x] 2.4 Implement `GET /console/tasks/{task_id}/timeline` reading `Timeline.read(task_id)` / `ArtifactStore`, inlining screenshots as base64
|
||||
- [x] 2.5 Add tests for all four endpoints covering populated and empty states, and the 404 case
|
||||
|
||||
## 3. Console config API
|
||||
|
||||
- [x] 3.1 Implement `POST /console/devices`: validate `driver_type` against a supported-driver registry (`{"wda": ...}`), build the `driver_factory`, call `DeviceManager.register_device`, persist via `DeviceConfigStore`; return `400` for unsupported driver types
|
||||
- [x] 3.2 Implement `DELETE /console/devices/{device_id}`: call `DeviceManager.unregister_device` and remove the persisted config; return `404` for unknown device ids
|
||||
- [x] 3.3 Implement `GET /console/config` and `PUT /console/config` for `max_steps`, validating positive values (`400` otherwise), applying the change to the live `TaskRunner.config` and persisting it via the settings table
|
||||
- [x] 3.4 Wire startup reload: in `create_app()`, read all persisted device configs and the persisted `max_steps` setting, register devices and construct `TaskRunnerConfig` accordingly, before returning the app
|
||||
- [x] 3.5 Mount the `console` router into `create_app()` in `api/rest.py` and enable permissive CORS for local frontend development
|
||||
- [x] 3.6 Add tests for register/unregister (success + validation errors), config get/update (success + invalid value), and startup reload of persisted devices/settings
|
||||
|
||||
## 4. Web console frontend (independent SPA)
|
||||
|
||||
- [x] 4.1 Scaffold an independent Vue 3 + Vite SPA project (in-repo `console/` per design's default) with its own `package.json`/build tooling
|
||||
- [x] 4.2 Implement the device status dashboard view (list + empty state) against `GET /console/devices`
|
||||
- [x] 4.3 Implement the task list view (with device/status filters) and task detail + timeline replay view (screenshots) against the console-status-api endpoints
|
||||
- [x] 4.4 Implement the device configuration screen (add/remove device forms, surfacing API validation errors) against the console-config-api endpoints
|
||||
- [x] 4.5 Implement the runtime parameter settings form (`max_steps`) against `GET/PUT /console/config`
|
||||
- [x] 4.6 Document how to run the frontend dev server against the local backend (base URL config, CORS expectations)
|
||||
|
||||
## 5. Verification
|
||||
|
||||
- [x] 5.1 Run the full backend test suite (`pytest`) and confirm no regressions to existing REST/MCP endpoints
|
||||
- [ ] 5.2 Manually verify end-to-end: register a device via the UI, run a task via the existing `/agent/task` endpoint, and confirm its status/timeline/screenshots render correctly in the console
|
||||
@@ -0,0 +1,2 @@
|
||||
schema: spec-driven
|
||||
created: 2026-07-06
|
||||
@@ -0,0 +1,90 @@
|
||||
## Context
|
||||
|
||||
Today's Agent Runtime (`agent-runtime`, `apex-agent-mvp`, code-complete but unapplied) is exactly one loop shape: `runtime/task.py`'s `TaskRunner.run(task)` repeats observe→plan→act until the (currently stub) `Planner.goal_reached()` returns true, a step fails, or `TaskRunnerConfig.max_steps` (default 20) is exceeded, for one flat `Task.goal` string. `TaskContext` (`runtime/context.py`) holds `scenes`/`step_results` for that one run and is discarded when it ends. Three other pending, unapplied changes extend what a single step can *mean* but not the *shape* of orchestration above it: `semantic-scene-runtime` enriches a `Scene` into a `SemanticScene` per step; `world-model-runtime` derives a per-task `WorldState` (`current_app`/`current_page`/`variables`/bounded `history`) from that same step loop; `skill-learning-runtime` synthesizes a parameterized `FlowTemplateSkill` from a *completed* task's `Timeline`, but explicitly does not build anything that can *run* a skill — "a synthesized skill is inert data until some future runner resolves parameters and drives `tools/`." None of these three changes, nor `agent-runtime` itself, gives a caller a way to express "do sub-goal A, then run skill B, then wait until condition C, then do sub-goal D" as one persisted, resumable unit — that is this change's entire scope.
|
||||
|
||||
Two stakeholders: a workflow author (human or a future higher-level Agent/orchestrator, out of scope here) who defines a `WorkflowDefinition` up front as data (not interactively via a UI — no visual editor exists or is planned in this change), and the process running `WorkflowRunner`, which must be able to pick a `WorkflowRun` back up after a restart without re-running already-completed mutating steps (a `tap`, `input_text`, or skill invocation is not safely re-runnable the way a read-only `describe_screen` is).
|
||||
|
||||
## Goals / Non-Goals
|
||||
|
||||
**Goals:**
|
||||
- Introduce `WorkflowDefinition` as an ordered, possibly-branching list of `WorkflowStep`s of four kinds: planned-goal, skill-invocation, wait-for-condition, branch.
|
||||
- Introduce `WorkflowRun` as the persisted, checkpointed execution record of one `WorkflowDefinition` run, survivable across process restarts.
|
||||
- Provide a `WorkflowRunner` that composes (imports, never edits or subclasses) the existing `agent-runtime` `TaskRunner`/`Planner`/`Executor` for planned-goal steps.
|
||||
- Provide the first real skill-invocation execution path: resolve a `FlowTemplateSkill`'s parameters against step-supplied argument values and drive the resolved tool calls through `tools/` — the gap `skill-learning-runtime` deliberately left open.
|
||||
- Provide a small, closed, registrable set of wait/branch condition kinds sufficient for common multi-stage flows (text appears, a tracked variable equals a value, elapsed time, prior step succeeded), extensible the same way the Driver Registry (`device-agent-runtime-foundation`) is.
|
||||
- Make resumability real: a `WorkflowRunner.resume(run_id)` reload continues from the last checkpointed step, never re-executing a step already marked completed.
|
||||
|
||||
**Non-Goals:**
|
||||
- No visual workflow editor/UI — `WorkflowDefinition` is authored as data (constructed programmatically or loaded from a file/API a future caller provides); any UI is `web-console`'s domain, untouched here.
|
||||
- No distributed/multi-device workflow execution or cross-node coordination — that is Milestone 10 (Cloud Runtime); this milestone assumes one `WorkflowRunner` process driving one workflow run against one device at a time.
|
||||
- Does not replace or remove the existing single-goal Planner/Executor loop (`agent-runtime`) — a `WorkflowDefinition` with exactly one planned-goal step is a valid but uninteresting workflow; simple one-shot tasks should keep calling `TaskRunner` directly.
|
||||
- No general expression/scripting language for wait/branch conditions — only a closed, registrable set of condition *kinds*, each a small typed evaluator function, not an embedded interpreter or arbitrary `eval`.
|
||||
- No generic skill-kind runner — `skill-learning-runtime`'s `Skill.kind` also allows `knowledge` skills (non-executable reference text); this change's skill-invocation step only executes `flow_template` skills. Attempting to invoke a `knowledge`-kind skill is a step-definition error, not a runtime capability gap to fill here.
|
||||
- No retrieval-driven skill *selection* at workflow-run time — a skill-invocation step names an explicit `skill_id`; using `skill-embedding-retrieval`'s `retrieve_candidate_skills(goal, top_k)` to pick which skill a step should invoke is an authoring-time or future-Planner concern, not built into `WorkflowRunner` here.
|
||||
- No changes to `runtime/`, `storage/`, `tools/`, `skill-catalog-subscription`, or `web-console`'s files or specs.
|
||||
|
||||
## Decisions
|
||||
|
||||
### D1: New `workflow/` package, sibling to `semantic/`/`world/`/`skills_learning/`, not folded into `runtime/`
|
||||
`workflow/` (`models.py`, `runner.py`, `conditions.py`, `skill_exec.py`, `store.py`, `config.py`) is its own package. `runtime/` stays focused on the single-goal Observe-Think-Act-Observe loop shape (its established job per `device-agent-runtime-foundation`'s layering ADR); multi-stage orchestration is a distinct concern one layer above it, matching the sibling-package precedent already set by `semantic-scene-runtime`'s D1, `world-model-runtime`'s D1, and `skill-learning-runtime`'s D1.
|
||||
**Alternative considered**: extend `runtime/task.py`'s `TaskRunner` in place to accept a list of goals/steps instead of one goal. Rejected — `TaskRunner` is a pending, unapplied, code-complete capability (`agent-runtime`) with no applied baseline in `openspec/specs/` to safely diff a `MODIFIED` delta against in this session; treating it as a stable, composed-over dependency (never edited) avoids retroactively changing another change's still-pending contract, and keeps "one flat goal, one loop" and "multi-stage workflow of possibly-heterogeneous steps" as two separate, independently testable concerns.
|
||||
|
||||
### D2: `WorkflowStep` as a tagged union of four concrete step dataclasses, not one flat schema with optional fields
|
||||
`workflow/models.py` defines `PlannedGoalStep{goal: str}`, `SkillInvocationStep{skill_id: str, args: dict}`, `WaitForConditionStep{condition: ConditionSpec, timeout_seconds: float, poll_interval_seconds: float}`, and `BranchStep{condition: ConditionSpec, on_true: str, on_false: str}` (all sharing a common `step_id: str` and optional explicit `next_step_id: str | None` for non-branch kinds — default is "fall through to the next step in `WorkflowDefinition.steps` order"), unioned as `WorkflowStep = PlannedGoalStep | SkillInvocationStep | WaitForConditionStep | BranchStep`. `WorkflowRunner` dispatches on `isinstance`/a `kind` discriminator to one handler function per step type.
|
||||
**Alternative considered**: one flat `WorkflowStep` dataclass with optional fields for every kind (`goal: str | None`, `skill_id: str | None`, `condition: ConditionSpec | None`, `on_true`/`on_false: str | None` all present simultaneously). Rejected — this makes invalid states representable (a step with both `goal` and `condition` populated, ambiguous about which the runner should honor) and pushes validation into every consumer; a tagged union makes constructing a step with two kinds' fields at once a type error, not a runtime validation rule to remember and enforce.
|
||||
|
||||
### D3: `WorkflowRun` checkpointed to `WorkflowStore` after every completed step, not only at start/end
|
||||
`WorkflowRunner.run(definition)` creates a `WorkflowRun{id, definition_id, status, current_step_id, variables, step_results}` row via `WorkflowStore.create_run()`, then after each step completes (success or failure) calls `WorkflowStore.update_run()` to persist the new `current_step_id`/`status`/`step_results` entry before advancing. `WorkflowRunner.resume(run_id)` loads the persisted `WorkflowRun`, looks up its `current_step_id` in the `WorkflowDefinition`, and continues from there.
|
||||
**Alternative considered**: persist only the initial definition and the final result, reconstructing "how far did we get" by scanning `task-memory`'s `Timeline`/`storage/task_metadata.py` for related task ids. Rejected — `Timeline`/task-metadata know nothing about workflow-step boundaries (a workflow step is not a 1:1 mapping to a `Task`; a wait-for-condition or branch step produces no `Task` at all), so re-deriving workflow progress from task-memory records would require workflow-step-boundary metadata to be smuggled into another capability's storage; an explicit, workflow-owned checkpoint after each step is simpler and self-contained.
|
||||
|
||||
### D4: Resumability is "reload-and-continue from last checkpoint," not "replay from step 0 assuming idempotent actions"
|
||||
On `resume(run_id)`, `WorkflowRunner` never re-executes a step whose result is already recorded in the persisted `WorkflowRun.step_results` for `current_step_id`'s predecessors; it resumes execution starting at the step recorded as the current (in-progress or not-yet-started) one.
|
||||
**Alternative considered**: on resume, replay the entire workflow from its first step, relying on `tools/` actions being idempotent (e.g., re-tapping an already-tapped button is harmless) to make re-execution safe. Rejected — many mutating actions are not idempotent (sending a chat message twice creates two messages; launching a payment flow twice is unacceptable), so assuming idempotency across arbitrary drivers/tools is unsafe; explicit step-level checkpointing avoids re-invoking already-completed mutating steps, at the cost (see Risks) of not providing a stronger exactly-once guarantee across a crash *during* a single step's execution.
|
||||
|
||||
### D5: Skill-invocation steps get a first, minimal parameter-resolution + tool-dispatch path in `workflow/skill_exec.py`
|
||||
`skill_exec.run_flow_template_skill(skill: FlowTemplateSkill, args: dict) -> list[StepResult]` validates `args` against `skill.parameters` (missing required parameter or unknown extra key fails the step before any tool call is issued), substitutes `{param}` placeholders in the skill's stored `steps` (tool name + args-template, as produced by `skill-learning-runtime`'s synthesis) with the resolved values, and calls the corresponding `tools/*` functions directly (the same functions `Executor.execute()` already dispatches to for planned-goal steps), returning one `StepResult` per resolved step. This is deliberately the "some future runner" `skill-learning-runtime`'s design.md left as an open non-goal — Workflow Runtime is the first concrete need for it.
|
||||
**Alternative considered**: leave skill-invocation steps declared-but-unimplemented in this change too (a no-op stub, like `Planner.plan()`'s unused `world` kwarg in `world-model-runtime`), deferring actual skill execution to yet another future change. Rejected — a workflow step kind that can never execute would make this proposal's own headline example ("run a Skill instead of re-planning from scratch") impossible to demonstrate; scoping it down instead (only `flow_template`-kind skills, no retry/backoff beyond what a direct `tools/` call already offers, no retrieval-based selection — see Non-Goals) keeps the executable surface small without leaving it entirely unbuilt.
|
||||
|
||||
### D6: `ConditionEvaluator` port + registry for wait/branch conditions, mirroring the Driver Registry pattern
|
||||
`workflow/conditions.py` defines `ConditionEvaluator` (one method, `evaluate(spec: ConditionSpec, *, scene, world_state) -> bool`) and a registry mapping a condition `kind` string (`scene_contains_text`, `world_variable_equals`, `elapsed_seconds`, `step_result_success`) to an evaluator, the same "string key → pluggable implementation" shape as `driver/registry.py`'s `SUPPORTED_DRIVER_TYPES` (`device-agent-runtime-foundation`, D3) and `perception/provider.py`'s `PerceptionProvider` port (D8). A `WaitForConditionStep`/`BranchStep`'s `ConditionSpec{kind: str, params: dict}` is looked up in this registry at evaluation time; adding a new condition kind is a one-function addition to `conditions.py`, not a change to `WorkflowRunner`'s control flow.
|
||||
**Alternative considered**: hardcode condition evaluation as an `if/elif` chain over a `condition_type` string directly inside `WorkflowRunner`. Rejected — this is precisely the extension-point shape the project has already standardized on twice (Driver Registry, `PerceptionProvider`); a third ad hoc if/elif chain for the same "pluggable-by-string-key" problem would be inconsistent with established project convention for no benefit.
|
||||
|
||||
### D7: Branching via explicit `on_true`/`on_false` step-id targets, not a general expression/scripting language
|
||||
`BranchStep.condition` is evaluated via the same `ConditionEvaluator` registry as wait steps; the runner then sets `current_step_id` to `on_true` or `on_false` (both required, explicit step ids present in the same `WorkflowDefinition.steps`) rather than falling through sequentially.
|
||||
**Alternative considered**: support an embedded expression language (e.g., a small boolean-expression evaluator over `WorldState.variables`) for branch conditions, allowing arbitrary author-supplied predicates. Rejected — an embedded expression evaluator large enough to be genuinely useful risks becoming a code-injection-adjacent surface in a system that drives real devices (a crafted expression reaching into more than intended), and the proposal's own Non-Goals already scope out a general DSL; a closed, registrable set of condition kinds (D6) plus explicit `goto`-style step-id targets is sufficient for the "wait/branch" step types this milestone commits to, and is easy to extend later without an interpreter.
|
||||
|
||||
### D8: `WorkflowRunner` composes `TaskRunner` as a black box for planned-goal steps; never edits or subclasses it
|
||||
A `PlannedGoalStep` is executed by constructing a `core.models.Task(goal=step.goal, device_id=run.device_id)`, calling `TaskRunner(...).run(task)` (letting `agent-runtime`'s existing Planner/Executor/retry logic run unmodified), and reading back `task.status`/`task.failure_reason` to decide the step's `StepResult`. `WorkflowRunner` never imports or touches `runtime/task.py`'s internals beyond this public `run(task) -> Task` contract.
|
||||
**Alternative considered**: give `WorkflowRunner` its own lower-level loop that calls `Planner.plan()`/`Executor.execute()` directly per planned-goal step, skipping `TaskRunner`. Rejected — this would duplicate `TaskRunner`'s already-implemented max-steps/retry/failure-reason bookkeeping inside `workflow/`, doubling the surface area that has to stay behaviorally consistent with `agent-runtime`'s own spec; composing the existing public `run(task) -> Task` entry point is simpler and automatically inherits any future `agent-runtime` improvement (e.g., a smarter Planner) with zero change to `workflow/`.
|
||||
|
||||
### D9: `WorkflowStore` is a new, independently-owned SQLite database file, not a new table in `storage/task_metadata.py`'s existing database
|
||||
`workflow/store.py`'s `WorkflowStore` follows the same connect-per-call `sqlite3` pattern as `storage/task_metadata.py`, but opens its own file (default `workflows/workflows.sqlite3`) with its own schema (`workflow_definitions`, `workflow_runs`, `workflow_step_results` tables) rather than adding tables to `tasks/tasks.sqlite3`.
|
||||
**Alternative considered**: add `workflow_runs`/`workflow_step_results` tables directly into `storage/task_metadata.py`'s existing database and module. Rejected — `storage/` is owned by the pending, unapplied `task-memory` capability; adding tables/schema-migration code to it would be a de facto modification of another change's owned artifact with no corresponding spec delta to justify it, contradicting this session's explicit constraint not to touch other pending changes' files. A separate, `workflow/`-owned store with its own schema-creation code is fully additive and requires no coordination with `task-memory`'s eventual implementation.
|
||||
|
||||
## Risks / Trade-offs
|
||||
|
||||
- **[Risk]** Step-level checkpointing (D3/D4) does not provide a true exactly-once guarantee: if the process crashes *during* a mutating step's execution (after the tool call was issued but before the checkpoint write completes), resuming will re-execute that step, potentially double-sending a message or double-tapping an action → **Mitigation**: documented as an accepted limitation, not solved in this change (mirrors `task-memory`'s own per-tool-call durability limits); a future revision could require mutating skill/tool calls to carry a caller-supplied idempotency key threaded through `tools/`, but no such mechanism exists in `tools/` today to build on, so it is left as an Open Question rather than invented here.
|
||||
- **[Risk]** `WorkflowRunner`'s composition of `TaskRunner` (D8) depends on `agent-runtime`'s public `run(task) -> Task` contract remaining stable while `apex-agent-mvp` itself is still pending/unapplied; if that change's spec evolves before archiving, this composition point could silently drift → **Mitigation**: treat only `TaskRunner.run(task) -> Task` and the `Task.status`/`Task.failure_reason` fields as the relied-upon contract (not `TaskRunner`'s internals); cover this in an integration-style test that runs a real (not mocked) `TaskRunner` instance so any breaking drift fails a test, not silently.
|
||||
- **[Risk]** Skill-invocation execution (D5) drives `tools/` with parameter values resolved from step-supplied `args`, which could contain a malformed or unexpectedly-typed value that passes schema validation but produces an unintended device action (e.g., an empty string where a search query was expected) → **Mitigation**: parameter validation checks presence/type against `skill.parameters`' declared schema before any tool call is issued (fail the step, no partial execution), but does not attempt semantic validation of *values* (e.g., "is this a real search term") — that remains the same trust boundary `Executor.execute()` already has for planned-goal steps' tool-call arguments.
|
||||
- **[Trade-off]** Explicit `on_true`/`on_false` step-id branching (D7) instead of a general expression DSL means a workflow author must hand-wire every branch target and cannot express a condition beyond the registered kinds (D6) without a code change to `conditions.py` → **Acceptable**, matches this milestone's stated Non-Goal (no visual editor, no general DSL); a registrable-by-string-key extension point keeps adding a new kind cheap even though it requires a code change, not a data-only change.
|
||||
- **[Trade-off]** Requiring an explicit `skill_id` on `SkillInvocationStep` (D5's Non-Goal on retrieval-driven selection) means a workflow cannot dynamically pick "whichever skill best matches this sub-goal" at run time — only `skill-embedding-retrieval`'s `retrieve_candidate_skills()` at authoring time (outside this runtime) could inform which `skill_id` to hardcode into the step → **Acceptable** for this milestone; wiring retrieval into live step dispatch is future Planner-integration work, consistent with `world-model-runtime`'s and `skill-learning-runtime`'s shared precedent of exposing a capability without yet teaching a decision-maker to use it dynamically.
|
||||
- **[Trade-off]** A new, separate SQLite file (D9) means "all workflow runs" and "all tasks" are two different databases with no foreign-key enforcement between a `WorkflowRun`'s planned-goal step and the `Task`/`task_metadata` row it produced (only a `task_id` string reference stored in `WorkflowStepResult`) → **Acceptable**; this mirrors `skill-learning-runtime`'s D6 acceptance of "two stores, composed in prose, merge is a future concern" rather than forcing a schema dependency on another pending change's not-yet-applied storage.
|
||||
|
||||
## Migration Plan
|
||||
|
||||
This is purely additive; no existing module is edited:
|
||||
1. Add the `workflow/` package: `models.py` (`WorkflowDefinition`, the four `WorkflowStep` variants, `WorkflowRun`, `WorkflowStepResult`, `ConditionSpec`), `conditions.py` (`ConditionEvaluator` port + registry + the four starting condition kinds), `skill_exec.py` (`run_flow_template_skill`), `runner.py` (`WorkflowRunner.run()`/`.resume()` and one private dispatch method per step kind), `store.py` (`WorkflowStore`, schema creation for its own `workflows.sqlite3`), `config.py` (default poll interval, default wait timeout, default db path).
|
||||
2. Wire `WorkflowRunner`'s planned-goal step handler to construct a `Task` and call the existing `runtime.task.TaskRunner(...).run(task)` — import only, zero edits to `runtime/`.
|
||||
3. Wire `WorkflowRunner`'s skill-invocation step handler to `skill_exec.run_flow_template_skill`, importing `Skill`/`FlowTemplateSkill` dataclass shapes from `skill-learning-runtime`'s planned `skills_learning/` package (or, if implemented first, `skill-catalog-subscription`'s canonical `skills/models.py`) by import, not redefinition; if neither is implemented yet when `workflow/` is built, define the minimal shared shape locally with a note to reconcile once one of those changes lands (same caveat `skill-learning-runtime`'s own migration plan already carries for the same dependency direction).
|
||||
4. Wire `WorkflowRunner`'s wait/branch step handlers to `conditions.py`'s registry; optionally read `TaskContext.world`/`WorldState.variables` (from `world-model-runtime`, if that change is applied and enabled) for `world_variable_equals` conditions — degrade to "condition never satisfied until timeout" if `WorldState` is unavailable, never raise.
|
||||
5. Add `workflow*` to `pyproject.toml`'s `[tool.setuptools.packages.find].include` list; no new third-party dependency.
|
||||
6. Add new tests (`tests/test_workflow_runner.py`-style): a linear multi-step workflow (planned-goal → skill-invocation → wait → planned-goal), a branch workflow taking each path, a wait step that times out, and a resume test that constructs a fresh `WorkflowRunner`/`WorkflowStore` pointed at the same db file mid-run to simulate a process restart and asserts already-completed steps are not re-executed.
|
||||
7. Run `pytest` — the full existing suite stays green with zero edits to existing test files, confirming this change touches no existing behavior.
|
||||
8. Rollback: net-new package plus a net-new SQLite file with its own schema; reverting is `git revert` of the commit(s), with no data migration of any existing store and no external system involved beyond whatever device the composed `TaskRunner`/`tools/` calls already target.
|
||||
|
||||
## Open Questions
|
||||
|
||||
- Whether mutating skill-invocation/planned-goal steps need a stronger idempotency mechanism (e.g., a caller-supplied idempotency key threaded through `tools/`) to close the crash-during-step-execution gap noted in Risks, once resumability is exercised against real, non-idempotent device flows — left unsolved here, no such mechanism exists in `tools/` to build on yet.
|
||||
- Whether `WorkflowRun.variables` should automatically bridge with a planned-goal step's underlying `TaskContext`/`WorldState.variables` (so a value a Planner "remembers" mid-task is visible to a later branch/wait step in the same workflow) or stay a separate, workflow-only variable scope populated only by explicit step outputs — this change keeps them separate (no automatic bridging) and leaves richer variable-flow as a follow-on decision once real multi-stage workflows using both are observed.
|
||||
- Whether `skill-embedding-retrieval`'s `retrieve_candidate_skills()` should eventually be wired into `SkillInvocationStep` resolution (author supplies a goal instead of a fixed `skill_id`, runner picks the top candidate at execution time) rather than requiring an explicit `skill_id` — left to a future revision once retrieval-based selection has been exercised standalone.
|
||||
- Whether Milestone 10 (Cloud Runtime)'s distributed/multi-device execution will require redesigning `WorkflowRun`'s persistence schema for multi-node coordination (e.g., leader election over which node resumes a given run, or sharding runs by device) — this milestone's schema assumes single-process, single-node execution only, and that assumption should be revisited when Cloud Runtime is scoped.
|
||||
@@ -0,0 +1,29 @@
|
||||
## Why
|
||||
|
||||
`agent-runtime` (`apex-agent-mvp`, code-complete, unapplied) gives the runtime exactly one shape of work: a single flat goal driven by one Planner/Executor Observe-Think-Act-Observe loop until it succeeds, fails, or exceeds `max_steps`. Real device-automation usage is rarely one flat goal — it is a sequence of distinct sub-goals with control flow between them: open an app, send a message, **wait** for a reply to arrive, take a screenshot, then end; or run a locally-learned Skill (`skill-learning-runtime`, Milestone 7) instead of re-planning from scratch, only re-planning if the skill's assumptions don't hold. Nothing in the current runtime can express "do A, then B, then wait until C is true, then D," track where a multi-stage run is, or resume it if the process restarts mid-way. Workflow Runtime introduces `Workflow` as a first-class, persisted, resumable object composed of planner-derived steps, skill-invocation steps, wait-for-condition steps, and simple branch steps — sitting one layer above the existing single-goal loop, not replacing it.
|
||||
|
||||
## What Changes
|
||||
|
||||
- Add a new `workflow/` package defining a `WorkflowDefinition` (an ordered, possibly-branching list of `WorkflowStep`s) as a discriminated union of four step kinds: a **planned-goal step** (delegates a sub-goal string to the existing `agent-runtime` Planner/Executor loop), a **skill-invocation step** (resolves parameters and executes a locally-synthesized `FlowTemplateSkill` from `skill-learning-runtime`, Milestone 7), a **wait-for-condition step** (polls a named condition — e.g. scene text present, a world variable equals a value, elapsed time — up to a timeout), and a **branch step** (evaluates a condition and jumps to a named step id instead of falling through sequentially).
|
||||
- Add a `WorkflowRun` — the persisted, mutable execution record for one execution of a `WorkflowDefinition`: status (`pending`/`running`/`waiting`/`completed`/`failed`/`cancelled`), `current_step_id`, workflow-scoped `variables`, and a per-step result log — checkpointed to storage after every completed step so a `WorkflowRunner.resume(run_id)` can continue from the last checkpoint instead of re-running from step 0, without re-executing already-completed mutating steps.
|
||||
- Add a `WorkflowRunner` orchestration component that composes (imports, never subclasses or edits) `runtime/task.py`'s existing `TaskRunner` for planned-goal steps, and drives skill-invocation/wait/branch steps itself.
|
||||
- Add a first, minimal skill-invocation execution path (`workflow/skill_exec.py`): resolve a `FlowTemplateSkill`'s declared parameters against a step's supplied argument values, validate against the skill's parameter schema, and drive the resolved tool calls through `tools/` — the "some future runner resolves parameters and drives `tools/`" gap `skill-learning-runtime` explicitly left open.
|
||||
- Add a `ConditionEvaluator` port + registry (`workflow/conditions.py`) for wait/branch conditions, mirroring the Driver Registry / `PerceptionProvider` extension-point pattern already established by `device-agent-runtime-foundation`, with a small starting set of condition kinds (`scene_contains_text`, `world_variable_equals`, `elapsed_seconds`, `step_result_success`).
|
||||
- Add a new, independently-owned SQLite-backed `WorkflowStore` (`workflow/store.py`) for `WorkflowDefinition`/`WorkflowRun` persistence, following the same connect-per-call `sqlite3` pattern as `storage/task_metadata.py` but in its own database file — not a schema change to `storage/`, which remains owned by the pending `task-memory` capability.
|
||||
- No changes to `runtime/task.py`, `runtime/planner.py`, `runtime/executor.py`, `runtime/context.py`, `storage/*`, `tools/*`, or any other existing module's public behavior — this change is purely additive composition on top of them.
|
||||
|
||||
## Capabilities
|
||||
|
||||
### New Capabilities
|
||||
- `workflow-orchestration`: A `Workflow` model (planned-goal / skill-invocation / wait-for-condition / branch step kinds), executed by a `WorkflowRunner` that composes the existing single-goal Planner/Executor loop and the not-yet-built skill-execution path, persisted via a task-memory-style store, and resumable from its last checkpoint if interrupted mid-workflow.
|
||||
|
||||
### Modified Capabilities
|
||||
(none — `openspec/specs/` has no applied baseline for `agent-runtime`, `task-memory`, `skill-authoring`, or `skill-embedding-retrieval` yet, so this change cannot and does not write a `MODIFIED Requirements` delta against any of them; it composes with their pending, unapplied designs in prose only, and does not alter their specified behavior.)
|
||||
|
||||
## Impact
|
||||
|
||||
- **New package**: `workflow/` — `models.py` (`WorkflowDefinition`, `WorkflowStep` variants, `WorkflowRun`, `WorkflowStepResult`), `runner.py` (`WorkflowRunner`), `conditions.py` (`ConditionEvaluator` port + registry), `skill_exec.py` (parameter resolution + tool dispatch for skill-invocation steps), `store.py` (`WorkflowStore`, SQLite-backed), `config.py` (enable flags, default poll interval, default wait timeout).
|
||||
- **New storage**: a new `workflows/workflows.sqlite3` database file (own schema: `workflow_definitions`, `workflow_runs`, `workflow_step_results` tables), independent of `storage/task_metadata.py`'s `tasks/tasks.sqlite3`.
|
||||
- **Composed, not modified, dependencies**: `runtime/task.py`'s `TaskRunner` (planned-goal steps construct a `Task` and call `TaskRunner(...).run(task)`, reading back `task.status`/`task.failure_reason`), `skill-learning-runtime`'s `FlowTemplateSkill`/`Skill` dataclass shape (imported, not redefined, matching that change's own D6 precedent for composing with `skill-catalog-subscription`), and optionally `world-model-runtime`'s `WorldState.variables` (read-only, for `world_variable_equals` conditions — degrades to "condition never satisfied until timeout" if `WorldState` is absent, never raises).
|
||||
- **Config**: add `workflow*` to `pyproject.toml`'s `[tool.setuptools.packages.find].include` list; no new third-party dependency beyond the standard library `sqlite3` already used by `storage/task_metadata.py`.
|
||||
- **Out of scope**: no visual workflow editor/UI (a future `web-console` concern, not touched here); no distributed/multi-device workflow execution (Milestone 10, Cloud Runtime); does not replace or remove the existing single-goal Planner/Executor loop, which remains the right tool for simple one-shot tasks; no generic expression/scripting language for branch conditions (a closed, registrable set of condition kinds only); no changes to `skill-catalog-subscription`, `web-console`, or any other pending change's files.
|
||||
@@ -0,0 +1,97 @@
|
||||
## ADDED Requirements
|
||||
|
||||
### Requirement: Workflow definition as an ordered, branchable list of typed steps
|
||||
The system SHALL provide a `WorkflowDefinition` model representing an ordered, possibly-branching list of `WorkflowStep`s, where each step is exactly one of four kinds: a planned-goal step (a natural-language sub-goal delegated to the existing single-goal Planner/Executor loop), a skill-invocation step (a reference to a locally-synthesized flow-template skill plus argument values), a wait-for-condition step (a named condition, timeout, and poll interval), or a branch step (a named condition plus two target step ids). Each step SHALL have a unique `step_id` within its `WorkflowDefinition`.
|
||||
|
||||
#### Scenario: Workflow with heterogeneous step kinds is constructed
|
||||
- **WHEN** a `WorkflowDefinition` is built with a planned-goal step, a skill-invocation step, a wait-for-condition step, and a branch step in sequence
|
||||
- **THEN** the system accepts the definition and each step retains its declared kind and fields without requiring fields belonging to another step kind
|
||||
|
||||
#### Scenario: Duplicate step id is rejected
|
||||
- **WHEN** a `WorkflowDefinition` is constructed with two steps sharing the same `step_id`
|
||||
- **THEN** the system rejects the definition before any run is created from it
|
||||
|
||||
### Requirement: Persisted, checkpointed workflow run
|
||||
The system SHALL persist a `WorkflowRun` record for each execution of a `WorkflowDefinition`, containing the run's status, the currently active step id, workflow-scoped variables, and a per-step result log, and SHALL update this record after every completed step before advancing to the next one.
|
||||
|
||||
#### Scenario: Run status transitions as steps execute
|
||||
- **WHEN** a `WorkflowRun` is started for a `WorkflowDefinition`
|
||||
- **THEN** the system creates a persisted run record with status `running` and, as each step completes, updates the persisted `current_step_id` and per-step result log before the next step begins
|
||||
|
||||
#### Scenario: Run reaches terminal status
|
||||
- **WHEN** all steps in a `WorkflowDefinition` complete successfully, or a step fails without a defined recovery path
|
||||
- **THEN** the system updates the persisted `WorkflowRun` status to `completed` or `failed` respectively, and records a failure reason when failed
|
||||
|
||||
### Requirement: Resume from last checkpoint without re-executing completed steps
|
||||
The system SHALL support resuming an interrupted `WorkflowRun` from its last persisted checkpoint, continuing execution from the currently active step without re-executing any step already recorded as completed in that run's step result log.
|
||||
|
||||
#### Scenario: Resume after simulated process restart
|
||||
- **WHEN** a `WorkflowRun` has completed its first two steps and the process driving it stops before the third step completes, and a new `WorkflowRunner` instance is later pointed at the same persisted run id
|
||||
- **THEN** the system resumes execution starting at the third step and does not re-invoke the tool calls or sub-goal already recorded as completed for the first two steps
|
||||
|
||||
#### Scenario: Resume on an already-completed run is a no-op
|
||||
- **WHEN** `resume` is called with the id of a `WorkflowRun` whose status is already `completed`
|
||||
- **THEN** the system returns the run's existing final state without executing any further steps
|
||||
|
||||
### Requirement: Planned-goal step delegates to the existing single-goal loop
|
||||
The system SHALL execute a planned-goal step by delegating its sub-goal to the existing Planner/Executor Observe-Think-Act-Observe loop for a single task, and SHALL derive that step's success or failure from the resulting task's final status.
|
||||
|
||||
#### Scenario: Planned-goal step succeeds
|
||||
- **WHEN** a planned-goal step's delegated task reaches a completed status
|
||||
- **THEN** the workflow step is recorded as succeeded and the run advances to the next step
|
||||
|
||||
#### Scenario: Planned-goal step fails
|
||||
- **WHEN** a planned-goal step's delegated task reaches a failed status
|
||||
- **THEN** the workflow step is recorded as failed with the task's failure reason and the run's status becomes `failed` unless a branch step defines an alternate path
|
||||
|
||||
### Requirement: Skill-invocation step resolves parameters and executes a flow-template skill
|
||||
The system SHALL execute a skill-invocation step by validating its supplied argument values against the referenced flow-template skill's declared parameters, substituting the validated values into the skill's stored tool-call template, and executing the resolved tool calls in order.
|
||||
|
||||
#### Scenario: Skill invocation with valid parameters executes resolved tool calls
|
||||
- **WHEN** a skill-invocation step supplies argument values that satisfy the referenced skill's declared required parameters
|
||||
- **THEN** the system substitutes those values into the skill's stored steps and executes the resulting tool calls in the skill's recorded order
|
||||
|
||||
#### Scenario: Skill invocation with a missing required parameter fails without executing any tool call
|
||||
- **WHEN** a skill-invocation step omits a value for a parameter the referenced skill declares as required
|
||||
- **THEN** the system fails the step before issuing any tool call and records the missing-parameter reason
|
||||
|
||||
#### Scenario: Skill invocation referencing a non-flow-template skill is rejected
|
||||
- **WHEN** a skill-invocation step references a skill whose kind is not a flow-template
|
||||
- **THEN** the system fails the step as a step-definition error rather than attempting to execute it
|
||||
|
||||
### Requirement: Wait-for-condition step polls until satisfied or timed out
|
||||
The system SHALL execute a wait-for-condition step by repeatedly evaluating its named condition at the step's configured poll interval until the condition is satisfied or the step's configured timeout elapses.
|
||||
|
||||
#### Scenario: Condition becomes true before timeout
|
||||
- **WHEN** a wait-for-condition step's condition evaluates true within its configured timeout
|
||||
- **THEN** the system stops polling, records the step as succeeded, and advances the run to the next step
|
||||
|
||||
#### Scenario: Condition never becomes true before timeout
|
||||
- **WHEN** a wait-for-condition step's condition has not evaluated true by its configured timeout
|
||||
- **THEN** the system records the step as failed with a timeout reason and the run's status becomes `failed` unless a branch step defines an alternate path
|
||||
|
||||
### Requirement: Branch step selects the next step from a condition
|
||||
The system SHALL execute a branch step by evaluating its named condition and setting the run's next active step to the branch's configured true-target step id or false-target step id accordingly, instead of advancing to the next step in definition order.
|
||||
|
||||
#### Scenario: Branch condition true selects the true-target step
|
||||
- **WHEN** a branch step's condition evaluates true
|
||||
- **THEN** the system sets the run's current step to the branch's `on_true` target step id
|
||||
|
||||
#### Scenario: Branch condition false selects the false-target step
|
||||
- **WHEN** a branch step's condition evaluates false
|
||||
- **THEN** the system sets the run's current step to the branch's `on_false` target step id
|
||||
|
||||
### Requirement: Condition kinds are pluggable via a registry
|
||||
The system SHALL evaluate wait-for-condition and branch step conditions through a registry mapping a condition kind name to an evaluator, SHALL provide at least `scene_contains_text`, `world_variable_equals`, `elapsed_seconds`, and `step_result_success` as built-in kinds, and SHALL allow a new condition kind to be added without modifying the workflow runner's step-dispatch logic.
|
||||
|
||||
#### Scenario: Built-in condition kind evaluates against current state
|
||||
- **WHEN** a wait-for-condition or branch step specifies the `scene_contains_text` kind with a target text value
|
||||
- **THEN** the system evaluates the condition against the most recently observed scene and returns true only when the target text is present
|
||||
|
||||
#### Scenario: Unregistered condition kind fails the step
|
||||
- **WHEN** a step specifies a condition kind that is not present in the registry
|
||||
- **THEN** the system fails that step with an unrecognized-condition-kind reason instead of executing an undefined check
|
||||
|
||||
#### Scenario: World-state-dependent condition degrades safely when world state is absent
|
||||
- **WHEN** a `world_variable_equals` condition is evaluated for a run whose task context has no `WorldState` available
|
||||
- **THEN** the system treats the condition as not satisfied rather than raising an error, allowing the step to continue polling until its timeout
|
||||
@@ -0,0 +1,67 @@
|
||||
## 1. Package scaffolding
|
||||
|
||||
- [ ] 1.1 Create the `workflow/` package (`__init__.py`, `models.py`, `conditions.py`, `skill_exec.py`, `runner.py`, `store.py`, `config.py`)
|
||||
- [ ] 1.2 Add `workflow*` to `[tool.setuptools.packages.find].include` in `pyproject.toml` (no new third-party dependency; `sqlite3` is stdlib, already used by `storage/task_metadata.py`)
|
||||
- [ ] 1.3 Add Workflow Runtime configuration in `workflow/config.py`: default poll interval, default wait-step timeout, default `WorkflowStore` db path (`workflows/workflows.sqlite3`)
|
||||
- [ ] 1.4 Extend the project's smoke test (that imports every package) to import `workflow`
|
||||
|
||||
## 2. Workflow and step data model (capability: workflow-orchestration)
|
||||
|
||||
- [ ] 2.1 Implement `workflow/models.py`: `ConditionSpec{kind: str, params: dict}`, the four step dataclasses (`PlannedGoalStep{step_id, goal, next_step_id}`, `SkillInvocationStep{step_id, skill_id, args, next_step_id}`, `WaitForConditionStep{step_id, condition, timeout_seconds, poll_interval_seconds, next_step_id}`, `BranchStep{step_id, condition, on_true, on_false}`), and the `WorkflowStep` union type
|
||||
- [ ] 2.2 Implement `WorkflowDefinition{id, name, steps: list[WorkflowStep], entry_step_id}` with a constructor-time validation pass rejecting duplicate `step_id`s and any `next_step_id`/`on_true`/`on_false`/`entry_step_id` that does not reference an existing step in the same definition
|
||||
- [ ] 2.3 Implement `WorkflowRun{id, definition_id, status, current_step_id, variables: dict, step_results: list[WorkflowStepResult], created_at, updated_at}` and `WorkflowStepResult{step_id, kind, success, detail, task_id: str | None, timestamp}`, with `to_dict()`/`from_dict()` helpers mirroring `core/models.py`'s style
|
||||
- [ ] 2.4 Write unit tests for `WorkflowDefinition` construction: valid heterogeneous-step definition accepted; duplicate `step_id` rejected; dangling `next_step_id`/`on_true`/`on_false`/`entry_step_id` reference rejected
|
||||
|
||||
## 3. WorkflowStore persistence (capability: workflow-orchestration)
|
||||
|
||||
- [ ] 3.1 Implement `workflow/store.py`: `WorkflowStore(db_path)` with schema creation for `workflow_definitions`, `workflow_runs`, `workflow_step_results` tables in its own SQLite file, following `storage/task_metadata.py`'s connect-per-call pattern
|
||||
- [ ] 3.2 Implement `WorkflowStore.save_definition(definition)` / `get_definition(definition_id)`
|
||||
- [ ] 3.3 Implement `WorkflowStore.create_run(definition_id, initial_variables) -> WorkflowRun` (status `pending`/`running`, `current_step_id` set to the definition's `entry_step_id`)
|
||||
- [ ] 3.4 Implement `WorkflowStore.append_step_result(run_id, step_result)` and `WorkflowStore.update_run(run_id, *, status=None, current_step_id=None, variables=None)`, both persisting immediately (no in-memory-only buffering)
|
||||
- [ ] 3.5 Implement `WorkflowStore.get_run(run_id) -> WorkflowRun` reconstructing the full run (status, current step, variables, ordered step results) from persisted rows
|
||||
- [ ] 3.6 Write unit tests for `WorkflowStore`: create/get round-trip, step-result append ordering, run survives being re-opened via a fresh `WorkflowStore(same db_path)` instance (simulating a process restart)
|
||||
|
||||
## 4. ConditionEvaluator registry (capability: workflow-orchestration)
|
||||
|
||||
- [ ] 4.1 Implement `workflow/conditions.py`: `ConditionEvaluator` protocol/ABC with `evaluate(spec, *, scene, world_state) -> bool`, and a registry `dict[str, ConditionEvaluator]`
|
||||
- [ ] 4.2 Implement the `scene_contains_text` evaluator (checks the most recent `Scene`'s elements/text for a target substring from `spec.params`)
|
||||
- [ ] 4.3 Implement the `world_variable_equals` evaluator (compares `world_state.variables.get(spec.params["name"])` to `spec.params["value"]`; returns `False` — never raises — when `world_state` is `None` or the key is absent)
|
||||
- [ ] 4.4 Implement the `elapsed_seconds` evaluator (returns `True` once at least `spec.params["seconds"]` have elapsed since the wait step started polling)
|
||||
- [ ] 4.5 Implement the `step_result_success` evaluator (looks up a prior step's recorded `WorkflowStepResult.success` by `spec.params["step_id"]` from the run's step-result log)
|
||||
- [ ] 4.6 Implement `evaluate_condition(spec, *, scene, world_state) -> bool` as the registry lookup entry point, raising a distinguishable `UnknownConditionKindError` for an unregistered `kind` (caught by the runner and turned into a failed step, not an uncaught exception)
|
||||
- [ ] 4.7 Write unit tests for each built-in evaluator (true case, false case) and for the unregistered-kind error path
|
||||
|
||||
## 5. Skill-invocation execution path (capability: workflow-orchestration)
|
||||
|
||||
- [ ] 5.1 Implement `workflow/skill_exec.py`: `validate_skill_args(skill, args) -> None` raising a descriptive error listing missing required parameters or unrecognized argument keys, called before any tool dispatch
|
||||
- [ ] 5.2 Implement `resolve_skill_steps(skill, args) -> list[dict]` substituting `{param}` placeholders in the skill's stored tool-call template with validated `args` values
|
||||
- [ ] 5.3 Implement `run_flow_template_skill(skill, args) -> list[StepResult]` calling `validate_skill_args`, `resolve_skill_steps`, then dispatching each resolved tool call through the same `tools/*` functions `Executor.execute()` already uses, collecting one `StepResult` per resolved step
|
||||
- [ ] 5.4 Implement the `Skill.kind != "flow_template"` guard: reject with a step-definition error before attempting resolution
|
||||
- [ ] 5.5 Define (or import, if `skill-learning-runtime`/`skill-catalog-subscription` is already implemented) the minimal `Skill`/`FlowTemplateSkill` dataclass shape `skill_exec.py` depends on, with a code comment flagging reconciliation once one of those changes lands
|
||||
- [ ] 5.6 Write unit tests for `skill_exec`: valid-args resolution produces the expected resolved tool-call sequence; missing required parameter fails before any tool call; non-flow-template skill kind is rejected
|
||||
|
||||
## 6. WorkflowRunner orchestration (capability: workflow-orchestration)
|
||||
|
||||
- [ ] 6.1 Implement `workflow/runner.py`: `WorkflowRunner(store, task_runner_factory, ...)` with `run(definition, device_id, initial_variables=None) -> WorkflowRun` that creates a run via `WorkflowStore.create_run()` and drives steps until a terminal status
|
||||
- [ ] 6.2 Implement the planned-goal step handler: construct a `core.models.Task(goal=step.goal, device_id=...)`, call `TaskRunner(...).run(task)`, map `task.status`/`task.failure_reason` to a `WorkflowStepResult`
|
||||
- [ ] 6.3 Implement the skill-invocation step handler: look up the referenced skill by `skill_id`, call `skill_exec.run_flow_template_skill`, map the returned `StepResult`s to one aggregate `WorkflowStepResult`
|
||||
- [ ] 6.4 Implement the wait-for-condition step handler: poll `conditions.evaluate_condition()` at `poll_interval_seconds` until `True` or `timeout_seconds` elapses; record success or a timeout failure
|
||||
- [ ] 6.5 Implement the branch step handler: evaluate the condition once and set the run's next step to `on_true`/`on_false` accordingly, bypassing default sequential advancement
|
||||
- [ ] 6.6 Implement default sequential advancement (no explicit `next_step_id`/branch target) as "the next step in `WorkflowDefinition.steps` order" for non-branch step kinds
|
||||
- [ ] 6.7 After each step handler returns, call `WorkflowStore.append_step_result()` and `WorkflowStore.update_run()` (new `current_step_id`, and `status` if terminal) before advancing — no step is considered "done" until this checkpoint write completes
|
||||
- [ ] 6.8 Implement `WorkflowRunner.resume(run_id) -> WorkflowRun`: load the persisted run via `WorkflowStore.get_run()`, and continue driving from its persisted `current_step_id`, skipping any step already present in the loaded `step_results` log
|
||||
- [ ] 6.9 Implement `resume()` on an already-`completed`/`failed`/`cancelled` run as a no-op that returns the existing run state unchanged
|
||||
- [ ] 6.10 Write unit tests for `WorkflowRunner.run()` covering: a linear planned-goal-only workflow reaching `completed`; a workflow with a failing planned-goal step reaching `failed` with a recorded reason; a skill-invocation step executing resolved tool calls (mocked `tools/*`); a wait step succeeding before timeout and timing out after
|
||||
- [ ] 6.11 Write unit tests for branch step selection (true path and false path) and for default sequential advancement between non-branch steps
|
||||
|
||||
## 7. Resumability end-to-end validation (capability: workflow-orchestration)
|
||||
|
||||
- [ ] 7.1 Write an end-to-end test that runs a multi-step workflow through `WorkflowRunner`, stops after two steps (simulating a crash by discarding the in-memory `WorkflowRunner`/`TaskRunner` instances), constructs a fresh `WorkflowRunner`/`WorkflowStore` pointed at the same db file, and asserts `resume(run_id)` continues from the third step without re-invoking the first two steps' tool calls or delegated tasks
|
||||
- [ ] 7.2 Write an end-to-end test asserting a `WorkflowRun`'s persisted `variables` and `step_results` log are readable and correctly ordered after a full run via `WorkflowStore.get_run()`
|
||||
- [ ] 7.3 Write an end-to-end test combining a branch step with a wait step and a skill-invocation step in one workflow, asserting the run reaches `completed` via the expected branch path
|
||||
|
||||
## 8. Composition safety checks and full-suite validation
|
||||
|
||||
- [ ] 8.1 Confirm no existing file under `runtime/`, `storage/`, `tools/`, or `core/` is modified by this change (composition via import only, per design.md's D1/D8/D9)
|
||||
- [ ] 8.2 Write a test that runs a real (non-mocked) `runtime.task.TaskRunner` instance inside a `WorkflowRunner`-driven planned-goal step, guarding against silent drift in `agent-runtime`'s public `run(task) -> Task` contract this change composes over
|
||||
- [ ] 8.3 Run the full test suite (`pytest`) and confirm every existing test in `tests/` passes unmodified, with only new `tests/test_workflow_*.py`-style files added
|
||||
@@ -0,0 +1,2 @@
|
||||
schema: spec-driven
|
||||
created: 2026-07-06
|
||||
@@ -0,0 +1,73 @@
|
||||
## Context
|
||||
|
||||
`runtime/task.py`'s `TaskRunner.run()` is the Observe→Think→Act→Observe loop (`agent-runtime` capability, `apex-agent-mvp`, code-complete but unapplied): each iteration calls `self.observer(device_id)` to get a `Scene`, appends it to `TaskContext.scenes`, calls `self.planner.plan(goal=..., scene=..., context=...)`, executes each `PlannedStep` via `Executor.execute()`, and appends the `StepResult` to `TaskContext.step_results`. `TaskContext` (`runtime/context.py`) is a flat dataclass — `scenes: list[Scene]`, `step_results: list[StepResult]` — with no derived/summarized state; `context.latest_scene` is the only convenience accessor today. `semantic-scene-runtime` (Milestone 5, drafted alongside this change) adds `enrich_scene(scene) -> SemanticScene | None`, a same-step, non-persisted artifact — its own design.md explicitly leaves open "should `SemanticScene` be recorded in `TaskContext`/the timeline" as a question for this milestone to answer. Today's `Planner` is a two-step stub (`runtime/planner.py`) that emits one hardcoded `describe_screen` step and declares the goal reached once any step has succeeded — it does not yet reason about page identity at all, so nothing currently consumes anything like "have I already navigated here." A future LLM-driven Planner is the actual reason this capability needs to exist: without a persisted, bounded summary of "where am I / what have I already established," a real Planner has no way to skip re-navigating to a screen it visited two steps ago except by re-scanning the entire `scenes`/`step_results` history and re-deriving semantics itself, every single step.
|
||||
|
||||
Two stakeholders: `runtime/task.py`'s `TaskRunner` (must gain a well-defined, optional integration point without changing behavior when `WorldModel` is not configured), and a future LLM-driven Planner (not built in this change) that will be the actual consumer reading `WorldState` to make cheaper plans.
|
||||
|
||||
## Goals / Non-Goals
|
||||
|
||||
**Goals:**
|
||||
- Maintain one `WorldState` per task run — `current_app`, `current_page`, a `variables` dict, and a bounded `history` of recent semantic-scene/action pairs — updated incrementally after each executed step.
|
||||
- Make the update purely a **derivation over data the loop already produces** (the just-observed `Scene`, the optional `SemanticScene`, the `PlannedStep`, and its `StepResult`) — no new I/O, no new LLM call, no new external dependency introduced by this change.
|
||||
- Keep `history` genuinely bounded (a fixed-size ring buffer) so a long-running task (up to `TaskRunnerConfig.max_steps`, currently 20, but not contractually capped) cannot grow `WorldState` without limit.
|
||||
- Expose `WorldState` to the Planner strictly **read-only** and **additively** — `Planner.plan()` gains an optional keyword argument with a safe default, so every existing caller and test keeps working unmodified.
|
||||
- Make `WorldModel` degrade sensibly when `SemanticScene` is absent (semantic enrichment disabled, per `semantic-scene-runtime`'s default-disabled config) — page/app tracking falls back to a raw-`Scene`/action heuristic rather than going stale or raising.
|
||||
|
||||
**Non-Goals:**
|
||||
- No cross-task or cross-device world sharing — `WorldState` is scoped to exactly one `Task`/`TaskContext` and is discarded when the task ends; a follow-on "world persistence" or "world sharing across a device's tasks" capability is a separate future decision, not made here.
|
||||
- No UI for viewing or editing world state — that belongs to `web-console`'s `console-status-api`/`console-config-api`/`web-console-ui` capabilities (a separate pending change), not touched here.
|
||||
- No real LLM-driven Planner — this change only makes `WorldState` available as a Planner input; teaching a Planner to actually *use* it (e.g. skip a navigation step because `current_page` already matches) is future Planner-capability work, out of scope here.
|
||||
- No second LLM call to summarize/update world state — updates are deterministic rules over already-available per-step data, not a new enrichment call (see D2).
|
||||
- No persistence of `WorldState` into `storage/timeline.py` or `task-memory`'s timeline format — `WorldState` lives only in the in-memory `TaskContext` for the duration of a run (see Open Questions).
|
||||
|
||||
## Decisions
|
||||
|
||||
### D1: New `world/` package, sibling to `semantic/`, not folded into `runtime/context.py`
|
||||
`WorldState`/`WorldModel` get their own package (`world/models.py`, `world/model.py`, `world/config.py`) rather than growing `runtime/context.py` in place. `TaskContext` gains only a thin `world: WorldState | None` field pointing at an object `world/` owns and updates; the update *rules* (how `current_page` is derived from a `SemanticScene`, how `history` eviction works) live in `world/model.py`, not in `runtime/`. This mirrors `semantic-scene-runtime`'s own D1 (a sibling package rather than folding into the layer below) and keeps `runtime/` focused on orchestration (loop shape, retries, tool dispatch) rather than accumulating every future kind of derived state inline.
|
||||
**Alternative considered**: add `current_app`/`current_page`/`variables`/`history` fields directly onto `TaskContext` and put the update logic in `runtime/task.py`'s `TaskRunner.run()`. Rejected — `TaskContext` would become a grab-bag mixing raw per-step history (`scenes`, `step_results`, owned by `agent-runtime`) with derived cross-step summarization (owned by this new capability); a future World-related change (persistence, cross-task sharing) would then have to reach into `runtime/` internals instead of a self-contained `world/` package, repeating the exact "grab-bag package" problem `device-agent-runtime-foundation` already diagnosed and fixed for `core/`.
|
||||
|
||||
### D2: Deterministic rule-based update, not a second LLM call
|
||||
`WorldModel.observe(scene, semantic_scene, step, result)` is pure Python: if `semantic_scene` is present, `current_page = semantic_scene.page`; `current_app` is refreshed whenever `step.action in {"launch_app", "terminate_app"}` succeeds, using `step.args["bundle_id"]`/`step.args["app_id"]` (falling back to leaving `current_app` unchanged if the action fails); `variables` are updated only when `step.args` contains an explicit `"remember"` mapping (`{key: value}`) a Planner opted into; `history` appends one `WorldEvent(semantic_scene_or_scene, step.action, result.success)` per call and evicts the oldest entry once past the configured bound. No network call, no additional latency, no new failure mode beyond "field stays stale if inputs are absent."
|
||||
**Alternative considered**: a second structured-output LLM call (reusing `semantic-scene-runtime`'s `llm_client.py` pattern) that looks at the accumulated history and produces a compact world-state summary each step, similar to how `SemanticScene` itself is produced. Rejected for this milestone — doubling the per-step LLM call count (one for `SemanticScene`, one for `WorldState`) doubles latency/cost for a problem (bookkeeping already-known fields) that does not need model reasoning to solve; a rule-based derivation over the same `SemanticScene`/`PlannedStep`/`StepResult` data that already exists is sufficient and strictly cheaper. This can be revisited if a future milestone's Planner needs richer world summarization than simple field-tracking + bounded history provides.
|
||||
|
||||
### D3: `WorldState` exposed via `TaskContext.world`, not a second context object threaded separately
|
||||
`TaskRunner` sets `context.world = world_model.state` once (or updates it in place) rather than introducing a `WorldContext` parameter threaded alongside `TaskContext` through `Planner.plan()`/`Executor.execute()`. `Planner.plan()`'s new `world: WorldState | None = None` keyword is a convenience mirror of `context.world` (some future Planner implementations may prefer an explicit parameter over reaching into `context`), but the source of truth is always `TaskContext.world`.
|
||||
**Alternative considered**: keep `WorldState` entirely outside `TaskContext`, passed as a wholly separate argument to `Planner.plan()`/`Executor.execute()`/`TaskRunner.run()`. Rejected — `agent-runtime`'s existing requirement is "task context/memory available during a run" as one accessible object; splitting per-run state across two parallel objects (`TaskContext` for raw history, a bare `WorldState` for derived summary) makes every future call site that needs "everything about this run" thread two parameters rather than one, for no benefit since `WorldState` is inherently `TaskContext`-scoped (one task, one world) anyway.
|
||||
|
||||
### D4: Bounded `history` via a fixed-size ring buffer, not unbounded list or time-based eviction
|
||||
`WorldState.history: deque[WorldEvent]` is a `collections.deque(maxlen=N)` with `N` from `world/config.py` (default 10), so the oldest event is dropped automatically once the buffer is full — no separate cleanup pass, no unbounded memory growth even if a task runs far beyond `TaskRunnerConfig.max_steps`'s current default of 20 (e.g. if a future change raises that ceiling).
|
||||
**Alternative considered**: keep an unbounded `list[WorldEvent]` (simplest, matches `TaskContext.scenes`/`step_results` which are also unbounded lists today). Rejected — `scenes`/`step_results` are raw historical record (their unboundedness is intentional, mirroring the timeline), whereas `history` here exists specifically to give the Planner a *recent* window without asking it to reason about a growing list; an explicit bound keeps the read-only Planner-facing contract ("recent" is a stated, fixed size) rather than an implicit "whatever the task length happens to be."
|
||||
|
||||
### D5: World Runtime tracking defaults to enabled (unlike Semantic Scene's default-disabled enrichment)
|
||||
`world/config.py`'s enable flag defaults to `True`, and `TaskRunner`'s optional `world_model` argument, when left `None`, still gets a default `WorldModel` constructed internally (not skipped) unless the config flag is explicitly off. This differs from `semantic-scene-runtime`'s enrichment, which defaults to **disabled** specifically because it makes a paid network call every step.
|
||||
**Alternative considered**: default World Runtime to disabled as well, for consistency with the Semantic Scene precedent and to minimize behavior change on apply. Rejected — `WorldModel.observe()` has no network call and no meaningful cost/latency (it is a few dict/deque operations over data the loop already holds), so the reason Semantic Scene defaults off (protect cost/latency-sensitive environments from a silent new LLM bill) does not apply here; defaulting on means the capability's read-only benefit is available to a Planner immediately, and an explicit off-switch remains for anyone who wants to opt out (e.g. a test asserting exact `TaskContext` shape pre-this-change).
|
||||
|
||||
### D6: `Planner.plan()` gains an optional `world` kwarg with a safe default; the existing stub `Planner` ignores it
|
||||
`runtime/planner.py`'s `Planner.plan(self, *, goal, scene, context, world=None)` accepts but does not use `world` — the existing stub planner's behavior (one hardcoded `describe_screen` step, then done) is unchanged. This is intentionally a no-op integration point in this change: making a Planner *smart enough* to skip steps using `WorldState` is real LLM-Planner work reserved for a future milestone/change, not implied by adding the parameter.
|
||||
**Alternative considered**: leave `Planner.plan()`'s signature untouched and require future Planner implementations to read `context.world` directly instead of a dedicated parameter. Rejected — an explicit `world` parameter documents, at the call boundary, that world-state read access is a first-class, expected part of planning (matching how `scene`/`context` are already explicit parameters rather than everything being reached off one context blob), which matters for future callers/tests constructing a `Planner` subclass against a stable, self-documenting signature.
|
||||
|
||||
## Risks / Trade-offs
|
||||
|
||||
- **[Risk]** A rule-based `current_app`/`current_page` derivation can drift from reality faster than an LLM-based one would (e.g. a page transition not captured by any `launch_app`/`terminate_app` action, or `SemanticScene.page` phrased inconsistently step to step since `semantic-scene-runtime`'s Open Questions leave intents/page labels open-vocabulary) → **Mitigation**: `WorldState` is documented and enforced as **read-only, advisory** context for the Planner, never a source of truth the Executor trusts blindly; a future Planner consuming it is expected to still verify against the current `Scene`/`SemanticScene` before skipping a step, not skip purely on stale `WorldState` says-so. This change does not build that Planner logic, only the state it would read.
|
||||
- **[Risk]** Enabling World Runtime by default (D5) means every existing `TaskRunner` caller/test that doesn't explicitly configure `world_model=None` starts accumulating `WorldState` it previously didn't have → **Mitigation**: this is additive-only (`TaskContext.world` is a new field with no effect on `scenes`/`step_results`/existing assertions), the update hook cannot raise (all derivation is defensive: missing `SemanticScene`, missing `step.args` keys, etc. are all "no-op, leave field unchanged," never an exception), and `world/config.py` provides an explicit off-switch for any caller/test that wants exact pre-this-change `TaskContext` shape.
|
||||
- **[Risk]** `variables`'s `step.args["remember"]` convention has no schema/validation — a Planner could stuff arbitrary or unbounded data into `variables` over many steps → **Mitigation**: out of scope to solve generally in this change (no Planner exists yet that writes to it); `world/config.py` can later gain a max-`variables`-size guard if a real Planner's usage pattern demands it, but there is no real caller to validate against yet, so adding a limit now would be speculative.
|
||||
- **[Trade-off]** `WorldState` is not persisted (Non-Goal) — if a task fails partway and is retried, or if `task-memory`'s timeline is later replayed, the accumulated world state from the failed run is not recoverable, only re-derivable from scratch on the retry → acceptable per this milestone's explicit scope (single task's world only); persistence is a `task-memory`-capability change to propose separately if needed, matching how `semantic-scene-runtime` deferred the same question for its own artifact.
|
||||
- **[Trade-off]** Choosing rule-based derivation (D2) over LLM-based world summarization (rejected alternative) means `WorldState`'s quality is bounded by how good `SemanticScene.page`/action-name heuristics are — if a future milestone finds this insufficient (e.g. genuinely needs to reason "the user probably navigated back," not just track the last known page), upgrading to an LLM-based summarizer is a `world/model.py`-internal change, not a `WorldState` shape change, since the shape (`current_app`/`current_page`/`variables`/`history`) is intentionally derivation-method-agnostic.
|
||||
|
||||
## Migration Plan
|
||||
|
||||
This is purely additive with optional, defensively-defaulted integration points:
|
||||
1. Add the `world/` package (`models.py`: `WorldState`, `WorldEvent`; `model.py`: `WorldModel.observe()`; `config.py`: enable flag + history size).
|
||||
2. Add `TaskContext.world: WorldState | None = None` to `runtime/context.py`.
|
||||
3. Add an optional `world_model: WorldModel | None` constructor argument to `TaskRunner` (`runtime/task.py`); when `None` and World Runtime is enabled in config, `TaskRunner` constructs a default `WorldModel` internally; call `world_model.observe(scene, semantic_scene, step, result)` once per executed step, immediately after the existing `context.add_step_result(result)` line, and set/refresh `context.world` from the model's current state.
|
||||
4. Add the optional `world: WorldState | None = None` keyword argument to `Planner.plan()` (`runtime/planner.py`); `TaskRunner` passes `context.world` through; the existing stub `Planner` implementation ignores it (no behavior change).
|
||||
5. Add `world*` to `pyproject.toml`'s `[tool.setuptools.packages.find].include` list; no new third-party dependency (no LLM SDK, no new I/O library — `collections.deque` is stdlib).
|
||||
6. Run `pytest` — the full existing suite must remain green with zero test-content changes (only additive assertions in new `tests/test_world_model.py`-style tests are expected, not edits to existing tests), confirming the default-enabled World Runtime tracking does not alter any existing task-completion/failure behavior.
|
||||
7. Rollback: since this is a net-new package plus two small additive fields/kwargs with safe defaults, reverting is `git revert` of the commit(s); no data migration, no persisted format touched, no external system involved.
|
||||
|
||||
## Open Questions
|
||||
|
||||
- Whether `WorldState` should eventually be persisted into `storage/timeline.py`'s entries (so a replayed/inspected task shows what the runtime "believed" at each step, not just the raw `Scene`/tool-call/result) — left open per this change's Non-Goals; would be a `task-memory`-capability change to propose separately once there is a concrete consumer (e.g. `web-console`'s status API wanting to show "current believed page").
|
||||
- Whether `variables` needs any validation, size bound, or namespacing convention once a real Planner starts writing to it via the `"remember"` args convention — deferred until a real Planner (a future milestone) exists to observe actual usage patterns against.
|
||||
- Whether a future Skill-synthesis milestone would want `WorldState.history`'s bounded window to be configurable per-task rather than one global default — left as a global-default-only setting for this milestone, matching `semantic-scene-runtime`'s equivalent "leaning global-default-only" stance on its own model-selection config.
|
||||
- Whether `current_page` derivation should fall back to something richer than "last known value, unchanged" when `SemanticScene` is absent (e.g. a raw-`Scene`-shape heuristic keyed on element text/layout signature) — this change only specifies the fallback as "leave unchanged," which is safe but potentially stale; worth revisiting once semantic enrichment's real-world enable rate is known.
|
||||
@@ -0,0 +1,28 @@
|
||||
## Why
|
||||
|
||||
`agent-runtime`'s `TaskRunner` (from `apex-agent-mvp`, code-complete but unapplied) re-derives everything about a task from scratch every step: `TaskContext` accumulates a flat list of `Scene`s and `StepResult`s, but nothing in the loop distills "what app am I in," "what page am I on," "am I already logged in," or "did I already navigate into the chat with Zhang San" into a queryable form. Even with `semantic-scene-runtime`'s per-step `SemanticScene` (page identity, intents, widget purposes), that artifact is deliberately ephemeral and same-step only — it is discarded, not accumulated, so a real Planner (a later milestone) still cannot ask "have I already done this" without re-scanning raw scene/step history itself. This change adds a **World Runtime**: a `WorldState` that persists across a task's steps (current app, current page, a small variables dict, and a bounded history of recent semantic scenes/actions), updated incrementally by a hook in `TaskRunner`'s step loop after each executed step, and exposed read-only to the Planner alongside the current `SemanticScene` so plans can skip redundant navigation or re-discovery. This is Milestone 6 (World) of the device-agnostic runtime roadmap established by `device-agent-runtime-foundation`, sitting directly on top of Milestone 5's `semantic-scene` capability.
|
||||
|
||||
## What Changes
|
||||
|
||||
- Add a new `world/` package that defines a `WorldState` dataclass (`current_app: str | None`, `current_page: str | None`, `variables: dict[str, Any]`, a bounded `history: deque[WorldEvent]` of recent `(semantic_scene | scene, action)` pairs) and a `WorldModel` that owns one `WorldState` per task and knows how to update it.
|
||||
- Introduce an **update hook** (`WorldModel.observe(scene, semantic_scene, step, result)`) called once per executed step from `runtime/task.py`'s `TaskRunner.run()` loop, after `context.add_step_result(result)`, so `WorldState` is derived incrementally from exactly the same per-step data the timeline already records — no new I/O, no new LLM call in this change.
|
||||
- Define the **update rule set** as a small, deterministic, rule-based derivation (not a second LLM call): app/page fields are refreshed from the current `SemanticScene.page` (falling back to a heuristic derived from the raw `Scene`/last `launch_app` action when semantic enrichment is disabled or unavailable), `variables` are updated only via an explicit `PlannedStep.args["remember"]` convention a Planner can opt into, and `history` is a fixed-size ring buffer (bounded, oldest evicted first) so `WorldState` cannot grow unboundedly across a long-running task.
|
||||
- Expose `WorldState` **read-only** to the Planner: extend `Planner.plan()`'s call signature with an optional `world: WorldState | None` keyword argument (default `None`, so the existing stub `Planner` and any test constructing `PlannedStep`s directly keep working unmodified) that a future LLM-driven Planner (not built in this change) can read to decide "already there, skip this step."
|
||||
- Add `TaskContext.world` (a `WorldState | None` field, populated by `TaskRunner` when a `WorldModel` is configured) so a single object continues to carry all per-run state the Planner/Executor need, matching `agent-runtime`'s existing "task context/memory available during a run" requirement instead of introducing a second parallel context object.
|
||||
- Add configuration to enable/disable World Runtime tracking globally (default **enabled**, since this is a pure derivation over data the loop already produces — unlike `semantic-scene`'s LLM call, there is no cost/latency reason to default it off) and to size the bounded history (default a small fixed window, e.g. 10 events).
|
||||
- **BREAKING**: none. `WorldState`/`WorldModel` are additive; `TaskContext.world` defaults to `None` when no `WorldModel` is configured, `Planner.plan()`'s new `world` kwarg defaults to `None`, and `TaskRunner`'s constructor accepts an optional `world_model` with `None` preserving today's behavior exactly.
|
||||
|
||||
## Capabilities
|
||||
|
||||
### New Capabilities
|
||||
- `world-model`: A `WorldState` store (current app, current page, a variables dict, and a bounded history of recent semantic-scene/action pairs) for a single task, updated by a hook in the agent runtime's step loop after each executed step, and exposed read-only to the Planner as additional context alongside the current `SemanticScene`, so plans can skip redundant navigation or re-discovery.
|
||||
|
||||
### Modified Capabilities
|
||||
(none — `agent-runtime`'s Observe→Think→Act→Observe loop shape and `semantic-scene`'s `SemanticScene` output are read-only inputs to this change; neither capability's existing requirements are altered. `agent-runtime`'s "task context/memory available during a run" requirement is extended in spirit — `TaskContext` gains a `world` field — but this change does not itself modify `agent-runtime`'s spec deltas since `openspec/specs/` has no applied baseline for it yet; see Impact.)
|
||||
|
||||
## Impact
|
||||
|
||||
- **New package**: `world/` — `models.py` (`WorldState`, `WorldEvent` dataclasses), `model.py` (`WorldModel`, the per-task owner + `observe()` update hook + rule-based derivation logic), `config.py` (enable/disable + history-size settings).
|
||||
- **Modified**: `runtime/context.py` (`TaskContext` gains a `world: WorldState | None = None` field); `runtime/task.py` (`TaskRunner` gains an optional `world_model: WorldModel | None` constructor argument, calls `world_model.observe(...)` once per executed step inside the existing loop, and passes `context.world` into `self.planner.plan(...)`); `runtime/planner.py` (`Planner.plan()` gains an optional `world: WorldState | None = None` keyword argument, unused by today's stub `Planner` but available to a future LLM-driven Planner).
|
||||
- **No change** to `core/models.py`'s `Scene`/`Task`/`Step`, `semantic/`'s `SemanticScene` shape or enrichment logic, `storage/timeline.py`'s persisted format, or any existing tool signature — `WorldState` is derived in-memory per task run and is not persisted by this change (see `design.md` Open Questions for whether a later change should persist it).
|
||||
- **Out of scope**: no cross-task or cross-device world sharing (single task's world only, discarded when the task ends); no UI for viewing/editing world state (that is `web-console`'s domain, not touched here); no change to `skill-catalog-subscription`'s or `web-console`'s pending capabilities.
|
||||
@@ -0,0 +1,86 @@
|
||||
## ADDED Requirements
|
||||
|
||||
### Requirement: Persistent per-task WorldState
|
||||
The system SHALL maintain one `WorldState` per task, consisting of a current app identifier, a current page identifier, a `variables` mapping, and a bounded history of recent semantic-scene/action pairs, that persists across the task's steps rather than being re-derived from scratch each step.
|
||||
|
||||
#### Scenario: WorldState survives across steps within a task
|
||||
- **WHEN** a task executes multiple steps in sequence
|
||||
- **THEN** the `WorldState` object associated with the task is the same object (or reflects continuously accumulated updates) across those steps, not reset between steps
|
||||
|
||||
#### Scenario: WorldState is scoped to a single task
|
||||
- **WHEN** two different tasks run (sequentially or concurrently) against the same or different devices
|
||||
- **THEN** each task has its own independent `WorldState`, and neither task's `WorldState` reflects the other task's app/page/variables/history
|
||||
|
||||
### Requirement: Incremental update after each executed step
|
||||
The system SHALL update a task's `WorldState` via a hook invoked once per executed step in the agent runtime's step loop, deriving the update from the step's observed `Scene`, optional `SemanticScene`, the `PlannedStep` that was executed, and its `StepResult`, without introducing a new LLM call or new network I/O.
|
||||
|
||||
#### Scenario: Update runs after a successful step
|
||||
- **WHEN** the agent runtime's step loop executes a step and records a successful `StepResult`
|
||||
- **THEN** the task's `WorldState` update hook is invoked with that step's `Scene`, `SemanticScene` (if any), the executed `PlannedStep`, and the `StepResult`, and updates the persisted `WorldState` accordingly
|
||||
|
||||
#### Scenario: Update runs after a failed step
|
||||
- **WHEN** the agent runtime's step loop executes a step and records a failed `StepResult`
|
||||
- **THEN** the task's `WorldState` update hook is still invoked with that step's data, and the update proceeds without raising an exception or blocking the loop's continuation/failure handling
|
||||
|
||||
#### Scenario: Update derivation makes no external calls
|
||||
- **WHEN** the `WorldState` update hook runs for any step
|
||||
- **THEN** the update completes using only the data already passed into the hook, without making any LLM call or other network request
|
||||
|
||||
### Requirement: Current app and page tracking
|
||||
The system SHALL derive and refresh `current_app` from successful app-lifecycle actions (e.g. `launch_app`, `terminate_app`) and SHALL derive and refresh `current_page` from the current step's `SemanticScene` page identity when a `SemanticScene` is available, leaving each field unchanged when no corresponding signal is present in a given step.
|
||||
|
||||
#### Scenario: Launching an app updates current_app
|
||||
- **WHEN** a step executes a successful `launch_app` action naming an app/bundle identifier
|
||||
- **THEN** the task's `WorldState.current_app` is updated to that identifier
|
||||
|
||||
#### Scenario: A page identity from SemanticScene updates current_page
|
||||
- **WHEN** a step's enrichment produces a `SemanticScene` with a non-empty `page` value
|
||||
- **THEN** the task's `WorldState.current_page` is updated to that `page` value
|
||||
|
||||
#### Scenario: No page signal leaves current_page unchanged
|
||||
- **WHEN** a step has no `SemanticScene` available (enrichment disabled, unavailable, or failed for that step)
|
||||
- **THEN** the task's `WorldState.current_page` retains its previous value rather than being cleared or set to an empty/placeholder value
|
||||
|
||||
### Requirement: Explicit variable memorization
|
||||
The system SHALL update `WorldState.variables` only when a step's `PlannedStep.args` contains an explicit memorization instruction, and SHALL NOT infer or write arbitrary variables from step content otherwise.
|
||||
|
||||
#### Scenario: A step explicitly remembers a value
|
||||
- **WHEN** a step's `PlannedStep.args` includes an explicit key/value pair designated for memorization
|
||||
- **THEN** the task's `WorldState.variables` is updated to include that key/value pair
|
||||
|
||||
#### Scenario: A step without a memorization instruction does not change variables
|
||||
- **WHEN** a step's `PlannedStep.args` contains no explicit memorization instruction
|
||||
- **THEN** the task's `WorldState.variables` is left unchanged by that step's update
|
||||
|
||||
### Requirement: Bounded history of recent scene/action pairs
|
||||
The system SHALL maintain `WorldState.history` as a fixed-size, bounded collection of the most recent semantic-scene-or-scene/action pairs, automatically evicting the oldest entry when a new entry is added past the configured bound, so that history size never grows unboundedly with task length.
|
||||
|
||||
#### Scenario: History accumulates recent entries up to the bound
|
||||
- **WHEN** a task executes a number of steps less than or equal to the configured history bound
|
||||
- **THEN** `WorldState.history` contains one entry per executed step, in order from oldest to newest
|
||||
|
||||
#### Scenario: History evicts the oldest entry once the bound is exceeded
|
||||
- **WHEN** a task executes more steps than the configured history bound
|
||||
- **THEN** `WorldState.history` retains only the most recent entries up to the bound, with earlier entries evicted, and never exceeds the configured bound in length
|
||||
|
||||
### Requirement: Read-only WorldState available to the Planner
|
||||
The system SHALL expose a task's current `WorldState` to the Planner as an additional, read-only input alongside the current `SemanticScene`/`Scene`, without requiring existing Planner implementations or call sites to change to keep working.
|
||||
|
||||
#### Scenario: Planner can read current WorldState
|
||||
- **WHEN** the agent runtime invokes the Planner to produce the next steps for a task
|
||||
- **THEN** the Planner is given access to the task's current `WorldState` (current app, current page, variables, bounded history) as of the most recently completed step
|
||||
|
||||
#### Scenario: Existing Planner call sites keep working unmodified
|
||||
- **WHEN** an existing caller invokes the Planner's planning entry point without passing any world-state argument
|
||||
- **THEN** the call succeeds exactly as it did before this capability existed, with the Planner treating the absence of world-state input as equivalent to "no world state available"
|
||||
|
||||
### Requirement: World Runtime tracking failure never blocks the task loop
|
||||
The system SHALL treat any failure or unavailable input during a `WorldState` update (e.g. missing `SemanticScene`, missing expected `PlannedStep` arguments, disabled configuration) as non-fatal, leaving the affected `WorldState` fields unchanged rather than raising an exception that would interrupt the agent runtime's step loop.
|
||||
|
||||
#### Scenario: Missing expected data during update does not raise
|
||||
- **WHEN** the `WorldState` update hook runs for a step whose data lacks a field an update rule expects (e.g. no app identifier on a `launch_app` step)
|
||||
- **THEN** the update hook completes without raising, leaving the corresponding `WorldState` field at its prior value
|
||||
|
||||
#### Scenario: World Runtime tracking disabled by configuration
|
||||
- **WHEN** World Runtime tracking is disabled in configuration for a task run
|
||||
- **THEN** the agent runtime's step loop proceeds normally without invoking the `WorldState` update hook, and the Planner receives an absence of world-state input rather than a partially-updated or stale `WorldState`
|
||||
@@ -0,0 +1,44 @@
|
||||
## 1. Package scaffolding
|
||||
|
||||
- [x] 1.1 Create the `world/` package (`__init__.py`, `models.py`, `model.py`, `config.py`)
|
||||
- [x] 1.2 Add `world*` to `[tool.setuptools.packages.find].include` in `pyproject.toml` (no new third-party dependency)
|
||||
- [x] 1.3 Add World Runtime configuration in `world/config.py`: an enabled/disabled flag (default enabled) and a history-size bound (default 10), sourced from environment/config in one place `world/` reads from
|
||||
- [x] 1.4 Extend the project's smoke test (that imports every package) to import `world`
|
||||
|
||||
## 2. WorldState data model (capability: world-model)
|
||||
|
||||
- [x] 2.1 Implement `world/models.py`: `WorldEvent` (`scene_summary: SemanticScene | Scene`, `action: str`, `success: bool`, `timestamp: datetime`) and `WorldState` (`current_app: str | None`, `current_page: str | None`, `variables: dict[str, Any]`, `history: deque[WorldEvent]`) dataclasses, with `to_dict()` mirroring the style of `core/models.py` (for future inspection/debugging use, not persistence in this change)
|
||||
- [x] 2.2 Implement `WorldState.history` as a `collections.deque(maxlen=<configured bound>)` so oldest entries are evicted automatically once the bound is exceeded
|
||||
- [x] 2.3 Write unit tests for `WorldState`/`WorldEvent` construction and for `history`'s bounded-eviction behavior (append past the configured `maxlen` and assert the oldest entry is gone, length stays at the bound)
|
||||
|
||||
## 3. WorldModel update hook (capability: world-model)
|
||||
|
||||
- [x] 3.1 Implement `world/model.py`: `WorldModel` owning one `WorldState` per task, with `observe(scene: Scene, semantic_scene: SemanticScene | None, step: PlannedStep, result: StepResult) -> None` as the single update entry point
|
||||
- [x] 3.2 Implement the `current_page` update rule: set `current_page = semantic_scene.page` when `semantic_scene` is not `None` and its `page` is non-empty; leave unchanged otherwise
|
||||
- [x] 3.3 Implement the `current_app` update rule: on a successful `StepResult` for `step.action in {"launch_app", "terminate_app"}`, set/clear `current_app` from `step.args` (e.g. `bundle_id`/`app_id`); leave unchanged for any other action or a failed result
|
||||
- [x] 3.4 Implement the `variables` update rule: merge `step.args["remember"]` (a `dict`) into `WorldState.variables` when present; leave `variables` unchanged when absent
|
||||
- [x] 3.5 Implement the `history` update rule: append one `WorldEvent` per call to `observe()`, using `semantic_scene` when available and falling back to `scene` otherwise
|
||||
- [x] 3.6 Make every update rule defensive: missing/malformed expected fields (e.g. no `bundle_id` on a `launch_app` step, non-dict `remember` value) are logged and skipped, never raised, so `observe()` never raises for any input shape
|
||||
- [x] 3.7 Write unit tests for `WorldModel.observe()` covering: page update from `SemanticScene`, page unchanged when `semantic_scene` is `None`, app update on successful `launch_app`, app unchanged on failed `launch_app` or unrelated actions, `variables` merge via `remember`, `variables` unchanged without `remember`, and history append/eviction across repeated `observe()` calls
|
||||
|
||||
## 4. TaskContext and TaskRunner integration (capability: world-model)
|
||||
|
||||
- [x] 4.1 Add `world: WorldState | None = None` field to `TaskContext` in `runtime/context.py`
|
||||
- [x] 4.2 Add an optional `world_model: WorldModel | None = None` constructor argument to `TaskRunner` in `runtime/task.py`; when `None` and World Runtime is enabled in config, construct a default `WorldModel` internally; when World Runtime is disabled in config, leave `context.world` as `None` and skip the update hook entirely
|
||||
- [x] 4.3 In `TaskRunner.run()`'s step loop, call `world_model.observe(scene, semantic_scene, step, result)` once per executed step, immediately after the existing `context.add_step_result(result)` line, and refresh `context.world` from the model's current `WorldState`
|
||||
- [x] 4.4 Confirm `TaskRunner.run()`'s existing control flow (retry/failure/max-steps handling) is unaffected by the new hook call — the hook must never change whether a step is treated as success/failure
|
||||
- [x] 4.5 Write unit tests for `TaskRunner` covering: `context.world` populated after a step when World Runtime is enabled (default), `context.world` remaining `None` when explicitly disabled via config, and a task run completing normally (unchanged pass/fail outcome) whether World Runtime is enabled or disabled
|
||||
|
||||
## 5. Planner integration (capability: world-model)
|
||||
|
||||
- [x] 5.1 Add an optional `world: WorldState | None = None` keyword argument to `Planner.plan()` in `runtime/planner.py`; the existing stub `Planner` implementation accepts but does not use it
|
||||
- [x] 5.2 Update `TaskRunner.run()`'s call to `self.planner.plan(...)` to pass `world=context.world`
|
||||
- [x] 5.3 Write a unit test asserting the existing stub `Planner.plan()` call sites (with and without a `world` argument) both continue to return the same steps as before this change
|
||||
- [x] 5.4 Write a unit test asserting `TaskRunner` passes the current `context.world` into `Planner.plan()`'s `world` argument by the second step of a multi-step task (using a custom test `Planner` subclass that records the `world` value it was given)
|
||||
|
||||
## 6. End-to-end validation
|
||||
|
||||
- [x] 6.1 Write an end-to-end test running a multi-step task through `TaskRunner` with a mocked `Driver`/`Scene`/`SemanticScene` sequence, asserting `WorldState.current_app`/`current_page`/`history` reflect the expected values after each step
|
||||
- [x] 6.2 Write an end-to-end test confirming World Runtime tracking failure modes (missing `SemanticScene`, missing expected step args) never fail or interrupt the task loop, only leave the corresponding `WorldState` field unchanged
|
||||
- [x] 6.3 Confirm World Runtime tracking is enabled by default after applying this change, and that disabling it via config fully restores pre-this-change `TaskContext`/`Planner.plan()` call behavior (no `world` state populated or passed)
|
||||
- [x] 6.4 Run the full test suite (`pytest`) and confirm no existing test in `tests/` needed a behavior change, only additive new tests
|
||||
Reference in New Issue
Block a user