feat: checkpoint device agent runtime milestones
This commit is contained in:
@@ -0,0 +1,2 @@
|
||||
schema: spec-driven
|
||||
created: 2026-07-06
|
||||
@@ -0,0 +1,94 @@
|
||||
## Context
|
||||
|
||||
Every prior milestone — `device-management`/`driver-registry` (`apex-agent-mvp`, `device-agent-runtime-foundation`), `agent-runtime`, `semantic-scene`, `world-model`, `skill-authoring`/`skill-versioning`/`skill-embedding-retrieval`, `workflow-orchestration`, `multi-agent-collaboration` — is designed and (mostly) built around one process: one `DeviceManager` instance holding an in-memory dict of devices, one `TaskRunner`/`WorkflowRunner` executing on whichever `device_id` the caller passes in, and one `driver/registry.py` extension point (`SUPPORTED_DRIVER_TYPES`) that a caller reads to build a driver by name. That single-process model is a deliberate, explicit non-goal-turned-goal in `apex-agent-mvp/design.md` ("No production task queue ... synchronous, single-process execution is sufficient for the MVP") and every later milestone has respected it rather than re-litigating it — `workflow-orchestration-runtime`'s own Non-Goals state plainly: "No distributed/multi-device workflow execution or cross-node coordination — that is Milestone 10 (Cloud Runtime)."
|
||||
|
||||
This change is that milestone. Three things are genuinely new here that no prior milestone had to solve: (1) device state now spans more than one host process, so "list devices" and "is this device idle" can no longer be answered by one in-memory dict; (2) a caller submitting a task no longer names a `device_id` up front — something has to *pick* one, from a fleet, based on constraints; (3) new drivers/tools/skills should be addable without editing this repository's packages at all, which the existing "add a dict entry to `driver/registry.py`" extension point does not yet support for a party who cannot modify the repo.
|
||||
|
||||
Two stakeholders: an **external integrator** (a system outside this repo, e.g. a customer's backend) who needs a stable, versioned API to submit work and query status without going through the LLM-facing MCP surface or the operator-facing web console; and a **fleet operator** who runs more than one host process (each with its own `DeviceManager` and set of physically-attached devices) and needs one place to see "how many devices are idle across all of them" and have tasks land on whichever is free.
|
||||
|
||||
## Goals / Non-Goals
|
||||
|
||||
**Goals:**
|
||||
- Define an internal `DevicePool` model that aggregates device state from potentially many hosts' independent `DeviceManager` instances, degrading gracefully (marking a host's devices `unreachable`, never crashing or blocking) when a host stops reporting in.
|
||||
- Define a `TaskScheduler` that accepts task submissions with optional device constraints and assigns them to an idle, matching `PooledDevice`, via a pluggable, registrable `AssignmentStrategy` — the same "string key → swappable implementation" shape as `driver/registry.py`'s `SUPPORTED_DRIVER_TYPES` and `workflow-orchestration-runtime`'s `ConditionEvaluator` registry.
|
||||
- Define a `TaskDispatcher` that actually executes a scheduler assignment by composing the existing `agent-runtime`/`workflow-orchestration` entry points — so device selection becomes automatic without duplicating any execution logic those capabilities already own.
|
||||
- Define a `PluginManifest`/`PluginRegistry` mechanism so a new driver (and, in schema if not yet in wiring, a new tool or skill) can register itself via a manifest file or Python entry point, without editing `driver/`, `device/`, `tools/`, or the skill stores.
|
||||
- Define a versioned, public `platform-sdk` (REST API + Python client) for external integrators, clearly scoped apart from `mcp-tool-server` (LLM tool-calling surface) and `console-status-api`/`console-config-api` (single-process operator UI surface).
|
||||
|
||||
**Non-Goals:**
|
||||
- No actual multi-host deployment or infrastructure tooling (no Docker/Kubernetes manifests, no service mesh, no real network transport implementation for host-to-pool sync beyond defining the `sync_host_devices(host_id, snapshot)` call signature this change's local reference implementation exercises in-process). This change defines the runtime's internal data model and control flow for a device pool and scheduler, not ops tooling to actually run one across real machines.
|
||||
- No cross-host **task execution** dispatch over the network: `TaskDispatcher` composes `TaskRunner`/`WorkflowRunner` directly, which only works when the assigned device is owned by the same process the dispatcher runs in. When `DevicePool` reports a `PooledDevice` owned by a *different* host, `TaskDispatcher` raises a clear, typed `RemoteDispatchNotSupportedError` rather than silently no-op'ing or pretending to execute — real cross-host RPC dispatch is left as an Open Question / follow-up, consistent with this change's own "no infra" non-goal.
|
||||
- No billing, multi-tenant authentication/authorization, or rate limiting on `platform-sdk` — a later change's concern; this change may add a single API-key placeholder hook (see D8) but does not implement real auth.
|
||||
- No generic dynamic tool/skill plugin *wiring* — `plugin-system`'s manifest schema accepts `tool`/`skill`-kind entries for forward compatibility, but only `driver`-kind plugins are concretely registered into an existing extension point (`driver-registry`) in this change; `tools/` and the skill stores have no comparable public registration hook yet to compose against.
|
||||
- No changes to `driver/`, `device/`, `runtime/`, `tools/`, `workflow/`, `agents/`, `storage/`, `api/console.py`, or `api/mcp.py` — every integration is by import/composition only.
|
||||
- No replacement of `mcp-tool-server` or `console-status-api`/`console-config-api` — `platform-sdk` is a third, additive surface, not a superset or a router in front of the other two.
|
||||
|
||||
## Decisions
|
||||
|
||||
### D1: New `cloud/` package, sibling to `agents/`/`workflow/`/`semantic/`/`world/`/`skills_learning/`, not folded into `runtime/`, `device/`, or `api/`
|
||||
`cloud/` (`pool.py`, `scheduler.py`, `dispatch.py`, `plugins.py`, `store.py`, `config.py`, `sdk/`) is its own top-level package. This continues the one-package-per-milestone precedent (`agents/` for Milestone 9, `workflow/` for Milestone 8, `semantic/`/`world/`/`skills_learning/` for Milestones 5–7) and keeps `runtime/`'s job (single-goal Observe-Think-Act loop) and `device/`'s job (one process's own device lifecycle) unchanged.
|
||||
- **Alternative considered**: Extend `device/manager.py`'s `DeviceManager` in place to understand multiple hosts. Rejected — `DeviceManager` is a pending, unapplied capability (`device-management`) with no applied baseline in `openspec/specs/` to safely diff a `MODIFIED` delta against in this session (the same reasoning `workflow-orchestration-runtime`'s D1 already applied to `TaskRunner`); treating it as a stable, composed-over dependency keeps "one process's device lifecycle" and "many processes' aggregated device state" as two separate, independently testable concerns, and avoids retroactively changing another change's still-pending contract.
|
||||
- **Alternative considered**: Put the four capabilities in four separate top-level packages (`device_pool/`, `task_scheduler/`, `plugins/`, `platform_sdk/`). Rejected — these four are tightly coupled (the scheduler reads the pool; the SDK exposes both; the dispatcher needs the scheduler's assignment) and the roadmap already names this single milestone "Cloud Runtime"; one package with clear submodules keeps the coupling visible in the import graph instead of scattering it across four independent top-level packages that would need to depend on each other anyway.
|
||||
|
||||
### D2: `DevicePool` aggregates by *push*, not by remotely calling each host's `DeviceManager`
|
||||
`DevicePool.sync_host_devices(host_id, snapshot: list[Device])` is called (by a periodic job, or by `platform-sdk`'s host-registration endpoint) with the *result* of that host's own `DeviceManager.list_devices()` — the pool never reaches out over the network to call a remote `DeviceManager` method itself. `HostRegistration.last_seen_at` updates on every sync call; a background staleness check (or a check performed lazily on read, see D3) marks a host's `PooledDevice`s `unreachable` once `last_seen_at` exceeds `config.stale_after_seconds`.
|
||||
- **Alternative considered**: Give `DevicePool` an RPC client that calls a remote host's `DeviceManager` over HTTP/gRPC directly (pull model). Rejected for this change — building a real inter-host RPC layer is exactly the "actual multi-host deployment" infrastructure work this change's Non-Goals defer; a push-based sync call has a stable, host-agnostic signature today (`sync_host_devices(host_id, snapshot)`) that a later change can wire to any real transport (HTTP call from a lightweight per-host agent, message queue, etc.) without changing `DevicePool`'s internal model.
|
||||
|
||||
### D3: Host staleness is checked lazily on read, not via a background timer thread
|
||||
`DevicePool.list_devices()` and `DevicePool.get_device(device_id)` compute each host's staleness (`now - last_seen_at > stale_after_seconds`) at call time and report `unreachable` status accordingly, rather than running a background thread that mutates stored status on a timer.
|
||||
- **Alternative considered**: Run a background thread/asyncio task that periodically sweeps `HostRegistration`s and flips stale ones to `unreachable` in the store. Rejected — a background thread adds lifecycle management (start/stop with the process, thread-safety with `CloudStore`'s SQLite connections) for a property (staleness) that is cheap to compute at read time from a single stored timestamp; lazy computation is simpler, always consistent with the current clock, and avoids a whole class of "the sweep thread died silently" failure modes for no loss of correctness.
|
||||
|
||||
### D4: `TaskScheduler`'s queue and `AssignmentStrategy` registry mirror the Driver Registry / `ConditionEvaluator` pattern
|
||||
`cloud/scheduler.py` defines `AssignmentStrategy` (one method, `select(task: ScheduledTask, candidates: list[PooledDevice]) -> PooledDevice | None`) and a registry mapping a strategy name (`fifo_match` default, reserved name `capability_score` for a future scoring strategy) to an implementation, the same "string key → pluggable implementation" shape already standardized on by `driver/registry.py` (`device-agent-runtime-foundation` D3), `perception/provider.py` (D8), and `workflow/conditions.py`'s `ConditionEvaluator` (`workflow-orchestration-runtime` D6). `TaskScheduler.submit(goal, constraints)` enqueues a `ScheduledTask{id, goal, constraints, status: queued|assigned|dispatched|done|failed}`; `TaskScheduler.assign()` (called by a poll loop or triggered on submission/device-freed events) pops the head of the FIFO queue, asks `DevicePool.list_devices()` for idle candidates matching `constraints.driver_type`/`constraints.capability_tags`, and calls the configured `AssignmentStrategy.select()`.
|
||||
- **Alternative considered**: Hardcode FIFO-first-idle-match as the only assignment behavior with no strategy abstraction. Rejected — this is precisely the third occurrence of the same "pluggable-by-string-key" problem (driver selection, condition evaluation, now assignment strategy); making it a registry from the start costs one extra indirection and avoids an `if/elif` chain being added later when a capability-scoring strategy is inevitably wanted.
|
||||
|
||||
### D5: `TaskDispatcher` composes `TaskRunner`/`WorkflowRunner` as black boxes; local-only in this change
|
||||
`cloud/dispatch.py`'s `TaskDispatcher.dispatch(assignment: Assignment)` builds a `core.models.Task(goal=assignment.task.goal, device_id=assignment.device_id)` (or, if the submission specified a `workflow_definition_id` instead of a bare `goal`, a `WorkflowDefinition`) and calls the existing, unmodified `runtime.task.TaskRunner(...).run(task)` or `workflow.runner.WorkflowRunner(...).run(definition)` — exactly the composition pattern `workflow-orchestration-runtime`'s D8 already established for `WorkflowRunner` composing `TaskRunner`. If `assignment.host_id` does not match the local process's own host id, `dispatch()` raises `RemoteDispatchNotSupportedError` instead of attempting anything — this change's scheduler can *decide* a remote host should take a task, but only a local dispatch is actually executed here (see Non-Goals and Open Questions).
|
||||
- **Alternative considered**: Build a lower-level dispatch loop that re-implements plan/execute/retry logic inside `cloud/` instead of calling `TaskRunner`/`WorkflowRunner`. Rejected — duplicates already-implemented, already-tested execution logic and creates two places that must stay behaviorally consistent; composing the existing public entry points automatically inherits any future improvement to either runner with zero change to `cloud/`.
|
||||
|
||||
### D6: `PluginManifest`/`PluginRegistry` discovers via `importlib.metadata` entry points and a local manifest-file scan, both feeding one validation+registration path
|
||||
`cloud/plugins.py`'s `PluginManifest` (`name: str`, `version: str`, `entry_point_kind: Literal["driver", "tool", "skill"]`, `target: str` — a dotted module:attribute path) can be produced two ways: (a) a package installed in the environment declares an entry point in group `device_agent_runtime.plugins` whose value is itself a `PluginManifest`-shaped object or a callable returning one; (b) a `plugins/<name>/plugin.json` file on a configured scan path is parsed directly into a `PluginManifest`. Both paths converge on `PluginRegistry.register(manifest)`, which validates shape/uniqueness (no duplicate `name`, `entry_point_kind` recognized) and then, only for `entry_point_kind == "driver"`, resolves `target` to a `DriverFactoryBuilder` callable and calls `driver_registry.register_driver_type(manifest.name, builder)`.
|
||||
- **Alternative considered**: Support only Python entry points (no manifest-file scan), matching a more conventional Python-plugin-ecosystem approach (e.g., `pluggy`). Rejected for this change — requiring every plugin author to publish an installable Python package with an entry point is a real adoption barrier for a first plugin mechanism; a plain `plugin.json` file scan lets a plugin be "drop a folder in `plugins/`" as well, and both paths sharing one validation+registration function means neither is a second, divergent code path to maintain.
|
||||
- **Note (dependency on a not-yet-existing function)**: `driver/registry.py` (as specified by the pending `driver-registry` capability) today only defines `SUPPORTED_DRIVER_TYPES` (a plain dict) and `build_driver_factory()`; it does not yet expose a `register_driver_type(name, builder)` function. This change's driver-plugin registration path is written against that function *by name*, on the expectation that `device-agent-runtime-foundation`'s own stated intent ("adding a new driver type is a one entry addition to `driver/registry.py`") is realized as a small public function, not only as "edit the dict inline." This is called out again in Open Questions since it is a real dependency on another pending change's still-evolving surface, not a formal spec delta this change is making to it.
|
||||
|
||||
### D7: `cloud/store.py`'s `CloudStore` is a new, independently-owned SQLite file, not new tables in `storage/task_metadata.py` or `workflow/store.py`
|
||||
`CloudStore` opens its own file (default `cloud/cloud.sqlite3`) with `host_registrations`, `pooled_devices`, `scheduled_tasks`, and `plugins` tables, using the same connect-per-call `sqlite3` pattern as `storage/task_metadata.py` and `workflow/store.py`.
|
||||
- **Alternative considered**: Add `scheduled_tasks`/`pooled_devices` tables into `workflow/store.py`'s existing `workflows.sqlite3` (since a scheduled task's eventual execution may become a `WorkflowRun`). Rejected — `workflow/` is owned by the pending, unapplied `workflow-orchestration` capability; adding schema/migration code to it without a corresponding spec delta would be a de facto modification of another change's owned artifact, the same reasoning `workflow-orchestration-runtime`'s own D9 already used to justify *not* writing into `storage/task_metadata.py`. A separate, `cloud/`-owned store composed in prose (a `scheduled_tasks` row stores a `task_id`/`workflow_run_id` string reference once dispatched, no foreign key) is fully additive.
|
||||
|
||||
### D8: `platform-sdk`'s REST API is versioned via a URL prefix (`/v1/...`), with a single pluggable `AuthProvider` hook (no-op default), not a full auth system
|
||||
`cloud/sdk/api.py` mounts all routes under `/v1/` (e.g., `POST /v1/tasks`, `GET /v1/tasks/{id}`, `GET /v1/devices`, `GET /v1/hosts`, `GET /v1/plugins`, `POST /v1/plugins`). Every route accepts an optional `AuthProvider.authenticate(request) -> Principal | None` hook (default: a `NullAuthProvider` that always returns an anonymous `Principal`, i.e., no enforcement) so a later change can add real API-key/OAuth checking without changing route signatures.
|
||||
- **Alternative considered**: Ship `platform-sdk` with no versioning prefix at all (bare `/tasks`, `/devices`, ...), deferring versioning until a breaking change is actually needed. Rejected — this is explicitly a *public-facing* surface for external integrators (unlike `console-status-api`, which is first-party/trusted-network only per `web-console`'s own scope); starting unversioned and retrofitting a prefix later would break every existing integrator's URLs on day one of the first breaking change, which a public SDK should not do.
|
||||
- **Alternative considered**: Build real API-key authentication now, since "public-facing" implies untrusted callers. Rejected for this change per its explicit Non-Goal — auth/authz is a distinct, non-trivial concern (key storage, rotation, revocation) that deserves its own change; shipping a no-op hook that is clearly the extension point (mirroring the Driver Registry/`ConditionEvaluator`/`AssignmentStrategy` pattern used everywhere else in this design) is enough to make "add real auth later" a localized change, not a rewrite.
|
||||
|
||||
### D9: `cloud/sdk/client.py` is a thin `requests`/`httpx`-based wrapper, not a code-generated client
|
||||
The Python SDK client (`CloudClient`) is hand-written, wrapping the same `/v1/...` routes `cloud/sdk/api.py` exposes, using `httpx` (already a dev dependency, promoted to a runtime dependency here since the client ships as installable code, not just test tooling).
|
||||
- **Alternative considered**: Generate the client from an OpenAPI spec (FastAPI already produces one) via `openapi-python-client` or similar. Rejected for this change — adds a codegen build step and a new dev-tool dependency for a surface with only ~6 routes; hand-writing a thin wrapper is faster to ship and easier to read/review, and nothing here forecloses generating a client later once the API surface is larger and more churn-prone.
|
||||
|
||||
## Risks / Trade-offs
|
||||
|
||||
- **[Risk]** Push-based host sync (D2) means `DevicePool`'s view of a host's devices is only as fresh as that host's last sync call; a device could go `busy`→`idle` on its host moments after a sync and the pool won't know until the next sync interval → **Mitigation**: `config.sync_interval_seconds` is deliberately short-default and documented as eventually-consistent, not real-time; `TaskScheduler.assign()` re-checks `DevicePool` state immediately before dispatch (not just at submission time) to shrink the staleness window for the assignment decision itself, though a race between "assign" and "actually dispatch" is still possible and is accepted, not solved, in this change.
|
||||
- **[Risk]** `TaskDispatcher`'s local-only execution (D5) means `TaskScheduler` can select a `PooledDevice` owned by a remote host, but `dispatch()` will then raise `RemoteDispatchNotSupportedError` rather than executing it → **Mitigation**: `AssignmentStrategy`'s default (`fifo_match`) is configured, in this change's reference wiring, to only consider `PooledDevice`s whose `host_id` equals the dispatcher's own host id, so the common single-dispatcher-process deployment never hits the error path; multi-host dispatch is explicitly flagged in Open Questions as unimplemented, not silently broken.
|
||||
- **[Risk]** `plugin-system`'s driver-plugin registration path (D6) depends on `driver/registry.py` exposing a `register_driver_type()` function that does not exist yet in the pending `driver-registry` capability's own current spec (only a dict + a lookup function) → **Mitigation**: documented explicitly as a cross-change dependency in D6 and Open Questions rather than silently assumed; `PluginRegistry.register()` for `driver`-kind manifests fails loudly (raises, does not silently no-op) if `driver_registry.register_driver_type` is not importable, so the gap surfaces as an explicit error at plugin-load time, not a silent failure to actually register the driver.
|
||||
- **[Risk]** Accepting `tool`/`skill`-kind plugin manifests in the schema (D6) without wiring them anywhere could mislead a plugin author into thinking registering a tool/skill plugin has an effect → **Mitigation**: `PluginRegistry.register()` for `tool`/`skill`-kind manifests stores the manifest (so it is listable via `platform-sdk`'s `GET /v1/plugins`) but returns a `wired: false` field in its result and logs a clear "accepted, not yet wired to any execution path" message, rather than pretending success.
|
||||
- **[Trade-off]** No real auth on `platform-sdk` (D8) means anything reachable at the API's network address can submit tasks/query status today → **Acceptable** for this change since it is not defining deployment/network topology (Non-Goals) and the `AuthProvider` hook makes adding real enforcement a scoped follow-up, not a redesign.
|
||||
- **[Trade-off]** A fourth independent SQLite file (`cloud/cloud.sqlite3`, alongside `tasks/tasks.sqlite3`, `workflows/workflows.sqlite3`, and any skill-store databases) means fleet-level state, workflow state, and task-memory state are three separate stores with only string-id cross-references, no foreign-key enforcement → **Acceptable**; this mirrors the same trade-off `workflow-orchestration-runtime`'s D9 already accepted for its own store relative to `task-memory`'s, and unifying them is explicitly out of scope for any one milestone to force on another's still-pending schema.
|
||||
|
||||
## Migration Plan
|
||||
|
||||
This is purely additive; no existing module is edited:
|
||||
1. Add the `cloud/` package: `pool.py` (`DevicePool`, `HostRegistration`, `PooledDevice`, `sync_host_devices()`), `scheduler.py` (`TaskScheduler`, `ScheduledTask`, `AssignmentStrategy` registry with `fifo_match`), `dispatch.py` (`TaskDispatcher`, `Assignment`, `RemoteDispatchNotSupportedError`), `plugins.py` (`PluginManifest`, `PluginRegistry`, entry-point + manifest-file discovery), `store.py` (`CloudStore`, schema creation for `cloud/cloud.sqlite3`), `config.py` (heartbeat/staleness/queue/strategy/API-version defaults).
|
||||
2. Wire `TaskDispatcher`'s planned-goal path to `runtime.task.TaskRunner(...).run(task)` and its workflow path to `workflow.runner.WorkflowRunner(...).run(definition)` — import only, zero edits to `runtime/` or `workflow/`.
|
||||
3. Wire `PluginRegistry.register()`'s driver-kind path to `driver_registry.register_driver_type(name, builder)` if importable; raise a clear, typed error naming the missing function if `device-agent-runtime-foundation` has not yet landed that function when this change is implemented (see D6's Note and Open Questions) — do not stub around it silently.
|
||||
4. Add `cloud/sdk/`: `models.py` (Pydantic request/response models for the `/v1/...` routes), `api.py` (`create_cloud_router(*, pool, scheduler, plugin_registry) -> APIRouter`, mountable by whatever process assembles the full FastAPI app, same shape as `api/console.py`'s `create_console_router`), `client.py` (`CloudClient`, thin `httpx` wrapper).
|
||||
5. Add `cloud*` to `pyproject.toml`'s `[tool.setuptools.packages.find].include`; promote `httpx` from `[dependency-groups].dev` to `[project].dependencies` (needed at runtime by `cloud/sdk/client.py`).
|
||||
6. Add new tests (`tests/test_device_pool.py`, `tests/test_task_scheduler.py`, `tests/test_task_dispatcher.py`, `tests/test_plugin_registry.py`, `tests/test_cloud_sdk_api.py`-style): multi-host sync + staleness degrade, FIFO assignment with constraint matching, local dispatch composing a stub `TaskRunner`, remote-assignment raising `RemoteDispatchNotSupportedError`, plugin manifest validation + driver registration (with a fake `driver_registry.register_driver_type`), and REST route round-trips using FastAPI's `TestClient`.
|
||||
7. Run `pytest` — the full existing suite stays green with zero edits to existing test files, confirming this change touches no existing behavior.
|
||||
8. Rollback: net-new package plus a net-new SQLite file with its own schema; reverting is `git revert` of the commit(s), with no data migration of any existing store and no external system involved beyond whatever device the composed `TaskRunner`/`WorkflowRunner` calls already target.
|
||||
|
||||
## Open Questions
|
||||
|
||||
- How real cross-host task dispatch should eventually work once `RemoteDispatchNotSupportedError` needs to actually go away — a lightweight per-host agent process that polls `CloudStore` for assignments targeting its own `host_id` and calls `TaskDispatcher` locally, versus a real RPC/HTTP call from the scheduler's process to a remote host's own dispatcher — left unresolved here since it requires the "no infra" Non-Goal to be revisited first.
|
||||
- Whether `driver/registry.py`'s `register_driver_type()` function (assumed by D6's driver-plugin registration path) should be proposed as an addition to `device-agent-runtime-foundation` before or alongside this change's implementation, given `openspec/specs/` is empty and that change is itself still unapplied — flagged for whoever implements `plugin-system` to resolve with that change's owner rather than silently diverging.
|
||||
- Whether `tool`/`skill`-kind plugin manifests should get a real wiring path in a follow-up change once `tools/` and the skill stores (`skill-catalog-subscription`, `skill-learning-runtime`) gain their own public registration hooks — deliberately left declared-but-unwired here (D6, mirrors `skill-learning-runtime`'s own precedent of leaving skill *execution* unbuilt until `workflow-orchestration-runtime` needed it).
|
||||
- Whether `ScheduledTask`'s `constraints` should eventually accept a `skill_id`/`workflow_definition_id` directly (submit "run this workflow on any matching device") versus only a bare `goal` string — this change's `TaskDispatcher` supports both today by inspecting which field is populated, but the scheduler's own matching logic (constraints → candidate devices) does not yet special-case workflow-shaped submissions differently from goal-shaped ones; worth revisiting once real workflow submissions are exercised through `platform-sdk`.
|
||||
- Whether `platform-sdk`'s `AuthProvider` hook should be promoted to a real auth capability in its own right (its own openspec change) once a concrete integrator/security requirement exists, rather than staying a documented no-op — intentionally left as a future decision, not prejudged here.
|
||||
@@ -0,0 +1,34 @@
|
||||
## Why
|
||||
|
||||
Every capability built through Milestone 9 — `device-management`/`driver-registry` (`apex-agent-mvp`, `device-agent-runtime-foundation`), `agent-runtime`, `workflow-orchestration`, `multi-agent-collaboration` — still assumes one process, one `DeviceManager`, and a caller who already knows which `device_id` to target. That is fine for a single developer driving a handful of physical devices from one machine, but it does not scale to "many devices across many hosts, tasks submitted by external systems without knowing which device is free, and new drivers/tools/skills added without editing this repository's core packages." Milestone 10 (Cloud Runtime) is the platform-completion step: it introduces a device pool that aggregates device state across hosts, a scheduler that assigns queued tasks to idle devices instead of requiring callers to pick one, a plugin mechanism so a second driver (or a tool/skill) can register itself without touching `driver/`, `device/`, or `tools/`, and a versioned public SDK/API for external integrators that is distinct from the LLM-facing MCP tool server. None of this replaces the existing single-process, single-goal execution path — it composes it, the same way `workflow-orchestration-runtime` composed `TaskRunner` rather than rewriting it.
|
||||
|
||||
## What Changes
|
||||
|
||||
- Add a new `cloud/` package, sibling to `agents/`, `workflow/`, `semantic/`, `world/`, `skills_learning/`, holding all four new capabilities below. No existing package (`driver/`, `device/`, `runtime/`, `tools/`, `workflow/`, `agents/`, `storage/`, `api/console.py`, `api/mcp.py`) is modified.
|
||||
- Add `cloud/pool.py`: a `DevicePool` that aggregates device state across multiple hosts, each still running its own unmodified single-process `DeviceManager` (`device-management`). A per-host sync call (`DevicePool.sync_host_devices(host_id, snapshot)`) pushes that host's `DeviceManager.list_devices()` result into the pool on a heartbeat interval; the pool tracks `HostRegistration` (host id, address, last-seen) and `PooledDevice` (device id, owning host id, driver_type, status, capability tags) records, and marks a host's devices `unreachable` (a new pool-level status, distinct from `device-management`'s own `idle`/`busy`/`offline`/`error`) once its heartbeat goes stale — never raises or blocks on a missing host.
|
||||
- Add `cloud/scheduler.py`: a `TaskScheduler` that accepts a queued task submission (goal + optional device constraints: `driver_type`, required capability tags), holds it in a bounded FIFO queue backed by `cloud/store.py`, and assigns it to the first matching `idle` `PooledDevice` the `DevicePool` reports, via a pluggable `AssignmentStrategy` registry (default: FIFO + constraint match; a priority/capability-scoring strategy can be added later without changing `TaskScheduler`'s control flow).
|
||||
- Add `cloud/dispatch.py`: a `TaskDispatcher` that takes a `TaskScheduler` assignment (task + `device_id` + `host_id`) and actually runs it by composing the existing, unmodified `agent-runtime` `TaskRunner` (single-goal tasks) or `workflow-orchestration`'s `WorkflowRunner` (multi-step `WorkflowDefinition`s), strictly through their existing public `run(task) -> Task` / `run(definition) -> WorkflowRun` entry points — closing the loop from "assigned" to "executed" without a caller ever hand-picking a `device_id` again.
|
||||
- Add `cloud/plugins.py`: a `PluginManifest` schema (name, version, `entry_point_kind: driver | tool | skill`, module/callable reference) and a `PluginRegistry` that discovers manifests via Python `entry_points` (group `device_agent_runtime.plugins`) and/or a local `plugins/*/plugin.json` directory scan, validates them, and — for `driver`-kind manifests only — registers them into `driver-registry`'s existing `driver_type -> factory` extension point. `tool`- and `skill`-kind manifests are accepted and validated by the same schema but are declared-not-yet-wired in this change (see Non-Goals in design.md), since `tools/` and the skill stores (`skill-catalog-subscription`, `skill-learning-runtime`) do not yet expose a comparable public registration hook to compose against.
|
||||
- Add `cloud/sdk/` (`api.py`, `client.py`, `models.py`): a versioned (`/v1/...`) public REST API plus a thin Python client, for external integrators to submit tasks/workflows to the pool+scheduler, poll task/device/host status, and list/register plugins — a fleet-facing surface distinct from `mcp-tool-server`'s per-device, LLM-facing tool calls and from `web-console`'s operator-facing status/config UI (`console-status-api`/`console-config-api`), which both continue to talk to one process's own `DeviceManager` directly.
|
||||
- Add `cloud/store.py`: a new, independently-owned SQLite file (`cloud/cloud.sqlite3`) with `host_registrations`, `pooled_devices`, `scheduled_tasks`, and `plugins` tables — mirrors `workflow-orchestration-runtime`'s precedent of a capability-owned store rather than adding tables to `storage/task_metadata.py`.
|
||||
- Add `cloud/config.py`: heartbeat interval, host-staleness threshold, queue depth limit, default `AssignmentStrategy` name, SDK API version prefix — all with conservative defaults so this change is inert until a caller actually registers a second host or submits through the new SDK.
|
||||
- **BREAKING**: none. `driver/`, `device/`, `runtime/`, `tools/`, `workflow/`, `agents/`, `storage/`, `api/console.py`, and `api/mcp.py` are not modified; `cloud/` is purely additive and composes all of them by import.
|
||||
|
||||
## Capabilities
|
||||
|
||||
### New Capabilities
|
||||
- `device-pool`: A multi-host device registry/discovery layer (`DevicePool`, `HostRegistration`, `PooledDevice`) aggregating the device state each host's existing single-process `DeviceManager` already tracks, with heartbeat-based staleness detection so an unreachable host degrades its devices' pool-visible status rather than blocking or crashing the pool.
|
||||
- `task-scheduler`: A queue (`TaskScheduler`) plus pluggable `AssignmentStrategy` matching submitted tasks (goal + device constraints) to an idle `PooledDevice`, and a `TaskDispatcher` that executes an assignment by composing the existing `agent-runtime`/`workflow-orchestration` execution paths as black boxes.
|
||||
- `plugin-system`: A registration mechanism (`PluginManifest` + `PluginRegistry`, entry-points or manifest-file discovery) for new drivers/tools/skills to register themselves without editing `driver/`, `device/`, `tools/`, or the skill stores; concretely wired for driver-kind plugins into `driver-registry`'s existing extension point, with tool/skill-kind plugins schema-defined for forward compatibility.
|
||||
- `platform-sdk`: A versioned, public-facing REST API and Python client for external integrators to submit tasks/workflows, query device/host/task status, and manage plugins across the device pool — separate from `mcp-tool-server` (LLM-facing) and `console-status-api`/`console-config-api` (operator-facing, single-process).
|
||||
|
||||
### Modified Capabilities
|
||||
(none — `openspec/specs/` is currently empty and none of `device-management`, `driver-registry`, `agent-runtime`, `workflow-orchestration`, `mcp-tool-server`, `console-status-api`, `console-config-api` have an applied baseline to diff against; this change composes all of them by import/prose dependency only and does not alter their specified behavior.)
|
||||
|
||||
## Impact
|
||||
|
||||
- **New package**: `cloud/` — `pool.py` (`DevicePool`, `HostRegistration`, `PooledDevice`), `scheduler.py` (`TaskScheduler`, `AssignmentStrategy` registry, `ScheduledTask`), `dispatch.py` (`TaskDispatcher`), `plugins.py` (`PluginManifest`, `PluginRegistry`), `store.py` (`CloudStore`, schema for `cloud/cloud.sqlite3`), `config.py`, and `cloud/sdk/` (`api.py`, `client.py`, `models.py`).
|
||||
- **No changes** to `core/` (or `driver/`/`device/` once `device-agent-runtime-foundation` is applied), `runtime/`, `tools/`, `storage/`, `workflow/`, `agents/`, `api/console.py`, `api/mcp.py`, `api/rest.py` — every integration point is by import/composition, matching the precedent set by `workflow-orchestration-runtime` (D1, D8) and `multi-agent-runtime` (`CollaborativeTaskRunner` composing `TaskRunner`).
|
||||
- **Reads from pending capabilities (composition only, no spec changes to them)**: `device-management`/`driver-registry` for the per-host `DeviceManager`/driver-type extension point `device-pool` aggregates and `plugin-system` registers into; `agent-runtime` and `workflow-orchestration` for the `TaskRunner`/`WorkflowRunner` entry points `task-scheduler`'s `TaskDispatcher` invokes; `mcp-tool-server` as the sibling LLM-facing surface `platform-sdk` sits alongside without replacing; `console-status-api`/`console-config-api` (`web-console`) as the sibling operator-facing surface this change does not extend or duplicate.
|
||||
- **Config**: `pyproject.toml` gains a `cloud*` entry in `[tool.setuptools.packages.find].include`; no new third-party dependency is required for the pool/scheduler/plugin data models (plugin discovery uses the standard-library `importlib.metadata`), though `cloud/sdk/api.py` reuses the already-declared `fastapi`/`uvicorn` dependencies and `cloud/sdk/client.py` will need an HTTP client (reuse `httpx`, already a dev dependency — promote to a runtime dependency in `tasks.md`).
|
||||
- **Out of scope**: no actual multi-host deployment/infra config (Docker/Kubernetes manifests, service discovery infra) — only the runtime's internal pool/scheduler/plugin/SDK model; no billing, multi-tenant auth, or rate limiting on `platform-sdk` (a later change's concern); no cross-host network dispatch mechanism for actually invoking a remote host's `DeviceManager` over the wire (`TaskDispatcher` composes `TaskRunner`/`WorkflowRunner` only when the assignment lands on the local host — see design.md's Non-Goals and Open Questions for the remote-dispatch gap); no changes to `skill-catalog-subscription`, `skill-learning-runtime`, `web-console`, or `multi-agent-runtime`.
|
||||
@@ -0,0 +1,45 @@
|
||||
## ADDED Requirements
|
||||
|
||||
### Requirement: Host registration and heartbeat sync
|
||||
The system SHALL provide a `DevicePool` that tracks a `HostRegistration` (host id, address, last-seen timestamp) for each host process that registers itself, and SHALL update a host's last-seen timestamp whenever that host pushes a device snapshot via `sync_host_devices(host_id, snapshot)`.
|
||||
|
||||
#### Scenario: New host registers and syncs devices
|
||||
- **WHEN** a previously-unknown `host_id` calls `sync_host_devices` with a list of devices
|
||||
- **THEN** the pool creates a new `HostRegistration` for that host, records the current time as its last-seen timestamp, and stores each synced device as a `PooledDevice` owned by that host
|
||||
|
||||
#### Scenario: Known host re-syncs
|
||||
- **WHEN** an already-registered `host_id` calls `sync_host_devices` again with an updated device snapshot
|
||||
- **THEN** the pool updates that host's last-seen timestamp and replaces its previously-stored `PooledDevice` records with the new snapshot, without duplicating or losing devices from other hosts
|
||||
|
||||
### Requirement: Aggregated device listing across hosts
|
||||
The system SHALL provide a way to list all `PooledDevice` records across every registered host, including each device's owning `host_id`, `driver_type`, status, and capability tags.
|
||||
|
||||
#### Scenario: Listing devices across multiple hosts
|
||||
- **WHEN** two hosts have each synced a non-empty device snapshot
|
||||
- **THEN** a caller listing pool devices sees devices from both hosts in one combined result, each tagged with its correct `host_id`
|
||||
|
||||
#### Scenario: No hosts registered
|
||||
- **WHEN** a caller lists pool devices before any host has ever synced
|
||||
- **THEN** the pool returns an empty list rather than raising an error
|
||||
|
||||
### Requirement: Stale host devices degrade to unreachable
|
||||
The system SHALL mark all `PooledDevice`s belonging to a host `unreachable` once that host's last-seen timestamp exceeds a configured staleness threshold, computed at read time, without requiring any background process and without raising an error for the stale host's absence.
|
||||
|
||||
#### Scenario: Host misses its sync interval
|
||||
- **WHEN** a host's last-seen timestamp is older than `config.stale_after_seconds` at the time of a `list_devices()`/`get_device()` call
|
||||
- **THEN** every `PooledDevice` owned by that host is reported with status `unreachable`, regardless of the status value in its last-synced snapshot
|
||||
|
||||
#### Scenario: Host resumes syncing after being stale
|
||||
- **WHEN** a host previously marked stale calls `sync_host_devices` again
|
||||
- **THEN** its devices immediately stop being reported `unreachable` and reflect the statuses in the new snapshot
|
||||
|
||||
### Requirement: Device lookup by id across the pool
|
||||
The system SHALL allow looking up a single `PooledDevice` by `device_id` regardless of which host owns it, returning a clear not-found result when no host has ever reported that device id.
|
||||
|
||||
#### Scenario: Lookup finds device on any host
|
||||
- **WHEN** a caller requests a device by id that exists in some host's synced snapshot
|
||||
- **THEN** the pool returns that `PooledDevice` including its owning `host_id`
|
||||
|
||||
#### Scenario: Lookup for unknown device id
|
||||
- **WHEN** a caller requests a device by id that no host has ever synced
|
||||
- **THEN** the pool returns a not-found result (e.g. `None`) rather than raising an unhandled exception
|
||||
@@ -0,0 +1,63 @@
|
||||
## ADDED Requirements
|
||||
|
||||
### Requirement: Versioned public API surface
|
||||
The system SHALL expose the platform SDK's REST endpoints under a versioned URL prefix (`/v1/...`), distinct from the `mcp-tool-server` and `console-status-api`/`console-config-api` surfaces, so external integrators have a stable base path that will not silently change shape.
|
||||
|
||||
#### Scenario: Routes are mounted under the version prefix
|
||||
- **WHEN** the platform SDK's router is mounted into an application
|
||||
- **THEN** every route it exposes (task submission, status queries, device/host listing, plugin listing/registration) is reachable only under the `/v1/` prefix
|
||||
|
||||
### Requirement: Task submission and status via the SDK
|
||||
The system SHALL allow an external integrator to submit a task (goal or workflow reference plus constraints) through the platform SDK's API, and to query that task's current status by id, backed by the `task-scheduler` capability.
|
||||
|
||||
#### Scenario: Submit a task via the API
|
||||
- **WHEN** an integrator calls the task-submission endpoint with a valid goal and optional constraints
|
||||
- **THEN** the API returns a task id that can be used to poll status, and the underlying `task-scheduler` records a new `queued` `ScheduledTask`
|
||||
|
||||
#### Scenario: Query status of a known task
|
||||
- **WHEN** an integrator requests status for a task id that exists
|
||||
- **THEN** the API returns that task's current status (`queued`, `assigned`, `dispatched`, `done`, or `failed`)
|
||||
|
||||
#### Scenario: Query status of an unknown task
|
||||
- **WHEN** an integrator requests status for a task id that does not exist
|
||||
- **THEN** the API returns a not-found response rather than an unhandled server error
|
||||
|
||||
### Requirement: Device and host visibility via the SDK
|
||||
The system SHALL allow an external integrator to list devices and hosts known to the `device-pool` capability through the platform SDK's API.
|
||||
|
||||
#### Scenario: List devices across the pool
|
||||
- **WHEN** an integrator calls the device-listing endpoint
|
||||
- **THEN** the API returns every `PooledDevice` known to the pool, including owning host id and current (possibly `unreachable`) status
|
||||
|
||||
#### Scenario: List registered hosts
|
||||
- **WHEN** an integrator calls the host-listing endpoint
|
||||
- **THEN** the API returns every `HostRegistration` known to the pool, including last-seen timestamp
|
||||
|
||||
### Requirement: Plugin listing and registration via the SDK
|
||||
The system SHALL allow an external integrator to list registered plugins and submit a new plugin manifest for registration through the platform SDK's API, backed by the `plugin-system` capability.
|
||||
|
||||
#### Scenario: List registered plugins
|
||||
- **WHEN** an integrator calls the plugin-listing endpoint
|
||||
- **THEN** the API returns every registered `PluginManifest`, including its `entry_point_kind` and whether it is wired to an execution path
|
||||
|
||||
#### Scenario: Register a new plugin manifest
|
||||
- **WHEN** an integrator submits a valid plugin manifest to the plugin-registration endpoint
|
||||
- **THEN** the API registers it via `plugin-system`'s `PluginRegistry` and returns the stored manifest, or a clear validation/conflict error if registration fails
|
||||
|
||||
### Requirement: Pluggable authentication hook with a safe default
|
||||
The system SHALL evaluate every platform SDK route through a configurable `AuthProvider` hook, defaulting to a no-op provider that treats every caller as an anonymous, authenticated principal, so real authentication can be added later without changing route signatures.
|
||||
|
||||
#### Scenario: Default configuration allows anonymous access
|
||||
- **WHEN** no `AuthProvider` is explicitly configured
|
||||
- **THEN** every route accepts requests without rejecting them for lack of credentials
|
||||
|
||||
#### Scenario: Custom AuthProvider is honored
|
||||
- **WHEN** a caller configures a custom `AuthProvider` that rejects a request
|
||||
- **THEN** the platform SDK's routes return an authorization error for that request instead of proceeding, without any route's own handler code needing to change
|
||||
|
||||
### Requirement: Python SDK client mirrors the REST API
|
||||
The system SHALL provide a Python client (`CloudClient`) exposing methods corresponding to each `/v1/...` route (submit task, get task status, list devices, list hosts, list plugins, register plugin), so integrators do not need to hand-construct HTTP requests.
|
||||
|
||||
#### Scenario: Client submits a task and retrieves status
|
||||
- **WHEN** a caller uses `CloudClient` to submit a task and then fetch its status by the returned id
|
||||
- **THEN** the client's methods produce the same result as calling the corresponding `/v1/...` endpoints directly over HTTP
|
||||
@@ -0,0 +1,53 @@
|
||||
## ADDED Requirements
|
||||
|
||||
### Requirement: Plugin manifest schema
|
||||
The system SHALL define a `PluginManifest` schema with a unique `name`, a `version`, an `entry_point_kind` restricted to `driver`, `tool`, or `skill`, and a `target` (a dotted module:attribute reference to the plugin's implementation), and SHALL reject a manifest missing any required field or using an unrecognized `entry_point_kind`.
|
||||
|
||||
#### Scenario: Valid manifest accepted
|
||||
- **WHEN** a manifest with all required fields and a recognized `entry_point_kind` is submitted for registration
|
||||
- **THEN** the registry accepts it and stores it as a known plugin
|
||||
|
||||
#### Scenario: Manifest with unrecognized entry_point_kind rejected
|
||||
- **WHEN** a manifest declares an `entry_point_kind` other than `driver`, `tool`, or `skill`
|
||||
- **THEN** the registry rejects it with a clear validation error and does not register it
|
||||
|
||||
#### Scenario: Duplicate plugin name rejected
|
||||
- **WHEN** a manifest is submitted whose `name` matches an already-registered plugin
|
||||
- **THEN** the registry rejects the new registration with a clear conflict error rather than silently overwriting the existing entry
|
||||
|
||||
### Requirement: Plugin discovery via entry points and manifest files
|
||||
The system SHALL discover plugin manifests both from installed Python packages declaring an entry point in the `device_agent_runtime.plugins` group and from local `plugin.json` files under a configured scan path, feeding both sources into the same validation-and-registration path.
|
||||
|
||||
#### Scenario: Discovery via installed entry point
|
||||
- **WHEN** an installed package declares an entry point in the `device_agent_runtime.plugins` group resolving to a valid manifest
|
||||
- **THEN** `PluginRegistry.discover()` finds and registers it
|
||||
|
||||
#### Scenario: Discovery via local manifest file
|
||||
- **WHEN** a `plugin.json` file exists under the configured plugin scan path and parses into a valid manifest
|
||||
- **THEN** `PluginRegistry.discover()` finds and registers it
|
||||
|
||||
#### Scenario: Malformed manifest file is skipped, not fatal
|
||||
- **WHEN** a `plugin.json` file under the scan path fails to parse or fails schema validation
|
||||
- **THEN** `PluginRegistry.discover()` skips that file, records it as a discovery error, and continues discovering remaining plugins rather than aborting the whole scan
|
||||
|
||||
### Requirement: Driver-kind plugins register into the driver registry extension point
|
||||
The system SHALL, for a manifest with `entry_point_kind == "driver"`, resolve its `target` to a driver-factory builder and register it under the manifest's `name` as a new `driver_type` in the existing driver-registry extension point, without requiring any edit to the `driver` package's own files.
|
||||
|
||||
#### Scenario: Driver plugin registered successfully
|
||||
- **WHEN** a valid `driver`-kind manifest is registered and its `target` resolves to a callable driver-factory builder
|
||||
- **THEN** the manifest's `name` becomes usable as a `driver_type` value by any caller building a driver factory, with no change to existing driver-registry code
|
||||
|
||||
#### Scenario: Driver registry extension point unavailable
|
||||
- **WHEN** a `driver`-kind manifest is registered but the driver-registry's registration function is not importable in the running environment
|
||||
- **THEN** the registry raises a clear, explicit error naming the missing integration point, rather than silently accepting the manifest without wiring it
|
||||
|
||||
### Requirement: Tool and skill plugin manifests are accepted but explicitly marked unwired
|
||||
The system SHALL accept and store `tool`- and `skill`-kind plugin manifests (listable like any other registered plugin) but SHALL report them as not wired to any execution path, rather than implying they are active.
|
||||
|
||||
#### Scenario: Tool-kind manifest registered
|
||||
- **WHEN** a valid `tool`-kind manifest is registered
|
||||
- **THEN** the registry stores it and it appears in a plugin listing with a `wired: false` indicator, and no tool dispatch path is modified as a result
|
||||
|
||||
#### Scenario: Skill-kind manifest registered
|
||||
- **WHEN** a valid `skill`-kind manifest is registered
|
||||
- **THEN** the registry stores it and it appears in a plugin listing with a `wired: false` indicator, and no skill store or execution path is modified as a result
|
||||
@@ -0,0 +1,56 @@
|
||||
## ADDED Requirements
|
||||
|
||||
### Requirement: Task submission enqueues a scheduled task
|
||||
The system SHALL allow a caller to submit a task (a goal string, or a reference to a `WorkflowDefinition`, plus optional device constraints: `driver_type`, required capability tags) and SHALL enqueue it as a `ScheduledTask` with status `queued`, returning a stable task id the caller can poll.
|
||||
|
||||
#### Scenario: Successful submission
|
||||
- **WHEN** a caller submits a task with a goal and no constraints
|
||||
- **THEN** the scheduler creates a `ScheduledTask` with status `queued`, assigns it a unique id, and returns that id to the caller without blocking for a device to become available
|
||||
|
||||
#### Scenario: Queue depth limit reached
|
||||
- **WHEN** a caller submits a task while the queue already holds `config.max_queue_depth` queued tasks
|
||||
- **THEN** the scheduler rejects the submission with a clear error rather than accepting an unbounded backlog
|
||||
|
||||
### Requirement: Assignment matches a queued task to an idle, constraint-matching device
|
||||
The system SHALL assign a queued `ScheduledTask` to an idle `PooledDevice` (as reported by the `device-pool` capability) whose `driver_type` and capability tags satisfy the task's constraints, using a named, registrable `AssignmentStrategy`.
|
||||
|
||||
#### Scenario: Matching idle device available
|
||||
- **WHEN** `assign()` runs and at least one idle `PooledDevice` matches the head-of-queue task's constraints
|
||||
- **THEN** the scheduler selects one such device via the configured `AssignmentStrategy`, transitions the task to status `assigned`, and records the chosen `device_id`/`host_id`
|
||||
|
||||
#### Scenario: No matching device available
|
||||
- **WHEN** `assign()` runs and no idle `PooledDevice` matches the head-of-queue task's constraints
|
||||
- **THEN** the task remains `queued` (not failed), and `assign()` returns without error, ready to be retried on a later call
|
||||
|
||||
#### Scenario: Unknown assignment strategy configured
|
||||
- **WHEN** `TaskScheduler` is configured with an `AssignmentStrategy` name that is not registered
|
||||
- **THEN** the scheduler raises a clear configuration error at startup/first-assign rather than silently falling back to a default strategy
|
||||
|
||||
### Requirement: Assignment strategies are pluggable by name
|
||||
The system SHALL provide an `AssignmentStrategy` registry mapping a strategy name to an implementation, with a default `fifo_match` strategy (oldest-queued matching task first, first matching idle device), and SHALL allow a new strategy to be added by registering a name without modifying `TaskScheduler`'s control flow.
|
||||
|
||||
#### Scenario: Default FIFO strategy orders by submission time
|
||||
- **WHEN** two tasks with satisfiable, overlapping constraints are queued in order A then B, and one matching idle device exists
|
||||
- **THEN** the default `fifo_match` strategy assigns the device to task A, leaving task B queued
|
||||
|
||||
#### Scenario: Adding a new strategy requires no scheduler edit
|
||||
- **WHEN** a new `AssignmentStrategy` implementation is registered under a new name
|
||||
- **THEN** `TaskScheduler` can be configured to use it by name alone, with no change to `scheduler.py`'s assignment control flow
|
||||
|
||||
### Requirement: Local dispatch executes an assignment via existing runners
|
||||
The system SHALL provide a `TaskDispatcher` that, for an assignment whose device is owned by the local process's own host, executes the assigned task by composing the existing `agent-runtime` task-execution entry point (for a goal-based submission) or the `workflow-orchestration` workflow-execution entry point (for a workflow-based submission), without reimplementing planning/execution/retry logic.
|
||||
|
||||
#### Scenario: Dispatching a goal-based assignment
|
||||
- **WHEN** `TaskDispatcher.dispatch()` is called with an assignment for a goal-based `ScheduledTask` whose device is local
|
||||
- **THEN** the dispatcher constructs and runs a `Task` through the existing task-execution entry point, and updates the `ScheduledTask`'s status to `done` or `failed` based on the resulting task's outcome
|
||||
|
||||
#### Scenario: Dispatching a workflow-based assignment
|
||||
- **WHEN** `TaskDispatcher.dispatch()` is called with an assignment referencing a `WorkflowDefinition` whose device is local
|
||||
- **THEN** the dispatcher runs the definition through the existing workflow-execution entry point and updates the `ScheduledTask`'s status based on the resulting workflow run's outcome
|
||||
|
||||
### Requirement: Remote assignments are rejected explicitly, not silently ignored
|
||||
The system SHALL raise a distinct, typed error when `TaskDispatcher.dispatch()` is called for an assignment whose device is owned by a host other than the dispatching process's own host, rather than attempting execution or silently no-op'ing.
|
||||
|
||||
#### Scenario: Assignment targets a remote host's device
|
||||
- **WHEN** `TaskDispatcher.dispatch()` is called with an assignment whose `host_id` does not match the local process's own host id
|
||||
- **THEN** the dispatcher raises a `RemoteDispatchNotSupportedError` and leaves the `ScheduledTask`'s status unchanged from `assigned`
|
||||
@@ -0,0 +1,76 @@
|
||||
## 1. Package scaffolding
|
||||
|
||||
- [ ] 1.1 Create the `cloud/` package (`__init__.py`, `pool.py`, `scheduler.py`, `dispatch.py`, `plugins.py`, `store.py`, `config.py`) and the `cloud/sdk/` sub-package (`__init__.py`, `api.py`, `client.py`, `models.py`)
|
||||
- [ ] 1.2 Add `cloud*` to `[tool.setuptools.packages.find].include` in `pyproject.toml`
|
||||
- [ ] 1.3 Promote `httpx` from `[dependency-groups].dev` to `[project].dependencies` in `pyproject.toml` (needed at runtime by `cloud/sdk/client.py`)
|
||||
- [ ] 1.4 Implement `cloud/config.py`: `CloudConfig` dataclass with `sync_interval_seconds`, `stale_after_seconds`, `max_queue_depth`, `default_assignment_strategy`, `api_version_prefix` (`"/v1"`), `db_path` (`cloud/cloud.sqlite3`), each with a conservative documented default
|
||||
- [ ] 1.5 Extend the project's smoke test (that imports every package) to import `cloud` and `cloud.sdk`
|
||||
|
||||
## 2. Device pool data model and store (capability: device-pool)
|
||||
|
||||
- [ ] 2.1 Implement `cloud/pool.py`'s data types: `HostRegistration{host_id, address, last_seen_at}`, `PooledDevice{device_id, host_id, driver_type, status, capability_tags, synced_at}`
|
||||
- [ ] 2.2 Implement `cloud/store.py`'s `CloudStore(db_path)` with schema creation for `host_registrations` and `pooled_devices` tables (connect-per-call `sqlite3` pattern, following `storage/task_metadata.py`/`workflow/store.py`)
|
||||
- [ ] 2.3 Implement `CloudStore.upsert_host(host_id, address, last_seen_at)` and `CloudStore.replace_host_devices(host_id, devices: list[PooledDevice])` (atomic replace of one host's device rows per sync)
|
||||
- [ ] 2.4 Implement `CloudStore.list_hosts()`, `CloudStore.list_devices()`, `CloudStore.get_device(device_id)`
|
||||
- [ ] 2.5 Write unit tests for `CloudStore`: host upsert + device replace round-trip; a second sync for the same host fully replaces (not appends to) its device rows; devices from two different hosts coexist without collision
|
||||
|
||||
## 3. DevicePool aggregation and staleness (capability: device-pool)
|
||||
|
||||
- [ ] 3.1 Implement `DevicePool(store: CloudStore, config: CloudConfig)` with `sync_host_devices(host_id, snapshot: list[core.models.Device])`, converting each `Device` into a `PooledDevice` and calling `CloudStore.upsert_host()`/`replace_host_devices()`
|
||||
- [ ] 3.2 Implement `DevicePool.list_devices() -> list[PooledDevice]` and `DevicePool.get_device(device_id) -> PooledDevice | None`, both computing per-host staleness lazily at call time (`now - last_seen_at > config.stale_after_seconds` implies status `unreachable`, overriding the last-synced status) rather than via a background thread
|
||||
- [ ] 3.3 Implement `DevicePool.list_hosts() -> list[HostRegistration]`
|
||||
- [ ] 3.4 Write unit tests: new host sync creates a `HostRegistration` + `PooledDevice`s; re-sync updates last-seen and replaces devices; a host whose last-seen exceeds the staleness threshold reports all its devices `unreachable` on the next `list_devices()`/`get_device()` call; a host that resyncs after being stale immediately stops being reported unreachable; lookup for an unknown `device_id` returns `None`; empty pool returns an empty list
|
||||
|
||||
## 4. TaskScheduler queue and assignment (capability: task-scheduler)
|
||||
|
||||
- [ ] 4.1 Implement `cloud/scheduler.py`'s data types: `TaskConstraints{driver_type: str | None, capability_tags: list[str]}`, `ScheduledTask{id, goal: str | None, workflow_definition_id: str | None, constraints, status, assigned_device_id, assigned_host_id, created_at}`
|
||||
- [ ] 4.2 Add `scheduled_tasks` table + `CloudStore.enqueue_task()`, `CloudStore.list_queued_tasks()`, `CloudStore.update_task()`, `CloudStore.get_task(task_id)` to `cloud/store.py`
|
||||
- [ ] 4.3 Implement `AssignmentStrategy` protocol/ABC (`select(task, candidates: list[PooledDevice]) -> PooledDevice | None`) and a registry `dict[str, AssignmentStrategy]`
|
||||
- [ ] 4.4 Implement the default `fifo_match` strategy: return the first candidate in `candidates` (oldest-synced-first is not required; caller already filters to idle+constraint-matching) or `None` if `candidates` is empty
|
||||
- [ ] 4.5 Implement `TaskScheduler(pool: DevicePool, store: CloudStore, config: CloudConfig)` with `submit(goal=None, workflow_definition_id=None, constraints=None) -> str` (returns task id), raising a clear error if `config.max_queue_depth` queued tasks already exist
|
||||
- [ ] 4.6 Implement `TaskScheduler.assign() -> list[Assignment]`: for each queued task (oldest first), filter `pool.list_devices()` to `idle` devices matching `constraints.driver_type`/`capability_tags`, call the configured `AssignmentStrategy`, and on a match transition the task to `assigned` recording `device_id`/`host_id`; leave unmatched tasks `queued`
|
||||
- [ ] 4.7 Raise a clear configuration error if `config.default_assignment_strategy` names a strategy not present in the registry
|
||||
- [ ] 4.8 Write unit tests: submission enqueues with status `queued`; queue-depth-limit rejection; assignment picks a matching idle device via `fifo_match`; assignment leaves a task queued when no device matches; two tasks queued in order are assigned in submission order when only one device is available; unregistered strategy name raises at configuration/first-assign time
|
||||
|
||||
## 5. TaskDispatcher composition (capability: task-scheduler)
|
||||
|
||||
- [ ] 5.1 Implement `cloud/dispatch.py`'s `Assignment{task_id, device_id, host_id, goal, workflow_definition_id}` and `RemoteDispatchNotSupportedError`
|
||||
- [ ] 5.2 Implement `TaskDispatcher(local_host_id: str, task_runner_factory, workflow_runner_factory, store: CloudStore)` with `dispatch(assignment: Assignment) -> None`
|
||||
- [ ] 5.3 Implement the goal-based dispatch path: construct `core.models.Task(goal=assignment.goal, device_id=assignment.device_id)`, call the existing `runtime.task.TaskRunner(...).run(task)` (import only, no edits to `runtime/`), and update the `ScheduledTask`'s status to `done`/`failed` from `task.status`/`task.failure_reason`
|
||||
- [ ] 5.4 Implement the workflow-based dispatch path: load the referenced `WorkflowDefinition` and call the existing `workflow.runner.WorkflowRunner(...).run(definition, device_id=assignment.device_id)` (import only, no edits to `workflow/`), mapping the resulting `WorkflowRun.status` to the `ScheduledTask`'s status
|
||||
- [ ] 5.5 Implement the remote-assignment guard: if `assignment.host_id != local_host_id`, raise `RemoteDispatchNotSupportedError` before constructing any `Task`/`WorkflowDefinition`, leaving the `ScheduledTask` status unchanged at `assigned`
|
||||
- [ ] 5.6 Write unit tests: local goal-based dispatch runs a stubbed `TaskRunner` and updates status to `done`/`failed` correctly; local workflow-based dispatch runs a stubbed `WorkflowRunner` and updates status correctly; remote-host assignment raises `RemoteDispatchNotSupportedError` and leaves status as `assigned`
|
||||
|
||||
## 6. Plugin manifest and registry (capability: plugin-system)
|
||||
|
||||
- [ ] 6.1 Implement `cloud/plugins.py`'s `PluginManifest{name, version, entry_point_kind: Literal["driver", "tool", "skill"], target}` with validation (all fields required, `entry_point_kind` restricted to the three literals)
|
||||
- [ ] 6.2 Add a `plugins` table + `CloudStore.save_plugin()`, `CloudStore.list_plugins()`, `CloudStore.get_plugin(name)` to `cloud/store.py`, storing `wired: bool` alongside each manifest
|
||||
- [ ] 6.3 Implement `PluginRegistry(store: CloudStore)` with `register(manifest: PluginManifest) -> PluginManifest`: reject unknown `entry_point_kind`, reject duplicate `name`, persist via `CloudStore.save_plugin()`
|
||||
- [ ] 6.4 Implement driver-kind wiring: resolve `manifest.target` (dotted `module:attribute` string) to a `DriverFactoryBuilder` callable via `importlib`, and call `driver_registry.register_driver_type(manifest.name, builder)` if importable; raise a clear, named error if the driver-registry function is not importable in the running environment
|
||||
- [ ] 6.5 Implement tool-/skill-kind handling: store the manifest with `wired=False` and do not attempt any further resolution or registration
|
||||
- [ ] 6.6 Implement `PluginRegistry.discover_entry_points() -> list[PluginManifest]` using `importlib.metadata.entry_points(group="device_agent_runtime.plugins")`, resolving each entry point and registering the resulting manifest
|
||||
- [ ] 6.7 Implement `PluginRegistry.discover_manifest_files(scan_path) -> list[PluginManifest]` globbing `plugin.json` under `scan_path`, parsing and registering each; on parse/validation failure, record the file path + error and continue (never abort the scan)
|
||||
- [ ] 6.8 Implement `PluginRegistry.discover() -> DiscoveryResult{registered: list[PluginManifest], errors: list[str]}` combining both discovery sources
|
||||
- [ ] 6.9 Write unit tests: valid manifest registers successfully; unrecognized `entry_point_kind` rejected; duplicate name rejected; driver-kind manifest registers into a fake `driver_registry.register_driver_type`; driver-kind manifest raises a named error when that function is not importable; tool-/skill-kind manifests register with `wired=False` and touch no other registry; entry-point discovery registers a fake installed plugin; manifest-file discovery registers a valid file and skips + records a malformed one without aborting
|
||||
|
||||
## 7. Platform SDK REST API (capability: platform-sdk)
|
||||
|
||||
- [ ] 7.1 Implement `cloud/sdk/models.py`: Pydantic request/response models for task submission, task status, device listing, host listing, plugin listing, and plugin registration
|
||||
- [ ] 7.2 Implement `AuthProvider` protocol (`authenticate(request) -> Principal | None`) and `NullAuthProvider` (always returns an anonymous `Principal`) in `cloud/sdk/api.py` or a small `cloud/sdk/auth.py`
|
||||
- [ ] 7.3 Implement `create_cloud_router(*, pool: DevicePool, scheduler: TaskScheduler, plugin_registry: PluginRegistry, auth_provider: AuthProvider = NullAuthProvider(), version_prefix: str = "/v1") -> APIRouter` in `cloud/sdk/api.py`, mirroring `api/console.py`'s `create_console_router` shape
|
||||
- [ ] 7.4 Implement `POST {prefix}/tasks` (submit via `TaskScheduler.submit()`), `GET {prefix}/tasks/{task_id}` (status via `CloudStore.get_task()`, 404 on unknown id)
|
||||
- [ ] 7.5 Implement `GET {prefix}/devices` (via `DevicePool.list_devices()`) and `GET {prefix}/hosts` (via `DevicePool.list_hosts()`)
|
||||
- [ ] 7.6 Implement `GET {prefix}/plugins` (via `PluginRegistry`/`CloudStore.list_plugins()`) and `POST {prefix}/plugins` (via `PluginRegistry.register()`, returning a validation/conflict error response on failure)
|
||||
- [ ] 7.7 Wire every route through `auth_provider.authenticate(request)`, returning an authorization error response when it returns `None`
|
||||
- [ ] 7.8 Write unit tests using FastAPI's `TestClient`: submit-then-status round trip; unknown task id returns 404; device/host listing reflects pool state; plugin listing/registration round trip; a custom rejecting `AuthProvider` causes every route to return an authorization error while the default `NullAuthProvider` allows all of the above through unchanged
|
||||
|
||||
## 8. Python SDK client (capability: platform-sdk)
|
||||
|
||||
- [ ] 8.1 Implement `cloud/sdk/client.py`'s `CloudClient(base_url, *, http_client=None)` using `httpx`, with `submit_task()`, `get_task_status()`, `list_devices()`, `list_hosts()`, `list_plugins()`, `register_plugin()` methods matching `cloud/sdk/api.py`'s routes
|
||||
- [ ] 8.2 Write unit tests for `CloudClient` against a live `TestClient`-backed instance of the router from task 7.3: submit + status round trip via the client returns the same result as calling the routes directly
|
||||
|
||||
## 9. Composition safety checks and full-suite validation
|
||||
|
||||
- [ ] 9.1 Confirm no existing file under `driver/`/`core/`, `device/`, `runtime/`, `tools/`, `workflow/`, `agents/`, `storage/`, `api/console.py`, or `api/mcp.py` is modified by this change (composition via import only, per design.md's D1/D5)
|
||||
- [ ] 9.2 Write a test that runs a real (non-mocked, stub-driver-backed) `runtime.task.TaskRunner` instance inside `TaskDispatcher.dispatch()`'s goal-based path, guarding against silent drift in `agent-runtime`'s public `run(task) -> Task` contract this change composes over
|
||||
- [ ] 9.3 Run the full test suite (`pytest`) and confirm every existing test in `tests/` passes unmodified, with only new `tests/test_device_pool.py`, `tests/test_task_scheduler.py`, `tests/test_task_dispatcher.py`, `tests/test_plugin_registry.py`, `tests/test_cloud_sdk_api.py`, `tests/test_cloud_client.py`-style files added
|
||||
Reference in New Issue
Block a user