agentic-mobile-control/openspec/changes/archive/2026-07-06-cloud-runtime/design.md

Context

Every prior milestone — device-management/driver-registry (apex-agent-mvp, device-agent-runtime-foundation), agent-runtime, semantic-scene, world-model, skill-authoring/skill-versioning/skill-embedding-retrieval, workflow-orchestration, multi-agent-collaboration — is designed and (mostly) built around one process: one DeviceManager instance holding an in-memory dict of devices, one TaskRunner/WorkflowRunner executing on whichever device_id the caller passes in, and one driver/registry.py extension point (SUPPORTED_DRIVER_TYPES) that a caller reads to build a driver by name. That single-process model is a deliberate, explicit non-goal-turned-goal in apex-agent-mvp/design.md ("No production task queue ... synchronous, single-process execution is sufficient for the MVP") and every later milestone has respected it rather than re-litigating it — workflow-orchestration-runtime's own Non-Goals state plainly: "No distributed/multi-device workflow execution or cross-node coordination — that is Milestone 10 (Cloud Runtime)."

This change is that milestone. Three things are genuinely new here that no prior milestone had to solve: (1) device state now spans more than one host process, so "list devices" and "is this device idle" can no longer be answered by one in-memory dict; (2) a caller submitting a task no longer names a device_id up front — something has to pick one, from a fleet, based on constraints; (3) new drivers/tools/skills should be addable without editing this repository's packages at all, which the existing "add a dict entry to driver/registry.py" extension point does not yet support for a party who cannot modify the repo.

Two stakeholders: an external integrator (a system outside this repo, e.g. a customer's backend) who needs a stable, versioned API to submit work and query status without going through the LLM-facing MCP surface or the operator-facing web console; and a fleet operator who runs more than one host process (each with its own DeviceManager and set of physically-attached devices) and needs one place to see "how many devices are idle across all of them" and have tasks land on whichever is free.

Goals / Non-Goals

Goals:

• Define an internal DevicePool model that aggregates device state from potentially many hosts' independent DeviceManager instances, degrading gracefully (marking a host's devices unreachable, never crashing or blocking) when a host stops reporting in.
• Define a TaskScheduler that accepts task submissions with optional device constraints and assigns them to an idle, matching PooledDevice, via a pluggable, registrable AssignmentStrategy — the same "string key → swappable implementation" shape as driver/registry.py's SUPPORTED_DRIVER_TYPES and workflow-orchestration-runtime's ConditionEvaluator registry.
• Define a TaskDispatcher that actually executes a scheduler assignment by composing the existing agent-runtime/workflow-orchestration entry points — so device selection becomes automatic without duplicating any execution logic those capabilities already own.
• Define a PluginManifest/PluginRegistry mechanism so a new driver (and, in schema if not yet in wiring, a new tool or skill) can register itself via a manifest file or Python entry point, without editing driver/, device/, tools/, or the skill stores.
• Define a versioned, public platform-sdk (REST API + Python client) for external integrators, clearly scoped apart from mcp-tool-server (LLM tool-calling surface) and console-status-api/console-config-api (single-process operator UI surface).

Non-Goals:

• No actual multi-host deployment or infrastructure tooling (no Docker/Kubernetes manifests, no service mesh, no real network transport implementation for host-to-pool sync beyond defining the sync_host_devices(host_id, snapshot) call signature this change's local reference implementation exercises in-process). This change defines the runtime's internal data model and control flow for a device pool and scheduler, not ops tooling to actually run one across real machines.
• No cross-host task execution dispatch over the network: TaskDispatcher composes TaskRunner/WorkflowRunner directly, which only works when the assigned device is owned by the same process the dispatcher runs in. When DevicePool reports a PooledDevice owned by a different host, TaskDispatcher raises a clear, typed RemoteDispatchNotSupportedError rather than silently no-op'ing or pretending to execute — real cross-host RPC dispatch is left as an Open Question / follow-up, consistent with this change's own "no infra" non-goal.
• No billing, multi-tenant authentication/authorization, or rate limiting on platform-sdk — a later change's concern; this change may add a single API-key placeholder hook (see D8) but does not implement real auth.
• No generic dynamic tool/skill plugin wiring — plugin-system's manifest schema accepts tool/skill-kind entries for forward compatibility, but only driver-kind plugins are concretely registered into an existing extension point (driver-registry) in this change; tools/ and the skill stores have no comparable public registration hook yet to compose against.
• No changes to driver/, device/, runtime/, tools/, workflow/, agents/, storage/, api/console.py, or api/mcp.py — every integration is by import/composition only.
• No replacement of mcp-tool-server or console-status-api/console-config-api — platform-sdk is a third, additive surface, not a superset or a router in front of the other two.

Decisions

D1: New cloud/ package, sibling to agents//workflow//semantic//world//skills_learning/, not folded into runtime/, device/, or api/

cloud/ (pool.py, scheduler.py, dispatch.py, plugins.py, store.py, config.py, sdk/) is its own top-level package. This continues the one-package-per-milestone precedent (agents/ for Milestone 9, workflow/ for Milestone 8, semantic//world//skills_learning/ for Milestones 5–7) and keeps runtime/'s job (single-goal Observe-Think-Act loop) and device/'s job (one process's own device lifecycle) unchanged.

• Alternative considered: Extend device/manager.py's DeviceManager in place to understand multiple hosts. Rejected — DeviceManager is a pending, unapplied capability (device-management) with no applied baseline in openspec/specs/ to safely diff a MODIFIED delta against in this session (the same reasoning workflow-orchestration-runtime's D1 already applied to TaskRunner); treating it as a stable, composed-over dependency keeps "one process's device lifecycle" and "many processes' aggregated device state" as two separate, independently testable concerns, and avoids retroactively changing another change's still-pending contract.
• Alternative considered: Put the four capabilities in four separate top-level packages (device_pool/, task_scheduler/, plugins/, platform_sdk/). Rejected — these four are tightly coupled (the scheduler reads the pool; the SDK exposes both; the dispatcher needs the scheduler's assignment) and the roadmap already names this single milestone "Cloud Runtime"; one package with clear submodules keeps the coupling visible in the import graph instead of scattering it across four independent top-level packages that would need to depend on each other anyway.

D2: DevicePool aggregates by push, not by remotely calling each host's DeviceManager

DevicePool.sync_host_devices(host_id, snapshot: list[Device]) is called (by a periodic job, or by platform-sdk's host-registration endpoint) with the result of that host's own DeviceManager.list_devices() — the pool never reaches out over the network to call a remote DeviceManager method itself. HostRegistration.last_seen_at updates on every sync call; a background staleness check (or a check performed lazily on read, see D3) marks a host's PooledDevices unreachable once last_seen_at exceeds config.stale_after_seconds.

• Alternative considered: Give DevicePool an RPC client that calls a remote host's DeviceManager over HTTP/gRPC directly (pull model). Rejected for this change — building a real inter-host RPC layer is exactly the "actual multi-host deployment" infrastructure work this change's Non-Goals defer; a push-based sync call has a stable, host-agnostic signature today (sync_host_devices(host_id, snapshot)) that a later change can wire to any real transport (HTTP call from a lightweight per-host agent, message queue, etc.) without changing DevicePool's internal model.

D3: Host staleness is checked lazily on read, not via a background timer thread

DevicePool.list_devices() and DevicePool.get_device(device_id) compute each host's staleness (now - last_seen_at > stale_after_seconds) at call time and report unreachable status accordingly, rather than running a background thread that mutates stored status on a timer.

• Alternative considered: Run a background thread/asyncio task that periodically sweeps HostRegistrations and flips stale ones to unreachable in the store. Rejected — a background thread adds lifecycle management (start/stop with the process, thread-safety with CloudStore's SQLite connections) for a property (staleness) that is cheap to compute at read time from a single stored timestamp; lazy computation is simpler, always consistent with the current clock, and avoids a whole class of "the sweep thread died silently" failure modes for no loss of correctness.

D4: TaskScheduler's queue and AssignmentStrategy registry mirror the Driver Registry / ConditionEvaluator pattern

cloud/scheduler.py defines AssignmentStrategy (one method, select(task: ScheduledTask, candidates: list[PooledDevice]) -> PooledDevice | None) and a registry mapping a strategy name (fifo_match default, reserved name capability_score for a future scoring strategy) to an implementation, the same "string key → pluggable implementation" shape already standardized on by driver/registry.py (device-agent-runtime-foundation D3), perception/provider.py (D8), and workflow/conditions.py's ConditionEvaluator (workflow-orchestration-runtime D6). TaskScheduler.submit(goal, constraints) enqueues a ScheduledTask{id, goal, constraints, status: queued|assigned|dispatched|done|failed}; TaskScheduler.assign() (called by a poll loop or triggered on submission/device-freed events) pops the head of the FIFO queue, asks DevicePool.list_devices() for idle candidates matching constraints.driver_type/constraints.capability_tags, and calls the configured AssignmentStrategy.select().

• Alternative considered: Hardcode FIFO-first-idle-match as the only assignment behavior with no strategy abstraction. Rejected — this is precisely the third occurrence of the same "pluggable-by-string-key" problem (driver selection, condition evaluation, now assignment strategy); making it a registry from the start costs one extra indirection and avoids an if/elif chain being added later when a capability-scoring strategy is inevitably wanted.

D5: TaskDispatcher composes TaskRunner/WorkflowRunner as black boxes; local-only in this change

cloud/dispatch.py's TaskDispatcher.dispatch(assignment: Assignment) builds a core.models.Task(goal=assignment.task.goal, device_id=assignment.device_id) (or, if the submission specified a workflow_definition_id instead of a bare goal, a WorkflowDefinition) and calls the existing, unmodified runtime.task.TaskRunner(...).run(task) or workflow.runner.WorkflowRunner(...).run(definition) — exactly the composition pattern workflow-orchestration-runtime's D8 already established for WorkflowRunner composing TaskRunner. If assignment.host_id does not match the local process's own host id, dispatch() raises RemoteDispatchNotSupportedError instead of attempting anything — this change's scheduler can decide a remote host should take a task, but only a local dispatch is actually executed here (see Non-Goals and Open Questions).

• Alternative considered: Build a lower-level dispatch loop that re-implements plan/execute/retry logic inside cloud/ instead of calling TaskRunner/WorkflowRunner. Rejected — duplicates already-implemented, already-tested execution logic and creates two places that must stay behaviorally consistent; composing the existing public entry points automatically inherits any future improvement to either runner with zero change to cloud/.

D6: PluginManifest/PluginRegistry discovers via importlib.metadata entry points and a local manifest-file scan, both feeding one validation+registration path

cloud/plugins.py's PluginManifest (name: str, version: str, entry_point_kind: Literal["driver", "tool", "skill"], target: str — a dotted module:attribute path) can be produced two ways: (a) a package installed in the environment declares an entry point in group device_agent_runtime.plugins whose value is itself a PluginManifest-shaped object or a callable returning one; (b) a plugins/<name>/plugin.json file on a configured scan path is parsed directly into a PluginManifest. Both paths converge on PluginRegistry.register(manifest), which validates shape/uniqueness (no duplicate name, entry_point_kind recognized) and then, only for entry_point_kind == "driver", resolves target to a DriverFactoryBuilder callable and calls driver_registry.register_driver_type(manifest.name, builder).

• Alternative considered: Support only Python entry points (no manifest-file scan), matching a more conventional Python-plugin-ecosystem approach (e.g., pluggy). Rejected for this change — requiring every plugin author to publish an installable Python package with an entry point is a real adoption barrier for a first plugin mechanism; a plain plugin.json file scan lets a plugin be "drop a folder in plugins/" as well, and both paths sharing one validation+registration function means neither is a second, divergent code path to maintain.
• Note (dependency on a not-yet-existing function): driver/registry.py (as specified by the pending driver-registry capability) today only defines SUPPORTED_DRIVER_TYPES (a plain dict) and build_driver_factory(); it does not yet expose a register_driver_type(name, builder) function. This change's driver-plugin registration path is written against that function by name, on the expectation that device-agent-runtime-foundation's own stated intent ("adding a new driver type is a one entry addition to driver/registry.py") is realized as a small public function, not only as "edit the dict inline." This is called out again in Open Questions since it is a real dependency on another pending change's still-evolving surface, not a formal spec delta this change is making to it.

D7: cloud/store.py's CloudStore is a new, independently-owned SQLite file, not new tables in storage/task_metadata.py or workflow/store.py

CloudStore opens its own file (default cloud/cloud.sqlite3) with host_registrations, pooled_devices, scheduled_tasks, and plugins tables, using the same connect-per-call sqlite3 pattern as storage/task_metadata.py and workflow/store.py.

• Alternative considered: Add scheduled_tasks/pooled_devices tables into workflow/store.py's existing workflows.sqlite3 (since a scheduled task's eventual execution may become a WorkflowRun). Rejected — workflow/ is owned by the pending, unapplied workflow-orchestration capability; adding schema/migration code to it without a corresponding spec delta would be a de facto modification of another change's owned artifact, the same reasoning workflow-orchestration-runtime's own D9 already used to justify not writing into storage/task_metadata.py. A separate, cloud/-owned store composed in prose (a scheduled_tasks row stores a task_id/workflow_run_id string reference once dispatched, no foreign key) is fully additive.

D8: platform-sdk's REST API is versioned via a URL prefix (/v1/...), with a single pluggable AuthProvider hook (no-op default), not a full auth system

cloud/sdk/api.py mounts all routes under /v1/ (e.g., POST /v1/tasks, GET /v1/tasks/{id}, GET /v1/devices, GET /v1/hosts, GET /v1/plugins, POST /v1/plugins). Every route accepts an optional AuthProvider.authenticate(request) -> Principal | None hook (default: a NullAuthProvider that always returns an anonymous Principal, i.e., no enforcement) so a later change can add real API-key/OAuth checking without changing route signatures.

• Alternative considered: Ship platform-sdk with no versioning prefix at all (bare /tasks, /devices, ...), deferring versioning until a breaking change is actually needed. Rejected — this is explicitly a public-facing surface for external integrators (unlike console-status-api, which is first-party/trusted-network only per web-console's own scope); starting unversioned and retrofitting a prefix later would break every existing integrator's URLs on day one of the first breaking change, which a public SDK should not do.
• Alternative considered: Build real API-key authentication now, since "public-facing" implies untrusted callers. Rejected for this change per its explicit Non-Goal — auth/authz is a distinct, non-trivial concern (key storage, rotation, revocation) that deserves its own change; shipping a no-op hook that is clearly the extension point (mirroring the Driver Registry/ConditionEvaluator/AssignmentStrategy pattern used everywhere else in this design) is enough to make "add real auth later" a localized change, not a rewrite.

D9: cloud/sdk/client.py is a thin requests/httpx-based wrapper, not a code-generated client

The Python SDK client (CloudClient) is hand-written, wrapping the same /v1/... routes cloud/sdk/api.py exposes, using httpx (already a dev dependency, promoted to a runtime dependency here since the client ships as installable code, not just test tooling).

• Alternative considered: Generate the client from an OpenAPI spec (FastAPI already produces one) via openapi-python-client or similar. Rejected for this change — adds a codegen build step and a new dev-tool dependency for a surface with only ~6 routes; hand-writing a thin wrapper is faster to ship and easier to read/review, and nothing here forecloses generating a client later once the API surface is larger and more churn-prone.

Risks / Trade-offs

• [Risk] Push-based host sync (D2) means DevicePool's view of a host's devices is only as fresh as that host's last sync call; a device could go busy→idle on its host moments after a sync and the pool won't know until the next sync interval → Mitigation: config.sync_interval_seconds is deliberately short-default and documented as eventually-consistent, not real-time; TaskScheduler.assign() re-checks DevicePool state immediately before dispatch (not just at submission time) to shrink the staleness window for the assignment decision itself, though a race between "assign" and "actually dispatch" is still possible and is accepted, not solved, in this change.
• [Risk] TaskDispatcher's local-only execution (D5) means TaskScheduler can select a PooledDevice owned by a remote host, but dispatch() will then raise RemoteDispatchNotSupportedError rather than executing it → Mitigation: AssignmentStrategy's default (fifo_match) is configured, in this change's reference wiring, to only consider PooledDevices whose host_id equals the dispatcher's own host id, so the common single-dispatcher-process deployment never hits the error path; multi-host dispatch is explicitly flagged in Open Questions as unimplemented, not silently broken.
• [Risk] plugin-system's driver-plugin registration path (D6) depends on driver/registry.py exposing a register_driver_type() function that does not exist yet in the pending driver-registry capability's own current spec (only a dict + a lookup function) → Mitigation: documented explicitly as a cross-change dependency in D6 and Open Questions rather than silently assumed; PluginRegistry.register() for driver-kind manifests fails loudly (raises, does not silently no-op) if driver_registry.register_driver_type is not importable, so the gap surfaces as an explicit error at plugin-load time, not a silent failure to actually register the driver.
• [Risk] Accepting tool/skill-kind plugin manifests in the schema (D6) without wiring them anywhere could mislead a plugin author into thinking registering a tool/skill plugin has an effect → Mitigation: PluginRegistry.register() for tool/skill-kind manifests stores the manifest (so it is listable via platform-sdk's GET /v1/plugins) but returns a wired: false field in its result and logs a clear "accepted, not yet wired to any execution path" message, rather than pretending success.
• [Trade-off] No real auth on platform-sdk (D8) means anything reachable at the API's network address can submit tasks/query status today → Acceptable for this change since it is not defining deployment/network topology (Non-Goals) and the AuthProvider hook makes adding real enforcement a scoped follow-up, not a redesign.
• [Trade-off] A fourth independent SQLite file (cloud/cloud.sqlite3, alongside tasks/tasks.sqlite3, workflows/workflows.sqlite3, and any skill-store databases) means fleet-level state, workflow state, and task-memory state are three separate stores with only string-id cross-references, no foreign-key enforcement → Acceptable; this mirrors the same trade-off workflow-orchestration-runtime's D9 already accepted for its own store relative to task-memory's, and unifying them is explicitly out of scope for any one milestone to force on another's still-pending schema.

Migration Plan

This is purely additive; no existing module is edited:

1. Add the cloud/ package: pool.py (DevicePool, HostRegistration, PooledDevice, sync_host_devices()), scheduler.py (TaskScheduler, ScheduledTask, AssignmentStrategy registry with fifo_match), dispatch.py (TaskDispatcher, Assignment, RemoteDispatchNotSupportedError), plugins.py (PluginManifest, PluginRegistry, entry-point + manifest-file discovery), store.py (CloudStore, schema creation for cloud/cloud.sqlite3), config.py (heartbeat/staleness/queue/strategy/API-version defaults).
2. Wire TaskDispatcher's planned-goal path to runtime.task.TaskRunner(...).run(task) and its workflow path to workflow.runner.WorkflowRunner(...).run(definition) — import only, zero edits to runtime/ or workflow/.
3. Wire PluginRegistry.register()'s driver-kind path to driver_registry.register_driver_type(name, builder) if importable; raise a clear, typed error naming the missing function if device-agent-runtime-foundation has not yet landed that function when this change is implemented (see D6's Note and Open Questions) — do not stub around it silently.
4. Add cloud/sdk/: models.py (Pydantic request/response models for the /v1/... routes), api.py (create_cloud_router(*, pool, scheduler, plugin_registry) -> APIRouter, mountable by whatever process assembles the full FastAPI app, same shape as api/console.py's create_console_router), client.py (CloudClient, thin httpx wrapper).
5. Add cloud* to pyproject.toml's [tool.setuptools.packages.find].include; promote httpx from [dependency-groups].dev to [project].dependencies (needed at runtime by cloud/sdk/client.py).
6. Add new tests (tests/test_device_pool.py, tests/test_task_scheduler.py, tests/test_task_dispatcher.py, tests/test_plugin_registry.py, tests/test_cloud_sdk_api.py-style): multi-host sync + staleness degrade, FIFO assignment with constraint matching, local dispatch composing a stub TaskRunner, remote-assignment raising RemoteDispatchNotSupportedError, plugin manifest validation + driver registration (with a fake driver_registry.register_driver_type), and REST route round-trips using FastAPI's TestClient.
7. Run pytest — the full existing suite stays green with zero edits to existing test files, confirming this change touches no existing behavior.
8. Rollback: net-new package plus a net-new SQLite file with its own schema; reverting is git revert of the commit(s), with no data migration of any existing store and no external system involved beyond whatever device the composed TaskRunner/WorkflowRunner calls already target.

Open Questions

• How real cross-host task dispatch should eventually work once RemoteDispatchNotSupportedError needs to actually go away — a lightweight per-host agent process that polls CloudStore for assignments targeting its own host_id and calls TaskDispatcher locally, versus a real RPC/HTTP call from the scheduler's process to a remote host's own dispatcher — left unresolved here since it requires the "no infra" Non-Goal to be revisited first.
• Whether driver/registry.py's register_driver_type() function (assumed by D6's driver-plugin registration path) should be proposed as an addition to device-agent-runtime-foundation before or alongside this change's implementation, given openspec/specs/ is empty and that change is itself still unapplied — flagged for whoever implements plugin-system to resolve with that change's owner rather than silently diverging.
• Whether tool/skill-kind plugin manifests should get a real wiring path in a follow-up change once tools/ and the skill stores (skill-catalog-subscription, skill-learning-runtime) gain their own public registration hooks — deliberately left declared-but-unwired here (D6, mirrors skill-learning-runtime's own precedent of leaving skill execution unbuilt until workflow-orchestration-runtime needed it).
• Whether ScheduledTask's constraints should eventually accept a skill_id/workflow_definition_id directly (submit "run this workflow on any matching device") versus only a bare goal string — this change's TaskDispatcher supports both today by inspecting which field is populated, but the scheduler's own matching logic (constraints → candidate devices) does not yet special-case workflow-shaped submissions differently from goal-shaped ones; worth revisiting once real workflow submissions are exercised through platform-sdk.
• Whether platform-sdk's AuthProvider hook should be promoted to a real auth capability in its own right (its own openspec change) once a concrete integrator/security requirement exists, rather than staying a documented no-op — intentionally left as a future decision, not prejudged here.