122 lines
12 KiB
Markdown
122 lines
12 KiB
Markdown
## Context
|
|
|
|
The archived cloud-runtime capabilities produced useful domain modules and tests, but they stop at an in-process boundary. `DevicePool.sync_host_devices()` is never exposed to a host process, `create_cloud_router()` is not mounted by a runnable cloud application, assignment is invoked manually, and `TaskDispatcher` rejects any assignment owned by another host. The completed `ai-planner-runtime` change now provides a production Planner path behind Runtime configuration, so a remote host can execute a cloud-submitted goal through the same `TaskRunner` used locally.
|
|
|
|
This change follows `uv-workspace-packaging`: the existing cloud domain remains in the outer `device-cloud-platform` distribution, while two thin application members provide composition and process lifecycle. Transport and persistence adapters remain outside `core`, `driver`, `device`, and `tools`.
|
|
|
|
## Goals / Non-Goals
|
|
|
|
**Goals:**
|
|
|
|
- Provide a deployable Cloud Control Plane process that exposes the public SDK API and owns scheduling, leases, persistence, and operational health.
|
|
- Provide a Device Host Agent process that makes outbound authenticated requests, synchronizes local device state, claims work for its own host, and executes through existing Runtime/workflow entry points.
|
|
- Prevent concurrent assignment of one device and recover work after host or control-plane restarts.
|
|
- Support PostgreSQL with migrations for deployment and SQLite for local development/tests through one repository contract.
|
|
- Require scoped authentication by default for public and host APIs.
|
|
- Preserve existing domain dependency direction and reuse existing scheduler, pool, plugin, Runtime, and workflow behavior where compatible.
|
|
|
|
**Non-Goals:**
|
|
|
|
- Exactly-once execution of device side effects.
|
|
- Multi-tenancy, billing, organization/user lifecycle, or a cloud administration frontend.
|
|
- A separate message broker, distributed scheduler cluster, or active-active control-plane deployment.
|
|
- Inbound network access to device hosts, WebSocket transport, or direct cloud access to WDA/Appium.
|
|
- Remote installation of arbitrary plugin packages; plugin registration only addresses already installed implementations.
|
|
- Task cancellation and live interactive device streaming in this change.
|
|
|
|
## Decisions
|
|
|
|
### D1: Two thin application members compose the existing packages
|
|
|
|
`apps/cloud-api` provides the `device-cloud-api` project and an app factory/CLI entry point. It depends on `device-cloud-platform` and composes the repository, pool, scheduler, plugin registry, auth providers, routers, and lifecycle workers.
|
|
|
|
`apps/device-host-agent` provides the `device-host-agent` project and CLI entry point. It depends on both `device-agent-runtime` and `device-cloud-platform`, because it adapts cloud protocol models to the local `DeviceManager`, `TaskRunner`, and `WorkflowRunner`.
|
|
|
|
The inner Runtime does not import either application or the cloud package. Alternative considered: mount cloud routes into `api/rest.py`. Rejected because that would make every local Runtime deployment own cloud persistence and scheduler lifecycle and would reverse the intended optional outer composition.
|
|
|
|
### D2: Host Agents use authenticated outbound long-polling
|
|
|
|
Each Host Agent periodically sends a heartbeat containing its address metadata and complete device snapshot, then long-polls an internal endpoint for work assigned to its `host_id`. The control plane never initiates a connection to the host.
|
|
|
|
Long-polling is selected over WebSockets because it works through common NAT/firewall configurations, is straightforward to recover after connection loss, and does not require connection-affinity infrastructure. Poll timeout and retry backoff are configurable. The protocol remains versioned under `/internal/v1` so a later streaming transport can coexist.
|
|
|
|
### D3: Assignments use database leases and at-least-once delivery
|
|
|
|
The existing task states remain externally recognizable: `queued`, `assigned`, `dispatched`, `done`, and `failed`. Assignment additionally stores `attempt_count`, `lease_id`, `lease_expires_at`, and result/failure metadata.
|
|
|
|
The scheduler atomically selects a queued task and eligible device, increments the attempt, creates a random lease, and marks the task `assigned`. An authenticated Host Agent for the owning host atomically claims that assignment and transitions it to `dispatched`; periodic renewal extends the same lease. Terminal result reporting succeeds only for the active lease and is idempotent when the same result is retried.
|
|
|
|
An expired `assigned` or `dispatched` lease is requeued while attempts remain, otherwise it becomes `failed`. Active assignments reserve their devices independently of possibly stale host snapshots, preventing a subsequent scheduler iteration from assigning the same apparently-idle device.
|
|
|
|
This is at-least-once execution. A host can perform a device side effect immediately before losing its lease, after which the task may be retried. Alternative considered: claim exactly-once semantics. Rejected because the control plane cannot transactionally coordinate its database with external device effects. Mitigation is short leases with renewal, stopping execution after renewal failure where possible, bounded attempts, and clear attempt/result history.
|
|
|
|
### D4: Host execution reuses existing TaskRunner and WorkflowRunner
|
|
|
|
For a goal assignment, the Host Agent creates a `Task` for the assigned device and invokes a locally composed `TaskRunner`. Runtime configuration determines whether the completed AI Planner implementation is enabled and which provider/model it uses. For a workflow assignment, it loads and invokes the existing `WorkflowRunner` contract.
|
|
|
|
The Host Agent reports normalized terminal status, failure reason, and execution metadata to the control plane. It does not reimplement planning, tool execution, retries, perception, or driver access.
|
|
|
|
### D5: One repository contract supports SQLite and PostgreSQL
|
|
|
|
The cloud package defines a repository protocol/facade covering hosts, devices, tasks, leases, plugins, and transactional assignment operations. A SQLAlchemy 2 implementation supports SQLite URLs for local/test use and PostgreSQL URLs for deployment. Alembic owns versioned schema migrations.
|
|
|
|
Synchronous SQLAlchemy sessions match the existing synchronous FastAPI handlers and Runtime execution model. Database-specific locking is isolated in repository methods: PostgreSQL uses row locking/skip-locked where appropriate; SQLite uses transactions suitable for the documented single-control-plane development mode.
|
|
|
|
Alternative considered: retain direct `sqlite3` and add a separate psycopg implementation. Rejected because duplicating schema and transactional logic would make lease correctness diverge between development and deployment.
|
|
|
|
### D6: Authentication is required and scope-aware
|
|
|
|
Public `/v1` routes use bearer principals with explicit scopes such as `tasks:submit`, `tasks:read`, `pool:read`, and `plugins:admin`. Host `/internal/v1` routes use host credentials bound to exactly one `host_id`; a host cannot synchronize or claim work for another identity.
|
|
|
|
The existing `AuthProvider` extension point is retained and expanded to return scopes. The default production composition uses configured bearer credentials and fails startup when none are configured. Anonymous access is available only through an explicit insecure-development flag and must be rejected when the environment is marked production.
|
|
|
|
Plugin listing may use a read scope, while plugin registration requires `plugins:admin` because resolving an installed entry point can load code. Credentials are compared in constant time and must not be emitted in logs.
|
|
|
|
### D7: Control-plane lifecycle owns scheduler and lease-reaper loops
|
|
|
|
The cloud app factory uses FastAPI lifespan to validate configuration, apply or verify migrations according to deployment policy, initialize dependencies, start a periodic scheduler loop and lease-reaper loop, and stop both cleanly. Each loop catches and records iteration failures without terminating the process or silently skipping future work.
|
|
|
|
Only one control-plane scheduler is supported for SQLite. PostgreSQL transactional assignment prevents duplicate claims if multiple API processes are later run, but active-active scheduler leadership is not claimed by this change and deployment documentation defaults to one scheduler-enabled process.
|
|
|
|
### D8: Health separates process liveness from readiness
|
|
|
|
`/health/live` reports that the process event loop is running and does not require database access. `/health/ready` verifies configuration, database connectivity/schema version, and lifecycle worker state. Readiness fails when the control plane cannot safely accept tasks.
|
|
|
|
The Host Agent exposes local process health only when explicitly configured; its primary observable state is structured heartbeat and execution logging.
|
|
|
|
### D9: Result and retry operations are idempotent
|
|
|
|
Heartbeat/snapshot replacement, assignment claim, lease renewal, and result reporting use stable host/task/lease identifiers. Retrying the same terminal result for the same lease returns the recorded outcome. A stale or foreign lease receives a conflict response and cannot overwrite a newer attempt.
|
|
|
|
### D10: Operational logging uses correlation identifiers
|
|
|
|
Structured logs include task id, host id, device id, attempt, and lease id where applicable, while excluding bearer credentials, screenshots, UI trees, and text input. Public and internal requests accept or generate a correlation id propagated into task lifecycle logs.
|
|
|
|
## Risks / Trade-offs
|
|
|
|
- [Risk] At-least-once retry can repeat a device action. -> Mitigation: lease renewal, bounded attempts, stop-on-lease-loss behavior, and visible attempt history; do not claim exactly-once guarantees.
|
|
- [Risk] Host snapshots can lag actual device state. -> Mitigation: active assignments reserve devices in the database and heartbeat staleness continues to mark unreachable hosts.
|
|
- [Risk] PostgreSQL and SQLite transaction semantics differ. -> Mitigation: centralize operations behind repository contract tests run against both engines, with concurrency tests required for PostgreSQL.
|
|
- [Risk] A mis-scoped token could expose plugin loading. -> Mitigation: deny anonymous access by default and require the separate `plugins:admin` scope.
|
|
- [Risk] Background loops inside the API process complicate horizontal scaling. -> Mitigation: document a single scheduler-enabled deployment and keep lifecycle services injectable so they can move to a dedicated worker later.
|
|
- [Trade-off] Long-poll adds request overhead compared with a broker or WebSocket. -> Accepted for the first deployable version because it is operationally simpler and NAT-friendly.
|
|
- [Risk] The Host Agent may be configured without AI provider credentials. -> Mitigation: readiness/config diagnostics report Planner configuration, and task failures preserve explicit Runtime failure reasons.
|
|
|
|
## Migration Plan
|
|
|
|
1. Apply and verify `uv-workspace-packaging` so cloud and application dependency ownership is stable.
|
|
2. Introduce the repository contract, SQLAlchemy models, and baseline migration while preserving existing `CloudStore` behavior through an adapter.
|
|
3. Add lease/reservation semantics and test scheduler/reaper transactions on SQLite and PostgreSQL.
|
|
4. Add scoped auth providers and the internal Host Agent protocol endpoints.
|
|
5. Add the `cloud-api` composition/lifespan application and health endpoints.
|
|
6. Add the `device-host-agent` sync, long-poll, renewal, execution, and result-reporting loop.
|
|
7. Extend the public SDK task status surface and update the Python client authentication support.
|
|
8. Run local single-host, multi-host fake-driver, restart-recovery, PostgreSQL concurrency, and full non-integration regression tests.
|
|
9. Publish container/deployment configuration and an upgrade procedure that runs database migrations before the new control plane accepts traffic.
|
|
|
|
Rollback requires stopping Host Agents first, then the new cloud API. Database migrations must include tested downgrade paths until the release is accepted; queued tasks can be exported or left in the database for a forward redeploy. The previous in-process cloud modules remain usable only for development and do not consume the new remote assignments.
|
|
|
|
## Open Questions
|
|
|
|
- The initial implementation should select concrete default lease, poll, and retry durations based on integration tests; all remain configuration values rather than protocol constants.
|