docs(openspec): sync cloud control plane specs
This commit is contained in:
@@ -0,0 +1,77 @@
|
||||
# cloud-control-plane Specification
|
||||
|
||||
## Purpose
|
||||
Provide a runnable, authenticated Cloud Control Plane with durable scheduling,
|
||||
lease maintenance, health reporting, and restart recovery.
|
||||
|
||||
## Requirements
|
||||
|
||||
### Requirement: Runnable Cloud Control Plane application
|
||||
The system SHALL provide an independently runnable Cloud Control Plane application that composes the cloud repository, device pool, task scheduler, plugin registry, authentication providers, public platform router, internal Host Agent router, and lifecycle services without modifying the local Runtime API application.
|
||||
|
||||
#### Scenario: Start the cloud application
|
||||
- **WHEN** an operator starts the Cloud Control Plane with valid configuration and an available database
|
||||
- **THEN** the application exposes its versioned public, internal, and health routes and starts its configured lifecycle workers
|
||||
|
||||
#### Scenario: Local Runtime remains separately runnable
|
||||
- **WHEN** an operator starts the existing local Runtime API without the cloud application
|
||||
- **THEN** local Runtime routes operate without initializing cloud persistence, scheduling, or Host Agent services
|
||||
|
||||
### Requirement: Scheduler and lease maintenance run through application lifecycle
|
||||
The Cloud Control Plane SHALL run configurable scheduler and expired-lease maintenance loops after application startup and SHALL stop them cleanly during shutdown.
|
||||
|
||||
#### Scenario: Queued work becomes assigned
|
||||
- **WHEN** a queued task has an eligible device and the scheduler loop runs
|
||||
- **THEN** the task receives an atomic assignment and becomes available to the owning Host Agent
|
||||
|
||||
#### Scenario: Lifecycle iteration fails transiently
|
||||
- **WHEN** one scheduler or lease-maintenance iteration raises an operational error
|
||||
- **THEN** the error is recorded and subsequent configured iterations continue rather than permanently terminating the worker
|
||||
|
||||
#### Scenario: Application shuts down
|
||||
- **WHEN** the Cloud Control Plane receives a graceful shutdown signal
|
||||
- **THEN** its lifecycle workers stop accepting new iterations and terminate without abandoning an in-process database transaction
|
||||
|
||||
### Requirement: Deployment and local persistence modes share one contract
|
||||
The cloud repository SHALL support PostgreSQL for deployed operation and SQLite for local development and tests through the same behavioral contract, including hosts, devices, tasks, leases, attempts, and plugins.
|
||||
|
||||
#### Scenario: Start with PostgreSQL
|
||||
- **WHEN** the configured database URL selects PostgreSQL and the schema is current
|
||||
- **THEN** the control plane uses PostgreSQL for all cloud state and transactional assignment operations
|
||||
|
||||
#### Scenario: Start in local SQLite mode
|
||||
- **WHEN** the configured database URL selects SQLite in a local or test environment
|
||||
- **THEN** the same repository contract is available with the documented single-control-plane concurrency limitation
|
||||
|
||||
### Requirement: Cloud schema is versioned with migrations
|
||||
The system SHALL provide versioned forward and downgrade database migrations and SHALL refuse readiness when the database schema is incompatible with the running application.
|
||||
|
||||
#### Scenario: Upgrade an existing cloud database
|
||||
- **WHEN** an operator applies the release's migrations to a supported previous schema
|
||||
- **THEN** existing hosts, devices, tasks, and plugins are retained and the new lease fields become available
|
||||
|
||||
#### Scenario: Schema is behind at startup
|
||||
- **WHEN** the application connects to a database whose schema version is not accepted by the running release
|
||||
- **THEN** readiness fails with a diagnostic that does not expose credentials
|
||||
|
||||
### Requirement: Restart recovery preserves durable work
|
||||
The Cloud Control Plane SHALL recover persisted queued tasks and SHALL requeue or fail expired assigned/dispatched attempts according to retry policy after process restart.
|
||||
|
||||
#### Scenario: Restart with queued tasks
|
||||
- **WHEN** the control plane restarts while tasks are queued
|
||||
- **THEN** those tasks remain queued and are considered by later scheduler iterations
|
||||
|
||||
#### Scenario: Restart after a lease expires
|
||||
- **WHEN** the control plane restarts and finds an assigned or dispatched task with an expired lease
|
||||
- **THEN** lease maintenance requeues it when attempts remain or marks it failed when the retry limit is exhausted
|
||||
|
||||
### Requirement: Liveness, readiness, and safe operational logging
|
||||
The application SHALL expose separate liveness and readiness endpoints and SHALL emit structured lifecycle logs with correlation identifiers while excluding credentials and sensitive device payloads.
|
||||
|
||||
#### Scenario: Database is unavailable
|
||||
- **WHEN** the process is running but cannot reach its configured database
|
||||
- **THEN** liveness succeeds and readiness fails
|
||||
|
||||
#### Scenario: Task lifecycle is logged
|
||||
- **WHEN** a task is assigned, claimed, renewed, completed, retried, or failed
|
||||
- **THEN** the log event includes available task, host, device, attempt, lease, and correlation identifiers but excludes bearer tokens, screenshots, UI trees, and typed text
|
||||
@@ -47,3 +47,24 @@ The system SHALL allow looking up a single `PooledDevice` by `device_id` regardl
|
||||
- **WHEN** a caller requests a device by id that no host has ever synced
|
||||
- **THEN** the pool returns a not-found result (e.g. `None`) rather than raising an unhandled exception
|
||||
|
||||
### Requirement: Authenticated network synchronization feeds the device pool
|
||||
The system SHALL expose an authenticated Host Agent operation that validates a host device snapshot and delegates it to the existing device-pool synchronization behavior.
|
||||
|
||||
#### Scenario: Valid remote snapshot
|
||||
- **WHEN** an authenticated Host Agent submits a valid complete snapshot for its bound host id
|
||||
- **THEN** the device pool refreshes that host and its devices with the same replacement and staleness semantics as an in-process synchronization call
|
||||
|
||||
#### Scenario: Invalid snapshot is rejected atomically
|
||||
- **WHEN** a Host Agent snapshot contains invalid device identifiers, driver types, statuses, or capability tags
|
||||
- **THEN** the control plane rejects the snapshot without partially replacing the host's previously stored devices
|
||||
|
||||
### Requirement: Device identity ownership conflicts are explicit
|
||||
The device pool SHALL reject a snapshot that claims a `device_id` actively owned by a different non-stale host, rather than silently transferring ownership.
|
||||
|
||||
#### Scenario: Two live hosts report the same device id
|
||||
- **WHEN** host B reports a device id currently owned by non-stale host A
|
||||
- **THEN** host B's conflicting snapshot is rejected with an ownership-conflict response and host A retains ownership
|
||||
|
||||
#### Scenario: Previous owner is stale
|
||||
- **WHEN** a configured ownership-recovery policy permits takeover and the prior owning host is stale beyond the recovery threshold
|
||||
- **THEN** the new host may claim the device id and the ownership transition is recorded
|
||||
|
||||
@@ -0,0 +1,84 @@
|
||||
# host-agent-protocol Specification
|
||||
|
||||
## Purpose
|
||||
Define the authenticated outbound protocol used by Device Host Agents to
|
||||
synchronize devices, claim leased work, execute it locally, and report results.
|
||||
|
||||
## Requirements
|
||||
|
||||
### Requirement: Host identity is authenticated and bound to one host id
|
||||
The internal Host Agent API SHALL require a host-scoped bearer principal and SHALL reject any request that attempts to act for a `host_id` different from the authenticated principal's bound host.
|
||||
|
||||
#### Scenario: Host authenticates as itself
|
||||
- **WHEN** a Host Agent presents valid credentials bound to its requested `host_id`
|
||||
- **THEN** the internal API authorizes permitted heartbeat, claim, renewal, and result operations
|
||||
|
||||
#### Scenario: Host attempts to impersonate another host
|
||||
- **WHEN** valid credentials bound to host A are used on a request for host B
|
||||
- **THEN** the internal API rejects the request without reading or modifying host B's state
|
||||
|
||||
### Requirement: Host Agent synchronizes heartbeat and complete device snapshots
|
||||
The Host Agent SHALL periodically submit its complete local device snapshot to the control plane, and the control plane SHALL atomically refresh the host heartbeat and replace only that host's pooled-device records.
|
||||
|
||||
#### Scenario: Host reports devices
|
||||
- **WHEN** a Host Agent submits a valid heartbeat containing its current devices
|
||||
- **THEN** the control plane updates the host's last-seen time and exposes the submitted devices through the aggregated pool
|
||||
|
||||
#### Scenario: Host reports no devices
|
||||
- **WHEN** a previously populated host submits an empty device snapshot
|
||||
- **THEN** only that host's prior device records are removed while devices owned by other hosts remain unchanged
|
||||
|
||||
### Requirement: Host Agent receives work through outbound long-polling
|
||||
The Host Agent SHALL request assigned work for its own host through a configurable long-poll endpoint, and the control plane SHALL return at most one atomically claimable assignment per response or an empty timeout response.
|
||||
|
||||
#### Scenario: Assigned work is available
|
||||
- **WHEN** a Host Agent long-polls and an unclaimed assignment exists for its host
|
||||
- **THEN** the control plane atomically transitions the assignment to dispatched and returns its task, device, attempt, lease, and execution payload
|
||||
|
||||
#### Scenario: No work becomes available
|
||||
- **WHEN** no assignment for the host becomes available before the configured poll timeout
|
||||
- **THEN** the endpoint returns a normal empty response and the Host Agent may poll again with backoff
|
||||
|
||||
### Requirement: Active execution renews its lease
|
||||
The Host Agent SHALL renew the active assignment lease before expiry while execution continues, and SHALL treat loss or rejection of the lease as a stop condition for further planned actions where interruption is possible.
|
||||
|
||||
#### Scenario: Lease renewal succeeds
|
||||
- **WHEN** the owning Host Agent renews an unexpired active lease
|
||||
- **THEN** the control plane extends its expiry without changing the task attempt or device assignment
|
||||
|
||||
#### Scenario: Lease is stale or foreign
|
||||
- **WHEN** a Host Agent attempts to renew an expired, replaced, or differently owned lease
|
||||
- **THEN** the control plane returns a conflict and does not revive or alter the current attempt
|
||||
|
||||
### Requirement: Host execution composes existing Runtime and workflow runners
|
||||
The Host Agent SHALL execute goal assignments through the existing `TaskRunner` and workflow assignments through the existing `WorkflowRunner`, using its local `DeviceManager` and Runtime configuration rather than reimplementing execution behavior.
|
||||
|
||||
#### Scenario: Execute a goal assignment
|
||||
- **WHEN** the Host Agent claims a goal-based assignment for a connected local device
|
||||
- **THEN** it runs a `Task` through the configured Runtime Planner/Executor loop and captures the terminal status and failure reason
|
||||
|
||||
#### Scenario: Execute a workflow assignment
|
||||
- **WHEN** the Host Agent claims an assignment referencing an available workflow definition
|
||||
- **THEN** it invokes the existing workflow runner for the assigned device and captures the terminal workflow outcome
|
||||
|
||||
### Requirement: Terminal result reporting is idempotent
|
||||
The Host Agent SHALL report a terminal result using the task, attempt, and lease identifiers, and repeating the same report SHALL return the already recorded outcome without duplicating state transitions.
|
||||
|
||||
#### Scenario: Report a successful result
|
||||
- **WHEN** the active lease owner reports successful completion
|
||||
- **THEN** the control plane marks the scheduled task done, releases the device reservation, and records the result metadata
|
||||
|
||||
#### Scenario: Retry a result after response loss
|
||||
- **WHEN** the Host Agent repeats the identical terminal report for an already completed active lease
|
||||
- **THEN** the control plane returns the recorded terminal result without creating a new attempt or error
|
||||
|
||||
#### Scenario: Stale attempt reports after requeue
|
||||
- **WHEN** an expired earlier attempt reports after a newer attempt has been created
|
||||
- **THEN** the control plane rejects the stale report and preserves the newer attempt's state
|
||||
|
||||
### Requirement: Host operation requires no inbound cloud connection
|
||||
The Host Agent SHALL perform synchronization, work retrieval, lease renewal, and result reporting using outbound requests only.
|
||||
|
||||
#### Scenario: Host is behind NAT
|
||||
- **WHEN** the Host Agent can reach the control-plane URL but exposes no inbound listener
|
||||
- **THEN** it can register devices and execute cloud assignments through the complete protocol
|
||||
@@ -48,15 +48,19 @@ The system SHALL allow an external integrator to list registered plugins and sub
|
||||
- **THEN** the API registers it via `plugin-system`'s `PluginRegistry` and returns the stored manifest, or a clear validation/conflict error if registration fails
|
||||
|
||||
### Requirement: Pluggable authentication hook with a safe default
|
||||
The system SHALL evaluate every platform SDK route through a configurable `AuthProvider` hook, defaulting to a no-op provider that treats every caller as an anonymous, authenticated principal, so real authentication can be added later without changing route signatures.
|
||||
The system SHALL evaluate every platform SDK route through a configurable scope-aware `AuthProvider` hook, and the deployable Cloud Control Plane SHALL reject anonymous access unless an explicit insecure-development override is enabled outside production.
|
||||
|
||||
#### Scenario: Default configuration allows anonymous access
|
||||
- **WHEN** no `AuthProvider` is explicitly configured
|
||||
- **THEN** every route accepts requests without rejecting them for lack of credentials
|
||||
#### Scenario: Production starts without configured credentials
|
||||
- **WHEN** the Cloud Control Plane is configured as production without a usable public authentication provider or credentials
|
||||
- **THEN** startup or readiness fails rather than exposing anonymous platform routes
|
||||
|
||||
#### Scenario: Explicit local anonymous override
|
||||
- **WHEN** a non-production operator explicitly enables the insecure anonymous-development override
|
||||
- **THEN** platform routes may use an anonymous principal and the application records that insecure mode is active
|
||||
|
||||
#### Scenario: Custom AuthProvider is honored
|
||||
- **WHEN** a caller configures a custom `AuthProvider` that rejects a request
|
||||
- **THEN** the platform SDK's routes return an authorization error for that request instead of proceeding, without any route's own handler code needing to change
|
||||
- **WHEN** a caller configures a custom `AuthProvider` that rejects a request or omits its required scope
|
||||
- **THEN** the platform SDK route returns an authentication or authorization error without executing its handler operation
|
||||
|
||||
### Requirement: Python SDK client mirrors the REST API
|
||||
The system SHALL provide a Python client (`CloudClient`) exposing methods corresponding to each `/v1/...` route (submit task, get task status, list devices, list hosts, list plugins, register plugin), so integrators do not need to hand-construct HTTP requests.
|
||||
@@ -65,3 +69,35 @@ The system SHALL provide a Python client (`CloudClient`) exposing methods corres
|
||||
- **WHEN** a caller uses `CloudClient` to submit a task and then fetch its status by the returned id
|
||||
- **THEN** the client's methods produce the same result as calling the corresponding `/v1/...` endpoints directly over HTTP
|
||||
|
||||
### Requirement: Public API operations enforce scopes
|
||||
The public platform API SHALL require operation-specific scopes, including task submission, task reading, pool reading, plugin reading, and plugin administration.
|
||||
|
||||
#### Scenario: Submit token has task scope
|
||||
- **WHEN** a principal with `tasks:submit` calls the task-submission endpoint
|
||||
- **THEN** the request is authorized subject to normal task validation
|
||||
|
||||
#### Scenario: Non-admin token attempts plugin registration
|
||||
- **WHEN** an authenticated principal without `plugins:admin` calls plugin registration
|
||||
- **THEN** the API rejects the request before resolving or loading the plugin target
|
||||
|
||||
### Requirement: Distributed task status exposes attempt outcomes
|
||||
The task-status API SHALL expose the existing lifecycle status and SHALL include non-secret assignment, attempt, and terminal failure metadata needed to diagnose distributed execution.
|
||||
|
||||
#### Scenario: Query an active remote task
|
||||
- **WHEN** an authorized caller queries an assigned or dispatched task
|
||||
- **THEN** the response includes its status, assigned host/device, current attempt number, and lease expiry without exposing the lease credential
|
||||
|
||||
#### Scenario: Query a failed remote task
|
||||
- **WHEN** an authorized caller queries a task that exhausted retries or failed during Runtime execution
|
||||
- **THEN** the response includes the terminal failure reason and attempt count
|
||||
|
||||
### Requirement: Python SDK supports authenticated requests
|
||||
The Python `CloudClient` SHALL accept bearer credentials or an injectable authentication mechanism and SHALL apply authentication consistently to every public API method.
|
||||
|
||||
#### Scenario: Client configured with bearer token
|
||||
- **WHEN** a caller constructs `CloudClient` with a valid bearer token and invokes a permitted method
|
||||
- **THEN** the client sends the authorization credential and returns the corresponding API result
|
||||
|
||||
#### Scenario: Client receives authorization failure
|
||||
- **WHEN** the configured credential is missing, invalid, or lacks the required scope
|
||||
- **THEN** the client raises a typed HTTP/API error that preserves the response status without exposing the credential
|
||||
|
||||
@@ -51,10 +51,46 @@ The system SHALL provide a `TaskDispatcher` that, for an assignment whose device
|
||||
- **WHEN** `TaskDispatcher.dispatch()` is called with an assignment referencing a `WorkflowDefinition` whose device is local
|
||||
- **THEN** the dispatcher runs the definition through the existing workflow-execution entry point and updates the `ScheduledTask`'s status based on the resulting workflow run's outcome
|
||||
|
||||
### Requirement: Remote assignments are rejected explicitly, not silently ignored
|
||||
The system SHALL raise a distinct, typed error when `TaskDispatcher.dispatch()` is called for an assignment whose device is owned by a host other than the dispatching process's own host, rather than attempting execution or silently no-op'ing.
|
||||
### Requirement: Assignment and device reservation are atomic
|
||||
The scheduler SHALL atomically bind a queued task to one eligible device, create a bounded lease attempt, and reserve that device so no other active task can be assigned to it.
|
||||
|
||||
#### Scenario: Assignment targets a remote host's device
|
||||
- **WHEN** `TaskDispatcher.dispatch()` is called with an assignment whose `host_id` does not match the local process's own host id
|
||||
- **THEN** the dispatcher raises a `RemoteDispatchNotSupportedError` and leaves the `ScheduledTask`'s status unchanged from `assigned`
|
||||
#### Scenario: Scheduler assigns an idle device
|
||||
- **WHEN** a queued task matches an idle pooled device with no active reservation
|
||||
- **THEN** one transaction records the assigned task, owning host/device, incremented attempt, lease identifier, lease expiry, and device reservation
|
||||
|
||||
#### Scenario: Later scheduler iteration sees stale idle snapshot
|
||||
- **WHEN** the host snapshot still reports a device idle but that device has an active assignment lease
|
||||
- **THEN** the scheduler excludes the device from candidates for every other queued task
|
||||
|
||||
### Requirement: Host claim transitions assigned work to dispatched
|
||||
The scheduler repository SHALL allow only the authenticated owning host to atomically claim an unexpired assigned attempt and transition it to `dispatched`.
|
||||
|
||||
#### Scenario: Owning host claims once
|
||||
- **WHEN** the owning host requests available work and an unexpired assigned attempt exists
|
||||
- **THEN** exactly one request receives the assignment and its status becomes dispatched
|
||||
|
||||
#### Scenario: Concurrent claims race
|
||||
- **WHEN** multiple requests concurrently attempt to claim the same assignment
|
||||
- **THEN** at most one request succeeds and every other request receives no assignment or a conflict
|
||||
|
||||
### Requirement: Expired attempts follow bounded retry policy
|
||||
The system SHALL detect expired assigned or dispatched leases and SHALL either requeue the task with its reservation released or mark it failed when the configured attempt limit is reached.
|
||||
|
||||
#### Scenario: Lease expires with attempts remaining
|
||||
- **WHEN** an active lease expires before a terminal result and the task has remaining attempts
|
||||
- **THEN** the task returns to queued, the previous device reservation is released, and the expired attempt remains auditable
|
||||
|
||||
#### Scenario: Lease expires at attempt limit
|
||||
- **WHEN** an active lease expires and the task has reached its maximum attempts
|
||||
- **THEN** the task becomes failed with a lease-expiry reason and its device reservation is released
|
||||
|
||||
### Requirement: Terminal transitions validate the active lease
|
||||
The system SHALL accept a `done` or `failed` result only from the current active task attempt and lease and SHALL make repeated identical terminal reports idempotent.
|
||||
|
||||
#### Scenario: Active lease reports completion
|
||||
- **WHEN** the active lease owner reports a terminal result
|
||||
- **THEN** the task transitions once to done or failed and releases its device reservation
|
||||
|
||||
#### Scenario: Superseded lease reports completion
|
||||
- **WHEN** a result references a lease superseded by expiry and retry
|
||||
- **THEN** the result is rejected and cannot overwrite the current task attempt
|
||||
|
||||
Reference in New Issue
Block a user