feat(cloud): add targeted task governance foundation
Tests / Test passed: 659

This commit is contained in:
2026-07-13 22:21:12 +08:00
parent a3ba94be04
commit 2cd314b183
41 changed files with 2099 additions and 15 deletions
@@ -0,0 +1,50 @@
## MODIFIED Requirements
### Requirement: Pluggable dual-provider tool-calling abstraction
The system SHALL support at least two interchangeable LLM providers
(Anthropic native tool use and OpenAI function calling) for the AI Planner's
decision calls, selectable via configuration, with both providers constrained
to return exactly one tool call per request. Independently of provider
selection, the system SHALL support at least two transports for making that
decision call -- direct-to-provider (the tool-calling client calls the
provider's SDK itself, using locally configured credentials) and cloud-proxy
(the tool-calling client calls the Cloud Control Plane's planner-decision
endpoint, which calls the provider using cloud-held credentials) -- selectable
via configuration without requiring any change to `AIPlanner`'s own decision
logic. A provider decision result SHALL also carry available non-secret
provider token-usage metadata so outer Cloud adapters can meter calls without
changing that decision logic.
#### Scenario: Provider selected via configuration
- **WHEN** the AI Planner is configured with a given provider identifier
- **THEN** it constructs and uses the tool-calling client for that provider
without requiring any change to `AIPlanner`'s own decision logic
#### Scenario: Provider response resolves to a single decision
- **WHEN** either supported provider returns a response to a tool-calling
request
- **THEN** the response is parsed into exactly one tool name and one arguments
object, regardless of which provider produced it
#### Scenario: Transport selected via configuration
- **WHEN** the Host Agent is configured with a given transport (direct or
cloud-proxy)
- **THEN** `AIPlanner` is constructed with the tool-calling client for that
transport, and its own decision logic is unchanged regardless of which
transport is in effect
#### Scenario: Direct transport remains available and default
- **WHEN** no transport is explicitly configured
- **THEN** the AI Planner uses the direct-to-provider transport, matching its
behavior before the cloud-proxy transport existed
#### Scenario: Cloud-proxy transport resolves a decision without a local provider client
- **WHEN** the Host Agent is configured with the cloud-proxy transport
- **THEN** its tool-calling client sends the decision request to the Cloud
Control Plane's planner-decision endpoint instead of constructing a local
Anthropic or OpenAI SDK client
#### Scenario: Provider reports token usage
- **WHEN** a provider response includes input or output token counts
- **THEN** the tool-calling result makes those non-secret counts available to
its outer adapter while `AIPlanner` selects the same single decision
@@ -0,0 +1,52 @@
## ADDED Requirements
### Requirement: Cloud-proxied planner calls produce durable non-secret usage events
After the Cloud planner proxy resolves a provider call, the system SHALL record
a durable usage event with Host, provider/model, provider-reported token
counts, request outcome, and known task-attempt context, while excluding raw
prompts, screenshots, provider credentials, cookies, and lease secrets.
#### Scenario: Provider returns token usage
- **WHEN** a Cloud-proxied planner call completes with provider-reported token
counts
- **THEN** the system stores those counts against the authenticated Host and
returns the planner decision without exposing secret request content
#### Scenario: Assignment context is available
- **WHEN** the Host makes a proxied planner call while executing a claimed
assignment
- **THEN** the recorded event includes that task id and attempt identifier
### Requirement: Host token budgets use atomic reservation and settlement
The Cloud planner proxy SHALL atomically reserve a conservative per-call token
amount before a provider invocation for every Host with an effective AI token
budget using Cloud planner transport, SHALL reject over-budget calls before
invoking the provider, and SHALL settle the reservation to provider-reported
usage when available.
#### Scenario: Remaining budget permits a call
- **WHEN** used tokens plus active reservations and the next conservative
reservation are within the Host's UTC-day budget
- **THEN** the proxy reserves budget, invokes the provider, and settles the
reservation after the provider response
#### Scenario: Remaining budget is insufficient
- **WHEN** the next reservation would exceed the Host's effective budget
- **THEN** the proxy rejects the planner call without invoking the provider
#### Scenario: Provider outcome has unknown usage
- **WHEN** a transport failure prevents the proxy from determining actual
provider usage after reservation
- **THEN** the reservation remains active until bounded expiry cleanup rather
than being released optimistically
### Requirement: Direct planner transport is explicitly unmetered
The system SHALL represent a Host using direct-to-provider planner transport as
unmetered for Cloud token accounting and SHALL not report it as complying with
a Cloud-enforced token budget.
#### Scenario: Console inspects a direct-transport Host
- **WHEN** an authorized operator views AI usage for a Host not using Cloud
planner transport
- **THEN** the system reports the Host as unmetered rather than zero usage or
hard-budget-enforced
@@ -0,0 +1,50 @@
## ADDED Requirements
### Requirement: Console composes explicitly targeted tasks
The Console SHALL provide a task-creation flow to principals with
`tasks:submit`, exposing goal/workflow input, Host selection, Device selection
scoped to the selected Host, and visible driver/capability constraints.
#### Scenario: Operator creates a Device-targeted task
- **WHEN** an authorized operator selects a Host and one of its Devices and
submits a valid task
- **THEN** the Console sends the explicit target to the public task API and
displays the resulting queued task with that target
#### Scenario: Restricted operator has no permitted target
- **WHEN** the Cloud API reports that the operator's policy disallows the
selected target or targetless submission
- **THEN** the Console shows the authorization error and does not imply that
the task was accepted or rerouted
### Requirement: Console administers governance policy within effective scopes
The Console SHALL expose user-submission and Host-policy views only to a
principal with `governance:admin`, and SHALL display effective non-secret
policy revisions and mutations without treating hidden controls as security.
#### Scenario: Administrator changes a Host budget
- **WHEN** a governance administrator saves a valid Host policy containing an
AI budget or active-task limit
- **THEN** the Console displays the returned newer policy revision and its
effective values
#### Scenario: Non-governance user opens the Console
- **WHEN** the current principal lacks `governance:admin`
- **THEN** the Console hides governance administration navigation while the
backend remains responsible for rejecting unauthorized requests
### Requirement: Console reports Host AI budget state without sensitive prompts
The Console SHALL render per-Host budget state and non-secret token usage for
authorized governance users, including an explicit unmetered state for Hosts
using direct planner transport.
#### Scenario: Administrator views a Cloud-proxied Host
- **WHEN** a governance administrator selects a Host with recorded proxy usage
- **THEN** the Console displays configured budget, used/reserved/remaining
tokens, provider/model, and non-secret event metadata
#### Scenario: Administrator views a direct-transport Host
- **WHEN** a governance administrator selects a Host that does not use Cloud
planner transport
- **THEN** the Console labels the Host unmetered and does not display zero as
a budget-compliant usage value
@@ -0,0 +1,53 @@
## ADDED Requirements
### Requirement: User task-submission policy is durable and authoritative
The system SHALL persist an optional versioned submission policy for each
human user, containing submission enablement and explicit Host/Device target
allow-lists, and SHALL enforce that policy after scope authentication on every
public task-submission request.
#### Scenario: Restricted user submits to an allowed Device
- **WHEN** a principal with `tasks:submit` has a policy allowing a Host/Device
target and submits a task for that exact target
- **THEN** the system accepts the task subject to normal validation and records
the authenticated principal as its submitter
#### Scenario: Restricted user omits or exceeds a target allow-list
- **WHEN** a principal with a restrictive policy submits without a target or
names a Host/Device outside its allow-list
- **THEN** the system rejects the request before creating a task
#### Scenario: Submission is disabled
- **WHEN** a principal with `tasks:submit` has a policy with submission
disabled
- **THEN** the system rejects task submission without changing the queue
### Requirement: Host governance policy is versioned and enforceable
The system SHALL persist a monotonic revisioned policy for each Host with an
optional active-task limit and optional UTC-day AI token budget, and SHALL
enforce active-task limits during Cloud scheduling.
#### Scenario: Host active-task limit is reached
- **WHEN** a Host already has its configured maximum number of assigned or
dispatched attempts and another queued task targets that Host
- **THEN** the scheduler leaves the later task queued until capacity becomes
available
#### Scenario: Host policy is updated
- **WHEN** an authorized governance administrator updates a Host policy
- **THEN** the system stores a strictly newer revision and later Cloud
authorization, scheduling, and proxy operations use the new policy
### Requirement: Governance mutations and reads use dedicated scopes
The system SHALL protect governance policy and AI-usage operations with
`governance:read` or `governance:admin` as appropriate, and SHALL record safe
audits for policy mutations.
#### Scenario: Administrator changes a policy
- **WHEN** a principal with `governance:admin` changes a user or Host policy
- **THEN** the system records actor, target, old/new revision, timestamp, and
non-secret policy metadata without recording credentials or request secrets
#### Scenario: Non-governance principal attempts a policy mutation
- **WHEN** a principal lacking `governance:admin` updates a governance policy
- **THEN** the system rejects the request before changing durable policy state
@@ -0,0 +1,38 @@
## ADDED Requirements
### Requirement: Planner proxy enforces effective Host token budgets before provider invocation
The Cloud planner-decision endpoint SHALL atomically reserve a conservative
bounded token amount before invoking its provider for an authenticated Host
using Cloud planner transport with an effective token budget, and SHALL reject
the request without invoking the provider when the reservation would exceed
the remaining budget.
#### Scenario: Budget permits a planner decision
- **WHEN** the Host has sufficient budget after used and reserved tokens are
considered
- **THEN** the endpoint creates a reservation and invokes the configured
provider exactly once
#### Scenario: Budget is exhausted
- **WHEN** a next planner decision would exceed the Host's effective budget
- **THEN** the endpoint returns a structured failure without invoking the
configured provider
### Requirement: Planner proxy settles provider-reported token usage without persisting prompts
The Cloud planner-decision endpoint SHALL settle its reservation to the
provider-reported token usage when available and SHALL retain only non-secret
metering metadata, never the raw prompt, screenshot, provider credentials, or
session/lease secret.
#### Scenario: Provider response includes usage
- **WHEN** the configured provider returns a valid tool-call decision and
token-usage metadata
- **THEN** the endpoint records and returns the decision, settles the Host's
reservation to the reported usage, and does not durably store request text
or screenshot bytes
#### Scenario: Usage is indeterminate after failure
- **WHEN** a reservation exists but the endpoint cannot determine provider
usage after a transport failure
- **THEN** the reservation remains until bounded expiry cleanup rather than
being released as unused
@@ -0,0 +1,61 @@
## MODIFIED Requirements
### Requirement: Host identity is authenticated and bound to one host id
The internal Host Agent API SHALL require a host-scoped bearer principal and
SHALL reject any request that attempts to act for a `host_id` different from
the authenticated principal's bound host.
#### Scenario: Host authenticates as itself
- **WHEN** a Host Agent presents valid credentials bound to its requested
`host_id`
- **THEN** the internal API authorizes permitted heartbeat, claim, renewal,
result, policy retrieval, and self-submission operations
#### Scenario: Host attempts to impersonate another host
- **WHEN** valid credentials bound to host A are used on a request for host B
- **THEN** the internal API rejects the request without reading or modifying
host B's state
### Requirement: Host Agent synchronizes heartbeat and complete device snapshots
The Host Agent SHALL periodically submit its complete local device snapshot
and last applied policy revision to the control plane, and the control plane
SHALL atomically refresh the host heartbeat, replace only that host's
pooled-device records, and return the effective Host policy whenever its
revision differs.
#### Scenario: Host reports devices
- **WHEN** a Host Agent submits a valid heartbeat containing its current devices
- **THEN** the control plane updates the host's last-seen time and exposes the
submitted devices through the aggregated pool
#### Scenario: Host policy revision changed
- **WHEN** a Host heartbeat presents a revision older than the effective
Cloud-host policy
- **THEN** the response includes the complete newer policy and revision for the
Host to cache
#### Scenario: Host policy revision is current
- **WHEN** a Host heartbeat presents the current effective policy revision
- **THEN** the response confirms that revision without resending an unrelated
policy representation
#### Scenario: Host reports no devices
- **WHEN** a previously populated host submits an empty device snapshot
- **THEN** only that host's prior device records are removed while devices
owned by other hosts remain unchanged
## ADDED Requirements
### Requirement: Host can submit a self-targeted goal through the outbound protocol
The Host Agent SHALL be able to submit a goal-only task through its existing
outbound authenticated protocol, and the control plane SHALL derive its target
from the authenticated Host rather than accepting another Host identifier.
#### Scenario: Host submits a local goal
- **WHEN** an authenticated Host submits a valid goal through its internal
self-submission operation
- **THEN** the control plane returns a queued task targeted to that Host
#### Scenario: Host submits an ineligible local Device
- **WHEN** a Host names a Device that is not owned by its authenticated Host
- **THEN** the control plane rejects the request without creating a task
@@ -0,0 +1,30 @@
## ADDED Requirements
### Requirement: Host can submit a goal task only for itself
The internal Host Agent API SHALL allow an authenticated Host to submit a
goal-based task that the Cloud Control Plane irrevocably targets to that same
Host, with an optional Device target owned by that Host.
#### Scenario: Host submits a goal for one of its Devices
- **WHEN** Host A presents valid Host A credentials and submits a valid goal
naming one of Host A's enrolled Devices
- **THEN** the control plane creates a queued task targeted to Host A and that
Device without granting Host A public task-submission authority
#### Scenario: Host omits a Device target
- **WHEN** an authenticated Host submits a valid goal without a Device target
- **THEN** the control plane creates a queued task targeted to that Host and
lets the scheduler select only an eligible Device owned by it
### Requirement: Host self-submission preserves host isolation
The internal Host task-submission operation SHALL reject a foreign Host,
foreign Device, workflow reference, or disabled self-submission policy before
creating any task.
#### Scenario: Host attempts a foreign target
- **WHEN** Host A submits a request naming Host B or a Device not owned by A
- **THEN** the control plane rejects the request and does not create a task
#### Scenario: Host self-submission is disabled
- **WHEN** a Host policy disables Host self-submission
- **THEN** the control plane rejects that Host's self-submission request
@@ -0,0 +1,84 @@
## MODIFIED Requirements
### Requirement: Task submission and status via the SDK
The system SHALL allow an external integrator to submit a task (goal or
workflow reference plus constraints and an optional explicit Host/Device
target) through the platform SDK's API, and to query that task's current
status by id, backed by the `task-scheduler` capability. A Device target MUST
include its owning Host, and the status representation SHALL expose any
explicit target separately from the eventual assignment.
#### Scenario: Submit a task via the API
- **WHEN** an integrator calls the task-submission endpoint with a valid goal
and optional constraints
- **THEN** the API returns a task id that can be used to poll status, and the
underlying `task-scheduler` records a new `queued` `ScheduledTask`
#### Scenario: Submit a task for one Host and Device
- **WHEN** a `tasks:submit` principal submits a valid target containing an
eligible Host and Device it is permitted to use
- **THEN** the API records that exact target and the scheduler cannot assign
the task outside it
#### Scenario: Device target lacks its Host
- **WHEN** an integrator submits a Device target without an owning Host
- **THEN** the API rejects the request before creating a task
#### Scenario: Query status of a known task
- **WHEN** an integrator requests status for a task id that exists
- **THEN** the API returns that task's current status (`queued`, `assigned`,
`dispatched`, `done`, or `failed`) and its explicit target when present
#### Scenario: Query status of an unknown task
- **WHEN** an integrator requests status for a task id that does not exist
- **THEN** the API returns a not-found response rather than an unhandled server
error
### Requirement: Python SDK client mirrors the REST API
The system SHALL provide a Python client (`CloudClient`) exposing methods
corresponding to every `/v1/...` resource, user-authentication,
user-administration, and governance route, including targeted task submission
and read-only Host AI-usage access, so integrators do not need to
hand-construct HTTP requests.
#### Scenario: Client submits a targeted task and retrieves status
- **WHEN** a caller uses `CloudClient` to submit a task with an explicit target
and then fetches status by the returned id
- **THEN** the client produces the same target and lifecycle result as direct
REST calls
#### Scenario: Client administers governance with a scoped bearer
- **WHEN** a caller configures `CloudClient` with `governance:admin` and
invokes a policy operation
- **THEN** the client sends that authentication and returns the corresponding
non-secret policy representation
### Requirement: Public API operations enforce scopes
The public platform API SHALL require operation-specific scopes for task
submission, task reading, pool reading, plugin reading, plugin
administration, user administration, governance reading, and governance
administration, regardless of whether the principal came from a bearer
credential or user session.
#### Scenario: Submit principal has task scope
- **WHEN** a bearer or user principal with `tasks:submit` calls the
task-submission endpoint
- **THEN** the request is authorized subject to normal task validation and
any effective user-submission policy
#### Scenario: Non-admin principal attempts plugin registration
- **WHEN** an authenticated principal without `plugins:admin` calls plugin
registration
- **THEN** the API rejects the request before resolving or loading the plugin
target
#### Scenario: Non-admin principal attempts user administration
- **WHEN** an authenticated principal without `users:admin` calls a
user-administration endpoint
- **THEN** the API rejects the request before reading or changing protected
user state
#### Scenario: Principal lacks governance administration scope
- **WHEN** an authenticated principal without `governance:admin` changes a
user or Host policy
- **THEN** the API rejects the request before changing policy or usage state
@@ -0,0 +1,58 @@
## MODIFIED Requirements
### Requirement: Task submission enqueues a scheduled task
The system SHALL allow a caller to submit a task (a goal string, or a
reference to a `WorkflowDefinition`, plus optional device constraints:
`driver_type`, required capability tags, and an explicit Host/Device target)
and SHALL enqueue it as a `ScheduledTask` with status `queued`, returning a
stable task id the caller can poll. A Device target MUST name its owning Host;
an explicit target remains attached to the task for its entire lifecycle.
#### Scenario: Successful submission
- **WHEN** a caller submits a task with a goal and no constraints
- **THEN** the scheduler creates a `ScheduledTask` with status `queued`,
assigns it a unique id, and returns that id to the caller without blocking
for a device to become available
#### Scenario: Targeted submission
- **WHEN** a caller submits a valid task targeted to a Host and one of its
Devices
- **THEN** the queued task retains that Host/Device target until it is assigned
or reaches a terminal lifecycle state
#### Scenario: Queue depth limit reached
- **WHEN** a caller submits a task while the queue already holds
`config.max_queue_depth` queued tasks
- **THEN** the scheduler rejects the submission with a clear error rather than
accepting an unbounded backlog
### Requirement: Assignment matches a queued task to an idle, constraint-matching device
The system SHALL assign a queued `ScheduledTask` to an idle `PooledDevice`
(as reported by the `device-pool` capability) whose explicit target, if any,
`driver_type`, and capability tags satisfy the task's constraints, using a
named, registrable `AssignmentStrategy`.
#### Scenario: Matching idle device available
- **WHEN** `assign()` runs and at least one idle `PooledDevice` matches the
head-of-queue task's constraints
- **THEN** the scheduler selects one such device via the configured
`AssignmentStrategy`, transitions the task to status `assigned`, and records
the chosen `device_id`/`host_id`
#### Scenario: Targeted Device is unavailable
- **WHEN** `assign()` runs and the only explicitly targeted Device is busy,
stale, absent, or otherwise ineligible
- **THEN** the task remains `queued` and the scheduler does not assign a
different Device or Host
#### Scenario: No matching device available
- **WHEN** `assign()` runs and no idle `PooledDevice` matches the head-of-queue
task's constraints
- **THEN** the task remains `queued` (not failed), and `assign()` returns
without error, ready to be retried on a later call
#### Scenario: Unknown assignment strategy configured
- **WHEN** `TaskScheduler` is configured with an `AssignmentStrategy` name
that is not registered
- **THEN** the scheduler raises a clear configuration error at startup/first-
assign rather than silently falling back to a default strategy