578 lines
26 KiB
Markdown
578 lines
26 KiB
Markdown
# Cloud Control Plane Deployment
|
|
|
|
This guide covers the deployable Cloud API and outbound Device Host Agent.
|
|
Run commands from the repository root after synchronizing the uv workspace:
|
|
|
|
```bash
|
|
uv sync --locked --all-packages
|
|
```
|
|
|
|
## Local SQLite
|
|
|
|
SQLite is intended for local development and tests with one Cloud API process.
|
|
The Cloud API uses persistent user sessions for human access and a Host-generated
|
|
secret for later Host operations; no static bearer credential configuration is
|
|
required.
|
|
|
|
```powershell
|
|
$env:CLOUD_ENVIRONMENT = "local"
|
|
$env:CLOUD_DATABASE_URL = "sqlite:///cloud/cloud.sqlite3"
|
|
$env:CLOUD_SESSION_COOKIE_SECURE = "false"
|
|
uv run --package device-cloud-api device-cloud-api --host 127.0.0.1 --port 8001
|
|
```
|
|
|
|
Create the first administrator interactively, then open the Console and sign
|
|
in with that account:
|
|
|
|
```bash
|
|
uv run --package device-cloud-api device-cloud-admin users create --username admin --display-name "Local Administrator" --role admin
|
|
```
|
|
|
|
In a second terminal, configure the local Host Agent to reach this Cloud API:
|
|
|
|
```powershell
|
|
$env:HOST_AGENT_CONTROL_PLANE_URL = "http://127.0.0.1:8001"
|
|
uv run --package device-host-agent device-host-agent setup
|
|
uv run --package device-host-agent device-host-agent
|
|
```
|
|
|
|
At startup, the Host Agent loads device registrations from
|
|
`tasks/device_config.sqlite3`, the same `DeviceConfigStore` used by the local
|
|
Runtime console API. Register or update devices before starting the Host Agent,
|
|
then restart it to reload changes. In Compose,
|
|
`HOST_AGENT_TASKS_PATH` selects the host directory mounted at `/app/tasks`; it
|
|
defaults to `./tasks`.
|
|
|
|
The Host Agent only initiates outbound HTTP requests. It does not expose an
|
|
inbound port.
|
|
|
|
## Direct Edge Enrollment
|
|
|
|
The Host Agent's default `HOST_AGENT_CONTROL_PLANE_URL` is
|
|
`https://amcp.home.jerryyan.top`. This is a single-operator home deployment
|
|
default; override the environment variable for local/dev/test runs pointed at
|
|
a different Cloud API.
|
|
|
|
When no cached identity exists, the Host Agent creates a random instance
|
|
identifier and Host secret, registers directly with the Cloud API, persists the
|
|
assigned `host_id`, and uses that secret for all later device enrollment,
|
|
heartbeat, claim, renewal, and result calls. The Cloud stores only the secret
|
|
digest. There is no configured enrollment-token or static Host-credential path.
|
|
|
|
This trades away any approval step: **any caller that can reach the control
|
|
plane can register itself as a new Host.** There is no rate limiting or
|
|
throttling on this path by design; restrict access at the firewall or reverse
|
|
proxy before exposing the endpoint.
|
|
|
|
Before the Host Agent's first unattended start, create the one-time local
|
|
operator account interactively:
|
|
|
|
```bash
|
|
uv run --package device-host-agent device-host-agent setup
|
|
```
|
|
|
|
This prompts for a username/password and writes a PBKDF2-hashed credential
|
|
file to `HOST_AGENT_LOCAL_ACCOUNT_PATH` (default
|
|
`tasks/host_local_account.json`), gating only this first-run bootstrap step —
|
|
it is not re-checked on subsequent unattended restarts. Running the daemon's
|
|
default command without a controlling terminal before this file exists fails
|
|
fast with a message naming the `setup` step, instead of hanging on a prompt
|
|
no one can answer.
|
|
|
|
Running under Compose, create the account once before `docker compose up`:
|
|
|
|
```bash
|
|
docker compose run --rm host-agent device-host-agent setup
|
|
docker compose up -d
|
|
```
|
|
|
|
Keep `HOST_AGENT_IDENTITY_PATH`, the local-account file, and
|
|
`tasks/device_config.sqlite3` on persistent storage with permissions limited to
|
|
the service account. The identity file is a bearer secret: do not put it in an
|
|
image, repository, log, or general backup.
|
|
|
|
### Local Web Console
|
|
|
|
The Host Agent always serves a small local-only web console on the
|
|
edge machine: heartbeat/enrollment status, registered local devices, current
|
|
assignment progress, local device add/edit/remove, a local account password
|
|
change, and recent assignment/heartbeat history. It authenticates with the
|
|
same local account created by `device-host-agent setup` above — there is no
|
|
separate console credential.
|
|
|
|
```text
|
|
HOST_AGENT_CONSOLE_BIND_HOST=127.0.0.1
|
|
HOST_AGENT_CONSOLE_PORT=8765
|
|
HOST_AGENT_CONSOLE_ALLOW_NON_LOOPBACK=false
|
|
HOST_AGENT_CONSOLE_SESSION_TTL_SECONDS=43200
|
|
HOST_AGENT_CONSOLE_HISTORY_LIMIT=200
|
|
```
|
|
|
|
- The Console starts with every Host Agent and binds to `127.0.0.1:8765` by
|
|
default.
|
|
- `HOST_AGENT_CONSOLE_BIND_HOST` — the address the console binds to; defaults
|
|
to loopback-only.
|
|
- `HOST_AGENT_CONSOLE_PORT` — the TCP port the console listens on.
|
|
- `HOST_AGENT_CONSOLE_ALLOW_NON_LOOPBACK` — required opt-in before
|
|
`HOST_AGENT_CONSOLE_BIND_HOST` may be a non-loopback address; the Host
|
|
Agent refuses to start otherwise.
|
|
- `HOST_AGENT_CONSOLE_SESSION_TTL_SECONDS` — sliding idle timeout, in seconds,
|
|
for an authenticated console session.
|
|
- `HOST_AGENT_CONSOLE_HISTORY_LIMIT` — number of recent assignment/heartbeat
|
|
entries the console retains before pruning older ones.
|
|
|
|
### Task progress storage and retention
|
|
|
|
The Host Agent persists step-by-step task execution state (metadata +
|
|
timeline screenshots) to local SQLite/files on the edge machine. These
|
|
paths are independent from the Runtime's own `tasks/tasks.sqlite3` and
|
|
do not collide when both processes run on the same host.
|
|
|
|
```text
|
|
HOST_AGENT_TASK_PROGRESS_DB_PATH=host_agent_data/task_progress.sqlite3
|
|
HOST_AGENT_TASK_ARTIFACT_DIR=host_agent_data/history
|
|
HOST_AGENT_TASK_RETENTION_MAX_COUNT=50
|
|
HOST_AGENT_TASK_RETENTION_MAX_AGE_DAYS=7
|
|
```
|
|
|
|
- `HOST_AGENT_TASK_PROGRESS_DB_PATH` — SQLite path for the Host Agent's
|
|
task metadata store. Default: `host_agent_data/task_progress.sqlite3`.
|
|
- `HOST_AGENT_TASK_ARTIFACT_DIR` — directory for timeline screenshots and
|
|
step artifacts. Default: `host_agent_data/history`.
|
|
- `HOST_AGENT_TASK_RETENTION_MAX_COUNT` — maximum number of completed
|
|
tasks to retain. Default: `50`.
|
|
- `HOST_AGENT_TASK_RETENTION_MAX_AGE_DAYS` — maximum age in days for
|
|
retained tasks. Default: `7`. The more restrictive of count vs. age
|
|
always wins.
|
|
|
|
**Viewing live and historical task progress:**
|
|
|
|
- **Host Agent console**: Open `http://127.0.0.1:8765/tasks` for the task
|
|
list (status, device, timestamps). Click a task ID to see the detail page
|
|
with full step-by-step timeline and inlined screenshots.
|
|
- **Cloud console**: The Cloud Console task detail page shows the latest
|
|
coarse-grained progress badge (step index, status, summary) that the Host
|
|
Agent piggybacks on each lease renewal.
|
|
|
|
Treat `HOST_AGENT_CONSOLE_ALLOW_NON_LOOPBACK` as an explicit,
|
|
operator-accepted risk: the console has no built-in TLS and no rate
|
|
limiting, so a non-loopback bind exposes an unencrypted login form to
|
|
whatever network can reach that port. To reach the console from another
|
|
machine instead, keep it bound to loopback and open an SSH local
|
|
port-forward to the edge machine:
|
|
|
|
```bash
|
|
ssh -L 8765:127.0.0.1:8765 user@edge-host
|
|
# then open http://127.0.0.1:8765 from the local browser
|
|
```
|
|
|
|
## PostgreSQL Deployment
|
|
|
|
Start from `.env.example`, replace every `change-me-*` value, and keep the
|
|
resulting `.env` file outside version control. The Compose stack contains
|
|
PostgreSQL, the Cloud API, and one Host Agent:
|
|
|
|
```bash
|
|
docker compose up --build -d
|
|
docker compose ps
|
|
```
|
|
|
|
The Cloud API container waits for PostgreSQL, runs the committed Alembic
|
|
migrations, then starts the API on port `8001`. Readiness is available at:
|
|
|
|
```text
|
|
GET http://127.0.0.1:8001/health/ready
|
|
```
|
|
|
|
## Deploying A Jenkins-Built Image
|
|
|
|
`docker compose up --build` above builds the image from source on the target
|
|
host. For environments that pull a pre-built image instead, `Jenkinsfile` runs
|
|
the pytest suite (`-m "not integration"`), builds the image from the same
|
|
`Dockerfile`, smoke-tests both console scripts (`device-cloud-api --help`,
|
|
`device-host-agent --help`), and pushes `<REGISTRY>/<IMAGE_NAME>:<BUILD_NUMBER>-<git short sha>`
|
|
plus `:latest` to the configured registry.
|
|
|
|
`compose.deploy.yaml` contains only PostgreSQL and the Cloud API; Host Agents
|
|
run at their edge sites rather than beside the Cloud API. It references the
|
|
fixed Jenkins registry image and interpolates only `IMAGE_TAG` (see
|
|
`.env.example`). Deploy without a local build step:
|
|
|
|
```bash
|
|
docker compose -f compose.deploy.yaml pull
|
|
docker compose -f compose.deploy.yaml up -d
|
|
docker compose -f compose.deploy.yaml ps
|
|
```
|
|
|
|
For a deployment that manages processes outside Compose, apply migrations
|
|
before starting the new Cloud API version:
|
|
|
|
```powershell
|
|
$env:CLOUD_DATABASE_URL = "postgresql+psycopg://USER:PASSWORD@HOST:5432/device_cloud"
|
|
uv run alembic -c packages/cloud-platform/cloud/migrations/alembic.ini upgrade head
|
|
uv run --package device-cloud-api device-cloud-api --host 0.0.0.0 --port 8001
|
|
```
|
|
|
|
Production startup requires `CLOUD_ENVIRONMENT=production`, a current schema,
|
|
and HTTPS before browser sessions are exposed. It does not require credential
|
|
JSON. Create the first administrator interactively after startup, before
|
|
opening the Console to operators.
|
|
|
|
## Authentication And Host Identities
|
|
|
|
Cloud users are the human authorization boundary. Their `viewer`, `operator`,
|
|
and `admin` roles map to the existing API scopes, and the browser sends an
|
|
`HttpOnly` session cookie plus CSRF proof for unsafe operations. The deployment
|
|
does not accept `CLOUD_PUBLIC_CREDENTIALS_JSON`,
|
|
`CLOUD_HOST_CREDENTIALS_JSON`, or `CLOUD_ENROLLMENT_TOKENS_JSON`.
|
|
|
|
A fresh Host sends its generated candidate secret only during direct
|
|
registration. The Cloud stores its digest and returns a `host_id`; later Host
|
|
operations use that secret and are strictly bound to the returned `host_id`.
|
|
Protect the persisted Host identity file as a bearer secret.
|
|
|
|
Dynamically enrolled Host credentials are stored as digests in the cloud
|
|
database. This release exposes repository-level revocation rather than a public
|
|
administration endpoint. An operator with database deployment access can revoke
|
|
a Host without deleting its task history:
|
|
|
|
```bash
|
|
export HOST_ID="host-..."
|
|
uv run --package device-cloud-platform python - <<'PY'
|
|
import os
|
|
from cloud.database import CloudDatabase
|
|
from core.models import utc_now
|
|
|
|
database = CloudDatabase(os.environ["CLOUD_DATABASE_URL"], create_schema=False)
|
|
try:
|
|
changed = database.repository.revoke_enrolled_host(
|
|
os.environ["HOST_ID"],
|
|
revoked_at=utc_now(),
|
|
)
|
|
print("revoked" if changed else "not an enrolled host")
|
|
finally:
|
|
database.close()
|
|
PY
|
|
```
|
|
|
|
## Cloud Console (Web UI)
|
|
|
|
The repository ships an independent Vue 3 + Vite SPA at `cloud-console/` that
|
|
renders task history, devices, hosts, and plugins. Human operators sign in with
|
|
a username and password; the Cloud API creates an expiring, revocable
|
|
`HttpOnly` session cookie and uses a separate CSRF cookie/header for writes.
|
|
An administrator can manage accounts, submission policies, Host operational
|
|
limits, and non-secret Cloud-proxy token usage from **Users & limits**. Bearer
|
|
tokens remain a compatibility path for API/SDK automation.
|
|
|
|
### HTTPS and session configuration
|
|
|
|
`CLOUD_ENVIRONMENT=production` requires `CLOUD_SESSION_COOKIE_SECURE=true`.
|
|
Terminate TLS at a reverse proxy and open the same-origin Console through HTTPS,
|
|
for example `https://cloud.example.com/console/`. Direct `http://host:8001`
|
|
access is for local/test mode only; it cannot retain production login cookies.
|
|
|
|
The defaults are an 8-hour idle session TTL, 7-day absolute TTL, and a temporary
|
|
block after five failed logins in a 15-minute username/client-address window:
|
|
|
|
```text
|
|
CLOUD_USER_SESSION_IDLE_SECONDS=28800
|
|
CLOUD_USER_SESSION_ABSOLUTE_SECONDS=604800
|
|
CLOUD_LOGIN_FAILURE_LIMIT=5
|
|
CLOUD_LOGIN_FAILURE_WINDOW_SECONDS=900
|
|
CLOUD_LOGIN_BLOCK_SECONDS=900
|
|
CLOUD_SESSION_COOKIE_SECURE=true
|
|
CLOUD_TRUST_PROXY_HEADERS=false
|
|
```
|
|
|
|
Set `CLOUD_TRUST_PROXY_HEADERS=true` only when a trusted proxy overwrites
|
|
`X-Forwarded-For` before requests reach the Cloud API. This affects the
|
|
client-address bucket used for login throttling; the default is safe when the
|
|
application is reached directly.
|
|
|
|
### Create and recover administrator accounts
|
|
|
|
After migrations and Cloud API startup, create the first account interactively:
|
|
|
|
```bash
|
|
docker compose exec cloud-api \
|
|
device-cloud-admin users create \
|
|
--username admin --display-name "Cloud Administrator" --role admin
|
|
```
|
|
|
|
The command prompts twice for the password, so it does not enter shell history,
|
|
Compose configuration, process arguments, logs, or container inspection output.
|
|
Recovery commands are also interactive:
|
|
|
|
```bash
|
|
docker compose exec cloud-api device-cloud-admin users reset-password --username admin
|
|
docker compose exec cloud-api device-cloud-admin users enable --username admin
|
|
docker compose exec cloud-api device-cloud-admin users revoke-sessions --username admin
|
|
```
|
|
|
|
Roles are fixed: `viewer` can read tasks/pool/plugins; `operator` additionally
|
|
submits tasks; `admin` has unrestricted Cloud API access. The administration
|
|
CLI creates users, resets passwords, changes roles, enables accounts, and
|
|
revokes sessions. New and reset users must change their temporary password
|
|
before accessing other resources, and the API will not disable or demote the
|
|
last enabled administrator.
|
|
|
|
### Configure the CORS allow-list
|
|
|
|
The Cloud API has no CORS middleware by default. Before a browser can call it
|
|
cross-origin, set `CLOUD_CONSOLE_CORS_ORIGINS` to a comma-separated allow-list
|
|
that includes the exact origin (scheme + host + port, no trailing slash) the
|
|
operator's browser will load the console from:
|
|
|
|
```bash
|
|
# Allow a local Vite dev server
|
|
export CLOUD_CONSOLE_CORS_ORIGINS="http://127.0.0.1:5173"
|
|
# Or a deployed origin
|
|
export CLOUD_CONSOLE_CORS_ORIGINS="https://console.example.com"
|
|
```
|
|
|
|
Restart the Cloud API after changing this env. The middleware is added only
|
|
when the allow-list is non-empty — existing deployments see no behavior change
|
|
until an operator opts in. Blanket `allow_origins=["*"]` is intentionally not
|
|
supported because browser sessions are credentialed.
|
|
|
|
### Run the console
|
|
|
|
```bash
|
|
cd cloud-console
|
|
cp .env.example .env.local
|
|
# Edit .env.local if your Cloud API is not at http://127.0.0.1:8001
|
|
npm install
|
|
npm run dev
|
|
```
|
|
|
|
Vite prints a local URL (default `http://127.0.0.1:5173`). That exact origin
|
|
must be in `CLOUD_CONSOLE_CORS_ORIGINS` on the Cloud API. For local development
|
|
set `CLOUD_SESSION_COOKIE_SECURE=false`, then open the dev URL and sign in with
|
|
a user account. The Console sends credentialed requests and attaches CSRF proof
|
|
to writes.
|
|
|
|
For a production build, run `npm run build` and serve the resulting `dist/`
|
|
behind any static file server or CDN, with `VITE_CLOUD_API_BASE_URL` baked in
|
|
at build time. The deployed origin must be in `CLOUD_CONSOLE_CORS_ORIGINS`.
|
|
|
|
### Same-origin deployment (baked into the Cloud API image)
|
|
|
|
The Jenkins-built Docker image already carries the SPA at `/app/console-static`
|
|
and sets `CLOUD_CONSOLE_STATIC_DIR` in the image itself. The Cloud API serves
|
|
the Console at `/console/` (visiting `/` 307-redirects there). Put that origin
|
|
behind an HTTPS reverse proxy, then open for example
|
|
`https://cloud.example.com/` directly — no separate dev server, static host, or
|
|
CORS allow-list is needed because the SPA and API share one origin.
|
|
|
|
The SPA shell (`index.html`, JS, CSS) is served without credentials by design
|
|
so it can render the login page. All `/v1/*` resource calls remain scope-gated,
|
|
and unsafe cookie-authenticated calls require CSRF proof. The browser receives
|
|
only the non-secret CSRF value; it never receives the `HttpOnly` session secret.
|
|
|
|
Source-based local development leaves `CLOUD_CONSOLE_STATIC_DIR` unset; the
|
|
mount remains conditional on that image-provided setting.
|
|
|
|
Jenkins build args (`NODE_IMAGE`, `NPM_REGISTRY`, `UV_IMAGE`, `APT_MIRROR`,
|
|
`UV_INDEX_URL`) default to CN mirrors so builds don't time out pulling from
|
|
Docker Hub / ghcr.io / npmjs.org / deb.debian.org. Blank any of them to fall
|
|
back to the upstream.
|
|
|
|
## Runtime AI Planner
|
|
|
|
The Host Agent reuses the local Runtime planner. Unlike the shared Runtime
|
|
library (whose own default is the deterministic stub planner), the **Host
|
|
Agent defaults `AI_PLANNER_ENABLED` to on** -- it is the actual device-control
|
|
path, so goal assignments use a model unless an operator explicitly opts out.
|
|
With `AI_PLANNER_TRANSPORT` unset, the Host Agent uses the **`cloud`**
|
|
transport. Configure an active Cloud Provider profile before deploying; without
|
|
one, every planning step raises immediately and the task fails on its first
|
|
step (no silent fallback to the stub planner). Set `AI_PLANNER_ENABLED=false`
|
|
to opt back out to the deterministic stub planner (e.g. for offline/dev hosts
|
|
with no provider credentials).
|
|
|
|
### Cloud-proxy transport (default)
|
|
|
|
With `AI_PLANNER_TRANSPORT` unset or set to `cloud`, every planning decision
|
|
routes through the Cloud API's
|
|
`POST /internal/v1/hosts/{host_id}/planner/decide` endpoint (the same
|
|
host-scoped bearer credential used for heartbeat/claim/renew/result). In this
|
|
mode:
|
|
|
|
- **Provider configuration lives in the Cloud database.** Set
|
|
`CLOUD_LLM_PROVIDER_ENCRYPTION_KEY` from the deployment secret manager, then
|
|
sign in to `/console/` as an administrator and create an active entry under
|
|
**LLM providers**. Provider API keys are encrypted in the database and are
|
|
never returned by the API or Console. Edge Hosts do not hold Provider keys.
|
|
The profile's provider, model, base URL, and timeout (1 to 120 seconds) are
|
|
the authority for every Cloud-proxy call. The Cloud API does not read
|
|
`AI_PLANNER_PROVIDER`, `AI_PLANNER_MODEL`, `AI_PLANNER_TIMEOUT_SECONDS`,
|
|
`ANTHROPIC_API_KEY`, or `OPENAI_API_KEY`; those variables apply only to the
|
|
explicit `direct` transport.
|
|
- **Profile types:** choose **Anthropic** for native Anthropic tool use, or
|
|
**OpenAI-compatible** for the OpenAI Chat Completions tool-calling protocol.
|
|
Both accept an optional absolute HTTP(S) Base URL; leave it blank for the
|
|
official provider endpoint. The configured endpoint must accept the selected
|
|
provider's existing request schema and authentication; custom headers or
|
|
incompatible parameter dialects are not supported by this path.
|
|
- **Activation is immediate:** a newly activated enabled profile becomes the
|
|
Provider/model/timeout for the next Cloud-proxy planner decision. A
|
|
Cloud-planner request fails closed until one enabled profile is active; it
|
|
never falls back to a Cloud API environment credential.
|
|
- **Trade-offs to accept before enabling:**
|
|
- *Latency*: every planning step now makes a round trip to the Cloud API in
|
|
addition to the LLM provider call.
|
|
- *Availability coupling*: unlike heartbeat/claim (which tolerate transient
|
|
Cloud API outages via retry/backoff), a planning step fails immediately if
|
|
the Cloud API or its configured provider is unreachable -- there is no
|
|
fallback to the stub planner or to a local direct call.
|
|
- *Expanded data path and retained history*: goal/scene prompts and
|
|
screenshots transit the Cloud API. Application logs retain only metadata
|
|
(host id, resolved tool name, latency, error class), but every successful
|
|
Cloud-proxy decision with task context is also stored as system prompt,
|
|
user prompt, resolved tool name, arguments, and step index. The Cloud
|
|
Console task detail exposes that history to authorized task readers. The
|
|
decision log never stores screenshot bytes; direct-transport Hosts produce
|
|
no Cloud-side LLM history.
|
|
- *Retention*: `CLOUD_PLANNER_DECISION_LOG_RETENTION_DAYS` defaults to `7`.
|
|
The Cloud prunes a terminal task's decision rows after that window;
|
|
`CLOUD_PLANNER_DECISION_LOG_PRUNE_INTERVAL_SECONDS` defaults to `3600`.
|
|
Prompt retention is therefore a deliberate operational and data-handling
|
|
choice, not merely transient request processing.
|
|
|
|
### Direct transport (explicit opt-out)
|
|
|
|
Set `AI_PLANNER_TRANSPORT=direct` only for Hosts that must call a provider
|
|
without the Cloud proxy. Those Hosts hold their own provider credentials:
|
|
|
|
```text
|
|
AI_PLANNER_TRANSPORT=direct
|
|
AI_PLANNER_PROVIDER=anthropic
|
|
AI_PLANNER_MODEL=claude-sonnet-5
|
|
AI_PLANNER_TIMEOUT_SECONDS=30
|
|
ANTHROPIC_API_KEY=<secret manager reference>
|
|
```
|
|
|
|
For OpenAI, set `AI_PLANNER_PROVIDER=openai` and provide `OPENAI_API_KEY`.
|
|
Direct Hosts are not covered by Cloud token budgets or Cloud-side Provider key
|
|
rotation. `AI_PLANNER_TIMEOUT_SECONDS` controls the provider call only in this
|
|
direct mode.
|
|
|
|
### Host governance and Cloud-proxy token budgets
|
|
|
|
After deploying the API migration, an administrator can use **Users & limits**
|
|
in `/console/` to restrict a user's task submission to explicit Host/Device
|
|
targets, enable or disable Host self-submission, and set a Host active-task
|
|
limit. Policies are enforced by the API and scheduler; hiding Console controls
|
|
is not an authorization boundary.
|
|
|
|
Cloud-enforced token budgets require `AI_PLANNER_TRANSPORT=cloud`. Before each
|
|
proxy decision, the API atomically reserves
|
|
`CLOUD_PLANNER_TOKEN_RESERVATION_CEILING` tokens (default `4096`) for the
|
|
current UTC day. Set `CLOUD_PLANNER_TOKEN_RESERVATION_TTL_SECONDS` (default
|
|
`300`) to bound an unknown-usage reservation after provider/transport failure.
|
|
The daily Host budget must accommodate the reservation ceiling; otherwise the
|
|
proxy rejects before calling the provider. On a provider response, the
|
|
reservation is settled to reported usage and the separate usage ledger retains
|
|
only timestamp, provider/model, token counts, and optional task/attempt
|
|
identifiers. This ledger is distinct from the bounded planner-decision history
|
|
described above.
|
|
|
|
Hosts reporting `AI_PLANNER_TRANSPORT=direct` are explicitly shown as
|
|
**unmetered**. Cloud cannot enforce or verify their provider token use. Do not
|
|
interpret an unmetered Host's absence of usage events as budget compliance.
|
|
|
|
Roll out in this order: migrate the Cloud database, provision
|
|
`CLOUD_LLM_PROVIDER_ENCRYPTION_KEY`, deploy the Cloud API, create and activate
|
|
a Provider profile in `/console/`, then switch a pilot Host to Cloud transport.
|
|
Configure a budget above the reservation ceiling and review usage events before
|
|
enabling budgets fleet-wide. Rotate a Provider API key by editing that profile;
|
|
the encryption master key is deployment-managed and must be preserved with the
|
|
database backups. A rollback to an older Cloud API requires restoring its
|
|
legacy Provider environment configuration, while a rollback to `direct`
|
|
transport requires valid provider credentials on that Host; preserve usage and
|
|
policy rows rather than deleting accounting history.
|
|
|
|
## Skill Management
|
|
|
|
Cloud-origin Skills are administrator-managed through the Cloud Console's
|
|
**Skills** view (requires the `skills:admin` scope, which administrators
|
|
hold via the `*` scope). Create/edit/delete skills (knowledge or flow-template
|
|
kinds), and grant or revoke per-host entitlement — an agent only ever sees the
|
|
cloud skills entitled to its own host.
|
|
|
|
Agents pull their entitled cloud skills incrementally from the Cloud API
|
|
(`GET /internal/v1/hosts/{host_id}/skills/sync`) on a configurable cadence
|
|
(`HOST_AGENT_SKILL_SYNC_INTERVAL_SECONDS`, default 300s) and cache them in a
|
|
local SQLite file (`tasks/skills.sqlite3`). Agents may also author their own
|
|
**local skills** (persisted in a separate `tasks/local_skills.sqlite3`) and
|
|
**override** a cloud skill locally via the MCP authoring tools; an override
|
|
shadows the cloud skill until removed, and forks into a standalone local skill
|
|
if the cloud entitlement is revoked. Agents report a best-effort read-only
|
|
inventory of their local skills to the Cloud so the Console can display them
|
|
per host.
|
|
|
|
The cloud skill store, entitlement mapping, per-host sync versioning, and
|
|
inventory readback live in the Cloud platform database (migration
|
|
`0010_skill_management`).
|
|
|
|
## Operational Limitations
|
|
|
|
Run exactly one scheduler-enabled Cloud API process. SQLite supports only the
|
|
documented single-control-plane development mode. PostgreSQL row locking makes
|
|
assignment and claim transactions safe if requests overlap, but this release
|
|
does not implement scheduler leader election or claim active-active scheduler
|
|
operation. Starting multiple Cloud API replicas would start one scheduler loop
|
|
per replica and is outside the supported deployment topology.
|
|
|
|
Device execution provides at-least-once side-effect semantics, not exactly-once
|
|
semantics. A device action can succeed immediately before the Host Agent loses
|
|
its lease or its result response, after which the control plane may retry the
|
|
task. Lease renewal and cooperative stop checks prevent later interruptible
|
|
actions where possible, but they cannot roll back an action already sent to a
|
|
device or safely terminate an in-progress synchronous driver call.
|
|
|
|
Use bounded attempts, inspect task attempt and failure metadata, and design
|
|
device workflows to tolerate repeated actions when the target operation allows
|
|
it. Do not use this release for operations that require a transactional
|
|
exactly-once guarantee across the cloud database and an external device.
|
|
|
|
## Shutdown And Rollback
|
|
|
|
For a normal shutdown, stop Host Agents first so they stop polling, interrupt
|
|
later cooperative actions, finish terminal reporting where the lease remains
|
|
valid, and attempt a final heartbeat. Stop the Cloud API after Host Agents have
|
|
exited, then stop PostgreSQL only if the database itself is being maintained:
|
|
|
|
```bash
|
|
docker compose stop host-agent
|
|
docker compose stop cloud-api
|
|
docker compose stop postgres
|
|
```
|
|
|
|
For rollback:
|
|
|
|
1. Stop all Host Agents and the Cloud API.
|
|
2. Back up PostgreSQL or the SQLite database file.
|
|
3. Before rolling back to an image that still requires static credentials,
|
|
restore that image's matching deployment configuration and provision the
|
|
required legacy credentials outside this release's Compose contract.
|
|
4. If the previous application version cannot use the current schema, run the
|
|
tested downgrade while no application process is connected:
|
|
|
|
```bash
|
|
uv run alembic -c packages/cloud-platform/cloud/migrations/alembic.ini downgrade -1
|
|
```
|
|
|
|
5. Restore the previous application image or checkout and start the Cloud API.
|
|
6. Verify `/health/ready`, then restart Host Agents with identities compatible
|
|
with the restored Cloud API.
|
|
|
|
Downgrading revision 0002 removes dynamic credential bindings and durable
|
|
device enrollment mappings. It retains the revision-0001 Host heartbeat rows,
|
|
pooled devices, queued tasks, attempts, and plugins.
|
|
|
|
Do not remove the PostgreSQL volume during an application rollback. Queued and
|
|
attempt history are durable database state and should remain available to the
|
|
restored or forward-deployed control plane.
|