agentic-mobile-control/docs/CLOUD_DEPLOYMENT.md

# Cloud Control Plane Deployment

This guide covers the deployable Cloud API and outbound Device Host Agent.
Run commands from the repository root after synchronizing the uv workspace:

```bash
uv sync --locked --all-packages
```

## Local SQLite

SQLite is intended for local development and tests with one Cloud API process.
The Cloud API uses persistent user sessions for human access and a Host-generated
secret for later Host operations; no static bearer credential configuration is
required.

```powershell
$env:CLOUD_ENVIRONMENT = "local"
$env:CLOUD_DATABASE_URL = "sqlite:///cloud/cloud.sqlite3"
$env:CLOUD_SESSION_COOKIE_SECURE = "false"
uv run --package device-cloud-api device-cloud-api --host 127.0.0.1 --port 8001
```

Create the first administrator interactively, then open the Console and sign
in with that account:

```bash
uv run --package device-cloud-api device-cloud-admin users create --username admin --display-name "Local Administrator" --role admin
```

In a second terminal, configure the local Host Agent to reach this Cloud API:

```powershell
$env:HOST_AGENT_CONTROL_PLANE_URL = "http://127.0.0.1:8001"
uv run --package device-host-agent device-host-agent setup
uv run --package device-host-agent device-host-agent
```

At startup, the Host Agent loads device registrations from
`tasks/device_config.sqlite3`, the same `DeviceConfigStore` used by the local
Runtime console API. Register or update devices before starting the Host Agent,
then restart it to reload changes. In Compose,
`HOST_AGENT_TASKS_PATH` selects the host directory mounted at `/app/tasks`; it
defaults to `./tasks`.

The Host Agent only initiates outbound HTTP requests. It does not expose an
inbound port.

## Direct Edge Enrollment

The Host Agent's default `HOST_AGENT_CONTROL_PLANE_URL` is
`https://amcp.home.jerryyan.top`. This is a single-operator home deployment
default; override the environment variable for local/dev/test runs pointed at
a different Cloud API.

When no cached identity exists, the Host Agent creates a random instance
identifier and Host secret, registers directly with the Cloud API, persists the
assigned `host_id`, and uses that secret for all later device enrollment,
heartbeat, claim, renewal, and result calls. The Cloud stores only the secret
digest. There is no configured enrollment-token or static Host-credential path.

This trades away any approval step: **any caller that can reach the control
plane can register itself as a new Host.** There is no rate limiting or
throttling on this path by design; restrict access at the firewall or reverse
proxy before exposing the endpoint.

Before the Host Agent's first unattended start, create the one-time local
operator account interactively:

```bash
uv run --package device-host-agent device-host-agent setup
```

This prompts for a username/password and writes a PBKDF2-hashed credential
file to `HOST_AGENT_LOCAL_ACCOUNT_PATH` (default
`tasks/host_local_account.json`), gating only this first-run bootstrap step —
it is not re-checked on subsequent unattended restarts. Running the daemon's
default command without a controlling terminal before this file exists fails
fast with a message naming the `setup` step, instead of hanging on a prompt
no one can answer.

Running under Compose, create the account once before `docker compose up`:

```bash
docker compose run --rm host-agent device-host-agent setup
docker compose up -d
```

Keep `HOST_AGENT_IDENTITY_PATH`, the local-account file, and
`tasks/device_config.sqlite3` on persistent storage with permissions limited to
the service account. The identity file is a bearer secret: do not put it in an
image, repository, log, or general backup.

### Local Web Console

The Host Agent always serves a small local-only web console on the
edge machine: heartbeat/enrollment status, registered local devices, current
assignment progress, local device add/edit/remove, a local account password
change, and recent assignment/heartbeat history. It authenticates with the
same local account created by `device-host-agent setup` above — there is no
separate console credential.

```text
HOST_AGENT_CONSOLE_BIND_HOST=127.0.0.1
HOST_AGENT_CONSOLE_PORT=8765
HOST_AGENT_CONSOLE_ALLOW_NON_LOOPBACK=false
HOST_AGENT_CONSOLE_SESSION_TTL_SECONDS=43200
HOST_AGENT_CONSOLE_HISTORY_LIMIT=200
```

- The Console starts with every Host Agent and binds to `127.0.0.1:8765` by
  default.
- `HOST_AGENT_CONSOLE_BIND_HOST` — the address the console binds to; defaults
  to loopback-only.
- `HOST_AGENT_CONSOLE_PORT` — the TCP port the console listens on.
- `HOST_AGENT_CONSOLE_ALLOW_NON_LOOPBACK` — required opt-in before
  `HOST_AGENT_CONSOLE_BIND_HOST` may be a non-loopback address; the Host
  Agent refuses to start otherwise.
- `HOST_AGENT_CONSOLE_SESSION_TTL_SECONDS` — sliding idle timeout, in seconds,
  for an authenticated console session.
- `HOST_AGENT_CONSOLE_HISTORY_LIMIT` — number of recent assignment/heartbeat
  entries the console retains before pruning older ones.

Treat `HOST_AGENT_CONSOLE_ALLOW_NON_LOOPBACK` as an explicit,
operator-accepted risk: the console has no built-in TLS and no rate
limiting, so a non-loopback bind exposes an unencrypted login form to
whatever network can reach that port. To reach the console from another
machine instead, keep it bound to loopback and open an SSH local
port-forward to the edge machine:

```bash
ssh -L 8765:127.0.0.1:8765 user@edge-host
# then open http://127.0.0.1:8765 from the local browser
```

## PostgreSQL Deployment

Start from `.env.example`, replace every `change-me-*` value, and keep the
resulting `.env` file outside version control. The Compose stack contains
PostgreSQL, the Cloud API, and one Host Agent:

```bash
docker compose up --build -d
docker compose ps
```

The Cloud API container waits for PostgreSQL, runs the committed Alembic
migrations, then starts the API on port `8001`. Readiness is available at:

```text
GET http://127.0.0.1:8001/health/ready
```

## Deploying A Jenkins-Built Image

`docker compose up --build` above builds the image from source on the target
host. For environments that pull a pre-built image instead, `Jenkinsfile` runs
the pytest suite (`-m "not integration"`), builds the image from the same
`Dockerfile`, smoke-tests both console scripts (`device-cloud-api --help`,
`device-host-agent --help`), and pushes `<REGISTRY>/<IMAGE_NAME>:<BUILD_NUMBER>-<git short sha>`
plus `:latest` to the configured registry.

`compose.deploy.yaml` contains only PostgreSQL and the Cloud API; Host Agents
run at their edge sites rather than beside the Cloud API. It references the
fixed Jenkins registry image and interpolates only `IMAGE_TAG` (see
`.env.example`). Deploy without a local build step:

```bash
docker compose -f compose.deploy.yaml pull
docker compose -f compose.deploy.yaml up -d
docker compose -f compose.deploy.yaml ps
```

For a deployment that manages processes outside Compose, apply migrations
before starting the new Cloud API version:

```powershell
$env:CLOUD_DATABASE_URL = "postgresql+psycopg://USER:PASSWORD@HOST:5432/device_cloud"
uv run alembic -c packages/cloud-platform/cloud/migrations/alembic.ini upgrade head
uv run --package device-cloud-api device-cloud-api --host 0.0.0.0 --port 8001
```

Production startup requires `CLOUD_ENVIRONMENT=production`, a current schema,
and HTTPS before browser sessions are exposed. It does not require credential
JSON. Create the first administrator interactively after startup, before
opening the Console to operators.

## Authentication And Host Identities

Cloud users are the human authorization boundary. Their `viewer`, `operator`,
and `admin` roles map to the existing API scopes, and the browser sends an
`HttpOnly` session cookie plus CSRF proof for unsafe operations. The deployment
does not accept `CLOUD_PUBLIC_CREDENTIALS_JSON`,
`CLOUD_HOST_CREDENTIALS_JSON`, or `CLOUD_ENROLLMENT_TOKENS_JSON`.

A fresh Host sends its generated candidate secret only during direct
registration. The Cloud stores its digest and returns a `host_id`; later Host
operations use that secret and are strictly bound to the returned `host_id`.
Protect the persisted Host identity file as a bearer secret.

Dynamically enrolled Host credentials are stored as digests in the cloud
database. This release exposes repository-level revocation rather than a public
administration endpoint. An operator with database deployment access can revoke
a Host without deleting its task history:

```bash
export HOST_ID="host-..."
uv run --package device-cloud-platform python - <<'PY'
import os
from cloud.database import CloudDatabase
from core.models import utc_now

database = CloudDatabase(os.environ["CLOUD_DATABASE_URL"], create_schema=False)
try:
    changed = database.repository.revoke_enrolled_host(
        os.environ["HOST_ID"],
        revoked_at=utc_now(),
    )
    print("revoked" if changed else "not an enrolled host")
finally:
    database.close()
PY
```

## Cloud Console (Web UI)

The repository ships an independent Vue 3 + Vite SPA at `cloud-console/` that
renders task history, devices, hosts, and plugins. Human operators sign in with
a username and password; the Cloud API creates an expiring, revocable
`HttpOnly` session cookie and uses a separate CSRF cookie/header for writes.
An administrator can manage accounts, submission policies, Host operational
limits, and non-secret Cloud-proxy token usage from **Users & limits**. Bearer
tokens remain a compatibility path for API/SDK automation.

### HTTPS and session configuration

`CLOUD_ENVIRONMENT=production` requires `CLOUD_SESSION_COOKIE_SECURE=true`.
Terminate TLS at a reverse proxy and open the same-origin Console through HTTPS,
for example `https://cloud.example.com/console/`. Direct `http://host:8001`
access is for local/test mode only; it cannot retain production login cookies.

The defaults are an 8-hour idle session TTL, 7-day absolute TTL, and a temporary
block after five failed logins in a 15-minute username/client-address window:

```text
CLOUD_USER_SESSION_IDLE_SECONDS=28800
CLOUD_USER_SESSION_ABSOLUTE_SECONDS=604800
CLOUD_LOGIN_FAILURE_LIMIT=5
CLOUD_LOGIN_FAILURE_WINDOW_SECONDS=900
CLOUD_LOGIN_BLOCK_SECONDS=900
CLOUD_SESSION_COOKIE_SECURE=true
CLOUD_TRUST_PROXY_HEADERS=false
```

Set `CLOUD_TRUST_PROXY_HEADERS=true` only when a trusted proxy overwrites
`X-Forwarded-For` before requests reach the Cloud API. This affects the
client-address bucket used for login throttling; the default is safe when the
application is reached directly.

### Create and recover administrator accounts

After migrations and Cloud API startup, create the first account interactively:

```bash
docker compose exec cloud-api \
  device-cloud-admin users create \
  --username admin --display-name "Cloud Administrator" --role admin
```

The command prompts twice for the password, so it does not enter shell history,
Compose configuration, process arguments, logs, or container inspection output.
Recovery commands are also interactive:

```bash
docker compose exec cloud-api device-cloud-admin users reset-password --username admin
docker compose exec cloud-api device-cloud-admin users enable --username admin
docker compose exec cloud-api device-cloud-admin users revoke-sessions --username admin
```

Roles are fixed: `viewer` can read tasks/pool/plugins; `operator` additionally
submits tasks; `admin` has unrestricted Cloud API access. The administration
CLI creates users, resets passwords, changes roles, enables accounts, and
revokes sessions. New and reset users must change their temporary password
before accessing other resources, and the API will not disable or demote the
last enabled administrator.

### Configure the CORS allow-list

The Cloud API has no CORS middleware by default. Before a browser can call it
cross-origin, set `CLOUD_CONSOLE_CORS_ORIGINS` to a comma-separated allow-list
that includes the exact origin (scheme + host + port, no trailing slash) the
operator's browser will load the console from:

```bash
# Allow a local Vite dev server
export CLOUD_CONSOLE_CORS_ORIGINS="http://127.0.0.1:5173"
# Or a deployed origin
export CLOUD_CONSOLE_CORS_ORIGINS="https://console.example.com"
```

Restart the Cloud API after changing this env. The middleware is added only
when the allow-list is non-empty — existing deployments see no behavior change
until an operator opts in. Blanket `allow_origins=["*"]` is intentionally not
supported because browser sessions are credentialed.

### Run the console

```bash
cd cloud-console
cp .env.example .env.local
# Edit .env.local if your Cloud API is not at http://127.0.0.1:8001
npm install
npm run dev
```

Vite prints a local URL (default `http://127.0.0.1:5173`). That exact origin
must be in `CLOUD_CONSOLE_CORS_ORIGINS` on the Cloud API. For local development
set `CLOUD_SESSION_COOKIE_SECURE=false`, then open the dev URL and sign in with
a user account. The Console sends credentialed requests and attaches CSRF proof
to writes.

For a production build, run `npm run build` and serve the resulting `dist/`
behind any static file server or CDN, with `VITE_CLOUD_API_BASE_URL` baked in
at build time. The deployed origin must be in `CLOUD_CONSOLE_CORS_ORIGINS`.

### Same-origin deployment (baked into the Cloud API image)

The Jenkins-built Docker image already carries the SPA at `/app/console-static`
and sets `CLOUD_CONSOLE_STATIC_DIR` in the image itself. The Cloud API serves
the Console at `/console/` (visiting `/` 307-redirects there). Put that origin
behind an HTTPS reverse proxy, then open for example
`https://cloud.example.com/` directly — no separate dev server, static host, or
CORS allow-list is needed because the SPA and API share one origin.

The SPA shell (`index.html`, JS, CSS) is served without credentials by design
so it can render the login page. All `/v1/*` resource calls remain scope-gated,
and unsafe cookie-authenticated calls require CSRF proof. The browser receives
only the non-secret CSRF value; it never receives the `HttpOnly` session secret.

Source-based local development leaves `CLOUD_CONSOLE_STATIC_DIR` unset; the
mount remains conditional on that image-provided setting.

Jenkins build args (`NODE_IMAGE`, `NPM_REGISTRY`, `UV_IMAGE`, `APT_MIRROR`,
`UV_INDEX_URL`) default to CN mirrors so builds don't time out pulling from
Docker Hub / ghcr.io / npmjs.org / deb.debian.org. Blank any of them to fall
back to the upstream.

## Runtime AI Planner

The Host Agent reuses the local Runtime planner. Unlike the shared Runtime
library (whose own default is the deterministic stub planner), the **Host
Agent defaults `AI_PLANNER_ENABLED` to on** -- it is the actual device-control
path, so goal assignments use a model unless an operator explicitly opts out.
Provide provider credentials before deploying:

```text
AI_PLANNER_PROVIDER=anthropic
AI_PLANNER_MODEL=claude-sonnet-5
AI_PLANNER_TIMEOUT_SECONDS=30
ANTHROPIC_API_KEY=<secret manager reference>
```

Without a valid API key, every planning step raises immediately and the task
fails on its first step (no silent fallback to the stub planner). Set
`AI_PLANNER_ENABLED=false` to opt back out to the deterministic stub planner
(e.g. for offline/dev hosts with no provider credentials).

For OpenAI, set `AI_PLANNER_PROVIDER=openai`, choose the deployed model through
`AI_PLANNER_MODEL`, and provide `OPENAI_API_KEY`. This is the **`direct`
transport** (the default): the Host Agent holds provider credentials and
calls Anthropic/OpenAI itself.

### Cloud-proxy transport (`AI_PLANNER_TRANSPORT=cloud`)

Set `AI_PLANNER_TRANSPORT=cloud` on the Host Agent to instead route every
planning decision through the Cloud API's
`POST /internal/v1/hosts/{host_id}/planner/decide` endpoint (the same
host-scoped bearer credential used for heartbeat/claim/renew/result). In this
mode:

- **Provider configuration lives in the Cloud database.** Set
  `CLOUD_LLM_PROVIDER_ENCRYPTION_KEY` from the deployment secret manager, then
  sign in to `/console/` as an administrator and create an active entry under
  **LLM providers**. Provider API keys are encrypted in the database and are
  never returned by the API or Console. Edge Hosts do not hold Provider keys.
  The Cloud API does not read `AI_PLANNER_PROVIDER`, `AI_PLANNER_MODEL`,
  `AI_PLANNER_TIMEOUT_SECONDS`, `ANTHROPIC_API_KEY`, or `OPENAI_API_KEY`.
- **Profile types:** choose **Anthropic** for native Anthropic tool use, or
  **OpenAI-compatible** for the OpenAI Chat Completions tool-calling protocol.
  Leave its Base URL blank for official OpenAI, or provide the compatible
  provider's absolute HTTP(S) `/v1` endpoint. Providers requiring another
  request schema or custom authentication are not supported by this path.
- **Activation is immediate:** a newly activated enabled profile becomes the
  Provider/model for the next Cloud-proxy planner decision. A Cloud-planner
  request fails closed until one enabled profile is active; it never falls back
  to a Cloud API environment credential.
- **Trade-offs to accept before enabling:**
  - *Latency*: every planning step now makes a round trip to the Cloud API in
    addition to the LLM provider call.
  - *Availability coupling*: unlike heartbeat/claim (which tolerate transient
    Cloud API outages via retry/backoff), a planning step fails immediately if
    the Cloud API or its configured provider is unreachable -- there is no
    fallback to the stub planner or to a local direct call.
  - *Expanded data path*: goal/scene prompts and screenshots now transit the
    Cloud API. The endpoint logs only metadata (host id, resolved tool name,
    latency, error class) and never prompt text or screenshot bytes, but the
    request bodies themselves do cross the network to the control plane.

`AI_PLANNER_TRANSPORT` unset or `direct` preserves the existing
direct-to-provider behavior with no change.

### Host governance and Cloud-proxy token budgets

After deploying the API migration, an administrator can use **Users & limits**
in `/console/` to restrict a user's task submission to explicit Host/Device
targets, enable or disable Host self-submission, and set a Host active-task
limit. Policies are enforced by the API and scheduler; hiding Console controls
is not an authorization boundary.

Cloud-enforced token budgets require `AI_PLANNER_TRANSPORT=cloud`. Before each
proxy decision, the API atomically reserves
`CLOUD_PLANNER_TOKEN_RESERVATION_CEILING` tokens (default `4096`) for the
current UTC day. Set `CLOUD_PLANNER_TOKEN_RESERVATION_TTL_SECONDS` (default
`300`) to bound an unknown-usage reservation after provider/transport failure.
The daily Host budget must accommodate the reservation ceiling; otherwise the
proxy rejects before calling the provider. On a provider response, the
reservation is settled to reported usage and the Console retains only timestamp,
provider/model, token counts, and optional task/attempt identifiers.

Hosts reporting `AI_PLANNER_TRANSPORT=direct` are explicitly shown as
**unmetered**. Cloud cannot enforce or verify their provider token use. Do not
interpret an unmetered Host's absence of usage events as budget compliance.

Roll out in this order: migrate the Cloud database, provision
`CLOUD_LLM_PROVIDER_ENCRYPTION_KEY`, deploy the Cloud API, create and activate
a Provider profile in `/console/`, then switch a pilot Host to Cloud transport.
Configure a budget above the reservation ceiling and review usage events before
enabling budgets fleet-wide. Rotate a Provider API key by editing that profile;
the encryption master key is deployment-managed and must be preserved with the
database backups. A rollback to an older Cloud API requires restoring its
legacy Provider environment configuration, while a rollback to `direct`
transport requires valid provider credentials on that Host; preserve usage and
policy rows rather than deleting accounting history.

## Operational Limitations

Run exactly one scheduler-enabled Cloud API process. SQLite supports only the
documented single-control-plane development mode. PostgreSQL row locking makes
assignment and claim transactions safe if requests overlap, but this release
does not implement scheduler leader election or claim active-active scheduler
operation. Starting multiple Cloud API replicas would start one scheduler loop
per replica and is outside the supported deployment topology.

Device execution provides at-least-once side-effect semantics, not exactly-once
semantics. A device action can succeed immediately before the Host Agent loses
its lease or its result response, after which the control plane may retry the
task. Lease renewal and cooperative stop checks prevent later interruptible
actions where possible, but they cannot roll back an action already sent to a
device or safely terminate an in-progress synchronous driver call.

Use bounded attempts, inspect task attempt and failure metadata, and design
device workflows to tolerate repeated actions when the target operation allows
it. Do not use this release for operations that require a transactional
exactly-once guarantee across the cloud database and an external device.

## Shutdown And Rollback

For a normal shutdown, stop Host Agents first so they stop polling, interrupt
later cooperative actions, finish terminal reporting where the lease remains
valid, and attempt a final heartbeat. Stop the Cloud API after Host Agents have
exited, then stop PostgreSQL only if the database itself is being maintained:

```bash
docker compose stop host-agent
docker compose stop cloud-api
docker compose stop postgres
```

For rollback:

1. Stop all Host Agents and the Cloud API.
2. Back up PostgreSQL or the SQLite database file.
3. Before rolling back to an image that still requires static credentials,
   restore that image's matching deployment configuration and provision the
   required legacy credentials outside this release's Compose contract.
4. If the previous application version cannot use the current schema, run the
   tested downgrade while no application process is connected:

   ```bash
   uv run alembic -c packages/cloud-platform/cloud/migrations/alembic.ini downgrade -1
   ```

5. Restore the previous application image or checkout and start the Cloud API.
6. Verify `/health/ready`, then restart Host Agents with identities compatible
   with the restored Cloud API.

Downgrading revision 0002 removes dynamic credential bindings and durable
device enrollment mappings. It retains the revision-0001 Host heartbeat rows,
pooled devices, queued tasks, attempts, and plugins.

Do not remove the PostgreSQL volume during an application rollback. Queued and
attempt history are durable database state and should remain available to the
restored or forward-deployed control plane.