225 lines
13 KiB
Markdown
225 lines
13 KiB
Markdown
## Context
|
|
|
|
The Host Agent's `AIPlanner` (`runtime/ai_planner.py`) delegates the actual
|
|
LLM call to a `ToolCallingClient` (`runtime/tool_calling_client.py`):
|
|
`AnthropicToolCallingClient`/`OpenAIToolCallingClient` each build a provider
|
|
SDK client that implicitly reads `ANTHROPIC_API_KEY`/`OPENAI_API_KEY` from
|
|
the Host Agent's own process environment, and `build_client()` selects
|
|
between them based on `PlannerConfig.provider` (from
|
|
`runtime/planner_config.py`, itself read from the Host Agent's local env).
|
|
None of this touches the Cloud Control Plane today.
|
|
|
|
The Cloud Control Plane (`apps/cloud-api` + `packages/cloud-platform/cloud`)
|
|
already authenticates every Host Agent with a host-scoped bearer credential
|
|
for heartbeat, claim/long-poll, lease renewal, and terminal result reporting
|
|
(`cloud/internal_api/*`), with the Host Agent importing shared Pydantic
|
|
models directly from `cloud.internal_api.models` rather than duplicating
|
|
schemas. Separately, `packages/cloud-platform` already depends on the root
|
|
Runtime package (`device-agent-runtime`, workspace dependency), the same way
|
|
`apps/device-host-agent` does -- so cloud-side code can import
|
|
`runtime.tool_calling_client` directly instead of reimplementing
|
|
Anthropic/OpenAI wire-format handling a second time.
|
|
|
|
A prior change in this session made the Host Agent default
|
|
`AI_PLANNER_ENABLED` to on (`apps/device-host-agent/host_agent/execution.py`).
|
|
That means every Host Agent now needs a working planner path by default,
|
|
which raises the stakes on where its provider credentials live.
|
|
|
|
## Goals / Non-Goals
|
|
|
|
**Goals:**
|
|
- Let an operator configure LLM provider, model, and credentials once, on
|
|
the Cloud Control Plane, instead of per Host Agent.
|
|
- Let a Host Agent execute AI-planned tasks without holding
|
|
`ANTHROPIC_API_KEY`/`OPENAI_API_KEY` locally, by proxying its planner
|
|
decision calls through the Cloud Control Plane.
|
|
- Reuse existing Host<->Cloud authentication and existing provider
|
|
wire-format code; add no new auth scope and no duplicated Anthropic/OpenAI
|
|
translation logic.
|
|
- Preserve the direct-to-provider path as a fully supported explicit opt-out
|
|
for Host Agent deployments that require local provider keys.
|
|
|
|
**Non-Goals:**
|
|
- Not moving prompt construction (`runtime/planner_prompts.py`, Scene
|
|
serialization, world/history summarization) to the cloud. The Host Agent
|
|
keeps building `system_prompt`/`user_prompt` locally; the cloud endpoint
|
|
is a thin "make this one LLM call and return one tool call" proxy, not a
|
|
re-implementation of planning.
|
|
- Not building per-tenant/per-host provider overrides, usage metering,
|
|
rate limiting, or a management UI for the proxy. Single cloud-wide
|
|
provider/model/credential configuration only, matching how
|
|
`AI_PLANNER_*` is configured today (env-based, one value cluster).
|
|
- Not changing the existing heartbeat/claim/renew/result protocol or its
|
|
spec (`host-agent-protocol`).
|
|
- Not deprecating or removing the direct-to-provider transport.
|
|
|
|
## Decisions
|
|
|
|
### D1: New endpoint reuses `runtime.tool_calling_client` server-side instead of reimplementing provider calls
|
|
`cloud.internal_api` adds a route whose handler constructs an
|
|
`AnthropicToolCallingClient`/`OpenAIToolCallingClient` (imported from
|
|
`runtime.tool_calling_client`, already an allowed dependency direction since
|
|
`packages/cloud-platform` depends on `device-agent-runtime`) using
|
|
Cloud-side provider/model/credential configuration, and calls `.decide(...)`
|
|
with the request payload. This eliminates a second implementation of
|
|
Anthropic/OpenAI tool-calling wire-format translation, which
|
|
`ai-planner-runtime/design.md` already flagged as a drift risk for the
|
|
*two-provider* case -- a *four-implementation* case (two providers x two
|
|
transports) would be worse.
|
|
|
|
Alternative considered: hand-roll a minimal cloud-side Anthropic/OpenAI
|
|
client in `cloud.internal_api`. Rejected: duplicates parsing logic
|
|
(`_decision_from_anthropic_response`, `_decision_from_openai_response`) that
|
|
must stay in lockstep with the Host Agent's local transport as providers'
|
|
APIs evolve.
|
|
|
|
### D2: New transport axis, orthogonal to provider selection
|
|
Host Agent config gains `AI_PLANNER_TRANSPORT` (`cloud` default | `direct`),
|
|
independent of `AI_PLANNER_PROVIDER`. `cloud` builds a new
|
|
`CloudProxyToolCallingClient` instead; provider/model selection and
|
|
credentials for that path live in the Cloud API's own
|
|
active Provider profile, not the Host Agent's. `direct` remains an explicit
|
|
opt-out that builds the provider SDK client locally.
|
|
|
|
Alternative considered: overload `AI_PLANNER_PROVIDER=cloud` as a third
|
|
provider value. Rejected: provider and transport are different axes (a
|
|
`cloud` transport still ultimately calls `anthropic` or `openai`), and
|
|
conflating them would make the Cloud API's own provider config harder to
|
|
reason about ("provider" would mean different things on each side).
|
|
|
|
### D3: `CloudProxyToolCallingClient` lives in `host_agent`, not `runtime`
|
|
`runtime/tool_calling_client.py`'s `ToolCallingClient` stays a plain
|
|
`Protocol`; the new client is added in `apps/device-host-agent/host_agent/`
|
|
(`host_agent/cloud_planner_client.py`) and satisfies that Protocol
|
|
structurally. This preserves the existing boundary enforced by
|
|
`apps/device-host-agent/tests/test_execution.py::test_runtime_owned_packages_do_not_import_host_or_cloud_concerns`,
|
|
which forbids `runtime` (and `core`/`device`/`driver`/`tools`) from
|
|
importing `cloud` or `host_agent`.
|
|
|
|
Implementation note (revised during implementation from the original plan
|
|
of wrapping `HostAgentClient`): `ToolCallingClient.decide()` is a
|
|
**synchronous** Protocol method, and `AIPlanner.plan()` -> `TaskRunner`'s
|
|
step loop runs inside a worker thread spawned via `asyncio.to_thread` (see
|
|
`host_agent/lease.py`'s `ActiveAssignmentRunner`), off the main event loop.
|
|
`HostAgentClient` holds an `httpx.AsyncClient` bound to that main loop, so
|
|
calling it from the worker thread would require event-loop bridging
|
|
(`asyncio.run_coroutine_threadsafe` or similar) for no benefit over a
|
|
simpler alternative. Instead, `CloudProxyToolCallingClient` holds its own
|
|
synchronous `httpx.Client`, mirroring the existing
|
|
`host_agent/client.py::HostAgentEnrollmentClient` pattern (same
|
|
`Authorization: Bearer` header construction, same base URL from
|
|
`HostAgentConfig`), rather than reusing `HostAgentClient`'s async session.
|
|
It still imports request/response models directly from
|
|
`cloud.internal_api.models` -- no duplicated schemas -- so the "no new
|
|
wire-format definitions" intent of D1 is preserved even though the HTTP
|
|
transport itself isn't literally shared with `HostAgentClient`.
|
|
|
|
### D4: Reuse the existing host-scoped bearer credential; no new auth scope
|
|
The new endpoint sits on the same internal router and auth dependency as
|
|
heartbeat/claim/renew/result. A Host Agent that can already reach the Cloud
|
|
Control Plane for task assignment can reach the planner-decide endpoint;
|
|
there is no separate enrollment or credential to provision for this
|
|
capability.
|
|
|
|
Alternative considered: a distinct scope/credential just for planner-proxy
|
|
calls (defense in depth, so a compromised heartbeat credential couldn't run
|
|
up LLM cost). Rejected for this proposal to keep the change additive and
|
|
consistent with how the rest of the internal API already treats "any
|
|
authenticated host" uniformly; revisit if abuse/cost containment becomes a
|
|
concrete concern (see Open Questions).
|
|
|
|
### D5: Request/response payload mirrors the local `ToolCallingClient.decide()` signature
|
|
The proxy request carries `system_prompt`, `user_prompt`, an optional
|
|
base64 screenshot, the tool specs (already JSON-serializable
|
|
`ToolSpec`/dict shapes used to build provider tool schemas), and a timeout.
|
|
The response carries the resolved `tool_name`/`arguments` (or a structured
|
|
error). This keeps the Host Agent and Cloud Control Plane in the same
|
|
shape as today's local call, so `AIPlanner` itself needs zero changes --
|
|
only the client it's constructed with changes.
|
|
|
|
### D6: Cloud does not durably persist prompt text or screenshot bytes from proxy requests
|
|
The endpoint handler processes the request in memory and returns; only
|
|
metadata (host id, tool name decided, latency, error class if any) may be
|
|
logged for observability. This bounds the new sensitive-data surface
|
|
created by routing screenshots and prompts through the cloud (an accepted
|
|
trade-off from the earlier feasibility discussion) to "in transit and in
|
|
process," not "at rest in cloud logs/DB."
|
|
|
|
### D7: Proxy failures surface as `ToolCallUnavailable`, matching direct-transport failure behavior
|
|
Any failure calling the cloud endpoint (network error, auth rejection,
|
|
non-2xx, provider error surfaced by the cloud side, timeout) raises
|
|
`ToolCallUnavailable` from `CloudProxyToolCallingClient.decide()`, exactly
|
|
like today's direct-transport failures. This preserves the existing,
|
|
already-accepted behavior that a planner failure fails the current task
|
|
step immediately with no silent fallback to the stub planner -- this
|
|
proposal does not change that risk profile, only where the call happens.
|
|
|
|
## Risks / Trade-offs
|
|
|
|
- **[Risk] Cloud Control Plane is now in the hot path of every planning
|
|
step by default** -> Mitigation: `direct` transport remains fully
|
|
supported as an explicit opt-out for operators who need offline or
|
|
low-latency operation. Bounded by the
|
|
existing `AI_PLANNER_TIMEOUT_SECONDS`, same as today.
|
|
- **[Risk] Cloud Control Plane outage now stalls planning (not just new
|
|
task assignment) for opted-in hosts** -> Mitigation: same
|
|
`ToolCallUnavailable` -> task-fails-fast behavior as any other planner
|
|
error; no new silent-hang mode. Documented as an explicit trade-off of
|
|
opting into `cloud` transport.
|
|
- **[Risk] Screenshots and prompt text now transit through the Cloud
|
|
Control Plane** -> Mitigation: D6 (no durable persistence); still an
|
|
expansion of the data path operators should account for versus
|
|
direct-to-provider, which never touches the cloud.
|
|
- **[Risk] Cloud API becomes a second place holding provider credentials**
|
|
-> Mitigation: reusing `runtime.tool_calling_client` (D1) keeps this to
|
|
configuration and routing, not new provider-integration code; credential
|
|
handling follows the same "environment or secret manager" pattern already
|
|
documented for the Host Agent in `docs/CLOUD_DEPLOYMENT.md`. Note: this is
|
|
not a *new* package-dependency footprint -- `device-cloud-platform`
|
|
already depends unconditionally on `device-agent-runtime`, which declares
|
|
`anthropic`/`openai`, so both SDKs are already installed wherever the
|
|
Cloud API runs today (verified with `uv run`); only the credentials
|
|
themselves are new.
|
|
- **[Risk] Any authenticated host can drive cloud-held LLM spend** (D4) ->
|
|
Mitigation: none in this proposal beyond existing per-host authentication;
|
|
flagged as an Open Question rather than silently accepted, since it's a
|
|
cost/abuse concern rather than a correctness one.
|
|
- **[Risk] `ai-planner-runtime`'s accepted spec text ("AI Planner is
|
|
disabled by default") is already stale relative to the Host Agent's
|
|
actual default (changed in a prior, separate session change without an
|
|
openspec artifact)** -> Not a risk introduced by this proposal, but this
|
|
proposal's delta spec is written against that same pending, unarchived
|
|
`agent-runtime` capability, so the discrepancy should be reconciled
|
|
once, when `ai-planner-runtime` is archived (see Open Questions).
|
|
|
|
## Migration Plan
|
|
|
|
1. Add Cloud API Provider profile management and the new internal route,
|
|
guarded by the existing host-scoped auth. Configure and activate a
|
|
Cloud-held Provider key before running a Host Agent.
|
|
2. Add the Host Agent's `AI_PLANNER_TRANSPORT` setting (default `cloud`)
|
|
and `CloudProxyToolCallingClient`. Existing deployments that require a
|
|
local provider key set `AI_PLANNER_TRANSPORT=direct` explicitly.
|
|
3. Update `docs/CLOUD_DEPLOYMENT.md` with the proxy configuration path and
|
|
its trade-offs (latency, availability coupling, data-path expansion).
|
|
4. Rollback is setting `AI_PLANNER_TRANSPORT=direct` on affected Hosts and
|
|
restoring their local provider key; the Cloud API route can remain
|
|
deployed but unused.
|
|
|
|
## Open Questions
|
|
|
|
- Should planner-proxy calls eventually get their own credential scope
|
|
separate from heartbeat/claim/renew/result, to bound blast radius and
|
|
allow independent cost/abuse controls? Deferred; revisit if this becomes
|
|
a real deployment.
|
|
- Should the Cloud API expose any per-request cost/usage observability
|
|
(e.g. token counts) given it now brokers every LLM call for `cloud`-
|
|
transport hosts? Out of scope for this proposal's tasks; worth deciding
|
|
before recommending `cloud` transport for cost-sensitive fleets.
|
|
- When `ai-planner-runtime` is archived, its "disabled by default" spec
|
|
text needs reconciling against both the Host Agent's actual default
|
|
(from the earlier, separate change) and this proposal's transport
|
|
addition -- noted here so it isn't lost, matching how
|
|
`edge-host-self-enrollment`'s design.md tracked its own dependency on an
|
|
unarchived pending spec.
|