agentic-mobile-control/openspec/changes/cloud-planner-proxy/design.md

Context

The Host Agent's AIPlanner (runtime/ai_planner.py) delegates the actual LLM call to a ToolCallingClient (runtime/tool_calling_client.py): AnthropicToolCallingClient/OpenAIToolCallingClient each build a provider SDK client that implicitly reads ANTHROPIC_API_KEY/OPENAI_API_KEY from the Host Agent's own process environment, and build_client() selects between them based on PlannerConfig.provider (from runtime/planner_config.py, itself read from the Host Agent's local env). None of this touches the Cloud Control Plane today.

The Cloud Control Plane (apps/cloud-api + packages/cloud-platform/cloud) already authenticates every Host Agent with a host-scoped bearer credential for heartbeat, claim/long-poll, lease renewal, and terminal result reporting (cloud/internal_api/*), with the Host Agent importing shared Pydantic models directly from cloud.internal_api.models rather than duplicating schemas. Separately, packages/cloud-platform already depends on the root Runtime package (device-agent-runtime, workspace dependency), the same way apps/device-host-agent does -- so cloud-side code can import runtime.tool_calling_client directly instead of reimplementing Anthropic/OpenAI wire-format handling a second time.

A prior change in this session made the Host Agent default AI_PLANNER_ENABLED to on (apps/device-host-agent/host_agent/execution.py). That means every Host Agent now needs a working planner path by default, which raises the stakes on where its provider credentials live.

Goals / Non-Goals

Goals:

• Let an operator configure LLM provider, model, and credentials once, on the Cloud Control Plane, instead of per Host Agent.
• Let a Host Agent execute AI-planned tasks without holding ANTHROPIC_API_KEY/OPENAI_API_KEY locally, by proxying its planner decision calls through the Cloud Control Plane.
• Reuse existing Host<->Cloud authentication and existing provider wire-format code; add no new auth scope and no duplicated Anthropic/OpenAI translation logic.
• Preserve today's direct-to-provider path as a fully supported, still-default option, so existing Host Agent deployments with local keys are unaffected.

Non-Goals:

• Not moving prompt construction (runtime/planner_prompts.py, Scene serialization, world/history summarization) to the cloud. The Host Agent keeps building system_prompt/user_prompt locally; the cloud endpoint is a thin "make this one LLM call and return one tool call" proxy, not a re-implementation of planning.
• Not building per-tenant/per-host provider overrides, usage metering, rate limiting, or a management UI for the proxy. Single cloud-wide provider/model/credential configuration only, matching how AI_PLANNER_* is configured today (env-based, one value cluster).
• Not changing the existing heartbeat/claim/renew/result protocol or its spec (host-agent-protocol).
• Not deprecating or removing the direct-to-provider transport.

Decisions

D1: New endpoint reuses runtime.tool_calling_client server-side instead of reimplementing provider calls

cloud.internal_api adds a route whose handler constructs an AnthropicToolCallingClient/OpenAIToolCallingClient (imported from runtime.tool_calling_client, already an allowed dependency direction since packages/cloud-platform depends on device-agent-runtime) using Cloud-side provider/model/credential configuration, and calls .decide(...) with the request payload. This eliminates a second implementation of Anthropic/OpenAI tool-calling wire-format translation, which ai-planner-runtime/design.md already flagged as a drift risk for the two-provider case -- a four-implementation case (two providers x two transports) would be worse.

Alternative considered: hand-roll a minimal cloud-side Anthropic/OpenAI client in cloud.internal_api. Rejected: duplicates parsing logic (_decision_from_anthropic_response, _decision_from_openai_response) that must stay in lockstep with the Host Agent's local transport as providers' APIs evolve.

D2: New transport axis, orthogonal to provider selection

Host Agent config gains AI_PLANNER_TRANSPORT (direct default | cloud), independent of AI_PLANNER_PROVIDER. direct is today's behavior unchanged (Host Agent builds the SDK client itself). cloud builds a new CloudProxyToolCallingClient instead; provider/model selection and credentials for that path live in the Cloud API's own AI_PLANNER_PROVIDER/AI_PLANNER_MODEL/ANTHROPIC_API_KEY/OPENAI_API_KEY configuration, not the Host Agent's.

Alternative considered: overload AI_PLANNER_PROVIDER=cloud as a third provider value. Rejected: provider and transport are different axes (a cloud transport still ultimately calls anthropic or openai), and conflating them would make the Cloud API's own provider config harder to reason about ("provider" would mean different things on each side).

D3: CloudProxyToolCallingClient lives in host_agent, not runtime

runtime/tool_calling_client.py's ToolCallingClient stays a plain Protocol; the new client is added in apps/device-host-agent/host_agent/ (host_agent/cloud_planner_client.py) and satisfies that Protocol structurally. This preserves the existing boundary enforced by apps/device-host-agent/tests/test_execution.py::test_runtime_owned_packages_do_not_import_host_or_cloud_concerns, which forbids runtime (and core/device/driver/tools) from importing cloud or host_agent.

Implementation note (revised during implementation from the original plan of wrapping HostAgentClient): ToolCallingClient.decide() is a synchronous Protocol method, and AIPlanner.plan() -> TaskRunner's step loop runs inside a worker thread spawned via asyncio.to_thread (see host_agent/lease.py's ActiveAssignmentRunner), off the main event loop. HostAgentClient holds an httpx.AsyncClient bound to that main loop, so calling it from the worker thread would require event-loop bridging (asyncio.run_coroutine_threadsafe or similar) for no benefit over a simpler alternative. Instead, CloudProxyToolCallingClient holds its own synchronous httpx.Client, mirroring the existing host_agent/client.py::HostAgentEnrollmentClient pattern (same Authorization: Bearer header construction, same base URL from HostAgentConfig), rather than reusing HostAgentClient's async session. It still imports request/response models directly from cloud.internal_api.models -- no duplicated schemas -- so the "no new wire-format definitions" intent of D1 is preserved even though the HTTP transport itself isn't literally shared with HostAgentClient.

D4: Reuse the existing host-scoped bearer credential; no new auth scope

The new endpoint sits on the same internal router and auth dependency as heartbeat/claim/renew/result. A Host Agent that can already reach the Cloud Control Plane for task assignment can reach the planner-decide endpoint; there is no separate enrollment or credential to provision for this capability.

Alternative considered: a distinct scope/credential just for planner-proxy calls (defense in depth, so a compromised heartbeat credential couldn't run up LLM cost). Rejected for this proposal to keep the change additive and consistent with how the rest of the internal API already treats "any authenticated host" uniformly; revisit if abuse/cost containment becomes a concrete concern (see Open Questions).

D5: Request/response payload mirrors the local ToolCallingClient.decide() signature

The proxy request carries system_prompt, user_prompt, an optional base64 screenshot, the tool specs (already JSON-serializable ToolSpec/dict shapes used to build provider tool schemas), and a timeout. The response carries the resolved tool_name/arguments (or a structured error). This keeps the Host Agent and Cloud Control Plane in the same shape as today's local call, so AIPlanner itself needs zero changes -- only the client it's constructed with changes.

D6: Cloud does not durably persist prompt text or screenshot bytes from proxy requests

The endpoint handler processes the request in memory and returns; only metadata (host id, tool name decided, latency, error class if any) may be logged for observability. This bounds the new sensitive-data surface created by routing screenshots and prompts through the cloud (an accepted trade-off from the earlier feasibility discussion) to "in transit and in process," not "at rest in cloud logs/DB."

D7: Proxy failures surface as ToolCallUnavailable, matching direct-transport failure behavior

Any failure calling the cloud endpoint (network error, auth rejection, non-2xx, provider error surfaced by the cloud side, timeout) raises ToolCallUnavailable from CloudProxyToolCallingClient.decide(), exactly like today's direct-transport failures. This preserves the existing, already-accepted behavior that a planner failure fails the current task step immediately with no silent fallback to the stub planner -- this proposal does not change that risk profile, only where the call happens.

Risks / Trade-offs

• [Risk] Cloud Control Plane is now in the hot path of every planning step for hosts on the cloud transport -> Mitigation: direct transport remains the default and fully supported; operators who need offline/low-latency operation simply don't opt in. Bounded by the existing AI_PLANNER_TIMEOUT_SECONDS, same as today.
• [Risk] Cloud Control Plane outage now stalls planning (not just new task assignment) for opted-in hosts -> Mitigation: same ToolCallUnavailable -> task-fails-fast behavior as any other planner error; no new silent-hang mode. Documented as an explicit trade-off of opting into cloud transport.
• [Risk] Screenshots and prompt text now transit through the Cloud Control Plane -> Mitigation: D6 (no durable persistence); still an expansion of the data path operators should account for versus direct-to-provider, which never touches the cloud.
• [Risk] Cloud API becomes a second place holding provider credentials -> Mitigation: reusing runtime.tool_calling_client (D1) keeps this to configuration and routing, not new provider-integration code; credential handling follows the same "environment or secret manager" pattern already documented for the Host Agent in docs/CLOUD_DEPLOYMENT.md. Note: this is not a new package-dependency footprint -- device-cloud-platform already depends unconditionally on device-agent-runtime, which declares anthropic/openai, so both SDKs are already installed wherever the Cloud API runs today (verified with uv run); only the credentials themselves are new.
• [Risk] Any authenticated host can drive cloud-held LLM spend (D4) -> Mitigation: none in this proposal beyond existing per-host authentication; flagged as an Open Question rather than silently accepted, since it's a cost/abuse concern rather than a correctness one.
• [Risk] ai-planner-runtime's accepted spec text ("AI Planner is disabled by default") is already stale relative to the Host Agent's actual default (changed in a prior, separate session change without an openspec artifact) -> Not a risk introduced by this proposal, but this proposal's delta spec is written against that same pending, unarchived agent-runtime capability, so the discrepancy should be reconciled once, when ai-planner-runtime is archived (see Open Questions).

Migration Plan

1. Add Cloud API configuration (AI_PLANNER_PROVIDER/AI_PLANNER_MODEL/ AI_PLANNER_TIMEOUT_SECONDS/provider API keys) and the new internal route, guarded by the existing host-scoped auth. Off by default in the sense that no Host Agent calls it until configured to use cloud transport.
2. Add the Host Agent's AI_PLANNER_TRANSPORT setting (default direct) and CloudProxyToolCallingClient. Existing deployments are unaffected until an operator sets AI_PLANNER_TRANSPORT=cloud and removes the local provider key.
3. Update docs/CLOUD_DEPLOYMENT.md with the proxy configuration path and its trade-offs (latency, availability coupling, data-path expansion).
4. Rollback is setting AI_PLANNER_TRANSPORT=direct (or unsetting it) on affected hosts and restoring their local provider key; the Cloud API route can remain deployed but unused.

Open Questions

• Should planner-proxy calls eventually get their own credential scope separate from heartbeat/claim/renew/result, to bound blast radius and allow independent cost/abuse controls? Deferred; revisit if this becomes a real deployment.
• Should the Cloud API expose any per-request cost/usage observability (e.g. token counts) given it now brokers every LLM call for cloud- transport hosts? Out of scope for this proposal's tasks; worth deciding before recommending cloud transport for cost-sensitive fleets.
• When ai-planner-runtime is archived, its "disabled by default" spec text needs reconciling against both the Host Agent's actual default (from the earlier, separate change) and this proposal's transport addition -- noted here so it isn't lost, matching how edge-host-self-enrollment's design.md tracked its own dependency on an unarchived pending spec.