agentic-mobile-control/openspec/specs/agent-runtime/spec.md

agent-runtime Specification

Purpose

Define the behavior of the AI Planner runtime, including LLM-driven action selection, completion signaling, provider abstraction, default-disabling, and the narrow screenshot exception to the Perception Boundary.

Requirements

Requirement: LLM-driven Planner selects exactly one grounded action per turn

The system SHALL provide a Planner implementation that, given a goal, the current Scene, and recent task history, uses native LLM tool/function calling to select exactly one action (or the completion signal defined below) per plan() invocation, grounding any coordinates in the current turn's Scene element bounds.

Scenario: Planner selects a single action for the current turn

• WHEN the AI Planner is invoked with a goal and the current Scene
• THEN it returns at most one PlannedStep, whose action and arguments come from exactly one tool call chosen by the underlying LLM for that turn

Scenario: Planner re-decides every turn from the current Scene

• WHEN the AI Planner is invoked again after a prior step has executed
• THEN its decision is grounded in the newly observed Scene for that turn, not in coordinates or assumptions carried over from a previous turn

Requirement: Explicit finish_task completion and failure signal

The system SHALL treat task completion and task failure as explicit, model-driven signals via a dedicated finish_task(success, reason) tool, rather than inferring either outcome from the model declining to call any tool.

Scenario: Model signals successful completion

• WHEN the model calls finish_task with success=True
• THEN the Planner returns an empty step list and the task is marked completed

Scenario: Model signals it cannot complete the goal

• WHEN the model calls finish_task with success=False and a reason
• THEN the task is marked failed with that reason, without attempting any further planning steps

Requirement: Pluggable dual-provider tool-calling abstraction

The system SHALL support at least two interchangeable LLM providers (Anthropic native tool use and OpenAI function calling) for the AI Planner's decision calls, selectable via configuration, with both providers constrained to return exactly one tool call per request. Independently of provider selection, the system SHALL support at least two transports for making that decision call -- direct-to-provider (the tool-calling client calls the provider's SDK itself, using locally configured credentials) and cloud-proxy (the tool-calling client calls the Cloud Control Plane's planner-decision endpoint, which calls the provider using cloud-held credentials) -- selectable via configuration without requiring any change to AIPlanner's own decision logic.

Scenario: Provider selected via configuration

• WHEN the AI Planner is configured with a given provider identifier
• THEN it constructs and uses the tool-calling client for that provider without requiring any change to AIPlanner's own decision logic

Scenario: Provider response resolves to a single decision

• WHEN either supported provider returns a response to a tool-calling request
• THEN the response is parsed into exactly one tool name and one arguments object, regardless of which provider produced it

Scenario: Transport selected via configuration

• WHEN the Host Agent is configured with a given transport (direct or cloud-proxy)
• THEN AIPlanner is constructed with the tool-calling client for that transport, and its own decision logic is unchanged regardless of which transport is in effect

Scenario: Cloud-proxy transport is the default

• WHEN no transport is explicitly configured
• THEN the AI Planner uses the cloud-proxy transport and the Cloud Control Plane's planner-decision endpoint

Scenario: Direct transport remains available by explicit configuration

• WHEN the Host Agent is configured with the direct transport
• THEN the AI Planner uses the direct-to-provider transport with locally configured credentials

Scenario: Cloud-proxy transport resolves a decision without a local provider client

• WHEN the Host Agent is configured with the cloud-proxy transport
• THEN its tool-calling client sends the decision request to the Cloud Control Plane's planner-decision endpoint instead of constructing a local Anthropic or OpenAI SDK client

Requirement: AI Planner is disabled by default and additive to the existing Planner

The system SHALL default to the existing non-LLM Planner unless the AI Planner is explicitly enabled via configuration, and SHALL NOT alter the existing Planner's behavior, dependencies, or any caller's construction of TaskRunner when left disabled.

Scenario: AI Planner disabled (default)

• WHEN TaskRunner is constructed without an explicit planner and without the AI Planner enabled in configuration
• THEN it uses the existing non-LLM Planner, unchanged from before this capability existed

Scenario: AI Planner enabled via configuration

• WHEN TaskRunner is constructed without an explicit planner and with the AI Planner enabled in configuration
• THEN it uses the AI Planner, configured with the selected provider and model

Requirement: Screenshot access is a Planner-only, narrow exception to the Perception Boundary

The system SHALL allow the AI Planner, and only the AI Planner, to receive the current step's raw screenshot bytes alongside the Scene for vision-grounded decision-making, while every other perception consumer SHALL continue to receive only the Scene.

Scenario: Planner receives both Scene and screenshot

• WHEN a screenshot for the current step is available
• THEN the AI Planner's decision call includes both the Scene JSON and the raw screenshot bytes for that step

Scenario: Screenshot unavailable does not block planning

• WHEN a screenshot for the current step cannot be obtained
• THEN the AI Planner still produces a decision using the Scene alone, and this is not treated as a task failure

Scenario: No other consumer receives raw screenshot bytes

• WHEN any component other than the AI Planner (for example, api, tools, perception, or storage) consumes perception output
• THEN it receives only the Scene, never raw screenshot bytes