agentic-mobile-control/openspec/specs/agent-runtime/spec.md

# agent-runtime Specification

## Purpose
Define the behavior of the AI Planner runtime, including LLM-driven action
selection, completion signaling, provider abstraction, default-disabling, and
the narrow screenshot exception to the Perception Boundary.

## Requirements

### Requirement: LLM-driven Planner selects exactly one grounded action per turn
The system SHALL provide a Planner implementation that, given a goal, the
current Scene, and recent task history, uses native LLM tool/function
calling to select exactly one action (or the completion signal defined
below) per `plan()` invocation, grounding any coordinates in the current
turn's Scene element bounds.

#### Scenario: Planner selects a single action for the current turn
- **WHEN** the AI Planner is invoked with a goal and the current Scene
- **THEN** it returns at most one `PlannedStep`, whose action and arguments
  come from exactly one tool call chosen by the underlying LLM for that turn

#### Scenario: Planner re-decides every turn from the current Scene
- **WHEN** the AI Planner is invoked again after a prior step has executed
- **THEN** its decision is grounded in the newly observed Scene for that
  turn, not in coordinates or assumptions carried over from a previous turn

### Requirement: Explicit finish_task completion and failure signal
The system SHALL treat task completion and task failure as explicit,
model-driven signals via a dedicated `finish_task(success, reason)` tool,
rather than inferring either outcome from the model declining to call any
tool.

#### Scenario: Model signals successful completion
- **WHEN** the model calls `finish_task` with `success=True`
- **THEN** the Planner returns an empty step list and the task is marked
  completed

#### Scenario: Model signals it cannot complete the goal
- **WHEN** the model calls `finish_task` with `success=False` and a `reason`
- **THEN** the task is marked failed with that reason, without attempting
  any further planning steps

### Requirement: Pluggable dual-provider tool-calling abstraction
The system SHALL support at least two interchangeable LLM providers
(Anthropic native tool use and OpenAI function calling) for the AI
Planner's decision calls, selectable via configuration, with both providers
constrained to return exactly one tool call per request. Independently of
provider selection, the system SHALL support at least two transports for
making that decision call -- direct-to-provider (the tool-calling client
calls the provider's SDK itself, using locally configured credentials) and
cloud-proxy (the tool-calling client calls the Cloud Control Plane's
planner-decision endpoint, which calls the provider using cloud-held
credentials) -- selectable via configuration without requiring any change
to `AIPlanner`'s own decision logic.

#### Scenario: Provider selected via configuration
- **WHEN** the AI Planner is configured with a given provider identifier
- **THEN** it constructs and uses the tool-calling client for that provider
  without requiring any change to `AIPlanner`'s own decision logic

#### Scenario: Provider response resolves to a single decision
- **WHEN** either supported provider returns a response to a tool-calling
  request
- **THEN** the response is parsed into exactly one tool name and one
  arguments object, regardless of which provider produced it

#### Scenario: Transport selected via configuration
- **WHEN** the Host Agent is configured with a given transport (direct or
  cloud-proxy)
- **THEN** `AIPlanner` is constructed with the tool-calling client for that
  transport, and its own decision logic is unchanged regardless of which
  transport is in effect

#### Scenario: Cloud-proxy transport is the default
- **WHEN** no transport is explicitly configured
- **THEN** the AI Planner uses the cloud-proxy transport and the Cloud
  Control Plane's planner-decision endpoint

#### Scenario: Direct transport remains available by explicit configuration
- **WHEN** the Host Agent is configured with the direct transport
- **THEN** the AI Planner uses the direct-to-provider transport with locally
  configured credentials

#### Scenario: Cloud-proxy transport resolves a decision without a local provider client
- **WHEN** the Host Agent is configured with the cloud-proxy transport
- **THEN** its tool-calling client sends the decision request to the Cloud
  Control Plane's planner-decision endpoint instead of constructing a local
  Anthropic or OpenAI SDK client

### Requirement: AI Planner is disabled by default and additive to the existing Planner
The system SHALL default to the existing non-LLM Planner unless the AI
Planner is explicitly enabled via configuration, and SHALL NOT alter the
existing Planner's behavior, dependencies, or any caller's construction of
`TaskRunner` when left disabled.

#### Scenario: AI Planner disabled (default)
- **WHEN** `TaskRunner` is constructed without an explicit `planner` and
  without the AI Planner enabled in configuration
- **THEN** it uses the existing non-LLM Planner, unchanged from before this
  capability existed

#### Scenario: AI Planner enabled via configuration
- **WHEN** `TaskRunner` is constructed without an explicit `planner` and with
  the AI Planner enabled in configuration
- **THEN** it uses the AI Planner, configured with the selected provider and
  model

### Requirement: Screenshot access is a Planner-only, narrow exception to the Perception Boundary
The system SHALL allow the AI Planner, and only the AI Planner, to receive
the current step's raw screenshot bytes alongside the Scene for
vision-grounded decision-making, while every other perception consumer
SHALL continue to receive only the Scene.

#### Scenario: Planner receives both Scene and screenshot
- **WHEN** a screenshot for the current step is available
- **THEN** the AI Planner's decision call includes both the Scene JSON and
  the raw screenshot bytes for that step

#### Scenario: Screenshot unavailable does not block planning
- **WHEN** a screenshot for the current step cannot be obtained
- **THEN** the AI Planner still produces a decision using the Scene alone,
  and this is not treated as a task failure

#### Scenario: No other consumer receives raw screenshot bytes
- **WHEN** any component other than the AI Planner (for example, `api`,
  `tools`, `perception`, or `storage`) consumes perception output
- **THEN** it receives only the Scene, never raw screenshot bytes