agentic-mobile-control/openspec/changes/ai-planner-runtime/specs/agent-runtime/spec.md at 5db68bfa0d0827161f58294ba3b0aec4197259a3

Files

T

q792602257andClaude Sonnet 5 61ff3b425d feat(agent-runtime): add LLM-driven AI Planner with dual-provider tool calling

Replaces the stub Planner's fixed describe_screen/[] behavior with a real
decision-maker: AIPlanner uses native tool/function calling (Anthropic or
OpenAI, pluggable via AI_PLANNER_PROVIDER) to select exactly one grounded
action per turn, with an explicit finish_task(success, reason) tool for
completion/failure instead of an ambiguous "no tool call" signal. Default
disabled (AI_PLANNER_ENABLED=false) and additive; TaskRunner falls back to
the existing stub Planner unchanged when disabled.

Amends CONSTITUTION.md's Perception Boundary with one narrow exception:
only the AI Planner may receive the current step's raw screenshot bytes
alongside Scene, for vision-grounded coordinate grounding. Also fixes a
latent gap in TaskRunner.run(): observe/plan exceptions are now caught per
iteration and turned into a failed task with a failure_reason, instead of
propagating uncaught.

openspec change: ai-planner-runtime.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

2026-07-12 13:48:50 +08:00

4.5 KiB

Raw Blame History

ADDED Requirements

Requirement: LLM-driven Planner selects exactly one grounded action per turn

The system SHALL provide a Planner implementation that, given a goal, the current Scene, and recent task history, uses native LLM tool/function calling to select exactly one action (or the completion signal defined below) per plan() invocation, grounding any coordinates in the current turn's Scene element bounds.

Scenario: Planner selects a single action for the current turn

WHEN the AI Planner is invoked with a goal and the current Scene
THEN it returns at most one PlannedStep, whose action and arguments come from exactly one tool call chosen by the underlying LLM for that turn

Scenario: Planner re-decides every turn from the current Scene

WHEN the AI Planner is invoked again after a prior step has executed
THEN its decision is grounded in the newly observed Scene for that turn, not in coordinates or assumptions carried over from a previous turn

Requirement: Explicit finish_task completion and failure signal

The system SHALL treat task completion and task failure as explicit, model-driven signals via a dedicated finish_task(success, reason) tool, rather than inferring either outcome from the model declining to call any tool.

Scenario: Model signals successful completion

WHEN the model calls finish_task with success=True
THEN the Planner returns an empty step list and the task is marked completed

Scenario: Model signals it cannot complete the goal

WHEN the model calls finish_task with success=False and a reason
THEN the task is marked failed with that reason, without attempting any further planning steps

Requirement: Pluggable dual-provider tool-calling abstraction

The system SHALL support at least two interchangeable LLM providers (Anthropic native tool use and OpenAI function calling) for the AI Planner's decision calls, selectable via configuration, with both providers constrained to return exactly one tool call per request.

Scenario: Provider selected via configuration

WHEN the AI Planner is configured with a given provider identifier
THEN it constructs and uses the tool-calling client for that provider without requiring any change to AIPlanner's own decision logic

Scenario: Provider response resolves to a single decision

WHEN either supported provider returns a response to a tool-calling request
THEN the response is parsed into exactly one tool name and one arguments object, regardless of which provider produced it

Requirement: AI Planner is disabled by default and additive to the existing Planner

The system SHALL default to the existing non-LLM Planner unless the AI Planner is explicitly enabled via configuration, and SHALL NOT alter the existing Planner's behavior, dependencies, or any caller's construction of TaskRunner when left disabled.

Scenario: AI Planner disabled (default)

WHEN TaskRunner is constructed without an explicit planner and without the AI Planner enabled in configuration
THEN it uses the existing non-LLM Planner, unchanged from before this capability existed

Scenario: AI Planner enabled via configuration

WHEN TaskRunner is constructed without an explicit planner and with the AI Planner enabled in configuration
THEN it uses the AI Planner, configured with the selected provider and model

Requirement: Screenshot access is a Planner-only, narrow exception to the Perception Boundary

The system SHALL allow the AI Planner, and only the AI Planner, to receive the current step's raw screenshot bytes alongside the Scene for vision-grounded decision-making, while every other perception consumer SHALL continue to receive only the Scene.

Scenario: Planner receives both Scene and screenshot

WHEN a screenshot for the current step is available
THEN the AI Planner's decision call includes both the Scene JSON and the raw screenshot bytes for that step

Scenario: Screenshot unavailable does not block planning

WHEN a screenshot for the current step cannot be obtained
THEN the AI Planner still produces a decision using the Scene alone, and this is not treated as a task failure

Scenario: No other consumer receives raw screenshot bytes

WHEN any component other than the AI Planner (for example, api, tools, perception, or storage) consumes perception output
THEN it receives only the Scene, never raw screenshot bytes

4.5 KiB Raw Blame History