feat(agent-runtime): add LLM-driven AI Planner with dual-provider tool calling

Replaces the stub Planner's fixed describe_screen/[] behavior with a real
decision-maker: AIPlanner uses native tool/function calling (Anthropic or
OpenAI, pluggable via AI_PLANNER_PROVIDER) to select exactly one grounded
action per turn, with an explicit finish_task(success, reason) tool for
completion/failure instead of an ambiguous "no tool call" signal. Default
disabled (AI_PLANNER_ENABLED=false) and additive; TaskRunner falls back to
the existing stub Planner unchanged when disabled.

Amends CONSTITUTION.md's Perception Boundary with one narrow exception:
only the AI Planner may receive the current step's raw screenshot bytes
alongside Scene, for vision-grounded coordinate grounding. Also fixes a
latent gap in TaskRunner.run(): observe/plan exceptions are now caught per
iteration and turned into a failed task with a failure_reason, instead of
propagating uncaught.

openspec change: ai-planner-runtime.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
2026-07-12 13:48:50 +08:00
co-authored by Claude Sonnet 5
parent b94abde92a
commit 61ff3b425d
19 changed files with 1977 additions and 19 deletions
@@ -0,0 +1,90 @@
## ADDED Requirements
### Requirement: LLM-driven Planner selects exactly one grounded action per turn
The system SHALL provide a Planner implementation that, given a goal, the
current Scene, and recent task history, uses native LLM tool/function
calling to select exactly one action (or the completion signal defined
below) per `plan()` invocation, grounding any coordinates in the current
turn's Scene element bounds.
#### Scenario: Planner selects a single action for the current turn
- **WHEN** the AI Planner is invoked with a goal and the current Scene
- **THEN** it returns at most one `PlannedStep`, whose action and arguments
come from exactly one tool call chosen by the underlying LLM for that turn
#### Scenario: Planner re-decides every turn from the current Scene
- **WHEN** the AI Planner is invoked again after a prior step has executed
- **THEN** its decision is grounded in the newly observed Scene for that
turn, not in coordinates or assumptions carried over from a previous turn
### Requirement: Explicit finish_task completion and failure signal
The system SHALL treat task completion and task failure as explicit,
model-driven signals via a dedicated `finish_task(success, reason)` tool,
rather than inferring either outcome from the model declining to call any
tool.
#### Scenario: Model signals successful completion
- **WHEN** the model calls `finish_task` with `success=True`
- **THEN** the Planner returns an empty step list and the task is marked
completed
#### Scenario: Model signals it cannot complete the goal
- **WHEN** the model calls `finish_task` with `success=False` and a `reason`
- **THEN** the task is marked failed with that reason, without attempting
any further planning steps
### Requirement: Pluggable dual-provider tool-calling abstraction
The system SHALL support at least two interchangeable LLM providers
(Anthropic native tool use and OpenAI function calling) for the AI Planner's
decision calls, selectable via configuration, with both providers
constrained to return exactly one tool call per request.
#### Scenario: Provider selected via configuration
- **WHEN** the AI Planner is configured with a given provider identifier
- **THEN** it constructs and uses the tool-calling client for that provider
without requiring any change to `AIPlanner`'s own decision logic
#### Scenario: Provider response resolves to a single decision
- **WHEN** either supported provider returns a response to a tool-calling
request
- **THEN** the response is parsed into exactly one tool name and one
arguments object, regardless of which provider produced it
### Requirement: AI Planner is disabled by default and additive to the existing Planner
The system SHALL default to the existing non-LLM Planner unless the AI
Planner is explicitly enabled via configuration, and SHALL NOT alter the
existing Planner's behavior, dependencies, or any caller's construction of
`TaskRunner` when left disabled.
#### Scenario: AI Planner disabled (default)
- **WHEN** `TaskRunner` is constructed without an explicit `planner` and
without the AI Planner enabled in configuration
- **THEN** it uses the existing non-LLM Planner, unchanged from before this
capability existed
#### Scenario: AI Planner enabled via configuration
- **WHEN** `TaskRunner` is constructed without an explicit `planner` and with
the AI Planner enabled in configuration
- **THEN** it uses the AI Planner, configured with the selected provider and
model
### Requirement: Screenshot access is a Planner-only, narrow exception to the Perception Boundary
The system SHALL allow the AI Planner, and only the AI Planner, to receive
the current step's raw screenshot bytes alongside the Scene for
vision-grounded decision-making, while every other perception consumer
SHALL continue to receive only the Scene.
#### Scenario: Planner receives both Scene and screenshot
- **WHEN** a screenshot for the current step is available
- **THEN** the AI Planner's decision call includes both the Scene JSON and
the raw screenshot bytes for that step
#### Scenario: Screenshot unavailable does not block planning
- **WHEN** a screenshot for the current step cannot be obtained
- **THEN** the AI Planner still produces a decision using the Scene alone,
and this is not treated as a task failure
#### Scenario: No other consumer receives raw screenshot bytes
- **WHEN** any component other than the AI Planner (for example, `api`,
`tools`, `perception`, or `storage`) consumes perception output
- **THEN** it receives only the Scene, never raw screenshot bytes