q792602257andClaude Sonnet 5
61ff3b425d
Replaces the stub Planner's fixed describe_screen/[] behavior with a real decision-maker: AIPlanner uses native tool/function calling (Anthropic or OpenAI, pluggable via AI_PLANNER_PROVIDER) to select exactly one grounded action per turn, with an explicit finish_task(success, reason) tool for completion/failure instead of an ambiguous "no tool call" signal. Default disabled (AI_PLANNER_ENABLED=false) and additive; TaskRunner falls back to the existing stub Planner unchanged when disabled. Amends CONSTITUTION.md's Perception Boundary with one narrow exception: only the AI Planner may receive the current step's raw screenshot bytes alongside Scene, for vision-grounded coordinate grounding. Also fixes a latent gap in TaskRunner.run(): observe/plan exceptions are now caught per iteration and turned into a failed task with a failure_reason, instead of propagating uncaught. openspec change: ai-planner-runtime. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
91 lines
4.5 KiB
Markdown
91 lines
4.5 KiB
Markdown
## ADDED Requirements
|
|
|
|
### Requirement: LLM-driven Planner selects exactly one grounded action per turn
|
|
The system SHALL provide a Planner implementation that, given a goal, the
|
|
current Scene, and recent task history, uses native LLM tool/function
|
|
calling to select exactly one action (or the completion signal defined
|
|
below) per `plan()` invocation, grounding any coordinates in the current
|
|
turn's Scene element bounds.
|
|
|
|
#### Scenario: Planner selects a single action for the current turn
|
|
- **WHEN** the AI Planner is invoked with a goal and the current Scene
|
|
- **THEN** it returns at most one `PlannedStep`, whose action and arguments
|
|
come from exactly one tool call chosen by the underlying LLM for that turn
|
|
|
|
#### Scenario: Planner re-decides every turn from the current Scene
|
|
- **WHEN** the AI Planner is invoked again after a prior step has executed
|
|
- **THEN** its decision is grounded in the newly observed Scene for that
|
|
turn, not in coordinates or assumptions carried over from a previous turn
|
|
|
|
### Requirement: Explicit finish_task completion and failure signal
|
|
The system SHALL treat task completion and task failure as explicit,
|
|
model-driven signals via a dedicated `finish_task(success, reason)` tool,
|
|
rather than inferring either outcome from the model declining to call any
|
|
tool.
|
|
|
|
#### Scenario: Model signals successful completion
|
|
- **WHEN** the model calls `finish_task` with `success=True`
|
|
- **THEN** the Planner returns an empty step list and the task is marked
|
|
completed
|
|
|
|
#### Scenario: Model signals it cannot complete the goal
|
|
- **WHEN** the model calls `finish_task` with `success=False` and a `reason`
|
|
- **THEN** the task is marked failed with that reason, without attempting
|
|
any further planning steps
|
|
|
|
### Requirement: Pluggable dual-provider tool-calling abstraction
|
|
The system SHALL support at least two interchangeable LLM providers
|
|
(Anthropic native tool use and OpenAI function calling) for the AI Planner's
|
|
decision calls, selectable via configuration, with both providers
|
|
constrained to return exactly one tool call per request.
|
|
|
|
#### Scenario: Provider selected via configuration
|
|
- **WHEN** the AI Planner is configured with a given provider identifier
|
|
- **THEN** it constructs and uses the tool-calling client for that provider
|
|
without requiring any change to `AIPlanner`'s own decision logic
|
|
|
|
#### Scenario: Provider response resolves to a single decision
|
|
- **WHEN** either supported provider returns a response to a tool-calling
|
|
request
|
|
- **THEN** the response is parsed into exactly one tool name and one
|
|
arguments object, regardless of which provider produced it
|
|
|
|
### Requirement: AI Planner is disabled by default and additive to the existing Planner
|
|
The system SHALL default to the existing non-LLM Planner unless the AI
|
|
Planner is explicitly enabled via configuration, and SHALL NOT alter the
|
|
existing Planner's behavior, dependencies, or any caller's construction of
|
|
`TaskRunner` when left disabled.
|
|
|
|
#### Scenario: AI Planner disabled (default)
|
|
- **WHEN** `TaskRunner` is constructed without an explicit `planner` and
|
|
without the AI Planner enabled in configuration
|
|
- **THEN** it uses the existing non-LLM Planner, unchanged from before this
|
|
capability existed
|
|
|
|
#### Scenario: AI Planner enabled via configuration
|
|
- **WHEN** `TaskRunner` is constructed without an explicit `planner` and with
|
|
the AI Planner enabled in configuration
|
|
- **THEN** it uses the AI Planner, configured with the selected provider and
|
|
model
|
|
|
|
### Requirement: Screenshot access is a Planner-only, narrow exception to the Perception Boundary
|
|
The system SHALL allow the AI Planner, and only the AI Planner, to receive
|
|
the current step's raw screenshot bytes alongside the Scene for
|
|
vision-grounded decision-making, while every other perception consumer
|
|
SHALL continue to receive only the Scene.
|
|
|
|
#### Scenario: Planner receives both Scene and screenshot
|
|
- **WHEN** a screenshot for the current step is available
|
|
- **THEN** the AI Planner's decision call includes both the Scene JSON and
|
|
the raw screenshot bytes for that step
|
|
|
|
#### Scenario: Screenshot unavailable does not block planning
|
|
- **WHEN** a screenshot for the current step cannot be obtained
|
|
- **THEN** the AI Planner still produces a decision using the Scene alone,
|
|
and this is not treated as a task failure
|
|
|
|
#### Scenario: No other consumer receives raw screenshot bytes
|
|
- **WHEN** any component other than the AI Planner (for example, `api`,
|
|
`tools`, `perception`, or `storage`) consumes perception output
|
|
- **THEN** it receives only the Scene, never raw screenshot bytes
|