q792602257andClaude Opus 4.6
c01dd4c6b2
Change is complete (22/22 tasks) and its delta spec has been synced into a new main spec openspec/specs/agent-runtime/spec.md. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
98 lines
4.7 KiB
Markdown
98 lines
4.7 KiB
Markdown
# agent-runtime Specification
|
|
|
|
## Purpose
|
|
Define the behavior of the AI Planner runtime, including LLM-driven action
|
|
selection, completion signaling, provider abstraction, default-disabling, and
|
|
the narrow screenshot exception to the Perception Boundary.
|
|
|
|
## Requirements
|
|
|
|
### Requirement: LLM-driven Planner selects exactly one grounded action per turn
|
|
The system SHALL provide a Planner implementation that, given a goal, the
|
|
current Scene, and recent task history, uses native LLM tool/function
|
|
calling to select exactly one action (or the completion signal defined
|
|
below) per `plan()` invocation, grounding any coordinates in the current
|
|
turn's Scene element bounds.
|
|
|
|
#### Scenario: Planner selects a single action for the current turn
|
|
- **WHEN** the AI Planner is invoked with a goal and the current Scene
|
|
- **THEN** it returns at most one `PlannedStep`, whose action and arguments
|
|
come from exactly one tool call chosen by the underlying LLM for that turn
|
|
|
|
#### Scenario: Planner re-decides every turn from the current Scene
|
|
- **WHEN** the AI Planner is invoked again after a prior step has executed
|
|
- **THEN** its decision is grounded in the newly observed Scene for that
|
|
turn, not in coordinates or assumptions carried over from a previous turn
|
|
|
|
### Requirement: Explicit finish_task completion and failure signal
|
|
The system SHALL treat task completion and task failure as explicit,
|
|
model-driven signals via a dedicated `finish_task(success, reason)` tool,
|
|
rather than inferring either outcome from the model declining to call any
|
|
tool.
|
|
|
|
#### Scenario: Model signals successful completion
|
|
- **WHEN** the model calls `finish_task` with `success=True`
|
|
- **THEN** the Planner returns an empty step list and the task is marked
|
|
completed
|
|
|
|
#### Scenario: Model signals it cannot complete the goal
|
|
- **WHEN** the model calls `finish_task` with `success=False` and a `reason`
|
|
- **THEN** the task is marked failed with that reason, without attempting
|
|
any further planning steps
|
|
|
|
### Requirement: Pluggable dual-provider tool-calling abstraction
|
|
The system SHALL support at least two interchangeable LLM providers
|
|
(Anthropic native tool use and OpenAI function calling) for the AI Planner's
|
|
decision calls, selectable via configuration, with both providers
|
|
constrained to return exactly one tool call per request.
|
|
|
|
#### Scenario: Provider selected via configuration
|
|
- **WHEN** the AI Planner is configured with a given provider identifier
|
|
- **THEN** it constructs and uses the tool-calling client for that provider
|
|
without requiring any change to `AIPlanner`'s own decision logic
|
|
|
|
#### Scenario: Provider response resolves to a single decision
|
|
- **WHEN** either supported provider returns a response to a tool-calling
|
|
request
|
|
- **THEN** the response is parsed into exactly one tool name and one
|
|
arguments object, regardless of which provider produced it
|
|
|
|
### Requirement: AI Planner is disabled by default and additive to the existing Planner
|
|
The system SHALL default to the existing non-LLM Planner unless the AI
|
|
Planner is explicitly enabled via configuration, and SHALL NOT alter the
|
|
existing Planner's behavior, dependencies, or any caller's construction of
|
|
`TaskRunner` when left disabled.
|
|
|
|
#### Scenario: AI Planner disabled (default)
|
|
- **WHEN** `TaskRunner` is constructed without an explicit `planner` and
|
|
without the AI Planner enabled in configuration
|
|
- **THEN** it uses the existing non-LLM Planner, unchanged from before this
|
|
capability existed
|
|
|
|
#### Scenario: AI Planner enabled via configuration
|
|
- **WHEN** `TaskRunner` is constructed without an explicit `planner` and with
|
|
the AI Planner enabled in configuration
|
|
- **THEN** it uses the AI Planner, configured with the selected provider and
|
|
model
|
|
|
|
### Requirement: Screenshot access is a Planner-only, narrow exception to the Perception Boundary
|
|
The system SHALL allow the AI Planner, and only the AI Planner, to receive
|
|
the current step's raw screenshot bytes alongside the Scene for
|
|
vision-grounded decision-making, while every other perception consumer
|
|
SHALL continue to receive only the Scene.
|
|
|
|
#### Scenario: Planner receives both Scene and screenshot
|
|
- **WHEN** a screenshot for the current step is available
|
|
- **THEN** the AI Planner's decision call includes both the Scene JSON and
|
|
the raw screenshot bytes for that step
|
|
|
|
#### Scenario: Screenshot unavailable does not block planning
|
|
- **WHEN** a screenshot for the current step cannot be obtained
|
|
- **THEN** the AI Planner still produces a decision using the Scene alone,
|
|
and this is not treated as a task failure
|
|
|
|
#### Scenario: No other consumer receives raw screenshot bytes
|
|
- **WHEN** any component other than the AI Planner (for example, `api`,
|
|
`tools`, `perception`, or `storage`) consumes perception output
|
|
- **THEN** it receives only the Scene, never raw screenshot bytes
|