diff --git a/openspec/changes/ai-planner-runtime/.openspec.yaml b/openspec/changes/archive/2026-07-14-ai-planner-runtime/.openspec.yaml similarity index 100% rename from openspec/changes/ai-planner-runtime/.openspec.yaml rename to openspec/changes/archive/2026-07-14-ai-planner-runtime/.openspec.yaml diff --git a/openspec/changes/ai-planner-runtime/design.md b/openspec/changes/archive/2026-07-14-ai-planner-runtime/design.md similarity index 100% rename from openspec/changes/ai-planner-runtime/design.md rename to openspec/changes/archive/2026-07-14-ai-planner-runtime/design.md diff --git a/openspec/changes/ai-planner-runtime/proposal.md b/openspec/changes/archive/2026-07-14-ai-planner-runtime/proposal.md similarity index 100% rename from openspec/changes/ai-planner-runtime/proposal.md rename to openspec/changes/archive/2026-07-14-ai-planner-runtime/proposal.md diff --git a/openspec/changes/ai-planner-runtime/specs/agent-runtime/spec.md b/openspec/changes/archive/2026-07-14-ai-planner-runtime/specs/agent-runtime/spec.md similarity index 100% rename from openspec/changes/ai-planner-runtime/specs/agent-runtime/spec.md rename to openspec/changes/archive/2026-07-14-ai-planner-runtime/specs/agent-runtime/spec.md diff --git a/openspec/changes/ai-planner-runtime/tasks.md b/openspec/changes/archive/2026-07-14-ai-planner-runtime/tasks.md similarity index 100% rename from openspec/changes/ai-planner-runtime/tasks.md rename to openspec/changes/archive/2026-07-14-ai-planner-runtime/tasks.md diff --git a/openspec/specs/agent-runtime/spec.md b/openspec/specs/agent-runtime/spec.md new file mode 100644 index 0000000..fa9003c --- /dev/null +++ b/openspec/specs/agent-runtime/spec.md @@ -0,0 +1,97 @@ +# agent-runtime Specification + +## Purpose +Define the behavior of the AI Planner runtime, including LLM-driven action +selection, completion signaling, provider abstraction, default-disabling, and +the narrow screenshot exception to the Perception Boundary. + +## Requirements + +### Requirement: LLM-driven Planner selects exactly one grounded action per turn +The system SHALL provide a Planner implementation that, given a goal, the +current Scene, and recent task history, uses native LLM tool/function +calling to select exactly one action (or the completion signal defined +below) per `plan()` invocation, grounding any coordinates in the current +turn's Scene element bounds. + +#### Scenario: Planner selects a single action for the current turn +- **WHEN** the AI Planner is invoked with a goal and the current Scene +- **THEN** it returns at most one `PlannedStep`, whose action and arguments + come from exactly one tool call chosen by the underlying LLM for that turn + +#### Scenario: Planner re-decides every turn from the current Scene +- **WHEN** the AI Planner is invoked again after a prior step has executed +- **THEN** its decision is grounded in the newly observed Scene for that + turn, not in coordinates or assumptions carried over from a previous turn + +### Requirement: Explicit finish_task completion and failure signal +The system SHALL treat task completion and task failure as explicit, +model-driven signals via a dedicated `finish_task(success, reason)` tool, +rather than inferring either outcome from the model declining to call any +tool. + +#### Scenario: Model signals successful completion +- **WHEN** the model calls `finish_task` with `success=True` +- **THEN** the Planner returns an empty step list and the task is marked + completed + +#### Scenario: Model signals it cannot complete the goal +- **WHEN** the model calls `finish_task` with `success=False` and a `reason` +- **THEN** the task is marked failed with that reason, without attempting + any further planning steps + +### Requirement: Pluggable dual-provider tool-calling abstraction +The system SHALL support at least two interchangeable LLM providers +(Anthropic native tool use and OpenAI function calling) for the AI Planner's +decision calls, selectable via configuration, with both providers +constrained to return exactly one tool call per request. + +#### Scenario: Provider selected via configuration +- **WHEN** the AI Planner is configured with a given provider identifier +- **THEN** it constructs and uses the tool-calling client for that provider + without requiring any change to `AIPlanner`'s own decision logic + +#### Scenario: Provider response resolves to a single decision +- **WHEN** either supported provider returns a response to a tool-calling + request +- **THEN** the response is parsed into exactly one tool name and one + arguments object, regardless of which provider produced it + +### Requirement: AI Planner is disabled by default and additive to the existing Planner +The system SHALL default to the existing non-LLM Planner unless the AI +Planner is explicitly enabled via configuration, and SHALL NOT alter the +existing Planner's behavior, dependencies, or any caller's construction of +`TaskRunner` when left disabled. + +#### Scenario: AI Planner disabled (default) +- **WHEN** `TaskRunner` is constructed without an explicit `planner` and + without the AI Planner enabled in configuration +- **THEN** it uses the existing non-LLM Planner, unchanged from before this + capability existed + +#### Scenario: AI Planner enabled via configuration +- **WHEN** `TaskRunner` is constructed without an explicit `planner` and with + the AI Planner enabled in configuration +- **THEN** it uses the AI Planner, configured with the selected provider and + model + +### Requirement: Screenshot access is a Planner-only, narrow exception to the Perception Boundary +The system SHALL allow the AI Planner, and only the AI Planner, to receive +the current step's raw screenshot bytes alongside the Scene for +vision-grounded decision-making, while every other perception consumer +SHALL continue to receive only the Scene. + +#### Scenario: Planner receives both Scene and screenshot +- **WHEN** a screenshot for the current step is available +- **THEN** the AI Planner's decision call includes both the Scene JSON and + the raw screenshot bytes for that step + +#### Scenario: Screenshot unavailable does not block planning +- **WHEN** a screenshot for the current step cannot be obtained +- **THEN** the AI Planner still produces a decision using the Scene alone, + and this is not treated as a task failure + +#### Scenario: No other consumer receives raw screenshot bytes +- **WHEN** any component other than the AI Planner (for example, `api`, + `tools`, `perception`, or `storage`) consumes perception output +- **THEN** it receives only the Scene, never raw screenshot bytes