## ADDED Requirements ### Requirement: LLM-driven Planner selects exactly one grounded action per turn The system SHALL provide a Planner implementation that, given a goal, the current Scene, and recent task history, uses native LLM tool/function calling to select exactly one action (or the completion signal defined below) per `plan()` invocation, grounding any coordinates in the current turn's Scene element bounds. #### Scenario: Planner selects a single action for the current turn - **WHEN** the AI Planner is invoked with a goal and the current Scene - **THEN** it returns at most one `PlannedStep`, whose action and arguments come from exactly one tool call chosen by the underlying LLM for that turn #### Scenario: Planner re-decides every turn from the current Scene - **WHEN** the AI Planner is invoked again after a prior step has executed - **THEN** its decision is grounded in the newly observed Scene for that turn, not in coordinates or assumptions carried over from a previous turn ### Requirement: Explicit finish_task completion and failure signal The system SHALL treat task completion and task failure as explicit, model-driven signals via a dedicated `finish_task(success, reason)` tool, rather than inferring either outcome from the model declining to call any tool. #### Scenario: Model signals successful completion - **WHEN** the model calls `finish_task` with `success=True` - **THEN** the Planner returns an empty step list and the task is marked completed #### Scenario: Model signals it cannot complete the goal - **WHEN** the model calls `finish_task` with `success=False` and a `reason` - **THEN** the task is marked failed with that reason, without attempting any further planning steps ### Requirement: Pluggable dual-provider tool-calling abstraction The system SHALL support at least two interchangeable LLM providers (Anthropic native tool use and OpenAI function calling) for the AI Planner's decision calls, selectable via configuration, with both providers constrained to return exactly one tool call per request. #### Scenario: Provider selected via configuration - **WHEN** the AI Planner is configured with a given provider identifier - **THEN** it constructs and uses the tool-calling client for that provider without requiring any change to `AIPlanner`'s own decision logic #### Scenario: Provider response resolves to a single decision - **WHEN** either supported provider returns a response to a tool-calling request - **THEN** the response is parsed into exactly one tool name and one arguments object, regardless of which provider produced it ### Requirement: AI Planner is disabled by default and additive to the existing Planner The system SHALL default to the existing non-LLM Planner unless the AI Planner is explicitly enabled via configuration, and SHALL NOT alter the existing Planner's behavior, dependencies, or any caller's construction of `TaskRunner` when left disabled. #### Scenario: AI Planner disabled (default) - **WHEN** `TaskRunner` is constructed without an explicit `planner` and without the AI Planner enabled in configuration - **THEN** it uses the existing non-LLM Planner, unchanged from before this capability existed #### Scenario: AI Planner enabled via configuration - **WHEN** `TaskRunner` is constructed without an explicit `planner` and with the AI Planner enabled in configuration - **THEN** it uses the AI Planner, configured with the selected provider and model ### Requirement: Screenshot access is a Planner-only, narrow exception to the Perception Boundary The system SHALL allow the AI Planner, and only the AI Planner, to receive the current step's raw screenshot bytes alongside the Scene for vision-grounded decision-making, while every other perception consumer SHALL continue to receive only the Scene. #### Scenario: Planner receives both Scene and screenshot - **WHEN** a screenshot for the current step is available - **THEN** the AI Planner's decision call includes both the Scene JSON and the raw screenshot bytes for that step #### Scenario: Screenshot unavailable does not block planning - **WHEN** a screenshot for the current step cannot be obtained - **THEN** the AI Planner still produces a decision using the Scene alone, and this is not treated as a task failure #### Scenario: No other consumer receives raw screenshot bytes - **WHEN** any component other than the AI Planner (for example, `api`, `tools`, `perception`, or `storage`) consumes perception output - **THEN** it receives only the Scene, never raw screenshot bytes