feat(planner): persist reusable action semantics
Tests / Test passed: 879

This commit is contained in:
2026-07-15 18:14:28 +08:00
parent 361dada276
commit d69be48f96
41 changed files with 733 additions and 116 deletions
@@ -23,6 +23,10 @@ The system SHALL provide a Planner implementation that, given a goal, the curren
- **WHEN** the LLM response contains only a tool call with no preceding text block
- **THEN** `PlannedStep.rationale` is `None` and the step is returned normally
#### Scenario: Device action includes reusable purpose and expected outcome
- **WHEN** the AI Planner selects a device action (`tap`, `swipe`, `input_text`, `launch_app`, or `terminate_app`)
- **THEN** its tool call requires non-empty `purpose` and `expected_outcome` values, and the returned `PlannedStep` carries both separately from the executable action arguments
### Requirement: Pluggable dual-provider tool-calling abstraction
The system SHALL support at least two interchangeable LLM providers (Anthropic native tool use and OpenAI function calling) for the AI Planner's decision calls, selectable via configuration, with both providers constrained to return exactly one tool call per request. The Anthropic client SHALL additionally support optional extended thinking via a configurable `thinking_budget_tokens` value. The OpenAI client SHALL capture `reasoning_content` from responses when present. Independently of provider selection, the system SHALL support at least two transports for making that decision call — direct-to-provider and cloud-proxy — selectable via configuration without requiring any change to `AIPlanner`'s own decision logic.
@@ -50,6 +54,6 @@ The system SHALL support at least two interchangeable LLM providers (Anthropic n
- **WHEN** the Host Agent is configured with the cloud-proxy transport
- **THEN** its tool-calling client sends the decision request to the Cloud Control Plane's planner-decision endpoint instead of constructing a local Anthropic or OpenAI SDK client
#### Scenario: Cloud-proxy transport returns None for thinking and text_output
- **WHEN** the Host Agent uses cloud-proxy transport
- **THEN** `ToolCallDecision.thinking` and `ToolCallDecision.text_output` are `None` because the proxy surface does not expose them
#### Scenario: Cloud-proxy transport returns planner metadata
- **WHEN** the Host Agent uses cloud-proxy transport and the Cloud Planner returns a decision
- **THEN** the proxy returns its rationale, thinking, purpose, and expected outcome alongside the tool name and executable arguments
@@ -1,7 +1,7 @@
## MODIFIED Requirements
### Requirement: Cloud Console displays a task's full LLM interaction history
Cloud Console SHALL provide a view, for a given task, listing each persisted planner decision in step order, including its full prompt, resulting decision, and — when available — the AI's rationale (pre-tool text reflection) and thinking (extended thinking block), sourced from the Cloud Control Plane's persisted planner-decision log.
Cloud Console SHALL provide a view, for a given task, listing each persisted planner decision in step order, including its full prompt, resulting decision, and — when available — the AI's rationale (pre-tool text reflection), thinking (extended thinking block), action purpose, and expected outcome, sourced from the Cloud Control Plane's persisted planner-decision log.
#### Scenario: Task has persisted planner decisions with rationale
- **WHEN** an operator opens the LLM interaction history view for a task that has one or more persisted planner decisions with non-null rationale
@@ -20,7 +20,7 @@ Cloud Console SHALL provide a view, for a given task, listing each persisted pla
- **THEN** Cloud Console indicates that no LLM interaction history is available because the Host does not report it, rather than showing an empty history with no explanation
### Requirement: Cloud Control Plane persists rationale and thinking in the planner decision log
The Cloud Control Plane's planner-decision log SHALL store the AI's rationale and thinking fields alongside the existing prompt and tool-call fields for each persisted decision. Both fields SHALL be nullable; absence of either field SHALL NOT prevent a decision record from being stored or queried.
The Cloud Control Plane's planner-decision log SHALL store the AI's rationale, thinking, action purpose, and expected outcome fields alongside the existing prompt and tool-call fields for each persisted decision. These fields SHALL be nullable for backward compatibility; absence of a legacy or non-AI value SHALL NOT prevent a decision record from being stored or queried.
#### Scenario: Decision record includes rationale
- **WHEN** the Host Agent reports a planner decision with a non-null rationale
@@ -34,6 +34,10 @@ The Cloud Control Plane's planner-decision log SHALL store the AI's rationale an
- **WHEN** the Host Agent reports a planner decision with null rationale and null thinking (e.g., cloud-proxy transport where these are not surfaced)
- **THEN** the persisted row stores NULL for both columns without error
#### Scenario: Decision record includes reusable action metadata
- **WHEN** the Host reports a device-action planner decision with a purpose and expected outcome
- **THEN** the persisted row stores both values separately from the executable tool arguments and the task API returns them to authorized readers
#### Scenario: Existing decision records without rationale or thinking remain readable
- **WHEN** the system queries a `planner_decision_log` row created before this migration
- **THEN** both `rationale` and `thinking` read as NULL, and the row is returned normally
@@ -34,6 +34,17 @@ The system SHALL extract and preserve the AI model's thinking block (when extend
- **WHEN** the LLM response contains only a tool call block (no thinking, no text)
- **THEN** `ToolCallDecision.thinking` and `ToolCallDecision.text_output` are both `None`, and the decision is returned normally
### Requirement: Device actions return required reusable metadata
The system SHALL require each device-action tool call to include concise, non-empty `purpose` and `expected_outcome` strings in its structured arguments. The Runtime SHALL preserve these values as planner metadata while excluding them from the arguments supplied to the physical device tool.
#### Scenario: Tool schema requires purpose and expected outcome
- **WHEN** the Planner sends an action tool schema to an LLM provider
- **THEN** each device-action schema requires `purpose` and `expected_outcome` in addition to its physical-action arguments
#### Scenario: Action metadata is not passed to the device tool
- **WHEN** a planned action is executed
- **THEN** the device tool receives only its physical-action arguments while the purpose and expected outcome remain available on the planned step and execution record
### Requirement: Extended thinking is opt-in via configuration
The system SHALL support enabling Anthropic extended thinking for the AI planner via a `thinking_budget_tokens` configuration value. When not configured, the planner SHALL operate identically to its pre-existing behavior.
@@ -50,12 +61,16 @@ The system SHALL support enabling Anthropic extended thinking for the AI planner
- **THEN** the OpenAI client does not apply the Anthropic thinking parameter; reasoning content is captured only if the model returns it naturally
### Requirement: Execution history uses compact rationale-based representation
The system SHALL construct the AI planner's history prompt from a compact per-step record containing the page context, rationale, action, and success flag — not the full scene JSON. This compact history SHALL be the sole format used when constructing the `history_summary` passed to `planner_user_prompt`.
The system SHALL construct the AI planner's history prompt from a compact per-step record containing the page context, rationale, action, executable action arguments, purpose, expected outcome, and success flag — not the full scene JSON. This compact history SHALL be the sole format used when constructing the `history_summary` passed to `planner_user_prompt`.
#### Scenario: History prompt uses compact format
- **WHEN** `_history_summary()` is called with a `WorldState` that has one or more history entries
- **THEN** each entry in the returned list contains `page`, `rationale`, `action`, and `success` fields only, without any scene element data
- **THEN** each entry in the returned list contains `page`, `rationale`, `action`, `arguments`, `purpose`, `expected_outcome`, and `success` fields only, without any scene element data
#### Scenario: History prompt handles None rationale
- **WHEN** a `WorldEvent` in history has `rationale=None`
- **THEN** the compact history entry for that step includes `"rationale": null` without omitting the field or raising an error
#### Scenario: History prompt preserves executed action arguments
- **WHEN** a prior action tapped a coordinate or otherwise supplied tool arguments
- **THEN** the compact history entry includes those exact executable arguments alongside the action name
@@ -1,7 +1,7 @@
## MODIFIED Requirements
### Requirement: Bounded history of recent scene/action pairs
The system SHALL maintain `WorldState.history` as a fixed-size, bounded collection of the most recent per-step records, automatically evicting the oldest entry when a new entry is added past the configured bound. Each history record SHALL store the action name, success flag, page context (from `WorldState.current_page` at the time of recording), and optional rationale and thinking fields sourced from the executed `PlannedStep`. The `scene_summary` field SHALL be retained as an optional field for backward compatibility but SHALL NOT be required for new entries.
The system SHALL maintain `WorldState.history` as a fixed-size, bounded collection of the most recent per-step records, automatically evicting the oldest entry when a new entry is added past the configured bound. Each history record SHALL store the action name, executable action arguments, success flag, page context (from `WorldState.current_page` at the time of recording), and optional rationale, thinking, purpose, and expected-outcome fields sourced from the executed `PlannedStep`. The `scene_summary` field SHALL be retained as an optional field for backward compatibility but SHALL NOT be required for new entries.
#### Scenario: WorldState survives across steps within a task
- **WHEN** a task executes multiple steps in sequence
@@ -23,6 +23,14 @@ The system SHALL maintain `WorldState.history` as a fixed-size, bounded collecti
- **WHEN** the executed `PlannedStep` carries a non-None `thinking`
- **THEN** the resulting `WorldEvent` stores that thinking string
#### Scenario: History record includes reusable action metadata
- **WHEN** an executed `PlannedStep` carries a purpose and expected outcome
- **THEN** the resulting `WorldEvent` stores both values for the next planning turn and later reuse
#### Scenario: History record includes executed action arguments
- **WHEN** a `PlannedStep` executes with action arguments such as a tap's `x` and `y` coordinates
- **THEN** the resulting `WorldEvent` stores those executable arguments with the action name
#### Scenario: History record captures current page at time of recording
- **WHEN** a step is appended to history and `WorldState.current_page` is non-None at that moment
- **THEN** `WorldEvent.page` is set to that page value