feat(runtime): add planner reflection history with rationale and thinking
Tests / Test failed: 2, passed: 849
Tests / Test failed: 2, passed: 849
- ToolCallDecision captures thinking blocks and pre-tool text output
- AnthropicToolCallingClient supports optional extended thinking (budget_tokens + beta header)
- PlannedStep carries rationale and thinking from each LLM decision
- WorldEvent replaces scene_summary with rationale/thinking/page fields (backward-compatible)
- AI planner system prompt instructs reflection before each tool call
- _history_summary() emits compact {page, rationale, action, success} dicts
- Cloud DB migration 0011 adds nullable rationale/thinking columns to planner_decision_log
- OpenAI client extracts reasoning_content into thinking field
This commit is contained in:
@@ -0,0 +1,55 @@
|
||||
## MODIFIED Requirements
|
||||
|
||||
### Requirement: LLM-driven Planner selects exactly one grounded action per turn
|
||||
The system SHALL provide a Planner implementation that, given a goal, the current Scene, recent task history in compact rationale-based format, and optional prior-step context, uses native LLM tool/function calling to select exactly one action (or the completion signal defined below) per `plan()` invocation, grounding any coordinates in the current turn's Scene element bounds. The Planner SHALL instruct the LLM to output a short text block (rationale) before the tool call, reflecting on the previous step's outcome and stating the current step's intent.
|
||||
|
||||
#### Scenario: Planner selects a single action for the current turn
|
||||
- **WHEN** the AI Planner is invoked with a goal and the current Scene
|
||||
- **THEN** it returns at most one `PlannedStep`, whose action and arguments come from exactly one tool call chosen by the underlying LLM for that turn
|
||||
|
||||
#### Scenario: Planner re-decides every turn from the current Scene
|
||||
- **WHEN** the AI Planner is invoked again after a prior step has executed
|
||||
- **THEN** its decision is grounded in the newly observed Scene for that turn, not in coordinates or assumptions carried over from a previous turn
|
||||
|
||||
#### Scenario: PlannedStep carries rationale when the LLM outputs a text block
|
||||
- **WHEN** the LLM response includes a text block before the tool call
|
||||
- **THEN** the returned `PlannedStep` has a non-None `rationale` containing that text
|
||||
|
||||
#### Scenario: PlannedStep carries thinking when extended thinking is enabled and the LLM outputs a thinking block
|
||||
- **WHEN** extended thinking is enabled and the LLM response includes a thinking block
|
||||
- **THEN** the returned `PlannedStep` has a non-None `thinking` containing the thinking block text
|
||||
|
||||
#### Scenario: PlannedStep carries None rationale when no text block is present
|
||||
- **WHEN** the LLM response contains only a tool call with no preceding text block
|
||||
- **THEN** `PlannedStep.rationale` is `None` and the step is returned normally
|
||||
|
||||
### Requirement: Pluggable dual-provider tool-calling abstraction
|
||||
The system SHALL support at least two interchangeable LLM providers (Anthropic native tool use and OpenAI function calling) for the AI Planner's decision calls, selectable via configuration, with both providers constrained to return exactly one tool call per request. The Anthropic client SHALL additionally support optional extended thinking via a configurable `thinking_budget_tokens` value. The OpenAI client SHALL capture `reasoning_content` from responses when present. Independently of provider selection, the system SHALL support at least two transports for making that decision call — direct-to-provider and cloud-proxy — selectable via configuration without requiring any change to `AIPlanner`'s own decision logic.
|
||||
|
||||
#### Scenario: Provider selected via configuration
|
||||
- **WHEN** the AI Planner is configured with a given provider identifier
|
||||
- **THEN** it constructs and uses the tool-calling client for that provider without requiring any change to `AIPlanner`'s own decision logic
|
||||
|
||||
#### Scenario: Provider response resolves to a single decision
|
||||
- **WHEN** either supported provider returns a response to a tool-calling request
|
||||
- **THEN** the response is parsed into exactly one tool name and one arguments object, regardless of which provider produced it
|
||||
|
||||
#### Scenario: Transport selected via configuration
|
||||
- **WHEN** the Host Agent is configured with a given transport (direct or cloud-proxy)
|
||||
- **THEN** `AIPlanner` is constructed with the tool-calling client for that transport, and its own decision logic is unchanged regardless of which transport is in effect
|
||||
|
||||
#### Scenario: Cloud-proxy transport is the default
|
||||
- **WHEN** no transport is explicitly configured
|
||||
- **THEN** the AI Planner uses the cloud-proxy transport and the Cloud Control Plane's planner-decision endpoint
|
||||
|
||||
#### Scenario: Direct transport remains available by explicit configuration
|
||||
- **WHEN** the Host Agent is configured with the direct transport
|
||||
- **THEN** the AI Planner uses the direct-to-provider transport with locally configured credentials
|
||||
|
||||
#### Scenario: Cloud-proxy transport resolves a decision without a local provider client
|
||||
- **WHEN** the Host Agent is configured with the cloud-proxy transport
|
||||
- **THEN** its tool-calling client sends the decision request to the Cloud Control Plane's planner-decision endpoint instead of constructing a local Anthropic or OpenAI SDK client
|
||||
|
||||
#### Scenario: Cloud-proxy transport returns None for thinking and text_output
|
||||
- **WHEN** the Host Agent uses cloud-proxy transport
|
||||
- **THEN** `ToolCallDecision.thinking` and `ToolCallDecision.text_output` are `None` because the proxy surface does not expose them
|
||||
+39
@@ -0,0 +1,39 @@
|
||||
## MODIFIED Requirements
|
||||
|
||||
### Requirement: Cloud Console displays a task's full LLM interaction history
|
||||
Cloud Console SHALL provide a view, for a given task, listing each persisted planner decision in step order, including its full prompt, resulting decision, and — when available — the AI's rationale (pre-tool text reflection) and thinking (extended thinking block), sourced from the Cloud Control Plane's persisted planner-decision log.
|
||||
|
||||
#### Scenario: Task has persisted planner decisions with rationale
|
||||
- **WHEN** an operator opens the LLM interaction history view for a task that has one or more persisted planner decisions with non-null rationale
|
||||
- **THEN** Cloud Console shows each decision in step order with its prompt, resulting tool call, and rationale text
|
||||
|
||||
#### Scenario: Task has persisted planner decisions with thinking
|
||||
- **WHEN** an operator opens the LLM interaction history view for a task whose planner decisions include a non-null thinking field
|
||||
- **THEN** Cloud Console shows the thinking content alongside the prompt and tool call for that step
|
||||
|
||||
#### Scenario: Task has persisted planner decisions without rationale or thinking
|
||||
- **WHEN** an operator opens the LLM interaction history view for a task whose decisions have null rationale and null thinking
|
||||
- **THEN** Cloud Console shows each decision without those fields, without error or placeholder text
|
||||
|
||||
#### Scenario: Task's Host used direct-to-provider transport
|
||||
- **WHEN** an operator opens the LLM interaction history view for a task whose Host used direct-to-provider transport
|
||||
- **THEN** Cloud Console indicates that no LLM interaction history is available because the Host does not report it, rather than showing an empty history with no explanation
|
||||
|
||||
### Requirement: Cloud Control Plane persists rationale and thinking in the planner decision log
|
||||
The Cloud Control Plane's planner-decision log SHALL store the AI's rationale and thinking fields alongside the existing prompt and tool-call fields for each persisted decision. Both fields SHALL be nullable; absence of either field SHALL NOT prevent a decision record from being stored or queried.
|
||||
|
||||
#### Scenario: Decision record includes rationale
|
||||
- **WHEN** the Host Agent reports a planner decision with a non-null rationale
|
||||
- **THEN** the persisted `planner_decision_log` row stores that rationale text in the `rationale` column
|
||||
|
||||
#### Scenario: Decision record includes thinking
|
||||
- **WHEN** the Host Agent reports a planner decision with a non-null thinking block
|
||||
- **THEN** the persisted `planner_decision_log` row stores that thinking text in the `thinking` column
|
||||
|
||||
#### Scenario: Decision record has no rationale or thinking
|
||||
- **WHEN** the Host Agent reports a planner decision with null rationale and null thinking (e.g., cloud-proxy transport where these are not surfaced)
|
||||
- **THEN** the persisted row stores NULL for both columns without error
|
||||
|
||||
#### Scenario: Existing decision records without rationale or thinking remain readable
|
||||
- **WHEN** the system queries a `planner_decision_log` row created before this migration
|
||||
- **THEN** both `rationale` and `thinking` read as NULL, and the row is returned normally
|
||||
@@ -0,0 +1,61 @@
|
||||
## ADDED Requirements
|
||||
|
||||
### Requirement: AI planner outputs a rationale text block before each tool call
|
||||
The system SHALL instruct the AI planner to output a short text block (1–2 sentences) before each tool call, first assessing whether the previous action achieved its intended effect and then stating the intent of the current action. This reflection SHALL be produced within the same LLM call as the tool selection, without a separate round-trip.
|
||||
|
||||
#### Scenario: Planner outputs rationale on a non-first step
|
||||
- **WHEN** the AI planner is invoked for a step that has at least one prior executed step in history
|
||||
- **THEN** the planner's response includes a text block that references whether the previous action succeeded and states the intent of the current action, alongside the tool call
|
||||
|
||||
#### Scenario: Planner outputs rationale on the first step
|
||||
- **WHEN** the AI planner is invoked for the first step of a task (no prior history)
|
||||
- **THEN** the planner's response may include a text block stating the intent of the first action, and the absence of a text block does not constitute a failure
|
||||
|
||||
#### Scenario: Absent rationale degrades gracefully
|
||||
- **WHEN** the AI planner produces a tool call with no preceding text block
|
||||
- **THEN** the system stores `rationale=None` for that step and continues normally without error or retry
|
||||
|
||||
### Requirement: Tool-calling client captures thinking and text output from LLM responses
|
||||
The system SHALL extract and preserve the AI model's thinking block (when extended thinking is enabled) and any pre-tool text block (rationale) from the raw LLM response, making both available on the `ToolCallDecision` returned from `decide()`.
|
||||
|
||||
#### Scenario: Anthropic response contains a thinking block
|
||||
- **WHEN** the Anthropic client receives a response with a `thinking`-type content block
|
||||
- **THEN** `ToolCallDecision.thinking` is populated with the text of that thinking block
|
||||
|
||||
#### Scenario: Anthropic response contains a text block before the tool call
|
||||
- **WHEN** the Anthropic client receives a response with a `text`-type content block preceding the `tool_use` block
|
||||
- **THEN** `ToolCallDecision.text_output` is populated with that text
|
||||
|
||||
#### Scenario: OpenAI response contains reasoning content
|
||||
- **WHEN** the OpenAI client receives a response whose message includes a `reasoning_content` field (o-series models)
|
||||
- **THEN** `ToolCallDecision.thinking` is populated with that reasoning content
|
||||
|
||||
#### Scenario: Neither thinking nor text block is present
|
||||
- **WHEN** the LLM response contains only a tool call block (no thinking, no text)
|
||||
- **THEN** `ToolCallDecision.thinking` and `ToolCallDecision.text_output` are both `None`, and the decision is returned normally
|
||||
|
||||
### Requirement: Extended thinking is opt-in via configuration
|
||||
The system SHALL support enabling Anthropic extended thinking for the AI planner via a `thinking_budget_tokens` configuration value. When not configured, the planner SHALL operate identically to its pre-existing behavior.
|
||||
|
||||
#### Scenario: Extended thinking enabled
|
||||
- **WHEN** `AI_PLANNER_THINKING_BUDGET_TOKENS` is set to a positive integer and the provider is `anthropic`
|
||||
- **THEN** the Anthropic client includes the thinking parameter in the API request with the configured budget, and the `interleaved-thinking` beta header is sent
|
||||
|
||||
#### Scenario: Extended thinking not configured (default)
|
||||
- **WHEN** `AI_PLANNER_THINKING_BUDGET_TOKENS` is not set
|
||||
- **THEN** the Anthropic client makes requests without the thinking parameter, identical to prior behavior
|
||||
|
||||
#### Scenario: Extended thinking with OpenAI provider
|
||||
- **WHEN** `AI_PLANNER_THINKING_BUDGET_TOKENS` is set and the provider is `openai`
|
||||
- **THEN** the OpenAI client does not apply the Anthropic thinking parameter; reasoning content is captured only if the model returns it naturally
|
||||
|
||||
### Requirement: Execution history uses compact rationale-based representation
|
||||
The system SHALL construct the AI planner's history prompt from a compact per-step record containing the page context, rationale, action, and success flag — not the full scene JSON. This compact history SHALL be the sole format used when constructing the `history_summary` passed to `planner_user_prompt`.
|
||||
|
||||
#### Scenario: History prompt uses compact format
|
||||
- **WHEN** `_history_summary()` is called with a `WorldState` that has one or more history entries
|
||||
- **THEN** each entry in the returned list contains `page`, `rationale`, `action`, and `success` fields only, without any scene element data
|
||||
|
||||
#### Scenario: History prompt handles None rationale
|
||||
- **WHEN** a `WorldEvent` in history has `rationale=None`
|
||||
- **THEN** the compact history entry for that step includes `"rationale": null` without omitting the field or raising an error
|
||||
@@ -0,0 +1,36 @@
|
||||
## MODIFIED Requirements
|
||||
|
||||
### Requirement: Bounded history of recent scene/action pairs
|
||||
The system SHALL maintain `WorldState.history` as a fixed-size, bounded collection of the most recent per-step records, automatically evicting the oldest entry when a new entry is added past the configured bound. Each history record SHALL store the action name, success flag, page context (from `WorldState.current_page` at the time of recording), and optional rationale and thinking fields sourced from the executed `PlannedStep`. The `scene_summary` field SHALL be retained as an optional field for backward compatibility but SHALL NOT be required for new entries.
|
||||
|
||||
#### Scenario: WorldState survives across steps within a task
|
||||
- **WHEN** a task executes multiple steps in sequence
|
||||
- **THEN** the `WorldState` object associated with the task is the same object (or reflects continuously accumulated updates) across those steps, not reset between steps
|
||||
|
||||
#### Scenario: WorldState is scoped to a single task
|
||||
- **WHEN** two different tasks run (sequentially or concurrently) against the same or different devices
|
||||
- **THEN** each task has its own independent `WorldState`, and neither task's `WorldState` reflects the other task's app/page/variables/history
|
||||
|
||||
#### Scenario: History record includes rationale when planner provides it
|
||||
- **WHEN** the executed `PlannedStep` carries a non-None `rationale`
|
||||
- **THEN** the resulting `WorldEvent` stores that rationale string
|
||||
|
||||
#### Scenario: History record stores None rationale when planner does not provide one
|
||||
- **WHEN** the executed `PlannedStep` has `rationale=None`
|
||||
- **THEN** the resulting `WorldEvent` stores `rationale=None` without error
|
||||
|
||||
#### Scenario: History record includes thinking when planner provides it
|
||||
- **WHEN** the executed `PlannedStep` carries a non-None `thinking`
|
||||
- **THEN** the resulting `WorldEvent` stores that thinking string
|
||||
|
||||
#### Scenario: History record captures current page at time of recording
|
||||
- **WHEN** a step is appended to history and `WorldState.current_page` is non-None at that moment
|
||||
- **THEN** `WorldEvent.page` is set to that page value
|
||||
|
||||
#### Scenario: History record stores None page when current_page is unavailable
|
||||
- **WHEN** a step is appended to history and `WorldState.current_page` is None
|
||||
- **THEN** `WorldEvent.page` is `None` without error
|
||||
|
||||
#### Scenario: History size remains bounded
|
||||
- **WHEN** steps are appended beyond the configured history bound
|
||||
- **THEN** the oldest entries are evicted so history size never exceeds the bound
|
||||
Reference in New Issue
Block a user