This commit is contained in:
@@ -36,9 +36,7 @@ Constraints:
|
||||
|
||||
**Non-Goals:**
|
||||
- Not adding a dedicated reflection LLM call (explicit method B) — method C (pre-tool text block) achieves the intent within the existing call budget.
|
||||
- Not changing `CloudProxyToolCallingClient` — it transparently proxies and is not aware of thinking/text blocks.
|
||||
- Not changing the `scene_summary` field in `WorldEvent` for stored/displayed task history in the Console — only the prompt-construction path (`_history_summary()`) is changed.
|
||||
- Not surfacing thinking/rationale in the Cloud Console UI (that's a follow-on concern).
|
||||
- Not making `thinking_budget_tokens` configurable per-task at runtime (only via environment variable).
|
||||
|
||||
## Decisions
|
||||
@@ -112,13 +110,42 @@ Constraints:
|
||||
|
||||
**Why separate columns rather than a JSON blob**: The existing table uses discrete columns for `system_prompt`, `user_prompt`, `tool_name`, `tool_arguments` — consistency favours discrete columns. Both fields are optional (cloud-proxy path only; `direct` transport never produces cloud decision records).
|
||||
|
||||
### D9: Device-action tool calls carry required purpose and expected outcome
|
||||
|
||||
**Decision**: Every device-action schema (`tap`, `swipe`, `input_text`,
|
||||
`launch_app`, and `terminate_app`) adds required non-empty `purpose` and
|
||||
`expected_outcome` string fields. The tool-call response parser removes these
|
||||
metadata fields from executable arguments and assigns them to
|
||||
`ToolCallDecision`; `AIPlanner` carries them on `PlannedStep`. The completion
|
||||
signal remains unchanged because it already has a required terminal `reason`.
|
||||
|
||||
The Cloud planner response returns rationale, thinking, purpose, and expected
|
||||
outcome to the Host. The Cloud decision log persists the four values as
|
||||
nullable, additive columns so historic rows and non-AI callers remain
|
||||
readable. `WorldEvent` retains the executable action arguments together with
|
||||
purpose and expected outcome; its compact history summary includes that action
|
||||
record. Timeline records and learned `FlowStep` instances retain the same
|
||||
arguments and metadata; flow embedding text includes available semantics so
|
||||
retrieval can use them.
|
||||
|
||||
**Why structured tool arguments rather than rationale**: the pre-tool text
|
||||
block is optional by API design and can be suppressed by a forced retry. Tool
|
||||
schemas are the provider-enforced structured-output boundary, so requiring
|
||||
purpose and expected outcome there makes them available for every submitted
|
||||
device action without relying on free-form rationale.
|
||||
|
||||
**Why strip metadata before execution**: device tool functions only accept
|
||||
their physical-action arguments. Keeping metadata off `step.args` preserves
|
||||
their API contracts while still making it available to execution history,
|
||||
verification, Cloud audit, and skill synthesis.
|
||||
|
||||
## Risks / Trade-offs
|
||||
|
||||
- **AI may not always output a text block**: even with `tool_choice: "auto"` (D8), the model is not guaranteed to prefix a text block before the tool call. When absent, `text_output` is `None` and `rationale` is `None`. History degrades gracefully to `{page, rationale: null, action, success}`.
|
||||
- **`tool_choice: "auto"` occasionally yields no tool call at all**: unlike forced `tool_choice`, `"auto"` permits the model to respond with text only and no tool call. D8's forced retry (no thinking, no rationale on that path) guards this case so a step never stalls; this trades away rationale/thinking for that single step, not overall reliability.
|
||||
- **Extended thinking increases latency**: `budget_tokens` directly adds to minimum response time. This is opt-in and accepted by the operator who enables it.
|
||||
- **`WorldEvent` schema divergence from stored data**: Existing `WorldEvent` instances in memory or serialised timelines lack `rationale`/`thinking`. The `to_dict()` method will emit `null` for these fields; downstream consumers should treat `null` as absent, not as a failure.
|
||||
- **Cloud-proxy transport never produces thinking/rationale at the client layer**: The proxy returns only `tool_name`/`arguments`. `ToolCallDecision.thinking` and `.text_output` will always be `None` for cloud-transport tasks. The `planner_decision_log` on the cloud side will be populated from the cloud-proxied call itself (D7), which does see the full LLM response.
|
||||
- **Cloud/Host deployment order**: a new Host requires a Cloud API that returns the additive metadata fields to preserve them locally. The Cloud response models remain nullable so an old peer remains readable during a rolling deployment, but it cannot provide the new semantic records.
|
||||
|
||||
## Migration Plan
|
||||
|
||||
|
||||
@@ -8,7 +8,8 @@ The AI planner's execution history currently stores raw `scene_summary` (full UI
|
||||
- **Rationale capture**: `ToolCallDecision` captures the AI's pre-tool text output (`text_output`) and, when extended thinking is enabled, the thinking block (`thinking`). Both flow through `PlannedStep` into `WorldEvent`.
|
||||
- **Extended thinking support**: `AnthropicToolCallingClient` gains optional `thinking_budget_tokens` config. When set, the Anthropic API is called with `thinking` enabled (interleaved thinking beta); the thinking block is extracted and stored.
|
||||
- **OpenAI reasoning capture**: `OpenAIToolCallingClient` extracts `reasoning_content` from responses when present (o-series models).
|
||||
- **WorldEvent schema change**: `scene_summary` is replaced by `rationale` (from `text_output`) and `thinking` (from thinking block), plus `current_page` from `WorldState` for minimal page-level verification context. This reduces per-step history token cost by an order of magnitude.
|
||||
- **Required action metadata**: every device-action tool call must return a concise `purpose` and observable `expected_outcome`; the Runtime stores them separately from executable tool arguments so successful executions can be reused as semantically meaningful flows.
|
||||
- **WorldEvent schema change**: `scene_summary` is replaced by rationale/thinking plus the action name, executable arguments, purpose, expected outcome, and `current_page` from `WorldState`, giving later planning and skill reuse a compact but complete action record.
|
||||
- **History format**: `_history_summary()` in `ai_planner.py` switches from full `WorldEvent.to_dict()` to a compact `{page, rationale, action, success}` format.
|
||||
- **planner_decision_log extension**: The Cloud-side decision log table adds `thinking` and `rationale` columns to persist these fields alongside existing prompt/tool records.
|
||||
|
||||
@@ -28,6 +29,7 @@ The AI planner's execution history currently stores raw `scene_summary` (full UI
|
||||
|
||||
- `runtime/tool_calling_client.py` — `ToolCallDecision`, `AnthropicToolCallingClient`, `OpenAIToolCallingClient`, response parsers
|
||||
- `runtime/planner.py` — `PlannedStep`
|
||||
- `runtime/tool_specs.py` — required purpose/expected-outcome fields for device actions
|
||||
- `runtime/ai_planner.py` — `AIPlanner.plan()`, `_history_summary()`
|
||||
- `runtime/planner_prompts.py` — `PLANNER_SYSTEM_PROMPT`, `planner_user_prompt`
|
||||
- `runtime/planner_config.py` — new `thinking_budget_tokens` field
|
||||
@@ -35,7 +37,8 @@ The AI planner's execution history currently stores raw `scene_summary` (full UI
|
||||
- `world/model.py` — `_append_history()`
|
||||
- `packages/cloud-platform/cloud/db_models.py` — `planner_decision_log` table
|
||||
- `packages/cloud-platform/cloud/schema.py` — Alembic migration
|
||||
- `packages/cloud-platform/cloud/internal_api/api.py` — `record_planner_decision()`
|
||||
- `packages/cloud-platform/cloud/internal_api/models.py` — `PlannerDecisionRecord`
|
||||
- No changes to `CloudProxyToolCallingClient` — thinking/text are surfaced at the local client layer only; the cloud proxy is transparent to them.
|
||||
- `packages/cloud-platform/cloud/internal_api/api.py` — `record_planner_decision()` and planner decision response
|
||||
- `packages/cloud-platform/cloud/internal_api/models.py` — planner decision transport models
|
||||
- `apps/device-host-agent/host_agent/cloud_planner_client.py` — return reflection and action metadata from the Cloud proxy
|
||||
- `skills_learning/` — retain action purpose/expected outcome in synthesized flow steps
|
||||
- No new external dependencies; Anthropic extended thinking uses existing SDK via beta header.
|
||||
|
||||
@@ -23,6 +23,10 @@ The system SHALL provide a Planner implementation that, given a goal, the curren
|
||||
- **WHEN** the LLM response contains only a tool call with no preceding text block
|
||||
- **THEN** `PlannedStep.rationale` is `None` and the step is returned normally
|
||||
|
||||
#### Scenario: Device action includes reusable purpose and expected outcome
|
||||
- **WHEN** the AI Planner selects a device action (`tap`, `swipe`, `input_text`, `launch_app`, or `terminate_app`)
|
||||
- **THEN** its tool call requires non-empty `purpose` and `expected_outcome` values, and the returned `PlannedStep` carries both separately from the executable action arguments
|
||||
|
||||
### Requirement: Pluggable dual-provider tool-calling abstraction
|
||||
The system SHALL support at least two interchangeable LLM providers (Anthropic native tool use and OpenAI function calling) for the AI Planner's decision calls, selectable via configuration, with both providers constrained to return exactly one tool call per request. The Anthropic client SHALL additionally support optional extended thinking via a configurable `thinking_budget_tokens` value. The OpenAI client SHALL capture `reasoning_content` from responses when present. Independently of provider selection, the system SHALL support at least two transports for making that decision call — direct-to-provider and cloud-proxy — selectable via configuration without requiring any change to `AIPlanner`'s own decision logic.
|
||||
|
||||
@@ -50,6 +54,6 @@ The system SHALL support at least two interchangeable LLM providers (Anthropic n
|
||||
- **WHEN** the Host Agent is configured with the cloud-proxy transport
|
||||
- **THEN** its tool-calling client sends the decision request to the Cloud Control Plane's planner-decision endpoint instead of constructing a local Anthropic or OpenAI SDK client
|
||||
|
||||
#### Scenario: Cloud-proxy transport returns None for thinking and text_output
|
||||
- **WHEN** the Host Agent uses cloud-proxy transport
|
||||
- **THEN** `ToolCallDecision.thinking` and `ToolCallDecision.text_output` are `None` because the proxy surface does not expose them
|
||||
#### Scenario: Cloud-proxy transport returns planner metadata
|
||||
- **WHEN** the Host Agent uses cloud-proxy transport and the Cloud Planner returns a decision
|
||||
- **THEN** the proxy returns its rationale, thinking, purpose, and expected outcome alongside the tool name and executable arguments
|
||||
|
||||
+6
-2
@@ -1,7 +1,7 @@
|
||||
## MODIFIED Requirements
|
||||
|
||||
### Requirement: Cloud Console displays a task's full LLM interaction history
|
||||
Cloud Console SHALL provide a view, for a given task, listing each persisted planner decision in step order, including its full prompt, resulting decision, and — when available — the AI's rationale (pre-tool text reflection) and thinking (extended thinking block), sourced from the Cloud Control Plane's persisted planner-decision log.
|
||||
Cloud Console SHALL provide a view, for a given task, listing each persisted planner decision in step order, including its full prompt, resulting decision, and — when available — the AI's rationale (pre-tool text reflection), thinking (extended thinking block), action purpose, and expected outcome, sourced from the Cloud Control Plane's persisted planner-decision log.
|
||||
|
||||
#### Scenario: Task has persisted planner decisions with rationale
|
||||
- **WHEN** an operator opens the LLM interaction history view for a task that has one or more persisted planner decisions with non-null rationale
|
||||
@@ -20,7 +20,7 @@ Cloud Console SHALL provide a view, for a given task, listing each persisted pla
|
||||
- **THEN** Cloud Console indicates that no LLM interaction history is available because the Host does not report it, rather than showing an empty history with no explanation
|
||||
|
||||
### Requirement: Cloud Control Plane persists rationale and thinking in the planner decision log
|
||||
The Cloud Control Plane's planner-decision log SHALL store the AI's rationale and thinking fields alongside the existing prompt and tool-call fields for each persisted decision. Both fields SHALL be nullable; absence of either field SHALL NOT prevent a decision record from being stored or queried.
|
||||
The Cloud Control Plane's planner-decision log SHALL store the AI's rationale, thinking, action purpose, and expected outcome fields alongside the existing prompt and tool-call fields for each persisted decision. These fields SHALL be nullable for backward compatibility; absence of a legacy or non-AI value SHALL NOT prevent a decision record from being stored or queried.
|
||||
|
||||
#### Scenario: Decision record includes rationale
|
||||
- **WHEN** the Host Agent reports a planner decision with a non-null rationale
|
||||
@@ -34,6 +34,10 @@ The Cloud Control Plane's planner-decision log SHALL store the AI's rationale an
|
||||
- **WHEN** the Host Agent reports a planner decision with null rationale and null thinking (e.g., cloud-proxy transport where these are not surfaced)
|
||||
- **THEN** the persisted row stores NULL for both columns without error
|
||||
|
||||
#### Scenario: Decision record includes reusable action metadata
|
||||
- **WHEN** the Host reports a device-action planner decision with a purpose and expected outcome
|
||||
- **THEN** the persisted row stores both values separately from the executable tool arguments and the task API returns them to authorized readers
|
||||
|
||||
#### Scenario: Existing decision records without rationale or thinking remain readable
|
||||
- **WHEN** the system queries a `planner_decision_log` row created before this migration
|
||||
- **THEN** both `rationale` and `thinking` read as NULL, and the row is returned normally
|
||||
|
||||
+17
-2
@@ -34,6 +34,17 @@ The system SHALL extract and preserve the AI model's thinking block (when extend
|
||||
- **WHEN** the LLM response contains only a tool call block (no thinking, no text)
|
||||
- **THEN** `ToolCallDecision.thinking` and `ToolCallDecision.text_output` are both `None`, and the decision is returned normally
|
||||
|
||||
### Requirement: Device actions return required reusable metadata
|
||||
The system SHALL require each device-action tool call to include concise, non-empty `purpose` and `expected_outcome` strings in its structured arguments. The Runtime SHALL preserve these values as planner metadata while excluding them from the arguments supplied to the physical device tool.
|
||||
|
||||
#### Scenario: Tool schema requires purpose and expected outcome
|
||||
- **WHEN** the Planner sends an action tool schema to an LLM provider
|
||||
- **THEN** each device-action schema requires `purpose` and `expected_outcome` in addition to its physical-action arguments
|
||||
|
||||
#### Scenario: Action metadata is not passed to the device tool
|
||||
- **WHEN** a planned action is executed
|
||||
- **THEN** the device tool receives only its physical-action arguments while the purpose and expected outcome remain available on the planned step and execution record
|
||||
|
||||
### Requirement: Extended thinking is opt-in via configuration
|
||||
The system SHALL support enabling Anthropic extended thinking for the AI planner via a `thinking_budget_tokens` configuration value. When not configured, the planner SHALL operate identically to its pre-existing behavior.
|
||||
|
||||
@@ -50,12 +61,16 @@ The system SHALL support enabling Anthropic extended thinking for the AI planner
|
||||
- **THEN** the OpenAI client does not apply the Anthropic thinking parameter; reasoning content is captured only if the model returns it naturally
|
||||
|
||||
### Requirement: Execution history uses compact rationale-based representation
|
||||
The system SHALL construct the AI planner's history prompt from a compact per-step record containing the page context, rationale, action, and success flag — not the full scene JSON. This compact history SHALL be the sole format used when constructing the `history_summary` passed to `planner_user_prompt`.
|
||||
The system SHALL construct the AI planner's history prompt from a compact per-step record containing the page context, rationale, action, executable action arguments, purpose, expected outcome, and success flag — not the full scene JSON. This compact history SHALL be the sole format used when constructing the `history_summary` passed to `planner_user_prompt`.
|
||||
|
||||
#### Scenario: History prompt uses compact format
|
||||
- **WHEN** `_history_summary()` is called with a `WorldState` that has one or more history entries
|
||||
- **THEN** each entry in the returned list contains `page`, `rationale`, `action`, and `success` fields only, without any scene element data
|
||||
- **THEN** each entry in the returned list contains `page`, `rationale`, `action`, `arguments`, `purpose`, `expected_outcome`, and `success` fields only, without any scene element data
|
||||
|
||||
#### Scenario: History prompt handles None rationale
|
||||
- **WHEN** a `WorldEvent` in history has `rationale=None`
|
||||
- **THEN** the compact history entry for that step includes `"rationale": null` without omitting the field or raising an error
|
||||
|
||||
#### Scenario: History prompt preserves executed action arguments
|
||||
- **WHEN** a prior action tapped a coordinate or otherwise supplied tool arguments
|
||||
- **THEN** the compact history entry includes those exact executable arguments alongside the action name
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
## MODIFIED Requirements
|
||||
|
||||
### Requirement: Bounded history of recent scene/action pairs
|
||||
The system SHALL maintain `WorldState.history` as a fixed-size, bounded collection of the most recent per-step records, automatically evicting the oldest entry when a new entry is added past the configured bound. Each history record SHALL store the action name, success flag, page context (from `WorldState.current_page` at the time of recording), and optional rationale and thinking fields sourced from the executed `PlannedStep`. The `scene_summary` field SHALL be retained as an optional field for backward compatibility but SHALL NOT be required for new entries.
|
||||
The system SHALL maintain `WorldState.history` as a fixed-size, bounded collection of the most recent per-step records, automatically evicting the oldest entry when a new entry is added past the configured bound. Each history record SHALL store the action name, executable action arguments, success flag, page context (from `WorldState.current_page` at the time of recording), and optional rationale, thinking, purpose, and expected-outcome fields sourced from the executed `PlannedStep`. The `scene_summary` field SHALL be retained as an optional field for backward compatibility but SHALL NOT be required for new entries.
|
||||
|
||||
#### Scenario: WorldState survives across steps within a task
|
||||
- **WHEN** a task executes multiple steps in sequence
|
||||
@@ -23,6 +23,14 @@ The system SHALL maintain `WorldState.history` as a fixed-size, bounded collecti
|
||||
- **WHEN** the executed `PlannedStep` carries a non-None `thinking`
|
||||
- **THEN** the resulting `WorldEvent` stores that thinking string
|
||||
|
||||
#### Scenario: History record includes reusable action metadata
|
||||
- **WHEN** an executed `PlannedStep` carries a purpose and expected outcome
|
||||
- **THEN** the resulting `WorldEvent` stores both values for the next planning turn and later reuse
|
||||
|
||||
#### Scenario: History record includes executed action arguments
|
||||
- **WHEN** a `PlannedStep` executes with action arguments such as a tap's `x` and `y` coordinates
|
||||
- **THEN** the resulting `WorldEvent` stores those executable arguments with the action name
|
||||
|
||||
#### Scenario: History record captures current page at time of recording
|
||||
- **WHEN** a step is appended to history and `WorldState.current_page` is non-None at that moment
|
||||
- **THEN** `WorldEvent.page` is set to that page value
|
||||
|
||||
@@ -60,3 +60,11 @@
|
||||
- [x] 9.6 Add/rename unit tests in `tests/test_tool_calling_client.py` covering: default request uses `tool_choice: "auto"`; retry sequence when first response has no tool call (Anthropic and OpenAI); thinking/`betas` dropped on the forced retry; OpenAI `text_output` capture
|
||||
- [x] 9.7 Update `design.md` (D1 correction + new D8) and this file to document the bug and fix
|
||||
- [x] 9.8 Re-run `uv run --all-packages pytest -m "not integration"`, `ruff check`/`ruff format --check`, `python -m compileall`, and `openspec validate --strict --change "planner-reflection-history"` after the fix
|
||||
|
||||
## 10. Required reusable action metadata (post-completion amendment)
|
||||
|
||||
- [x] 10.1 Require `purpose` and `expected_outcome` in every device-action tool schema; extract them from executable arguments into `ToolCallDecision` and `PlannedStep`.
|
||||
- [x] 10.2 Return rationale, thinking, purpose, and expected outcome through the Cloud planner response; persist the new action metadata in `planner_decision_log` with an additive migration and expose it through the task decision API.
|
||||
- [x] 10.3 Preserve executable arguments, purpose, and expected outcome in `WorldEvent`, Timeline records, and synthesized `FlowStep` values; include available metadata in skill embedding text for semantic retrieval.
|
||||
- [x] 10.4 Render available rationale, thinking, purpose, and expected outcome in Cloud Console planner-decision history.
|
||||
- [x] 10.5 Add focused regression tests and run the relevant Python, frontend, migration, lint/format, and OpenSpec validation checks.
|
||||
|
||||
Reference in New Issue
Block a user