q792602257
a5aeb8889c
Tests / Test failed: 2, passed: 849
- ToolCallDecision captures thinking blocks and pre-tool text output
- AnthropicToolCallingClient supports optional extended thinking (budget_tokens + beta header)
- PlannedStep carries rationale and thinking from each LLM decision
- WorldEvent replaces scene_summary with rationale/thinking/page fields (backward-compatible)
- AI planner system prompt instructs reflection before each tool call
- _history_summary() emits compact {page, rationale, action, success} dicts
- Cloud DB migration 0011 adds nullable rationale/thinking columns to planner_decision_log
- OpenAI client extracts reasoning_content into thinking field
42 lines
3.8 KiB
Markdown
42 lines
3.8 KiB
Markdown
## Why
|
|
|
|
The AI planner's execution history currently stores raw `scene_summary` (full UI tree / semantic scene JSON) per step, which bloats context tokens rapidly and conveys no semantic intent. The LLM receives a long list of "tap succeeded" entries with no understanding of why each action was taken or whether it achieved its intended effect, making it prone to repeating mistakes and unable to self-correct mid-task.
|
|
|
|
## What Changes
|
|
|
|
- **Planner reflection loop (method C)**: The system prompt is updated to require the AI to output a short text block before each tool call — first evaluating whether the previous step achieved its intended effect, then stating the intent of the current step. This reflection happens within the same LLM call (no extra API round-trip).
|
|
- **Rationale capture**: `ToolCallDecision` captures the AI's pre-tool text output (`text_output`) and, when extended thinking is enabled, the thinking block (`thinking`). Both flow through `PlannedStep` into `WorldEvent`.
|
|
- **Extended thinking support**: `AnthropicToolCallingClient` gains optional `thinking_budget_tokens` config. When set, the Anthropic API is called with `thinking` enabled (interleaved thinking beta); the thinking block is extracted and stored.
|
|
- **OpenAI reasoning capture**: `OpenAIToolCallingClient` extracts `reasoning_content` from responses when present (o-series models).
|
|
- **WorldEvent schema change**: `scene_summary` is replaced by `rationale` (from `text_output`) and `thinking` (from thinking block), plus `current_page` from `WorldState` for minimal page-level verification context. This reduces per-step history token cost by an order of magnitude.
|
|
- **History format**: `_history_summary()` in `ai_planner.py` switches from full `WorldEvent.to_dict()` to a compact `{page, rationale, action, success}` format.
|
|
- **planner_decision_log extension**: The Cloud-side decision log table adds `thinking` and `rationale` columns to persist these fields alongside existing prompt/tool records.
|
|
|
|
## Capabilities
|
|
|
|
### New Capabilities
|
|
|
|
- `planner-reflection-history`: AI planner captures per-step rationale (pre-tool text reflection) and optional thinking (extended thinking block), stored in execution history and surfaced in the Cloud planner decision log.
|
|
|
|
### Modified Capabilities
|
|
|
|
- `world-model`: `WorldEvent` schema changes — `scene_summary` becomes optional (backward-compat), new `rationale: str | None` and `thinking: str | None` fields added; `history_summary` format for prompt construction changes to compact representation.
|
|
- `agent-runtime`: `PlannedStep` gains `rationale` and `thinking` fields; `ToolCallDecision` gains `text_output` and `thinking` fields.
|
|
- `cloud-task-progress-visibility`: `planner_decision_log` table extended with `thinking` and `rationale` columns.
|
|
|
|
## Impact
|
|
|
|
- `runtime/tool_calling_client.py` — `ToolCallDecision`, `AnthropicToolCallingClient`, `OpenAIToolCallingClient`, response parsers
|
|
- `runtime/planner.py` — `PlannedStep`
|
|
- `runtime/ai_planner.py` — `AIPlanner.plan()`, `_history_summary()`
|
|
- `runtime/planner_prompts.py` — `PLANNER_SYSTEM_PROMPT`, `planner_user_prompt`
|
|
- `runtime/planner_config.py` — new `thinking_budget_tokens` field
|
|
- `world/models.py` — `WorldEvent`
|
|
- `world/model.py` — `_append_history()`
|
|
- `packages/cloud-platform/cloud/db_models.py` — `planner_decision_log` table
|
|
- `packages/cloud-platform/cloud/schema.py` — Alembic migration
|
|
- `packages/cloud-platform/cloud/internal_api/api.py` — `record_planner_decision()`
|
|
- `packages/cloud-platform/cloud/internal_api/models.py` — `PlannerDecisionRecord`
|
|
- No changes to `CloudProxyToolCallingClient` — thinking/text are surfaced at the local client layer only; the cloud proxy is transparent to them.
|
|
- No new external dependencies; Anthropic extended thinking uses existing SDK via beta header.
|