q792602257
a5aeb8889c
Tests / Test failed: 2, passed: 849
- ToolCallDecision captures thinking blocks and pre-tool text output
- AnthropicToolCallingClient supports optional extended thinking (budget_tokens + beta header)
- PlannedStep carries rationale and thinking from each LLM decision
- WorldEvent replaces scene_summary with rationale/thinking/page fields (backward-compatible)
- AI planner system prompt instructs reflection before each tool call
- _history_summary() emits compact {page, rationale, action, success} dicts
- Cloud DB migration 0011 adds nullable rationale/thinking columns to planner_decision_log
- OpenAI client extracts reasoning_content into thinking field
3.8 KiB
3.8 KiB
Why
The AI planner's execution history currently stores raw scene_summary (full UI tree / semantic scene JSON) per step, which bloats context tokens rapidly and conveys no semantic intent. The LLM receives a long list of "tap succeeded" entries with no understanding of why each action was taken or whether it achieved its intended effect, making it prone to repeating mistakes and unable to self-correct mid-task.
What Changes
- Planner reflection loop (method C): The system prompt is updated to require the AI to output a short text block before each tool call — first evaluating whether the previous step achieved its intended effect, then stating the intent of the current step. This reflection happens within the same LLM call (no extra API round-trip).
- Rationale capture:
ToolCallDecisioncaptures the AI's pre-tool text output (text_output) and, when extended thinking is enabled, the thinking block (thinking). Both flow throughPlannedStepintoWorldEvent. - Extended thinking support:
AnthropicToolCallingClientgains optionalthinking_budget_tokensconfig. When set, the Anthropic API is called withthinkingenabled (interleaved thinking beta); the thinking block is extracted and stored. - OpenAI reasoning capture:
OpenAIToolCallingClientextractsreasoning_contentfrom responses when present (o-series models). - WorldEvent schema change:
scene_summaryis replaced byrationale(fromtext_output) andthinking(from thinking block), pluscurrent_pagefromWorldStatefor minimal page-level verification context. This reduces per-step history token cost by an order of magnitude. - History format:
_history_summary()inai_planner.pyswitches from fullWorldEvent.to_dict()to a compact{page, rationale, action, success}format. - planner_decision_log extension: The Cloud-side decision log table adds
thinkingandrationalecolumns to persist these fields alongside existing prompt/tool records.
Capabilities
New Capabilities
planner-reflection-history: AI planner captures per-step rationale (pre-tool text reflection) and optional thinking (extended thinking block), stored in execution history and surfaced in the Cloud planner decision log.
Modified Capabilities
world-model:WorldEventschema changes —scene_summarybecomes optional (backward-compat), newrationale: str | Noneandthinking: str | Nonefields added;history_summaryformat for prompt construction changes to compact representation.agent-runtime:PlannedStepgainsrationaleandthinkingfields;ToolCallDecisiongainstext_outputandthinkingfields.cloud-task-progress-visibility:planner_decision_logtable extended withthinkingandrationalecolumns.
Impact
runtime/tool_calling_client.py—ToolCallDecision,AnthropicToolCallingClient,OpenAIToolCallingClient, response parsersruntime/planner.py—PlannedStepruntime/ai_planner.py—AIPlanner.plan(),_history_summary()runtime/planner_prompts.py—PLANNER_SYSTEM_PROMPT,planner_user_promptruntime/planner_config.py— newthinking_budget_tokensfieldworld/models.py—WorldEventworld/model.py—_append_history()packages/cloud-platform/cloud/db_models.py—planner_decision_logtablepackages/cloud-platform/cloud/schema.py— Alembic migrationpackages/cloud-platform/cloud/internal_api/api.py—record_planner_decision()packages/cloud-platform/cloud/internal_api/models.py—PlannerDecisionRecord- No changes to
CloudProxyToolCallingClient— thinking/text are surfaced at the local client layer only; the cloud proxy is transparent to them. - No new external dependencies; Anthropic extended thinking uses existing SDK via beta header.