agentic-mobile-control/openspec/changes/planner-reflection-history/proposal.md

Why

The AI planner's execution history currently stores raw scene_summary (full UI tree / semantic scene JSON) per step, which bloats context tokens rapidly and conveys no semantic intent. The LLM receives a long list of "tap succeeded" entries with no understanding of why each action was taken or whether it achieved its intended effect, making it prone to repeating mistakes and unable to self-correct mid-task.

What Changes

• Planner reflection loop (method C): The system prompt is updated to require the AI to output a short text block before each tool call — first evaluating whether the previous step achieved its intended effect, then stating the intent of the current step. This reflection happens within the same LLM call (no extra API round-trip).
• Rationale capture: ToolCallDecision captures the AI's pre-tool text output (text_output) and, when extended thinking is enabled, the thinking block (thinking). Both flow through PlannedStep into WorldEvent.
• Extended thinking support: AnthropicToolCallingClient gains optional thinking_budget_tokens config. When set, the Anthropic API is called with thinking enabled (interleaved thinking beta); the thinking block is extracted and stored.
• OpenAI reasoning capture: OpenAIToolCallingClient extracts reasoning_content from responses when present (o-series models).
• Required action metadata: every device-action tool call must return a concise purpose and observable expected_outcome; the Runtime stores them separately from executable tool arguments so successful executions can be reused as semantically meaningful flows.
• WorldEvent schema change: scene_summary is replaced by rationale/thinking plus the action name, executable arguments, purpose, expected outcome, and current_page from WorldState, giving later planning and skill reuse a compact but complete action record.
• History format: _history_summary() in ai_planner.py switches from full WorldEvent.to_dict() to a compact {page, rationale, action, success} format.
• planner_decision_log extension: The Cloud-side decision log table adds thinking and rationale columns to persist these fields alongside existing prompt/tool records.

Capabilities

New Capabilities

• planner-reflection-history: AI planner captures per-step rationale (pre-tool text reflection) and optional thinking (extended thinking block), stored in execution history and surfaced in the Cloud planner decision log.

Modified Capabilities

• world-model: WorldEvent schema changes — scene_summary becomes optional (backward-compat), new rationale: str | None and thinking: str | None fields added; history_summary format for prompt construction changes to compact representation.
• agent-runtime: PlannedStep gains rationale and thinking fields; ToolCallDecision gains text_output and thinking fields.
• cloud-task-progress-visibility: planner_decision_log table extended with thinking and rationale columns.

Impact

• runtime/tool_calling_client.py — ToolCallDecision, AnthropicToolCallingClient, OpenAIToolCallingClient, response parsers
• runtime/planner.py — PlannedStep
• runtime/tool_specs.py — required purpose/expected-outcome fields for device actions
• runtime/ai_planner.py — AIPlanner.plan(), _history_summary()
• runtime/planner_prompts.py — PLANNER_SYSTEM_PROMPT, planner_user_prompt
• runtime/planner_config.py — new thinking_budget_tokens field
• world/models.py — WorldEvent
• world/model.py — _append_history()
• packages/cloud-platform/cloud/db_models.py — planner_decision_log table
• packages/cloud-platform/cloud/schema.py — Alembic migration
• packages/cloud-platform/cloud/internal_api/api.py — record_planner_decision() and planner decision response
• packages/cloud-platform/cloud/internal_api/models.py — planner decision transport models
• apps/device-host-agent/host_agent/cloud_planner_client.py — return reflection and action metadata from the Cloud proxy
• skills_learning/ — retain action purpose/expected outcome in synthesized flow steps
• No new external dependencies; Anthropic extended thinking uses existing SDK via beta header.