45 lines
4.2 KiB
Markdown
45 lines
4.2 KiB
Markdown
## Why
|
|
|
|
The AI planner's execution history currently stores raw `scene_summary` (full UI tree / semantic scene JSON) per step, which bloats context tokens rapidly and conveys no semantic intent. The LLM receives a long list of "tap succeeded" entries with no understanding of why each action was taken or whether it achieved its intended effect, making it prone to repeating mistakes and unable to self-correct mid-task.
|
|
|
|
## What Changes
|
|
|
|
- **Planner reflection loop (method C)**: The system prompt is updated to require the AI to output a short text block before each tool call — first evaluating whether the previous step achieved its intended effect, then stating the intent of the current step. This reflection happens within the same LLM call (no extra API round-trip).
|
|
- **Rationale capture**: `ToolCallDecision` captures the AI's pre-tool text output (`text_output`) and, when extended thinking is enabled, the thinking block (`thinking`). Both flow through `PlannedStep` into `WorldEvent`.
|
|
- **Extended thinking support**: `AnthropicToolCallingClient` gains optional `thinking_budget_tokens` config. When set, the Anthropic API is called with `thinking` enabled (interleaved thinking beta); the thinking block is extracted and stored.
|
|
- **OpenAI reasoning capture**: `OpenAIToolCallingClient` extracts `reasoning_content` from responses when present (o-series models).
|
|
- **Required action metadata**: every device-action tool call must return a concise `purpose` and observable `expected_outcome`; the Runtime stores them separately from executable tool arguments so successful executions can be reused as semantically meaningful flows.
|
|
- **WorldEvent schema change**: `scene_summary` is replaced by rationale/thinking plus the action name, executable arguments, purpose, expected outcome, and `current_page` from `WorldState`, giving later planning and skill reuse a compact but complete action record.
|
|
- **History format**: `_history_summary()` in `ai_planner.py` switches from full `WorldEvent.to_dict()` to a compact `{page, rationale, action, success}` format.
|
|
- **planner_decision_log extension**: The Cloud-side decision log table adds `thinking` and `rationale` columns to persist these fields alongside existing prompt/tool records.
|
|
|
|
## Capabilities
|
|
|
|
### New Capabilities
|
|
|
|
- `planner-reflection-history`: AI planner captures per-step rationale (pre-tool text reflection) and optional thinking (extended thinking block), stored in execution history and surfaced in the Cloud planner decision log.
|
|
|
|
### Modified Capabilities
|
|
|
|
- `world-model`: `WorldEvent` schema changes — `scene_summary` becomes optional (backward-compat), new `rationale: str | None` and `thinking: str | None` fields added; `history_summary` format for prompt construction changes to compact representation.
|
|
- `agent-runtime`: `PlannedStep` gains `rationale` and `thinking` fields; `ToolCallDecision` gains `text_output` and `thinking` fields.
|
|
- `cloud-task-progress-visibility`: `planner_decision_log` table extended with `thinking` and `rationale` columns.
|
|
|
|
## Impact
|
|
|
|
- `runtime/tool_calling_client.py` — `ToolCallDecision`, `AnthropicToolCallingClient`, `OpenAIToolCallingClient`, response parsers
|
|
- `runtime/planner.py` — `PlannedStep`
|
|
- `runtime/tool_specs.py` — required purpose/expected-outcome fields for device actions
|
|
- `runtime/ai_planner.py` — `AIPlanner.plan()`, `_history_summary()`
|
|
- `runtime/planner_prompts.py` — `PLANNER_SYSTEM_PROMPT`, `planner_user_prompt`
|
|
- `runtime/planner_config.py` — new `thinking_budget_tokens` field
|
|
- `world/models.py` — `WorldEvent`
|
|
- `world/model.py` — `_append_history()`
|
|
- `packages/cloud-platform/cloud/db_models.py` — `planner_decision_log` table
|
|
- `packages/cloud-platform/cloud/schema.py` — Alembic migration
|
|
- `packages/cloud-platform/cloud/internal_api/api.py` — `record_planner_decision()` and planner decision response
|
|
- `packages/cloud-platform/cloud/internal_api/models.py` — planner decision transport models
|
|
- `apps/device-host-agent/host_agent/cloud_planner_client.py` — return reflection and action metadata from the Cloud proxy
|
|
- `skills_learning/` — retain action purpose/expected outcome in synthesized flow steps
|
|
- No new external dependencies; Anthropic extended thinking uses existing SDK via beta header.
|