feat(planner): persist reusable action semantics
Tests / Test passed: 879

This commit is contained in:
2026-07-15 18:14:28 +08:00
parent 361dada276
commit d69be48f96
41 changed files with 733 additions and 116 deletions
@@ -36,9 +36,7 @@ Constraints:
**Non-Goals:**
- Not adding a dedicated reflection LLM call (explicit method B) — method C (pre-tool text block) achieves the intent within the existing call budget.
- Not changing `CloudProxyToolCallingClient` — it transparently proxies and is not aware of thinking/text blocks.
- Not changing the `scene_summary` field in `WorldEvent` for stored/displayed task history in the Console — only the prompt-construction path (`_history_summary()`) is changed.
- Not surfacing thinking/rationale in the Cloud Console UI (that's a follow-on concern).
- Not making `thinking_budget_tokens` configurable per-task at runtime (only via environment variable).
## Decisions
@@ -112,13 +110,42 @@ Constraints:
**Why separate columns rather than a JSON blob**: The existing table uses discrete columns for `system_prompt`, `user_prompt`, `tool_name`, `tool_arguments` — consistency favours discrete columns. Both fields are optional (cloud-proxy path only; `direct` transport never produces cloud decision records).
### D9: Device-action tool calls carry required purpose and expected outcome
**Decision**: Every device-action schema (`tap`, `swipe`, `input_text`,
`launch_app`, and `terminate_app`) adds required non-empty `purpose` and
`expected_outcome` string fields. The tool-call response parser removes these
metadata fields from executable arguments and assigns them to
`ToolCallDecision`; `AIPlanner` carries them on `PlannedStep`. The completion
signal remains unchanged because it already has a required terminal `reason`.
The Cloud planner response returns rationale, thinking, purpose, and expected
outcome to the Host. The Cloud decision log persists the four values as
nullable, additive columns so historic rows and non-AI callers remain
readable. `WorldEvent` retains the executable action arguments together with
purpose and expected outcome; its compact history summary includes that action
record. Timeline records and learned `FlowStep` instances retain the same
arguments and metadata; flow embedding text includes available semantics so
retrieval can use them.
**Why structured tool arguments rather than rationale**: the pre-tool text
block is optional by API design and can be suppressed by a forced retry. Tool
schemas are the provider-enforced structured-output boundary, so requiring
purpose and expected outcome there makes them available for every submitted
device action without relying on free-form rationale.
**Why strip metadata before execution**: device tool functions only accept
their physical-action arguments. Keeping metadata off `step.args` preserves
their API contracts while still making it available to execution history,
verification, Cloud audit, and skill synthesis.
## Risks / Trade-offs
- **AI may not always output a text block**: even with `tool_choice: "auto"` (D8), the model is not guaranteed to prefix a text block before the tool call. When absent, `text_output` is `None` and `rationale` is `None`. History degrades gracefully to `{page, rationale: null, action, success}`.
- **`tool_choice: "auto"` occasionally yields no tool call at all**: unlike forced `tool_choice`, `"auto"` permits the model to respond with text only and no tool call. D8's forced retry (no thinking, no rationale on that path) guards this case so a step never stalls; this trades away rationale/thinking for that single step, not overall reliability.
- **Extended thinking increases latency**: `budget_tokens` directly adds to minimum response time. This is opt-in and accepted by the operator who enables it.
- **`WorldEvent` schema divergence from stored data**: Existing `WorldEvent` instances in memory or serialised timelines lack `rationale`/`thinking`. The `to_dict()` method will emit `null` for these fields; downstream consumers should treat `null` as absent, not as a failure.
- **Cloud-proxy transport never produces thinking/rationale at the client layer**: The proxy returns only `tool_name`/`arguments`. `ToolCallDecision.thinking` and `.text_output` will always be `None` for cloud-transport tasks. The `planner_decision_log` on the cloud side will be populated from the cloud-proxied call itself (D7), which does see the full LLM response.
- **Cloud/Host deployment order**: a new Host requires a Cloud API that returns the additive metadata fields to preserve them locally. The Cloud response models remain nullable so an old peer remains readable during a rolling deployment, but it cannot provide the new semantic records.
## Migration Plan