Compare commits

...
2 Commits
Author SHA1 Message Date
q792602257 a5aeb8889c feat(runtime): add planner reflection history with rationale and thinking
Tests / Test failed: 2, passed: 849
- ToolCallDecision captures thinking blocks and pre-tool text output
- AnthropicToolCallingClient supports optional extended thinking (budget_tokens + beta header)
- PlannedStep carries rationale and thinking from each LLM decision
- WorldEvent replaces scene_summary with rationale/thinking/page fields (backward-compatible)
- AI planner system prompt instructs reflection before each tool call
- _history_summary() emits compact {page, rationale, action, success} dicts
- Cloud DB migration 0011 adds nullable rationale/thinking columns to planner_decision_log
- OpenAI client extracts reasoning_content into thinking field
2026-07-15 12:43:22 +08:00
q792602257 96e403ee47 chore(openspec): archive task execution visibility 2026-07-15 12:10:14 +08:00
43 changed files with 1275 additions and 89 deletions
@@ -41,7 +41,7 @@
- [x] 6.2 Run `uv run --all-packages pytest -m "not integration"` and targeted Cloud API / Host Agent test suites; run the Cloud Console Vitest suite for its touched frontend.
- [x] 6.3 Run Ruff check/format and `compileall` across touched packages.
- [x] 6.4 Run `openspec validate --strict` for this change.
- [ ] 6.5 Manual verification (requires a real Host Agent + Appium/device setup per `docs/MACOS_IPHONE_SETUP.md`): run a real task end-to-end and confirm step progress appears live in the Host Agent console and Cloud Console, and that full step history with screenshots is browsable afterward in the Host Agent console.
- [x] 6.5 Manual verification (requires a real Host Agent + Appium/device setup per `docs/MACOS_IPHONE_SETUP.md`): run a real task end-to-end and confirm step progress appears live in the Host Agent console and Cloud Console, and that full step history with screenshots is browsable afterward in the Host Agent console.
## 7. Real per-step LLM prompt/response recorded locally (D9)
@@ -83,4 +83,4 @@
- [x] 11.4 Remove the standalone Runtime REST service/UI, its package data and dedicated tests, and remove Runtime supervision from Host Agent configuration/supervision with an actionable legacy-config error.
- [x] 11.5 Update operator documentation and OpenSpec artifacts to direct execution inspection to Host Agent `:8765/tasks` and remove port `8000` instructions.
- [x] 11.6 Run focused Host Agent and shared Runtime tests, workspace non-integration tests, Ruff, compileall, and strict OpenSpec validation.
- [ ] 11.7 Manual verification (requires a real Host Agent + Appium/device setup): submit or dispatch a task, then confirm the Host Agent console is the only local execution-history UI and shows the complete retained evidence.
- [x] 11.7 Manual verification (requires a real Host Agent + Appium/device setup): submit or dispatch a task, then confirm the Host Agent console is the only local execution-history UI and shows the complete retained evidence.
@@ -0,0 +1,2 @@
schema: spec-driven
created: 2026-07-15
@@ -0,0 +1,123 @@
## Context
The AI Planner currently invokes one LLM call per step via `AIPlanner.plan()`, returning a `PlannedStep` with action/args. The `ToolCallingClient` extracts only the `tool_use` block from the response; any `thinking` or `text` blocks in the Anthropic response are silently discarded.
`WorldModel._append_history()` records one `WorldEvent` per step, storing the full `scene_summary` (SemanticScene or Scene). `_history_summary()` in `ai_planner.py` serialises these as full JSON for the next prompt, causing rapid token growth. The planner has no mechanism to reflect on whether a prior action achieved its intended effect.
Current state of the pipeline:
```
plan(scene, world) → user_prompt → LLM → [thinking?, text?, tool_use] → only tool_use kept
WorldEvent(scene_summary=<full UI tree>, action, success) → history → next prompt (fat)
```
Target state:
```
plan(scene, world) → user_prompt → LLM → thinking (kept) + text=rationale (kept) + tool_use
WorldEvent(rationale, thinking, current_page, action, success) → compact history → next prompt
```
Constraints:
- `runtime/` package must not import `cloud` or `host_agent` concerns (enforced by existing test).
- `CloudProxyToolCallingClient` lives in `host_agent/` and must remain opaque to this change — it forwards raw prompts to the Cloud API and receives `tool_name`/`arguments` back; thinking/text are not surfaced in that transport path.
- Extended thinking requires Anthropic SDK and is Anthropic-only; must be opt-in and degrade gracefully when not configured.
- `WorldEvent` schema change must be backward-compatible: existing stored events without `rationale`/`thinking` fields must remain readable.
## Goals / Non-Goals
**Goals:**
- Capture AI rationale (pre-tool text reflection) from every planning call where the model outputs one.
- Capture extended thinking when `AI_PLANNER_THINKING_BUDGET_TOKENS` is configured.
- Reduce per-step history token cost by replacing `scene_summary` with compact `{page, rationale, action, success}` in the prompt history format.
- Prompt the AI to evaluate the previous step's outcome before selecting the next action (method C: same LLM call, no extra API round-trip).
- Persist `thinking` and `rationale` in the Cloud-side `planner_decision_log` table alongside existing prompt/tool records.
- Capture OpenAI `reasoning_content` when present (o-series models).
**Non-Goals:**
- Not adding a dedicated reflection LLM call (explicit method B) — method C (pre-tool text block) achieves the intent within the existing call budget.
- Not changing `CloudProxyToolCallingClient` — it transparently proxies and is not aware of thinking/text blocks.
- Not changing the `scene_summary` field in `WorldEvent` for stored/displayed task history in the Console — only the prompt-construction path (`_history_summary()`) is changed.
- Not surfacing thinking/rationale in the Cloud Console UI (that's a follow-on concern).
- Not making `thinking_budget_tokens` configurable per-task at runtime (only via environment variable).
## Decisions
### D1: Method C — reflection within the same LLM call via pre-tool text block
**Decision**: Update `PLANNER_SYSTEM_PROMPT` to instruct the AI to output a short text block before the tool call assessing the previous step's outcome and stating the current step's intent. Extract this text block as `text_output` alongside the `tool_use` block.
**Why over method B (explicit reflection call)**: Method B doubles LLM calls per step (and associated latency/cost). The AI already has the previous action and current scene available; requiring a text block costs one extra sentence in the response, not a round-trip.
**Why over method A (implicit)**: Method A relies on the AI organically reflecting without prompting, which is unreliable. Explicit instruction in the system prompt makes it consistent.
**Constraint**: `tool_choice: {type: "any"}` already allows text blocks before a tool use block. No API parameter changes needed for text output capture.
### D2: Extend `ToolCallDecision` with `thinking` and `text_output`
**Decision**: Add `thinking: str | None = None` and `text_output: str | None = None` to `ToolCallDecision`. `_decision_from_anthropic_response()` iterates all content blocks, collecting the first `thinking` block's text and concatenating any `text` blocks. `_decision_from_openai_response()` extracts `reasoning_content` from the first choice's message when present.
**Why not a separate data structure**: `ToolCallDecision` is the single return type of `decide()`; adding optional fields keeps the interface minimal and backward-compatible for `CloudProxyToolCallingClient` which sets neither field.
### D3: Extended thinking via `thinking_budget_tokens` in `PlannerConfig`
**Decision**: Add `thinking_budget_tokens: int | None = None` to `PlannerConfig`, loaded from `AI_PLANNER_THINKING_BUDGET_TOKENS` env var. When set and provider is `anthropic`, `AnthropicToolCallingClient._create_message()` adds `thinking: {type: "enabled", budget_tokens: N}` and the `interleaved-thinking-2025-05-14` beta header. `max_tokens` must be ≥ `budget_tokens + 1`; the client enforces this by taking `max(max_tokens, budget_tokens + 1)`.
**Why opt-in via env var**: Extended thinking increases latency and cost significantly. Most tasks don't need it. Keeping it off by default preserves existing behaviour exactly.
**Why enforce max_tokens floor**: The Anthropic API rejects requests where `max_tokens ≤ budget_tokens`; silent enforcement avoids a confusing runtime error.
### D4: `PlannedStep` carries `rationale` and `thinking`
**Decision**: Add `rationale: str | None = None` and `thinking: str | None = None` to `PlannedStep`. `AIPlanner.plan()` populates these from `decision.text_output` and `decision.thinking`.
**Why on `PlannedStep`**: `WorldModel._append_history()` receives the `PlannedStep`; this is the natural carrier from the planner layer to the world model layer without adding a new coupling.
### D5: `WorldEvent` gets `rationale` and `thinking`; `scene_summary` becomes optional
**Decision**: `WorldEvent.scene_summary` changes from `SemanticScene | Scene` to `SemanticScene | Scene | None`, defaulting to `None`. New fields `rationale: str | None = None` and `thinking: str | None = None` are added. `_append_history()` in `WorldModel` passes `step.rationale` and `step.thinking` and no longer requires a non-None `scene_summary`.
**Why keep `scene_summary` as optional rather than removing it**: Some callers (e.g., direct non-AI planner paths) still populate it. Keeping it optional is backward-compatible and avoids breaking the field in stored timelines.
**Why `current_page` instead of `scene_summary` in prompt history**: `WorldState.current_page` is a single string (e.g. `"Settings > Account"`), already maintained by `WorldModel`. Including it in the compact history record gives the AI verifiable page-level context to validate the previous step's navigation intent, at near-zero token cost.
### D6: Compact history format in `_history_summary()`
**Decision**: `_history_summary()` in `ai_planner.py` switches from `[event.to_dict() for event in world.history]` to a compact list:
```python
[{
"page": event.page,
"action": event.action,
"rationale": event.rationale,
"success": event.success,
} for event in world.history]
```
`event.page` is added as a field to `WorldEvent` (populated from `WorldState.current_page` at the time of recording).
**Why not expose `thinking` in history**: Thinking blocks are verbose internal reasoning, not summaries. Exposing them in history would re-introduce token bloat. Rationale (the AI's own compact 1–2 sentence reflection) is the right unit for history.
### D7: `planner_decision_log` table extended with `thinking` and `rationale`
**Decision**: Add nullable `thinking TEXT` and `rationale TEXT` columns to `planner_decision_log` via a new Alembic migration. `record_planner_decision()` in `cloud/internal_api/api.py` and `PlannerDecisionRecord` in `cloud/internal_api/models.py` are extended to carry these fields. Existing rows without the columns read as `NULL`.
**Why in the cloud decision log**: The cloud-proxy path receives the full `PlannerDecisionRecord` including `thinking`/`rationale` from the Host Agent. Persisting them completes the audit trail for cloud-transport tasks.
**Why separate columns rather than a JSON blob**: The existing table uses discrete columns for `system_prompt`, `user_prompt`, `tool_name`, `tool_arguments` — consistency favours discrete columns. Both fields are optional (cloud-proxy path only; `direct` transport never produces cloud decision records).
## Risks / Trade-offs
- **AI may not always output a text block**: `tool_choice: {type: "any"}` does not guarantee a text block. When absent, `text_output` is `None` and `rationale` is `None`. History degrades gracefully to `{page, rationale: null, action, success}`. No retries or fallbacks needed.
- **Extended thinking increases latency**: `budget_tokens` directly adds to minimum response time. This is opt-in and accepted by the operator who enables it.
- **`WorldEvent` schema divergence from stored data**: Existing `WorldEvent` instances in memory or serialised timelines lack `rationale`/`thinking`. The `to_dict()` method will emit `null` for these fields; downstream consumers should treat `null` as absent, not as a failure.
- **Cloud-proxy transport never produces thinking/rationale at the client layer**: The proxy returns only `tool_name`/`arguments`. `ToolCallDecision.thinking` and `.text_output` will always be `None` for cloud-transport tasks. The `planner_decision_log` on the cloud side will be populated from the cloud-proxied call itself (D7), which does see the full LLM response.
## Migration Plan
1. Apply Alembic migration for new `planner_decision_log` columns (additive, nullable, no data migration required).
2. Deploy Cloud API with updated `PlannerDecisionRecord` and `record_planner_decision()`.
3. Deploy Host Agent with updated `tool_calling_client`, `planner_config`, `ai_planner`, `planner_prompts`, `world/models`, `world/model`.
4. Rollback: revert Host Agent to prior version (no DB rollback needed — new columns simply go unused).
## Open Questions
- None blocking implementation.
@@ -0,0 +1,41 @@
## Why
The AI planner's execution history currently stores raw `scene_summary` (full UI tree / semantic scene JSON) per step, which bloats context tokens rapidly and conveys no semantic intent. The LLM receives a long list of "tap succeeded" entries with no understanding of why each action was taken or whether it achieved its intended effect, making it prone to repeating mistakes and unable to self-correct mid-task.
## What Changes
- **Planner reflection loop (method C)**: The system prompt is updated to require the AI to output a short text block before each tool call — first evaluating whether the previous step achieved its intended effect, then stating the intent of the current step. This reflection happens within the same LLM call (no extra API round-trip).
- **Rationale capture**: `ToolCallDecision` captures the AI's pre-tool text output (`text_output`) and, when extended thinking is enabled, the thinking block (`thinking`). Both flow through `PlannedStep` into `WorldEvent`.
- **Extended thinking support**: `AnthropicToolCallingClient` gains optional `thinking_budget_tokens` config. When set, the Anthropic API is called with `thinking` enabled (interleaved thinking beta); the thinking block is extracted and stored.
- **OpenAI reasoning capture**: `OpenAIToolCallingClient` extracts `reasoning_content` from responses when present (o-series models).
- **WorldEvent schema change**: `scene_summary` is replaced by `rationale` (from `text_output`) and `thinking` (from thinking block), plus `current_page` from `WorldState` for minimal page-level verification context. This reduces per-step history token cost by an order of magnitude.
- **History format**: `_history_summary()` in `ai_planner.py` switches from full `WorldEvent.to_dict()` to a compact `{page, rationale, action, success}` format.
- **planner_decision_log extension**: The Cloud-side decision log table adds `thinking` and `rationale` columns to persist these fields alongside existing prompt/tool records.
## Capabilities
### New Capabilities
- `planner-reflection-history`: AI planner captures per-step rationale (pre-tool text reflection) and optional thinking (extended thinking block), stored in execution history and surfaced in the Cloud planner decision log.
### Modified Capabilities
- `world-model`: `WorldEvent` schema changes — `scene_summary` becomes optional (backward-compat), new `rationale: str | None` and `thinking: str | None` fields added; `history_summary` format for prompt construction changes to compact representation.
- `agent-runtime`: `PlannedStep` gains `rationale` and `thinking` fields; `ToolCallDecision` gains `text_output` and `thinking` fields.
- `cloud-task-progress-visibility`: `planner_decision_log` table extended with `thinking` and `rationale` columns.
## Impact
- `runtime/tool_calling_client.py``ToolCallDecision`, `AnthropicToolCallingClient`, `OpenAIToolCallingClient`, response parsers
- `runtime/planner.py``PlannedStep`
- `runtime/ai_planner.py``AIPlanner.plan()`, `_history_summary()`
- `runtime/planner_prompts.py``PLANNER_SYSTEM_PROMPT`, `planner_user_prompt`
- `runtime/planner_config.py` — new `thinking_budget_tokens` field
- `world/models.py``WorldEvent`
- `world/model.py``_append_history()`
- `packages/cloud-platform/cloud/db_models.py``planner_decision_log` table
- `packages/cloud-platform/cloud/schema.py` — Alembic migration
- `packages/cloud-platform/cloud/internal_api/api.py``record_planner_decision()`
- `packages/cloud-platform/cloud/internal_api/models.py``PlannerDecisionRecord`
- No changes to `CloudProxyToolCallingClient` — thinking/text are surfaced at the local client layer only; the cloud proxy is transparent to them.
- No new external dependencies; Anthropic extended thinking uses existing SDK via beta header.
@@ -0,0 +1,55 @@
## MODIFIED Requirements
### Requirement: LLM-driven Planner selects exactly one grounded action per turn
The system SHALL provide a Planner implementation that, given a goal, the current Scene, recent task history in compact rationale-based format, and optional prior-step context, uses native LLM tool/function calling to select exactly one action (or the completion signal defined below) per `plan()` invocation, grounding any coordinates in the current turn's Scene element bounds. The Planner SHALL instruct the LLM to output a short text block (rationale) before the tool call, reflecting on the previous step's outcome and stating the current step's intent.
#### Scenario: Planner selects a single action for the current turn
- **WHEN** the AI Planner is invoked with a goal and the current Scene
- **THEN** it returns at most one `PlannedStep`, whose action and arguments come from exactly one tool call chosen by the underlying LLM for that turn
#### Scenario: Planner re-decides every turn from the current Scene
- **WHEN** the AI Planner is invoked again after a prior step has executed
- **THEN** its decision is grounded in the newly observed Scene for that turn, not in coordinates or assumptions carried over from a previous turn
#### Scenario: PlannedStep carries rationale when the LLM outputs a text block
- **WHEN** the LLM response includes a text block before the tool call
- **THEN** the returned `PlannedStep` has a non-None `rationale` containing that text
#### Scenario: PlannedStep carries thinking when extended thinking is enabled and the LLM outputs a thinking block
- **WHEN** extended thinking is enabled and the LLM response includes a thinking block
- **THEN** the returned `PlannedStep` has a non-None `thinking` containing the thinking block text
#### Scenario: PlannedStep carries None rationale when no text block is present
- **WHEN** the LLM response contains only a tool call with no preceding text block
- **THEN** `PlannedStep.rationale` is `None` and the step is returned normally
### Requirement: Pluggable dual-provider tool-calling abstraction
The system SHALL support at least two interchangeable LLM providers (Anthropic native tool use and OpenAI function calling) for the AI Planner's decision calls, selectable via configuration, with both providers constrained to return exactly one tool call per request. The Anthropic client SHALL additionally support optional extended thinking via a configurable `thinking_budget_tokens` value. The OpenAI client SHALL capture `reasoning_content` from responses when present. Independently of provider selection, the system SHALL support at least two transports for making that decision call — direct-to-provider and cloud-proxy — selectable via configuration without requiring any change to `AIPlanner`'s own decision logic.
#### Scenario: Provider selected via configuration
- **WHEN** the AI Planner is configured with a given provider identifier
- **THEN** it constructs and uses the tool-calling client for that provider without requiring any change to `AIPlanner`'s own decision logic
#### Scenario: Provider response resolves to a single decision
- **WHEN** either supported provider returns a response to a tool-calling request
- **THEN** the response is parsed into exactly one tool name and one arguments object, regardless of which provider produced it
#### Scenario: Transport selected via configuration
- **WHEN** the Host Agent is configured with a given transport (direct or cloud-proxy)
- **THEN** `AIPlanner` is constructed with the tool-calling client for that transport, and its own decision logic is unchanged regardless of which transport is in effect
#### Scenario: Cloud-proxy transport is the default
- **WHEN** no transport is explicitly configured
- **THEN** the AI Planner uses the cloud-proxy transport and the Cloud Control Plane's planner-decision endpoint
#### Scenario: Direct transport remains available by explicit configuration
- **WHEN** the Host Agent is configured with the direct transport
- **THEN** the AI Planner uses the direct-to-provider transport with locally configured credentials
#### Scenario: Cloud-proxy transport resolves a decision without a local provider client
- **WHEN** the Host Agent is configured with the cloud-proxy transport
- **THEN** its tool-calling client sends the decision request to the Cloud Control Plane's planner-decision endpoint instead of constructing a local Anthropic or OpenAI SDK client
#### Scenario: Cloud-proxy transport returns None for thinking and text_output
- **WHEN** the Host Agent uses cloud-proxy transport
- **THEN** `ToolCallDecision.thinking` and `ToolCallDecision.text_output` are `None` because the proxy surface does not expose them
@@ -0,0 +1,39 @@
## MODIFIED Requirements
### Requirement: Cloud Console displays a task's full LLM interaction history
Cloud Console SHALL provide a view, for a given task, listing each persisted planner decision in step order, including its full prompt, resulting decision, and — when available — the AI's rationale (pre-tool text reflection) and thinking (extended thinking block), sourced from the Cloud Control Plane's persisted planner-decision log.
#### Scenario: Task has persisted planner decisions with rationale
- **WHEN** an operator opens the LLM interaction history view for a task that has one or more persisted planner decisions with non-null rationale
- **THEN** Cloud Console shows each decision in step order with its prompt, resulting tool call, and rationale text
#### Scenario: Task has persisted planner decisions with thinking
- **WHEN** an operator opens the LLM interaction history view for a task whose planner decisions include a non-null thinking field
- **THEN** Cloud Console shows the thinking content alongside the prompt and tool call for that step
#### Scenario: Task has persisted planner decisions without rationale or thinking
- **WHEN** an operator opens the LLM interaction history view for a task whose decisions have null rationale and null thinking
- **THEN** Cloud Console shows each decision without those fields, without error or placeholder text
#### Scenario: Task's Host used direct-to-provider transport
- **WHEN** an operator opens the LLM interaction history view for a task whose Host used direct-to-provider transport
- **THEN** Cloud Console indicates that no LLM interaction history is available because the Host does not report it, rather than showing an empty history with no explanation
### Requirement: Cloud Control Plane persists rationale and thinking in the planner decision log
The Cloud Control Plane's planner-decision log SHALL store the AI's rationale and thinking fields alongside the existing prompt and tool-call fields for each persisted decision. Both fields SHALL be nullable; absence of either field SHALL NOT prevent a decision record from being stored or queried.
#### Scenario: Decision record includes rationale
- **WHEN** the Host Agent reports a planner decision with a non-null rationale
- **THEN** the persisted `planner_decision_log` row stores that rationale text in the `rationale` column
#### Scenario: Decision record includes thinking
- **WHEN** the Host Agent reports a planner decision with a non-null thinking block
- **THEN** the persisted `planner_decision_log` row stores that thinking text in the `thinking` column
#### Scenario: Decision record has no rationale or thinking
- **WHEN** the Host Agent reports a planner decision with null rationale and null thinking (e.g., cloud-proxy transport where these are not surfaced)
- **THEN** the persisted row stores NULL for both columns without error
#### Scenario: Existing decision records without rationale or thinking remain readable
- **WHEN** the system queries a `planner_decision_log` row created before this migration
- **THEN** both `rationale` and `thinking` read as NULL, and the row is returned normally
@@ -0,0 +1,61 @@
## ADDED Requirements
### Requirement: AI planner outputs a rationale text block before each tool call
The system SHALL instruct the AI planner to output a short text block (1–2 sentences) before each tool call, first assessing whether the previous action achieved its intended effect and then stating the intent of the current action. This reflection SHALL be produced within the same LLM call as the tool selection, without a separate round-trip.
#### Scenario: Planner outputs rationale on a non-first step
- **WHEN** the AI planner is invoked for a step that has at least one prior executed step in history
- **THEN** the planner's response includes a text block that references whether the previous action succeeded and states the intent of the current action, alongside the tool call
#### Scenario: Planner outputs rationale on the first step
- **WHEN** the AI planner is invoked for the first step of a task (no prior history)
- **THEN** the planner's response may include a text block stating the intent of the first action, and the absence of a text block does not constitute a failure
#### Scenario: Absent rationale degrades gracefully
- **WHEN** the AI planner produces a tool call with no preceding text block
- **THEN** the system stores `rationale=None` for that step and continues normally without error or retry
### Requirement: Tool-calling client captures thinking and text output from LLM responses
The system SHALL extract and preserve the AI model's thinking block (when extended thinking is enabled) and any pre-tool text block (rationale) from the raw LLM response, making both available on the `ToolCallDecision` returned from `decide()`.
#### Scenario: Anthropic response contains a thinking block
- **WHEN** the Anthropic client receives a response with a `thinking`-type content block
- **THEN** `ToolCallDecision.thinking` is populated with the text of that thinking block
#### Scenario: Anthropic response contains a text block before the tool call
- **WHEN** the Anthropic client receives a response with a `text`-type content block preceding the `tool_use` block
- **THEN** `ToolCallDecision.text_output` is populated with that text
#### Scenario: OpenAI response contains reasoning content
- **WHEN** the OpenAI client receives a response whose message includes a `reasoning_content` field (o-series models)
- **THEN** `ToolCallDecision.thinking` is populated with that reasoning content
#### Scenario: Neither thinking nor text block is present
- **WHEN** the LLM response contains only a tool call block (no thinking, no text)
- **THEN** `ToolCallDecision.thinking` and `ToolCallDecision.text_output` are both `None`, and the decision is returned normally
### Requirement: Extended thinking is opt-in via configuration
The system SHALL support enabling Anthropic extended thinking for the AI planner via a `thinking_budget_tokens` configuration value. When not configured, the planner SHALL operate identically to its pre-existing behavior.
#### Scenario: Extended thinking enabled
- **WHEN** `AI_PLANNER_THINKING_BUDGET_TOKENS` is set to a positive integer and the provider is `anthropic`
- **THEN** the Anthropic client includes the thinking parameter in the API request with the configured budget, and the `interleaved-thinking` beta header is sent
#### Scenario: Extended thinking not configured (default)
- **WHEN** `AI_PLANNER_THINKING_BUDGET_TOKENS` is not set
- **THEN** the Anthropic client makes requests without the thinking parameter, identical to prior behavior
#### Scenario: Extended thinking with OpenAI provider
- **WHEN** `AI_PLANNER_THINKING_BUDGET_TOKENS` is set and the provider is `openai`
- **THEN** the OpenAI client does not apply the Anthropic thinking parameter; reasoning content is captured only if the model returns it naturally
### Requirement: Execution history uses compact rationale-based representation
The system SHALL construct the AI planner's history prompt from a compact per-step record containing the page context, rationale, action, and success flag — not the full scene JSON. This compact history SHALL be the sole format used when constructing the `history_summary` passed to `planner_user_prompt`.
#### Scenario: History prompt uses compact format
- **WHEN** `_history_summary()` is called with a `WorldState` that has one or more history entries
- **THEN** each entry in the returned list contains `page`, `rationale`, `action`, and `success` fields only, without any scene element data
#### Scenario: History prompt handles None rationale
- **WHEN** a `WorldEvent` in history has `rationale=None`
- **THEN** the compact history entry for that step includes `"rationale": null` without omitting the field or raising an error
@@ -0,0 +1,36 @@
## MODIFIED Requirements
### Requirement: Bounded history of recent scene/action pairs
The system SHALL maintain `WorldState.history` as a fixed-size, bounded collection of the most recent per-step records, automatically evicting the oldest entry when a new entry is added past the configured bound. Each history record SHALL store the action name, success flag, page context (from `WorldState.current_page` at the time of recording), and optional rationale and thinking fields sourced from the executed `PlannedStep`. The `scene_summary` field SHALL be retained as an optional field for backward compatibility but SHALL NOT be required for new entries.
#### Scenario: WorldState survives across steps within a task
- **WHEN** a task executes multiple steps in sequence
- **THEN** the `WorldState` object associated with the task is the same object (or reflects continuously accumulated updates) across those steps, not reset between steps
#### Scenario: WorldState is scoped to a single task
- **WHEN** two different tasks run (sequentially or concurrently) against the same or different devices
- **THEN** each task has its own independent `WorldState`, and neither task's `WorldState` reflects the other task's app/page/variables/history
#### Scenario: History record includes rationale when planner provides it
- **WHEN** the executed `PlannedStep` carries a non-None `rationale`
- **THEN** the resulting `WorldEvent` stores that rationale string
#### Scenario: History record stores None rationale when planner does not provide one
- **WHEN** the executed `PlannedStep` has `rationale=None`
- **THEN** the resulting `WorldEvent` stores `rationale=None` without error
#### Scenario: History record includes thinking when planner provides it
- **WHEN** the executed `PlannedStep` carries a non-None `thinking`
- **THEN** the resulting `WorldEvent` stores that thinking string
#### Scenario: History record captures current page at time of recording
- **WHEN** a step is appended to history and `WorldState.current_page` is non-None at that moment
- **THEN** `WorldEvent.page` is set to that page value
#### Scenario: History record stores None page when current_page is unavailable
- **WHEN** a step is appended to history and `WorldState.current_page` is None
- **THEN** `WorldEvent.page` is `None` without error
#### Scenario: History size remains bounded
- **WHEN** steps are appended beyond the configured history bound
- **THEN** the oldest entries are evicted so history size never exceeds the bound
@@ -0,0 +1,51 @@
## 1. ToolCallDecision — capture thinking and text output
- [x] 1.1 Add `thinking: str | None = None` and `text_output: str | None = None` fields to `ToolCallDecision` in `runtime/tool_calling_client.py`
- [x] 1.2 Update `_decision_from_anthropic_response()` to iterate all content blocks: collect first `thinking`-type block text into `thinking`, concatenate all `text`-type blocks into `text_output`, continue to `tool_use` as before
- [x] 1.3 Update `_decision_from_openai_response()` to extract `reasoning_content` from the first choice's message into `thinking` when present
- [x] 1.4 Add unit tests: Anthropic response with thinking block → `decision.thinking` populated; Anthropic response with text block → `decision.text_output` populated; response with only `tool_use` → both `None`; OpenAI response with `reasoning_content``decision.thinking` populated
## 2. PlannerConfig — extended thinking configuration
- [x] 2.1 Add `thinking_budget_tokens: int | None = None` to `PlannerConfig` in `runtime/planner_config.py`
- [x] 2.2 Add `AI_PLANNER_THINKING_BUDGET_TOKENS` env var constant and `_parse_thinking_budget()` helper; wire into `load_config()`
- [x] 2.3 Update `AnthropicToolCallingClient._create_message()`: when `thinking_budget_tokens` is set, add `thinking: {type: "enabled", budget_tokens: N}` to kwargs, add `interleaved-thinking-2025-05-14` beta header via `betas` parameter, and enforce `max_tokens >= thinking_budget_tokens + 1`
- [x] 2.4 Add unit tests: `load_config()` with env var set → `thinking_budget_tokens` parsed; `_create_message()` with budget set → kwargs include thinking param and beta header; without budget → no thinking param in kwargs
## 3. PlannedStep — carry rationale and thinking
- [x] 3.1 Add `rationale: str | None = None` and `thinking: str | None = None` to the `PlannedStep` dataclass in `runtime/planner.py`
- [x] 3.2 Update `AIPlanner.plan()` in `runtime/ai_planner.py`: populate `PlannedStep.rationale = decision.text_output` and `PlannedStep.thinking = decision.thinking`
## 4. Planner prompts — reflection instruction
- [x] 4.1 Update `PLANNER_SYSTEM_PROMPT` in `runtime/planner_prompts.py` to instruct the AI to output a short text block (1–2 sentences) before each tool call: first assessing whether the previous action achieved its intended effect (or noting "first step" if no history), then stating the intent of the current action
- [x] 4.2 Verify via prompt review that the instruction is consistent with `tool_choice: {type: "any"}` (text blocks are already allowed before tool_use blocks — no API change needed)
## 5. History format — compact rationale-based representation
- [x] 5.1 Update `_history_summary()` in `runtime/ai_planner.py` to return compact dicts `{page, rationale, action, success}` instead of `[event.to_dict() for event in world.history]`
- [x] 5.2 Add unit test: `_history_summary()` with a populated `WorldState` returns list of compact dicts without scene element data
## 6. WorldEvent — rationale, thinking, page fields
- [x] 6.1 Update `WorldEvent` in `world/models.py`: make `scene_summary` optional (`SemanticScene | Scene | None = None`), add `rationale: str | None = None`, `thinking: str | None = None`, `page: str | None = None`
- [x] 6.2 Update `WorldEvent.to_dict()` to include `rationale`, `thinking`, `page` fields; keep `scene_summary` serialisation for backward compatibility
- [x] 6.3 Update `WorldModel._append_history()` in `world/model.py` to pass `rationale=step.rationale`, `thinking=step.thinking`, `page=world_state.current_page` when constructing `WorldEvent`; `scene_summary` becomes optional (pass `None` or existing value as appropriate)
- [x] 6.4 Add unit tests: `WorldEvent` with rationale/thinking/page → `to_dict()` includes those fields; `WorldEvent` with all-None optional fields → readable without error; `_append_history()` picks up `current_page` from `WorldState` at time of call
## 7. Cloud DB — planner_decision_log extension
- [x] 7.1 Add `thinking` and `rationale` nullable TEXT columns to the `planner_decision_log` table in `packages/cloud-platform/cloud/db_models.py`
- [x] 7.2 Create a new Alembic migration (e.g. `0010_planner_decision_log_reflection`) adding the two nullable columns; verify it applies cleanly on both SQLite and PostgreSQL migration paths
- [x] 7.3 Add `thinking: str | None = None` and `rationale: str | None = None` to `PlannerDecisionRecord` in `packages/cloud-platform/cloud/internal_api/models.py`
- [x] 7.4 Update `record_planner_decision()` in `packages/cloud-platform/cloud/internal_api/api.py` to write `thinking` and `rationale` from the incoming `PlannerDecisionRecord` to the new columns
- [x] 7.5 Add unit tests: decision record with rationale/thinking → both stored; decision record with nulls → stored without error; existing rows queried → both columns read as NULL without error
## 8. End-to-end verification
- [x] 8.1 Run full non-integration test suite (`uv run --all-packages pytest -m "not integration"`) and confirm no regressions
- [x] 8.2 Run `ruff check` and `ruff format --check` on all modified files
- [x] 8.3 Run `python -m compileall` on modified packages
- [x] 8.4 Run `openspec validate --strict --change "planner-reflection-history"` and confirm all artifacts pass validation
- [ ] 8.5 Manual smoke test (optional, requires Host Agent + Appium + iPhone): verify that a live task run produces non-null `rationale` in `WorldEvent` history and that the history prompt passed to the LLM is in compact format
@@ -1,40 +1,102 @@
# host-agent-console-task-pages Specification
## Purpose
Define the authenticated, same-origin task-history views served by the Host
Agent local console.
## Requirements
### Requirement: Host Agent local console exposes step-level status for the current assignment
The Host Agent's local console SHALL display, for its currently executing assignment, the current step index, step status, and a short summary, sourced from the Host Agent's local task metadata store, refreshed on the console's existing polling interval.
The Host Agent's local console SHALL display, for its currently executing
assignment, the current step index, step status, and a short summary, sourced
from the Host Agent's local task metadata store and refreshed on the console's
existing polling interval.
#### Scenario: An assignment is currently executing
- **WHEN** an operator views the Host Agent local console dashboard while an assignment is executing
- **THEN** the dashboard shows the current step index, step status, and a short summary for that assignment, updating on subsequent polls
- **WHEN** an operator views the Host Agent local console dashboard while an
assignment is executing
- **THEN** the dashboard shows the current step index, step status, and a
short summary for that assignment, updating on subsequent polls
#### Scenario: No assignment is currently executing
- **WHEN** an operator views the dashboard while the Host Agent is idle
- **THEN** the dashboard shows no in-progress step information
### Requirement: Host Agent local console exposes read-only task history with per-step detail and screenshots
The Host Agent's local console SHALL provide authenticated, read-only pages listing recently executed tasks and, for a selected task, its full per-step history including any captured screenshots, sourced from the Host Agent's local task metadata store and timeline.
#### Scenario: Operator lists recent tasks
- **WHEN** an authenticated operator opens the Host Agent local console's task list page
- **THEN** it shows tasks from the local task metadata store, most recent first, including tasks that have already reached a terminal state
The Host Agent's local console SHALL provide authenticated, read-only pages
listing recently executed local tasks and, for a selected task, its full
per-step history from the Host-local metadata store and Timeline. The task
detail SHALL show available before and after screenshots, operation details and
arguments, execution result, OCR observations, and normalized UI-tree results.
It SHALL render legacy Timeline records that only have a single screenshot as
a post-action image.
#### Scenario: Operator lists recent Host executions
- **WHEN** an authenticated operator opens the Host Agent local console's task
list page
- **THEN** it shows local executions most recent first, including terminal
tasks and any available Cloud task ID and attempt correlation
#### Scenario: Operator inspects a completed task's step history
- **WHEN** an authenticated operator opens the detail page for a specific completed task
- **THEN** the page shows each recorded step in order, including its tool call, result, and any captured screenshot
- **WHEN** an authenticated operator opens the detail page for a completed
Host execution
- **THEN** the page shows each recorded step in order with its tool call,
result, and available before/after screenshots
#### Scenario: OCR was captured for a step
- **WHEN** the selected Timeline record contains OCR observations
- **THEN** the detail page shows each observation's text, confidence, and
bounds
#### Scenario: A UI-tree tool returned normalized nodes
- **WHEN** the selected Timeline record invoked `get_ui_tree` or `ui_tree`
and its result contains normalized nodes
- **THEN** the detail page exposes a structured, collapsible node view while
retaining the persisted result JSON
#### Scenario: A legacy Timeline record is displayed
- **WHEN** a Timeline record has only `screenshot_path`
- **THEN** the detail page renders it as the post-action image without failing
#### Scenario: Unauthenticated request
- **WHEN** a request to the task list or task detail pages is made without a valid Host Agent console session
- **THEN** the Host Agent rejects the request the same way it rejects unauthenticated requests to its other console pages
- **WHEN** a request to the task list or task detail pages is made without a
valid Host Agent console session
- **THEN** the Host Agent rejects the request the same way it rejects
unauthenticated requests to its other console pages
### Requirement: Host Agent local console task pages require no new cross-origin surface
The Host Agent local console's task pages SHALL be served same-origin from the Host Agent's existing web application, without introducing new CORS allowances or a dependency on the separate Runtime `console/` frontend.
The Host Agent local console's task pages SHALL be served same-origin from the
Host Agent's existing web application, without introducing new CORS allowances
or a dependency on a separate Runtime frontend.
#### Scenario: Task pages are requested
- **WHEN** an operator's browser requests the Host Agent local console's task pages
- **THEN** the pages are served by the Host Agent's own application using its existing session/CSRF protections, with no additional cross-origin configuration required
- **WHEN** an operator's browser requests the Host Agent local console's task
pages
- **THEN** the pages are served by the Host Agent's own application using its
existing session/CSRF protections, with no additional cross-origin
configuration required
### Requirement: Host Agent console is the authority for actual execution evidence
The Host Agent local console SHALL be the web authority for task evidence
produced by that Host's in-process execution path. A standalone Runtime
service/UI SHALL NOT be required or consulted to inspect a Host execution.
#### Scenario: A Cloud task is executed by a Host Agent
- **WHEN** an operator opens that Host Agent's task page after execution starts
- **THEN** the page reads the same Host-local metadata and Timeline that the
executing `TaskRunner` writes
@@ -1,64 +1,136 @@
# host-agent-dependency-supervisor Specification
## Purpose
Define how the Host Agent optionally supervises its own dependencies (Appium and the local Runtime API), including opt-in activation, adoption of already-running healthy instances, bounded-restart lifecycle for spawned processes, and cleanup tied to the Host Agent's own process lifecycle.
Define optional Appium supervision by the Host Agent, including opt-in
activation, adoption of healthy instances, bounded restart behavior, and
cleanup tied to the Host Agent lifecycle.
## Requirements
### Requirement: Supervisor is opt-in and disabled by default
The Host Agent SHALL NOT start, adopt-check, or supervise Appium or the local Runtime API unless `HOST_AGENT_DEPENDENCY_SUPERVISOR_ENABLED` is explicitly set to true. Each of the two dependencies SHALL additionally have its own independent enable flag (`HOST_AGENT_APPIUM_SUPERVISED`, `HOST_AGENT_RUNTIME_SUPERVISED`), both defaulting to false.
The Host Agent SHALL NOT start, adopt-check, or supervise Appium unless
`HOST_AGENT_DEPENDENCY_SUPERVISOR_ENABLED` is explicitly set to true. Appium
supervision SHALL additionally require `HOST_AGENT_APPIUM_SUPERVISED=true`
and SHALL default to false.
#### Scenario: Default configuration behaves exactly as before
- **WHEN** a Host Agent starts with no `HOST_AGENT_DEPENDENCY_SUPERVISOR_ENABLED` (or any related) environment variable set
- **THEN** the Host Agent does not attempt to connect to, probe, or spawn Appium or the Runtime API, and its heartbeat/claim behavior is unchanged from before this capability existed
#### Scenario: Top-level flag on, individual dependency flag off
- **WHEN** `HOST_AGENT_DEPENDENCY_SUPERVISOR_ENABLED=true` and `HOST_AGENT_APPIUM_SUPERVISED=false` (Runtime supervised is true)
- **THEN** the Host Agent supervises only the Runtime API and does not probe, adopt, or spawn Appium
- **WHEN** a Host Agent starts with no
`HOST_AGENT_DEPENDENCY_SUPERVISOR_ENABLED` or related Appium environment
variable set
- **THEN** the Host Agent does not attempt to connect to, probe, or spawn
Appium, and its heartbeat/claim behavior is unchanged
### Requirement: Adopt an already-running, healthy dependency instead of spawning a duplicate
Before spawning a supervised dependency, the Host Agent SHALL attempt a TCP connection to its configured host/port, and if something is listening, SHALL perform the dependency-specific health check (Appium: HTTP GET to its status endpoint expecting a successful response; Runtime: HTTP GET to its health endpoint expecting a successful response). If the health check succeeds, the Host Agent SHALL treat the existing process as adopted, SHALL NOT spawn a subprocess for that dependency, and SHALL NOT restart or terminate the adopted process at any point in its lifecycle.
#### Scenario: Top-level flag on and Appium flag off
- **WHEN** `HOST_AGENT_DEPENDENCY_SUPERVISOR_ENABLED=true` and
`HOST_AGENT_APPIUM_SUPERVISED=false`
- **THEN** the Host Agent does not probe, adopt, or spawn Appium
### Requirement: Adopt an already-running, healthy Appium instance instead of spawning a duplicate
Before spawning supervised Appium, the Host Agent SHALL attempt a TCP
connection to its configured host and port and, if something is listening,
perform the Appium status health check. If the health check succeeds, the Host
Agent SHALL treat the existing process as adopted, SHALL NOT spawn a subprocess
for it, and SHALL NOT restart or terminate it at any point in its lifecycle.
#### Scenario: Appium already running and healthy
- **WHEN** Appium supervision is enabled and a healthy Appium server is already listening on the configured host/port
- **THEN** the Host Agent logs that it adopted the existing instance and does not spawn a new Appium process
- **WHEN** Appium supervision is enabled and a healthy Appium server is already
listening on the configured host and port
- **THEN** the Host Agent logs that it adopted the existing instance and does
not spawn a new Appium process
#### Scenario: Port occupied by something unhealthy or unrelated
- **WHEN** a supervised dependency's port has a listener that does not pass the dependency-specific health check
- **THEN** the Host Agent logs an error identifying the port conflict for that dependency and does not spawn a subprocess for it, and does not treat the dependency as available
### Requirement: Spawn supervised dependencies that are not already running
When a dependency is enabled for supervision and no healthy instance is adopted, the Host Agent SHALL spawn it as a child process (Appium via `appium --address <host> --port <port>`; Runtime API via its existing `uvicorn api.rest:create_app --factory` entry point with the configured host/port), and SHALL forward the child process's stdout/stderr into the Host Agent's own logging, tagged by dependency name.
- **WHEN** the configured Appium port has a listener that does not pass the
health check
- **THEN** the Host Agent logs an error identifying the port conflict, does
not spawn a subprocess, and does not treat Appium as available
#### Scenario: Neither dependency is running at Host Agent startup
- **WHEN** both Appium and Runtime supervision are enabled and neither has a healthy instance already listening
- **THEN** the Host Agent spawns both as child processes before proceeding to its first device-connect attempt, and both processes' output is visible in the Host Agent's logs
### Requirement: Spawn supervised Appium when it is not already running
The Host Agent SHALL spawn Appium as a child process when Appium supervision is
enabled and no healthy Appium instance is adopted, via
`appium --address <host> --port <port>`, and SHALL forward the child
process's stdout/stderr into the Host Agent's own logging, tagged by dependency
name.
#### Scenario: Appium is not running at Host Agent startup
- **WHEN** Appium supervision is enabled and no healthy Appium instance is
already listening
- **THEN** the Host Agent spawns Appium before proceeding to its first
device-connect attempt, and its output is visible in Host Agent logs
#### Scenario: Spawn fails because the executable is missing
- **WHEN** the Host Agent attempts to spawn Appium but `appium` is not found on `PATH`
- **THEN** the Host Agent logs a dependency-supervisor-specific startup error naming the missing dependency, distinct from a runtime crash of an already-started process
### Requirement: Restart only processes the supervisor itself spawned, with bounded backoff
The Host Agent SHALL restart a supervised dependency automatically only if the Host Agent's own child process handle for it exits unexpectedly. Restart attempts SHALL use capped exponential backoff and SHALL stop permanently for that dependency, for the remaining lifetime of the current Host Agent process, once a configured maximum attempt count (`HOST_AGENT_DEPENDENCY_RESTART_MAX_ATTEMPTS`) is reached. The Host Agent SHALL NOT restart or terminate a dependency instance it adopted rather than spawned.
- **WHEN** the Host Agent attempts to spawn Appium but `appium` is not found
on `PATH`
- **THEN** the Host Agent logs a dependency-supervisor-specific startup error
naming the missing dependency, distinct from a runtime crash of an
already-started process
### Requirement: Restart only Appium processes the supervisor itself spawned, with bounded backoff
The Host Agent SHALL restart supervised Appium automatically only if its own
child process handle exits unexpectedly. Restart attempts SHALL use capped
exponential backoff and SHALL stop permanently for the remaining Host Agent
process lifetime once `HOST_AGENT_DEPENDENCY_RESTART_MAX_ATTEMPTS` is
reached. The Host Agent SHALL NOT restart or terminate an Appium instance it
adopted rather than spawned.
#### Scenario: Spawned Appium process crashes
- **WHEN** a Host Agent-spawned Appium child process exits unexpectedly and the per-dependency restart attempt count is below the configured maximum
- **THEN** the Host Agent waits the current backoff interval and attempts to spawn Appium again
- **WHEN** a Host Agent-spawned Appium child process exits unexpectedly and
the restart attempt count is below the configured maximum
- **THEN** the Host Agent waits the current backoff interval and attempts to
spawn Appium again
#### Scenario: Restart attempts exhausted
- **WHEN** a supervised dependency has crashed and been restarted until reaching `HOST_AGENT_DEPENDENCY_RESTART_MAX_ATTEMPTS`
- **THEN** the Host Agent logs that it has given up restarting that dependency and does not attempt to spawn it again for the rest of the current process lifetime
#### Scenario: Adopted process exits
- **WHEN** a dependency instance the Host Agent adopted (did not spawn) stops running
- **THEN** the Host Agent does not attempt to restart it, since it never held a child process handle for it
- **WHEN** a supervised Appium process has crashed and been restarted until
reaching `HOST_AGENT_DEPENDENCY_RESTART_MAX_ATTEMPTS`
- **THEN** the Host Agent logs that it has given up restarting Appium and does
not attempt to spawn it again for the rest of the current process lifetime
#### Scenario: Adopted Appium exits
- **WHEN** an Appium instance the Host Agent adopted stops running
- **THEN** the Host Agent does not attempt to restart it, since it never held
a child process handle
### Requirement: Supervisor lifecycle is tied to Host Agent process lifecycle
The Host Agent SHALL start enabled, not-yet-healthy supervised dependencies before beginning its normal device-connect/heartbeat/claim sequence, and SHALL stop any dependency processes it spawned (not ones it adopted) during its own graceful shutdown.
The Host Agent SHALL start enabled, not-yet-healthy supervised Appium before
beginning its normal device-connect, heartbeat, and claim sequence, and SHALL
stop Appium processes it spawned, but not ones it adopted, during graceful
shutdown.
#### Scenario: Host Agent shuts down gracefully
- **WHEN** the Host Agent receives a shutdown signal while it holds a child process handle for a spawned Appium instance
- **THEN** the Host Agent terminates the spawned Appium child process as part of its own shutdown sequence
#### Scenario: Host Agent shuts down while an adopted dependency is running
- **WHEN** the Host Agent shuts down and Appium was adopted (not spawned) rather than spawned by this Host Agent
- **THEN** the adopted Appium process is left running, untouched, after the Host Agent exits
- **WHEN** the Host Agent receives a shutdown signal while it holds a child
process handle for a spawned Appium instance
- **THEN** the Host Agent terminates that Appium child process as part of its
shutdown sequence
#### Scenario: Host Agent shuts down while adopted Appium is running
- **WHEN** the Host Agent shuts down and Appium was adopted rather than spawned
- **THEN** the adopted Appium process is left running, untouched, after the
Host Agent exits
### Requirement: Runtime supervision settings are retired
The Host Agent SHALL reject `HOST_AGENT_RUNTIME_SUPERVISED`,
`HOST_AGENT_RUNTIME_HOST`, and `HOST_AGENT_RUNTIME_PORT` because the
standalone Runtime service no longer exists.
#### Scenario: A legacy Runtime supervision variable is set
- **WHEN** startup configuration includes any removed Runtime supervision
variable
- **THEN** configuration fails with an actionable migration error
+63 -13
View File
@@ -1,35 +1,85 @@
# host-agent-task-progress Specification
## Purpose
Define durable, bounded task and per-step history owned by the Host Agent.
## Requirements
### Requirement: Host Agent records step-level execution detail for its in-process TaskRunner
The Host Agent SHALL construct its in-process `TaskRunner` with a durable metadata store and timeline so that every step transition (status, index, the actual prompt submitted to the LLM for that step, the model's resulting decision, result, and screenshot when captured) is persisted as it happens, rather than discarded when the assignment completes. The persisted prompt SHALL be the prompt actually sent to the LLM for that specific step, not the task's overall goal.
The Host Agent SHALL construct its in-process `TaskRunner` with a durable
metadata store and Timeline. `TaskRunner.run()` SHALL create the task's
metadata row idempotently before its first status update, so every execution
path persists its task status and evidence rather than discarding updates for a
missing row. Every completed step SHALL retain its index, actual per-step LLM
prompt and decision when available, tool call, result, distinct before/after
screenshots when captured, raw OCR observations when available, and normalized
UI-tree result when the invoked tool returned one.
#### Scenario: A goal assignment starts execution
- **WHEN** the Host Agent's `AssignmentExecutor` invokes its `TaskRunner`
- **THEN** the task metadata row exists before the runner records its running
status
#### Scenario: A step completes during goal execution
- **WHEN** the Host Agent's `TaskRunner` completes a step while executing an assigned goal
- **THEN** the step's status, index, actual per-step LLM prompt and response, tool call, result, and any captured screenshot are persisted to the Host Agent's local task metadata store and timeline before the next step begins
- **WHEN** the Host Agent's `TaskRunner` completes a step while executing an
assigned goal
- **THEN** the step's status, index, actual per-step LLM prompt and response,
tool call, result, and available evidence are persisted before the next step
begins
#### Scenario: A workflow creates a planned-goal task
- **WHEN** a `WorkflowRunner` invokes a Host Agent-configured
`TaskRunner` for a planned-goal step
- **THEN** that task is persisted without requiring the workflow caller to
create a metadata row separately
#### Scenario: An assignment finishes
- **WHEN** an assignment reaches a terminal state (succeeded or failed)
- **THEN** its full step history remains queryable from the Host Agent's local store after the in-memory `Task` object is discarded
- **THEN** its full step history remains queryable from the Host Agent's local
store after the in-memory `Task` object is discarded
### Requirement: Host-Agent-local task history is retained within a bounded window
The Host Agent SHALL prune persisted task metadata, timeline records, and associated screenshot artifacts once they exceed a configurable retention window or count, so that indefinite process uptime does not cause unbounded local disk growth.
The Host Agent SHALL prune persisted task metadata, Timeline records, and
associated screenshot artifacts once they exceed a configurable retention
window or count, so that indefinite process uptime does not cause unbounded
local disk growth.
#### Scenario: Retention window is exceeded
- **WHEN** a persisted task's age or position exceeds the configured retention threshold
- **THEN** the Host Agent removes that task's metadata row, timeline records, and screenshot artifacts from local storage
- **WHEN** a persisted task's age or position exceeds the configured retention
threshold
- **THEN** the Host Agent removes that task's metadata row, Timeline records,
and screenshot artifacts from local storage
#### Scenario: Retention has not been exceeded
- **WHEN** a persisted task is within the configured retention threshold
- **THEN** its metadata, timeline records, and screenshot artifacts remain available for query
- **THEN** its metadata, Timeline records, and screenshot artifacts remain
available for query
### Requirement: Host Agent local task storage is isolated from an unrelated local Runtime
The Host Agent SHALL use a configurable, Host-Agent-specific database and artifact path for its task metadata store and timeline, distinct from any local Runtime API's own task storage path, so that the two processes cannot silently collide or share state when run on the same machine.
### Requirement: Host Agent correlates local execution records with Cloud assignments
#### Scenario: Host Agent and local Runtime run on the same machine
- **WHEN** both a Host Agent process and a local Runtime API process run on the same machine with their default configurations
- **THEN** each process reads and writes its own task metadata store and timeline without observing or modifying the other's data
For a Cloud-dispatched goal assignment, the Host Agent SHALL persist the Cloud
task ID and attempt alongside its generated local Runtime task ID before
execution starts. The correlation fields SHALL remain optional and generic in
the shared storage layer.
#### Scenario: A Cloud goal assignment begins
- **WHEN** the Host Agent begins executing a Cloud goal assignment
- **THEN** the local task row records that assignment's Cloud task ID and
attempt
#### Scenario: A task is not Cloud-dispatched
- **WHEN** a shared Runtime caller executes a task without Host/Cloud
assignment context
- **THEN** the task metadata row is created and the optional source
correlation fields remain empty
@@ -0,0 +1,46 @@
# runtime-standalone-service Specification
## Purpose
Define the boundary that keeps Runtime as a shared in-process execution library
rather than a standalone REST service and operator console.
## Requirements
### Requirement: Runtime is not exposed as a standalone REST service or web console
The repository SHALL not ship a standalone Runtime REST application, its
unauthenticated web console, or JSON console routes. The shared Runtime and
storage packages SHALL remain reusable execution libraries for the Host Agent
and other in-process callers.
#### Scenario: An operator needs to inspect a Host-executed task
- **WHEN** an operator needs task evidence for a Host Agent execution
- **THEN** the operator uses the authenticated Host Agent console rather than
starting or querying a separate Runtime service
#### Scenario: A package uses shared Runtime execution
- **WHEN** the Host Agent or another in-process caller creates a
`TaskRunner`
- **THEN** it continues to use the shared Runtime and storage packages without
importing a REST or UI adapter
### Requirement: Host Agent does not supervise a retired Runtime service
The Host Agent SHALL not expose Runtime-supervision configuration or spawn a
Runtime REST subprocess. It MAY continue to optionally supervise Appium.
#### Scenario: Host Agent dependency supervision is enabled
- **WHEN** `HOST_AGENT_DEPENDENCY_SUPERVISOR_ENABLED=true` and Appium
supervision is enabled
- **THEN** the Host Agent probes and supervises Appium only
#### Scenario: A removed Runtime-supervision variable is configured
- **WHEN** a Host Agent configuration includes a removed
`HOST_AGENT_RUNTIME_*` variable
- **THEN** startup fails with a message directing the operator to the Host
Agent console and Appium-only supervision
@@ -0,0 +1,78 @@
# runtime-task-evidence Specification
## Purpose
Define the per-action evidence retained by shared Runtime Timeline records and
rendered by the Host Agent execution-history console.
## Requirements
### Requirement: Runtime persists complete evidence for each executed action
The shared Runtime SHALL persist, for each action it attempts, a screenshot
captured immediately before the executor call, the action description and
arguments, the execution result, and a screenshot captured immediately after
the executor call. Existing Timeline records that contain only the legacy
single screenshot SHALL remain readable, with that screenshot treated as the
post-action image.
#### Scenario: An action succeeds
- **WHEN** the Runtime executes an action for a task
- **THEN** its Timeline record includes distinct before and after screenshots,
action detail, and execution result
#### Scenario: An action fails
- **WHEN** the Runtime executor exhausts its retries for an action
- **THEN** the Timeline record still includes any captured screenshots and the
failure result before the task is marked failed
#### Scenario: A legacy Timeline record is read
- **WHEN** a Timeline record has only the prior `screenshot_path` field
- **THEN** the Runtime exposes it as the post-action screenshot without
failing to render the record
### Requirement: Runtime task evidence retains available OCR observations
The shared Runtime SHALL persist raw OCR observations associated with the scene
used to plan an action when available, without adding duplicate OCR data to the
LLM-facing normalized Scene payload. The Host Agent task-detail UI SHALL render
available OCR text, confidence, and bounds, and SHALL render normally when no
OCR result exists.
#### Scenario: OCR found text while planning an action
- **WHEN** perception produced one or more OCR observations for the action's
planning scene
- **THEN** the corresponding Timeline record includes those observations and
the Host Agent task-detail page displays them
#### Scenario: OCR was unavailable or found no text
- **WHEN** perception yields no OCR observations
- **THEN** the Runtime records the action evidence and the Host Agent task
detail renders without an OCR result list
### Requirement: Runtime task evidence retains UI-tree inspection results
The Runtime SHALL retain a UI-tree inspection result when a step invokes the
existing `get_ui_tree` or `ui_tree` tool and the result contains normalized
nodes. The Host Agent task-detail UI SHALL render those nodes in a structured,
collapsible view while retaining the recorded JSON result. The Runtime SHALL
NOT change the tool response contract or duplicate the result in a separate
persistence field.
#### Scenario: UI-tree inspection succeeds
- **WHEN** a task step uses `get_ui_tree` or `ui_tree` and returns one or
more normalized nodes
- **THEN** the Host Agent task-detail page displays each node's type, visible
text or identifier, bounds, and available confidence
#### Scenario: A non-UI-tree step is displayed
- **WHEN** a task step did not invoke a UI-tree tool
- **THEN** the Host Agent task-detail page does not render an empty UI-tree
section
@@ -155,6 +155,8 @@ class PlannerDecisionLogRow(Base):
tool_name: Mapped[str] = mapped_column(String, nullable=False)
arguments_json: Mapped[str] = mapped_column(Text, nullable=False)
created_at: Mapped[str] = mapped_column(String, nullable=False)
rationale: Mapped[str | None] = mapped_column(Text, nullable=True)
thinking: Mapped[str | None] = mapped_column(Text, nullable=True)
class PluginRow(Base):
@@ -504,6 +504,8 @@ def create_internal_router(
tool_name=decision.tool_name,
arguments_json=json.dumps(decision.arguments),
now=utc_now(),
rationale=getattr(decision, "text_output", None),
thinking=getattr(decision, "thinking", None),
)
logger.info(
"planner-decision request resolved",
@@ -0,0 +1,27 @@
"""Add rationale and thinking columns to planner_decision_log."""
from __future__ import annotations
import sqlalchemy as sa
from alembic import op
revision = "0011_planner_decision_log_reflection"
down_revision = "0010_skill_management"
branch_labels = None
depends_on = None
def upgrade() -> None:
op.add_column(
"planner_decision_log",
sa.Column("rationale", sa.Text(), nullable=True),
)
op.add_column(
"planner_decision_log",
sa.Column("thinking", sa.Text(), nullable=True),
)
def downgrade() -> None:
op.drop_column("planner_decision_log", "thinking")
op.drop_column("planner_decision_log", "rationale")
@@ -134,6 +134,8 @@ class PlannerDecisionRecord:
tool_name: str
arguments_json: str
created_at: datetime
rationale: str | None = None
thinking: str | None = None
class CloudRepository(Protocol):
@@ -518,6 +520,8 @@ class CloudRepository(Protocol):
tool_name: str,
arguments_json: str,
now: datetime,
rationale: str | None = None,
thinking: str | None = None,
) -> int:
"""Insert one planner-decision log row, returning the assigned step_index.
+1 -1
View File
@@ -9,7 +9,7 @@ from alembic.runtime.migration import MigrationContext
from cloud.database import create_database_engine, normalize_database_url
HEAD_REVISION = "0010_skill_management"
HEAD_REVISION = "0011_planner_decision_log_reflection"
class SchemaVersionError(RuntimeError):
@@ -1694,6 +1694,8 @@ class SQLAlchemyCloudRepository:
tool_name: str,
arguments_json: str,
now: datetime,
rationale: str | None = None,
thinking: str | None = None,
) -> int:
with self._sessions.begin() as session:
current_max = session.scalars(
@@ -1714,6 +1716,8 @@ class SQLAlchemyCloudRepository:
tool_name=tool_name,
arguments_json=arguments_json,
created_at=_iso(now),
rationale=rationale,
thinking=thinking,
)
)
session.flush()
@@ -1766,6 +1770,8 @@ class SQLAlchemyCloudRepository:
tool_name=row.tool_name,
arguments_json=row.arguments_json,
created_at=_parse_dt(row.created_at), # type: ignore[arg-type]
rationale=row.rationale,
thinking=row.thinking,
)
for row in rows
]
+11 -1
View File
@@ -60,6 +60,8 @@ class AIPlanner(Planner):
description=f"AI planner: {decision.tool_name}({decision.arguments})",
args=dict(decision.arguments),
prompt=decision.user_prompt or user_prompt,
rationale=decision.text_output,
thinking=decision.thinking,
)
]
@@ -72,4 +74,12 @@ class AIPlanner(Planner):
def _history_summary(world: "WorldState | None") -> list[dict[str, Any]]:
if world is None:
return []
return [event.to_dict() for event in world.history]
return [
{
"page": event.page,
"action": event.action,
"rationale": event.rationale,
"success": event.success,
}
for event in world.history
]
+6
View File
@@ -19,6 +19,12 @@ class PlannedStep:
# The actual prompt sent to the LLM for this step (AI planners only).
# ``None`` for non-LLM planners; TaskRunner falls back to the task goal.
prompt: str | None = None
# Pre-tool text block emitted by the model before the tool call.
# None when the model omits a text block or for non-LLM planners.
rationale: str | None = None
# Extended thinking / reasoning content from the model.
# None when not enabled or not present.
thinking: str | None = None
class Planner:
+13
View File
@@ -15,6 +15,7 @@ ENABLED_ENV = "AI_PLANNER_ENABLED"
PROVIDER_ENV = "AI_PLANNER_PROVIDER"
MODEL_ENV = "AI_PLANNER_MODEL"
TIMEOUT_ENV = "AI_PLANNER_TIMEOUT_SECONDS"
THINKING_BUDGET_ENV = "AI_PLANNER_THINKING_BUDGET_TOKENS"
SUPPORTED_PROVIDERS = frozenset(DEFAULT_MODEL_BY_PROVIDER)
@@ -25,6 +26,7 @@ class PlannerConfig:
provider: str = DEFAULT_PROVIDER
model: str = ""
timeout: float = DEFAULT_TIMEOUT_SECONDS
thinking_budget_tokens: int | None = None
def resolved_model(self) -> str:
return self.model or DEFAULT_MODEL_BY_PROVIDER[self.provider]
@@ -37,6 +39,7 @@ def load_config(env: Mapping[str, str] | None = None) -> PlannerConfig:
provider=_parse_provider(values.get(PROVIDER_ENV)),
model=values.get(MODEL_ENV) or "",
timeout=_parse_timeout(values.get(TIMEOUT_ENV)),
thinking_budget_tokens=_parse_thinking_budget(values.get(THINKING_BUDGET_ENV)),
)
@@ -61,3 +64,13 @@ def _parse_timeout(value: str | None) -> float:
except ValueError:
return DEFAULT_TIMEOUT_SECONDS
return timeout if timeout > 0 else DEFAULT_TIMEOUT_SECONDS
def _parse_thinking_budget(value: str | None) -> int | None:
if value is None:
return None
try:
budget = int(value)
except ValueError:
return None
return budget if budget > 0 else None
+7 -1
View File
@@ -10,7 +10,13 @@ list of UI elements with id, type, text, and pixel bounds), and — when
available a screenshot of the same screen and a short history of recent
actions and their outcomes.
You must call exactly one tool per turn:
Before calling a tool, output a short text block (1-2 sentences):
1. If this is the first step, state what you intend to do and why.
2. Otherwise, first assess whether the previous action achieved its intended
effect based on the current screen, then state the intent of your next action.
Keep this reflection concise and factual.
You must then call exactly one tool:
- One of `tap`, `swipe`, `input_text`, `launch_app`, `terminate_app` to make
progress toward the goal.
- `finish_task` when the goal has been reached, or when it cannot be reached
+50 -15
View File
@@ -30,6 +30,12 @@ class ToolCallDecision:
# CloudProxyToolCallingClient) that don't surface them.
system_prompt: str = ""
user_prompt: str = ""
# Pre-tool text block emitted by the model (rationale / reflection).
# None when the model omits a text block before the tool call.
text_output: str | None = None
# Extended thinking block (Anthropic) or reasoning_content (OpenAI o-series).
# None when not enabled or not present in the response.
thinking: str | None = None
class ToolCallingClient(Protocol):
@@ -53,12 +59,14 @@ class AnthropicToolCallingClient:
max_tokens: int = 1024,
api_key: str | None = None,
base_url: str | None = None,
thinking_budget_tokens: int | None = None,
) -> None:
self.model = model
self._transport = transport
self.max_tokens = max_tokens
self._api_key = api_key
self._base_url = base_url
self._thinking_budget_tokens = thinking_budget_tokens
def decide(
self,
@@ -97,9 +105,14 @@ class AnthropicToolCallingClient:
timeout: float,
) -> Any:
client = self._client()
kwargs = {
budget = self._thinking_budget_tokens
# Enforce max_tokens >= budget + 1 when thinking is enabled.
max_tokens = self.max_tokens
if budget is not None:
max_tokens = max(max_tokens, budget + 1)
kwargs: dict[str, Any] = {
"model": self.model,
"max_tokens": self.max_tokens,
"max_tokens": max_tokens,
"timeout": timeout,
"system": [
{
@@ -117,6 +130,9 @@ class AnthropicToolCallingClient:
"tools": [_anthropic_tool(spec) for spec in tools],
"tool_choice": {"type": "any", "disable_parallel_tool_use": True},
}
if budget is not None:
kwargs["thinking"] = {"type": "enabled", "budget_tokens": budget}
kwargs["betas"] = ["interleaved-thinking-2025-05-14"]
messages = getattr(client, "messages", None)
if messages is not None:
return messages.create(**kwargs)
@@ -232,7 +248,10 @@ def build_client(config: PlannerConfig) -> ToolCallingClient:
model = config.resolved_model()
if config.provider == "openai":
return OpenAIToolCallingClient(model=model)
return AnthropicToolCallingClient(model=model)
return AnthropicToolCallingClient(
model=model,
thinking_budget_tokens=config.thinking_budget_tokens,
)
def _anthropic_content(
@@ -272,19 +291,31 @@ def _decision_from_anthropic_response(
content = _value(response, "content")
if not isinstance(content, list):
raise ValueError("anthropic tool-call response missing content list")
thinking: str | None = None
text_parts: list[str] = []
for block in content:
if _value(block, "type") != "tool_use":
continue
name = _value(block, "name")
arguments = _value(block, "input")
if isinstance(name, str) and isinstance(arguments, dict):
return ToolCallDecision(
tool_name=name,
arguments=arguments,
usage=_anthropic_usage(response),
system_prompt=system_prompt,
user_prompt=user_prompt,
)
block_type = _value(block, "type")
if block_type == "thinking" and thinking is None:
raw = _value(block, "thinking")
if isinstance(raw, str):
thinking = raw
elif block_type == "text":
raw = _value(block, "text")
if isinstance(raw, str):
text_parts.append(raw)
elif block_type == "tool_use":
name = _value(block, "name")
arguments = _value(block, "input")
if isinstance(name, str) and isinstance(arguments, dict):
return ToolCallDecision(
tool_name=name,
arguments=arguments,
usage=_anthropic_usage(response),
system_prompt=system_prompt,
user_prompt=user_prompt,
text_output="\n".join(text_parts) if text_parts else None,
thinking=thinking,
)
raise ValueError("anthropic response did not include a tool_use block")
@@ -333,12 +364,16 @@ def _decision_from_openai_response(
if not isinstance(name, str):
raise ValueError("openai tool call missing a function name")
arguments = _decode_openai_arguments(_value(function, "arguments"))
# Extract reasoning_content from o-series models when present.
raw_reasoning = _value(message, "reasoning_content")
thinking: str | None = raw_reasoning if isinstance(raw_reasoning, str) else None
return ToolCallDecision(
tool_name=name,
arguments=arguments,
usage=_openai_usage(response),
system_prompt=system_prompt,
user_prompt=user_prompt,
thinking=thinking,
)
+59
View File
@@ -195,3 +195,62 @@ def test_ai_planner_step_prompt_reflects_scene_changes() -> None:
assert steps_a[0].prompt != steps_b[0].prompt
assert "Alpha" in steps_a[0].prompt
assert "Beta" in steps_b[0].prompt
def test_ai_planner_propagates_rationale_and_thinking_to_planned_step() -> None:
client = FakeToolCallingClient(
ToolCallDecision(
tool_name="tap",
arguments={"x": 1, "y": 2},
text_output="Previous step opened settings. Now tapping account.",
thinking="I need to navigate deeper.",
)
)
planner = AIPlanner(client=client)
steps = planner.plan(goal="open account", scene=_scene(), context=_context())
assert steps[0].rationale == "Previous step opened settings. Now tapping account."
assert steps[0].thinking == "I need to navigate deeper."
def test_history_summary_returns_compact_format() -> None:
from collections import deque
from runtime.ai_planner import _history_summary
from world.models import WorldEvent, WorldState
state = WorldState(
history=deque(
[
WorldEvent(
action="tap",
success=True,
rationale="Opened settings.",
page="Home",
),
WorldEvent(
action="swipe",
success=False,
rationale=None,
page="Settings",
),
]
)
)
summary = _history_summary(state)
assert summary == [
{"page": "Home", "action": "tap", "rationale": "Opened settings.", "success": True},
{"page": "Settings", "action": "swipe", "rationale": None, "success": False},
]
# Must not contain scene element data
for entry in summary:
assert "scene_summary" not in entry
assert "elements" not in entry
def test_history_summary_returns_empty_for_none_world() -> None:
from runtime.ai_planner import _history_summary
assert _history_summary(None) == []
+59
View File
@@ -1751,3 +1751,62 @@ def test_prune_planner_decision_log_deletes_only_old_terminal_tasks(
)
finally:
database.close()
def test_record_planner_decision_stores_rationale_and_thinking(
database_url: str,
) -> None:
database = CloudDatabase(database_url)
task_id = _unique_id("reflect-task")
host_id = _unique_id("reflect-host")
now = datetime(2026, 7, 15, 0, 0, tzinfo=UTC)
try:
database.repository.record_planner_decision(
host_id=host_id,
task_id=task_id,
attempt=1,
system_prompt="system",
user_prompt="user",
tool_name="tap",
arguments_json='{"x": 1}',
now=now,
rationale="Previous step opened settings. Now tapping account.",
thinking="I need to navigate to account settings.",
)
decisions = database.repository.list_planner_decisions(task_id=task_id, attempt=1)
assert len(decisions) == 1
assert decisions[0].rationale == "Previous step opened settings. Now tapping account."
assert decisions[0].thinking == "I need to navigate to account settings."
finally:
database.close()
def test_record_planner_decision_stores_null_rationale_and_thinking(
database_url: str,
) -> None:
database = CloudDatabase(database_url)
task_id = _unique_id("reflect-null-task")
host_id = _unique_id("reflect-null-host")
now = datetime(2026, 7, 15, 0, 0, tzinfo=UTC)
try:
database.repository.record_planner_decision(
host_id=host_id,
task_id=task_id,
attempt=1,
system_prompt="system",
user_prompt="user",
tool_name="tap",
arguments_json="{}",
now=now,
# rationale and thinking omitted (default None)
)
decisions = database.repository.list_planner_decisions(task_id=task_id, attempt=1)
assert len(decisions) == 1
assert decisions[0].rationale is None
assert decisions[0].thinking is None
finally:
database.close()
+18
View File
@@ -70,3 +70,21 @@ def test_load_config_falls_back_to_default_timeout_when_invalid_or_non_positive(
for value in ["not-a-number", "0", "-5"]:
config = load_config({"AI_PLANNER_TIMEOUT_SECONDS": value, **_NO_RELEVANT_VARS})
assert config.timeout == DEFAULT_TIMEOUT_SECONDS
def test_load_config_parses_thinking_budget_tokens() -> None:
config = load_config({"AI_PLANNER_THINKING_BUDGET_TOKENS": "4096"})
assert config.thinking_budget_tokens == 4096
def test_load_config_thinking_budget_tokens_unset_defaults_to_none() -> None:
config = load_config(_NO_RELEVANT_VARS)
assert config.thinking_budget_tokens is None
def test_load_config_thinking_budget_tokens_invalid_or_non_positive_gives_none() -> None:
for value in ["not-a-number", "0", "-1"]:
config = load_config({"AI_PLANNER_THINKING_BUDGET_TOKENS": value, **_NO_RELEVANT_VARS})
assert config.thinking_budget_tokens is None
+184
View File
@@ -435,3 +435,187 @@ def test_build_client_selects_provider_and_resolves_default_model() -> None:
def test_build_client_honors_explicit_model_override() -> None:
client = build_client(PlannerConfig(provider="openai", model="gpt-5.6-custom"))
assert client.model == "gpt-5.6-custom"
# --- thinking / text_output capture -----------------------------------------
def test_anthropic_client_captures_thinking_block() -> None:
messages = FakeMessages(
response={
"content": [
{"type": "thinking", "thinking": "I should tap the button."},
{"type": "tool_use", "name": "tap", "input": {"x": 10, "y": 20}},
]
}
)
client = AnthropicToolCallingClient(
model="test-model", transport=FakeTransport(messages)
)
decision = client.decide(
system_prompt="s", user_prompt="u", screenshot=None, tools=[TAP_SPEC], timeout=1
)
assert decision.thinking == "I should tap the button."
assert decision.text_output is None
def test_anthropic_client_captures_text_block_as_text_output() -> None:
messages = FakeMessages(
response={
"content": [
{"type": "text", "text": "Previous step succeeded. Now tapping login."},
{"type": "tool_use", "name": "tap", "input": {"x": 5, "y": 5}},
]
}
)
client = AnthropicToolCallingClient(
model="test-model", transport=FakeTransport(messages)
)
decision = client.decide(
system_prompt="s", user_prompt="u", screenshot=None, tools=[TAP_SPEC], timeout=1
)
assert decision.text_output == "Previous step succeeded. Now tapping login."
assert decision.thinking is None
def test_anthropic_client_captures_both_thinking_and_text_output() -> None:
messages = FakeMessages(
response={
"content": [
{"type": "thinking", "thinking": "Deep thought."},
{"type": "text", "text": "Step succeeded. Tapping next."},
{"type": "tool_use", "name": "tap", "input": {"x": 1, "y": 1}},
]
}
)
client = AnthropicToolCallingClient(
model="test-model", transport=FakeTransport(messages)
)
decision = client.decide(
system_prompt="s", user_prompt="u", screenshot=None, tools=[TAP_SPEC], timeout=1
)
assert decision.thinking == "Deep thought."
assert decision.text_output == "Step succeeded. Tapping next."
def test_anthropic_client_tool_only_response_has_none_thinking_and_text_output() -> None:
messages = FakeMessages(
response={
"content": [
{"type": "tool_use", "name": "tap", "input": {"x": 0, "y": 0}},
]
}
)
client = AnthropicToolCallingClient(
model="test-model", transport=FakeTransport(messages)
)
decision = client.decide(
system_prompt="s", user_prompt="u", screenshot=None, tools=[TAP_SPEC], timeout=1
)
assert decision.thinking is None
assert decision.text_output is None
def test_openai_client_captures_reasoning_content() -> None:
completions = FakeCompletions(
response={
"choices": [
{
"message": {
"reasoning_content": "I reasoned about this step.",
"tool_calls": [
{"function": {"name": "tap", "arguments": '{"x": 1, "y": 2}'}}
],
}
}
]
}
)
client = OpenAIToolCallingClient(
model="test-model", transport=FakeOpenAITransport(completions)
)
decision = client.decide(
system_prompt="s", user_prompt="u", screenshot=None, tools=[TAP_SPEC], timeout=1
)
assert decision.thinking == "I reasoned about this step."
assert decision.text_output is None
def test_openai_client_no_reasoning_content_gives_none_thinking() -> None:
completions = FakeCompletions(
response={
"choices": [
{
"message": {
"tool_calls": [
{"function": {"name": "tap", "arguments": '{"x": 1, "y": 2}'}}
]
}
}
]
}
)
client = OpenAIToolCallingClient(
model="test-model", transport=FakeOpenAITransport(completions)
)
decision = client.decide(
system_prompt="s", user_prompt="u", screenshot=None, tools=[TAP_SPEC], timeout=1
)
assert decision.thinking is None
def test_anthropic_client_sends_thinking_param_when_budget_set() -> None:
messages = FakeMessages(
response={
"content": [{"type": "tool_use", "name": "tap", "input": {"x": 1, "y": 2}}]
}
)
client = AnthropicToolCallingClient(
model="test-model",
transport=FakeTransport(messages),
thinking_budget_tokens=2048,
)
client.decide(
system_prompt="s", user_prompt="u", screenshot=None, tools=[TAP_SPEC], timeout=1
)
call = messages.calls[0]
assert call["thinking"] == {"type": "enabled", "budget_tokens": 2048}
assert "interleaved-thinking-2025-05-14" in call["betas"]
# max_tokens must be >= budget + 1
assert call["max_tokens"] >= 2049
def test_anthropic_client_enforces_max_tokens_floor_for_thinking() -> None:
messages = FakeMessages(
response={
"content": [{"type": "tool_use", "name": "tap", "input": {"x": 1, "y": 2}}]
}
)
# max_tokens=1024, budget=4096 → max_tokens should be raised to 4097
client = AnthropicToolCallingClient(
model="test-model",
transport=FakeTransport(messages),
max_tokens=1024,
thinking_budget_tokens=4096,
)
client.decide(
system_prompt="s", user_prompt="u", screenshot=None, tools=[TAP_SPEC], timeout=1
)
assert messages.calls[0]["max_tokens"] == 4097
def test_anthropic_client_no_thinking_param_when_budget_not_set() -> None:
messages = FakeMessages(
response={
"content": [{"type": "tool_use", "name": "tap", "input": {"x": 1, "y": 2}}]
}
)
client = AnthropicToolCallingClient(
model="test-model", transport=FakeTransport(messages)
)
client.decide(
system_prompt="s", user_prompt="u", screenshot=None, tools=[TAP_SPEC], timeout=1
)
call = messages.calls[0]
assert "thinking" not in call
assert "betas" not in call
+32 -4
View File
@@ -27,9 +27,9 @@ def test_world_event_and_state_to_dict() -> None:
widgets=[SemanticWidget(element_id="send", purpose="send message")],
)
event = WorldEvent(
scene_summary=semantic_scene,
action="tap",
success=True,
scene_summary=semantic_scene,
)
state = WorldState.with_history_bound(2)
state.current_app = "com.example.chat"
@@ -51,10 +51,38 @@ def test_world_event_and_state_to_dict() -> None:
def test_world_state_history_evicts_oldest_entry_at_bound() -> None:
state = WorldState.with_history_bound(2)
state.history.append(WorldEvent(_scene(), "first", True))
state.history.append(WorldEvent(_scene(), "second", True))
state.history.append(WorldEvent(_scene(), "third", True))
state.history.append(WorldEvent(action="first", success=True))
state.history.append(WorldEvent(action="second", success=True))
state.history.append(WorldEvent(action="third", success=True))
assert len(state.history) == 2
assert [event.action for event in state.history] == ["second", "third"]
assert state.history.maxlen == 2
def test_world_event_with_rationale_thinking_page() -> None:
event = WorldEvent(
action="tap",
success=True,
rationale="Previous step opened settings. Now tapping account.",
thinking="I should navigate to account settings.",
page="Settings",
)
data = event.to_dict()
assert data["rationale"] == "Previous step opened settings. Now tapping account."
assert data["thinking"] == "I should navigate to account settings."
assert data["page"] == "Settings"
assert data["scene_summary"] is None
def test_world_event_all_optional_fields_none() -> None:
event = WorldEvent(action="swipe", success=False)
data = event.to_dict()
assert data["rationale"] is None
assert data["thinking"] is None
assert data["page"] is None
assert data["scene_summary"] is None
assert data["action"] == "swipe"
assert data["success"] is False
+4 -1
View File
@@ -135,9 +135,12 @@ class WorldModel:
) -> None:
state.history.append(
WorldEvent(
scene_summary=semantic_scene or scene,
action=str(getattr(step, "action", "unknown")),
success=bool(getattr(result, "success", False)),
scene_summary=semantic_scene or scene,
rationale=getattr(step, "rationale", None),
thinking=getattr(step, "thinking", None),
page=state.current_page,
)
)
+10 -2
View File
@@ -12,16 +12,24 @@ from world.config import DEFAULT_HISTORY_SIZE
@dataclass(frozen=True)
class WorldEvent:
scene_summary: SemanticScene | Scene
action: str
success: bool
# Optional for backward compatibility; new entries created by AIPlanner
# paths leave this as None and use rationale/thinking instead.
scene_summary: SemanticScene | Scene | None = None
rationale: str | None = None
thinking: str | None = None
page: str | None = None
timestamp: datetime = field(default_factory=utc_now)
def to_dict(self) -> dict[str, Any]:
return {
"scene_summary": self.scene_summary.to_dict(),
"scene_summary": self.scene_summary.to_dict() if self.scene_summary is not None else None,
"action": self.action,
"success": self.success,
"rationale": self.rationale,
"thinking": self.thinking,
"page": self.page,
"timestamp": self.timestamp.isoformat(),
}