## Why `runtime/planner.py::Planner` — the Planner half of Milestone 3 (Agent Runtime)'s `agent-runtime` capability, implemented by `apex-agent-mvp` and still the only Planner in the codebase — is a stub: it always returns exactly one hardcoded `describe_screen` step on a task's first call, then `[]` on every call after any step has executed, regardless of the goal or the current `Scene`. `TaskRunner`'s Observe→Plan→Act→Observe loop, `Executor`'s retry/backoff, `TaskContext`'s per-task memory, and the already-implemented `semantic-scene` and `world-model` capabilities are all real and wired end-to-end — every later milestone (Task Memory, Semantic Scene, World Model, Skill Learning, Workflow Orchestration, Multi-Agent Runtime) has been built on top of, or alongside, this stub without ever replacing it. Nothing in the runtime actually decides what to do. This change gives the runtime its first real decision-maker: an LLM-driven Planner that uses native tool/function calling to choose exactly one grounded action per turn, completing what Milestone 3 always intended the Planner role to be. ## What Changes - Add `runtime/ai_planner.py::AIPlanner`, a `Planner` implementation that calls an LLM with native tool/function calling once per `plan()` invocation, translating the model's single chosen tool call into 0 or 1 `PlannedStep`. Single-step, ReAct-style: `TaskRunner.run()` already re-observes and re-plans every iteration, so `AIPlanner` never attempts multi-step lookahead. - Add a **pluggable dual-provider** LLM abstraction (`runtime/tool_calling_client.py`): both Anthropic native tool use and OpenAI function calling are supported, selected via configuration (`AI_PLANNER_PROVIDER`), each forced to return exactly one tool call per turn so a response always resolves to a single, unambiguous decision. - Add a fixed, six-tool tool-face (`runtime/tool_specs.py`): `tap`, `swipe`, `input_text`, `launch_app`, `terminate_app` for action, plus an explicit `finish_task(success, reason)` control tool the model calls to end the task (on success or failure) instead of relying on an ambiguous "no tool call" signal. - **Amend the Perception Boundary invariant** (`docs/CONSTITUTION.md`) with one narrow, explicit exception: the AI `Planner` (only that Planner) may additionally receive the current step's raw screenshot bytes alongside `Scene`, to support vision-grounded action grounding (precise tap/swipe coordinates). No other layer or LLM consumer gains access to raw screenshot bytes; `Scene` remains the only perception artifact everywhere else. - Add configuration (`runtime/planner_config.py`: `AI_PLANNER_ENABLED`, `AI_PLANNER_PROVIDER`, `AI_PLANNER_MODEL`, `AI_PLANNER_TIMEOUT_SECONDS`) defaulting `AI_PLANNER_ENABLED=False`, matching the existing enable/disable convention used by every other LLM-backed capability (`semantic-scene`, `skill-learning`). `runtime/task.py::TaskRunner` builds an `AIPlanner` by default only when enabled; otherwise its existing stub `Planner` behavior is unchanged. - Fix a latent correctness gap in `TaskRunner.run()`'s loop, exposed by giving the Planner a real chance to fail: observe/plan exceptions are now caught per iteration and turned into a `status="failed"` task with a `failure_reason`, instead of propagating uncaught (previously harmless only because the stub Planner never raised). - **BREAKING**: none. Default-disabled; when disabled, `TaskRunner`'s constructed `Planner` and control flow are unchanged from before this change. ## Capabilities ### New Capabilities (none — this change fulfills the Planner role already scoped by Milestone 3's `agent-runtime` capability; see Impact for why no new capability is declared) ### Modified Capabilities - `agent-runtime`: the Planner requirement ("given a goal and the current Scene, produces steps") gains a real, LLM-backed implementation with native tool calling, a defined single-action-per-turn contract, an explicit task-completion/failure signal (`finish_task`), and — as a Planner-only exception to the Perception Boundary — optional access to the current step's screenshot. `agent-runtime`'s base spec was never archived into `openspec/specs/` (a pre-existing gap from `apex-agent-mvp`, out of scope for this change); this change's delta spec below uses `## ADDED Requirements` against that as-yet-unarchived baseline, the same way `semantic-scene-runtime` and `world-model-runtime` each layered their own delta on top of it without attempting to backfill it (see `design.md`). ## Impact - **New files**: `runtime/planner_config.py`, `runtime/tool_specs.py`, `runtime/tool_calling_client.py`, `runtime/planner_prompts.py`, `runtime/ai_planner.py`. - **Modified**: `runtime/planner.py` (base `Planner.plan()` gains an optional `screenshot` parameter, stub behavior unchanged), `runtime/task.py` (`TaskRunner` builds an `AIPlanner` when enabled; generalized its existing `_planner_accepts_world()` reflection helper to also conditionally inject `screenshot`; wrapped the observe+plan step in try/except), `docs/CONSTITUTION.md` (Perception Boundary amendment described above). - **No change** to `api/rest.py` — `TaskRunner()`'s existing zero-argument construction picks up the new behavior automatically once `AI_PLANNER_ENABLED=true` is set in the environment; no new request parameters, no MCP surface change (`api/mcp.py` is a separate, curated tool-handler surface unaffected by this change). - **No new dependencies**: `pyproject.toml` already declares both `anthropic` and `openai` SDKs (the former added for `semantic-scene-runtime`); this change is the first to actually construct an OpenAI client. - **Out of scope**: no `wait`/no-op tool (v1.1 candidate, see `design.md`); no dynamic tool subset selection; no provider/model choice exposed as an API request parameter; no automatic retry of failed LLM calls inside `AIPlanner` (transient failures surface as a failed task; tool-execution retry remains solely `Executor`'s concern, unchanged by this change); no backfill of the missing `openspec/specs/agent-runtime/` base spec.