agentic-mobile-control/openspec/changes/ai-planner-runtime/design.md

## Context

`docs/ROADMAP.md` places the Planner inside Milestone 3 (Agent Runtime):
"Planner/Executor structure, tool execution, retry behavior, and task
runner," already implemented by `apex-agent-mvp`. What `apex-agent-mvp`
actually shipped for the Planner half is `runtime/planner.py::Planner` — a
stub that returns one hardcoded `describe_screen` step, then `[]` forever.
Every later milestone built real capability around this stub (`Executor`
retry/backoff, `TaskContext` memory, `semantic-scene` enrichment,
`world-model` cross-step state) without ever giving the loop a real decision
-maker. This change closes that gap: an `AIPlanner` that uses native LLM
tool/function calling to choose one grounded action per turn, wired into
`TaskRunner` behind a default-off flag so applying this change does not
change any existing task's behavior, cost, or dependencies unless explicitly
enabled.

Two decisions were fixed before design work started (not re-litigated here):
a **pluggable dual-provider** abstraction (Anthropic tool use and OpenAI
function calling, switchable via config) rather than a single hardcoded
provider, and **Scene + screenshot** (vision multimodal) as the Planner's
perception input rather than Scene-only — the latter requires the Constitution
amendment covered in Decision D5.

## Goals / Non-Goals

**Goals:**
- Replace the stub decision logic with a real LLM call that chooses exactly
  one tool (action or `finish_task`) per `plan()` invocation, using the
  current `Scene`, optionally the current screenshot, and recent
  `WorldState` history as input.
- Support both Anthropic and OpenAI as interchangeable providers behind one
  internal `ToolCallingClient` interface, selected by configuration alone.
- Keep the change strictly additive and default-disabled: with
  `AI_PLANNER_ENABLED` unset (or false), `TaskRunner`'s constructed planner,
  control flow, and dependencies are byte-for-byte the same as before this
  change.
- Make task completion and task failure both explicit, model-driven signals
  (`finish_task(success, reason)`) rather than inferring completion from the
  absence of a tool call.
- Preserve every Constitution invariant except the one explicitly amended
  (Perception Boundary), and amend that one as narrowly as the vision
  requirement allows.

**Non-Goals:**
- No multi-step lookahead planning — `AIPlanner.plan()` always returns 0 or 1
  step; `TaskRunner.run()` already re-observes and re-plans every loop
  iteration, so pre-planning multiple steps ahead would only drift from the
  actual screen state.
- No `wait`/no-op tool. Forcing "exactly one tool call per turn" means the
  model cannot express "do nothing this turn" — a real but low-severity gap
  (a spurious action self-corrects next turn once the loop re-observes).
  Candidate for a v1.1 follow-up, not this change.
- No dynamic tool subset per task or app — the six-tool face
  (`ACTION_TOOL_SPECS` + `finish_task`) is fixed.
- No `find_text_on_screen`/`find_icon_on_screen`/`describe_screen_semantic`
  exposed as Planner tools — the per-turn `Scene` JSON already enumerates
  every element's id/type/text/bounds, so a "search for X" tool call would
  only add latency and hallucination surface without adding information the
  model doesn't already have.
- No automatic retry of a failed LLM call inside `AIPlanner` or
  `ToolCallingClient` — a transient failure surfaces as a failed task via the
  `TaskRunner.run()` fix in D7, matching the project's existing convention
  that retry is `Executor`'s concern for tool execution, not the Planner's
  concern for its own decision calls.
- No provider/model selection exposed as an API request parameter — v1 is a
  deployment-time/environment-variable choice only.
- No backfill of `openspec/specs/agent-runtime/`, which does not exist
  because `apex-agent-mvp`'s `agent-runtime` capability was never archived
  into `openspec/specs/`. This is a pre-existing gap unrelated to this
  change; this change's `specs/agent-runtime/spec.md` uses `## ADDED
  Requirements` against that unarchived baseline, the same way
  `semantic-scene-runtime` and `world-model-runtime` each did.

## Decisions

### D1: Single-step, ReAct-style `plan()` — never multi-step
`AIPlanner.plan()` returns a list of 0 or 1 `PlannedStep`. `TaskRunner.run()`
already calls `observer()` then `_plan()` fresh on every loop iteration
(`runtime/task.py`), so a Planner that tried to hand back several steps at
once would either have those extra steps silently ignored by the loop's
current per-iteration contract, or require changing that contract to consume
a queue — both worse than just re-deciding every turn against the freshly
observed `Scene`.
**Alternative considered**: return a short queued plan (e.g. up to 3 steps)
and only re-plan when a step's expectation is violated. Rejected — UI state
can change after any single action (a dialog appears, a keyboard covers an
element), so a queued step is frequently stale by the time it would execute;
single-step keeps every action grounded in the turn's actual `Scene`.

### D2: Explicit `finish_task(success, reason)` control tool, not an implicit "no call" signal
Task completion and task failure are both signaled by the model calling
`finish_task` — never by the model declining to call any tool (which native
tool-calling APIs do not reliably support as a distinguishable "done" state
across both providers) and never by a heuristic on the Planner's side.
`finish_task(success=True)` maps to `AIPlanner.plan()` returning `[]`, which
reuses `TaskRunner.run()`'s existing `if not steps or ...: return
self._complete_task(task)` short-circuit unchanged.  `finish_task(success=
False, reason)` maps to `AIPlanner.plan()` raising `TaskFailedError(reason)`
— reusing `core/errors.py`'s `TaskFailedError`, defined since
`apex-agent-mvp` but never previously constructed anywhere in the codebase.
These two outcomes are deliberately routed through different mechanisms
(empty list vs. exception) rather than both returning `[]` with a status
flag, because `run()`'s short-circuit condition is `if not steps or
self.planner.goal_reached(...)` — a bare empty list on failure would be
silently read as success by that exact line.
**Alternative considered**: signal success via `goal_reached()` returning
`True`. Rejected — see D3.

### D3: `AIPlanner.goal_reached()` always returns `False`
This is a required invariant, not a style choice. `run()` evaluates `if not
steps or self.planner.goal_reached(...)` — the `or` means `goal_reached()` is
still consulted even when `steps` is non-empty. If `goal_reached()` could
ever return `True` on a turn where `plan()` also returned a real action, that
action would be silently discarded and the task would be marked complete one
turn early. Routing every completion/failure signal exclusively through
`finish_task` (D2) removes any reason for `goal_reached()` to do anything;
it is a permanent no-op override, documented as such at the call site.

### D4: Dual-provider abstraction via `Protocol`, forced single-tool-call on both
`runtime/tool_calling_client.py` defines `ToolCallingClient` as a
structural `Protocol` (matching the existing precedent of
`semantic/enricher.py::SemanticLLMClient` and
`skills_learning/embeddings.py::EmbeddingClient` — no ABC is used anywhere in
this codebase for a swappable single-method client), with concrete
`AnthropicToolCallingClient` and `OpenAIToolCallingClient` implementations
selected by `build_client(config)` based on `AI_PLANNER_PROVIDER`. Both
implementations force the API to return exactly one tool call per turn —
Anthropic via `tool_choice={"type": "any", "disable_parallel_tool_use":
True}`, OpenAI via `tool_choice="required"` plus the top-level
`parallel_tool_calls=False` — so a single `ToolCallDecision` can always be
parsed deterministically regardless of provider. Both implementations use
the lazy-import-plus-injectable-transport pattern already established by
`semantic/llm_client.py::AnthropicSemanticClient` (SDK imported only inside a
`_client()` method; constructor accepts an optional `transport` for tests),
and wrap every SDK/network/parse failure into one internal
`ToolCallUnavailable` exception, mirroring `EnrichmentUnavailable`.
**Alternative considered**: a single hardcoded Anthropic-only client (the
simpler default for a first LLM-driven Planner). Explicitly rejected by
product decision before this design was drafted — dual-provider
pluggability is a requirement, not a nice-to-have, for this change.

### D5: Perception Boundary amendment — narrow, Planner-only screenshot exception
The Constitution's Perception Boundary previously stated `Scene` is the
*only* perception artifact the LLM sees. Vision-grounded action selection
(precise tap/swipe coordinates, disambiguating visually-similar elements
that OCR/UI-tree fusion can conflate) requires the Planner to also see the
raw screenshot for the current step — a deliberate product decision, not an
oversight. The amendment is written to be as narrow as that requirement
actually is: it names one consumer (`the runtime-layer AI Planner, and only
that Planner`) and explicitly reaffirms that every other layer/consumer
(`api`, `tools`, `perception`, `storage`, or any other LLM consumer) still
never receives raw screenshot bytes, and that `Scene` itself is still
produced exclusively through `PerceptionProvider`. See "Constitution
Compliance" below for the exact before/after text.
**Alternative considered**: keep `Scene` as the Planner's only input
(Scene-only, no vision). Explicitly rejected by product decision before this
design was drafted, for the reasons above.
**Alternative considered**: broaden the exception to "any runtime-layer LLM
consumer" instead of naming the Planner specifically, anticipating future
vision consumers. Rejected — Change Discipline requires each change to
justify its own scope; a hypothetical future consumer should justify its own
amendment when it exists, not inherit a pre-approved blank check today.

### D6: Default-disabled via `AI_PLANNER_ENABLED`, mirroring existing LLM-feature flags
`PlannerConfig.enabled` defaults to `False`, and `load_config()` reads
`AI_PLANNER_ENABLED` the same way `semantic/config.py`,
`skills_learning/config.py`, and `agents/config.py` gate their own
LLM-backed behavior — off unless explicitly turned on. (`world/config.py`'s
`WorldConfig.enabled` defaults `True`, but that capability makes zero LLM
calls and is not a counterexample to this convention.) `TaskRunner`'s
`_default_planner()` follows the same "self-loaded config, enabled flag
picks the implementation" shape already used for `world_model` and
`on_task_succeeded` (`runtime/task.py`), so no change to `api/rest.py` or any
other caller of `TaskRunner()` is required to preserve current behavior.

### D7: `TaskRunner.run()` gains a try/except around observe+plan
Before this change, `run()` had no error handling around
`self.observer(...)` or `self._plan(...)` — safe only because the stub
`Planner` never raised. `AIPlanner` can raise for real reasons (network
failure, malformed provider response, `TaskFailedError` from D2's
`finish_task(success=False)` path), so `run()` now catches any exception
from that per-iteration observe-then-plan sequence and marks the task
`status="failed"` with `failure_reason=f"{type(exc).__name__}: {exc}"`,
returning immediately. Two failure modes are avoided by doing this
explicitly rather than skipping it: an uncaught exception leaving a
`BackgroundTasks`-invoked task stuck at `status="running"` forever, and a
catch-and-return-`[]` degrade pattern (as used by
`semantic/enricher.py::enrich_scene()`) that would be misread as task
success by the same `if not steps or ...` short-circuit discussed in D2/D3.
**Alternative considered**: copy `enrich_scene()`'s catch-and-degrade-to-None
pattern. Rejected — that pattern is correct for an optional enrichment pass
where the caller has a defined fallback (use the raw `Scene`). The Planner
has no such fallback: if it cannot decide, the task cannot proceed, and
pretending otherwise (via an empty list) means silent, incorrect "success."

### D8: `_planner_accepts_world()` generalized to `_planner_accepts(name)`
`TaskRunner` already used `inspect.signature` reflection to decide whether to
pass a `world=` kwarg to `Planner.plan()`, so that narrow-signature test
Planners (e.g. `NoWorldPlanner`) would not receive an unexpected keyword
argument. This change generalizes that one method to take the parameter name
as an argument and reuses it for both `"world"` and the new `"screenshot"`
kwarg, preserving backward compatibility with any existing `Planner`
subclass that does not declare a `screenshot` parameter (or `**kwargs`).

## Constitution Compliance

- **Device Boundary**: unaffected — this change adds no driver code and does
  not touch `driver/`/`device/`.
- **Tool Boundary**: unaffected — `AIPlanner` calls tools exclusively through
  `PlannedStep` → `Executor`, the same as the stub `Planner`; no new code
  imports a concrete driver or bypasses `tools/`.
- **Perception Boundary**: amended, narrowly, per D5. The invariant that
  `Scene` is the only perception artifact for every consumer *except* the
  named Planner is preserved and made explicit in the same sentence as the
  exception, rather than being loosened globally.
- **Runtime Boundary**: preserved and, in fact, fulfilled by this change —
  the existing text already states "LLM dependencies enter at `runtime`
  through Planner behavior," which is exactly what `AIPlanner` is. The
  Executor remains the only component that calls tools and handles
  tool-execution retries; `AIPlanner` does not call tools directly or
  implement its own retry loop (D7 fails the task instead of retrying).
- **Change Discipline**: the dependency direction (`core -> driver/device ->
  tools -> perception -> storage -> runtime -> api`) is preserved — all new
  code lives in `runtime/`, imports flow downward only (`runtime/ai_planner.py`
  imports from `runtime/`, `core/`; nothing in `core`, `driver`, `device`, or
  `tools` imports anything LLM-related). The new external integration (the
  OpenAI SDK, newly *used* though already declared as a dependency) is added
  at the adapter layer that owns the concern (`runtime/tool_calling_client.py`),
  not at the domain or device boundary.

## Risks / Trade-offs

- **[Risk] Per-step LLM call adds latency and cost to every planning turn**
  → Mitigation: default-disabled (D6); `AI_PLANNER_TIMEOUT_SECONDS` bounds
  worst-case latency; no in-`AIPlanner` retry (D7) means a slow/failing
  provider fails fast instead of compounding delay.
- **[Risk] Hallucinated tool arguments (e.g. tap coordinates outside any
  element's bounds)** → Mitigation: the system prompt instructs the model to
  ground every coordinate in the current turn's `Scene` element bounds (and
  screenshot, when present); a bad tap still degrades gracefully into an
  ordinary failed/retried step through `Executor`'s existing mechanism,
  unchanged by this design.
- **[Risk] Provider wire-format drift (Anthropic/OpenAI SDK or API changes)
  silently breaks request construction or response parsing** → Mitigation:
  all format-specific logic is isolated into small, independently unit-
  tested functions (`_anthropic_tool`, `_openai_tool`,
  `_decision_from_anthropic_response`, `_decision_from_openai_response`)
  against fake transports, so a drift shows up as a specific, localized test
  failure rather than a silent behavior change.
- **[Risk] Two providers could behave inconsistently** (e.g. one honors
  `finish_task` semantics more reliably than the other) → Mitigation: both
  are constrained by the identical `ALL_TOOL_SPECS` JSON Schema and the same
  system prompt; provider-specific behavior differences are a model-quality
  concern to observe via the optional real-integration test, not something
  this design can fully eliminate structurally.
- **[Risk] Perception Boundary amendment could be read as a precedent for
  loosening the invariant further** → Mitigation: the amendment text itself
  names exactly one consumer and restates the "no one else" constraint in
  the same breath (D5); any future consumer needs its own amendment.

## Migration Plan

Purely additive; no data migration:
1. Add the five new `runtime/*` files and the `runtime/planner.py`/
   `runtime/task.py` edits described in Impact. With `AI_PLANNER_ENABLED`
   unset, `TaskRunner()` continues to construct the stub `Planner` exactly as
   before.
2. Amend `docs/CONSTITUTION.md`'s Perception Boundary section (D5).
3. No changes to `storage/`'s timeline format, `api/rest.py`, or `api/mcp.py`.
4. Rollback is simply leaving `AI_PLANNER_ENABLED` unset/false, or reverting
   the changed files; no other capability depends on this one.
5. Enabling in a real environment requires setting `AI_PLANNER_ENABLED=true`,
   `AI_PLANNER_PROVIDER` (`anthropic` or `openai`), and the corresponding
   provider's API key in the process environment (already-existing SDK
   convention, not a new config surface this change introduces).

## Open Questions

- Whether a `wait`/no-op tool is actually needed in practice, or whether the
  self-correcting-next-turn behavior of a spurious action is good enough
  indefinitely — deferred to real usage observation, not decided here.
- Whether `AI_PLANNER_TIMEOUT_SECONDS`'s default (30s) is well-tuned against
  real provider latency for image-bearing requests — needs tuning against
  real usage data once this is enabled somewhere with real traffic; not
  fixed by this design.
- Whether provider/model selection should eventually move from
  environment-variable/deployment-time to a per-task or per-request choice —
  leaning toward "not until a concrete need appears" (YAGNI), consistent
  with `semantic-scene-runtime`'s D4 reasoning for its own model choice.