feat(agent-runtime): add LLM-driven AI Planner with dual-provider tool calling
Replaces the stub Planner's fixed describe_screen/[] behavior with a real decision-maker: AIPlanner uses native tool/function calling (Anthropic or OpenAI, pluggable via AI_PLANNER_PROVIDER) to select exactly one grounded action per turn, with an explicit finish_task(success, reason) tool for completion/failure instead of an ambiguous "no tool call" signal. Default disabled (AI_PLANNER_ENABLED=false) and additive; TaskRunner falls back to the existing stub Planner unchanged when disabled. Amends CONSTITUTION.md's Perception Boundary with one narrow exception: only the AI Planner may receive the current step's raw screenshot bytes alongside Scene, for vision-grounded coordinate grounding. Also fixes a latent gap in TaskRunner.run(): observe/plan exceptions are now caught per iteration and turned into a failed task with a failure_reason, instead of propagating uncaught. openspec change: ai-planner-runtime. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
Claude Sonnet 5
@@ -0,0 +1,2 @@
|
||||
schema: spec-driven
|
||||
created: 2026-07-12
|
||||
@@ -0,0 +1,289 @@
|
||||
## Context
|
||||
|
||||
`docs/ROADMAP.md` places the Planner inside Milestone 3 (Agent Runtime):
|
||||
"Planner/Executor structure, tool execution, retry behavior, and task
|
||||
runner," already implemented by `apex-agent-mvp`. What `apex-agent-mvp`
|
||||
actually shipped for the Planner half is `runtime/planner.py::Planner` — a
|
||||
stub that returns one hardcoded `describe_screen` step, then `[]` forever.
|
||||
Every later milestone built real capability around this stub (`Executor`
|
||||
retry/backoff, `TaskContext` memory, `semantic-scene` enrichment,
|
||||
`world-model` cross-step state) without ever giving the loop a real decision
|
||||
-maker. This change closes that gap: an `AIPlanner` that uses native LLM
|
||||
tool/function calling to choose one grounded action per turn, wired into
|
||||
`TaskRunner` behind a default-off flag so applying this change does not
|
||||
change any existing task's behavior, cost, or dependencies unless explicitly
|
||||
enabled.
|
||||
|
||||
Two decisions were fixed before design work started (not re-litigated here):
|
||||
a **pluggable dual-provider** abstraction (Anthropic tool use and OpenAI
|
||||
function calling, switchable via config) rather than a single hardcoded
|
||||
provider, and **Scene + screenshot** (vision multimodal) as the Planner's
|
||||
perception input rather than Scene-only — the latter requires the Constitution
|
||||
amendment covered in Decision D5.
|
||||
|
||||
## Goals / Non-Goals
|
||||
|
||||
**Goals:**
|
||||
- Replace the stub decision logic with a real LLM call that chooses exactly
|
||||
one tool (action or `finish_task`) per `plan()` invocation, using the
|
||||
current `Scene`, optionally the current screenshot, and recent
|
||||
`WorldState` history as input.
|
||||
- Support both Anthropic and OpenAI as interchangeable providers behind one
|
||||
internal `ToolCallingClient` interface, selected by configuration alone.
|
||||
- Keep the change strictly additive and default-disabled: with
|
||||
`AI_PLANNER_ENABLED` unset (or false), `TaskRunner`'s constructed planner,
|
||||
control flow, and dependencies are byte-for-byte the same as before this
|
||||
change.
|
||||
- Make task completion and task failure both explicit, model-driven signals
|
||||
(`finish_task(success, reason)`) rather than inferring completion from the
|
||||
absence of a tool call.
|
||||
- Preserve every Constitution invariant except the one explicitly amended
|
||||
(Perception Boundary), and amend that one as narrowly as the vision
|
||||
requirement allows.
|
||||
|
||||
**Non-Goals:**
|
||||
- No multi-step lookahead planning — `AIPlanner.plan()` always returns 0 or 1
|
||||
step; `TaskRunner.run()` already re-observes and re-plans every loop
|
||||
iteration, so pre-planning multiple steps ahead would only drift from the
|
||||
actual screen state.
|
||||
- No `wait`/no-op tool. Forcing "exactly one tool call per turn" means the
|
||||
model cannot express "do nothing this turn" — a real but low-severity gap
|
||||
(a spurious action self-corrects next turn once the loop re-observes).
|
||||
Candidate for a v1.1 follow-up, not this change.
|
||||
- No dynamic tool subset per task or app — the six-tool face
|
||||
(`ACTION_TOOL_SPECS` + `finish_task`) is fixed.
|
||||
- No `find_text_on_screen`/`find_icon_on_screen`/`describe_screen_semantic`
|
||||
exposed as Planner tools — the per-turn `Scene` JSON already enumerates
|
||||
every element's id/type/text/bounds, so a "search for X" tool call would
|
||||
only add latency and hallucination surface without adding information the
|
||||
model doesn't already have.
|
||||
- No automatic retry of a failed LLM call inside `AIPlanner` or
|
||||
`ToolCallingClient` — a transient failure surfaces as a failed task via the
|
||||
`TaskRunner.run()` fix in D7, matching the project's existing convention
|
||||
that retry is `Executor`'s concern for tool execution, not the Planner's
|
||||
concern for its own decision calls.
|
||||
- No provider/model selection exposed as an API request parameter — v1 is a
|
||||
deployment-time/environment-variable choice only.
|
||||
- No backfill of `openspec/specs/agent-runtime/`, which does not exist
|
||||
because `apex-agent-mvp`'s `agent-runtime` capability was never archived
|
||||
into `openspec/specs/`. This is a pre-existing gap unrelated to this
|
||||
change; this change's `specs/agent-runtime/spec.md` uses `## ADDED
|
||||
Requirements` against that unarchived baseline, the same way
|
||||
`semantic-scene-runtime` and `world-model-runtime` each did.
|
||||
|
||||
## Decisions
|
||||
|
||||
### D1: Single-step, ReAct-style `plan()` — never multi-step
|
||||
`AIPlanner.plan()` returns a list of 0 or 1 `PlannedStep`. `TaskRunner.run()`
|
||||
already calls `observer()` then `_plan()` fresh on every loop iteration
|
||||
(`runtime/task.py`), so a Planner that tried to hand back several steps at
|
||||
once would either have those extra steps silently ignored by the loop's
|
||||
current per-iteration contract, or require changing that contract to consume
|
||||
a queue — both worse than just re-deciding every turn against the freshly
|
||||
observed `Scene`.
|
||||
**Alternative considered**: return a short queued plan (e.g. up to 3 steps)
|
||||
and only re-plan when a step's expectation is violated. Rejected — UI state
|
||||
can change after any single action (a dialog appears, a keyboard covers an
|
||||
element), so a queued step is frequently stale by the time it would execute;
|
||||
single-step keeps every action grounded in the turn's actual `Scene`.
|
||||
|
||||
### D2: Explicit `finish_task(success, reason)` control tool, not an implicit "no call" signal
|
||||
Task completion and task failure are both signaled by the model calling
|
||||
`finish_task` — never by the model declining to call any tool (which native
|
||||
tool-calling APIs do not reliably support as a distinguishable "done" state
|
||||
across both providers) and never by a heuristic on the Planner's side.
|
||||
`finish_task(success=True)` maps to `AIPlanner.plan()` returning `[]`, which
|
||||
reuses `TaskRunner.run()`'s existing `if not steps or ...: return
|
||||
self._complete_task(task)` short-circuit unchanged. `finish_task(success=
|
||||
False, reason)` maps to `AIPlanner.plan()` raising `TaskFailedError(reason)`
|
||||
— reusing `core/errors.py`'s `TaskFailedError`, defined since
|
||||
`apex-agent-mvp` but never previously constructed anywhere in the codebase.
|
||||
These two outcomes are deliberately routed through different mechanisms
|
||||
(empty list vs. exception) rather than both returning `[]` with a status
|
||||
flag, because `run()`'s short-circuit condition is `if not steps or
|
||||
self.planner.goal_reached(...)` — a bare empty list on failure would be
|
||||
silently read as success by that exact line.
|
||||
**Alternative considered**: signal success via `goal_reached()` returning
|
||||
`True`. Rejected — see D3.
|
||||
|
||||
### D3: `AIPlanner.goal_reached()` always returns `False`
|
||||
This is a required invariant, not a style choice. `run()` evaluates `if not
|
||||
steps or self.planner.goal_reached(...)` — the `or` means `goal_reached()` is
|
||||
still consulted even when `steps` is non-empty. If `goal_reached()` could
|
||||
ever return `True` on a turn where `plan()` also returned a real action, that
|
||||
action would be silently discarded and the task would be marked complete one
|
||||
turn early. Routing every completion/failure signal exclusively through
|
||||
`finish_task` (D2) removes any reason for `goal_reached()` to do anything;
|
||||
it is a permanent no-op override, documented as such at the call site.
|
||||
|
||||
### D4: Dual-provider abstraction via `Protocol`, forced single-tool-call on both
|
||||
`runtime/tool_calling_client.py` defines `ToolCallingClient` as a
|
||||
structural `Protocol` (matching the existing precedent of
|
||||
`semantic/enricher.py::SemanticLLMClient` and
|
||||
`skills_learning/embeddings.py::EmbeddingClient` — no ABC is used anywhere in
|
||||
this codebase for a swappable single-method client), with concrete
|
||||
`AnthropicToolCallingClient` and `OpenAIToolCallingClient` implementations
|
||||
selected by `build_client(config)` based on `AI_PLANNER_PROVIDER`. Both
|
||||
implementations force the API to return exactly one tool call per turn —
|
||||
Anthropic via `tool_choice={"type": "any", "disable_parallel_tool_use":
|
||||
True}`, OpenAI via `tool_choice="required"` plus the top-level
|
||||
`parallel_tool_calls=False` — so a single `ToolCallDecision` can always be
|
||||
parsed deterministically regardless of provider. Both implementations use
|
||||
the lazy-import-plus-injectable-transport pattern already established by
|
||||
`semantic/llm_client.py::AnthropicSemanticClient` (SDK imported only inside a
|
||||
`_client()` method; constructor accepts an optional `transport` for tests),
|
||||
and wrap every SDK/network/parse failure into one internal
|
||||
`ToolCallUnavailable` exception, mirroring `EnrichmentUnavailable`.
|
||||
**Alternative considered**: a single hardcoded Anthropic-only client (the
|
||||
simpler default for a first LLM-driven Planner). Explicitly rejected by
|
||||
product decision before this design was drafted — dual-provider
|
||||
pluggability is a requirement, not a nice-to-have, for this change.
|
||||
|
||||
### D5: Perception Boundary amendment — narrow, Planner-only screenshot exception
|
||||
The Constitution's Perception Boundary previously stated `Scene` is the
|
||||
*only* perception artifact the LLM sees. Vision-grounded action selection
|
||||
(precise tap/swipe coordinates, disambiguating visually-similar elements
|
||||
that OCR/UI-tree fusion can conflate) requires the Planner to also see the
|
||||
raw screenshot for the current step — a deliberate product decision, not an
|
||||
oversight. The amendment is written to be as narrow as that requirement
|
||||
actually is: it names one consumer (`the runtime-layer AI Planner, and only
|
||||
that Planner`) and explicitly reaffirms that every other layer/consumer
|
||||
(`api`, `tools`, `perception`, `storage`, or any other LLM consumer) still
|
||||
never receives raw screenshot bytes, and that `Scene` itself is still
|
||||
produced exclusively through `PerceptionProvider`. See "Constitution
|
||||
Compliance" below for the exact before/after text.
|
||||
**Alternative considered**: keep `Scene` as the Planner's only input
|
||||
(Scene-only, no vision). Explicitly rejected by product decision before this
|
||||
design was drafted, for the reasons above.
|
||||
**Alternative considered**: broaden the exception to "any runtime-layer LLM
|
||||
consumer" instead of naming the Planner specifically, anticipating future
|
||||
vision consumers. Rejected — Change Discipline requires each change to
|
||||
justify its own scope; a hypothetical future consumer should justify its own
|
||||
amendment when it exists, not inherit a pre-approved blank check today.
|
||||
|
||||
### D6: Default-disabled via `AI_PLANNER_ENABLED`, mirroring existing LLM-feature flags
|
||||
`PlannerConfig.enabled` defaults to `False`, and `load_config()` reads
|
||||
`AI_PLANNER_ENABLED` the same way `semantic/config.py`,
|
||||
`skills_learning/config.py`, and `agents/config.py` gate their own
|
||||
LLM-backed behavior — off unless explicitly turned on. (`world/config.py`'s
|
||||
`WorldConfig.enabled` defaults `True`, but that capability makes zero LLM
|
||||
calls and is not a counterexample to this convention.) `TaskRunner`'s
|
||||
`_default_planner()` follows the same "self-loaded config, enabled flag
|
||||
picks the implementation" shape already used for `world_model` and
|
||||
`on_task_succeeded` (`runtime/task.py`), so no change to `api/rest.py` or any
|
||||
other caller of `TaskRunner()` is required to preserve current behavior.
|
||||
|
||||
### D7: `TaskRunner.run()` gains a try/except around observe+plan
|
||||
Before this change, `run()` had no error handling around
|
||||
`self.observer(...)` or `self._plan(...)` — safe only because the stub
|
||||
`Planner` never raised. `AIPlanner` can raise for real reasons (network
|
||||
failure, malformed provider response, `TaskFailedError` from D2's
|
||||
`finish_task(success=False)` path), so `run()` now catches any exception
|
||||
from that per-iteration observe-then-plan sequence and marks the task
|
||||
`status="failed"` with `failure_reason=f"{type(exc).__name__}: {exc}"`,
|
||||
returning immediately. Two failure modes are avoided by doing this
|
||||
explicitly rather than skipping it: an uncaught exception leaving a
|
||||
`BackgroundTasks`-invoked task stuck at `status="running"` forever, and a
|
||||
catch-and-return-`[]` degrade pattern (as used by
|
||||
`semantic/enricher.py::enrich_scene()`) that would be misread as task
|
||||
success by the same `if not steps or ...` short-circuit discussed in D2/D3.
|
||||
**Alternative considered**: copy `enrich_scene()`'s catch-and-degrade-to-None
|
||||
pattern. Rejected — that pattern is correct for an optional enrichment pass
|
||||
where the caller has a defined fallback (use the raw `Scene`). The Planner
|
||||
has no such fallback: if it cannot decide, the task cannot proceed, and
|
||||
pretending otherwise (via an empty list) means silent, incorrect "success."
|
||||
|
||||
### D8: `_planner_accepts_world()` generalized to `_planner_accepts(name)`
|
||||
`TaskRunner` already used `inspect.signature` reflection to decide whether to
|
||||
pass a `world=` kwarg to `Planner.plan()`, so that narrow-signature test
|
||||
Planners (e.g. `NoWorldPlanner`) would not receive an unexpected keyword
|
||||
argument. This change generalizes that one method to take the parameter name
|
||||
as an argument and reuses it for both `"world"` and the new `"screenshot"`
|
||||
kwarg, preserving backward compatibility with any existing `Planner`
|
||||
subclass that does not declare a `screenshot` parameter (or `**kwargs`).
|
||||
|
||||
## Constitution Compliance
|
||||
|
||||
- **Device Boundary**: unaffected — this change adds no driver code and does
|
||||
not touch `driver/`/`device/`.
|
||||
- **Tool Boundary**: unaffected — `AIPlanner` calls tools exclusively through
|
||||
`PlannedStep` → `Executor`, the same as the stub `Planner`; no new code
|
||||
imports a concrete driver or bypasses `tools/`.
|
||||
- **Perception Boundary**: amended, narrowly, per D5. The invariant that
|
||||
`Scene` is the only perception artifact for every consumer *except* the
|
||||
named Planner is preserved and made explicit in the same sentence as the
|
||||
exception, rather than being loosened globally.
|
||||
- **Runtime Boundary**: preserved and, in fact, fulfilled by this change —
|
||||
the existing text already states "LLM dependencies enter at `runtime`
|
||||
through Planner behavior," which is exactly what `AIPlanner` is. The
|
||||
Executor remains the only component that calls tools and handles
|
||||
tool-execution retries; `AIPlanner` does not call tools directly or
|
||||
implement its own retry loop (D7 fails the task instead of retrying).
|
||||
- **Change Discipline**: the dependency direction (`core -> driver/device ->
|
||||
tools -> perception -> storage -> runtime -> api`) is preserved — all new
|
||||
code lives in `runtime/`, imports flow downward only (`runtime/ai_planner.py`
|
||||
imports from `runtime/`, `core/`; nothing in `core`, `driver`, `device`, or
|
||||
`tools` imports anything LLM-related). The new external integration (the
|
||||
OpenAI SDK, newly *used* though already declared as a dependency) is added
|
||||
at the adapter layer that owns the concern (`runtime/tool_calling_client.py`),
|
||||
not at the domain or device boundary.
|
||||
|
||||
## Risks / Trade-offs
|
||||
|
||||
- **[Risk] Per-step LLM call adds latency and cost to every planning turn**
|
||||
→ Mitigation: default-disabled (D6); `AI_PLANNER_TIMEOUT_SECONDS` bounds
|
||||
worst-case latency; no in-`AIPlanner` retry (D7) means a slow/failing
|
||||
provider fails fast instead of compounding delay.
|
||||
- **[Risk] Hallucinated tool arguments (e.g. tap coordinates outside any
|
||||
element's bounds)** → Mitigation: the system prompt instructs the model to
|
||||
ground every coordinate in the current turn's `Scene` element bounds (and
|
||||
screenshot, when present); a bad tap still degrades gracefully into an
|
||||
ordinary failed/retried step through `Executor`'s existing mechanism,
|
||||
unchanged by this design.
|
||||
- **[Risk] Provider wire-format drift (Anthropic/OpenAI SDK or API changes)
|
||||
silently breaks request construction or response parsing** → Mitigation:
|
||||
all format-specific logic is isolated into small, independently unit-
|
||||
tested functions (`_anthropic_tool`, `_openai_tool`,
|
||||
`_decision_from_anthropic_response`, `_decision_from_openai_response`)
|
||||
against fake transports, so a drift shows up as a specific, localized test
|
||||
failure rather than a silent behavior change.
|
||||
- **[Risk] Two providers could behave inconsistently** (e.g. one honors
|
||||
`finish_task` semantics more reliably than the other) → Mitigation: both
|
||||
are constrained by the identical `ALL_TOOL_SPECS` JSON Schema and the same
|
||||
system prompt; provider-specific behavior differences are a model-quality
|
||||
concern to observe via the optional real-integration test, not something
|
||||
this design can fully eliminate structurally.
|
||||
- **[Risk] Perception Boundary amendment could be read as a precedent for
|
||||
loosening the invariant further** → Mitigation: the amendment text itself
|
||||
names exactly one consumer and restates the "no one else" constraint in
|
||||
the same breath (D5); any future consumer needs its own amendment.
|
||||
|
||||
## Migration Plan
|
||||
|
||||
Purely additive; no data migration:
|
||||
1. Add the five new `runtime/*` files and the `runtime/planner.py`/
|
||||
`runtime/task.py` edits described in Impact. With `AI_PLANNER_ENABLED`
|
||||
unset, `TaskRunner()` continues to construct the stub `Planner` exactly as
|
||||
before.
|
||||
2. Amend `docs/CONSTITUTION.md`'s Perception Boundary section (D5).
|
||||
3. No changes to `storage/`'s timeline format, `api/rest.py`, or `api/mcp.py`.
|
||||
4. Rollback is simply leaving `AI_PLANNER_ENABLED` unset/false, or reverting
|
||||
the changed files; no other capability depends on this one.
|
||||
5. Enabling in a real environment requires setting `AI_PLANNER_ENABLED=true`,
|
||||
`AI_PLANNER_PROVIDER` (`anthropic` or `openai`), and the corresponding
|
||||
provider's API key in the process environment (already-existing SDK
|
||||
convention, not a new config surface this change introduces).
|
||||
|
||||
## Open Questions
|
||||
|
||||
- Whether a `wait`/no-op tool is actually needed in practice, or whether the
|
||||
self-correcting-next-turn behavior of a spurious action is good enough
|
||||
indefinitely — deferred to real usage observation, not decided here.
|
||||
- Whether `AI_PLANNER_TIMEOUT_SECONDS`'s default (30s) is well-tuned against
|
||||
real provider latency for image-bearing requests — needs tuning against
|
||||
real usage data once this is enabled somewhere with real traffic; not
|
||||
fixed by this design.
|
||||
- Whether provider/model selection should eventually move from
|
||||
environment-variable/deployment-time to a per-task or per-request choice —
|
||||
leaning toward "not until a concrete need appears" (YAGNI), consistent
|
||||
with `semantic-scene-runtime`'s D4 reasoning for its own model choice.
|
||||
@@ -0,0 +1,106 @@
|
||||
## Why
|
||||
|
||||
`runtime/planner.py::Planner` — the Planner half of Milestone 3 (Agent
|
||||
Runtime)'s `agent-runtime` capability, implemented by `apex-agent-mvp` and
|
||||
still the only Planner in the codebase — is a stub: it always returns exactly
|
||||
one hardcoded `describe_screen` step on a task's first call, then `[]` on
|
||||
every call after any step has executed, regardless of the goal or the current
|
||||
`Scene`. `TaskRunner`'s Observe→Plan→Act→Observe loop, `Executor`'s
|
||||
retry/backoff, `TaskContext`'s per-task memory, and the already-implemented
|
||||
`semantic-scene` and `world-model` capabilities are all real and wired
|
||||
end-to-end — every later milestone (Task Memory, Semantic Scene, World Model,
|
||||
Skill Learning, Workflow Orchestration, Multi-Agent Runtime) has been built on
|
||||
top of, or alongside, this stub without ever replacing it. Nothing in the
|
||||
runtime actually decides what to do. This change gives the runtime its first
|
||||
real decision-maker: an LLM-driven Planner that uses native tool/function
|
||||
calling to choose exactly one grounded action per turn, completing what
|
||||
Milestone 3 always intended the Planner role to be.
|
||||
|
||||
## What Changes
|
||||
|
||||
- Add `runtime/ai_planner.py::AIPlanner`, a `Planner` implementation that
|
||||
calls an LLM with native tool/function calling once per `plan()`
|
||||
invocation, translating the model's single chosen tool call into 0 or 1
|
||||
`PlannedStep`. Single-step, ReAct-style: `TaskRunner.run()` already
|
||||
re-observes and re-plans every iteration, so `AIPlanner` never attempts
|
||||
multi-step lookahead.
|
||||
- Add a **pluggable dual-provider** LLM abstraction
|
||||
(`runtime/tool_calling_client.py`): both Anthropic native tool use and
|
||||
OpenAI function calling are supported, selected via configuration
|
||||
(`AI_PLANNER_PROVIDER`), each forced to return exactly one tool call per
|
||||
turn so a response always resolves to a single, unambiguous decision.
|
||||
- Add a fixed, six-tool tool-face (`runtime/tool_specs.py`): `tap`, `swipe`,
|
||||
`input_text`, `launch_app`, `terminate_app` for action, plus an explicit
|
||||
`finish_task(success, reason)` control tool the model calls to end the task
|
||||
(on success or failure) instead of relying on an ambiguous "no tool call"
|
||||
signal.
|
||||
- **Amend the Perception Boundary invariant** (`docs/CONSTITUTION.md`) with
|
||||
one narrow, explicit exception: the AI `Planner` (only that Planner) may
|
||||
additionally receive the current step's raw screenshot bytes alongside
|
||||
`Scene`, to support vision-grounded action grounding (precise tap/swipe
|
||||
coordinates). No other layer or LLM consumer gains access to raw screenshot
|
||||
bytes; `Scene` remains the only perception artifact everywhere else.
|
||||
- Add configuration (`runtime/planner_config.py`: `AI_PLANNER_ENABLED`,
|
||||
`AI_PLANNER_PROVIDER`, `AI_PLANNER_MODEL`, `AI_PLANNER_TIMEOUT_SECONDS`)
|
||||
defaulting `AI_PLANNER_ENABLED=False`, matching the existing enable/disable
|
||||
convention used by every other LLM-backed capability (`semantic-scene`,
|
||||
`skill-learning`). `runtime/task.py::TaskRunner` builds an `AIPlanner` by
|
||||
default only when enabled; otherwise its existing stub `Planner` behavior
|
||||
is unchanged.
|
||||
- Fix a latent correctness gap in `TaskRunner.run()`'s loop, exposed by
|
||||
giving the Planner a real chance to fail: observe/plan exceptions are now
|
||||
caught per iteration and turned into a `status="failed"` task with a
|
||||
`failure_reason`, instead of propagating uncaught (previously harmless only
|
||||
because the stub Planner never raised).
|
||||
- **BREAKING**: none. Default-disabled; when disabled, `TaskRunner`'s
|
||||
constructed `Planner` and control flow are unchanged from before this
|
||||
change.
|
||||
|
||||
## Capabilities
|
||||
|
||||
### New Capabilities
|
||||
|
||||
(none — this change fulfills the Planner role already scoped by Milestone
|
||||
3's `agent-runtime` capability; see Impact for why no new capability is
|
||||
declared)
|
||||
|
||||
### Modified Capabilities
|
||||
|
||||
- `agent-runtime`: the Planner requirement ("given a goal and the current
|
||||
Scene, produces steps") gains a real, LLM-backed implementation with
|
||||
native tool calling, a defined single-action-per-turn contract, an
|
||||
explicit task-completion/failure signal (`finish_task`), and — as a
|
||||
Planner-only exception to the Perception Boundary — optional access to the
|
||||
current step's screenshot. `agent-runtime`'s base spec was never archived
|
||||
into `openspec/specs/` (a pre-existing gap from `apex-agent-mvp`, out of
|
||||
scope for this change); this change's delta spec below uses `## ADDED
|
||||
Requirements` against that as-yet-unarchived baseline, the same way
|
||||
`semantic-scene-runtime` and `world-model-runtime` each layered their own
|
||||
delta on top of it without attempting to backfill it (see `design.md`).
|
||||
|
||||
## Impact
|
||||
|
||||
- **New files**: `runtime/planner_config.py`, `runtime/tool_specs.py`,
|
||||
`runtime/tool_calling_client.py`, `runtime/planner_prompts.py`,
|
||||
`runtime/ai_planner.py`.
|
||||
- **Modified**: `runtime/planner.py` (base `Planner.plan()` gains an optional
|
||||
`screenshot` parameter, stub behavior unchanged), `runtime/task.py`
|
||||
(`TaskRunner` builds an `AIPlanner` when enabled; generalized its existing
|
||||
`_planner_accepts_world()` reflection helper to also conditionally inject
|
||||
`screenshot`; wrapped the observe+plan step in try/except),
|
||||
`docs/CONSTITUTION.md` (Perception Boundary amendment described above).
|
||||
- **No change** to `api/rest.py` — `TaskRunner()`'s existing zero-argument
|
||||
construction picks up the new behavior automatically once
|
||||
`AI_PLANNER_ENABLED=true` is set in the environment; no new request
|
||||
parameters, no MCP surface change (`api/mcp.py` is a separate, curated
|
||||
tool-handler surface unaffected by this change).
|
||||
- **No new dependencies**: `pyproject.toml` already declares both
|
||||
`anthropic` and `openai` SDKs (the former added for
|
||||
`semantic-scene-runtime`); this change is the first to actually construct
|
||||
an OpenAI client.
|
||||
- **Out of scope**: no `wait`/no-op tool (v1.1 candidate, see `design.md`);
|
||||
no dynamic tool subset selection; no provider/model choice exposed as an
|
||||
API request parameter; no automatic retry of failed LLM calls inside
|
||||
`AIPlanner` (transient failures surface as a failed task; tool-execution
|
||||
retry remains solely `Executor`'s concern, unchanged by this change); no
|
||||
backfill of the missing `openspec/specs/agent-runtime/` base spec.
|
||||
@@ -0,0 +1,90 @@
|
||||
## ADDED Requirements
|
||||
|
||||
### Requirement: LLM-driven Planner selects exactly one grounded action per turn
|
||||
The system SHALL provide a Planner implementation that, given a goal, the
|
||||
current Scene, and recent task history, uses native LLM tool/function
|
||||
calling to select exactly one action (or the completion signal defined
|
||||
below) per `plan()` invocation, grounding any coordinates in the current
|
||||
turn's Scene element bounds.
|
||||
|
||||
#### Scenario: Planner selects a single action for the current turn
|
||||
- **WHEN** the AI Planner is invoked with a goal and the current Scene
|
||||
- **THEN** it returns at most one `PlannedStep`, whose action and arguments
|
||||
come from exactly one tool call chosen by the underlying LLM for that turn
|
||||
|
||||
#### Scenario: Planner re-decides every turn from the current Scene
|
||||
- **WHEN** the AI Planner is invoked again after a prior step has executed
|
||||
- **THEN** its decision is grounded in the newly observed Scene for that
|
||||
turn, not in coordinates or assumptions carried over from a previous turn
|
||||
|
||||
### Requirement: Explicit finish_task completion and failure signal
|
||||
The system SHALL treat task completion and task failure as explicit,
|
||||
model-driven signals via a dedicated `finish_task(success, reason)` tool,
|
||||
rather than inferring either outcome from the model declining to call any
|
||||
tool.
|
||||
|
||||
#### Scenario: Model signals successful completion
|
||||
- **WHEN** the model calls `finish_task` with `success=True`
|
||||
- **THEN** the Planner returns an empty step list and the task is marked
|
||||
completed
|
||||
|
||||
#### Scenario: Model signals it cannot complete the goal
|
||||
- **WHEN** the model calls `finish_task` with `success=False` and a `reason`
|
||||
- **THEN** the task is marked failed with that reason, without attempting
|
||||
any further planning steps
|
||||
|
||||
### Requirement: Pluggable dual-provider tool-calling abstraction
|
||||
The system SHALL support at least two interchangeable LLM providers
|
||||
(Anthropic native tool use and OpenAI function calling) for the AI Planner's
|
||||
decision calls, selectable via configuration, with both providers
|
||||
constrained to return exactly one tool call per request.
|
||||
|
||||
#### Scenario: Provider selected via configuration
|
||||
- **WHEN** the AI Planner is configured with a given provider identifier
|
||||
- **THEN** it constructs and uses the tool-calling client for that provider
|
||||
without requiring any change to `AIPlanner`'s own decision logic
|
||||
|
||||
#### Scenario: Provider response resolves to a single decision
|
||||
- **WHEN** either supported provider returns a response to a tool-calling
|
||||
request
|
||||
- **THEN** the response is parsed into exactly one tool name and one
|
||||
arguments object, regardless of which provider produced it
|
||||
|
||||
### Requirement: AI Planner is disabled by default and additive to the existing Planner
|
||||
The system SHALL default to the existing non-LLM Planner unless the AI
|
||||
Planner is explicitly enabled via configuration, and SHALL NOT alter the
|
||||
existing Planner's behavior, dependencies, or any caller's construction of
|
||||
`TaskRunner` when left disabled.
|
||||
|
||||
#### Scenario: AI Planner disabled (default)
|
||||
- **WHEN** `TaskRunner` is constructed without an explicit `planner` and
|
||||
without the AI Planner enabled in configuration
|
||||
- **THEN** it uses the existing non-LLM Planner, unchanged from before this
|
||||
capability existed
|
||||
|
||||
#### Scenario: AI Planner enabled via configuration
|
||||
- **WHEN** `TaskRunner` is constructed without an explicit `planner` and with
|
||||
the AI Planner enabled in configuration
|
||||
- **THEN** it uses the AI Planner, configured with the selected provider and
|
||||
model
|
||||
|
||||
### Requirement: Screenshot access is a Planner-only, narrow exception to the Perception Boundary
|
||||
The system SHALL allow the AI Planner, and only the AI Planner, to receive
|
||||
the current step's raw screenshot bytes alongside the Scene for
|
||||
vision-grounded decision-making, while every other perception consumer
|
||||
SHALL continue to receive only the Scene.
|
||||
|
||||
#### Scenario: Planner receives both Scene and screenshot
|
||||
- **WHEN** a screenshot for the current step is available
|
||||
- **THEN** the AI Planner's decision call includes both the Scene JSON and
|
||||
the raw screenshot bytes for that step
|
||||
|
||||
#### Scenario: Screenshot unavailable does not block planning
|
||||
- **WHEN** a screenshot for the current step cannot be obtained
|
||||
- **THEN** the AI Planner still produces a decision using the Scene alone,
|
||||
and this is not treated as a task failure
|
||||
|
||||
#### Scenario: No other consumer receives raw screenshot bytes
|
||||
- **WHEN** any component other than the AI Planner (for example, `api`,
|
||||
`tools`, `perception`, or `storage`) consumes perception output
|
||||
- **THEN** it receives only the Scene, never raw screenshot bytes
|
||||
@@ -0,0 +1,93 @@
|
||||
## 1. Configuration and tool face
|
||||
|
||||
- [x] 1.1 Create `runtime/planner_config.py`: `PlannerConfig` (`enabled`,
|
||||
`provider`, `model`, `timeout`), `load_config()` reading
|
||||
`AI_PLANNER_ENABLED`/`AI_PLANNER_PROVIDER`/`AI_PLANNER_MODEL`/
|
||||
`AI_PLANNER_TIMEOUT_SECONDS`, `enabled` defaulting `False`
|
||||
- [x] 1.2 Create `runtime/tool_specs.py`: vendor-neutral `ToolSpec` dataclass
|
||||
plus `TAP_SPEC`/`SWIPE_SPEC`/`INPUT_TEXT_SPEC`/`LAUNCH_APP_SPEC`/
|
||||
`TERMINATE_APP_SPEC`/`FINISH_TASK_SPEC`, matching real tool signatures,
|
||||
exported as `ACTION_TOOL_SPECS` (5) and `ALL_TOOL_SPECS` (6)
|
||||
- [x] 1.3 Create `runtime/planner_prompts.py`: `PLANNER_SYSTEM_PROMPT` and
|
||||
`planner_user_prompt(*, goal, scene_json, history_summary)`
|
||||
|
||||
## 2. Dual-provider tool-calling client
|
||||
|
||||
- [x] 2.1 Create `runtime/tool_calling_client.py`: `ToolCallDecision`,
|
||||
`ToolCallUnavailable`, `ToolCallingClient` Protocol
|
||||
- [x] 2.2 Implement `AnthropicToolCallingClient` (lazy SDK import, injectable
|
||||
transport, forced single tool call via
|
||||
`tool_choice={"type": "any", "disable_parallel_tool_use": True}`,
|
||||
image content block when a screenshot is present)
|
||||
- [x] 2.3 Implement `OpenAIToolCallingClient` (lazy SDK import, injectable
|
||||
transport, `max_completion_tokens`, forced single tool call via
|
||||
`tool_choice="required"` + `parallel_tool_calls=False`, `image_url`
|
||||
data-URI block when a screenshot is present)
|
||||
- [x] 2.4 Implement `build_client(config)` provider selection
|
||||
|
||||
## 3. AI Planner and runtime wiring
|
||||
|
||||
- [x] 3.1 Create `runtime/ai_planner.py::AIPlanner` (single-step decision,
|
||||
`finish_task` → empty plan / `TaskFailedError`, `goal_reached()`
|
||||
permanently `False`)
|
||||
- [x] 3.2 Add `screenshot: bytes | None = None` to base
|
||||
`runtime/planner.py::Planner.plan()`
|
||||
- [x] 3.3 Wire `runtime/task.py::TaskRunner`: `planner_config` constructor
|
||||
param, `_default_planner()` (AI Planner when enabled, stub otherwise),
|
||||
generalize `_planner_accepts_world()` into `_planner_accepts(name)`,
|
||||
add `_planning_screenshot()`, conditionally inject `screenshot=` in
|
||||
`_plan()`
|
||||
- [x] 3.4 Wrap `TaskRunner.run()`'s per-iteration observe+plan in try/except,
|
||||
marking the task `status="failed"` with a `failure_reason` on any
|
||||
exception instead of propagating it uncaught
|
||||
|
||||
## 4. Constitution amendment
|
||||
|
||||
- [x] 4.1 Amend `docs/CONSTITUTION.md`'s Perception Boundary section with the
|
||||
narrow, Planner-only screenshot exception
|
||||
|
||||
## 5. openspec change artifacts
|
||||
|
||||
- [x] 5.1 Write `proposal.md`, `design.md` (with explicit Constitution
|
||||
Compliance section), `tasks.md`
|
||||
- [x] 5.2 Write `specs/agent-runtime/spec.md` (`## ADDED Requirements` only)
|
||||
|
||||
## 6. Tests
|
||||
|
||||
- [x] 6.1 `tests/test_planner_config.py`: env var parsing, defaults,
|
||||
`enabled` defaults `False`, invalid/negative timeout falls back to
|
||||
default
|
||||
- [x] 6.2 `tests/test_tool_specs.py`: each `ToolSpec.parameters` schema
|
||||
(required fields, `additionalProperties: False`), `ALL_TOOL_SPECS` has
|
||||
exactly 6 entries
|
||||
- [x] 6.3 `tests/test_tool_calling_client.py`: fake-transport tests per
|
||||
provider — forced single-tool-call fields present, image block
|
||||
present/absent based on screenshot, successful response parses to
|
||||
`ToolCallDecision`, malformed/error response raises
|
||||
`ToolCallUnavailable`
|
||||
- [x] 6.4 `tests/test_ai_planner.py`: fake `ToolCallingClient` — action
|
||||
decision → single `PlannedStep`; `finish_task(success=True)` → `[]`;
|
||||
`finish_task(success=False, reason=...)` → raises `TaskFailedError`;
|
||||
`goal_reached()` always `False`
|
||||
- [x] 6.5 Added `tests/test_ai_planner_task_runner.py` (kept
|
||||
`tests/test_task_loop.py` untouched rather than extending it): planner
|
||||
exception mid-task → task ends `status="failed"` with a
|
||||
`failure_reason` (not stuck `running`); observer exception is caught
|
||||
the same way; a narrow-signature Planner does not receive an
|
||||
unexpected `screenshot` kwarg while one that declares it does;
|
||||
`PlannerConfig(enabled=False)` (default) still yields the stub
|
||||
`Planner`; `PlannerConfig(enabled=True)` without an explicit
|
||||
`planner=` yields an `AIPlanner`
|
||||
- [x] 6.6 `tests/test_ai_planner_integration.py` (`@pytest.mark.integration`,
|
||||
skipped without a real API key): one real call per provider
|
||||
|
||||
## 7. Verification
|
||||
|
||||
- [x] 7.1 Ran `pytest -m "not integration"`: 314 passed, 6 deselected (up
|
||||
from the pre-existing 269-test baseline plus the 45 new tests added by
|
||||
this change)
|
||||
- [x] 7.2 Confirmed default-off behavior: no `AI_PLANNER_*` environment
|
||||
variables set in the shell → `TaskRunner()` constructs
|
||||
`planner_config=PlannerConfig(enabled=False, ...)` and
|
||||
`type(runner.planner) is Planner`; `uvicorn api.rest:create_app
|
||||
--factory` starts cleanly with no API key configured
|
||||
Reference in New Issue
Block a user