agentic-mobile-control/openspec/changes/archive/2026-07-06-multi-agent-runtime/proposal.md

Why

agent-runtime (apex-agent-mvp, code-complete but unapplied) runs a single Observe→Think→Act loop: TaskRunner.run() calls Planner.plan() once per iteration and hands every produced PlannedStep to Executor.execute(), whose only failure signal is "did the tool call raise" — retried with bounded backoff. That is a mechanical success signal, not a semantic one: a tap on the send button can return success=True from the driver while the message never actually sent (wrong element, stale screen, silent app-level rejection), and nothing in the current loop would ever notice, because no role ever asks "did the intended effect actually happen" using anything richer than the tool call's own return value. As the runtime grows richer state to reason with — SemanticScene (Milestone 5, page identity/intents/widget purposes) and WorldState (Milestone 6, current app/page/variables/bounded history) — a single Planner+Executor pair has no natural place to use that richer state for verification or recovery; today it is planning input only. This change splits the loop's responsibilities across five collaborating roles — Observer, Planner, Executor, Verifier, Reflector — so that "did this step actually work" and "what should we do about it if not" become first-class, semantically-informed steps in the loop, layered above (not replacing) the Executor's existing low-level retry/backoff safety net.

What Changes

• Add a new agents/ package implementing three new roles — Observer (perceives and summarizes current device state via SemanticScene/WorldState, producing an Observation), Verifier (checks whether an already-executed step actually achieved its intended effect, comparing pre-step and post-step Observations against the PlannedStep's stated intent, distinct from and running after the Executor's own low-level retry), and Reflector (invoked only when the Verifier flags a step as not-achieved; analyzes the likely failure cause from the Observation/PlannedStep/StepResult/Verifier verdict and proposes either a bounded recovery action or a replan request, never a blind re-issue of the same step).
• Add a CollaborativeTaskRunner (agents/collab_runner.py) that composes the existing, unmodified Planner and Executor (agent-runtime) with the three new roles in a defined handoff loop: Observer → Planner → Executor → Verifier → (Reflector only on a Verifier-flagged failure) → back to Observer. It reuses runtime/task.py's TaskRunner as the underlying step-execution engine (composed by import, not forked) rather than reimplementing planning/execution/timeline-append logic.
• Define the handoff protocol as typed data passed between roles (Observation, VerificationVerdict, ReflectionOutcome), each a plain dataclass with no role-specific side effects, so the protocol itself — not any one role's internal implementation — is the stable, spec'd contract.
• Add configuration to enable/disable multi-agent collaboration per task run, defaulting to disabled (mirrors semantic-scene-runtime's default-off precedent) so that applying this change does not silently change any existing task's cost, latency, or step count; when disabled, a task runs exactly as runtime/task.py's existing TaskRunner.run() today.
• Bound Reflector-triggered replanning with an explicit ceiling (a configurable max reflection-recovery attempts per task, separate from and on top of the Executor's own max_retries), so a persistently-failing step cannot loop forever between Verifier and Reflector.
• BREAKING: none. runtime/planner.py's Planner, runtime/executor.py's Executor, and runtime/task.py's TaskRunner are not modified by this change; multi-agent collaboration is an opt-in alternate driver of the same underlying Task/Planner/Executor contracts, not a replacement.

Capabilities

New Capabilities

• multi-agent-collaboration: Defines the Observer/Verifier/Reflector roles, their Observation/VerificationVerdict/ReflectionOutcome data contracts, and the Observer→Planner→Executor→Verifier→(Reflector)→Observer handoff protocol that composes with the existing Planner/Executor (agent-runtime) as a higher semantic-verification layer above (not a replacement for) the Executor's own bounded low-level retries, usable both for a single-goal task and for one step of a multi-step Workflow (workflow-orchestration).

Modified Capabilities

(none — agent-runtime's Planner, Executor, and TaskRunner (apex-agent-mvp) keep their existing requirements unchanged; CollaborativeTaskRunner composes them strictly by import as an opt-in alternate driver, and neither semantic-scene nor world-model has their specified behavior altered by this change, see Impact.)

Impact

• New package: agents/ — models.py (Observation, VerificationVerdict, ReflectionOutcome, ReflectionAction dataclasses), observer.py (Observer.observe(...) -> Observation), verifier.py (Verifier.verify(...) -> VerificationVerdict), reflector.py (Reflector.reflect(...) -> ReflectionOutcome), collab_runner.py (CollaborativeTaskRunner, the handoff loop), config.py (enable flag, max-reflection-attempts ceiling).
• No change to runtime/planner.py, runtime/executor.py, runtime/task.py, core/models.py, storage/timeline.py, or any existing tool in tools/ — all existing tests and callers keep working unmodified; CollaborativeTaskRunner composes TaskRunner/Planner/Executor strictly by import, as workflow-orchestration-runtime's WorkflowRunner already does for its planned-goal steps.
• Reads from pending capabilities (read-only composition, no spec changes to them): semantic-scene (Milestone 5) for SemanticScene as Observer/Verifier input (with the existing skip/fallback degrade path honored — Observer must work when SemanticScene is None), world-model (Milestone 6) for WorldState as additional Observer/Verifier context, and agent-runtime (apex-agent-mvp) for Task/PlannedStep/StepResult/TaskContext as the underlying vocabulary every new role's inputs/outputs are built from.
• Composable with workflow-orchestration (Milestone 8): a WorkflowDefinition's planned-goal step may be driven by CollaborativeTaskRunner instead of the plain TaskRunner, without any change to workflow/runner.py's step-handler contract (run(task) -> Task in, Task out) — this change does not itself modify workflow/.
• Config: pyproject.toml gains an agents* entry in [tool.setuptools.packages.find].include; no new third-party dependency (roles reuse semantic/llm_client.py's existing LLM client abstraction for Verifier/Reflector reasoning, added in Milestone 5, not a new SDK).
• Out of scope: no new LLM provider/vendor integration; no removal or modification of the Executor's existing retry/backoff logic; no persistence of Observation/VerificationVerdict/ReflectionOutcome into storage/timeline.py (left as an open question, mirroring world-model-runtime's deferred WorldState persistence question); no change to skill-catalog-subscription, web-console, or skill-learning-runtime.