agentic-mobile-control/openspec/changes/world-model-runtime/design.md

Context

runtime/task.py's TaskRunner.run() is the Observe→Think→Act→Observe loop (agent-runtime capability, apex-agent-mvp, code-complete but unapplied): each iteration calls self.observer(device_id) to get a Scene, appends it to TaskContext.scenes, calls self.planner.plan(goal=..., scene=..., context=...), executes each PlannedStep via Executor.execute(), and appends the StepResult to TaskContext.step_results. TaskContext (runtime/context.py) is a flat dataclass — scenes: list[Scene], step_results: list[StepResult] — with no derived/summarized state; context.latest_scene is the only convenience accessor today. semantic-scene-runtime (Milestone 5, drafted alongside this change) adds enrich_scene(scene) -> SemanticScene | None, a same-step, non-persisted artifact — its own design.md explicitly leaves open "should SemanticScene be recorded in TaskContext/the timeline" as a question for this milestone to answer. Today's Planner is a two-step stub (runtime/planner.py) that emits one hardcoded describe_screen step and declares the goal reached once any step has succeeded — it does not yet reason about page identity at all, so nothing currently consumes anything like "have I already navigated here." A future LLM-driven Planner is the actual reason this capability needs to exist: without a persisted, bounded summary of "where am I / what have I already established," a real Planner has no way to skip re-navigating to a screen it visited two steps ago except by re-scanning the entire scenes/step_results history and re-deriving semantics itself, every single step.

Two stakeholders: runtime/task.py's TaskRunner (must gain a well-defined, optional integration point without changing behavior when WorldModel is not configured), and a future LLM-driven Planner (not built in this change) that will be the actual consumer reading WorldState to make cheaper plans.

Goals / Non-Goals

Goals:

• Maintain one WorldState per task run — current_app, current_page, a variables dict, and a bounded history of recent semantic-scene/action pairs — updated incrementally after each executed step.
• Make the update purely a derivation over data the loop already produces (the just-observed Scene, the optional SemanticScene, the PlannedStep, and its StepResult) — no new I/O, no new LLM call, no new external dependency introduced by this change.
• Keep history genuinely bounded (a fixed-size ring buffer) so a long-running task (up to TaskRunnerConfig.max_steps, currently 20, but not contractually capped) cannot grow WorldState without limit.
• Expose WorldState to the Planner strictly read-only and additively — Planner.plan() gains an optional keyword argument with a safe default, so every existing caller and test keeps working unmodified.
• Make WorldModel degrade sensibly when SemanticScene is absent (semantic enrichment disabled, per semantic-scene-runtime's default-disabled config) — page/app tracking falls back to a raw-Scene/action heuristic rather than going stale or raising.

Non-Goals:

• No cross-task or cross-device world sharing — WorldState is scoped to exactly one Task/TaskContext and is discarded when the task ends; a follow-on "world persistence" or "world sharing across a device's tasks" capability is a separate future decision, not made here.
• No UI for viewing or editing world state — that belongs to web-console's console-status-api/console-config-api/web-console-ui capabilities (a separate pending change), not touched here.
• No real LLM-driven Planner — this change only makes WorldState available as a Planner input; teaching a Planner to actually use it (e.g. skip a navigation step because current_page already matches) is future Planner-capability work, out of scope here.
• No second LLM call to summarize/update world state — updates are deterministic rules over already-available per-step data, not a new enrichment call (see D2).
• No persistence of WorldState into storage/timeline.py or task-memory's timeline format — WorldState lives only in the in-memory TaskContext for the duration of a run (see Open Questions).

Decisions

D1: New world/ package, sibling to semantic/, not folded into runtime/context.py

WorldState/WorldModel get their own package (world/models.py, world/model.py, world/config.py) rather than growing runtime/context.py in place. TaskContext gains only a thin world: WorldState | None field pointing at an object world/ owns and updates; the update rules (how current_page is derived from a SemanticScene, how history eviction works) live in world/model.py, not in runtime/. This mirrors semantic-scene-runtime's own D1 (a sibling package rather than folding into the layer below) and keeps runtime/ focused on orchestration (loop shape, retries, tool dispatch) rather than accumulating every future kind of derived state inline. Alternative considered: add current_app/current_page/variables/history fields directly onto TaskContext and put the update logic in runtime/task.py's TaskRunner.run(). Rejected — TaskContext would become a grab-bag mixing raw per-step history (scenes, step_results, owned by agent-runtime) with derived cross-step summarization (owned by this new capability); a future World-related change (persistence, cross-task sharing) would then have to reach into runtime/ internals instead of a self-contained world/ package, repeating the exact "grab-bag package" problem device-agent-runtime-foundation already diagnosed and fixed for core/.

D2: Deterministic rule-based update, not a second LLM call

WorldModel.observe(scene, semantic_scene, step, result) is pure Python: if semantic_scene is present, current_page = semantic_scene.page; current_app is refreshed whenever step.action in {"launch_app", "terminate_app"} succeeds, using step.args["bundle_id"]/step.args["app_id"] (falling back to leaving current_app unchanged if the action fails); variables are updated only when step.args contains an explicit "remember" mapping ({key: value}) a Planner opted into; history appends one WorldEvent(semantic_scene_or_scene, step.action, result.success) per call and evicts the oldest entry once past the configured bound. No network call, no additional latency, no new failure mode beyond "field stays stale if inputs are absent." Alternative considered: a second structured-output LLM call (reusing semantic-scene-runtime's llm_client.py pattern) that looks at the accumulated history and produces a compact world-state summary each step, similar to how SemanticScene itself is produced. Rejected for this milestone — doubling the per-step LLM call count (one for SemanticScene, one for WorldState) doubles latency/cost for a problem (bookkeeping already-known fields) that does not need model reasoning to solve; a rule-based derivation over the same SemanticScene/PlannedStep/StepResult data that already exists is sufficient and strictly cheaper. This can be revisited if a future milestone's Planner needs richer world summarization than simple field-tracking + bounded history provides.

D3: WorldState exposed via TaskContext.world, not a second context object threaded separately

TaskRunner sets context.world = world_model.state once (or updates it in place) rather than introducing a WorldContext parameter threaded alongside TaskContext through Planner.plan()/Executor.execute(). Planner.plan()'s new world: WorldState | None = None keyword is a convenience mirror of context.world (some future Planner implementations may prefer an explicit parameter over reaching into context), but the source of truth is always TaskContext.world. Alternative considered: keep WorldState entirely outside TaskContext, passed as a wholly separate argument to Planner.plan()/Executor.execute()/TaskRunner.run(). Rejected — agent-runtime's existing requirement is "task context/memory available during a run" as one accessible object; splitting per-run state across two parallel objects (TaskContext for raw history, a bare WorldState for derived summary) makes every future call site that needs "everything about this run" thread two parameters rather than one, for no benefit since WorldState is inherently TaskContext-scoped (one task, one world) anyway.

D4: Bounded history via a fixed-size ring buffer, not unbounded list or time-based eviction

WorldState.history: deque[WorldEvent] is a collections.deque(maxlen=N) with N from world/config.py (default 10), so the oldest event is dropped automatically once the buffer is full — no separate cleanup pass, no unbounded memory growth even if a task runs far beyond TaskRunnerConfig.max_steps's current default of 20 (e.g. if a future change raises that ceiling). Alternative considered: keep an unbounded list[WorldEvent] (simplest, matches TaskContext.scenes/step_results which are also unbounded lists today). Rejected — scenes/step_results are raw historical record (their unboundedness is intentional, mirroring the timeline), whereas history here exists specifically to give the Planner a recent window without asking it to reason about a growing list; an explicit bound keeps the read-only Planner-facing contract ("recent" is a stated, fixed size) rather than an implicit "whatever the task length happens to be."

D5: World Runtime tracking defaults to enabled (unlike Semantic Scene's default-disabled enrichment)

world/config.py's enable flag defaults to True, and TaskRunner's optional world_model argument, when left None, still gets a default WorldModel constructed internally (not skipped) unless the config flag is explicitly off. This differs from semantic-scene-runtime's enrichment, which defaults to disabled specifically because it makes a paid network call every step. Alternative considered: default World Runtime to disabled as well, for consistency with the Semantic Scene precedent and to minimize behavior change on apply. Rejected — WorldModel.observe() has no network call and no meaningful cost/latency (it is a few dict/deque operations over data the loop already holds), so the reason Semantic Scene defaults off (protect cost/latency-sensitive environments from a silent new LLM bill) does not apply here; defaulting on means the capability's read-only benefit is available to a Planner immediately, and an explicit off-switch remains for anyone who wants to opt out (e.g. a test asserting exact TaskContext shape pre-this-change).

D6: Planner.plan() gains an optional world kwarg with a safe default; the existing stub Planner ignores it

runtime/planner.py's Planner.plan(self, *, goal, scene, context, world=None) accepts but does not use world — the existing stub planner's behavior (one hardcoded describe_screen step, then done) is unchanged. This is intentionally a no-op integration point in this change: making a Planner smart enough to skip steps using WorldState is real LLM-Planner work reserved for a future milestone/change, not implied by adding the parameter. Alternative considered: leave Planner.plan()'s signature untouched and require future Planner implementations to read context.world directly instead of a dedicated parameter. Rejected — an explicit world parameter documents, at the call boundary, that world-state read access is a first-class, expected part of planning (matching how scene/context are already explicit parameters rather than everything being reached off one context blob), which matters for future callers/tests constructing a Planner subclass against a stable, self-documenting signature.

Risks / Trade-offs

• [Risk] A rule-based current_app/current_page derivation can drift from reality faster than an LLM-based one would (e.g. a page transition not captured by any launch_app/terminate_app action, or SemanticScene.page phrased inconsistently step to step since semantic-scene-runtime's Open Questions leave intents/page labels open-vocabulary) → Mitigation: WorldState is documented and enforced as read-only, advisory context for the Planner, never a source of truth the Executor trusts blindly; a future Planner consuming it is expected to still verify against the current Scene/SemanticScene before skipping a step, not skip purely on stale WorldState says-so. This change does not build that Planner logic, only the state it would read.
• [Risk] Enabling World Runtime by default (D5) means every existing TaskRunner caller/test that doesn't explicitly configure world_model=None starts accumulating WorldState it previously didn't have → Mitigation: this is additive-only (TaskContext.world is a new field with no effect on scenes/step_results/existing assertions), the update hook cannot raise (all derivation is defensive: missing SemanticScene, missing step.args keys, etc. are all "no-op, leave field unchanged," never an exception), and world/config.py provides an explicit off-switch for any caller/test that wants exact pre-this-change TaskContext shape.
• [Risk] variables's step.args["remember"] convention has no schema/validation — a Planner could stuff arbitrary or unbounded data into variables over many steps → Mitigation: out of scope to solve generally in this change (no Planner exists yet that writes to it); world/config.py can later gain a max-variables-size guard if a real Planner's usage pattern demands it, but there is no real caller to validate against yet, so adding a limit now would be speculative.
• [Trade-off] WorldState is not persisted (Non-Goal) — if a task fails partway and is retried, or if task-memory's timeline is later replayed, the accumulated world state from the failed run is not recoverable, only re-derivable from scratch on the retry → acceptable per this milestone's explicit scope (single task's world only); persistence is a task-memory-capability change to propose separately if needed, matching how semantic-scene-runtime deferred the same question for its own artifact.
• [Trade-off] Choosing rule-based derivation (D2) over LLM-based world summarization (rejected alternative) means WorldState's quality is bounded by how good SemanticScene.page/action-name heuristics are — if a future milestone finds this insufficient (e.g. genuinely needs to reason "the user probably navigated back," not just track the last known page), upgrading to an LLM-based summarizer is a world/model.py-internal change, not a WorldState shape change, since the shape (current_app/current_page/variables/history) is intentionally derivation-method-agnostic.

Migration Plan

This is purely additive with optional, defensively-defaulted integration points:

1. Add the world/ package (models.py: WorldState, WorldEvent; model.py: WorldModel.observe(); config.py: enable flag + history size).
2. Add TaskContext.world: WorldState | None = None to runtime/context.py.
3. Add an optional world_model: WorldModel | None constructor argument to TaskRunner (runtime/task.py); when None and World Runtime is enabled in config, TaskRunner constructs a default WorldModel internally; call world_model.observe(scene, semantic_scene, step, result) once per executed step, immediately after the existing context.add_step_result(result) line, and set/refresh context.world from the model's current state.
4. Add the optional world: WorldState | None = None keyword argument to Planner.plan() (runtime/planner.py); TaskRunner passes context.world through; the existing stub Planner implementation ignores it (no behavior change).
5. Add world* to pyproject.toml's [tool.setuptools.packages.find].include list; no new third-party dependency (no LLM SDK, no new I/O library — collections.deque is stdlib).
6. Run pytest — the full existing suite must remain green with zero test-content changes (only additive assertions in new tests/test_world_model.py-style tests are expected, not edits to existing tests), confirming the default-enabled World Runtime tracking does not alter any existing task-completion/failure behavior.
7. Rollback: since this is a net-new package plus two small additive fields/kwargs with safe defaults, reverting is git revert of the commit(s); no data migration, no persisted format touched, no external system involved.

Open Questions

• Whether WorldState should eventually be persisted into storage/timeline.py's entries (so a replayed/inspected task shows what the runtime "believed" at each step, not just the raw Scene/tool-call/result) — left open per this change's Non-Goals; would be a task-memory-capability change to propose separately once there is a concrete consumer (e.g. web-console's status API wanting to show "current believed page").
• Whether variables needs any validation, size bound, or namespacing convention once a real Planner starts writing to it via the "remember" args convention — deferred until a real Planner (a future milestone) exists to observe actual usage patterns against.
• Whether a future Skill-synthesis milestone would want WorldState.history's bounded window to be configurable per-task rather than one global default — left as a global-default-only setting for this milestone, matching semantic-scene-runtime's equivalent "leaning global-default-only" stance on its own model-selection config.
• Whether current_page derivation should fall back to something richer than "last known value, unchanged" when SemanticScene is absent (e.g. a raw-Scene-shape heuristic keyed on element text/layout signature) — this change only specifies the fallback as "leave unchanged," which is safe but potentially stale; worth revisiting once semantic enrichment's real-world enable rate is known.