agentic-mobile-control/openspec/changes/archive/2026-07-06-world-model-runtime/proposal.md

Why

agent-runtime's TaskRunner (from apex-agent-mvp, code-complete but unapplied) re-derives everything about a task from scratch every step: TaskContext accumulates a flat list of Scenes and StepResults, but nothing in the loop distills "what app am I in," "what page am I on," "am I already logged in," or "did I already navigate into the chat with Zhang San" into a queryable form. Even with semantic-scene-runtime's per-step SemanticScene (page identity, intents, widget purposes), that artifact is deliberately ephemeral and same-step only — it is discarded, not accumulated, so a real Planner (a later milestone) still cannot ask "have I already done this" without re-scanning raw scene/step history itself. This change adds a World Runtime: a WorldState that persists across a task's steps (current app, current page, a small variables dict, and a bounded history of recent semantic scenes/actions), updated incrementally by a hook in TaskRunner's step loop after each executed step, and exposed read-only to the Planner alongside the current SemanticScene so plans can skip redundant navigation or re-discovery. This is Milestone 6 (World) of the device-agnostic runtime roadmap established by device-agent-runtime-foundation, sitting directly on top of Milestone 5's semantic-scene capability.

What Changes

• Add a new world/ package that defines a WorldState dataclass (current_app: str | None, current_page: str | None, variables: dict[str, Any], a bounded history: deque[WorldEvent] of recent (semantic_scene | scene, action) pairs) and a WorldModel that owns one WorldState per task and knows how to update it.
• Introduce an update hook (WorldModel.observe(scene, semantic_scene, step, result)) called once per executed step from runtime/task.py's TaskRunner.run() loop, after context.add_step_result(result), so WorldState is derived incrementally from exactly the same per-step data the timeline already records — no new I/O, no new LLM call in this change.
• Define the update rule set as a small, deterministic, rule-based derivation (not a second LLM call): app/page fields are refreshed from the current SemanticScene.page (falling back to a heuristic derived from the raw Scene/last launch_app action when semantic enrichment is disabled or unavailable), variables are updated only via an explicit PlannedStep.args["remember"] convention a Planner can opt into, and history is a fixed-size ring buffer (bounded, oldest evicted first) so WorldState cannot grow unboundedly across a long-running task.
• Expose WorldState read-only to the Planner: extend Planner.plan()'s call signature with an optional world: WorldState | None keyword argument (default None, so the existing stub Planner and any test constructing PlannedSteps directly keep working unmodified) that a future LLM-driven Planner (not built in this change) can read to decide "already there, skip this step."
• Add TaskContext.world (a WorldState | None field, populated by TaskRunner when a WorldModel is configured) so a single object continues to carry all per-run state the Planner/Executor need, matching agent-runtime's existing "task context/memory available during a run" requirement instead of introducing a second parallel context object.
• Add configuration to enable/disable World Runtime tracking globally (default enabled, since this is a pure derivation over data the loop already produces — unlike semantic-scene's LLM call, there is no cost/latency reason to default it off) and to size the bounded history (default a small fixed window, e.g. 10 events).
• BREAKING: none. WorldState/WorldModel are additive; TaskContext.world defaults to None when no WorldModel is configured, Planner.plan()'s new world kwarg defaults to None, and TaskRunner's constructor accepts an optional world_model with None preserving today's behavior exactly.

Capabilities

New Capabilities

• world-model: A WorldState store (current app, current page, a variables dict, and a bounded history of recent semantic-scene/action pairs) for a single task, updated by a hook in the agent runtime's step loop after each executed step, and exposed read-only to the Planner as additional context alongside the current SemanticScene, so plans can skip redundant navigation or re-discovery.

Modified Capabilities

(none — agent-runtime's Observe→Think→Act→Observe loop shape and semantic-scene's SemanticScene output are read-only inputs to this change; neither capability's existing requirements are altered. agent-runtime's "task context/memory available during a run" requirement is extended in spirit — TaskContext gains a world field — but this change does not itself modify agent-runtime's spec deltas since openspec/specs/ has no applied baseline for it yet; see Impact.)

Impact

• New package: world/ — models.py (WorldState, WorldEvent dataclasses), model.py (WorldModel, the per-task owner + observe() update hook + rule-based derivation logic), config.py (enable/disable + history-size settings).
• Modified: runtime/context.py (TaskContext gains a world: WorldState | None = None field); runtime/task.py (TaskRunner gains an optional world_model: WorldModel | None constructor argument, calls world_model.observe(...) once per executed step inside the existing loop, and passes context.world into self.planner.plan(...)); runtime/planner.py (Planner.plan() gains an optional world: WorldState | None = None keyword argument, unused by today's stub Planner but available to a future LLM-driven Planner).
• No change to core/models.py's Scene/Task/Step, semantic/'s SemanticScene shape or enrichment logic, storage/timeline.py's persisted format, or any existing tool signature — WorldState is derived in-memory per task run and is not persisted by this change (see design.md Open Questions for whether a later change should persist it).
• Out of scope: no cross-task or cross-device world sharing (single task's world only, discarded when the task ends); no UI for viewing/editing world state (that is web-console's domain, not touched here); no change to skill-catalog-subscription's or web-console's pending capabilities.