agentic-mobile-control/openspec/changes/archive/2026-07-06-semantic-scene-runtime/proposal.md

Why

scene-perception (from apex-agent-mvp, code-complete but unapplied) fuses a screenshot, the device's UI tree, and OCR output into a Scene: screen dimensions plus a flat list of typed elements (bounds, text, confidence, source). That is still a structural description of the screen — it tells the LLM "there is a button-shaped element at (x, y) with text 'Send'," not "this is the WeChat chat screen and tapping that button sends the message." Every planning step, the LLM has to re-derive page identity and intent from raw element geometry, which burns context on repeated low-level reasoning and is brittle to layout drift (the same page re-derived slightly differently step to step). This change adds a Semantic Runtime: a single additional LLM enrichment call that turns a Scene into a SemanticScene — page identity, a short list of plain-language supported intents, and a purpose label per widget — so most subsequent prompts can rely on a compact, stable JSON summary instead of re-parsing raw geometry or screenshots. This is Milestone 5 (Semantic) of the device-agnostic runtime roadmap established by device-agent-runtime-foundation, sitting directly on top of the existing Perception milestone's Scene output.

What Changes

• Add a new semantic/ package that consumes an existing Scene (from perception/, per the scene-perception capability) and produces a SemanticScene: {page: str, intents: list[str], widgets: [{element_id: str, purpose: str}]}.
• Introduce the first real LLM integration point in the codebase: a narrow, swappable LLM client abstraction used only by semantic/ to make one structured-output call per enrichment (model, prompt, and structured-schema details are an implementation decision in design.md, not part of this proposal's contract).
• Define a mandatory degrade path: if the enrichment LLM call is unavailable, times out, or fails for any reason, semantic enrichment is skipped and callers fall back to using the raw Scene directly — enrichment failure must never block or fail the Observe→Act loop.
• Add a new tool-level entry point (e.g. describe_screen_semantic or an enrich flag on the existing describe-screen flow) so runtime/ and api/ callers can opt into the enriched view without every existing caller of describe_screen needing to change.
• Add configuration to enable/disable semantic enrichment globally (so it can be turned off entirely in environments without LLM access, e.g. tests or offline development) without touching scene-perception.
• BREAKING: none. This is a purely additive layer; nothing in scene-perception, agent-runtime, or any other existing capability changes shape or behavior.

Capabilities

New Capabilities

• semantic-scene: Builds a SemanticScene (page identity, supported intents, per-widget purpose labels) from an existing Scene via one additional LLM call, with a defined skip/fallback degrade path when that call is unavailable or fails, so enrichment is strictly additive and non-blocking to the Observe→Act loop.

Modified Capabilities

(none — scene-perception's Scene format and requirements from apex-agent-mvp are read-only input to this change and are not modified; agent-runtime's Planner/Executor loop shape is not changed by this proposal, only optionally consumed by it, see design.md for how a future milestone might wire it in.)

Impact

• New package: semantic/ — models.py (SemanticScene, SemanticWidget dataclasses), llm_client.py (narrow LLM client wrapper + typed result), enricher.py (enrich_scene(scene, ...) -> SemanticScene | None), prompts.py (the enrichment system prompt / JSON schema description).
• New tool: tools/describe_screen_semantic.py (or equivalent), wrapping the existing tools/describe_screen.py output with an enrichment pass and degrade path, following the same manager/device_id kwarg pattern as other tools.
• Config: pyproject.toml gains an anthropic SDK dependency and a semantic* entry in [tool.setuptools.packages.find].include; a new settings surface (e.g. env var or config value) to enable/disable enrichment and select the model.
• No change to core/models.py's Scene/SceneElement, vision/perception's fusion pipeline, or any existing tool signatures — all existing callers of describe_screen keep working unmodified.
• Out of scope: no persistent cross-step semantic state or caching across steps (Milestone 6, World Runtime); no skill synthesis from semantic labels (Milestone 7); no change to skill-catalog-subscription or web-console.