agentic-mobile-control/openspec/changes/archive/2026-07-06-semantic-scene-runtime/design.md

Context

apex-agent-mvp (code-complete, unapplied) already produces a Scene for every Observe step: core/models.py defines Scene { width, height, elements: [SceneElement] } and SceneElement { id, type, bounds, text, confidence, source }, built by vision/scene_builder.py (planned rename: perception/scene_builder.py per device-agent-runtime-foundation) by fusing OCR output and the device's UI tree via bounding-box IoU. runtime/planner.py (Planner.plan()) is currently a stub that returns one hardcoded describe_screen step and then nothing — there is no real LLM call anywhere in the codebase yet. This change introduces the first real LLM integration point: a single enrichment call per Observe step that turns a Scene into a compact SemanticScene (page identity, supported intents, per-widget purpose labels), so a real LLM-driven Planner (a later milestone) and other prompts can consume a stable, low-token JSON summary instead of re-deriving page semantics from raw element geometry or a screenshot every step.

Because this is the first LLM call in the project, this design also sets the pattern (client shape, structured-output mechanism, failure handling) that later milestones (a real Planner, World Runtime's cross-step memory, Skill synthesis) are expected to reuse rather than inventing their own.

Goals / Non-Goals

Goals:

• Given a Scene, produce a SemanticScene — {page: str, intents: list[str], widgets: [{element_id, purpose}]} — using exactly one LLM call.
• Guarantee the enrichment call never blocks or fails the Observe→Act loop: any failure (timeout, rate limit, malformed response, disabled config) degrades to "no semantic scene," and callers fall back to the raw Scene.
• Keep the LLM client abstraction narrow and swappable (mockable in tests without network access), scoped inside the new semantic/ package rather than becoming a project-wide llm/ dependency other capabilities must adopt.
• Produce output that is schema-valid JSON by construction (not "usually valid, occasionally needs a repair pass"), since downstream code will parse it directly into SemanticScene.
• Make enrichment optional and cheap enough to run on every Observe step without materially changing task latency or cost profile.

Non-Goals:

• No change to scene-perception's Scene format or fusion logic — SemanticScene is a separate, additive artifact layered on top, never a replacement.
• No persistent cross-step semantic state, caching of SemanticScene across steps, or diffing between steps — that is Milestone 6 (World Runtime).
• No skill synthesis or action recommendation derived from intents/purposes — that is Milestone 7 (Skill).
• No change to Planner/Executor control flow itself — this change only makes a new artifact available; wiring a real LLM-driven Planner to consume SemanticScene is left to a later milestone (see Migration Plan).
• No multi-provider LLM abstraction (e.g. supporting both Anthropic and OpenAI behind a common interface) — a single concrete client is enough for this milestone; a provider-agnostic port can be extracted later if a second provider is actually needed (YAGNI).

Decisions

D1: New semantic/ package, not folded into perception/ or runtime/

SemanticScene is conceptually "one layer above Scene," but it has fundamentally different runtime characteristics: it makes a network call, it can fail/timeout, and it is optional. Mixing it into perception/ (which is currently synchronous, local-only, and always-on) would force every perception/ consumer to reason about network failure modes it doesn't otherwise have. A sibling semantic/ package (models.py, llm_client.py, enricher.py, prompts.py) keeps perception/'s contract unchanged and makes the enrichment layer's optionality explicit at the package boundary. Alternative considered: add an enrich() method directly on Scene or inside scene_builder.py. Rejected — it would make perception/ depend on an LLM SDK and network config, contradicting device-agent-runtime-foundation's stated dependency direction (perception must stay a local, deterministic pipeline; LLM-backed behavior is explicitly scoped to sit above it, closer to runtime).

D2: enrich_scene() returns SemanticScene | None, never raises for expected failure modes

enricher.enrich_scene(scene, *, context=None) -> SemanticScene | None catches all expected LLM-client failure modes (timeout, rate limit, connection error, malformed/schema-invalid response, enrichment disabled by config) internally and returns None on any of them, logging the reason. Callers (tools, later a real Planner) treat None as "use the raw Scene," never as an exception to handle. Alternative considered: raise a typed EnrichmentUnavailableError and require every caller to catch it. Rejected — experience from runtime/executor.py's existing retry/backoff pattern shows try/except-per-call-site is exactly the kind of boilerplate that leads to a caller eventually forgetting to catch it, silently turning a "nice-to-have" feature into a hard dependency. A None-returning function makes the degrade path the only path for callers to write, not an opt-in one.

D3: Structured output via schema-constrained response, not free-text parsing or prompt-based JSON coaxing

The enrichment call uses the Messages API's structured-output mechanism (output_config: {format: {type: "json_schema", schema: {...}}}, or the SDK's messages.parse() helper with a SemanticScene-shaped Pydantic model) so the response is schema-valid JSON by construction, rather than asking the model to "reply with JSON" in the prompt and parsing/repairing free text. This directly eliminates an entire class of degrade-path triggers (truncated/wrapped/commented JSON) that a prompt-only approach would otherwise have to detect and handle. Alternative considered: frame enrichment as a tool call with strict: true tool-input validation instead of a direct structured message reply. Rejected for this milestone — a direct structured response is simpler (no tool-loop bookkeeping) for a single, non-interactive extraction call; the tool-call pattern is more valuable when the model needs to choose between actions, which does not apply here.

D4: Default model is a small/fast tier (claude-haiku-4-5), overridable via config

Enrichment is a bounded, low-complexity extraction task (label a page, list a handful of intents, tag widgets already enumerated in the Scene) invoked once per Observe step, so latency and per-call cost dominate the choice over raw capability. claude-haiku-4-5 is the default: cheapest/fastest tier that still supports schema-constrained structured output. The model is a config value, not hardcoded, so a later milestone can upgrade the default if enrichment quality proves insufficient in practice without a code change. Alternative considered: default to the project's most capable tier (as a generic "always use the best model" policy would suggest). Rejected specifically for this call site — the enrichment task is simple extraction over an already-structured Scene (not open-ended reasoning), and this call sits in the hot path of every Observe step, where added latency directly slows the whole task loop. A stronger model remains available via config for callers who need it (e.g. a harder-to-classify page).

D5: No extended-thinking / effort tuning on the enrichment call

The enrichment call omits thinking entirely and does not set output_config.effort. claude-haiku-4-5 is a fast-extraction model; forcing extra reasoning depth on a bounded-output classification/labeling task increases latency and cost with no expected quality benefit here, and some effort/thinking parameter combinations are rejected outright on newer model tiers depending on model family. Keeping the request shape minimal (system prompt + schema + scene JSON) also maximizes what can be prompt-cached (see D6). Alternative considered: enable adaptive thinking for robustness on ambiguous screens. Rejected for the default path — if a later milestone finds enrichment quality insufficient on complex/ambiguous screens, this is a config-level model/parameter change, not a structural one.

D6: Prompt caching on the fixed system/schema prefix

The enrichment system prompt (instructions + JSON schema description) is static across every call within a task (and across tasks), while only the Scene JSON body varies per call. The client marks the system-prompt block with a cache_control: {"type": "ephemeral"} breakpoint so repeated per-step calls within a task reuse the cached prefix instead of paying full input-token price every step. This is a pure cost optimization with no behavior change; if the fixed prefix is ever short enough to fall under a given model's minimum cacheable-prefix length, caching simply has no effect (not an error) — the design must not depend on caching succeeding for correctness. Alternative considered: no caching, accept repeated system-prompt cost. Rejected — enrichment is invoked once per Observe step, so a multi-step task repeats the identical instructions/schema text on every call; caching this prefix is a low-effort, no-risk win once the per-step call pattern exists.

D7: New describe_screen_semantic tool wraps the existing tool, existing tool untouched

A new tools/describe_screen_semantic.py calls the existing tools/describe_screen.py to get a Scene, then calls enrich_scene(), and returns {scene, semantic_scene} (with semantic_scene possibly None). tools/describe_screen.py itself is not modified — existing callers (including the current Planner stub and any wiring in runtime/executor.py's default_tool_registry()) keep working unchanged, and the new tool is additive to the registry. Alternative considered: add an enrich: bool = False kwarg directly to describe_screen(). Rejected — it would make tools/describe_screen.py depend on the new semantic/ package (and transitively on LLM client config) even for callers who never pass enrich=True, which cuts against scene-perception's existing "no LLM dependency" boundary. A separate wrapper tool keeps the dependency opt-in at the import level, not just the call-site level.

Risks / Trade-offs

• [Risk] Enrichment latency adds to every Observe step even when unused by the caller → Mitigation: enrichment is only invoked through the new describe_screen_semantic tool / explicit enrich_scene() call, never automatically inside the existing describe_screen; callers who don't need it pay zero extra latency.
• [Risk] LLM cost scales with task step count (one call per Observe step) → Mitigation: cheap default model tier (D4) + prompt caching of the fixed prefix (D6); config exposes a global enable/disable switch so cost-sensitive environments (CI, offline dev) can turn enrichment off entirely.
• [Risk] Enrichment quality (wrong page label, missed intent, mislabeled widget purpose) is not directly testable against real model output in unit tests → Mitigation: llm_client is injected/mockable in enricher.py, so unit tests exercise the parsing/degrade-path logic against canned responses; only a small, explicitly-marked integration test exercises the real API (matching the existing apex-agent-mvp pattern of skippable hardware/network integration tests).
• [Risk] element_id values in widgets[].element_id could drift from the Scene's actual element IDs if the model hallucinates or omits one → Mitigation: enricher.py validates every returned element_id against the input Scene's element ID set post-response; any widget referencing an unknown ID is dropped (not treated as a hard failure) rather than propagated as a dangling reference.
• [Risk] Introducing the project's first external LLM dependency adds a new operational failure mode (network, auth, rate limits) that didn't exist before → Mitigation: this is exactly why D2's None-returning contract and the mandatory fallback-to-raw-Scene path exist; the entire capability is designed so this failure mode degrades gracefully rather than being a new way for tasks to fail.

Migration Plan

This is a purely additive change with no rollback complexity beyond removing the new package and config:

1. Add the semantic/ package and tools/describe_screen_semantic.py; wire the new tool into runtime/executor.py's default_tool_registry() under its own key (e.g. "describe_screen_semantic"), alongside (not replacing) the existing describe_screen entry.
2. Add the anthropic SDK dependency and semantic* to pyproject.toml's package-find include list; add enrichment config (enabled/disabled, model name) with enrichment defaulting to disabled until a caller explicitly opts in, so applying this change never silently changes existing task behavior or cost.
3. No data migration, no changes to storage/'s timeline format in this change — if a later milestone wants SemanticScene persisted per step, that is a task-memory capability change to propose separately.
4. Rollback is simply not calling the new tool / leaving enrichment disabled in config; no existing capability needs to be reverted.

Open Questions

• Should SemanticScene (when present) be recorded in TaskContext/the timeline alongside Scene, or is it purely a same-step, non-persisted artifact until World Runtime (Milestone 6) defines cross-step state? Leaning toward "not persisted in this change" to keep the boundary with Milestone 6 clean, but this affects whether runtime/task.py needs any change at all in this milestone.
• Should the default enrichment model become configurable per-task (e.g. a harder app might warrant a stronger tier) or is a single global default sufficient until real usage data exists? Leaning toward global-default-only for this milestone.
• Exact timeout budget for the enrichment call (e.g. 5s vs 10s) before triggering the degrade path — needs to be tuned against real latency data once implemented; not fixed in this design.
• Should intents be a fully open-vocabulary list (whatever the model says) or validated against a small controlled vocabulary to keep them stable/comparable across steps and pages? Leaning open-vocabulary for this milestone since no downstream consumer (yet) needs a fixed taxonomy; revisit if World/Skill milestones need one.