agentic-mobile-control/openspec/changes/archive/2026-07-06-semantic-scene-runtime/tasks.md

1. Package scaffolding

• 1.1 Create the semantic/ package (__init__.py, models.py, llm_client.py, enricher.py, prompts.py)
• 1.2 Add the anthropic SDK dependency and a semantic* entry to [tool.setuptools.packages.find].include in pyproject.toml
• 1.3 Add enrichment configuration: enabled/disabled flag (default disabled), model name, and timeout, sourced from environment/config in a single place semantic/ reads from
• 1.4 Extend the project's smoke test (that imports every package) to import semantic

2. SemanticScene data model (capability: semantic-scene)

• 2.1 Implement semantic/models.py: SemanticWidget (element_id: str, purpose: str) and SemanticScene (page: str, intents: list[str], widgets: list[SemanticWidget]) dataclasses with to_dict()/from_dict() mirroring the style of core/models.py
• 2.2 Define the JSON schema for SemanticScene used to constrain the LLM's structured output, matching the SemanticScene dataclass shape exactly
• 2.3 Write unit tests for SemanticScene/SemanticWidget round-tripping through to_dict()/from_dict()

3. LLM client abstraction (capability: semantic-scene)

• 3.1 Implement semantic/llm_client.py: a narrow client wrapper around the Anthropic SDK exposing a single enrich(scene_json: dict, *, timeout: float) -> dict method that issues one structured-output (output_config.format / schema-constrained) call and returns the parsed JSON body
• 3.2 Configure the enrichment system prompt/schema block with a cache_control: {"type": "ephemeral"} breakpoint so repeated per-step calls reuse the cached prefix
• 3.3 Map the SDK's typed exceptions (timeout, connection error, rate limit, authentication error, API status error) into a single internal EnrichmentUnavailable signal consumed only inside semantic/ (never re-raised past enricher.py)
• 3.4 Make the client injectable/mockable (constructor parameter or factory function) so tests can supply a fake client with canned responses instead of calling the network
• 3.5 Write unit tests for the client wrapper against a fake transport covering: success, timeout, rate limit, malformed/schema-invalid JSON response

4. Enrichment pass and degrade path (capability: semantic-scene)

• 4.1 Implement semantic/prompts.py: the enrichment system prompt describing the page-identity/intents/widget-purpose task and referencing the input Scene's elements by ID
• 4.2 Implement semantic/enricher.py: enrich_scene(scene: Scene, *, client=None) -> SemanticScene | None, serializing the Scene to JSON, calling the LLM client, and parsing the result into a SemanticScene
• 4.3 Add post-response validation in enricher.py: drop any widgets[] entry whose element_id does not match an element ID present in the input Scene
• 4.4 Make enrich_scene() return None (never raise) for every expected failure mode: disabled config, EnrichmentUnavailable from the client, or a SemanticScene that fails post-response validation entirely
• 4.5 Write unit tests for enrich_scene() covering: successful enrichment, disabled-by-config short-circuit (no client call made), degrade-to-None on client failure, and dangling-element_id filtering
• 4.6 Write an integration test (skippable without network/API credentials, following the existing apex-agent-mvp skippable-integration-test pattern) that calls the real LLM client against a fixture Scene and asserts a schema-valid SemanticScene is returned

5. Tool integration (capability: semantic-scene)

• 5.1 Implement tools/describe_screen_semantic.py: calls the existing tools/describe_screen.py for a Scene, then enrich_scene(), returning {"scene": Scene, "semantic_scene": SemanticScene | None}, following the same device_id/manager/ocr_engine kwarg pattern as describe_screen
• 5.2 Register the new tool under its own key (e.g. "describe_screen_semantic") in runtime/executor.py's default_tool_registry(), additive alongside the existing "describe_screen" entry
• 5.3 Write a unit test asserting tools/describe_screen.py's existing behavior and return type are unchanged after this change (no accidental coupling to semantic/)
• 5.4 Write a unit test for describe_screen_semantic covering both the enrichment-succeeds and enrichment-degrades-to-None cases, using a fake/mock LLM client

6. End-to-end validation

• 6.1 Write an end-to-end test simulating an Observe step that calls describe_screen_semantic against a mocked Driver/Scene and a mocked LLM client, asserting the task loop completes normally whether enrichment succeeds or is forced to fail
• 6.2 Confirm enrichment stays disabled by default after applying this change (no existing task's behavior, latency, or cost changes unless a caller explicitly enables it and opts into the new tool)
• 6.3 Run the full test suite (pytest) and confirm no existing test in tests/ needed a behavior change, only additive new tests