semantic-scene Specification

Purpose

TBD - created by archiving change semantic-scene-runtime. Update Purpose after archive.

Requirements

Requirement: Semantic scene enrichment from an existing Scene

The system SHALL provide a function that, given an existing Scene (as produced by the scene-perception capability), produces a SemanticScene consisting of a page identity string, a list of plain-language supported intents, and a list of per-widget purpose labels referencing the Scene's element IDs, using exactly one LLM call.

Scenario: Enrichment succeeds for a recognizable screen

WHEN enrich_scene() is called with a Scene describing a recognizable app screen (e.g. a chat screen with a text input and a send button)
THEN it returns a SemanticScene with a non-empty page string, a non-empty intents list of plain-language strings, and a widgets list where each entry's element_id matches an element ID present in the input Scene

WHEN the LLM response includes a widget purpose label whose element_id does not match any element ID in the input Scene
THEN enrich_scene() discards that widget entry from the returned SemanticScene rather than propagating a dangling element reference

Requirement: Enrichment failure degrades to no semantic scene, never blocks the loop

The system SHALL treat any enrichment failure (LLM call timeout, connection error, rate limit, malformed or schema-invalid response, or enrichment disabled by configuration) as a non-fatal condition, returning an absence of a semantic scene rather than raising an exception, so that callers always have a defined fallback of using the raw Scene.

Scenario: LLM call times out

WHEN the enrichment LLM call does not complete within the configured timeout
THEN enrich_scene() returns None and the caller proceeds using the raw Scene without the task step being marked as failed

Scenario: LLM response fails schema validation

WHEN the enrichment LLM call returns a response that does not conform to the expected SemanticScene JSON schema
THEN enrich_scene() returns None instead of raising, and no partially-parsed SemanticScene is returned

Scenario: Enrichment disabled by configuration

WHEN semantic enrichment is disabled in configuration
THEN enrich_scene() returns None immediately without making an LLM call

Requirement: Structured, schema-constrained LLM output

The system SHALL request the enrichment LLM call using a schema-constrained structured-output mechanism so that any successful response is guaranteed to be valid JSON matching the SemanticScene shape, rather than relying on free-text parsing of an unconstrained model reply.

Scenario: Successful call yields directly parseable output

WHEN the enrichment LLM call completes successfully
THEN the raw response body is valid JSON matching the declared SemanticScene schema without requiring text extraction, regex matching, or a JSON-repair step

Requirement: Semantic enrichment is opt-in and does not alter existing tool behavior

The system SHALL expose semantic enrichment through a new, separate tool entry point rather than modifying the existing describe_screen tool's signature or behavior, so that every existing caller of describe_screen continues to receive only a Scene, unchanged, unless it explicitly opts into the new semantic-enriched entry point.

Scenario: Existing describe_screen callers are unaffected

WHEN an existing caller invokes the describe_screen tool as it did before this change
THEN it receives the same Scene result as before, with no semantic enrichment attempted and no new LLM-related dependency invoked

Scenario: A caller opts into semantic enrichment

WHEN a caller invokes the new semantic-enrichment tool entry point for a given device
THEN it receives both the underlying Scene and, when enrichment succeeds, the corresponding SemanticScene; when enrichment fails or is disabled, it receives the Scene with an explicit absence of a SemanticScene rather than a partial or error result

4.2 KiB Raw Permalink Blame History