spec

2026-07-06 23:52:53 +08:00
parent d899b875ce
commit 8e29fcf3c2
54 changed files with 736 additions and 0 deletions
@@ -0,0 +1,49 @@
+# semantic-scene Specification
+
+## Purpose
+TBD - created by archiving change semantic-scene-runtime. Update Purpose after archive.
+## Requirements
+### Requirement: Semantic scene enrichment from an existing Scene
+The system SHALL provide a function that, given an existing `Scene` (as produced by the `scene-perception` capability), produces a `SemanticScene` consisting of a page identity string, a list of plain-language supported intents, and a list of per-widget purpose labels referencing the `Scene`'s element IDs, using exactly one LLM call.
+
+#### Scenario: Enrichment succeeds for a recognizable screen
+- **WHEN** `enrich_scene()` is called with a `Scene` describing a recognizable app screen (e.g. a chat screen with a text input and a send button)
+- **THEN** it returns a `SemanticScene` with a non-empty `page` string, a non-empty `intents` list of plain-language strings, and a `widgets` list where each entry's `element_id` matches an element ID present in the input `Scene`
+
+#### Scenario: Widget purpose labels reference only known elements
+- **WHEN** the LLM response includes a widget purpose label whose `element_id` does not match any element ID in the input `Scene`
+- **THEN** `enrich_scene()` discards that widget entry from the returned `SemanticScene` rather than propagating a dangling element reference
+
+### Requirement: Enrichment failure degrades to no semantic scene, never blocks the loop
+The system SHALL treat any enrichment failure (LLM call timeout, connection error, rate limit, malformed or schema-invalid response, or enrichment disabled by configuration) as a non-fatal condition, returning an absence of a semantic scene rather than raising an exception, so that callers always have a defined fallback of using the raw `Scene`.
+
+#### Scenario: LLM call times out
+- **WHEN** the enrichment LLM call does not complete within the configured timeout
+- **THEN** `enrich_scene()` returns `None` and the caller proceeds using the raw `Scene` without the task step being marked as failed
+
+#### Scenario: LLM response fails schema validation
+- **WHEN** the enrichment LLM call returns a response that does not conform to the expected `SemanticScene` JSON schema
+- **THEN** `enrich_scene()` returns `None` instead of raising, and no partially-parsed `SemanticScene` is returned
+
+#### Scenario: Enrichment disabled by configuration
+- **WHEN** semantic enrichment is disabled in configuration
+- **THEN** `enrich_scene()` returns `None` immediately without making an LLM call
+
+### Requirement: Structured, schema-constrained LLM output
+The system SHALL request the enrichment LLM call using a schema-constrained structured-output mechanism so that any successful response is guaranteed to be valid JSON matching the `SemanticScene` shape, rather than relying on free-text parsing of an unconstrained model reply.
+
+#### Scenario: Successful call yields directly parseable output
+- **WHEN** the enrichment LLM call completes successfully
+- **THEN** the raw response body is valid JSON matching the declared `SemanticScene` schema without requiring text extraction, regex matching, or a JSON-repair step
+
+### Requirement: Semantic enrichment is opt-in and does not alter existing tool behavior
+The system SHALL expose semantic enrichment through a new, separate tool entry point rather than modifying the existing `describe_screen` tool's signature or behavior, so that every existing caller of `describe_screen` continues to receive only a `Scene`, unchanged, unless it explicitly opts into the new semantic-enriched entry point.
+
+#### Scenario: Existing describe_screen callers are unaffected
+- **WHEN** an existing caller invokes the `describe_screen` tool as it did before this change
+- **THEN** it receives the same `Scene` result as before, with no semantic enrichment attempted and no new LLM-related dependency invoked
+
+#### Scenario: A caller opts into semantic enrichment
+- **WHEN** a caller invokes the new semantic-enrichment tool entry point for a given device
+- **THEN** it receives both the underlying `Scene` and, when enrichment succeeds, the corresponding `SemanticScene`; when enrichment fails or is disabled, it receives the `Scene` with an explicit absence of a `SemanticScene` rather than a partial or error result
+