43 lines
5.0 KiB
Markdown
43 lines
5.0 KiB
Markdown
## 1. Package scaffolding
|
|
|
|
- [x] 1.1 Create the `semantic/` package (`__init__.py`, `models.py`, `llm_client.py`, `enricher.py`, `prompts.py`)
|
|
- [x] 1.2 Add the `anthropic` SDK dependency and a `semantic*` entry to `[tool.setuptools.packages.find].include` in `pyproject.toml`
|
|
- [x] 1.3 Add enrichment configuration: enabled/disabled flag (default disabled), model name, and timeout, sourced from environment/config in a single place `semantic/` reads from
|
|
- [x] 1.4 Extend the project's smoke test (that imports every package) to import `semantic`
|
|
|
|
## 2. SemanticScene data model (capability: semantic-scene)
|
|
|
|
- [x] 2.1 Implement `semantic/models.py`: `SemanticWidget` (`element_id: str`, `purpose: str`) and `SemanticScene` (`page: str`, `intents: list[str]`, `widgets: list[SemanticWidget]`) dataclasses with `to_dict()`/`from_dict()` mirroring the style of `core/models.py`
|
|
- [x] 2.2 Define the JSON schema for `SemanticScene` used to constrain the LLM's structured output, matching the `SemanticScene` dataclass shape exactly
|
|
- [x] 2.3 Write unit tests for `SemanticScene`/`SemanticWidget` round-tripping through `to_dict()`/`from_dict()`
|
|
|
|
## 3. LLM client abstraction (capability: semantic-scene)
|
|
|
|
- [x] 3.1 Implement `semantic/llm_client.py`: a narrow client wrapper around the Anthropic SDK exposing a single `enrich(scene_json: dict, *, timeout: float) -> dict` method that issues one structured-output (`output_config.format` / schema-constrained) call and returns the parsed JSON body
|
|
- [x] 3.2 Configure the enrichment system prompt/schema block with a `cache_control: {"type": "ephemeral"}` breakpoint so repeated per-step calls reuse the cached prefix
|
|
- [x] 3.3 Map the SDK's typed exceptions (timeout, connection error, rate limit, authentication error, API status error) into a single internal `EnrichmentUnavailable` signal consumed only inside `semantic/` (never re-raised past `enricher.py`)
|
|
- [x] 3.4 Make the client injectable/mockable (constructor parameter or factory function) so tests can supply a fake client with canned responses instead of calling the network
|
|
- [x] 3.5 Write unit tests for the client wrapper against a fake transport covering: success, timeout, rate limit, malformed/schema-invalid JSON response
|
|
|
|
## 4. Enrichment pass and degrade path (capability: semantic-scene)
|
|
|
|
- [x] 4.1 Implement `semantic/prompts.py`: the enrichment system prompt describing the page-identity/intents/widget-purpose task and referencing the input `Scene`'s elements by ID
|
|
- [x] 4.2 Implement `semantic/enricher.py`: `enrich_scene(scene: Scene, *, client=None) -> SemanticScene | None`, serializing the `Scene` to JSON, calling the LLM client, and parsing the result into a `SemanticScene`
|
|
- [x] 4.3 Add post-response validation in `enricher.py`: drop any `widgets[]` entry whose `element_id` does not match an element ID present in the input `Scene`
|
|
- [x] 4.4 Make `enrich_scene()` return `None` (never raise) for every expected failure mode: disabled config, `EnrichmentUnavailable` from the client, or a `SemanticScene` that fails post-response validation entirely
|
|
- [x] 4.5 Write unit tests for `enrich_scene()` covering: successful enrichment, disabled-by-config short-circuit (no client call made), degrade-to-`None` on client failure, and dangling-`element_id` filtering
|
|
- [x] 4.6 Write an integration test (skippable without network/API credentials, following the existing `apex-agent-mvp` skippable-integration-test pattern) that calls the real LLM client against a fixture `Scene` and asserts a schema-valid `SemanticScene` is returned
|
|
|
|
## 5. Tool integration (capability: semantic-scene)
|
|
|
|
- [x] 5.1 Implement `tools/describe_screen_semantic.py`: calls the existing `tools/describe_screen.py` for a `Scene`, then `enrich_scene()`, returning `{"scene": Scene, "semantic_scene": SemanticScene | None}`, following the same `device_id`/`manager`/`ocr_engine` kwarg pattern as `describe_screen`
|
|
- [x] 5.2 Register the new tool under its own key (e.g. `"describe_screen_semantic"`) in `runtime/executor.py`'s `default_tool_registry()`, additive alongside the existing `"describe_screen"` entry
|
|
- [x] 5.3 Write a unit test asserting `tools/describe_screen.py`'s existing behavior and return type are unchanged after this change (no accidental coupling to `semantic/`)
|
|
- [x] 5.4 Write a unit test for `describe_screen_semantic` covering both the enrichment-succeeds and enrichment-degrades-to-`None` cases, using a fake/mock LLM client
|
|
|
|
## 6. End-to-end validation
|
|
|
|
- [x] 6.1 Write an end-to-end test simulating an Observe step that calls `describe_screen_semantic` against a mocked `Driver`/`Scene` and a mocked LLM client, asserting the task loop completes normally whether enrichment succeeds or is forced to fail
|
|
- [x] 6.2 Confirm enrichment stays disabled by default after applying this change (no existing task's behavior, latency, or cost changes unless a caller explicitly enables it and opts into the new tool)
|
|
- [x] 6.3 Run the full test suite (`pytest`) and confirm no existing test in `tests/` needed a behavior change, only additive new tests
|