feat: checkpoint device agent runtime milestones
This commit is contained in:
@@ -0,0 +1,2 @@
|
||||
schema: spec-driven
|
||||
created: 2026-07-06
|
||||
@@ -0,0 +1,74 @@
|
||||
## Context
|
||||
|
||||
`apex-agent-mvp` (code-complete, unapplied) already produces a `Scene` for every Observe step: `core/models.py` defines `Scene { width, height, elements: [SceneElement] }` and `SceneElement { id, type, bounds, text, confidence, source }`, built by `vision/scene_builder.py` (planned rename: `perception/scene_builder.py` per `device-agent-runtime-foundation`) by fusing OCR output and the device's UI tree via bounding-box IoU. `runtime/planner.py` (`Planner.plan()`) is currently a stub that returns one hardcoded `describe_screen` step and then nothing — there is no real LLM call anywhere in the codebase yet. This change introduces the **first real LLM integration point**: a single enrichment call per Observe step that turns a `Scene` into a compact `SemanticScene` (page identity, supported intents, per-widget purpose labels), so a real LLM-driven `Planner` (a later milestone) and other prompts can consume a stable, low-token JSON summary instead of re-deriving page semantics from raw element geometry or a screenshot every step.
|
||||
|
||||
Because this is the first LLM call in the project, this design also sets the pattern (client shape, structured-output mechanism, failure handling) that later milestones (a real `Planner`, World Runtime's cross-step memory, Skill synthesis) are expected to reuse rather than inventing their own.
|
||||
|
||||
## Goals / Non-Goals
|
||||
|
||||
**Goals:**
|
||||
- Given a `Scene`, produce a `SemanticScene` — `{page: str, intents: list[str], widgets: [{element_id, purpose}]}` — using exactly one LLM call.
|
||||
- Guarantee the enrichment call never blocks or fails the Observe→Act loop: any failure (timeout, rate limit, malformed response, disabled config) degrades to "no semantic scene," and callers fall back to the raw `Scene`.
|
||||
- Keep the LLM client abstraction narrow and swappable (mockable in tests without network access), scoped inside the new `semantic/` package rather than becoming a project-wide `llm/` dependency other capabilities must adopt.
|
||||
- Produce output that is schema-valid JSON by construction (not "usually valid, occasionally needs a repair pass"), since downstream code will parse it directly into `SemanticScene`.
|
||||
- Make enrichment optional and cheap enough to run on every Observe step without materially changing task latency or cost profile.
|
||||
|
||||
**Non-Goals:**
|
||||
- No change to `scene-perception`'s `Scene` format or fusion logic — `SemanticScene` is a separate, additive artifact layered on top, never a replacement.
|
||||
- No persistent cross-step semantic state, caching of `SemanticScene` across steps, or diffing between steps — that is Milestone 6 (World Runtime).
|
||||
- No skill synthesis or action recommendation derived from intents/purposes — that is Milestone 7 (Skill).
|
||||
- No change to `Planner`/`Executor` control flow itself — this change only makes a new artifact available; wiring a real LLM-driven `Planner` to consume `SemanticScene` is left to a later milestone (see Migration Plan).
|
||||
- No multi-provider LLM abstraction (e.g. supporting both Anthropic and OpenAI behind a common interface) — a single concrete client is enough for this milestone; a provider-agnostic port can be extracted later if a second provider is actually needed (YAGNI).
|
||||
|
||||
## Decisions
|
||||
|
||||
### D1: New `semantic/` package, not folded into `perception/` or `runtime/`
|
||||
`SemanticScene` is conceptually "one layer above Scene," but it has fundamentally different runtime characteristics: it makes a network call, it can fail/timeout, and it is optional. Mixing it into `perception/` (which is currently synchronous, local-only, and always-on) would force every `perception/` consumer to reason about network failure modes it doesn't otherwise have. A sibling `semantic/` package (`models.py`, `llm_client.py`, `enricher.py`, `prompts.py`) keeps `perception/`'s contract unchanged and makes the enrichment layer's optionality explicit at the package boundary.
|
||||
**Alternative considered**: add an `enrich()` method directly on `Scene` or inside `scene_builder.py`. Rejected — it would make `perception/` depend on an LLM SDK and network config, contradicting `device-agent-runtime-foundation`'s stated dependency direction (`perception` must stay a local, deterministic pipeline; LLM-backed behavior is explicitly scoped to sit above it, closer to `runtime`).
|
||||
|
||||
### D2: `enrich_scene()` returns `SemanticScene | None`, never raises for expected failure modes
|
||||
`enricher.enrich_scene(scene, *, context=None) -> SemanticScene | None` catches all expected LLM-client failure modes (timeout, rate limit, connection error, malformed/schema-invalid response, enrichment disabled by config) internally and returns `None` on any of them, logging the reason. Callers (tools, later a real `Planner`) treat `None` as "use the raw `Scene`," never as an exception to handle.
|
||||
**Alternative considered**: raise a typed `EnrichmentUnavailableError` and require every caller to catch it. Rejected — experience from `runtime/executor.py`'s existing retry/backoff pattern shows try/except-per-call-site is exactly the kind of boilerplate that leads to a caller eventually forgetting to catch it, silently turning a "nice-to-have" feature into a hard dependency. A `None`-returning function makes the degrade path the *only* path for callers to write, not an opt-in one.
|
||||
|
||||
### D3: Structured output via schema-constrained response, not free-text parsing or prompt-based JSON coaxing
|
||||
The enrichment call uses the Messages API's structured-output mechanism (`output_config: {format: {type: "json_schema", schema: {...}}}`, or the SDK's `messages.parse()` helper with a `SemanticScene`-shaped Pydantic model) so the response is schema-valid JSON by construction, rather than asking the model to "reply with JSON" in the prompt and parsing/repairing free text. This directly eliminates an entire class of degrade-path triggers (truncated/wrapped/commented JSON) that a prompt-only approach would otherwise have to detect and handle.
|
||||
**Alternative considered**: frame enrichment as a tool call with `strict: true` tool-input validation instead of a direct structured message reply. Rejected for this milestone — a direct structured response is simpler (no tool-loop bookkeeping) for a single, non-interactive extraction call; the tool-call pattern is more valuable when the model needs to choose between actions, which does not apply here.
|
||||
|
||||
### D4: Default model is a small/fast tier (`claude-haiku-4-5`), overridable via config
|
||||
Enrichment is a bounded, low-complexity extraction task (label a page, list a handful of intents, tag widgets already enumerated in the `Scene`) invoked once per Observe step, so latency and per-call cost dominate the choice over raw capability. `claude-haiku-4-5` is the default: cheapest/fastest tier that still supports schema-constrained structured output. The model is a config value, not hardcoded, so a later milestone can upgrade the default if enrichment quality proves insufficient in practice without a code change.
|
||||
**Alternative considered**: default to the project's most capable tier (as a generic "always use the best model" policy would suggest). Rejected specifically for this call site — the enrichment task is simple extraction over an already-structured `Scene` (not open-ended reasoning), and this call sits in the hot path of every Observe step, where added latency directly slows the whole task loop. A stronger model remains available via config for callers who need it (e.g. a harder-to-classify page).
|
||||
|
||||
### D5: No extended-thinking / effort tuning on the enrichment call
|
||||
The enrichment call omits `thinking` entirely and does not set `output_config.effort`. `claude-haiku-4-5` is a fast-extraction model; forcing extra reasoning depth on a bounded-output classification/labeling task increases latency and cost with no expected quality benefit here, and some effort/thinking parameter combinations are rejected outright on newer model tiers depending on model family. Keeping the request shape minimal (system prompt + schema + scene JSON) also maximizes what can be prompt-cached (see D6).
|
||||
**Alternative considered**: enable adaptive thinking for robustness on ambiguous screens. Rejected for the default path — if a later milestone finds enrichment quality insufficient on complex/ambiguous screens, this is a config-level model/parameter change, not a structural one.
|
||||
|
||||
### D6: Prompt caching on the fixed system/schema prefix
|
||||
The enrichment system prompt (instructions + JSON schema description) is static across every call within a task (and across tasks), while only the `Scene` JSON body varies per call. The client marks the system-prompt block with a `cache_control: {"type": "ephemeral"}` breakpoint so repeated per-step calls within a task reuse the cached prefix instead of paying full input-token price every step. This is a pure cost optimization with no behavior change; if the fixed prefix is ever short enough to fall under a given model's minimum cacheable-prefix length, caching simply has no effect (not an error) — the design must not depend on caching succeeding for correctness.
|
||||
**Alternative considered**: no caching, accept repeated system-prompt cost. Rejected — enrichment is invoked once per Observe step, so a multi-step task repeats the identical instructions/schema text on every call; caching this prefix is a low-effort, no-risk win once the per-step call pattern exists.
|
||||
|
||||
### D7: New `describe_screen_semantic` tool wraps the existing tool, existing tool untouched
|
||||
A new `tools/describe_screen_semantic.py` calls the existing `tools/describe_screen.py` to get a `Scene`, then calls `enrich_scene()`, and returns `{scene, semantic_scene}` (with `semantic_scene` possibly `None`). `tools/describe_screen.py` itself is not modified — existing callers (including the current `Planner` stub and any wiring in `runtime/executor.py`'s `default_tool_registry()`) keep working unchanged, and the new tool is additive to the registry.
|
||||
**Alternative considered**: add an `enrich: bool = False` kwarg directly to `describe_screen()`. Rejected — it would make `tools/describe_screen.py` depend on the new `semantic/` package (and transitively on LLM client config) even for callers who never pass `enrich=True`, which cuts against `scene-perception`'s existing "no LLM dependency" boundary. A separate wrapper tool keeps the dependency opt-in at the import level, not just the call-site level.
|
||||
|
||||
## Risks / Trade-offs
|
||||
|
||||
- **[Risk] Enrichment latency adds to every Observe step even when unused by the caller** → Mitigation: enrichment is only invoked through the new `describe_screen_semantic` tool / explicit `enrich_scene()` call, never automatically inside the existing `describe_screen`; callers who don't need it pay zero extra latency.
|
||||
- **[Risk] LLM cost scales with task step count (one call per Observe step)** → Mitigation: cheap default model tier (D4) + prompt caching of the fixed prefix (D6); config exposes a global enable/disable switch so cost-sensitive environments (CI, offline dev) can turn enrichment off entirely.
|
||||
- **[Risk] Enrichment quality (wrong page label, missed intent, mislabeled widget purpose) is not directly testable against real model output in unit tests** → Mitigation: `llm_client` is injected/mockable in `enricher.py`, so unit tests exercise the parsing/degrade-path logic against canned responses; only a small, explicitly-marked integration test exercises the real API (matching the existing `apex-agent-mvp` pattern of skippable hardware/network integration tests).
|
||||
- **[Risk] `element_id` values in `widgets[].element_id` could drift from the `Scene`'s actual element IDs if the model hallucinates or omits one** → Mitigation: `enricher.py` validates every returned `element_id` against the input `Scene`'s element ID set post-response; any widget referencing an unknown ID is dropped (not treated as a hard failure) rather than propagated as a dangling reference.
|
||||
- **[Risk] Introducing the project's first external LLM dependency adds a new operational failure mode (network, auth, rate limits) that didn't exist before** → Mitigation: this is exactly why D2's `None`-returning contract and the mandatory fallback-to-raw-`Scene` path exist; the entire capability is designed so this failure mode degrades gracefully rather than being a new way for tasks to fail.
|
||||
|
||||
## Migration Plan
|
||||
|
||||
This is a purely additive change with no rollback complexity beyond removing the new package and config:
|
||||
1. Add the `semantic/` package and `tools/describe_screen_semantic.py`; wire the new tool into `runtime/executor.py`'s `default_tool_registry()` under its own key (e.g. `"describe_screen_semantic"`), alongside (not replacing) the existing `describe_screen` entry.
|
||||
2. Add the `anthropic` SDK dependency and `semantic*` to `pyproject.toml`'s package-find include list; add enrichment config (enabled/disabled, model name) with enrichment defaulting to **disabled** until a caller explicitly opts in, so applying this change never silently changes existing task behavior or cost.
|
||||
3. No data migration, no changes to `storage/`'s timeline format in this change — if a later milestone wants `SemanticScene` persisted per step, that is a `task-memory` capability change to propose separately.
|
||||
4. Rollback is simply not calling the new tool / leaving enrichment disabled in config; no existing capability needs to be reverted.
|
||||
|
||||
## Open Questions
|
||||
|
||||
- Should `SemanticScene` (when present) be recorded in `TaskContext`/the timeline alongside `Scene`, or is it purely a same-step, non-persisted artifact until World Runtime (Milestone 6) defines cross-step state? Leaning toward "not persisted in this change" to keep the boundary with Milestone 6 clean, but this affects whether `runtime/task.py` needs any change at all in this milestone.
|
||||
- Should the default enrichment model become configurable per-task (e.g. a harder app might warrant a stronger tier) or is a single global default sufficient until real usage data exists? Leaning toward global-default-only for this milestone.
|
||||
- Exact timeout budget for the enrichment call (e.g. 5s vs 10s) before triggering the degrade path — needs to be tuned against real latency data once implemented; not fixed in this design.
|
||||
- Should intents be a fully open-vocabulary list (whatever the model says) or validated against a small controlled vocabulary to keep them stable/comparable across steps and pages? Leaning open-vocabulary for this milestone since no downstream consumer (yet) needs a fixed taxonomy; revisit if World/Skill milestones need one.
|
||||
@@ -0,0 +1,28 @@
|
||||
## Why
|
||||
|
||||
`scene-perception` (from `apex-agent-mvp`, code-complete but unapplied) fuses a screenshot, the device's UI tree, and OCR output into a `Scene`: screen dimensions plus a flat list of typed elements (bounds, text, confidence, source). That is still a *structural* description of the screen — it tells the LLM "there is a button-shaped element at (x, y) with text 'Send'," not "this is the WeChat chat screen and tapping that button sends the message." Every planning step, the LLM has to re-derive page identity and intent from raw element geometry, which burns context on repeated low-level reasoning and is brittle to layout drift (the same page re-derived slightly differently step to step). This change adds a **Semantic Runtime**: a single additional LLM enrichment call that turns a `Scene` into a `SemanticScene` — page identity, a short list of plain-language supported intents, and a purpose label per widget — so most subsequent prompts can rely on a compact, stable JSON summary instead of re-parsing raw geometry or screenshots. This is Milestone 5 (Semantic) of the device-agnostic runtime roadmap established by `device-agent-runtime-foundation`, sitting directly on top of the existing Perception milestone's `Scene` output.
|
||||
|
||||
## What Changes
|
||||
|
||||
- Add a new `semantic/` package that consumes an existing `Scene` (from `perception/`, per the `scene-perception` capability) and produces a `SemanticScene`: `{page: str, intents: list[str], widgets: [{element_id: str, purpose: str}]}`.
|
||||
- Introduce the **first real LLM integration point** in the codebase: a narrow, swappable LLM client abstraction used only by `semantic/` to make one structured-output call per enrichment (model, prompt, and structured-schema details are an implementation decision in `design.md`, not part of this proposal's contract).
|
||||
- Define a mandatory **degrade path**: if the enrichment LLM call is unavailable, times out, or fails for any reason, semantic enrichment is skipped and callers fall back to using the raw `Scene` directly — enrichment failure must never block or fail the Observe→Act loop.
|
||||
- Add a new tool-level entry point (e.g. `describe_screen_semantic` or an `enrich` flag on the existing describe-screen flow) so `runtime/` and `api/` callers can opt into the enriched view without every existing caller of `describe_screen` needing to change.
|
||||
- Add configuration to enable/disable semantic enrichment globally (so it can be turned off entirely in environments without LLM access, e.g. tests or offline development) without touching `scene-perception`.
|
||||
- **BREAKING**: none. This is a purely additive layer; nothing in `scene-perception`, `agent-runtime`, or any other existing capability changes shape or behavior.
|
||||
|
||||
## Capabilities
|
||||
|
||||
### New Capabilities
|
||||
- `semantic-scene`: Builds a `SemanticScene` (page identity, supported intents, per-widget purpose labels) from an existing `Scene` via one additional LLM call, with a defined skip/fallback degrade path when that call is unavailable or fails, so enrichment is strictly additive and non-blocking to the Observe→Act loop.
|
||||
|
||||
### Modified Capabilities
|
||||
(none — `scene-perception`'s `Scene` format and requirements from `apex-agent-mvp` are read-only input to this change and are not modified; `agent-runtime`'s Planner/Executor loop shape is not changed by this proposal, only optionally consumed by it, see `design.md` for how a future milestone might wire it in.)
|
||||
|
||||
## Impact
|
||||
|
||||
- **New package**: `semantic/` — `models.py` (`SemanticScene`, `SemanticWidget` dataclasses), `llm_client.py` (narrow LLM client wrapper + typed result), `enricher.py` (`enrich_scene(scene, ...) -> SemanticScene | None`), `prompts.py` (the enrichment system prompt / JSON schema description).
|
||||
- **New tool**: `tools/describe_screen_semantic.py` (or equivalent), wrapping the existing `tools/describe_screen.py` output with an enrichment pass and degrade path, following the same `manager`/`device_id` kwarg pattern as other tools.
|
||||
- **Config**: `pyproject.toml` gains an `anthropic` SDK dependency and a `semantic*` entry in `[tool.setuptools.packages.find].include`; a new settings surface (e.g. env var or config value) to enable/disable enrichment and select the model.
|
||||
- **No change** to `core/models.py`'s `Scene`/`SceneElement`, `vision`/`perception`'s fusion pipeline, or any existing tool signatures — all existing callers of `describe_screen` keep working unmodified.
|
||||
- **Out of scope**: no persistent cross-step semantic state or caching across steps (Milestone 6, World Runtime); no skill synthesis from semantic labels (Milestone 7); no change to `skill-catalog-subscription` or `web-console`.
|
||||
@@ -0,0 +1,45 @@
|
||||
## ADDED Requirements
|
||||
|
||||
### Requirement: Semantic scene enrichment from an existing Scene
|
||||
The system SHALL provide a function that, given an existing `Scene` (as produced by the `scene-perception` capability), produces a `SemanticScene` consisting of a page identity string, a list of plain-language supported intents, and a list of per-widget purpose labels referencing the `Scene`'s element IDs, using exactly one LLM call.
|
||||
|
||||
#### Scenario: Enrichment succeeds for a recognizable screen
|
||||
- **WHEN** `enrich_scene()` is called with a `Scene` describing a recognizable app screen (e.g. a chat screen with a text input and a send button)
|
||||
- **THEN** it returns a `SemanticScene` with a non-empty `page` string, a non-empty `intents` list of plain-language strings, and a `widgets` list where each entry's `element_id` matches an element ID present in the input `Scene`
|
||||
|
||||
#### Scenario: Widget purpose labels reference only known elements
|
||||
- **WHEN** the LLM response includes a widget purpose label whose `element_id` does not match any element ID in the input `Scene`
|
||||
- **THEN** `enrich_scene()` discards that widget entry from the returned `SemanticScene` rather than propagating a dangling element reference
|
||||
|
||||
### Requirement: Enrichment failure degrades to no semantic scene, never blocks the loop
|
||||
The system SHALL treat any enrichment failure (LLM call timeout, connection error, rate limit, malformed or schema-invalid response, or enrichment disabled by configuration) as a non-fatal condition, returning an absence of a semantic scene rather than raising an exception, so that callers always have a defined fallback of using the raw `Scene`.
|
||||
|
||||
#### Scenario: LLM call times out
|
||||
- **WHEN** the enrichment LLM call does not complete within the configured timeout
|
||||
- **THEN** `enrich_scene()` returns `None` and the caller proceeds using the raw `Scene` without the task step being marked as failed
|
||||
|
||||
#### Scenario: LLM response fails schema validation
|
||||
- **WHEN** the enrichment LLM call returns a response that does not conform to the expected `SemanticScene` JSON schema
|
||||
- **THEN** `enrich_scene()` returns `None` instead of raising, and no partially-parsed `SemanticScene` is returned
|
||||
|
||||
#### Scenario: Enrichment disabled by configuration
|
||||
- **WHEN** semantic enrichment is disabled in configuration
|
||||
- **THEN** `enrich_scene()` returns `None` immediately without making an LLM call
|
||||
|
||||
### Requirement: Structured, schema-constrained LLM output
|
||||
The system SHALL request the enrichment LLM call using a schema-constrained structured-output mechanism so that any successful response is guaranteed to be valid JSON matching the `SemanticScene` shape, rather than relying on free-text parsing of an unconstrained model reply.
|
||||
|
||||
#### Scenario: Successful call yields directly parseable output
|
||||
- **WHEN** the enrichment LLM call completes successfully
|
||||
- **THEN** the raw response body is valid JSON matching the declared `SemanticScene` schema without requiring text extraction, regex matching, or a JSON-repair step
|
||||
|
||||
### Requirement: Semantic enrichment is opt-in and does not alter existing tool behavior
|
||||
The system SHALL expose semantic enrichment through a new, separate tool entry point rather than modifying the existing `describe_screen` tool's signature or behavior, so that every existing caller of `describe_screen` continues to receive only a `Scene`, unchanged, unless it explicitly opts into the new semantic-enriched entry point.
|
||||
|
||||
#### Scenario: Existing describe_screen callers are unaffected
|
||||
- **WHEN** an existing caller invokes the `describe_screen` tool as it did before this change
|
||||
- **THEN** it receives the same `Scene` result as before, with no semantic enrichment attempted and no new LLM-related dependency invoked
|
||||
|
||||
#### Scenario: A caller opts into semantic enrichment
|
||||
- **WHEN** a caller invokes the new semantic-enrichment tool entry point for a given device
|
||||
- **THEN** it receives both the underlying `Scene` and, when enrichment succeeds, the corresponding `SemanticScene`; when enrichment fails or is disabled, it receives the `Scene` with an explicit absence of a `SemanticScene` rather than a partial or error result
|
||||
@@ -0,0 +1,42 @@
|
||||
## 1. Package scaffolding
|
||||
|
||||
- [x] 1.1 Create the `semantic/` package (`__init__.py`, `models.py`, `llm_client.py`, `enricher.py`, `prompts.py`)
|
||||
- [x] 1.2 Add the `anthropic` SDK dependency and a `semantic*` entry to `[tool.setuptools.packages.find].include` in `pyproject.toml`
|
||||
- [x] 1.3 Add enrichment configuration: enabled/disabled flag (default disabled), model name, and timeout, sourced from environment/config in a single place `semantic/` reads from
|
||||
- [x] 1.4 Extend the project's smoke test (that imports every package) to import `semantic`
|
||||
|
||||
## 2. SemanticScene data model (capability: semantic-scene)
|
||||
|
||||
- [x] 2.1 Implement `semantic/models.py`: `SemanticWidget` (`element_id: str`, `purpose: str`) and `SemanticScene` (`page: str`, `intents: list[str]`, `widgets: list[SemanticWidget]`) dataclasses with `to_dict()`/`from_dict()` mirroring the style of `core/models.py`
|
||||
- [x] 2.2 Define the JSON schema for `SemanticScene` used to constrain the LLM's structured output, matching the `SemanticScene` dataclass shape exactly
|
||||
- [x] 2.3 Write unit tests for `SemanticScene`/`SemanticWidget` round-tripping through `to_dict()`/`from_dict()`
|
||||
|
||||
## 3. LLM client abstraction (capability: semantic-scene)
|
||||
|
||||
- [x] 3.1 Implement `semantic/llm_client.py`: a narrow client wrapper around the Anthropic SDK exposing a single `enrich(scene_json: dict, *, timeout: float) -> dict` method that issues one structured-output (`output_config.format` / schema-constrained) call and returns the parsed JSON body
|
||||
- [x] 3.2 Configure the enrichment system prompt/schema block with a `cache_control: {"type": "ephemeral"}` breakpoint so repeated per-step calls reuse the cached prefix
|
||||
- [x] 3.3 Map the SDK's typed exceptions (timeout, connection error, rate limit, authentication error, API status error) into a single internal `EnrichmentUnavailable` signal consumed only inside `semantic/` (never re-raised past `enricher.py`)
|
||||
- [x] 3.4 Make the client injectable/mockable (constructor parameter or factory function) so tests can supply a fake client with canned responses instead of calling the network
|
||||
- [x] 3.5 Write unit tests for the client wrapper against a fake transport covering: success, timeout, rate limit, malformed/schema-invalid JSON response
|
||||
|
||||
## 4. Enrichment pass and degrade path (capability: semantic-scene)
|
||||
|
||||
- [x] 4.1 Implement `semantic/prompts.py`: the enrichment system prompt describing the page-identity/intents/widget-purpose task and referencing the input `Scene`'s elements by ID
|
||||
- [x] 4.2 Implement `semantic/enricher.py`: `enrich_scene(scene: Scene, *, client=None) -> SemanticScene | None`, serializing the `Scene` to JSON, calling the LLM client, and parsing the result into a `SemanticScene`
|
||||
- [x] 4.3 Add post-response validation in `enricher.py`: drop any `widgets[]` entry whose `element_id` does not match an element ID present in the input `Scene`
|
||||
- [x] 4.4 Make `enrich_scene()` return `None` (never raise) for every expected failure mode: disabled config, `EnrichmentUnavailable` from the client, or a `SemanticScene` that fails post-response validation entirely
|
||||
- [x] 4.5 Write unit tests for `enrich_scene()` covering: successful enrichment, disabled-by-config short-circuit (no client call made), degrade-to-`None` on client failure, and dangling-`element_id` filtering
|
||||
- [x] 4.6 Write an integration test (skippable without network/API credentials, following the existing `apex-agent-mvp` skippable-integration-test pattern) that calls the real LLM client against a fixture `Scene` and asserts a schema-valid `SemanticScene` is returned
|
||||
|
||||
## 5. Tool integration (capability: semantic-scene)
|
||||
|
||||
- [x] 5.1 Implement `tools/describe_screen_semantic.py`: calls the existing `tools/describe_screen.py` for a `Scene`, then `enrich_scene()`, returning `{"scene": Scene, "semantic_scene": SemanticScene | None}`, following the same `device_id`/`manager`/`ocr_engine` kwarg pattern as `describe_screen`
|
||||
- [x] 5.2 Register the new tool under its own key (e.g. `"describe_screen_semantic"`) in `runtime/executor.py`'s `default_tool_registry()`, additive alongside the existing `"describe_screen"` entry
|
||||
- [x] 5.3 Write a unit test asserting `tools/describe_screen.py`'s existing behavior and return type are unchanged after this change (no accidental coupling to `semantic/`)
|
||||
- [x] 5.4 Write a unit test for `describe_screen_semantic` covering both the enrichment-succeeds and enrichment-degrades-to-`None` cases, using a fake/mock LLM client
|
||||
|
||||
## 6. End-to-end validation
|
||||
|
||||
- [x] 6.1 Write an end-to-end test simulating an Observe step that calls `describe_screen_semantic` against a mocked `Driver`/`Scene` and a mocked LLM client, asserting the task loop completes normally whether enrichment succeeds or is forced to fail
|
||||
- [x] 6.2 Confirm enrichment stays disabled by default after applying this change (no existing task's behavior, latency, or cost changes unless a caller explicitly enables it and opts into the new tool)
|
||||
- [x] 6.3 Run the full test suite (`pytest`) and confirm no existing test in `tests/` needed a behavior change, only additive new tests
|
||||
Reference in New Issue
Block a user